From patchwork Wed Oct 30 15:12:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856676 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E033BD5CCB3 for ; Wed, 30 Oct 2024 15:13:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74E618D000A; Wed, 30 Oct 2024 11:13:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D54C8D0001; Wed, 30 Oct 2024 11:13:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 576708D000A; Wed, 30 Oct 2024 11:13:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 385448D0001 for ; Wed, 30 Oct 2024 11:13:32 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DCEC8120DDE for ; Wed, 30 Oct 2024 15:13:31 +0000 (UTC) X-FDA: 82730612412.29.600AE90 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf14.hostedemail.com (Postfix) with ESMTP id 49BF2100013 for ; Wed, 30 Oct 2024 15:13:01 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Ku8sigQY; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301079; a=rsa-sha256; cv=none; b=Xa7qhGlQu3e9cHxwnEkGAUSRIL2PE1LXo6KxVmqQDrFuDladSQbJ2RS2I24rKTAlphGv0g 0Z7zy0eTbkAd4P8KEJDKCiUqqnoK98gz63EO/VscOUEsjlZPgpypPBMRlLXugQMgQb4NWK OlIJtaJfDGCk/3pm5lY5VkTj4yRgtSo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Ku8sigQY; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301079; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bYQwrnJsq9OyAooRQxM8WiYuZnKdkfEZ0zRjUdzZmmo=; b=M++Efw5EPKTJZVZiU5Ix9gxciccf6BNmh5VRBR4G/ro85LUHu5zp5HwOcbznw7MIo+PcXW fLukhqCSpnMSo2cc0/BO/tc6D0NclQR1FPRm01ymcWArkpllBISczvPQQYb1ftT6OoNCbo AjqsrRycHDfqHpbRStpGj81zj9b6OiI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 722F25C5EC5; Wed, 30 Oct 2024 15:12:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 619F5C4CED1; Wed, 30 Oct 2024 15:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301209; bh=8wWhFd8jls3C/GXbekE7tJOMVMm116FEo4y9nYLUxtg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ku8sigQYl0/wvrz0IzAhk2bYpPZy+BolOqT2XILLlTkA7oGxNODEVz2Ij1N8q0WCn +zvlM8bnauIDnwvQElHubOqET9DG2u6c1jbBBDOkFdNc7c+w4w21C1gZGLZouD/IWm L+rX4/1ONfVf9lnvvfjPpwEpQ+MmyA6GSqrGaYKlj0Qhn7+m26fXQIwwGhJTkXI8hl +0pDEwzgy9Hw9qYTfU3n6htTvGsTUajvvT6MNjzQLpYqxEgWnl+AVBCTzz4kSJVB6k XJWb6GAQymYKsudThNnsZG5IIFibD8QZu2yOElMQ5L9cfExUopefN3JNYfAgxNKf3t MHyhP7QJWY4gw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 01/17] PCI/P2PDMA: Refactor the p2pdma mapping helpers Date: Wed, 30 Oct 2024 17:12:47 +0200 Message-ID: <0ac8fac1e98365a31093dd90d549da1f6b00977d.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 49BF2100013 X-Stat-Signature: jmishkh7onzs3hj6kaq65jptmnagzeik X-Rspam-User: X-HE-Tag: 1730301181-494127 X-HE-Meta: U2FsdGVkX18VCmfRMpQPNhpNyTuSiV3msjPazGO3AjY89QwpC+YuLp9FYV37e/1tu8EkGkK4F9Y1jV4jFJ2mwh+aQJzT+n4TuT7TVozxCFxOVWV4z8ObOREFvFSs+a1RPxEHHlu6xLgt9lqx6Z8789WScJklbuSzxCxFVSqSSG5Zz6zSuAgoou0XyrSy8qgNqXoA52i0FiPSEJBMiOCtxZpw1o3ISvM05pXoCDz0YsA93tPM3QdCTUST25H1S9sqjRMeEFWzBf58boYB5h4S9IURr6Emdt9rgJG6O+aZgQaRjZU2cxN7MXyvg+zLb35FbPV2r+ucFNBftopViF2z6EZyJcxOLL33r1OrzG4xQY4XLqPLNT1Nth4B0IMMdyEfaOL0wMg7Qc2ZSndqDgP315EjoXaePrtwnBfY7YH0XGJuBTWQWpipwfgZ0DsFtlehBzHaKypQLoe/hGZ5lCp0BzwbAFsBDtE8NwTs/iThaoNTbZFCgBodVj2aYC+e9I28MMOHTJNLq4Fq5zp4E/d0rVNVetmCPdTJX5cp0XyUoEEi4gVBUsyZJvhgoKXYRsWLcxZGe/nYMsvRWRiBw8D1dEpaCK6SRGD9PhBrisE6qrPcVkZEKxpcwrUCJndhgo63kq5u8jBoRAKl5XARijb3wbJv6Fl8akuDXfllK9NK9X1yEmW4ajF/xyMssPFr8lfCQS8SthxDh0/qNVuPC2nDjmk7TSuv8hIzIlk/dsgrcvyGI+ELGvqxsVSi8IEPT2ePET+Dom5sKaqf79l/HzEI7ZsEL0D4M7MQcojmddeMNFAg9IJn4hUyGRiinBp8few6i2RE2VmYjqKmvGYKiC4rawnUhkw3VW7XYADoUW41P6GRLeWg0LdQnkEPY8qVHTAcgEQsgwa1lqGEpmuap5spKdTAfzUNEeSWu04OsSkLYzIms9nR1G6+uNiE2FD65/KQrP41u9T7KBrjO9zmxIf jQotlg8f LR4vwux806huAXwUfbYOuyT9e/5bLbyZI/hAEu42re58sLsQaa+E2/FXN3R5SsMLJNd8vmjzsgN8O6O02kjBGh60Wj921J1dNI9lkILkzHqEXJzzh208if8YHdAJgk5DkTh3+y2fxVUoeDdooi+DGh/SfowNN9Rg17T1ZmBgoJ3GXJ7fKqb7PQagVi/BKSOcy1RUP3GMX3vmHpnoRdl4SK5WDnP+ZJP5Ver1BzIqKNiWUxRU9Jfwpqegub00QExbtZVxSWu+xCeXvVo+rvyaOeYF9c47nFPYW2M2bb2goUO+BovIHEPre+rnRcTOGeYSukcSG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig The current scheme with a single helper to determine the P2P status and map a scatterlist segment force users to always use the map_sg helper to DMA map, which we're trying to get away from because they are very cache inefficient. Refactor the code so that there is a single helper that checks the P2P state for a page, including the result that it is not a P2P page to simplify the callers, and a second one to perform the address translation for a bus mapped P2P transfer that does not depend on the scatterlist structure. Signed-off-by: Christoph Hellwig Reviewed-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 47 +++++++++++++++++----------------- drivers/pci/p2pdma.c | 38 ++++----------------------- include/linux/dma-map-ops.h | 51 +++++++++++++++++++++++++++++-------- kernel/dma/direct.c | 43 +++++++++++++++---------------- 4 files changed, 91 insertions(+), 88 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2a9fa0c8cc00..5746ffaf0061 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1382,7 +1382,6 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, struct scatterlist *s, *prev = NULL; int prot = dma_info_to_prot(dir, dev_is_dma_coherent(dev), attrs); struct pci_p2pdma_map_state p2pdma_state = {}; - enum pci_p2pdma_map_type map; dma_addr_t iova; size_t iova_len = 0; unsigned long mask = dma_get_seg_boundary(dev); @@ -1412,28 +1411,30 @@ int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, int nents, size_t s_length = s->length; size_t pad_len = (mask - iova_len + 1) & mask; - if (is_pci_p2pdma_page(sg_page(s))) { - map = pci_p2pdma_map_segment(&p2pdma_state, dev, s); - switch (map) { - case PCI_P2PDMA_MAP_BUS_ADDR: - /* - * iommu_map_sg() will skip this segment as - * it is marked as a bus address, - * __finalise_sg() will copy the dma address - * into the output segment. - */ - continue; - case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: - /* - * Mapping through host bridge should be - * mapped with regular IOVAs, thus we - * do nothing here and continue below. - */ - break; - default: - ret = -EREMOTEIO; - goto out_restore_sg; - } + switch (pci_p2pdma_state(&p2pdma_state, dev, sg_page(s))) { + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Mapping through host bridge should be mapped with + * regular IOVAs, thus we do nothing here and continue + * below. + */ + break; + case PCI_P2PDMA_MAP_NONE: + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + /* + * iommu_map_sg() will skip this segment as it is marked + * as a bus address, __finalise_sg() will copy the dma + * address into the output segment. + */ + s->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state, + sg_phys(s)); + sg_dma_len(s) = sg->length; + sg_dma_mark_bus_address(s); + continue; + default: + ret = -EREMOTEIO; + goto out_restore_sg; } sg_dma_address(s) = s_iova_off; diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 4f47a13cb500..f38d16d71dd5 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -995,40 +995,12 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, return type; } -/** - * pci_p2pdma_map_segment - map an sg segment determining the mapping type - * @state: State structure that should be declared outside of the for_each_sg() - * loop and initialized to zero. - * @dev: DMA device that's doing the mapping operation - * @sg: scatterlist segment to map - * - * This is a helper to be used by non-IOMMU dma_map_sg() implementations where - * the sg segment is the same for the page_link and the dma_address. - * - * Attempt to map a single segment in an SGL with the PCI bus address. - * The segment must point to a PCI P2PDMA page and thus must be - * wrapped in a is_pci_p2pdma_page(sg_page(sg)) check. - * - * Returns the type of mapping used and maps the page if the type is - * PCI_P2PDMA_MAP_BUS_ADDR. - */ -enum pci_p2pdma_map_type -pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, - struct scatterlist *sg) +void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, + struct device *dev, struct page *page) { - if (state->pgmap != sg_page(sg)->pgmap) { - state->pgmap = sg_page(sg)->pgmap; - state->map = pci_p2pdma_map_type(state->pgmap, dev); - state->bus_off = to_p2p_pgmap(state->pgmap)->bus_offset; - } - - if (state->map == PCI_P2PDMA_MAP_BUS_ADDR) { - sg->dma_address = sg_phys(sg) + state->bus_off; - sg_dma_len(sg) = sg->length; - sg_dma_mark_bus_address(sg); - } - - return state->map; + state->pgmap = page->pgmap; + state->map = pci_p2pdma_map_type(state->pgmap, dev); + state->bus_off = to_p2p_pgmap(state->pgmap)->bus_offset; } /** diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index b7773201414c..3480a28d1b9f 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -443,6 +443,11 @@ enum pci_p2pdma_map_type { */ PCI_P2PDMA_MAP_UNKNOWN = 0, + /* + * Not a PCI P2PDMA transfer. + */ + PCI_P2PDMA_MAP_NONE, + /* * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will * traverse the host bridge and the host bridge is not in the @@ -471,21 +476,47 @@ enum pci_p2pdma_map_type { struct pci_p2pdma_map_state { struct dev_pagemap *pgmap; - int map; + enum pci_p2pdma_map_type map; u64 bus_off; }; -#ifdef CONFIG_PCI_P2PDMA -enum pci_p2pdma_map_type -pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, - struct scatterlist *sg); -#else /* CONFIG_PCI_P2PDMA */ +/* helper for pci_p2pdma_state(), do not use directly */ +void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, + struct device *dev, struct page *page); + +/** + * pci_p2pdma_state - check the P2P transfer state of a page + * @state: P2P state structure + * @dev: device to transfer to/from + * @page: page to map + * + * Check if @page is a PCI P2PDMA page, and if yes of what kind. Returns the + * map type, and updates @state with all information needed for a P2P transfer. + */ static inline enum pci_p2pdma_map_type -pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, - struct scatterlist *sg) +pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, + struct page *page) +{ + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { + if (state->pgmap != page->pgmap) + __pci_p2pdma_update_state(state, dev, page); + return state->map; + } + return PCI_P2PDMA_MAP_NONE; +} + +/** + * pci_p2pdma_bus_addr_map - map a PCI_P2PDMA_MAP_BUS_ADDR P2P transfer + * @state: P2P state structure + * @paddr: physical address to map + * + * Map a physically contigous PCI_P2PDMA_MAP_BUS_ADDR transfer. + */ +static inline dma_addr_t +pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) { - return PCI_P2PDMA_MAP_NOT_SUPPORTED; + WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); + return paddr + state->bus_off; } -#endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_DMA_MAP_OPS_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 5b4e6d3bf7bc..e289ad27d1b5 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -462,34 +462,33 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { struct pci_p2pdma_map_state p2pdma_state = {}; - enum pci_p2pdma_map_type map; struct scatterlist *sg; int i, ret; for_each_sg(sgl, sg, nents, i) { - if (is_pci_p2pdma_page(sg_page(sg))) { - map = pci_p2pdma_map_segment(&p2pdma_state, dev, sg); - switch (map) { - case PCI_P2PDMA_MAP_BUS_ADDR: - continue; - case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: - /* - * Any P2P mapping that traverses the PCI - * host bridge must be mapped with CPU physical - * address and not PCI bus addresses. This is - * done with dma_direct_map_page() below. - */ - break; - default: - ret = -EREMOTEIO; + switch (pci_p2pdma_state(&p2pdma_state, dev, sg_page(sg))) { + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Any P2P mapping that traverses the PCI host bridge + * must be mapped with CPU physical address and not PCI + * bus addresses. + */ + break; + case PCI_P2PDMA_MAP_NONE: + sg->dma_address = dma_direct_map_page(dev, sg_page(sg), + sg->offset, sg->length, dir, attrs); + if (sg->dma_address == DMA_MAPPING_ERROR) { + ret = -EIO; goto out_unmap; } - } - - sg->dma_address = dma_direct_map_page(dev, sg_page(sg), - sg->offset, sg->length, dir, attrs); - if (sg->dma_address == DMA_MAPPING_ERROR) { - ret = -EIO; + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + sg->dma_address = pci_p2pdma_bus_addr_map(&p2pdma_state, + sg_phys(sg)); + sg_dma_mark_bus_address(sg); + continue; + default: + ret = -EREMOTEIO; goto out_unmap; } sg_dma_len(sg) = sg->length; From patchwork Wed Oct 30 15:12:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856675 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 616FAD5CCB8 for ; Wed, 30 Oct 2024 15:13:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E52678D0008; Wed, 30 Oct 2024 11:13:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DFFB28D0001; Wed, 30 Oct 2024 11:13:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C79208D0008; Wed, 30 Oct 2024 11:13:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A786D8D0001 for ; Wed, 30 Oct 2024 11:13:27 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2C9CEC0C3C for ; Wed, 30 Oct 2024 15:13:27 +0000 (UTC) X-FDA: 82730611488.25.2EB62F1 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf27.hostedemail.com (Postfix) with ESMTP id 2C75140008 for ; Wed, 30 Oct 2024 15:12:58 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=YYhKznON; spf=pass (imf27.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301125; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tBbrLHZhmCyqx90f2SbTThQY35eLeHqz2R0i9v7lpY0=; b=5F9t6ys3mztGJW6Vfpwpqz3XZFE+IKr1KMa1kGttXHjroYQcOXMCnbmvGoD5qDB3DqtEC6 UN8JLnicfaQ6+RQMWekeMQBDz/KQwaIg6PxvBvyn07o1qYIvzpyt3ECf5Gm+enmKlUVnPE 3KlaZGParX6zv6uRTd1WsG2S+OXu3P4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=YYhKznON; spf=pass (imf27.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301125; a=rsa-sha256; cv=none; b=BYIM9Etsv+UORtW/KP5OXrsTlny+bffKMViRqjO4sVhGY5kS35k+Er6rrAQrghzrAcAMlG 1JU6tWAzgotn/e1pKvWz7xXrR7pWjbaN9jUt3UXUNyb729py8nlgasbpQ2mCMG/yln9N1r iFFEQiMLqdJgg+KJKEkyfe3DFi4uco0= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id BD9825C5EA0; Wed, 30 Oct 2024 15:12:39 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BBB8EC4CECE; Wed, 30 Oct 2024 15:13:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301204; bh=wXz+z/X6ORsWWV4O6ZDdJLW6LBMrcitLsABF/mODEU8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YYhKznONnn9rqT63pSQgOqKTzzDJQyGAzTW3YL9iAPvAodYhF3J7QBs/B8Igm6tJ/ fbm86QN4597cF4c/zqMr8QKjMyT2Leaaob3eYYGvCVNsuPu7mUcpMXi7yPjg3b6W9d SRu6oAqaqWSKqW26bZECmPX0foH1N6k0LCIp8zf7jZaaWyUituxPsW4FoTnuyysJqr IVScxEOUPuV4XauuR+4Rz9knf8jyW1IHy38BraAP1r2OEsYRjKf7jNDmUg+B/6AVgi rJqfY19/qrlG39cKOxB7an7iW+JMCl4EWuowRdwUCMqPI0FlfFY1CODEVSAnRiVs5Z oEDgBk2KuwZ8w== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 02/17] dma-mapping: move the PCI P2PDMA mapping helpers to pci-p2pdma.h Date: Wed, 30 Oct 2024 17:12:48 +0200 Message-ID: X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 2C75140008 X-Stat-Signature: q8x3r5tmaxjmgfruqcjketowiz44w38y X-HE-Tag: 1730301178-781602 X-HE-Meta: U2FsdGVkX1/a9yp5ZkIP3f/iMepnEhyMxmrlq6cwxw0aZ3jPzI25ceSSbnofk1E3ZDleJbSQEszLsBFJhKegzgxDYK7/CcsOhskkKa6yFvip09rjS1nq2OJ2ZUQUbu09PFQ64ONiIrUQu0C1vZ94t9O5dqof/znEJFbslIgnzyTKoB3A+nm5nOJ8RV5EtEQ7KHe5xAhLVmN4sJSVxlYPdizmFdXPTnnnoqtG26YCRf1iKXmkZCIkwQ3O7ml7htJq0NqxqusYXZdlTQemFSnlhRRnMtwOvyGhTJFDdVyVQKQWUJc471UWle1SE+CFKOOQjfvCpFoahdv9ewwLGIElV+sKtnpD8g4XiJh4ItTStVKFa3Lf/m6Up9FITdFllu/t8uSyuu3Ufaby13ifzo7kkZS4yiCT+okZy6xOQXEpcmckUdqW/2jOrQITr35FkMLZwcxiiJ8IYJbfmWWhFDtBEZroLCboZR5Jjv8M28gnaO6Tb/khpDbz9cbw95826/AFQbw4v5FFEOni0PqnCZc1CnFsg/s9lsLD9E7hp1Hs/towGU8+NYmcwYVvK/sr17CCpiVjFvm5h1+zV3bKjG4k/I6Ia5Solaza/8eFriZCXwF+mxl4hl7r2GZ3fPFPOFvlDhAAPwoWt5/2N9n2Jdq9dZouIrawKKW+uAJ58/W+B5mO0y6xIs30JlZ8aDBFRJQfv0ieupg+BYTCIGuPvBuUHKYweM0Ck5rrHlEweQD0dIqTwjXR/f42wIBw0xlkLo1IRXouEPS4DcuCk2Fy4XrVTqVuOk0Uf74v71pA+VAKO4KOdhgF81+XGpiljM3bw+vf+1067HslYUI6cl80EIWB+iDdYEtvkblMxcjnH/0gzmBqlPvPbxih1j7p9KqwHRaI6l2A3zav9Bv17BYGgfLfmylSqZy6uSRKY8rf58Six/7g1+WEB1qsJ1sUcR0m632/6rYS7XSiUofTPzZf+eo fqTHfNpY hhAPzz7eoZ6rT/j8gROWOX6SVsshLYRm+pyUPPoo9c8zVktsB+sTDRX0M+5/VOmSOCtRtnwf2C9gruCFpQctmVmkXJL8TeBDHmY2e3ufU9oRcMQ9GKo3iPvxsIj2YS+e0zgDY7shxsue3+DqPki6c738AgR64kWzE+HyaVb7eStGm28YR4fC1Nh2ZTmJLecoHKHwU9g1wge7OxyItNqBxsWZijw2zXsKKB9OiMsfRReTd9T/E8TPn24IVu1pgSqYM/wS9KdB5pmOU+CkP/71MHtZv1EIge9YUjx+H+sBG3+qd99luoFin2Nwgd9/rJM3jGjyl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig To support the upcoming non-scatterlist mapping helpers, we need to go back to have them called outside of the DMA API. Thus move them out of dma-map-ops.h, which is only for DMA API implementations to pci-p2pdma.h, which is for driver use. Note that the core helper is still not exported as the mapping is expected to be done only by very highlevel subsystem code at least for now. Signed-off-by: Christoph Hellwig Reviewed-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 1 + include/linux/dma-map-ops.h | 85 ------------------------------------- include/linux/pci-p2pdma.h | 84 ++++++++++++++++++++++++++++++++++++ kernel/dma/direct.c | 1 + 4 files changed, 86 insertions(+), 85 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 5746ffaf0061..853247c42f7d 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include #include diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 3480a28d1b9f..dced37816ede 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -434,89 +434,4 @@ static inline void debug_dma_dump_mappings(struct device *dev) #endif /* CONFIG_DMA_API_DEBUG */ extern const struct dma_map_ops dma_dummy_ops; - -enum pci_p2pdma_map_type { - /* - * PCI_P2PDMA_MAP_UNKNOWN: Used internally for indicating the mapping - * type hasn't been calculated yet. Functions that return this enum - * never return this value. - */ - PCI_P2PDMA_MAP_UNKNOWN = 0, - - /* - * Not a PCI P2PDMA transfer. - */ - PCI_P2PDMA_MAP_NONE, - - /* - * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will - * traverse the host bridge and the host bridge is not in the - * allowlist. DMA Mapping routines should return an error when - * this is returned. - */ - PCI_P2PDMA_MAP_NOT_SUPPORTED, - - /* - * PCI_P2PDMA_BUS_ADDR: Indicates that two devices can talk to - * each other directly through a PCI switch and the transaction will - * not traverse the host bridge. Such a mapping should program - * the DMA engine with PCI bus addresses. - */ - PCI_P2PDMA_MAP_BUS_ADDR, - - /* - * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk - * to each other, but the transaction traverses a host bridge on the - * allowlist. In this case, a normal mapping either with CPU physical - * addresses (in the case of dma-direct) or IOVA addresses (in the - * case of IOMMUs) should be used to program the DMA engine. - */ - PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, -}; - -struct pci_p2pdma_map_state { - struct dev_pagemap *pgmap; - enum pci_p2pdma_map_type map; - u64 bus_off; -}; - -/* helper for pci_p2pdma_state(), do not use directly */ -void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, - struct device *dev, struct page *page); - -/** - * pci_p2pdma_state - check the P2P transfer state of a page - * @state: P2P state structure - * @dev: device to transfer to/from - * @page: page to map - * - * Check if @page is a PCI P2PDMA page, and if yes of what kind. Returns the - * map type, and updates @state with all information needed for a P2P transfer. - */ -static inline enum pci_p2pdma_map_type -pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, - struct page *page) -{ - if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { - if (state->pgmap != page->pgmap) - __pci_p2pdma_update_state(state, dev, page); - return state->map; - } - return PCI_P2PDMA_MAP_NONE; -} - -/** - * pci_p2pdma_bus_addr_map - map a PCI_P2PDMA_MAP_BUS_ADDR P2P transfer - * @state: P2P state structure - * @paddr: physical address to map - * - * Map a physically contigous PCI_P2PDMA_MAP_BUS_ADDR transfer. - */ -static inline dma_addr_t -pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) -{ - WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); - return paddr + state->bus_off; -} - #endif /* _LINUX_DMA_MAP_OPS_H */ diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h index 2c07aa6b7665..e839f52b512b 100644 --- a/include/linux/pci-p2pdma.h +++ b/include/linux/pci-p2pdma.h @@ -104,4 +104,88 @@ static inline struct pci_dev *pci_p2pmem_find(struct device *client) return pci_p2pmem_find_many(&client, 1); } +enum pci_p2pdma_map_type { + /* + * PCI_P2PDMA_MAP_UNKNOWN: Used internally for indicating the mapping + * type hasn't been calculated yet. Functions that return this enum + * never return this value. + */ + PCI_P2PDMA_MAP_UNKNOWN = 0, + + /* + * Not a PCI P2PDMA transfer. + */ + PCI_P2PDMA_MAP_NONE, + + /* + * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will + * traverse the host bridge and the host bridge is not in the + * allowlist. DMA Mapping routines should return an error when + * this is returned. + */ + PCI_P2PDMA_MAP_NOT_SUPPORTED, + + /* + * PCI_P2PDMA_BUS_ADDR: Indicates that two devices can talk to + * each other directly through a PCI switch and the transaction will + * not traverse the host bridge. Such a mapping should program + * the DMA engine with PCI bus addresses. + */ + PCI_P2PDMA_MAP_BUS_ADDR, + + /* + * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk + * to each other, but the transaction traverses a host bridge on the + * allowlist. In this case, a normal mapping either with CPU physical + * addresses (in the case of dma-direct) or IOVA addresses (in the + * case of IOMMUs) should be used to program the DMA engine. + */ + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, +}; + +struct pci_p2pdma_map_state { + struct dev_pagemap *pgmap; + enum pci_p2pdma_map_type map; + u64 bus_off; +}; + +/* helper for pci_p2pdma_state(), do not use directly */ +void __pci_p2pdma_update_state(struct pci_p2pdma_map_state *state, + struct device *dev, struct page *page); + +/** + * pci_p2pdma_state - check the P2P transfer state of a page + * @state: P2P state structure + * @dev: device to transfer to/from + * @page: page to map + * + * Check if @page is a PCI P2PDMA page, and if yes of what kind. Returns the + * map type, and updates @state with all information needed for a P2P transfer. + */ +static inline enum pci_p2pdma_map_type +pci_p2pdma_state(struct pci_p2pdma_map_state *state, struct device *dev, + struct page *page) +{ + if (IS_ENABLED(CONFIG_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { + if (state->pgmap != page->pgmap) + __pci_p2pdma_update_state(state, dev, page); + return state->map; + } + return PCI_P2PDMA_MAP_NONE; +} + +/** + * pci_p2pdma_bus_addr_map - map a PCI_P2PDMA_MAP_BUS_ADDR P2P transfer + * @state: P2P state structure + * @paddr: physical address to map + * + * Map a physically contigous PCI_P2PDMA_MAP_BUS_ADDR transfer. + */ +static inline dma_addr_t +pci_p2pdma_bus_addr_map(struct pci_p2pdma_map_state *state, phys_addr_t paddr) +{ + WARN_ON_ONCE(state->map != PCI_P2PDMA_MAP_BUS_ADDR); + return paddr + state->bus_off; +} + #endif /* _LINUX_PCI_P2P_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index e289ad27d1b5..c9b3893257d4 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "direct.h" /* From patchwork Wed Oct 30 15:12:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856682 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AF35D5CCB3 for ; Wed, 30 Oct 2024 15:13:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCCCF8E0006; Wed, 30 Oct 2024 11:13:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C7F8B8E0001; Wed, 30 Oct 2024 11:13:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF2F98E0006; Wed, 30 Oct 2024 11:13:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 916638E0001 for ; Wed, 30 Oct 2024 11:13:53 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 38E261A0B29 for ; Wed, 30 Oct 2024 15:13:53 +0000 (UTC) X-FDA: 82730612748.30.94A90BC Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf03.hostedemail.com (Postfix) with ESMTP id 7C1722002A for ; Wed, 30 Oct 2024 15:13:39 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dUjbzlfB; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf03.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301055; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v5SYRMY/doGJo/E57q9NuO+kOt01Crkg32kQAVBzJCY=; b=zcshUttlGEHNT6WBmFxf/HhW3LCxbZKzwTktkhzemmoyP7Z4KijIdU77kJVskUK6dGhenG 5MEyGqvPKZFV05k/qGYJrgm/mm+O9iYRdMRTh4yL2nMnIi/QFThHcCune6i9Ryk14nX8mM c9dvLis0yngCzsA58tmPGFDITiDRihE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301055; a=rsa-sha256; cv=none; b=e+MMXIfU0iagn68dFNvcU9NfoGoQHQNPkDlCZHjhh4L3YdSfhPMcbzEKIpGoNbn4bE/KEu E0zGLx/kqBpUTOenbe1v4llPxksrcvJvwpPmAFJSa0Y1yuQOR1fEv5ZbNsoU327R1KtcP8 +VwK7xGRDFjH5L49T2EKbevEbfu/hCE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dUjbzlfB; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf03.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id B70355C5E7F; Wed, 30 Oct 2024 15:13:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 864EFC4CECF; Wed, 30 Oct 2024 15:13:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301230; bh=RhcK3jpx80P74UKqRSLHZKtWTOJGp14TVgQFOHpK5uo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dUjbzlfB58nam+e/KuKFD86cA5mG8ygjLY9xQ0Z+h8sT3ni+00nHbhRibOA92d2hi 8xSFaFNOhS1hi7n1y1TP/+mgcST+mtaxk4HgBka7GNtvh7Mt9RzEEVIg91E8eZ4pX2 A/wD3SvwthfhxnGgkwocL11q550gsMwYXoTxqLkRkR/bcMUsXgtpEBnrlQ5MuVmOam 3KHo6w/cSCf3BLB2YJgJUjec/9ITBowgjGgsaPk5GvKbINh9JofcCiGOH7ZipyG9gL ZdHwhS8PLH6oDNV6SpMo+XFB1UPff4GK4X5kD5OYHis9J0N5UQL1Uyin1m7/ZYvKdf yHA0I8oajcNSg== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 03/17] iommu: generalize the batched sync after map interface Date: Wed, 30 Oct 2024 17:12:49 +0200 Message-ID: <589adb3a4b53121942e9a39051ae49a27f7a074c.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7C1722002A X-Stat-Signature: 7tux54t6rxqb4q3ktbidhgokb54t86hn X-Rspam-User: X-HE-Tag: 1730301219-826407 X-HE-Meta: U2FsdGVkX1+LXsxql6MFLmtTWFG6QzIX9KAPYVmkwZ45c8Pu8pa+q8bEcOGWHX5QJM7+qpz53B4/GSkqbPs/ede8fejgPnrDt7pDedfdEIluxgwybTscVr2V17WWfLx1Rieu9qGhBxBg17S6HMSuKCC3i6yPDNLm5LG3v0MuBbU1T+8uVK6YK2er3NQ2pYcIfiTV2qWCEblM6ELj/YjVVrOUntnXg7GCs+x9Htx088zjI20koi7ide8r/A1+5H2ozTd373GdF3giKxjR4j9GKCCLS8gXxGthu54kRLNIAZxezQOQw8UvBM/EYt/3RASLCYqwBbGg46xaWg7rRD4MV+eY4N85ob+Rx5MvOV+88jHv1v9Fc2Bf6Up95an5MVAk0F3YhQNWUQ8egka8IWkD4tgbjsJZKhneB0aDM7PBZFTZ2rfw1DyXe4rvnSDoHqjaLDWeex+FBT1N+WUM62Dp8vHDznXwzFF4ui1QmWL5shRNx8jYKeXsk+PC/8UTkbdfDfyJaeDhlvXIidUVRGpglb9sArrBWgjyKCCh3kxRHTIimxyzLVLHQpPI0/7VQIszcfG0RutvPBX9dMi4hXhNWeYhKBv5RHW/kHoUrPWgPVMZwcRyokMrueIHKylM74dgwzwkqv5b5GRc751aUyNh9S//mAqM8y9z5my4Ty9zpiOcFmkLpw4GIt2MwNrtBQmjHNPm3tNmhrHcBjim4Opdp8dZVvbP8/O4HVE7ywWQoF8hCirwYwzj56A8fudQWx5WcRpGKgadQ3+/wRAchpX8eg+qsuBeBvsYuRXZAh11OH5NIa8K/mu8bHVdGeaq7pNGnohQDtZArD4QLJpYmwfX0P5bkmEe+3hDr50e0xCrv6fVbLGJjTCsmsHOJeL716a8d45Mp24q04iK1DTPagH1+XToNlZBGlt3ik5mPhGHEd/VyGm8vgM0sh0hsITsw/6VDRMlzxXDzux73O2R+dY rsKUN4tk MCquWcTfkuNftoJLszAnT7XkTljngcRgy8b/EDzGjIeoU8bBw9+EdJShjUC7lfr913gUfqQpWQZuTib6mL6n4n8tSCZIDzOWPkCqFcjzvn7D/4C+wFMFNEz86ZNJQGiaEq+uv7HQf3n8cw9iE+zBXz3UBuLykD1PJ13GZN/LGhSlGBrqOqlSWB7+tIHrDFKKwePmahmWTOK4N7nXE8I/AN0v6P+dQE+sojth8tOL7Dn8oZ+4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig For the upcoming IOVA-based DMA API we want to use the interface batch the sync after mapping multiple entries from dma-iommu without having a scatterlist. For that move more sanity checks from the callers into __iommu_map and make that function available outside of iommu.c as iommu_map_nosync. Add a wrapper for the map_sync as iommu_sync_map so that callers don't need to poke into the methods directly. Signed-off-by: Christoph Hellwig Signed-off-by: Leon Romanovsky --- drivers/iommu/iommu.c | 65 +++++++++++++++++++------------------------ include/linux/iommu.h | 4 +++ 2 files changed, 33 insertions(+), 36 deletions(-) diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 83c8e617a2c5..6b0943397e1e 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -2439,8 +2439,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, unsigned long iova, return pgsize; } -static int __iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_domain_ops *ops = domain->ops; unsigned long orig_iova = iova; @@ -2449,12 +2449,19 @@ static int __iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t orig_paddr = paddr; int ret = 0; + might_sleep_if(gfpflags_allow_blocking(gfp)); + if (unlikely(!(domain->type & __IOMMU_DOMAIN_PAGING))) return -EINVAL; if (WARN_ON(!ops->map_pages || domain->pgsize_bitmap == 0UL)) return -ENODEV; + /* Discourage passing strange GFP flags */ + if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 | + __GFP_HIGHMEM))) + return -EINVAL; + /* find out the minimum page size supported */ min_pagesz = 1 << __ffs(domain->pgsize_bitmap); @@ -2502,31 +2509,27 @@ static int __iommu_map(struct iommu_domain *domain, unsigned long iova, return ret; } -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +int iommu_sync_map(struct iommu_domain *domain, unsigned long iova, size_t size) { const struct iommu_domain_ops *ops = domain->ops; - int ret; - - might_sleep_if(gfpflags_allow_blocking(gfp)); - /* Discourage passing strange GFP flags */ - if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 | - __GFP_HIGHMEM))) - return -EINVAL; + if (!ops->iotlb_sync_map) + return 0; + return ops->iotlb_sync_map(domain, iova, size); +} - ret = __iommu_map(domain, iova, paddr, size, prot, gfp); - if (ret == 0 && ops->iotlb_sync_map) { - ret = ops->iotlb_sync_map(domain, iova, size); - if (ret) - goto out_err; - } +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) +{ + int ret; - return ret; + ret = iommu_map_nosync(domain, iova, paddr, size, prot, gfp); + if (ret) + return ret; -out_err: - /* undo mappings already done */ - iommu_unmap(domain, iova, size); + ret = iommu_sync_map(domain, iova, size); + if (ret) + iommu_unmap(domain, iova, size); return ret; } @@ -2612,26 +2615,17 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, struct scatterlist *sg, unsigned int nents, int prot, gfp_t gfp) { - const struct iommu_domain_ops *ops = domain->ops; size_t len = 0, mapped = 0; phys_addr_t start; unsigned int i = 0; int ret; - might_sleep_if(gfpflags_allow_blocking(gfp)); - - /* Discourage passing strange GFP flags */ - if (WARN_ON_ONCE(gfp & (__GFP_COMP | __GFP_DMA | __GFP_DMA32 | - __GFP_HIGHMEM))) - return -EINVAL; - while (i <= nents) { phys_addr_t s_phys = sg_phys(sg); if (len && s_phys != start + len) { - ret = __iommu_map(domain, iova + mapped, start, + ret = iommu_map_nosync(domain, iova + mapped, start, len, prot, gfp); - if (ret) goto out_err; @@ -2654,11 +2648,10 @@ ssize_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, sg = sg_next(sg); } - if (ops->iotlb_sync_map) { - ret = ops->iotlb_sync_map(domain, iova, mapped); - if (ret) - goto out_err; - } + ret = iommu_sync_map(domain, iova, mapped); + if (ret) + goto out_err; + return mapped; out_err: diff --git a/include/linux/iommu.h b/include/linux/iommu.h index bd722f473635..8927e5f996c2 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -799,6 +799,10 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot, gfp_t gfp); +int iommu_map_nosync(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); +int iommu_sync_map(struct iommu_domain *domain, unsigned long iova, + size_t size); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, From patchwork Wed Oct 30 15:12:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856677 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA275D5CCB9 for ; Wed, 30 Oct 2024 15:13:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 418F98D000B; Wed, 30 Oct 2024 11:13:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C6238D0001; Wed, 30 Oct 2024 11:13:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23FC88D000B; Wed, 30 Oct 2024 11:13:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0293C8D0001 for ; Wed, 30 Oct 2024 11:13:35 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B8FAF120DC8 for ; Wed, 30 Oct 2024 15:13:35 +0000 (UTC) X-FDA: 82730611404.09.223AAC6 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 6249220009 for ; Wed, 30 Oct 2024 15:13:07 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OvNAG5mt; spf=pass (imf13.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3MQIVTpPUFkgHQ2kL+3oRPWGhhug+8ZedKPylFoOF7U=; b=KOf2kYIiC2bS/Mq1n2LYJKCDuFBskgtCDSGzyYw2+nsl3LHySbQCVmgz4u/NcVE02WLHBh 74OPecvT64xkV6GutOVPlKpNsSkV+NCgmJj9DpRWOEc1ByRH7SCKutHAGwEk3YXE0Nv/nw Djur2L7dFvg2a00NgpKZlN4lpB6kb7o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301053; a=rsa-sha256; cv=none; b=QrLneA6Lff1/FbL9Kf7+uMF2P+ctt/psnrJk63SwT2FOUgyN+FGFzs3+iZSHwkX2/hlK/f yGvQPY8sCXDM1cZEPQHZGq1hTXbWEf2//bctdn2GWcjclHCnw8zJbR/C1Jr9h6B6vj13e2 QyEZ2C5hvPVbXs684/ARqeAwWWSEfeY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=OvNAG5mt; spf=pass (imf13.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 3B60F5C5EFD; Wed, 30 Oct 2024 15:12:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 14BBEC4CECE; Wed, 30 Oct 2024 15:13:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301212; bh=Lk+QUmdhMcbaHNbOGFaFlupW/NnaXwQbfPX5iL2RGqE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OvNAG5mtLwTxS6yIXlexDZ921FgcNRyBgt2vMt4axYlqHFpW8SmDoqG5XbqxbzCM8 L7YW2Ct+xt9DP7bWW4sEvI5Wk9Wp/VMbr4ZP6P2mhmfX2LN/xOGuLsqWCbQA/Y4XCM ke1UZ9s0S7fVl2ZM1mE1d3tLVQ3/k+XbRZtRTXEq0Rn0ieVLumeCOVrHFxa/qmcS/D g36jMcwkC2j0oYQVEtJ2jNNs911O3lVzzsZ2W0PtVH26+srz/cOSgTXvIBCehUYB2y T3ONlc1oOu+WKGhena4V70d3gbHxgyVwE0sdFDVFAQhJXi8on9dtDc9xba76vFc57v v/H2iFVgh9BjQ== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 04/17] dma-mapping: Add check if IOVA can be used Date: Wed, 30 Oct 2024 17:12:50 +0200 Message-ID: <9515f330b9615de92a1864ab46acbd95e32634b6.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: wgxctu31pf8jyg35eq3sawhsp43zhhhk X-Rspamd-Queue-Id: 6249220009 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1730301187-706574 X-HE-Meta: U2FsdGVkX18DQur+CpDdDaMomIobiXQQKP47zTRPWMEhBHPEv9B6UWkNzL3qmdms8e6LwSlflu3J5l8UjtZWMs7FlLd+XLoBX1cP5/wGfGwHZ9bl9eJeWNcLBa1P77IphNGbwx9FXxTBCNxz5S3PX53ij9vA10Us7B2fYSnt6pg7srU+JJEXTihl/GKcdSvCS7JO06+XoNTEv/jaD8j1eyEYfeeu5Nb+D3Mwz7Vc5eKXPyH6ZGx8mTdiTXbDwbKFRYgYIJ6rsBdZr34ydEmFEndVcSufcb4z5VxOeXtRkgencYOnFASleDgWx1yvVxxEe48lh3ciMm7EXMi2wlDTSfF3OYY3jXJmrh5hYQQJt5eKEVrdTJeiHS9OER1/xPumY4HqUMf1zxRfTYCxdZscHI9ATb/fJVIABgjoJJ2KdlNi1f6nObQGf1QV3oJE5zZ5vJ9iPfdGJyImK8hjNjJV1XLZReh/jBLcYcav/ICyMrJUBh5yOeFuBnjUg8ZRuTnYQCXVY4lHTDl8qDgOx1jPYRJkCCqz4sZ5frWco6ZffrqBkbCHztyRxO4teU2uVWFYsw+jzGkj+bBb1sb7C+HE8Dsug91EG7h50dgmK8hX6naxRBQWyKZkoWAeRnBhQWpZqfBJfc9rwz8AZ4N01egHj3qNKzgH6yUz1yeN9HVmMkhf49p3mLR0Hn1sEpFI3E9HbbkoW9D6r1HGDPDViGnYS23Tnyq4YxTSSfqTLwcj5i+upLhKDXPpoGewO9IRLRkDLgWudLPKW5jDALQIhba3j2kE9ELlBCLMb5VlMecnykIAsSd8Pzgn8K2s9dk2xDLJrgZCHPNUKYU16njuZz99xhbvyGGEZ6HPuZjcQe4VKaCZ1FmpBZVGfHFwVANCoyCmPqy1uCfIOC+LZ3W7w0DHqfcsrCjZF87bmlCWcu+tw+uX7naRI9BZyGADjOkQkDgJi+VcmQmrxG5MN/jYFqn 2OEBtbOh YT9bikk7SIbP+Fbu7x1+xbv1FGg7BsLdrjc6X3HkLrBoJLwHQXp+jAxbN2x0qjLJgubhu2+Fjyn6HoXzinYX8kS0LPgtxUJSfFT6nucI5buDW3Mfw1s+br5h2koj+4oayJiSRG7RHAOUqgesHYPtNG064KK1UVaO9V+QwjXmABkJIV3I7xoEeJyebHKduS6wpCW8TgsjgJnUy1wIZHVLossyoM1D/l0t1VidNgc8HpFFujdI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky This patch adds a check if IOVA can be used for the specific transaction. In the new API a DMA mapping transaction is identified by a struct dma_iova_state, which holds some recomputed information for the transaction which does not change for each page being mapped. Signed-off-by: Leon Romanovsky --- include/linux/dma-mapping.h | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 1524da363734..6075e0708deb 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -76,6 +76,20 @@ #define DMA_BIT_MASK(n) (((n) == 64) ? ~0ULL : ((1ULL<<(n))-1)) +struct dma_iova_state { + size_t __size; +}; + +/* + * Use the high bit to mark if we used swiotlb for one or more ranges. + */ +#define DMA_IOVA_USE_SWIOTLB (1ULL << 63) + +static inline size_t dma_iova_size(struct dma_iova_state *state) +{ + return state->__size & ~DMA_IOVA_USE_SWIOTLB; +} + #ifdef CONFIG_DMA_API_DEBUG void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr); void debug_dma_map_single(struct device *dev, const void *addr, @@ -281,6 +295,25 @@ static inline int dma_mmap_noncontiguous(struct device *dev, } #endif /* CONFIG_HAS_DMA */ +#ifdef CONFIG_IOMMU_DMA +/** + * dma_use_iova - check if the IOVA API is used for this state + * @state: IOVA state + * + * Return %true if the DMA transfers uses the dma_iova_*() calls or %false if + * they can't be used. + */ +static inline bool dma_use_iova(struct dma_iova_state *state) +{ + return state->__size != 0; +} +#else /* CONFIG_IOMMU_DMA */ +static inline bool dma_use_iova(struct dma_iova_state *state) +{ + return false; +} +#endif /* CONFIG_IOMMU_DMA */ + #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) void __dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir); From patchwork Wed Oct 30 15:12:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856678 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74FE2D5CCB9 for ; Wed, 30 Oct 2024 15:13:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12FC48D0001; Wed, 30 Oct 2024 11:13:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E1988E0001; Wed, 30 Oct 2024 11:13:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEB118D000C; Wed, 30 Oct 2024 11:13:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D115E8D0001 for ; Wed, 30 Oct 2024 11:13:39 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5E37B1A0E18 for ; Wed, 30 Oct 2024 15:13:39 +0000 (UTC) X-FDA: 82730612202.24.B6C24A7 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf19.hostedemail.com (Postfix) with ESMTP id 57B561A001E for ; Wed, 30 Oct 2024 15:13:05 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SbwHDIv5; spf=pass (imf19.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301056; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hMQrq4damIqraIK5Hyv9GC3JunpgB2mwM/Flp+wYMaA=; b=HCdmyUMmqPoBOzrgvJClTZy96x302TsqbKypC9y7ritDzjBe7WA5FfKzWyImSk7uDTjCCz eGMIagYUtgJ8dg9p8LwRx5z0bmReZ2Q9aLpwOCS1ZKYywaT6Iqz+w7APt8NC5rmzFS5Gz/ K485opXGjo9OH8N5JApMQhwR8dh2NaU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301056; a=rsa-sha256; cv=none; b=SWCfyGFJjfLC/4To5D24K7/WvG3iYEQKwUFYqJJiYaQ61zMk+LsdjnkbWajyWf77A1dSYN cZUv7BK2XUJgBlsiPoGCHCiMxewKzymZNdBchP0BdwRwQSjYPodQWXsCHpIPdCgxb6Y5eq SqmEk1zWFQSkSDAIjdc4MjM2hNFe+cU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=SbwHDIv5; spf=pass (imf19.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id EFBCFA418C0; Wed, 30 Oct 2024 15:11:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2123C4CED2; Wed, 30 Oct 2024 15:13:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301216; bh=wHgEMwPQysWFRNninL/6ReSJNwvAIakyr6CEHJOLARw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SbwHDIv5BcBoTUVjjDQDtfvTuNMIDzcN5nDrlN1kiCBopZXLtJxiGHs8GmFNSny6i xM06GCTdJcEBYA0nsJrS5joJzq8XbFneIawU8UeLMrR9Przpd6vT94nB4ReBhUjG47 vFCiJgZi0F4NXugQ8xGts9u9XVa472CAeaLbQoKs0FtDWG2/its9c7DJ0KdsN2YYfI AGaSV2WzluQ4Awrsbu4697lpMRqZlfJvrPxoSk1xFj6foQmOhvnkzvbjB1WYp73wUP Ju0sDenS3aETSRn//mw80yUhcsCkQ35k//5CF6T/F2j6xOrFddNsTzfs4Glb9qf50Z zJeVW46+yVnuw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 05/17] dma: Provide an interface to allow allocate IOVA Date: Wed, 30 Oct 2024 17:12:51 +0200 Message-ID: X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: du4kzhe36n6zerts53jt6byzyzsqq8oo X-Rspamd-Queue-Id: 57B561A001E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1730301185-66205 X-HE-Meta: U2FsdGVkX1/tKzmlwK4Zx3W+X3KUBKznl0ZnZJan2eqY05bHzo9kapF9dTvq3m9Poa2BTIEo7ACU2w322xEr/ucqKeuN7mwaRKooOu6FqsY2JQ6jkwK70BMvOCmpCO8o1HySBy6ATVJTpKK25w92/FzE8k3Ijc7kJ1+a180kJvvqQPXn4LVzcAXfD8WbboRoUgJ1tPpagQ48kwMT+C3wDQWJoEA0r9PxyGSfBoRF8xHt+nH5aAGNdqezpsILtnVT84jPSLM3eDnOp+dhs6rkVJBxw+ys6LgfsCiOBohx5uVYIUqVfde01+v6NcYpPFXnakLsAmHvl50dz+vi3a/J6djSkUcr+temEsrN2n4UI3iS4/WQ8/9lrs2UAScAMKfDRuoo+7iNs2U/ZjTZ86soeG8/RZ4G5+EJH1iXqT94b2OvT4a54DW+NBwJ/FsvzNnJhjTbzyOV3rFvunrXFGUj6wnVqBmiNboH1T2+tUtIu6WYBbQULHxqb5VItefwPI7YoDuoJFRRJ+GvGqsH93dAGGCGchuyRFsUlSoHwrbFdRgll+RaCQZzcn5OnP025jv+JKAezsNGwfsfU75Erccq0jEcuSZnbmHqM6sDw7jI5OvUt4HKPEV5hmwQ2o8Y+RKxkt9lvDGpwPq47VfMiRJwhKm7um0bnuL9nEZ/eAhlnIjysNPB9qxK51czkOSXQcTULptGqWCBS5b4Qk+EZbq9paWKYUXODZge7kUvnfd4SCfVGCBvdgUjRWYHoYHRnyAzaitmS4vtlZTPVTsQtNXSOuosng5bEz0ejjirUPWAhvGlpkENB377+oQpBdI2HhfXGcXh4Wsjt+nWVZXpJR91T/NiSDKaM3w7uAay6WFGinh+HvQaCyN3QeUvTmrxrn3/C4Cwz7GqLqSdOlpueUDZkvEof/ZeZbZJ9FaDrEAtAefWCFlDXUk7oECH/JrSrHTVzO10A7VpjK1bwxL+aCY X6mHByI6 NOxvGuMFi+2hnoqSvcII2H+uMsiMmsxMs1ifwL64hQrO3gQG4NruuSDS1WGhtFEHsycGTUyjrWUdDONSK1a5wE+qramPEyu+YuPOSWFhN3+il9KaRXSjHsMCAzniQonsGICTopWNoNacvrpfnI7IglRtnP5xoFDmKASijt7XFsyLjYMbD8jZ1yVtvRfepYZqCZmXgNnrmfFrlMhVzG2I6A1yMB25AlBD/koLvmH8ucY3bgkA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky The existing .map_page() callback provides both allocating of IOVA and linking DMA pages. That combination works great for most of the callers who use it in control paths, but is less effective in fast paths where there may be multiple calls to map_page(). These advanced callers already manage their data in some sort of database and can perform IOVA allocation in advance, leaving range linkage operation to be in fast path. Provide an interface to allocate/deallocate IOVA and next patch link/unlink DMA ranges to that specific IOVA. The API is exported from dma-iommu as it is the only implementation supported, the namespace is clearly different from iommu_* functions which are not allowed to be used. This code layout allows us to save function call per API call used in datapath as well as a lot of boilerplate code. Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 79 +++++++++++++++++++++++++++++++++++++ include/linux/dma-mapping.h | 15 +++++++ 2 files changed, 94 insertions(+) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 853247c42f7d..127150f63c95 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1746,6 +1746,85 @@ size_t iommu_dma_max_mapping_size(struct device *dev) return SIZE_MAX; } +static bool iommu_dma_iova_alloc(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t size) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + size_t iova_off = iova_offset(iovad, phys); + dma_addr_t addr; + + if (WARN_ON_ONCE(!size)) + return false; + if (WARN_ON_ONCE(size & DMA_IOVA_USE_SWIOTLB)) + return false; + + addr = iommu_dma_alloc_iova(domain, + iova_align(iovad, size + iova_off), + dma_get_mask(dev), dev); + if (!addr) + return false; + + state->addr = addr + iova_off; + state->__size = size; + return true; +} + +/** + * dma_iova_try_alloc - Try to allocate an IOVA space + * @dev: Device to allocate the IOVA space for + * @state: IOVA state + * @phys: physical address + * @size: IOVA size + * + * Check if @dev supports the IOVA-based DMA API, and if yes allocate IOVA space + * for the given base address and size. + * + * Note: @phys is only used to calculate the IOVA alignment. Callers that always + * do PAGE_SIZE aligned transfers can safely pass 0 here. + * + * Returns %true if the IOVA-based DMA API can be used and IOVA space has been + * allocated, or %false if the regular DMA API should be used. + */ +bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size) +{ + memset(state, 0, sizeof(*state)); + if (!use_dma_iommu(dev)) + return false; + if (static_branch_unlikely(&iommu_deferred_attach_enabled) && + iommu_deferred_attach(dev, iommu_get_domain_for_dev(dev))) + return false; + return iommu_dma_iova_alloc(dev, state, phys, size); +} +EXPORT_SYMBOL_GPL(dma_iova_try_alloc); + +/** + * dma_iova_free - Free an IOVA space + * @dev: Device to free the IOVA space for + * @state: IOVA state + * + * Undoes a successful dma_try_iova_alloc(). + * + * Note that all dma_iova_link() calls need to be undone first. For callers + * that never call dma_iova_unlink(), dma_iova_destroy() can be used instead + * which unlinks all ranges and frees the IOVA space in a single efficient + * operation. + */ +void dma_iova_free(struct device *dev, struct dma_iova_state *state) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + size_t iova_start_pad = iova_offset(iovad, state->addr); + size_t size = dma_iova_size(state); + + iommu_dma_free_iova(cookie, state->addr - iova_start_pad, + iova_align(iovad, size + iova_start_pad), NULL); +} +EXPORT_SYMBOL_GPL(dma_iova_free); + void iommu_setup_dma_ops(struct device *dev) { struct iommu_domain *domain = iommu_get_domain_for_dev(dev); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 6075e0708deb..817f11bce7bc 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -11,6 +11,7 @@ #include #include #include +#include /** * List of possible attributes associated with a DMA mapping. The semantics @@ -77,6 +78,7 @@ #define DMA_BIT_MASK(n) (((n) == 64) ? ~0ULL : ((1ULL<<(n))-1)) struct dma_iova_state { + dma_addr_t addr; size_t __size; }; @@ -307,11 +309,24 @@ static inline bool dma_use_iova(struct dma_iova_state *state) { return state->__size != 0; } + +bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size); +void dma_iova_free(struct device *dev, struct dma_iova_state *state); #else /* CONFIG_IOMMU_DMA */ static inline bool dma_use_iova(struct dma_iova_state *state) { return false; } +static inline bool dma_iova_try_alloc(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t size) +{ + return false; +} +static inline void dma_iova_free(struct device *dev, + struct dma_iova_state *state) +{ +} #endif /* CONFIG_IOMMU_DMA */ #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) From patchwork Wed Oct 30 15:12:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856679 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 522CBD5CCB3 for ; Wed, 30 Oct 2024 15:13:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D8CEA8E0002; Wed, 30 Oct 2024 11:13:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CEF788E0001; Wed, 30 Oct 2024 11:13:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B67AE8E0002; Wed, 30 Oct 2024 11:13:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 992F78E0001 for ; Wed, 30 Oct 2024 11:13:42 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4FC071A0C67 for ; Wed, 30 Oct 2024 15:13:42 +0000 (UTC) X-FDA: 82730612580.04.F323C3D Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf27.hostedemail.com (Postfix) with ESMTP id 6653C4000E for ; Wed, 30 Oct 2024 15:13:14 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=LmPDkamq; spf=pass (imf27.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301061; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=x88jTcq6GLouiEokwKqr1ClmMP8uqV/wSbdaC6q3fFg=; b=XoMpRwbQwKmMIfE+tnP4EppSCNGa34ORYLKkYYfEkvsNadaLIws3MOibt5qd71mgoYzhtE eSeswG5kXkkzZKKHO7c/KMJKDHw7/78ogqGBxLrI9QuMuyGsOQUKgYtxbIA9bQPFIVb3uw jXFoAQapPen/fCUiL9hXFx7plXLPpfU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301061; a=rsa-sha256; cv=none; b=lJ76zdCEMZBbN4tfMbKT9VirKpiCZYV/5iwVrexwNfk0Mch9D54E5F+/sF9KLO3GgrPgok 2mDC4KLNvIjsfIs7pW2ytpOfAAgTe4MP4B1nvw0fSbXSve+BCUjBCqdfLTQ0h+oZz8eQyB kb8DSZqeoga2Ye5xPY4KxS9kRbKwOOg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=LmPDkamq; spf=pass (imf27.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 4BEECA4255F; Wed, 30 Oct 2024 15:11:44 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 13913C4CECE; Wed, 30 Oct 2024 15:13:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301219; bh=XGQ1NR355bwszo+JrznVV1hbHviA7CQrxv/BERy+y90=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LmPDkamqTEpX+kwqmBANPlqtlaFChfErvXzZhGcch88a2LmIU2YivyLFbxjH/hzDZ UNXh/XXgTFkZfi9o/NrVNl/D2NwKGhCbDbxrukedWRjZezPUNgu3cjHDkBmHwB1ipi ToolLhizyQGeQxxr9bWcbopr2T3qyqnWcemFq8LORVDrrnGTOu7T2CVKojWMqU0aQy +EYFhYN45wrl8n2fh86UD+fmcagGwz4cCmHXzXzOmvR9DGTd+10cF2ZIGLyWyxQLVv N2MVluX7uYAEpW4yfHdfhNE3yxm+LfwU2OgZ2tF67RMlGpA+rFfIfD9Nx5spoyzjvo M+eW1/GKIlmtg== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 06/17] iommu/dma: Factor out a iommu_dma_map_swiotlb helper Date: Wed, 30 Oct 2024 17:12:52 +0200 Message-ID: <38bd0f07e18f4c74f2ab77f268e9d0424569f69f.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 6653C4000E X-Stat-Signature: ddut3urowfieoxnyziycfxckrxdb59b7 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1730301194-358554 X-HE-Meta: U2FsdGVkX1+J7NySuwOtqK3kpf9JBZQFWQsxn27rk74phayS3nVQyaibs7oY3xm99gjkZhnb/9VlNoylsn1f1W4o5t1mu6XrIcQohUwL3nAignu6Vfbcrj9VZO6fXv5wIORxYmvjoCqny8hnRheCmMdr2VD19bfZ0VMpgpLh/+yU6wqfRTAd9mkxO/HkL277I2BM4u62pxTprtuWVINo4P3AB5WzEjY9BWbnfcARLNap/FoiXEZG+XL1JnOWXqZ3kzDTCWeG7zPKexIfQEXBatxiKLZ7aJ9YXkcoUlFyHyogFoA03ZKZOdGMa5vfEkQOBqE5UtdhER6bSJpbqUmn7n0H9pP9ba2UJcQSQ+Xm+jo/sW+QsrMMZoUNLSAf71bJLPbtIogW0jAEog/6EmixzIUkfiIe/CgiFJWjqSJHkrrkQVhyTSnw4pcwRUd7Df7EBIrx5mGaX0SS/AsI9vm6nUs1C6KwT6xmJ4HFWS/3eQR6PLPc/RLcjw46lR+FjBk1Sn48NoanAvYlkjSHmm83l3BzgVtkRyZZxzqWEXGRLqZpt97MJSxohTn8Dfnke1rdKy4EQLgv/GhmUfl0P7DwO6o1/F3TQazM6sqrUohlZXY41UFRn7mEie5IrMaS8GRnCBuoPsVe3/T5cle3rtzQyJmG6T6GeHDTA9QUukiIa16eHn+XXe1G47oYv/wZOZwHW59e9oUyzGa21EPsYdFVtlVcpiNgbYkhuwIQhLqjCGwI0bd2FTn4SvY3SGOHI0WlM+vH0tnSgw1K1a978OYOcebLfFS2A9UcP0aVBJdtyKxIGoRKniygOg9cM3OcWjC6i8dYfDnGG1nTwnwJnxj9RtiI6/nHyjY/UeEYY840v/MeGiRUGE9ZudZ5wL2TI8YYdP8msE+dO4jTvAbo1IF+CFcnLQpOBWkdjEirGm3u3Gs9rwdosdqAvUi4sUUL6Ir5jKdxVGDGLafGrrJmzyn Gp2D54vW wt7Y0jNgEjakjrrGCoL1duou4Wrhy+dEjLrDQjkHVIlTT3dzKqVL6hcPAL60hn2bhuI0XkP6UnGHIX7lx1CW4LhwQTSneM1ABlvLjJGIX3hYP6OLw6vZY4v3Kpl0lq7ogzf4Q4/IvHFYQEeziwsOnuNmYNrm0lWyco9vAsGbaPmCG81Y8ceBXlVqqN7YYOI+wxvYTBquXWV1Pt3VVgbduUyecwSaI2TrEWosPVue46n1q2GaKv/pawHbGDbTJU7+zAhHq+VFt6eGNBL3zgXdSCJpm26WVKTeb1XoN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig Split the iommu logic from iommu_dma_map_page into a separate helper. This not only keeps the code neatly separated, but will also allow for reuse in another caller. Signed-off-by: Christoph Hellwig Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 73 ++++++++++++++++++++++----------------- 1 file changed, 41 insertions(+), 32 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 127150f63c95..e1eaad500d27 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1161,6 +1161,43 @@ void iommu_dma_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, arch_sync_dma_for_device(sg_phys(sg), sg->length, dir); } +static phys_addr_t iommu_dma_map_swiotlb(struct device *dev, phys_addr_t phys, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iova_domain *iovad = &domain->iova_cookie->iovad; + + if (!is_swiotlb_active(dev)) { + dev_warn_once(dev, "DMA bounce buffers are inactive, unable to map unaligned transaction.\n"); + return DMA_MAPPING_ERROR; + } + + trace_swiotlb_bounced(dev, phys, size); + + phys = swiotlb_tbl_map_single(dev, phys, size, iova_mask(iovad), dir, + attrs); + + /* + * Untrusted devices should not see padding areas with random leftover + * kernel data, so zero the pre- and post-padding. + * swiotlb_tbl_map_single() has initialized the bounce buffer proper to + * the contents of the original memory buffer. + */ + if (phys != DMA_MAPPING_ERROR && dev_is_untrusted(dev)) { + size_t start, virt = (size_t)phys_to_virt(phys); + + /* Pre-padding */ + start = iova_align_down(iovad, virt); + memset((void *)start, 0, virt - start); + + /* Post-padding */ + start = virt + size; + memset((void *)start, 0, iova_align(iovad, start) - start); + } + + return phys; +} + dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -1174,42 +1211,14 @@ dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, dma_addr_t iova, dma_mask = dma_get_mask(dev); /* - * If both the physical buffer start address and size are - * page aligned, we don't need to use a bounce page. + * If both the physical buffer start address and size are page aligned, + * we don't need to use a bounce page. */ if (dev_use_swiotlb(dev, size, dir) && iova_offset(iovad, phys | size)) { - if (!is_swiotlb_active(dev)) { - dev_warn_once(dev, "DMA bounce buffers are inactive, unable to map unaligned transaction.\n"); - return DMA_MAPPING_ERROR; - } - - trace_swiotlb_bounced(dev, phys, size); - - phys = swiotlb_tbl_map_single(dev, phys, size, - iova_mask(iovad), dir, attrs); - + phys = iommu_dma_map_swiotlb(dev, phys, size, dir, attrs); if (phys == DMA_MAPPING_ERROR) - return DMA_MAPPING_ERROR; - - /* - * Untrusted devices should not see padding areas with random - * leftover kernel data, so zero the pre- and post-padding. - * swiotlb_tbl_map_single() has initialized the bounce buffer - * proper to the contents of the original memory buffer. - */ - if (dev_is_untrusted(dev)) { - size_t start, virt = (size_t)phys_to_virt(phys); - - /* Pre-padding */ - start = iova_align_down(iovad, virt); - memset((void *)start, 0, virt - start); - - /* Post-padding */ - start = virt + size; - memset((void *)start, 0, - iova_align(iovad, start) - start); - } + return phys; } if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) From patchwork Wed Oct 30 15:12:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856680 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A968D5CCB8 for ; Wed, 30 Oct 2024 15:13:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E03378E0003; Wed, 30 Oct 2024 11:13:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB2148E0001; Wed, 30 Oct 2024 11:13:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2D368E0003; Wed, 30 Oct 2024 11:13:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9FD3E8E0001 for ; Wed, 30 Oct 2024 11:13:46 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 510C3140C37 for ; Wed, 30 Oct 2024 15:13:46 +0000 (UTC) X-FDA: 82730612118.19.6826D03 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf14.hostedemail.com (Postfix) with ESMTP id 9F833100010 for ; Wed, 30 Oct 2024 15:13:15 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=XWPRFrM5; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SrCMSABCbwXM0u2TNmEkKmhJZz3GfMWSIn5xtZoxhpI=; b=amrZrJZh2m/o5W2SLlpkzC65uda8MwR02yHBzC3b61PG6/3Aa59dYlkzRb79VMd3BrKC/h jyYpSIHrfd5O2/LaaXnEoTCWigPD5DMetlS/G3XpTkI0oDfIzhMD0Oxp7RnW0twMJ1cvlv omsCj+UceOk16WQyTsl2jXpkuYatMDk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301063; a=rsa-sha256; cv=none; b=kQ9z4nXHlL9Rb2ojswaygOtrskJDimzTrs9NhHnmmLwFXzEAcHv8lOGrmaEvPlCErDXVPN o1LMomvJw/wXJyScKVOkU5W6oB8OxHjI87v0n7OrE0MwBViY+WW0KaHSFJcQslUEuZ228p T8TtRJcPyywd7rcuzEwK/jgi9WThL/c= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=XWPRFrM5; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id B92FA5C5E7F; Wed, 30 Oct 2024 15:12:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7139C4CECE; Wed, 30 Oct 2024 15:13:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301223; bh=IXzQBF3c/yZPTe8x0QJ0JrgIemF2ruKzaY/aVUXUcCc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XWPRFrM5Wl6m+dLBlaCdBj6Qm4Zw5gf7r4ikooFMjHGGjvlVhKOXmFsDVfXeGbn6a V+3nbQe8i9e3XgDRmfznzRzRVpnD0wL4c7Sj4qSPKm9ojdRfK0fkNvfxSCseJIAIfV 2j9reoSJHsWHX7TVWg2oQfigJHAyw1RA+Z+ugbKqa5QHhvW9/wiHRkKnqausYhBAC4 y0j990xxc/JfxcAzjhZZE411BzuI4ay8OxofXouSEysYsqODetq3MW7XrtwkfPcRQ4 89IjaioXYahXzXXZigTjOuXF762f9Xkc9avbjlVSDIMBFvbJbEniIiLEeRG0NXSzoq 3bI0xzw9zptBA== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 07/17] dma-mapping: Implement link/unlink ranges API Date: Wed, 30 Oct 2024 17:12:53 +0200 Message-ID: X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: ecwgdujmkgx96rxag5wigsggbbjzba89 X-Rspamd-Queue-Id: 9F833100010 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1730301195-552514 X-HE-Meta: U2FsdGVkX1/UeXbkXI11pPaw0FuYoGYqCMnsf6iG0SfGQTjI94gWrXYoR96RiTmL+LNePFwv8d5GEiEvJ6XySytN0NOUqKsq9NnRHyi9RsBil71QxMWtvLR0WXOG7EsfD7jyIaL7SyuJpIUfsqvguaq2r8172OP+tG0TAUGPX+LfHHJfTT4hGPeCubGq+p0/0Mp8Zl0ET75fY1oDODKXIJM5qYOSSWrKPBnl8b3N2RAA978sTnchVtX4ratmdhsi9C1+56zjuDVQHciorClexsyZNBDGEiJcBWJJeCNJ102kSyhJ62l8a8NaUc/2W/k1uE4Y0ons3FZfMMesY/f0wlueq5eXlUh8EYbogkxAcxOblwwt+rmvtBK5BSNnO1cmUNHDPkwE2F4InIYFHixTHfyjseViIWL43D4cAt/dSc67jPe1W0E9ZY5wUIW3rjoLKErGN0zmye9knGlcGJSz0FGzbTi8ub3rMbrdMHg4QNmToMeW/POQq3eudXYD/Kycud1hDMXaSqAcF/dgqoL9VLaNteCqLWsjjV9mRClyfnbauKYnBuiIVz7wt7xDckwwhSC6+RjoxIBnpHMQxCkbiBCZ4JQ1URVsOJ7umcsMvMT2qeBgt+9O6TMVpavONXsktlkIzUvQJKwZN6stBmnslJG3ICs2BVF0d5f1rYTrJdV9tCKzbAn5gbAZMBDqvJrdnZjAERPdE2VyOsGNPOjRerSKW9GOk4yJx1Fj7hHxuUG1nP2pUVXTkVYU/jYzolcBOdqjTQG+BBFnAgWreQzQy/S7h9R4f4K83AoYuRnOLDLjiJs494yfestG9k8LadJoqdUDDH5AHH65BaL+mgNG9Z/U/03KwOJwK+HrtDk6uYgDhLSa/xpDtWHBfG+zIiGrpQJaVG2QY8MOJp/QKSnhSU+xs54gJ+Fgd4QcxKLC+vqhFx51zf6Hv0KxtKtr43h1Buubgy2q4Ed90m6XNqd 5W+ZgAV/ 7hJJa8Q64910A0eOYTXk7ok4KMHHFBMaEbSYid04D2LbOdxAfnkxiP4aP3YCa8bZmMOJxXdOYJswVVdMrrkDR206IALPZt4XSJ8vBkzaPX/DH17WGlwn8dYIv+e6NN+Y22k/XC+SijOQNDYzTCzgyWZ3HLadSyvLwti7bhqeCyS7yPdOdU1I+N4Js9o0BK/Gttv8mBxgcb6d3uuph1zZCwpUViy4uz4LBtSggqPTlq/j5+9o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Introduce new DMA APIs to perform DMA linkage of buffers in layers higher than DMA. In proposed API, the callers will perform the following steps. In map path: if (dma_can_use_iova(...)) dma_iova_alloc() for (page in range) dma_iova_link_next(...) dma_iova_sync(...) else /* Fallback to legacy map pages */ for (all pages) dma_map_page(...) In unmap path: if (dma_can_use_iova(...)) dma_iova_destroy() else for (all pages) dma_unmap_page(...) Signed-off-by: Leon Romanovsky --- drivers/iommu/dma-iommu.c | 259 ++++++++++++++++++++++++++++++++++++ include/linux/dma-mapping.h | 32 +++++ 2 files changed, 291 insertions(+) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index e1eaad500d27..4a504a879cc0 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1834,6 +1834,265 @@ void dma_iova_free(struct device *dev, struct dma_iova_state *state) } EXPORT_SYMBOL_GPL(dma_iova_free); +static int __dma_iova_link(struct device *dev, dma_addr_t addr, + phys_addr_t phys, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + bool coherent = dev_is_dma_coherent(dev); + + if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_device(phys, size, dir); + + return iommu_map_nosync(iommu_get_dma_domain(dev), addr, phys, size, + dma_info_to_prot(dir, coherent, attrs), GFP_ATOMIC); +} + +static int iommu_dma_iova_bounce_and_link(struct device *dev, dma_addr_t addr, + phys_addr_t phys, size_t bounce_len, + enum dma_data_direction dir, unsigned long attrs, + size_t iova_start_pad) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iova_domain *iovad = &domain->iova_cookie->iovad; + phys_addr_t bounce_phys; + int error; + + bounce_phys = iommu_dma_map_swiotlb(dev, phys, bounce_len, dir, attrs); + if (bounce_phys == DMA_MAPPING_ERROR) + return -ENOMEM; + + error = __dma_iova_link(dev, addr - iova_start_pad, + bounce_phys - iova_start_pad, + iova_align(iovad, bounce_len), dir, attrs); + if (error) + swiotlb_tbl_unmap_single(dev, bounce_phys, bounce_len, dir, + attrs); + return error; +} + +static int iommu_dma_iova_link_swiotlb(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t offset, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + size_t iova_start_pad = iova_offset(iovad, phys); + size_t iova_end_pad = iova_offset(iovad, phys + size); + dma_addr_t addr = state->addr + offset; + size_t mapped = 0; + int error; + + if (iova_start_pad) { + size_t bounce_len = min(size, iovad->granule - iova_start_pad); + + error = iommu_dma_iova_bounce_and_link(dev, addr, phys, + bounce_len, dir, attrs, iova_start_pad); + if (error) + return error; + state->__size |= DMA_IOVA_USE_SWIOTLB; + + mapped += bounce_len; + size -= bounce_len; + if (!size) + return 0; + } + + size -= iova_end_pad; + error = __dma_iova_link(dev, addr + mapped, phys + mapped, size, dir, + attrs); + if (error) + goto out_unmap; + mapped += size; + + if (iova_end_pad) { + error = iommu_dma_iova_bounce_and_link(dev, addr + mapped, + phys + mapped, iova_end_pad, dir, attrs, 0); + if (error) + goto out_unmap; + state->__size |= DMA_IOVA_USE_SWIOTLB; + } + + return 0; + +out_unmap: + dma_iova_unlink(dev, state, 0, mapped, dir, attrs); + return error; +} + +/** + * dma_iova_link - Link a range of IOVA space + * @dev: DMA device + * @state: IOVA state + * @phys: physical address to link + * @offset: offset into the IOVA state to map into + * @size: size of the buffer + * @dir: DMA direction + * @attrs: attributes of mapping properties + * + * Link a range of IOVA space for the given IOVA state without IOTLB sync. + * This function is used to link multiple physical addresses in contigueous + * IOVA space without performing costly IOTLB sync. + * + * The caller is responsible to call to dma_iova_sync() to sync IOTLB at + * the end of linkage. + */ +int dma_iova_link(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + size_t iova_start_pad = iova_offset(iovad, phys); + + if (WARN_ON_ONCE(iova_start_pad && offset > 0)) + return -EIO; + + if (dev_use_swiotlb(dev, size, dir) && iova_offset(iovad, phys | size)) + return iommu_dma_iova_link_swiotlb(dev, state, phys, offset, + size, dir, attrs); + + return __dma_iova_link(dev, state->addr + offset - iova_start_pad, + phys - iova_start_pad, + iova_align(iovad, size + iova_start_pad), dir, attrs); +} +EXPORT_SYMBOL_GPL(dma_iova_link); + +/** + * dma_iova_sync - Sync IOTLB + * @dev: DMA device + * @state: IOVA state + * @offset: offset into the IOVA state to sync + * @size: size of the buffer + * + * Sync IOTLB for the given IOVA state. This function should be called on + * the IOVA-contigous range created by one ore more dma_iova_link() calls + * to sync the IOTLB. + */ +int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + dma_addr_t addr = state->addr + offset; + size_t iova_start_pad = iova_offset(iovad, addr); + + return iommu_sync_map(domain, addr - iova_start_pad, + iova_align(iovad, size + iova_start_pad)); +} +EXPORT_SYMBOL_GPL(dma_iova_sync); + +static void iommu_dma_iova_unlink_range_slow(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + size_t iova_start_pad = iova_offset(iovad, addr); + dma_addr_t end = addr + size; + + do { + phys_addr_t phys; + size_t len; + + phys = iommu_iova_to_phys(domain, addr); + if (WARN_ON(!phys)) + continue; + len = min_t(size_t, + end - addr, iovad->granule - iova_start_pad); + + if (!dev_is_dma_coherent(dev) && + !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_cpu(phys, len, dir); + + swiotlb_tbl_unmap_single(dev, phys, len, dir, attrs); + + addr += len; + iova_start_pad = 0; + } while (addr < end); +} + +static void __iommu_dma_iova_unlink(struct device *dev, + struct dma_iova_state *state, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs, + bool free_iova) +{ + struct iommu_domain *domain = iommu_get_dma_domain(dev); + struct iommu_dma_cookie *cookie = domain->iova_cookie; + struct iova_domain *iovad = &cookie->iovad; + dma_addr_t addr = state->addr + offset; + size_t iova_start_pad = iova_offset(iovad, addr); + struct iommu_iotlb_gather iotlb_gather; + size_t unmapped; + + if ((state->__size & DMA_IOVA_USE_SWIOTLB) || + (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC))) + iommu_dma_iova_unlink_range_slow(dev, addr, size, dir, attrs); + + iommu_iotlb_gather_init(&iotlb_gather); + iotlb_gather.queued = free_iova && READ_ONCE(cookie->fq_domain); + + size = iova_align(iovad, size + iova_start_pad); + addr -= iova_start_pad; + unmapped = iommu_unmap_fast(domain, addr, size, &iotlb_gather); + WARN_ON(unmapped != size); + + if (!iotlb_gather.queued) + iommu_iotlb_sync(domain, &iotlb_gather); + if (free_iova) + iommu_dma_free_iova(cookie, addr, size, &iotlb_gather); +} + +/** + * dma_iova_unlink - Unlink a range of IOVA space + * @dev: DMA device + * @state: IOVA state + * @offset: offset into the IOVA state to unlink + * @size: size of the buffer + * @dir: DMA direction + * @attrs: attributes of mapping properties + * + * Unlink a range of IOVA space for the given IOVA state. + */ +void dma_iova_unlink(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + __iommu_dma_iova_unlink(dev, state, offset, size, dir, attrs, false); +} +EXPORT_SYMBOL_GPL(dma_iova_unlink); + +/** + * dma_iova_destroy - Finish a DMA mapping transaction + * @dev: DMA device + * @state: IOVA state + * @mapped_len: number of bytes to unmap + * @dir: DMA direction + * @attrs: attributes of mapping properties + * + * Unlink the IOVA range up to @mapped_len and free the entire IOVA space. The + * range of IOVA from dma_addr to @mapped_len must all be linked, and be the + * only linked IOVA in state. + */ +void dma_iova_destroy(struct device *dev, struct dma_iova_state *state, + size_t mapped_len, enum dma_data_direction dir, + unsigned long attrs) +{ + if (mapped_len) + __iommu_dma_iova_unlink(dev, state, 0, mapped_len, dir, attrs, + true); + else + /* + * We can be here if first call to dma_iova_link() failed and + * there is nothing to unlink, so let's be more clear. + */ + dma_iova_free(dev, state); +} +EXPORT_SYMBOL_GPL(dma_iova_destroy); + void iommu_setup_dma_ops(struct device *dev) { struct iommu_domain *domain = iommu_get_domain_for_dev(dev); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 817f11bce7bc..8074a3b5c807 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -313,6 +313,17 @@ static inline bool dma_use_iova(struct dma_iova_state *state) bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, phys_addr_t phys, size_t size); void dma_iova_free(struct device *dev, struct dma_iova_state *state); +void dma_iova_destroy(struct device *dev, struct dma_iova_state *state, + size_t mapped_len, enum dma_data_direction dir, + unsigned long attrs); +int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size); +int dma_iova_link(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs); +void dma_iova_unlink(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); #else /* CONFIG_IOMMU_DMA */ static inline bool dma_use_iova(struct dma_iova_state *state) { @@ -327,6 +338,27 @@ static inline void dma_iova_free(struct device *dev, struct dma_iova_state *state) { } +static inline void dma_iova_destroy(struct device *dev, + struct dma_iova_state *state, size_t mapped_len, + enum dma_data_direction dir, unsigned long attrs) +{ +} +static inline int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size) +{ + return -EOPNOTSUPP; +} +static inline int dma_iova_link(struct device *dev, + struct dma_iova_state *state, phys_addr_t phys, size_t offset, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + return -EOPNOTSUPP; +} +static inline void dma_iova_unlink(struct device *dev, + struct dma_iova_state *state, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ +} #endif /* CONFIG_IOMMU_DMA */ #if defined(CONFIG_HAS_DMA) && defined(CONFIG_DMA_NEED_SYNC) From patchwork Wed Oct 30 15:12:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66EE6D5CCB3 for ; Wed, 30 Oct 2024 15:13:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E51938E0005; Wed, 30 Oct 2024 11:13:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E00F48E0001; Wed, 30 Oct 2024 11:13:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C548F8E0005; Wed, 30 Oct 2024 11:13:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A03F78E0001 for ; Wed, 30 Oct 2024 11:13:49 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6192080C8F for ; Wed, 30 Oct 2024 15:13:49 +0000 (UTC) X-FDA: 82730612286.27.2BD64C9 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf02.hostedemail.com (Postfix) with ESMTP id A95428001E for ; Wed, 30 Oct 2024 15:12:58 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=psdZpNNG; spf=pass (imf02.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301066; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=18NPGuceQo1bRh7brK8gmz6Fjwz/7GYqhz5ykldBY5M=; b=3fh7ubnZ7ddemhbAEfY2a9oLc1UakY3N2jbhy+t3/bs+Z8WX234ZWvnCXlGho8/4MTfAOI LnkFFbX99bZhoaFq+aPX8hHOAwkksBZXoOe9IxC5LVUKXpWhq8h9cgln/LssJtUQW9CmGe ZF3BbAyw+NrLbqmoSbUsy5qYGEzoduA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301066; a=rsa-sha256; cv=none; b=gC8/k9wxiyil4qd8ZgGb3xn15WG+6MaIofz4QH1cOkSc5Yd24Jiuce6mRv3Dmb8m9rnDqg 8Q0k3EyI4KFoaiNOUM0+VwmvRE3IuYOaWhnFFZxzT634eKyXsAsISRn8z6glt6Ao51DfHt NgXVA2ca4ARYF+8FystycjGJvnpfzkU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=psdZpNNG; spf=pass (imf02.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 66212A418C0; Wed, 30 Oct 2024 15:11:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3607CC4CECE; Wed, 30 Oct 2024 15:13:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301226; bh=QGhPEI3S06TBMrjanRSWwBEaYlCHA0taHYk6+dEHPl4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=psdZpNNGJcibprE+79HbooZm4yrYRQ0Yu1vOnpFXCYC6sQg7aj8A9kDoxv4O9iKda vpq6+UnL42B2GDmqpeaSgxnZCKXUatA37GWb2vauJWt+5W1nRqHtmPp04iI93g57tu KT11sUZqFdMvorZZWFhUeI1GodNfUEPOp44eQcLTwPEUO4I48Oa3RCy8kMqfiQy3y+ RLbv/eCj019iM7aJEV8UauLvHM/qFQekp0H0w4ukWgIMTJw+cem8+gNMj9xEujliUm wkqpnJG2GZLxNNnsYjzykT+M6nMB7eALtxtpI40Fb1ze4vYyPccY02MSuZC0u2g5z0 R8WSAEimTxIEA== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 08/17] dma-mapping: add a dma_need_unmap helper Date: Wed, 30 Oct 2024 17:12:54 +0200 Message-ID: <00385b3557fa074865d37b0ac613d2cb28bcb741.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: 8d8wfdhbte8z6mg7hu9gagccydpn9he8 X-Rspamd-Queue-Id: A95428001E X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1730301178-401748 X-HE-Meta: U2FsdGVkX1/NjImkcPw1c0myJzplgEUBJY2970GHKavTXSrbwg/qPZPAFKEwV+h869fWe4q43rCf1eYA72J2kGeGc1f7yvgFi4kpoEX7XrR0VqDGupD6ME4fqCbBbb0xEFehZtQr9FERB777Pra+gw+EFOv9nJSuCoLZ1M7N4C2ZmMJ7wzhaNWsMWwoZ31lj6kASmJEindUGPdkCCqRJOAvKff2sg22NWXR4+HThYPooKyB83pOlllTiGkFMQtso/kXKHH7fxfjomIqaUruFw5oByO0ukftj+PyVrgKvPlfUQTHvPZJSBU3C8UZjtaEi6RrpTzP3XJhZsdmyRXQuGsZuA8ykU9spsPcWAU8myntnLEeTqMTIrJVtnUGzae8wzEDYTCqsxj0iVqMN2Ut/WryKbzw9xZPkKVYnvmtqU4klhsKYKBUlQnTyLPXqpYsVdMbUpGyMrrUBpfPWj9j31hBIWIbFld4m5BBtHe5UpbACDX+P4gELQV/x0gmyv/4f6AswLq8vQLhAPHVg75FAX08QY32vG5BdjiUWZZSobOjNRfUr/o+QGHTiq2Lg5/JSIPPnJ6RrfQoUobStR8/xcYZP69aU+MM5Hs9AAgI+qXTm2e9Nq8qqo1IRogVlASFzePZbdzb0RltjvhHPR8XnY/h7DO79B8lKiEk0BmRaHE8DEz0sv2qWU1YQxYtFJ5Lk9YTpDoknwOlnuG4V3G345NXXp622/9rRd44pz3HCnMzCSAAi6F135ZrrRNlrpk/j1S/jy37Hy7fG6rVkkPRFGCGHTPxJ/n5nL3N7uR5I9KgVDE5ngfiF9shVUzw9VaaNswK2nwyyml0TT4m9u8aBikz0ypdumVtfZJoq0Rz75tunjwaQ5RDyBMdpUqhQzgNPmIsjOxn8z2iUw2bYifVW+dEkCmGOdKgc6oLV5vSHofHQuCRrlr/35NjbAAfi3RI88ikZKq35bTR7aN5+zzF 56nvaiCQ VmU/h96SrfLXx8QvWDhSm0fKgejQbe+6PBU+CH6tPwuOEQqDfeU4t01Om09bleXXX7XwCpr4wc4lyipY4V8bCKjJuzRPqXCpqHobkCYmMGqtqgWoVK7nkjYEIrrtGRziJnkOwuzT+ZRv+zSZYWQ0Xdg1YkrRZd2x1y3Q3FKk49Pr//T4VQ5lfjg2wfBIvl9Io5T5P+8EWI1TujEU8J8g7e0ZI1FH45Z7HRSTTkcjurrId7rQ818DZGHK32qNN7o4gJBg+FyvkSc6LGHWqRT5SLMLWx+FwFBRGK2IP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig Add helper that allows a driver to skip calling dma_unmap_* if the DMA layer can guarantee that they are no-nops. Signed-off-by: Christoph Hellwig Signed-off-by: Leon Romanovsky --- include/linux/dma-mapping.h | 5 +++++ kernel/dma/mapping.c | 20 ++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 8074a3b5c807..6906edde505d 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -410,6 +410,7 @@ static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) { return dma_dev_need_sync(dev) ? __dma_need_sync(dev, dma_addr) : false; } +bool dma_need_unmap(struct device *dev); #else /* !CONFIG_HAS_DMA || !CONFIG_DMA_NEED_SYNC */ static inline bool dma_dev_need_sync(const struct device *dev) { @@ -435,6 +436,10 @@ static inline bool dma_need_sync(struct device *dev, dma_addr_t dma_addr) { return false; } +static inline bool dma_need_unmap(struct device *dev) +{ + return false; +} #endif /* !CONFIG_HAS_DMA || !CONFIG_DMA_NEED_SYNC */ struct page *dma_alloc_pages(struct device *dev, size_t size, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 864a1121bf08..daa97a650778 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -442,6 +442,26 @@ bool __dma_need_sync(struct device *dev, dma_addr_t dma_addr) } EXPORT_SYMBOL_GPL(__dma_need_sync); +/** + * dma_need_unmap - does this device need dma_unmap_* operations + * @dev: device to check + * + * If this function returns %false, drivers can skip calling dma_unmap_* after + * finishing an I/O. This function must be called after all mappings that might + * need to be unmapped have been performed. + */ +bool dma_need_unmap(struct device *dev) +{ + if (!dma_map_direct(dev, get_dma_ops(dev))) + return true; +#ifdef CONFIG_DMA_NEED_SYNC + if (!dev->dma_skip_sync) + return true; +#endif + return IS_ENABLED(CONFIG_DMA_API_DEBUG); +} +EXPORT_SYMBOL_GPL(dma_need_unmap); + static void dma_setup_need_sync(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); From patchwork Wed Oct 30 15:12:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAC6ED5CCB3 for ; Wed, 30 Oct 2024 15:14:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 444356B0082; Wed, 30 Oct 2024 11:14:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F0F46B00A6; Wed, 30 Oct 2024 11:14:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 219FD6B00AA; Wed, 30 Oct 2024 11:14:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F1C376B0082 for ; Wed, 30 Oct 2024 11:14:10 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AFB02140E93 for ; Wed, 30 Oct 2024 15:14:10 +0000 (UTC) X-FDA: 82730612538.20.3F7172D Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf14.hostedemail.com (Postfix) with ESMTP id 2F98C100019 for ; Wed, 30 Oct 2024 15:13:40 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=YiKlu4D9; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DVzPaReK6bZLdA6DSR8HvqCwZ/5n23XgmH66ialyCvo=; b=ufMFe7Pd6O/LVEzunT2r+kr2436mmkcwassawml4K4HjBcPG8fBNySQlNMyPo6VPqaLluf s4rC5u3RoFbDE+Wt1xXsk8o7dgy/lY9asPEpbqRhr0tiZAKAgUl8W+e2c3tdv1ZCnrFtJU hZBS+SQJ61LxiD9Kw9pBUpoIyRgPatY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301088; a=rsa-sha256; cv=none; b=tukXgzP4RQVqxMXrPHp3jGpJ6IPbzDgRvMLZewzD459Hc6C4Y3I4XU4jOpbs//9QH1BJo7 7m05Y1CCHcWWw305gmsCHEoVvOFWYv5P5GSs7JaUcr5jjWIr4RyVWpXpBt7KrVsIF0A1I7 TqaopVds8R8+GpR46pOdDeAlT7nRpEM= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=YiKlu4D9; spf=pass (imf14.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 86FDDA42606; Wed, 30 Oct 2024 15:12:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A444C4CECF; Wed, 30 Oct 2024 15:14:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301248; bh=v8Tw1BMcIKkWkecbyVD2bjlFo11chEXusLhfEENOi8E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YiKlu4D9JMFXIyH5CmKlxglMedckpIJ1J4QcbKrlh88l8Zu1/lpOaJ+4iUCQ32VdQ 1/w/S5Ll4gaGts1LS/6zNykS3hYIj37SlotuNs/IhO5EgX3KFxe41mBrokI8wyI7ef Gp2Gs1Tzb7aT+be/27/nlttyo0HlPv0As+XQTeqwvrhqhAVeILkoq/Cwoi8TsWZpvI aBmJLDqQ92v/ZE4BX9FujIHUuLZQOX3O5CnOraEuYgV/e6eeIWHRzr4MvlA36P0xgm K2m6ssfopeA4RM+gktE/POocZtuUrG4q4SRR0Dqn+ON1d1X1m2VtsMaoVQg3qJTO5i ESfBwP4gKV70w== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 09/17] docs: core-api: document the IOVA-based API Date: Wed, 30 Oct 2024 17:12:55 +0200 Message-ID: <881ef0bcf9aa971e995fbdd00776c5140a7b5b3d.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2F98C100019 X-Stat-Signature: me8ekwqiisi6jdu3tmp7qpzntcna9koh X-HE-Tag: 1730301220-434284 X-HE-Meta: U2FsdGVkX19I7bJ+IOG0JjEXNON3PQSmHzhc5/tX+Sp2tO98I5335udSJLw0YIuIfQxbX7kJgi5+GnXA9dSSncTTujixyaQdZFSUWwNTlsgQBkhJdX41cPk6Lr9SClfQEq9dFt99HUNyO6q94ZEZML5HMzypqtkAHfv77dP51hVdyrm414/y9131uAMpXVg7PF72UhL3VQkmDCEAQfRYvGEu93evx8Q2cugX7T7F7oXJo+iFR7R92GpN08kB3uMuwnjvf5/tU70e8e9cWlfvluORYz2Rn8oY9gGGHmUVZBHHSxlmSwZZF8hxUXUrUUxht19A8PjftswzlnTfrcVrsTIHdg5dY4IEEUrUx69nr/LDqZyQW/Oa50t/Lxw0Rbmwa3zvcq0Pyru9gTzNmMaWkAcBym7TWdWA9GYY/UoGHC9J60Ui7YyvhbSL0Oj2LKuNmHQrv/SEI1nNI0q4uDpZmVyLv8Pr+hk5oLaJQAj52RQM+nMMB0p2DxZjzpfj7AaBDQ/7fwCWic0WDZyBzbg15r6qLwQJwr2/kZHK7uzBGZedPQxKLqVS5GqhmpI9v3hSN2sts7eJ+RvSTt0SUsVwCyQn4dIVoK3+w31b5/RDoaiSIBZWy3CBDewznBeOfKMprcoVLmNY9u4gCPV9s6zh6a0TJCxaSG9s3tQnZDWoelOTh5QfbW5ioHRoo53TdbrzyamEklm/mPKGBxxDl2Z3APHNtkQ2RuZlmQXhxpHPpQqRF2YtGzesSADjqWgpkxNj58+dgeNh9GFMpnvRH04u9EyKFa9bd5BAt/zS2kblVYrCxc1ZVEwQcQTsgcCw3uYXGhlqfGse+uJ1WsYPsYqzKn+pTl2azMhZlVnIv8RXTgWoAt0wqPiv8ZAiSuGHVeZfp8azZU8NVbza4bFUmgKOdvqJWpCzrkRu2KQuaCP1NF/ZbFJN1mW08Y5Dtfgd4BgKkUb/MmRmpqg2c2B3ODF W2pjuqwb Xs3z3Clzf+ffHwHkBWOF1E3g4rM7KcB6IyS9L1tU1wDlmCBbkAgIaIuYbOEZbTG2DZ/HG1GJFG1UC3BmG2J3vOF/A5x6kxQRSMqi6HFw6acfpXKZyaRfNrlH1TDWejDAKvA+GZ6vJszCQFkMuXdf3/UCSt6G7jyBuB18cjCp8HtOAcvcBpfpkJRMbypVffwRfksZbphyzMIa5p1sChBM8qqY/PTuYgw4QIopRTbdTsqBOnuc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Christoph Hellwig Add an explanation of the newly added IOVA-based mapping API. Signed-off-by: Christoph Hellwig Signed-off-by: Leon Romanovsky --- Documentation/core-api/dma-api.rst | 70 ++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/Documentation/core-api/dma-api.rst b/Documentation/core-api/dma-api.rst index 8e3cce3d0a23..6095696a65a7 100644 --- a/Documentation/core-api/dma-api.rst +++ b/Documentation/core-api/dma-api.rst @@ -530,6 +530,76 @@ routines, e.g.::: .... } +Part Ie - IOVA-based DMA mappings +--------------------------------- + +These APIs allow a very efficient mapping when using an IOMMU. They are an +optional path that requires extra code and are only recommended for drivers +where DMA mapping performance, or the space usage for storing the DMA addresses +matter. All the consideration from the previous section apply here as well. + +:: + + bool dma_iova_try_alloc(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t size); + +Is used to try to allocate IOVA space for mapping operation. If it returns +false this API can't be used for the given device and the normal streaming +DMA mapping API should be used. The ``struct dma_iova_state`` is allocated +by the driver and must be kept around until unmap time. + +:: + + static inline bool dma_use_iova(struct dma_iova_state *state) + +Can be used by the driver to check if the IOVA-based API is used after a +call to dma_iova_try_alloc. This can be useful in the unmap path. + +:: + + int dma_iova_link(struct device *dev, struct dma_iova_state *state, + phys_addr_t phys, size_t offset, size_t size, + enum dma_data_direction dir, unsigned long attrs); + +Is used to link ranges to the IOVA previously allocated. The start of all +but the first call to dma_iova_link for a given state must be aligned +to the DMA merge boundary returned by ``dma_get_merge_boundary())``, and +the size of all but the last range must be aligned to the DMA merge boundary +as well. + +:: + + int dma_iova_sync(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size); + +Must be called to sync the IOMMU page tables for IOVA-range mapped by one or +more calls to ``dma_iova_link()``. + +For drivers that use a one-shot mapping, all ranges can be unmapped and the +IOVA freed by calling: + +:: + + void dma_iova_destroy(struct device *dev, struct dma_iova_state *state, + enum dma_data_direction dir, unsigned long attrs); + +Alternatively drivers can dynamically manage the IOVA space by unmapping +and mapping individual regions. In that case + +:: + + void dma_iova_unlink(struct device *dev, struct dma_iova_state *state, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); + +is used to unmap a range previous mapped, and + +:: + + void dma_iova_free(struct device *dev, struct dma_iova_state *state); + +is used to free the IOVA space. All regions must have been unmapped using +``dma_iova_unlink()`` before calling ``dma_iova_free()``. Part II - Non-coherent DMA allocations -------------------------------------- From patchwork Wed Oct 30 15:12:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7DF9D5CCB3 for ; Wed, 30 Oct 2024 15:13:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74CA58E0007; Wed, 30 Oct 2024 11:13:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FA788E0001; Wed, 30 Oct 2024 11:13:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54DBE8E0007; Wed, 30 Oct 2024 11:13:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 377938E0001 for ; Wed, 30 Oct 2024 11:13:57 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BF18040DEB for ; Wed, 30 Oct 2024 15:13:56 +0000 (UTC) X-FDA: 82730613420.14.2667403 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf30.hostedemail.com (Postfix) with ESMTP id 3BA4E8002A for ; Wed, 30 Oct 2024 15:13:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=gnbQRLju; spf=pass (imf30.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301154; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tFfAVeRpTt9MOKvdbyKITP7haOM8Jd6d88FaRUe8ZOQ=; b=3IDCZecAKyVirR6qDrs4g/gzQeFE8sI55dije7xx3Ft4Rs/KdC303DHGxxbCZSdmKNmvvw 9EKABUj+W7JtrJ91Po24RQfRdoZNkWdX84x86OFgzMsYIjyn++9RVUIPy/m6qqh5Vxlm/I mrZ0D3eux1XcT/nsU0wGSrqvOeVCFYA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=gnbQRLju; spf=pass (imf30.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301154; a=rsa-sha256; cv=none; b=kYXnKQ/Duul/YK57czAJsXcknf7hZ0FVa19m4KRZLaQpXL+P6t7iOZHHwkOkZwmVtlJO+7 o5C/KK11Tpf7MPV8ixMudbLohKMS963mq2R5tCF0freCRgyP1pIg3SeE5xkffP94H8QkHR R5Q4KG7+N0akQdxbDPUKRPFJOHKoEWU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id C09D3A417D9; Wed, 30 Oct 2024 15:11:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91DDCC4CED1; Wed, 30 Oct 2024 15:13:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301234; bh=LyGbooKE736rWckmyvuR6Y2aUxxUKiIhQ25bI43LeHo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gnbQRLju1EGkWrMUq1IEnfCImY/sV9QqN3EtvIXwn6fwjiEiWAe2neBZdDCdV7yHr qcm05YMTX2HjYttwSkMdkPZu827K45fVD5z/d2S7M0nrrrP5/m9EqJ1slY09EmQN3a d2Cd6nFHM6OmwTNw2wEVvXTaKeNW4VfyiRLzEuKOs85WvxEzWcM/xRt73c66yBHci3 SKqb7lSPzJW49SZUIb5sj7omTVYloV7g4/qyWLtu5pchWIDw+VTxFln9bhApnufmPL csDsywMT+HisWLNFXI/t1D4sSzvZQOjKXw12aKaZsJqNIqTytD5aguAnmC+31kW8MT ywSaehtZC6bRw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 10/17] mm/hmm: let users to tag specific PFN with DMA mapped bit Date: Wed, 30 Oct 2024 17:12:56 +0200 Message-ID: <6ae52934e3f7652d8ae1630f9c16f2c8a15215ff.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 3BA4E8002A X-Stat-Signature: a8snbjky39mamfdyd5fi6daaytgr43pu X-HE-Tag: 1730301186-27293 X-HE-Meta: U2FsdGVkX1/1TsjLLVZiMHjOAkmGPmGJllTirGTSLsSLKymHm6lHMIu36ZExn/4Pb67bfQuHwxDBrli7rW0TBevoqY1XTslziZK2bJ/hYX0aRq2hDA174KU8t4ZUBRPkd3Pv0aIyg4IA4Gy6C3mzNdHHuBag3MjJSCKLyUOFMFCjE9Uc7N0wSNE81AUfBt+De+SV5SX9v/3XjfSek0J9huHMlxazYL+XCzcXf0UzSf8Jg7Vju0WVoeacGFga7AugxTVDBNNHgeg1UX2TrLIk4kq0hzxuOiw2KHBa2N+GrlUZKDxZLY4IM3q8gcyRkcba6a2MU5x3Oy4+aBM/2DellDUlHAci8dJcw9tD9xPltXyS/azEcJXBVmD0W3foSr8HTCpXccp/7J0kfblOXSe9YeoBEaIB3J5VoKDmL3C8lqcU+dKa0m2SmGUHBmIG+blNyitVnw9hlKfVn9FPWLUPQdjYOIgye/UCCnMTctCjWmcrkzF9AqBS2HBIChfjyaqW00QKHULw4+yaMAxVq2zLvN1zbKdG+484a383ZAshwgEOO0Vc624Ae56cipie43haMOj6JO1ECYJyFevFOQZpU82sPef/ZzR+Xrwv/h0WfLeoVzI9l+jALCA7e/9B6BLcO/WLQFsIGOvKRnJqh57zOpb7cfWYMLtFF8dHSSW1TU++9e0HhDrVUVyhN9nIxtcOlQ0rW11QQlJkWj+uDODYn3C0vo1szuZKKiXp/bmTwpR+7j01kxrGft2zYjRT5r8Q8fHz1Dt0LIPeyK3+Qi4ykYSOMry1A7sQ66JK19GGHhLX3+97NQm3ORHa0WUFhDtO/Yoipp6duNXP3ijMs8o9LS7LXdvJyefsqjh7TS+W8XnPtqksmpXBuwjxkfWUTXlEf84tn2fqjPUwUw+aSBu88PCnUu+J92VgaOpAyC7E7UfR1lljVjydQPXHI32BTtmy0ef/ZTf2S73HiGSeUcL zTXS8LLX BmnyPNG5HoZspO3QJUvxyzkdQvFUnUQzTkeJU6YCONDbJ0RHbdU15KVlsEZQwDW/OIkh/nUTDhS4BExrH3n6q5qAu6oAozbMBCsuKeR4wldxXMCUMxiMvsJcoi7HI2o019Gh7gv+WII674Le07hmrtorSCyn3rD580ZlcoNHihvWEmBgCkWsYTgQ+u72ralMwbjHLwRspsjsChKPOfy3hPdm8NO6hJplWWM8GRQm7pCkl1Rk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Introduce new sticky flag (HMM_PFN_DMA_MAPPED), which isn't overwritten by HMM range fault. Such flag allows users to tag specific PFNs with information if this specific PFN was already DMA mapped. Signed-off-by: Leon Romanovsky --- include/linux/hmm.h | 14 ++++++++++++++ mm/hmm.c | 34 +++++++++++++++++++++------------- 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..5dd655f6766b 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,8 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation + * to mark that page is already DMA mapped * * On input: * 0 - Return the current state of the page, do not fault it. @@ -36,6 +38,10 @@ enum hmm_pfn_flags { HMM_PFN_VALID = 1UL << (BITS_PER_LONG - 1), HMM_PFN_WRITE = 1UL << (BITS_PER_LONG - 2), HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3), + + /* Sticky flag, carried from Input to Output */ + HMM_PFN_DMA_MAPPED = 1UL << (BITS_PER_LONG - 7), + HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8), /* Input flags */ @@ -57,6 +63,14 @@ static inline struct page *hmm_pfn_to_page(unsigned long hmm_pfn) return pfn_to_page(hmm_pfn & ~HMM_PFN_FLAGS); } +/* + * hmm_pfn_to_phys() - return physical address pointed to by a device entry + */ +static inline phys_addr_t hmm_pfn_to_phys(unsigned long hmm_pfn) +{ + return __pfn_to_phys(hmm_pfn & ~HMM_PFN_FLAGS); +} + /* * hmm_pfn_to_map_order() - return the CPU mapping size order * diff --git a/mm/hmm.c b/mm/hmm.c index 7e0229ae4a5a..2a0c34d7cb2b 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -44,8 +44,10 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, { unsigned long i = (addr - range->start) >> PAGE_SHIFT; - for (; addr < end; addr += PAGE_SIZE, i++) - range->hmm_pfns[i] = cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++) { + range->hmm_pfns[i] &= HMM_PFN_DMA_MAPPED; + range->hmm_pfns[i] |= cpu_flags; + } return 0; } @@ -202,8 +204,10 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, return hmm_vma_fault(addr, end, required_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { + hmm_pfns[i] &= HMM_PFN_DMA_MAPPED; + hmm_pfns[i] |= pfn | cpu_flags; + } return 0; } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -236,7 +240,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) goto fault; - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_DMA_MAPPED; return 0; } @@ -253,14 +257,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |= HMM_PFN_WRITE; - *hmm_pfn = swp_offset_pfn(entry) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_DMA_MAPPED) | swp_offset_pfn(entry) | cpu_flags; return 0; } required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (!required_fault) { - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_DMA_MAPPED; return 0; } @@ -304,11 +308,11 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, pte_unmap(ptep); return -EFAULT; } - *hmm_pfn = HMM_PFN_ERROR; + *hmm_pfn = (*hmm_pfn & HMM_PFN_DMA_MAPPED) | HMM_PFN_ERROR; return 0; } - *hmm_pfn = pte_pfn(pte) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_DMA_MAPPED) | pte_pfn(pte) | cpu_flags; return 0; fault: @@ -448,8 +452,10 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - for (i = 0; i < npages; ++i, ++pfn) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; i < npages; ++i, ++pfn) { + hmm_pfns[i] &= HMM_PFN_DMA_MAPPED; + hmm_pfns[i] |= pfn | cpu_flags; + } goto out_unlock; } @@ -507,8 +513,10 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); - for (; addr < end; addr += PAGE_SIZE, i++, pfn++) - range->hmm_pfns[i] = pfn | cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++, pfn++) { + range->hmm_pfns[i] &= HMM_PFN_DMA_MAPPED; + range->hmm_pfns[i] |= pfn | cpu_flags; + } spin_unlock(ptl); return 0; From patchwork Wed Oct 30 15:12:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DC52D5CCB3 for ; Wed, 30 Oct 2024 15:14:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF5158E0008; Wed, 30 Oct 2024 11:14:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA4C28E0001; Wed, 30 Oct 2024 11:14:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91F728E0008; Wed, 30 Oct 2024 11:14:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6AEE48E0001 for ; Wed, 30 Oct 2024 11:14:00 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 21C5DA0D56 for ; Wed, 30 Oct 2024 15:14:00 +0000 (UTC) X-FDA: 82730613546.02.72E0E0A Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf07.hostedemail.com (Postfix) with ESMTP id 62E4F40021 for ; Wed, 30 Oct 2024 15:13:22 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Y324KZxb; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf07.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301108; a=rsa-sha256; cv=none; b=1RpjroQJoDTc4LOc/R3lRRPzLt2TKfBZxLaG6qlhmSFJSUScsexEUKKCr+w9LUlJ+DLspk cmIsIl3lda4D4Jfc4RWmCjs3DJXqnsQbkZnEpL303KcS+nR6+y4af3zGjjnUqLlEdTmlpG 8ivk8k8xh7+Ay+CpyN8cEwugT9NUM3o= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Y324KZxb; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf07.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301108; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GQ0bw/NEB2jBz2xPBGymFBm8toO4/v03Hw2X/I1dfQc=; b=W9TN4acU9OJKJgjHI4WU0JSeCqv0jdla4wvU+niLK3PwMnmCOVqcZe3bCgcFnPVtNSq7It dDwEuKLm5inPEAJQVoZvoAfUjNI9VF6aI7CuhJ8w12qhOoIAKwQeXayD6cvAkx9szb93yw ehrLXg20pqxfMMIKbO4oqAn87X4N+G0= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 2CA7DA418C0; Wed, 30 Oct 2024 15:12:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DEF74C4CED1; Wed, 30 Oct 2024 15:13:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301237; bh=87gpgRHIgXynVPMHMmjNp7ijROtjbPM+GbsYeyGrfQ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y324KZxbya3gSzwXzEuzoqUy3uEOTBvdqYmb4j57b954TWYJQTu2JBW9o6T5zzp+p YH2Ecop7h+tWnG3KasQOK1qxsPklEnDAHDfKyist8ki/3tStjtLM96QCQ34jF2bqG1 aVwKEBvysGIDSA3MdwJ4Ijd5OnDIzYg/39ZOXpNHgb0ATgjBGw6stBGeTjYD0fM81C qir52cXG0CqX7r0igDbdFaatlUENXehQeloTJAgCy1zAyQUVY1kwVaWpDJ5TvD4ytE Y3XcMfS1yaS2murETzmKjGMpIbtJQjeGiW8ei53Cr4wTQp2Gwxx9wyBg3Y59SDJmF8 kyRxG6tbZJMTw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 11/17] mm/hmm: provide generic DMA managing logic Date: Wed, 30 Oct 2024 17:12:57 +0200 Message-ID: <3cf57ecd01f42e1fe181329e65b95b4dacbb9443.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 62E4F40021 X-Stat-Signature: zgkpuoi9y9cifqpnenmxhbq4tjt8tusb X-Rspam-User: X-HE-Tag: 1730301202-756333 X-HE-Meta: U2FsdGVkX18kTr962j9wsYZzwmBdPGe6AcUvR9v2KUg1fmZK85ATxqgDu24LBHIubRumKu+t7Gh7X0g8Av9DiBL45s+iXCa37FAIh2RX/raeusrxOIgMrVP2qLQmtLbryD6nIdKGLbaHg/CVfFDQWjAxPuQ23cFPufsHwp2wHR9ZyABZiF1b/lOdFpWZuZ6teCqQMnpJ4+Rf+p3++FwD5EB5SUZUsgiQyjyGQSWz5KrGHWn1eACSdG+Lmz5lPXoQIxNRGcf+9IMVLtXEWfwK0X8d23jOtsb17v2b9msz57dvEvRf1+vUwi1ygJYUq9uzkMMKsAQ59CahYdienzO55ZAkiHxw3cIxvy/uvj6e5CRJL9ElalnpMAj4zPWZcwxjnRPtiQUtspWPCDLp1QFrswFUUp6dx7vov5m9jCd45zmifGrv7bWxM+7Qi2V5zYD8TaHERDBB2reDqFHQIG1yJvzOgH0e2Rui7JopYqMbixXZxaHB1673wVHvGOYCQPvpWoy7RtVxVR9JLi0/5t00HhJQR7BjLxoC3qMVMpA6gJbFg9H3EUgwr8Te6iNi7vUFg3wphVWysF8J8QDl9Ryd3ZqqQOw5HSbAeDcMSwWLNGZWM/BJ2pXHHmGdqesoC2O8RurDVX4de41SEzbwDAnxVkDni191rYGaUOIcAquPTmkg5ETmUnWx4Ik2btQOFs+UA0VF6NoSRdxqNXkylOdVZwWwnksqYLC83QPL9MJyaqPmE22nqnsNVzvXVODeeqGh/3NAxQO9RgAEDq5lvuUjzNdWBlxHzP9Pbg8jOFdyGjgyJiQsCIllMKmo2H2b52m0YCQq/AnLGtDa6LcXX0t4jV/0KelZez4eQcEPcFnun+L420H+DF7s5IwObuGT78odn3UVWfV+i7dGA39iqBByY0pR1k1FRwLEU9jqk1+tR8Ltf5A/ZPxzFLLfr/2ol0Luk0F+dV6RHvqAYHZwfAm E7yHDG1Y Mph7vfKRfVgVOoOqktZDJdye/AGsJRxeZio/9nWOQMGOVlNAVRal3tzmOca5JnpdRp0EYRhmoHa2hYth7LonaDeYfGnHfFbvNrOG749Z23o9scpFKvtCISnU3KfI7j7I9xRQQMLpLAyGSp0MTpsgmj/eZdmCfU8mdWTNAqh4pIXwChfINnFiWPemC1uSctUPC60Ht0TOxyNqx67ddp79N9NOsgS7Fs1Kz/oEPBUP7dvegQG4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky HMM callers use PFN list to populate range while calling to hmm_range_fault(), the conversion from PFN to DMA address is done by the callers with help of another DMA list. However, it is wasteful on any modern platform and by doing the right logic, that DMA list can be avoided. Provide generic logic to manage these lists and gave an interface to map/unmap PFNs to DMA addresses, without requiring from the callers to be an experts in DMA core API. Signed-off-by: Leon Romanovsky --- include/linux/hmm-dma.h | 32 +++++++ include/linux/hmm.h | 2 + mm/hmm.c | 197 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 231 insertions(+) create mode 100644 include/linux/hmm-dma.h diff --git a/include/linux/hmm-dma.h b/include/linux/hmm-dma.h new file mode 100644 index 000000000000..f6ce2a00d74d --- /dev/null +++ b/include/linux/hmm-dma.h @@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* Copyright (c) 2024 NVIDIA Corporation & Affiliates */ +#ifndef LINUX_HMM_DMA_H +#define LINUX_HMM_DMA_H + +#include + +struct dma_iova_state; +struct pci_p2pdma_map_state; + +/* + * struct hmm_dma_map - array of PFNs and DMA addresses + * + * @state: DMA IOVA state + * @pfns: array of PFNs + * @dma_list: array of DMA addresses + * @dma_entry_size: size of each DMA entry in the array + */ +struct hmm_dma_map { + struct dma_iova_state state; + unsigned long *pfn_list; + dma_addr_t *dma_list; + size_t dma_entry_size; +}; + +int hmm_dma_map_alloc(struct device *dev, struct hmm_dma_map *map, + size_t nr_entries, size_t dma_entry_size); +void hmm_dma_map_free(struct device *dev, struct hmm_dma_map *map); +dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, + size_t idx, struct pci_p2pdma_map_state *p2pdma_state); +bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx); +#endif /* LINUX_HMM_DMA_H */ diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 5dd655f6766b..62980ca8f3c5 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,7 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_P2PDMA_BUS - Bus mapped P2P transfer * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped * @@ -40,6 +41,7 @@ enum hmm_pfn_flags { HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3), /* Sticky flag, carried from Input to Output */ + HMM_PFN_P2PDMA_BUS = 1UL << (BITS_PER_LONG - 6), HMM_PFN_DMA_MAPPED = 1UL << (BITS_PER_LONG - 7), HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8), diff --git a/mm/hmm.c b/mm/hmm.c index 2a0c34d7cb2b..a852d8337c73 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -10,6 +10,7 @@ */ #include #include +#include #include #include #include @@ -23,6 +24,7 @@ #include #include #include +#include #include #include @@ -615,3 +617,198 @@ int hmm_range_fault(struct hmm_range *range) return ret; } EXPORT_SYMBOL(hmm_range_fault); + +/** + * hmm_dma_map_alloc - Allocate HMM map structure + * @dev: device to allocate structure for + * @map: HMM map to allocate + * @nr_entries: number of entries in the map + * @dma_entry_size: size of the DMA entry in the map + * + * Allocate the HMM map structure and all the lists it contains. + * Return 0 on success, -ENOMEM on failure. + */ +int hmm_dma_map_alloc(struct device *dev, struct hmm_dma_map *map, + size_t nr_entries, size_t dma_entry_size) +{ + bool dma_need_sync = false; + bool use_iova; + + if (!(nr_entries * PAGE_SIZE / dma_entry_size)) + return -EINVAL; + + /* + * The HMM API violates our normal DMA buffer ownership rules and can't + * transfer buffer ownership. The dma_addressing_limited() check is a + * best approximation to ensure no swiotlb buffering happens. + */ + if (IS_ENABLED(CONFIG_DMA_NEED_SYNC)) + dma_need_sync = !dev->dma_skip_sync; + if (dma_need_sync || dma_addressing_limited(dev)) + return -EOPNOTSUPP; + + map->dma_entry_size = dma_entry_size; + map->pfn_list = + kvcalloc(nr_entries, sizeof(*map->pfn_list), GFP_KERNEL); + if (!map->pfn_list) + return -ENOMEM; + + use_iova = dma_iova_try_alloc(dev, &map->state, 0, + nr_entries * PAGE_SIZE); + if (!use_iova && dma_need_unmap(dev)) { + map->dma_list = kvcalloc(nr_entries, sizeof(*map->dma_list), + GFP_KERNEL); + if (!map->dma_list) + goto err_dma; + } + return 0; + +err_dma: + kfree(map->pfn_list); + return -ENOMEM; +} +EXPORT_SYMBOL_GPL(hmm_dma_map_alloc); + +/** + * hmm_dma_map_free - iFree HMM map structure + * @dev: device to free structure from + * @map: HMM map containing the various lists and state + * + * Free the HMM map structure and all the lists it contains. + */ +void hmm_dma_map_free(struct device *dev, struct hmm_dma_map *map) +{ + if (dma_use_iova(&map->state)) + dma_iova_free(dev, &map->state); + kfree(map->pfn_list); + kfree(map->dma_list); +} +EXPORT_SYMBOL_GPL(hmm_dma_map_free); + +/** + * hmm_dma_map_pfn - Map a physical HMM page to DMA address + * @dev: Device to map the page for + * @map: HMM map + * @idx: Index into the PFN and dma address arrays + * @pci_p2pdma_map_state: PCI P2P state. + * + * dma_alloc_iova() allocates IOVA based on the size specified by their use in + * iova->size. Call this function after IOVA allocation to link whole @page + * to get the DMA address. Note that very first call to this function + * will have @offset set to 0 in the IOVA space allocated from + * dma_alloc_iova(). For subsequent calls to this function on same @iova, + * @offset needs to be advanced by the caller with the size of previous + * page that was linked + DMA address returned for the previous page that was + * linked by this function. + */ +dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, + size_t idx, struct pci_p2pdma_map_state *p2pdma_state) +{ + struct dma_iova_state *state = &map->state; + dma_addr_t *dma_addrs = map->dma_list; + unsigned long *pfns = map->pfn_list; + struct page *page = hmm_pfn_to_page(pfns[idx]); + phys_addr_t paddr = hmm_pfn_to_phys(pfns[idx]); + size_t offset = idx * map->dma_entry_size; + dma_addr_t dma_addr; + int ret; + + if ((pfns[idx] & HMM_PFN_DMA_MAPPED) && + !(pfns[idx] & HMM_PFN_P2PDMA_BUS)) { + /* + * We are in this flow when there is a need to resync flags, + * for example when page was already linked in prefetch call + * with READ flag and now we need to add WRITE flag + * + * This page was already programmed to HW and we don't want/need + * to unlink and link it again just to resync flags. + */ + if (dma_use_iova(state)) + return state->addr + offset; + + /* + * Without dma_need_unmap, the dma_addrs array is NULL, thus we + * need to regenerate the address below even if there already + * was a mapping. But !dma_need_unmap implies that the + * mapping stateless, so this is fine. + */ + if (dma_need_unmap(dev)) + return dma_addrs[idx]; + + /* Continue to remapping */ + } + + switch (pci_p2pdma_state(p2pdma_state, dev, page)) { + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + case PCI_P2PDMA_MAP_NONE: + break; + case PCI_P2PDMA_MAP_BUS_ADDR: + dma_addr = pci_p2pdma_bus_addr_map(p2pdma_state, paddr); + pfns[idx] |= HMM_PFN_P2PDMA_BUS; + goto done; + default: + return DMA_MAPPING_ERROR; + } + + if (dma_use_iova(state)) { + ret = dma_iova_link(dev, state, paddr, offset, + map->dma_entry_size, DMA_BIDIRECTIONAL, 0); + if (ret) + return DMA_MAPPING_ERROR; + + ret = dma_iova_sync(dev, state, offset, map->dma_entry_size); + if (ret) + return DMA_MAPPING_ERROR; + + dma_addr = state->addr + offset; + } else { + if (WARN_ON_ONCE(dma_need_unmap(dev) && !dma_addrs)) + return DMA_MAPPING_ERROR; + + dma_addr = dma_map_page(dev, page, 0, map->dma_entry_size, + DMA_BIDIRECTIONAL); + if (dma_mapping_error(dev, dma_addr)) + return DMA_MAPPING_ERROR; + + if (dma_need_unmap(dev)) + dma_addrs[idx] = dma_addr; + } + +done: + pfns[idx] |= HMM_PFN_DMA_MAPPED; + return dma_addr; +} +EXPORT_SYMBOL_GPL(hmm_dma_map_pfn); + +/** + * hmm_dma_unmap_pfn - Unmap a physical HMM page from DMA address + * @dev: Device to unmap the page from + * @map: HMM map + * @idx: Index of the PFN to unmap + * + * Returns true if the PFN was mapped and has been unmapped, false otherwise. + */ +bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx) +{ + struct dma_iova_state *state = &map->state; + dma_addr_t *dma_addrs = map->dma_list; + unsigned long *pfns = map->pfn_list; + +#define HMM_PFN_VALID_DMA (HMM_PFN_VALID | HMM_PFN_DMA_MAPPED) + if ((pfns[idx] & HMM_PFN_VALID_DMA) != HMM_PFN_VALID_DMA) + return false; +#undef HMM_PFN_VALID_DMA + + if (pfns[idx] & HMM_PFN_P2PDMA_BUS) + ; /* no need to unmap bus address P2P mappings */ + else if (dma_use_iova(state)) + dma_iova_unlink(dev, state, idx * map->dma_entry_size, + map->dma_entry_size, DMA_BIDIRECTIONAL, 0); + else if (dma_need_unmap(dev)) + dma_unmap_page(dev, dma_addrs[idx], map->dma_entry_size, + DMA_BIDIRECTIONAL); + + pfns[idx] &= ~(HMM_PFN_DMA_MAPPED | HMM_PFN_P2PDMA_BUS); + return true; +} +EXPORT_SYMBOL_GPL(hmm_dma_unmap_pfn); From patchwork Wed Oct 30 15:12:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 318E9D5CCB8 for ; Wed, 30 Oct 2024 15:14:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB8FD6B009F; Wed, 30 Oct 2024 11:14:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1AC36B00A1; Wed, 30 Oct 2024 11:14:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96E086B00A5; Wed, 30 Oct 2024 11:14:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 747376B009F for ; Wed, 30 Oct 2024 11:14:04 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BDA6CC0D81 for ; Wed, 30 Oct 2024 15:14:03 +0000 (UTC) X-FDA: 82730612832.08.EE29D54 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf11.hostedemail.com (Postfix) with ESMTP id 596B340027 for ; Wed, 30 Oct 2024 15:13:30 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QyNtuVUV; spf=pass (imf11.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zRilljuSlrHZeowBJUFk5kMExeeLfbXBNqxkEwyETU4=; b=XyXvcG2jEqwNIaFvwXex/IBmF0FVcjoVPEDrqTlD738JXCx/nlRBARtNU8XurZDUyncqzs fU7mrmZAjY4hrPOUrtclK9eQFzs0nxP1rlKqaIXqL2Q6VjjrSRcCbOFB/9WzVa6e4kMNu7 nGYW0TJSPH4m1c0MrGMCXdpCm/WBPQs= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=QyNtuVUV; spf=pass (imf11.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301029; a=rsa-sha256; cv=none; b=XezA1NnJcGHwt58LCVk7F3ro+gVn5S/2nLiX6iw2oRAsagnE2pEAU7o8XgwQLcPE82Qy+E oSwSa9l+dSsWKHwKgxusiAPmzy35Fyu5Rq+lE0/0DxoAcKAbBePXX0Ycsk5Q+UE+NWnFX/ 9PGs4VnZmIspciFqwjB+ByqryYRN+wA= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 9C7A9A4253F; Wed, 30 Oct 2024 15:12:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 70575C4CECF; Wed, 30 Oct 2024 15:14:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301241; bh=z2vnn5LCpeU9AGWM/Q4PQinA49Qn1NqHhrSnH32MJZk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QyNtuVUVzDsGpFLens+lziwIR5Dqzp/HXH0ru1Tl7fia+0ut+1aspiZrjYhUHFY8K PaGMNG3DaF8rBbPbK2r6a10ukog3NvLjdJJchmFXJQS/WK8sZosijyOiDQxfPlaRhi 1CVs0tTENvHQ4NiKgnBDId8ADYHYcqZVQ+TRy+ksF+t0gf+h1G2hkSHDDCsLLAfiI8 61udBG5X8vFA4ZPNENbT/Q2jYnqbdD4xvuux6AcRdoBeVEOEpAHDM0FMax434Ik8Ml Eu/1rCtMjLLvH0pImbndFAgXqziqSM4UodBiAvxuq4uvO28sY1dVcWaqxdgBzGlj+Y NyRcq3zum7Lbg== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 12/17] RDMA/umem: Store ODP access mask information in PFN Date: Wed, 30 Oct 2024 17:12:58 +0200 Message-ID: <445fb59878214beec591ddeb88ade724e01cb3ca.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 596B340027 X-Stat-Signature: 3b1dcu1ouxe4bnoezwkakgsar3keoirw X-Rspam-User: X-HE-Tag: 1730301210-128756 X-HE-Meta: U2FsdGVkX18p5tLseH5M+EucZmlDbf4hJtgPl5D02Ebv2T38DQlMzYwEuU9XRCfta+DvmCKcB3zFtkwigYIMXqxW4h6ZDCWE0NtYRDlPOiVAajA7FjZQqpggs19UIMifiuqV7Z2Yq8nxT2PoWGXsz/8/4wMqQ38wDQjeKX/Cu4IS/VwhlQNkM6w176KPVB5tk+SptlXvETLU/9xmjO+Hce+w9JifbbrEsPRFD4lpUDSjyk6WtDNL5yYr7hnCg72mlqYjznPHIfan5QpXDbv/mlKn0G8tHZwD4lbaOh7o2MVSylCmeFF2bY7to0tEq38k2wcH1lLSTnmumYsyGs06p33QaIPE7iHD0m8IdlXQkSby2EubeBMBfpx2qVOYkIzT+5s2bgrCYhPJyuJ5bMLy02+yVdMkS900xQIwMKHnJlNoXV8XHAhTwqGm5Jp31cu1oPL0/ktNZL5h4t2Pf1FyvKs2zMPkSn0VrWOGu8dqrNpXbh0NpqKa9gbPuWFy21H1LaVkZKQpfLZS6soY8dNb4BDygUKtC0z3aac1QabyEp0kcXYf6YuXp8x6dmrHWu+LEV9a5W0a01oBN9j27D2k4Sv6CXDuIJca18Vd/RpbnKWeLdZzsi5AGC9jR2J9tclTWU/EtPbOJnPxPIwjoBlzULoQTrEExeud57Cs0r+S/GMVGem3PXLr3TcVeF6P4UzAiCQyUqAtLDnPguiM40xf9Bs109agt5IxpCU+DxwP05RyaXU9hP32Ng/yN0N7OMrtuQMPvpyIBqLPn7j7SRv5zRtuHO5a3627F4ULZkTgwTegnwBal4nBxgy9GsAdkyLHaXNsA0FyFGtXEiwhX+5ooL9/B6ziTHpSgxLQH2tLKlHjXtL9JgvfT6OPISTn2y0Oneo9qfv6zGEfSh2IPGoKBSCmdhkaVSzwiojXL5Qgdv18hQSNOWXfvl1fS2GV4O3g6FxryMjgoPd/13kfqHn GaB8cyKC /wdMDqPlZaQ1N3m3wpT5JKZp+KV2mvybN/Yw+O+g0MH9u1WmW2JcjcHKqs5SDhIxBRH3WJUhcobdvxqrdIUkxoiOmRqAa/sfYOncrYGukHlN3JlNfDgV3b4YvBDM6uVV3UxYYGOt32bervcPsNNIFjgfVY+LZ9CxMMkbaf47CnTMpzZqp1UbsL0cOOBnPjJsirL03iPxmgsvJovwHy1RRjqlCUzBDPqvWZJxosPCIrVSIUv8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky As a preparation to remove dma_list, store access mask in PFN pointer and not in dma_addr_t. Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 100 +++++++++++---------------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/odp.c | 37 +++++----- include/rdma/ib_umem_odp.h | 14 +--- 4 files changed, 61 insertions(+), 91 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index e9fa22d31c23..9dba369365af 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -296,22 +296,11 @@ EXPORT_SYMBOL(ib_umem_odp_release); static int ib_umem_odp_map_dma_single_page( struct ib_umem_odp *umem_odp, unsigned int dma_index, - struct page *page, - u64 access_mask) + struct page *page) { struct ib_device *dev = umem_odp->umem.ibdev; dma_addr_t *dma_addr = &umem_odp->dma_list[dma_index]; - if (*dma_addr) { - /* - * If the page is already dma mapped it means it went through - * a non-invalidating trasition, like read-only to writable. - * Resync the flags. - */ - *dma_addr = (*dma_addr & ODP_DMA_ADDR_MASK) | access_mask; - return 0; - } - *dma_addr = ib_dma_map_page(dev, page, 0, 1 << umem_odp->page_shift, DMA_BIDIRECTIONAL); if (ib_dma_mapping_error(dev, *dma_addr)) { @@ -319,7 +308,6 @@ static int ib_umem_odp_map_dma_single_page( return -EFAULT; } umem_odp->npages++; - *dma_addr |= access_mask; return 0; } @@ -355,9 +343,6 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, struct hmm_range range = {}; unsigned long timeout; - if (access_mask == 0) - return -EINVAL; - if (user_virt < ib_umem_start(umem_odp) || user_virt + bcnt > ib_umem_end(umem_odp)) return -EFAULT; @@ -383,7 +368,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, if (fault) { range.default_flags = HMM_PFN_REQ_FAULT; - if (access_mask & ODP_WRITE_ALLOWED_BIT) + if (access_mask & HMM_PFN_WRITE) range.default_flags |= HMM_PFN_REQ_WRITE; } @@ -415,22 +400,17 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, for (pfn_index = 0; pfn_index < num_pfns; pfn_index += 1 << (page_shift - PAGE_SHIFT), dma_index++) { - if (fault) { - /* - * Since we asked for hmm_range_fault() to populate - * pages it shouldn't return an error entry on success. - */ - WARN_ON(range.hmm_pfns[pfn_index] & HMM_PFN_ERROR); - WARN_ON(!(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)); - } else { - if (!(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)) { - WARN_ON(umem_odp->dma_list[dma_index]); - continue; - } - access_mask = ODP_READ_ALLOWED_BIT; - if (range.hmm_pfns[pfn_index] & HMM_PFN_WRITE) - access_mask |= ODP_WRITE_ALLOWED_BIT; - } + /* + * Since we asked for hmm_range_fault() to populate + * pages it shouldn't return an error entry on success. + */ + WARN_ON(fault && range.hmm_pfns[pfn_index] & HMM_PFN_ERROR); + WARN_ON(fault && !(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)); + if (!(range.hmm_pfns[pfn_index] & HMM_PFN_VALID)) + continue; + + if (range.hmm_pfns[pfn_index] & HMM_PFN_DMA_MAPPED) + continue; hmm_order = hmm_pfn_to_map_order(range.hmm_pfns[pfn_index]); /* If a hugepage was detected and ODP wasn't set for, the umem @@ -445,13 +425,13 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, } ret = ib_umem_odp_map_dma_single_page( - umem_odp, dma_index, hmm_pfn_to_page(range.hmm_pfns[pfn_index]), - access_mask); + umem_odp, dma_index, hmm_pfn_to_page(range.hmm_pfns[pfn_index])); if (ret < 0) { ibdev_dbg(umem_odp->umem.ibdev, "ib_umem_odp_map_dma_single_page failed with error %d\n", ret); break; } + range.hmm_pfns[pfn_index] |= HMM_PFN_DMA_MAPPED; } /* upon success lock should stay on hold for the callee */ if (!ret) @@ -471,7 +451,6 @@ EXPORT_SYMBOL(ib_umem_odp_map_dma_and_lock); void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, u64 bound) { - dma_addr_t dma_addr; dma_addr_t dma; int idx; u64 addr; @@ -482,34 +461,35 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, virt = max_t(u64, virt, ib_umem_start(umem_odp)); bound = min_t(u64, bound, ib_umem_end(umem_odp)); for (addr = virt; addr < bound; addr += BIT(umem_odp->page_shift)) { + unsigned long pfn_idx = (addr - ib_umem_start(umem_odp)) >> PAGE_SHIFT; + struct page *page = hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); + idx = (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; dma = umem_odp->dma_list[idx]; - /* The access flags guaranteed a valid DMA address in case was NULL */ - if (dma) { - unsigned long pfn_idx = (addr - ib_umem_start(umem_odp)) >> PAGE_SHIFT; - struct page *page = hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); - - dma_addr = dma & ODP_DMA_ADDR_MASK; - ib_dma_unmap_page(dev, dma_addr, - BIT(umem_odp->page_shift), - DMA_BIDIRECTIONAL); - if (dma & ODP_WRITE_ALLOWED_BIT) { - struct page *head_page = compound_head(page); - /* - * set_page_dirty prefers being called with - * the page lock. However, MMU notifiers are - * called sometimes with and sometimes without - * the lock. We rely on the umem_mutex instead - * to prevent other mmu notifiers from - * continuing and allowing the page mapping to - * be removed. - */ - set_page_dirty(head_page); - } - umem_odp->dma_list[idx] = 0; - umem_odp->npages--; + if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_VALID)) + goto clear; + if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_DMA_MAPPED)) + goto clear; + + ib_dma_unmap_page(dev, dma, BIT(umem_odp->page_shift), + DMA_BIDIRECTIONAL); + if (umem_odp->pfn_list[pfn_idx] & HMM_PFN_WRITE) { + struct page *head_page = compound_head(page); + /* + * set_page_dirty prefers being called with + * the page lock. However, MMU notifiers are + * called sometimes with and sometimes without + * the lock. We rely on the umem_mutex instead + * to prevent other mmu notifiers from + * continuing and allowing the page mapping to + * be removed. + */ + set_page_dirty(head_page); } + umem_odp->npages--; +clear: + umem_odp->pfn_list[pfn_idx] &= ~HMM_PFN_FLAGS; } } EXPORT_SYMBOL(ib_umem_odp_unmap_dma_pages); diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 23fd72f7f63d..3e4aaa6319db 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -336,6 +336,7 @@ struct mlx5_ib_flow_db { #define MLX5_IB_UPD_XLT_PD BIT(4) #define MLX5_IB_UPD_XLT_ACCESS BIT(5) #define MLX5_IB_UPD_XLT_INDIRECT BIT(6) +#define MLX5_IB_UPD_XLT_DOWNGRADE BIT(7) /* Private QP creation flags to be passed in ib_qp_init_attr.create_flags. * diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 4b37446758fd..78887500ce15 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -34,6 +34,7 @@ #include #include #include +#include #include "mlx5_ib.h" #include "cmd.h" @@ -158,22 +159,12 @@ static void populate_klm(struct mlx5_klm *pklm, size_t idx, size_t nentries, } } -static u64 umem_dma_to_mtt(dma_addr_t umem_dma) -{ - u64 mtt_entry = umem_dma & ODP_DMA_ADDR_MASK; - - if (umem_dma & ODP_READ_ALLOWED_BIT) - mtt_entry |= MLX5_IB_MTT_READ; - if (umem_dma & ODP_WRITE_ALLOWED_BIT) - mtt_entry |= MLX5_IB_MTT_WRITE; - - return mtt_entry; -} - static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, struct mlx5_ib_mr *mr, int flags) { struct ib_umem_odp *odp = to_ib_umem_odp(mr->umem); + bool downgrade = flags & MLX5_IB_UPD_XLT_DOWNGRADE; + unsigned long pfn; dma_addr_t pa; size_t i; @@ -181,8 +172,17 @@ static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, return; for (i = 0; i < nentries; i++) { + pfn = odp->pfn_list[idx + i]; + if (!(pfn & HMM_PFN_VALID)) + /* ODP initialization */ + continue; + pa = odp->dma_list[idx + i]; - pas[i] = cpu_to_be64(umem_dma_to_mtt(pa)); + pa |= MLX5_IB_MTT_READ; + if ((pfn & HMM_PFN_WRITE) && !downgrade) + pa |= MLX5_IB_MTT_WRITE; + + pas[i] = cpu_to_be64(pa); } } @@ -286,8 +286,7 @@ static bool mlx5_ib_invalidate_range(struct mmu_interval_notifier *mni, * estimate the cost of another UMR vs. the cost of bigger * UMR. */ - if (umem_odp->dma_list[idx] & - (ODP_READ_ALLOWED_BIT | ODP_WRITE_ALLOWED_BIT)) { + if (umem_odp->pfn_list[idx] & HMM_PFN_VALID) { if (!in_block) { blk_start_idx = idx; in_block = 1; @@ -668,7 +667,7 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp, { int page_shift, ret, np; bool downgrade = flags & MLX5_PF_FLAGS_DOWNGRADE; - u64 access_mask; + u64 access_mask = 0; u64 start_idx; bool fault = !(flags & MLX5_PF_FLAGS_SNAPSHOT); u32 xlt_flags = MLX5_IB_UPD_XLT_ATOMIC; @@ -676,12 +675,14 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp, if (flags & MLX5_PF_FLAGS_ENABLE) xlt_flags |= MLX5_IB_UPD_XLT_ENABLE; + if (flags & MLX5_PF_FLAGS_DOWNGRADE) + xlt_flags |= MLX5_IB_UPD_XLT_DOWNGRADE; + page_shift = odp->page_shift; start_idx = (user_va - ib_umem_start(odp)) >> page_shift; - access_mask = ODP_READ_ALLOWED_BIT; if (odp->umem.writable && !downgrade) - access_mask |= ODP_WRITE_ALLOWED_BIT; + access_mask |= HMM_PFN_WRITE; np = ib_umem_odp_map_dma_and_lock(odp, user_va, bcnt, access_mask, fault); if (np < 0) diff --git a/include/rdma/ib_umem_odp.h b/include/rdma/ib_umem_odp.h index 0844c1d05ac6..a345c26a745d 100644 --- a/include/rdma/ib_umem_odp.h +++ b/include/rdma/ib_umem_odp.h @@ -8,6 +8,7 @@ #include #include +#include struct ib_umem_odp { struct ib_umem umem; @@ -67,19 +68,6 @@ static inline size_t ib_umem_odp_num_pages(struct ib_umem_odp *umem_odp) umem_odp->page_shift; } -/* - * The lower 2 bits of the DMA address signal the R/W permissions for - * the entry. To upgrade the permissions, provide the appropriate - * bitmask to the map_dma_pages function. - * - * Be aware that upgrading a mapped address might result in change of - * the DMA address for the page. - */ -#define ODP_READ_ALLOWED_BIT (1<<0ULL) -#define ODP_WRITE_ALLOWED_BIT (1<<1ULL) - -#define ODP_DMA_ADDR_MASK (~(ODP_READ_ALLOWED_BIT | ODP_WRITE_ALLOWED_BIT)) - #ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING struct ib_umem_odp * From patchwork Wed Oct 30 15:12:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856686 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36CA1D5CCB9 for ; Wed, 30 Oct 2024 15:14:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B91BC6B00A5; Wed, 30 Oct 2024 11:14:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B40776B00A6; Wed, 30 Oct 2024 11:14:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96D826B00AA; Wed, 30 Oct 2024 11:14:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 74D506B00A5 for ; Wed, 30 Oct 2024 11:14:07 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 30E6F1A0C09 for ; Wed, 30 Oct 2024 15:14:07 +0000 (UTC) X-FDA: 82730612370.20.B296E4C Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf26.hostedemail.com (Postfix) with ESMTP id F2AAB140021 for ; Wed, 30 Oct 2024 15:13:45 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=atfbnlC2; spf=pass (imf26.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NI4+esa6fCVcBQC++yJUbQGT81QwVH0x4mwOIGIz0fE=; b=ZLYekbfZRHnYowt5110e53wzTX/AO755yvmKlEhtre1pqwNzvKag6GD1blRqjZz+9ySOXp KMnn5TML1SQGwr2fBcGLsUj7/mSJ9IhSw4zTuElaPr3Qbfq27MfvpE18H3X74tYp1BBvCm uavbgvxA9/1T8evkQa4nd61HjT8Hnbg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=atfbnlC2; spf=pass (imf26.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301165; a=rsa-sha256; cv=none; b=ZjWlgk9Kq3EhB33LkJnof381lF+EDeBn1m1kbKkcJS7ivCvCa83YyfXqYScPVjBuSqwCRm sgrNrWg8iRR5hTesMHbRyKaHJYAM7WGwI1ooXUO2dpJqU1Wfewmgxm6eJeGa4wcvdCL2nR SqT32s4HngFjzYOP8K0EVKzIxdi8Jls= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 1FE8DA4256D; Wed, 30 Oct 2024 15:12:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E62D3C4CED1; Wed, 30 Oct 2024 15:14:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301244; bh=ufIIJTJY/y6wCplhr9nPwK7JHAPS3UUfFuT2KoVZ//k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=atfbnlC23aJ/unZHenCqSaPOHbChvvIiRIgKgnze7qCocelxJzJQrmFykPo5E2++W DAKJQ9dRzygh8UYnCu8UcAKYfd2jqxXNQPCL/lv0VTSMmaTCnDr4ILeIguX2E6UiRd Ce4dOyKxMCLJdRpyYjtDucWfGfzQGcAv2x1QH2m21hA5DhlTaoSyeUf3S75YhEmNL8 Tfmf6z+Kbq9cxYT55VWYSHE6G2isGZpz7SY9yLjyy2EPIk90sqWOJAEGHGRABmYBtE xK41KQAuf8VIGrUgBYbvgG0aaEo8nW9tZ8y3T8oRrYNX2rXb6hbhvxCZXo8R5UusFb pOjC/DEgHuDOw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 13/17] RDMA/core: Convert UMEM ODP DMA mapping to caching IOVA and page linkage Date: Wed, 30 Oct 2024 17:12:59 +0200 Message-ID: <8ab01c9be43ed9ae8739481cb2c4044bd1e1535c.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: F2AAB140021 X-Stat-Signature: jha7i5k8b3yz3mf7j5cxszhy97gsiyy5 X-HE-Tag: 1730301225-119164 X-HE-Meta: U2FsdGVkX1/qTCLo43tfHC7+iCCjMVr8A5vujhn32eKxJABYpCXn1TmrnAkeXSg9+cq+dPjcY0c3weP6mFgv7Dj0Mu3jQggz/1wS56QG2eiNn1LTfNXJv0U+NGy5493pOODnjSXnon9tbUIN42oDaiaYSsJILhaDLDBjLEqwwOZSpurnKu3DBHGpqCnLUQtp8AjkvNcqLbNwxDJ94mIOarExq8RUY3PWE1DgL9A+yGAhVlZVf6CMi07TNc3zlxyJ7ZXtT71Yhx4UGsUEShpO4827yOPBD3mgTFJaqmDb9Wmv5Ft4N0P9cKAoqO05yuH1oMdMl8jOqqVYKhlTJ7O8NbnWP9GVi6YdW8QpjNAs4C3hx7XptYoYFyv4eu2jNeaYDNQvbYELKIRkkbJT8vODvxT1OUY6ucmff9L4qjFIBWnFj2XKjgmj6eTRBdjnHaA0vYBdTVyKDRoaK8k/4gljwlqZ+2kDz8b4Q8pggBzojdrsBRxFsH5GJfLYFEQT6tKEr+sb57lavkubgz3gdmV9u2b89kSJ8Azu4nZjiW5U6Qpf2bGNn7GibDQ5syD+KChIYRC9r20oqNNPO8nXYvYKRfNkDh3/zesfyaAWoYYZnfbjTlq6NpctNRCcmmNst5Cqs2pMzkil1RH7Z3J0K9lfdhdflYetatEwUxOBwv6gZucQgeEST3GbAl4VHd4ygz/Ina49GnnDEmA15Ppf5wpkL0TDs6dbthLO+sMtiXj5DveD312NdlKL+eezAqb06O9KeegtnF9Fbk2M2/iDg8GHM6smJH7uZxnpsVermpyKZVZLCj8r1Rv7OqSDFNYY7NlCrxgisBoWP0JuyRfZaa8PvHixaSAyvEgAqYjYfCnZsOCrgVyyET+lIUyV/B/PIyfMt8qExVnGhi2bwMqZ2IRshLr7bOXwqvY+C7CuYxluPZ9yTDkyvaeevaShG4mNZguR6rQP9lHCy13dADruSyK YqQQarbi BQf+KgGFjDS+zXSTL3Edt7vSHzpiCn5mBuO/SwjeY4bDEUgVceJzSgKBRtb9ZN3/q3u2R1sSXYBUFbtGBlUtu/bR/3cvRUxDqAZiApNtcKwMznzElME+/9YTq+6e4Hoi+Vz3kmvrr2HbdnFrB2x+9nTaT3X7y/hMFTQmsTbSHEQrdf10ZQWfjNoAddVMqYytuOf0PJXu0hz80X/wxJHuZA6W/8vqACB72t2bI/v2FUZoAA10= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Reuse newly added DMA API to cache IOVA and only link/unlink pages in fast path for UMEM ODP flow. Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 101 ++++++--------------------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 11 +-- drivers/infiniband/hw/mlx5/odp.c | 40 +++++++---- drivers/infiniband/hw/mlx5/umr.c | 12 +++- include/rdma/ib_umem_odp.h | 13 +--- 5 files changed, 69 insertions(+), 108 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index 9dba369365af..30cd8f353476 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -50,6 +51,7 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, const struct mmu_interval_notifier_ops *ops) { + struct ib_device *dev = umem_odp->umem.ibdev; int ret; umem_odp->umem.is_odp = 1; @@ -59,7 +61,6 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, size_t page_size = 1UL << umem_odp->page_shift; unsigned long start; unsigned long end; - size_t ndmas, npfns; start = ALIGN_DOWN(umem_odp->umem.address, page_size); if (check_add_overflow(umem_odp->umem.address, @@ -70,36 +71,23 @@ static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, if (unlikely(end < page_size)) return -EOVERFLOW; - ndmas = (end - start) >> umem_odp->page_shift; - if (!ndmas) - return -EINVAL; - - npfns = (end - start) >> PAGE_SHIFT; - umem_odp->pfn_list = kvcalloc( - npfns, sizeof(*umem_odp->pfn_list), GFP_KERNEL); - if (!umem_odp->pfn_list) - return -ENOMEM; - - umem_odp->dma_list = kvcalloc( - ndmas, sizeof(*umem_odp->dma_list), GFP_KERNEL); - if (!umem_odp->dma_list) { - ret = -ENOMEM; - goto out_pfn_list; - } + ret = hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, + (end - start) >> PAGE_SHIFT, + 1 << umem_odp->page_shift); + if (ret) + return ret; ret = mmu_interval_notifier_insert(&umem_odp->notifier, umem_odp->umem.owning_mm, start, end - start, ops); if (ret) - goto out_dma_list; + goto out_free_map; } return 0; -out_dma_list: - kvfree(umem_odp->dma_list); -out_pfn_list: - kvfree(umem_odp->pfn_list); +out_free_map: + hmm_dma_map_free(dev->dma_device, &umem_odp->map); return ret; } @@ -262,6 +250,8 @@ EXPORT_SYMBOL(ib_umem_odp_get); void ib_umem_odp_release(struct ib_umem_odp *umem_odp) { + struct ib_device *dev = umem_odp->umem.ibdev; + /* * Ensure that no more pages are mapped in the umem. * @@ -274,48 +264,17 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) ib_umem_end(umem_odp)); mutex_unlock(&umem_odp->umem_mutex); mmu_interval_notifier_remove(&umem_odp->notifier); - kvfree(umem_odp->dma_list); - kvfree(umem_odp->pfn_list); + hmm_dma_map_free(dev->dma_device, &umem_odp->map); } put_pid(umem_odp->tgid); kfree(umem_odp); } EXPORT_SYMBOL(ib_umem_odp_release); -/* - * Map for DMA and insert a single page into the on-demand paging page tables. - * - * @umem: the umem to insert the page to. - * @dma_index: index in the umem to add the dma to. - * @page: the page struct to map and add. - * @access_mask: access permissions needed for this page. - * - * The function returns -EFAULT if the DMA mapping operation fails. - * - */ -static int ib_umem_odp_map_dma_single_page( - struct ib_umem_odp *umem_odp, - unsigned int dma_index, - struct page *page) -{ - struct ib_device *dev = umem_odp->umem.ibdev; - dma_addr_t *dma_addr = &umem_odp->dma_list[dma_index]; - - *dma_addr = ib_dma_map_page(dev, page, 0, 1 << umem_odp->page_shift, - DMA_BIDIRECTIONAL); - if (ib_dma_mapping_error(dev, *dma_addr)) { - *dma_addr = 0; - return -EFAULT; - } - umem_odp->npages++; - return 0; -} - /** * ib_umem_odp_map_dma_and_lock - DMA map userspace memory in an ODP MR and lock it. * * Maps the range passed in the argument to DMA addresses. - * The DMA addresses of the mapped pages is updated in umem_odp->dma_list. * Upon success the ODP MR will be locked to let caller complete its device * page table update. * @@ -372,7 +331,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, range.default_flags |= HMM_PFN_REQ_WRITE; } - range.hmm_pfns = &(umem_odp->pfn_list[pfn_start_idx]); + range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]); timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); retry: @@ -423,15 +382,6 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, __func__, hmm_order, page_shift); break; } - - ret = ib_umem_odp_map_dma_single_page( - umem_odp, dma_index, hmm_pfn_to_page(range.hmm_pfns[pfn_index])); - if (ret < 0) { - ibdev_dbg(umem_odp->umem.ibdev, - "ib_umem_odp_map_dma_single_page failed with error %d\n", ret); - break; - } - range.hmm_pfns[pfn_index] |= HMM_PFN_DMA_MAPPED; } /* upon success lock should stay on hold for the callee */ if (!ret) @@ -451,30 +401,23 @@ EXPORT_SYMBOL(ib_umem_odp_map_dma_and_lock); void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, u64 bound) { - dma_addr_t dma; - int idx; - u64 addr; struct ib_device *dev = umem_odp->umem.ibdev; + u64 addr; lockdep_assert_held(&umem_odp->umem_mutex); virt = max_t(u64, virt, ib_umem_start(umem_odp)); bound = min_t(u64, bound, ib_umem_end(umem_odp)); for (addr = virt; addr < bound; addr += BIT(umem_odp->page_shift)) { - unsigned long pfn_idx = (addr - ib_umem_start(umem_odp)) >> PAGE_SHIFT; - struct page *page = hmm_pfn_to_page(umem_odp->pfn_list[pfn_idx]); - - idx = (addr - ib_umem_start(umem_odp)) >> umem_odp->page_shift; - dma = umem_odp->dma_list[idx]; + u64 offset = addr - ib_umem_start(umem_odp); + size_t idx = offset >> umem_odp->page_shift; + unsigned long pfn = umem_odp->map.pfn_list[idx]; - if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_VALID)) - goto clear; - if (!(umem_odp->pfn_list[pfn_idx] & HMM_PFN_DMA_MAPPED)) + if (!hmm_dma_unmap_pfn(dev->dma_device, &umem_odp->map, idx)) goto clear; - ib_dma_unmap_page(dev, dma, BIT(umem_odp->page_shift), - DMA_BIDIRECTIONAL); - if (umem_odp->pfn_list[pfn_idx] & HMM_PFN_WRITE) { + if (pfn & HMM_PFN_WRITE) { + struct page *page = hmm_pfn_to_page(pfn); struct page *head_page = compound_head(page); /* * set_page_dirty prefers being called with @@ -489,7 +432,7 @@ void ib_umem_odp_unmap_dma_pages(struct ib_umem_odp *umem_odp, u64 virt, } umem_odp->npages--; clear: - umem_odp->pfn_list[pfn_idx] &= ~HMM_PFN_FLAGS; + umem_odp->map.pfn_list[idx] &= ~HMM_PFN_FLAGS; } } EXPORT_SYMBOL(ib_umem_odp_unmap_dma_pages); diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 3e4aaa6319db..1bae5595c729 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1444,8 +1444,8 @@ void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev); int __init mlx5_ib_odp_init(void); void mlx5_ib_odp_cleanup(void); int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev); -void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags); +int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, + struct mlx5_ib_mr *mr, int flags); int mlx5_ib_advise_mr_prefetch(struct ib_pd *pd, enum ib_uverbs_advise_mr_advice advice, @@ -1466,8 +1466,11 @@ static inline int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev) { return 0; } -static inline void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags) {} +static inline int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, + struct mlx5_ib_mr *mr, int flags) +{ + return -EOPNOTSUPP; +} static inline int mlx5_ib_advise_mr_prefetch(struct ib_pd *pd, diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 78887500ce15..fbb2a5670c32 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -35,6 +35,8 @@ #include #include #include +#include +#include #include "mlx5_ib.h" #include "cmd.h" @@ -159,40 +161,50 @@ static void populate_klm(struct mlx5_klm *pklm, size_t idx, size_t nentries, } } -static void populate_mtt(__be64 *pas, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags) +static int populate_mtt(__be64 *pas, size_t start, size_t nentries, + struct mlx5_ib_mr *mr, int flags) { struct ib_umem_odp *odp = to_ib_umem_odp(mr->umem); bool downgrade = flags & MLX5_IB_UPD_XLT_DOWNGRADE; - unsigned long pfn; - dma_addr_t pa; + struct pci_p2pdma_map_state p2pdma_state = {}; + struct ib_device *dev = odp->umem.ibdev; size_t i; if (flags & MLX5_IB_UPD_XLT_ZAP) - return; + return 0; for (i = 0; i < nentries; i++) { - pfn = odp->pfn_list[idx + i]; + unsigned long pfn = odp->map.pfn_list[start + i]; + dma_addr_t dma_addr; + + pfn = odp->map.pfn_list[start + i]; if (!(pfn & HMM_PFN_VALID)) /* ODP initialization */ continue; - pa = odp->dma_list[idx + i]; - pa |= MLX5_IB_MTT_READ; + dma_addr = hmm_dma_map_pfn(dev->dma_device, &odp->map, + start + i, &p2pdma_state); + if (ib_dma_mapping_error(dev, dma_addr)) + return -EFAULT; + + dma_addr |= MLX5_IB_MTT_READ; if ((pfn & HMM_PFN_WRITE) && !downgrade) - pa |= MLX5_IB_MTT_WRITE; + dma_addr |= MLX5_IB_MTT_WRITE; - pas[i] = cpu_to_be64(pa); + pas[i] = cpu_to_be64(dma_addr); + odp->npages++; } + return 0; } -void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, - struct mlx5_ib_mr *mr, int flags) +int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, + struct mlx5_ib_mr *mr, int flags) { if (flags & MLX5_IB_UPD_XLT_INDIRECT) { populate_klm(xlt, idx, nentries, mr, flags); + return 0; } else { - populate_mtt(xlt, idx, nentries, mr, flags); + return populate_mtt(xlt, idx, nentries, mr, flags); } } @@ -286,7 +298,7 @@ static bool mlx5_ib_invalidate_range(struct mmu_interval_notifier *mni, * estimate the cost of another UMR vs. the cost of bigger * UMR. */ - if (umem_odp->pfn_list[idx] & HMM_PFN_VALID) { + if (umem_odp->map.pfn_list[idx] & HMM_PFN_VALID) { if (!in_block) { blk_start_idx = idx; in_block = 1; diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c index 887fd6fa3ba9..d7fa94ab23cf 100644 --- a/drivers/infiniband/hw/mlx5/umr.c +++ b/drivers/infiniband/hw/mlx5/umr.c @@ -811,7 +811,17 @@ int mlx5r_umr_update_xlt(struct mlx5_ib_mr *mr, u64 idx, int npages, size_to_map = npages * desc_size; dma_sync_single_for_cpu(ddev, sg.addr, sg.length, DMA_TO_DEVICE); - mlx5_odp_populate_xlt(xlt, idx, npages, mr, flags); + /* + * npages is the maximum number of pages to map, but we + * can't guarantee that all pages are actually mapped. + * + * For example, if page is p2p of type which is not supported + * for mapping, the number of pages mapped will be less than + * requested. + */ + err = mlx5_odp_populate_xlt(xlt, idx, npages, mr, flags); + if (err) + return err; dma_sync_single_for_device(ddev, sg.addr, sg.length, DMA_TO_DEVICE); sg.length = ALIGN(size_to_map, MLX5_UMR_FLEX_ALIGNMENT); diff --git a/include/rdma/ib_umem_odp.h b/include/rdma/ib_umem_odp.h index a345c26a745d..2a24bf791c10 100644 --- a/include/rdma/ib_umem_odp.h +++ b/include/rdma/ib_umem_odp.h @@ -8,24 +8,17 @@ #include #include -#include +#include struct ib_umem_odp { struct ib_umem umem; struct mmu_interval_notifier notifier; struct pid *tgid; - /* An array of the pfns included in the on-demand paging umem. */ - unsigned long *pfn_list; + struct hmm_dma_map map; /* - * An array with DMA addresses mapped for pfns in pfn_list. - * The lower two bits designate access permissions. - * See ODP_READ_ALLOWED_BIT and ODP_WRITE_ALLOWED_BIT. - */ - dma_addr_t *dma_list; - /* - * The umem_mutex protects the page_list and dma_list fields of an ODP + * The umem_mutex protects the page_list field of an ODP * umem, allowing only a single thread to map/unmap pages. The mutex * also protects access to the mmu notifier counters. */ From patchwork Wed Oct 30 15:13:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A407CD5CCBB for ; Wed, 30 Oct 2024 15:14:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A2866B00B5; Wed, 30 Oct 2024 11:14:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 353876B00B8; Wed, 30 Oct 2024 11:14:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CC6A6B00B9; Wed, 30 Oct 2024 11:14:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EED5C6B00B5 for ; Wed, 30 Oct 2024 11:14:25 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9C6C5160B1D for ; Wed, 30 Oct 2024 15:14:25 +0000 (UTC) X-FDA: 82730614638.14.16D721A Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf08.hostedemail.com (Postfix) with ESMTP id BF24B16001A for ; Wed, 30 Oct 2024 15:14:06 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hnGwg4Jb; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301146; a=rsa-sha256; cv=none; b=yBVQluA3WxZfkq6hCIV+LIJddsk0FtJ/NF4p0wtGB+DyWs6UonPw6FasdC3OSMwN3BAoEV P1omBWxU3cofTridaKUiyPk6JcFhrmCLuqlRBET9C/KhjGbUQHW+UIjjGCisv/69FWdtue Ks73yECGdt82UzC9ZRiTcvYbXIFPo68= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=hnGwg4Jb; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf08.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301146; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zSWGIO8+eoNDo2boHntx+WY/7C1S7kHkrtB0FlbeWwg=; b=7+J6xEP3YpgKzHAT+2k2zpsK982g3/L6pcZfwvNHtQkXAa9priwB3uSlPbxa/5BYfobAoM V0ITF1YDCA7r8+wIAdxtvjacvNdGKvuxAzuCa7q5L7lKo/ATLVL585yDOyAw5DRWbMkPcl azZ5gvOawcSdwawdhibeZ056eTLiy6g= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 91F54A412E2; Wed, 30 Oct 2024 15:12:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6154CC4CED0; Wed, 30 Oct 2024 15:14:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301263; bh=OlIySM8DodN7dp6laXmryqK8XnG8NlruPsXd0qhOBbs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hnGwg4JbAdghWJybJUqla2ZegkylUzfKB5TcyGzLPzUbM/IHSIANk1/OjZfOMsIzE xqJbLAd9/auwfxV3a72k98W/n2OkiLBw+b4+AH4591LyeHiEd9zNYC0YcvAAcDzGWU MMfeU7lQ/4g02Dp3UlnJ8gcov4oVJueML9uJF2xMFn03x9KMvtYH7yg8crE8p4oVjp pSLm8g7ZJrlcHiAx4IL0pxQft6UasJBqcwplMv6IxDlVEGvZHcOeI29e4L9G0lUpSe mshIdIFqzfpeNCnjYDIv78wDqXwdSd35IiiouiRcox3+6KBKU3xNITLDmysnSqFBmq +2U4Rit5ap4vQ== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 14/17] RDMA/umem: Separate implicit ODP initialization from explicit ODP Date: Wed, 30 Oct 2024 17:13:00 +0200 Message-ID: <23253254adaf3a1e6953de0c5852ac4598b456bd.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: BF24B16001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: hmugx368zcfh9ginfj6rrnx7dkgpu98y X-HE-Tag: 1730301246-858778 X-HE-Meta: U2FsdGVkX19XHlFatw8NNcMs2CWRyurMeTHvSzMbMWWZxQL6VkHdGw4p1ueHJDwFcg8wcht0XN1LI8hIqzR+6CgqW3LCAEejIxEP82t9xDX2ZGhIVoDxYJM3HH9IZ2GdkdPtIJRG4DPC3B7Dc5mZz4KLP4FXYGeSGolwRWzxqJCqQ15olyGdW8qlNz/FdS2efChNMtMilve+3wDGNXfk96++mVhwFE2TKudBDJJIyUao81DdvMAYH63HzTympz3co0LQNnYU8mhgm7ZK4FVghuFQei29gpzFfAwvnXzHbd4cSwZlggJX7I6wQwgdFqeRm6rMv0XmD8n6yUT5g3g1Drfjhx03caNJSyiyJ6/0RHG0gMMUoXnni0QmW1yzdVnroHEZ+5ALMeZSoHa1E4T7pzg+WCR8GPGO1SQxDGDukhNahGaBRQ42qjPPCQictN7M6jGcL9MX5ZJqtQMErF8QLncRTUlvPh7tFiRZSAtafYOPaEIOjzL9wKUno5o8SXy7QQsfvAjg1tGt8xMkmBwwxU7mqLlsHT6EPkIz2D88M+ihehcSUc6cvl70YZGhPenaBCMHd2pZt3l3DVdo8F8/k+XpxPuEBgS8BgeUvYHtT/GBlRjNK8btARiwbq2Gl7VGemyHwBRcHIyYhDRnQdKNu2PpVXG+RwtAknSQWkyZf5LjtMxw1dhGWmKAN86QMs7r20b1BgWFJ/pR7TyNCy+r1bbhlMgcVBvd2jnF7K3njyphA31WkdqedYaHXj87BWOTaIQUV67g8GsEk9D342qXQZHqmXuHnMJvB6Pdw1WAYOh3IIfAkgph/+lQ+kJ3mwVijPGW1CPOUymFw1MW8L6OJjeyZ3IerKF6kisQhNiZJyHbkt40nUh5IXZlF0vHMcxMvs8QDRE9v8cxs2cGCsCH77aeiIWZjFXSbpmYlE0xtoqoM1fEPlTzYhtECbifHJJx1mUEUhz9HccSHs9B2eo nPR79TkY 2pveVs8D2STz40oTVUL8IAIpec/nl285302d1GS0RrlkeB2imCo1Wf134AgblyNbZdOvbduAOiEy5Ayg3ZZxIVfRRYIJADmm8gS2UwvfpLADLYhmOzeGIdF93qFvElxU4/FhOjaeh6Z1c8zDJ0rbBmSXqhsggyPxrVYVgr6vTqPsjVQ1i/WekVSqrXueLSy9KcjvWpsAgWiHkWplmJemzlAQZOxFEQM0oWXeltnoMx8Cw1D0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Create separate functions for the implicit ODP initialization which is different from the explicit ODP initialization. Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/umem_odp.c | 91 +++++++++++++++--------------- 1 file changed, 46 insertions(+), 45 deletions(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index 30cd8f353476..51d518989914 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -48,41 +48,44 @@ #include "uverbs.h" -static inline int ib_init_umem_odp(struct ib_umem_odp *umem_odp, - const struct mmu_interval_notifier_ops *ops) +static void ib_init_umem_implicit_odp(struct ib_umem_odp *umem_odp) +{ + umem_odp->is_implicit_odp = 1; + umem_odp->umem.is_odp = 1; + mutex_init(&umem_odp->umem_mutex); +} + +static int ib_init_umem_odp(struct ib_umem_odp *umem_odp, + const struct mmu_interval_notifier_ops *ops) { struct ib_device *dev = umem_odp->umem.ibdev; + size_t page_size = 1UL << umem_odp->page_shift; + unsigned long start; + unsigned long end; int ret; umem_odp->umem.is_odp = 1; mutex_init(&umem_odp->umem_mutex); - if (!umem_odp->is_implicit_odp) { - size_t page_size = 1UL << umem_odp->page_shift; - unsigned long start; - unsigned long end; - - start = ALIGN_DOWN(umem_odp->umem.address, page_size); - if (check_add_overflow(umem_odp->umem.address, - (unsigned long)umem_odp->umem.length, - &end)) - return -EOVERFLOW; - end = ALIGN(end, page_size); - if (unlikely(end < page_size)) - return -EOVERFLOW; - - ret = hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, - (end - start) >> PAGE_SHIFT, - 1 << umem_odp->page_shift); - if (ret) - return ret; - - ret = mmu_interval_notifier_insert(&umem_odp->notifier, - umem_odp->umem.owning_mm, - start, end - start, ops); - if (ret) - goto out_free_map; - } + start = ALIGN_DOWN(umem_odp->umem.address, page_size); + if (check_add_overflow(umem_odp->umem.address, + (unsigned long)umem_odp->umem.length, &end)) + return -EOVERFLOW; + end = ALIGN(end, page_size); + if (unlikely(end < page_size)) + return -EOVERFLOW; + + ret = hmm_dma_map_alloc(dev->dma_device, &umem_odp->map, + (end - start) >> PAGE_SHIFT, + 1 << umem_odp->page_shift); + if (ret) + return ret; + + ret = mmu_interval_notifier_insert(&umem_odp->notifier, + umem_odp->umem.owning_mm, start, + end - start, ops); + if (ret) + goto out_free_map; return 0; @@ -106,7 +109,6 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, { struct ib_umem *umem; struct ib_umem_odp *umem_odp; - int ret; if (access & IB_ACCESS_HUGETLB) return ERR_PTR(-EINVAL); @@ -118,16 +120,10 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, umem->ibdev = device; umem->writable = ib_access_writable(access); umem->owning_mm = current->mm; - umem_odp->is_implicit_odp = 1; umem_odp->page_shift = PAGE_SHIFT; umem_odp->tgid = get_task_pid(current->group_leader, PIDTYPE_PID); - ret = ib_init_umem_odp(umem_odp, NULL); - if (ret) { - put_pid(umem_odp->tgid); - kfree(umem_odp); - return ERR_PTR(ret); - } + ib_init_umem_implicit_odp(umem_odp); return umem_odp; } EXPORT_SYMBOL(ib_umem_odp_alloc_implicit); @@ -248,7 +244,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *device, } EXPORT_SYMBOL(ib_umem_odp_get); -void ib_umem_odp_release(struct ib_umem_odp *umem_odp) +static void ib_umem_odp_free(struct ib_umem_odp *umem_odp) { struct ib_device *dev = umem_odp->umem.ibdev; @@ -258,14 +254,19 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) * It is the driver's responsibility to ensure, before calling us, * that the hardware will not attempt to access the MR any more. */ - if (!umem_odp->is_implicit_odp) { - mutex_lock(&umem_odp->umem_mutex); - ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), - ib_umem_end(umem_odp)); - mutex_unlock(&umem_odp->umem_mutex); - mmu_interval_notifier_remove(&umem_odp->notifier); - hmm_dma_map_free(dev->dma_device, &umem_odp->map); - } + mutex_lock(&umem_odp->umem_mutex); + ib_umem_odp_unmap_dma_pages(umem_odp, ib_umem_start(umem_odp), + ib_umem_end(umem_odp)); + mutex_unlock(&umem_odp->umem_mutex); + mmu_interval_notifier_remove(&umem_odp->notifier); + hmm_dma_map_free(dev->dma_device, &umem_odp->map); +} + +void ib_umem_odp_release(struct ib_umem_odp *umem_odp) +{ + if (!umem_odp->is_implicit_odp) + ib_umem_odp_free(umem_odp); + put_pid(umem_odp->tgid); kfree(umem_odp); } From patchwork Wed Oct 30 15:13:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E81DD5CCB3 for ; Wed, 30 Oct 2024 15:14:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D0C96B00AA; Wed, 30 Oct 2024 11:14:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01E886B00AC; Wed, 30 Oct 2024 11:14:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB3956B00AE; Wed, 30 Oct 2024 11:14:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BB2226B00AA for ; Wed, 30 Oct 2024 11:14:14 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7AC2480C83 for ; Wed, 30 Oct 2024 15:14:14 +0000 (UTC) X-FDA: 82730613042.09.4D7A17C Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf23.hostedemail.com (Postfix) with ESMTP id F2290140012 for ; Wed, 30 Oct 2024 15:13:55 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=h4VaVnmI; spf=pass (imf23.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=u/onjaAzz4OkttdC/5YCGBTFG7PsSXENF3Yj4oybQ+I=; b=zW8WQ5JQKK/ngrPt3R6fjkiSN/AWxQ4f/FCXvIeOhIsvoxTMz+WKOgcvKnWWdtcmWPuCIB HHapfbyy49dXRXFL/yCmh7ZnLBHJH8yLR0Wd9YqIGSpbjAopOaQR6P6ms+8q6lMc2+znQr 6pmrGhHgqPcatuxeI/Tc5UiVLdXumvw= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=h4VaVnmI; spf=pass (imf23.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301197; a=rsa-sha256; cv=none; b=k0QHIwWirHnCBSKORcHvco1oJWd7D0CeOUbymDkZw2gT2hwzkYnkvcbp89vJ5Kc60I31hV aNEMGiceo6dgxh1KteNIIuXOD2OCm6sl2uVSbmIR5D1g3E1XWzjIWPzPOKf6aj752TsUrC SPxSczPdiwTcBYsNxzxIIdmIjAIWuRg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id E311D5C116F; Wed, 30 Oct 2024 15:13:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E4BACC4CED1; Wed, 30 Oct 2024 15:14:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301251; bh=Tou6h56yAf8zPXrsCYXLKq3TAhdexORUkVzCl7QijME=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h4VaVnmIky+hiiKr9zeB9+Uf2F83BYfpIDUK/675btqMGfzMBduAm96m26BXz31P3 5D7N0uheBlG1EkiNAmKwPs7a3KD4Io2CKqGYS5p8mXNOKcjdAz8JbJ4UBRmuqn30aN IoJffimX9q0yCGEHGD+BpiM7EQQg8IdYwgaWOl1tOJzz9lMTD35/n8QfBIJaUXtAvX km8vrnoDksYXodN3tq0Ucs+nkCcb0Y/T5YPZ+sEWV1V+jtxhp7CDSx1cDmdN3U+W/9 c9EhxHATtTP07AhHig1te53HVJqGXJZGCXo1OEk6WPN/uu7DMHKAOEQxy1Y6d0J14K vAhDVgYxRyK/w== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 15/17] vfio/mlx5: Explicitly use number of pages instead of allocated length Date: Wed, 30 Oct 2024 17:13:01 +0200 Message-ID: <3e91721157afc02c04beaec97247dafe251f5d2c.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: bmguq8tug3maxgewwr6yrnzwo9rzat8c X-Rspamd-Queue-Id: F2290140012 X-Rspamd-Server: rspam11 X-HE-Tag: 1730301235-86507 X-HE-Meta: U2FsdGVkX18ztfxqyCtjWIDuDQk/9JBamZKB8OnsAPB1JMu6QLVs2wCKDJKgBUFg7TzrLoJ76ZWCV5Vfmi3x2+nRAYfGpGLnEgfqE51dzmJXrOHmvVFXTGXADVVK2Y4gNbUtZWdS7Yk0gP4RksHcZO24o8w1N5KCrQySEQ0Y95wbgJopkw+q2zUoKrfiEpZwWBiqQUD6f4qg3y+Bm7D/dAwF+qrpJhySP3TBvUMfGwroH4aK6lSdMdALYAv8ZTtJrzLMwCmj3JhR0bWUuuuEccGeRRn69ggg03DUjqj6BTRbyAKrUF/6at4VwpAY7OxZ7wsJRTCAVXCEb/LQXnJd1fBwIgjVdMhUwvasyzUwAkttJsFkOlMJEQ4fESWcz67Zw0P2vlifRQoyG5i2hyTh8bwZtvN4p0AZcSKXLmNRBgxHpyhdc+C4rnf1bRo2Df9tP5x/SY7muokE17dWwsoUTWGI79xb8jIMPq+Hrg97BkwiSBwGcsEuN8nbfO0sCW0S0LUrCFqr1/TRnXYwKA0QfcYPgduLbTWw67EBT/zMnNiHFps2u2MH+/fQ53gtX+Wz5Twh9q91YOLTV2X8NWWGoHAjVIzx4ETYN8dyQ5JLuoboTM9+ftdX+/R6ZX6y9RHN7ZdVSDH5g26Z1Nqi2U5xE/Ra3akkNkDsMC961mZ8c4/KtAOrwP1l5H1ve4VbOdAcjjxEG38eHojKmNJLrXxWz6n0ym6V1B/4lGa0syQ2tSq3+CufdqH8XB6kUSQzTf5WpVHJx79tdoHockhQL8AzRNyeY0GC2S5+e640ndlJRRWZQsc2VWyHzZuRo4AFKiDUv1RwK6MrNYSvfSq0tulL5jf/bKJv0p9r8Ahr1aoVKM5nUl5KeIzMPn/wSQEmq2f1fhqM+cOBBK4oEbgYPY24f4cfv4goZnOoD2YoTSgOIVAmzyMgugi2kHsz/INT8cxzP3wRR1WZ+HH8iCPfrmz Oxm7Bu26 Yp1QhLyXvzu52RZfJyiLe3fADYXM9p+FKpd1DqhRM6JNAs2xfMwOPbQwxp+O53vkYYHzO02Okx6T7/Zqjw4YncP1AkGlVZYbP0bCNsjtYcsSXxax2pCimYzzBmV1nHP6xEQaPZzfRlENpoEM8YxO+oanDBr7EpijGixdPSv7pLxtEbYWHmE+g5nsXIhV20CBE+CZrts5o/Ss+DiK6C5xs5NWv8224YRlGUCfDdjEhlFltFHk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky allocated_length is a multiple of page size and number of pages, so let's change the functions to accept number of pages. It opens us a venue to combine receive and send paths together with code readability improvement. Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 32 ++++++++++----------- drivers/vfio/pci/mlx5/cmd.h | 10 +++---- drivers/vfio/pci/mlx5/main.c | 56 +++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 41 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 41a4b0cf4297..fdc3e515741f 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -318,8 +318,7 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn, struct mlx5_vhca_recv_buf *recv_buf, u32 *mkey) { - size_t npages = buf ? DIV_ROUND_UP(buf->allocated_length, PAGE_SIZE) : - recv_buf->npages; + size_t npages = buf ? buf->npages : recv_buf->npages; int err = 0, inlen; __be64 *mtt; void *mkc; @@ -375,7 +374,7 @@ static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) if (mvdev->mdev_detach) return -ENOTCONN; - if (buf->dmaed || !buf->allocated_length) + if (buf->dmaed || !buf->npages) return -EINVAL; ret = dma_map_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); @@ -444,7 +443,7 @@ static int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, if (ret) goto err; - buf->allocated_length += filled * PAGE_SIZE; + buf->npages += filled; /* clean input for another bulk allocation */ memset(page_list, 0, filled * sizeof(*page_list)); to_fill = min_t(unsigned int, to_alloc, @@ -460,8 +459,7 @@ static int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, } struct mlx5_vhca_data_buffer * -mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, +mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, enum dma_data_direction dma_dir) { struct mlx5_vhca_data_buffer *buf; @@ -473,9 +471,8 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, buf->dma_dir = dma_dir; buf->migf = migf; - if (length) { - ret = mlx5vf_add_migration_pages(buf, - DIV_ROUND_UP_ULL(length, PAGE_SIZE)); + if (npages) { + ret = mlx5vf_add_migration_pages(buf, npages); if (ret) goto end; @@ -501,8 +498,8 @@ void mlx5vf_put_data_buffer(struct mlx5_vhca_data_buffer *buf) } struct mlx5_vhca_data_buffer * -mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, enum dma_data_direction dma_dir) +mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, + enum dma_data_direction dma_dir) { struct mlx5_vhca_data_buffer *buf, *temp_buf; struct list_head free_list; @@ -517,7 +514,7 @@ mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, list_for_each_entry_safe(buf, temp_buf, &migf->avail_list, buf_elm) { if (buf->dma_dir == dma_dir) { list_del_init(&buf->buf_elm); - if (buf->allocated_length >= length) { + if (buf->npages >= npages) { spin_unlock_irq(&migf->list_lock); goto found; } @@ -531,7 +528,7 @@ mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, } } spin_unlock_irq(&migf->list_lock); - buf = mlx5vf_alloc_data_buffer(migf, length, dma_dir); + buf = mlx5vf_alloc_data_buffer(migf, npages, dma_dir); found: while ((temp_buf = list_first_entry_or_null(&free_list, @@ -712,7 +709,7 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, MLX5_SET(save_vhca_state_in, in, op_mod, 0); MLX5_SET(save_vhca_state_in, in, vhca_id, mvdev->vhca_id); MLX5_SET(save_vhca_state_in, in, mkey, buf->mkey); - MLX5_SET(save_vhca_state_in, in, size, buf->allocated_length); + MLX5_SET(save_vhca_state_in, in, size, buf->npages * PAGE_SIZE); MLX5_SET(save_vhca_state_in, in, incremental, inc); MLX5_SET(save_vhca_state_in, in, set_track, track); @@ -734,8 +731,11 @@ int mlx5vf_cmd_save_vhca_state(struct mlx5vf_pci_core_device *mvdev, } if (!header_buf) { - header_buf = mlx5vf_get_data_buffer(migf, - sizeof(struct mlx5_vf_migration_header), DMA_NONE); + header_buf = mlx5vf_get_data_buffer( + migf, + DIV_ROUND_UP(sizeof(struct mlx5_vf_migration_header), + PAGE_SIZE), + DMA_NONE); if (IS_ERR(header_buf)) { err = PTR_ERR(header_buf); goto err_free; diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index df421dc6de04..7d4a833b6900 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -56,7 +56,7 @@ struct mlx5_vhca_data_buffer { struct sg_append_table table; loff_t start_pos; u64 length; - u64 allocated_length; + u32 npages; u32 mkey; enum dma_data_direction dma_dir; u8 dmaed:1; @@ -217,12 +217,12 @@ int mlx5vf_cmd_alloc_pd(struct mlx5_vf_migration_file *migf); void mlx5vf_cmd_dealloc_pd(struct mlx5_vf_migration_file *migf); void mlx5fv_cmd_clean_migf_resources(struct mlx5_vf_migration_file *migf); struct mlx5_vhca_data_buffer * -mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, enum dma_data_direction dma_dir); +mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, + enum dma_data_direction dma_dir); void mlx5vf_free_data_buffer(struct mlx5_vhca_data_buffer *buf); struct mlx5_vhca_data_buffer * -mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, - size_t length, enum dma_data_direction dma_dir); +mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, + enum dma_data_direction dma_dir); void mlx5vf_put_data_buffer(struct mlx5_vhca_data_buffer *buf); struct page *mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, unsigned long offset); diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index 242c23eef452..a1dbee3be1e0 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -308,6 +308,7 @@ static struct mlx5_vhca_data_buffer * mlx5vf_mig_file_get_stop_copy_buf(struct mlx5_vf_migration_file *migf, u8 index, size_t required_length) { + u32 npages = DIV_ROUND_UP(required_length, PAGE_SIZE); struct mlx5_vhca_data_buffer *buf = migf->buf[index]; u8 chunk_num; @@ -315,12 +316,11 @@ mlx5vf_mig_file_get_stop_copy_buf(struct mlx5_vf_migration_file *migf, chunk_num = buf->stop_copy_chunk_num; buf->migf->buf[index] = NULL; /* Checking whether the pre-allocated buffer can fit */ - if (buf->allocated_length >= required_length) + if (buf->npages >= npages) return buf; mlx5vf_put_data_buffer(buf); - buf = mlx5vf_get_data_buffer(buf->migf, required_length, - DMA_FROM_DEVICE); + buf = mlx5vf_get_data_buffer(buf->migf, npages, DMA_FROM_DEVICE); if (IS_ERR(buf)) return buf; @@ -373,7 +373,8 @@ static int mlx5vf_add_stop_copy_header(struct mlx5_vf_migration_file *migf, u8 *to_buff; int ret; - header_buf = mlx5vf_get_data_buffer(migf, size, DMA_NONE); + header_buf = mlx5vf_get_data_buffer(migf, DIV_ROUND_UP(size, PAGE_SIZE), + DMA_NONE); if (IS_ERR(header_buf)) return PTR_ERR(header_buf); @@ -388,7 +389,7 @@ static int mlx5vf_add_stop_copy_header(struct mlx5_vf_migration_file *migf, to_buff = kmap_local_page(page); memcpy(to_buff, &header, sizeof(header)); header_buf->length = sizeof(header); - data.stop_copy_size = cpu_to_le64(migf->buf[0]->allocated_length); + data.stop_copy_size = cpu_to_le64(migf->buf[0]->npages * PAGE_SIZE); memcpy(to_buff + sizeof(header), &data, sizeof(data)); header_buf->length += sizeof(data); kunmap_local(to_buff); @@ -437,15 +438,20 @@ static int mlx5vf_prep_stop_copy(struct mlx5vf_pci_core_device *mvdev, num_chunks = mvdev->chunk_mode ? MAX_NUM_CHUNKS : 1; for (i = 0; i < num_chunks; i++) { - buf = mlx5vf_get_data_buffer(migf, inc_state_size, DMA_FROM_DEVICE); + buf = mlx5vf_get_data_buffer( + migf, DIV_ROUND_UP(inc_state_size, PAGE_SIZE), + DMA_FROM_DEVICE); if (IS_ERR(buf)) { ret = PTR_ERR(buf); goto err; } migf->buf[i] = buf; - buf = mlx5vf_get_data_buffer(migf, - sizeof(struct mlx5_vf_migration_header), DMA_NONE); + buf = mlx5vf_get_data_buffer( + migf, + DIV_ROUND_UP(sizeof(struct mlx5_vf_migration_header), + PAGE_SIZE), + DMA_NONE); if (IS_ERR(buf)) { ret = PTR_ERR(buf); goto err; @@ -553,7 +559,8 @@ static long mlx5vf_precopy_ioctl(struct file *filp, unsigned int cmd, * We finished transferring the current state and the device has a * dirty state, save a new state to be ready for. */ - buf = mlx5vf_get_data_buffer(migf, inc_length, DMA_FROM_DEVICE); + buf = mlx5vf_get_data_buffer(migf, DIV_ROUND_UP(inc_length, PAGE_SIZE), + DMA_FROM_DEVICE); if (IS_ERR(buf)) { ret = PTR_ERR(buf); mlx5vf_mark_err(migf); @@ -673,8 +680,8 @@ mlx5vf_pci_save_device_data(struct mlx5vf_pci_core_device *mvdev, bool track) if (track) { /* leave the allocated buffer ready for the stop-copy phase */ - buf = mlx5vf_alloc_data_buffer(migf, - migf->buf[0]->allocated_length, DMA_FROM_DEVICE); + buf = mlx5vf_alloc_data_buffer(migf, migf->buf[0]->npages, + DMA_FROM_DEVICE); if (IS_ERR(buf)) { ret = PTR_ERR(buf); goto out_pd; @@ -917,11 +924,14 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, goto out_unlock; break; case MLX5_VF_LOAD_STATE_PREP_HEADER_DATA: - if (vhca_buf_header->allocated_length < migf->record_size) { + { + u32 npages = DIV_ROUND_UP(migf->record_size, PAGE_SIZE); + + if (vhca_buf_header->npages < npages) { mlx5vf_free_data_buffer(vhca_buf_header); - migf->buf_header[0] = mlx5vf_alloc_data_buffer(migf, - migf->record_size, DMA_NONE); + migf->buf_header[0] = mlx5vf_alloc_data_buffer( + migf, npages, DMA_NONE); if (IS_ERR(migf->buf_header[0])) { ret = PTR_ERR(migf->buf_header[0]); migf->buf_header[0] = NULL; @@ -934,6 +944,7 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, vhca_buf_header->start_pos = migf->max_pos; migf->load_state = MLX5_VF_LOAD_STATE_READ_HEADER_DATA; break; + } case MLX5_VF_LOAD_STATE_READ_HEADER_DATA: ret = mlx5vf_resume_read_header_data(migf, vhca_buf_header, &buf, &len, pos, &done); @@ -944,12 +955,13 @@ static ssize_t mlx5vf_resume_write(struct file *filp, const char __user *buf, { u64 size = max(migf->record_size, migf->stop_copy_prep_size); + u32 npages = DIV_ROUND_UP(size, PAGE_SIZE); - if (vhca_buf->allocated_length < size) { + if (vhca_buf->npages < npages) { mlx5vf_free_data_buffer(vhca_buf); - migf->buf[0] = mlx5vf_alloc_data_buffer(migf, - size, DMA_TO_DEVICE); + migf->buf[0] = mlx5vf_alloc_data_buffer( + migf, npages, DMA_TO_DEVICE); if (IS_ERR(migf->buf[0])) { ret = PTR_ERR(migf->buf[0]); migf->buf[0] = NULL; @@ -1031,8 +1043,11 @@ mlx5vf_pci_resume_device_data(struct mlx5vf_pci_core_device *mvdev) } migf->buf[0] = buf; - buf = mlx5vf_alloc_data_buffer(migf, - sizeof(struct mlx5_vf_migration_header), DMA_NONE); + buf = mlx5vf_alloc_data_buffer( + migf, + DIV_ROUND_UP(sizeof(struct mlx5_vf_migration_header), + PAGE_SIZE), + DMA_NONE); if (IS_ERR(buf)) { ret = PTR_ERR(buf); goto out_buf; @@ -1149,7 +1164,8 @@ mlx5vf_pci_step_device_state_locked(struct mlx5vf_pci_core_device *mvdev, MLX5VF_QUERY_INC | MLX5VF_QUERY_CLEANUP); if (ret) return ERR_PTR(ret); - buf = mlx5vf_get_data_buffer(migf, size, DMA_FROM_DEVICE); + buf = mlx5vf_get_data_buffer(migf, + DIV_ROUND_UP(size, PAGE_SIZE), DMA_FROM_DEVICE); if (IS_ERR(buf)) return ERR_CAST(buf); /* pre_copy cleanup */ From patchwork Wed Oct 30 15:13:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09247D5CCB3 for ; Wed, 30 Oct 2024 15:14:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D2B86B00AF; Wed, 30 Oct 2024 11:14:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 85B686B00B2; Wed, 30 Oct 2024 11:14:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D4AA6B00B3; Wed, 30 Oct 2024 11:14:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4EA086B00AF for ; Wed, 30 Oct 2024 11:14:18 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 11F7614059E for ; Wed, 30 Oct 2024 15:14:18 +0000 (UTC) X-FDA: 82730613840.12.EFBB8E1 Received: from nyc.source.kernel.org (nyc.source.kernel.org [147.75.193.91]) by imf22.hostedemail.com (Postfix) with ESMTP id 64D74C000D for ; Wed, 30 Oct 2024 15:13:44 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=uGWltSBf; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf22.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301138; a=rsa-sha256; cv=none; b=KCXQuA5MIv14MIwF/37YRDByUcVo3F/lTtagk/FAYAEAe9teW2G/Pt3BjKvEoMIrHkW2WX FhhWnvt1UqaLZ7jmhjYtxuhyhFzUak+vprFNQLtMWJYxq4iRBJDl2Zz/zNV2QFm4WOg1J2 7+zuXtBp4+CwQwLpuiWaGMS3QFgjkKg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=uGWltSBf; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf22.hostedemail.com: domain of leon@kernel.org designates 147.75.193.91 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PhW4VjnvtzkDooyFGath+3T4tsAPk9cdglqFB5sUC1Q=; b=DILRN9r9Rx0rg0L4KizBtWoqkHcC7CjSwqkPRqMNURguZo60NwLyo/QJiRAg2dqCxDjezk 2wG1axW/LLAQQo0MrVL9Z0xfB/rS69Iw5rzCQB6H4S1BensipqwOXjYvunfYq1S5UtWERL 5KP2ss6qksVraSR1bvkLU4FngkUL6TA= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id D59DCA40257; Wed, 30 Oct 2024 15:12:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A57D9C4CED1; Wed, 30 Oct 2024 15:14:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301255; bh=RCW7z84/26BITUxg8PndrrKsB8pTf34fxPy1qnPVTz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uGWltSBfuWLOOW1uCjA96ZkRf/tc9Ofzcqd4yBlthycq0QuFcTDg1vtaJK2d3dEmJ j/O3jro7YQUDUh3DxANT/lEqBGuf8gX8pjaejHU1r3j/TQqSe9feErtiZKD2m9nXSi h+/WR6a7xsDnPgH6Ft8Ec2DlZ+deXW4uSXxyCPluFGBtOcD9zI5UdLjBa+BH4S6Bmo 7fF+NgcxheBw9unedEy68XY0Xinsf2FH7Fr7T/5xEp4rTjNIbFI600KVuS+YwrLtfE IT9WlhFIsQyXpXFR9PPqNDFPrigxE1BcEJ9/JHhP9PKeGG4tzzA2iCVhzFS+tEiTDt b1Fmfu9dSheXw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 16/17] vfio/mlx5: Rewrite create mkey flow to allow better code reuse Date: Wed, 30 Oct 2024 17:13:02 +0200 Message-ID: <0cecdfc196d028a8554f1d4564e0e4f3ef75ce70.1730298502.git.leon@kernel.org> X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 64D74C000D X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ipo4nd35pjrccc7otnyan7tnaugw1drw X-HE-Tag: 1730301224-143765 X-HE-Meta: U2FsdGVkX1/9/7piRNAYTasWAjIE4QlyTADrn1mAZugyUJ5LbjJXHdeJaGWn4+Eua/Eqt2R7sG0wkPwA55K/0JOzWSTLvK+mvHuyIPJVhuE3PYOGXax1Zl27Kp4LLvFEuzZhEc9kciAGQJeGE4m2dh0BuTJBdvoZc5GfAT0/3nyX+Vjxe/r+YEk+u2gKOkcx5ap0zV79G1DMmYghGcloIryBvrKXz9tnPwzfzFeW6awFukYha7AT5zpY2GAtqpeq271nuiRLM1M5lBO4xEOIjdeWZ6FCVUkjNV1/Io/SPGXK93/s/7qT/5+5ATYxz3sdv89VTURGyHLfdtaYnyljgcZfztRuRcuz39D5EDOJwhcp4EqxKW/iPzB1KLjo8Ql1LeHULJh0sDcbwIGFyDRDdbaDLqjfcZbVu3RBX1ctqZwmJ4CXNlqpg8Y1XbxGBKlYjeIHWbivEbvvauDNEMdKtilEK3l3dk/AsCYUIcn7EvXcotqDAeIeRlspF7p4P6cQSogZ6IaufezpL2AOwBMhv4mse3yuFhydL5tI3bzUMJv6bC8iW0XvOwbZ5Gvm2b4AorkXPAnhWON685wS82fUrAmmE1RQBJ+b1TmKwSsoIvlHwcMWnnOMkonKOmc0klW2F9tYMe4Xdm2vVWfaLootz2od6h/UMhE8Al939XvXYcoGSoNOpwHVLos8XMlxWZS4W7u6dbm6vuJtbwjTObLzINd6aelc0BjHUr1qedwXSDKIZ5E71Xv0ryXcDeUMxmCWuaGofqqNV6jj8vnk5j9L8DaAVBNukRtvXCGPSlGRmb1qXQ8gmlFuJoAzncR5DXqa0TZAPxshxLmIVJtFZkCvWLkCx7zLynSEJCEStaRBuQF4jEl3MXFu78vjstaHfsq7JOs8VvTA6d2HoAdgGvBbPEeBxU9GeWYqI69SotK0yf3vfvbc+jGt1T7aXTgVxJJjH2cNp5QAhf1PAFD70KO i58sgxN4 1qZ0M5/7tv/kw8bI/7kQRrQt8kb3/YSLiZ75ZcAf6WSdAhyN/EgyC1CdndLX7RBSslIOqLkbslhr8Xptg5z7IYBbZ20wlbaGO13kr+RZsyw1kEJEo724BmItVX2NsnqvGy66OjMYXrB2D/SjpKwJwBb7IRX+BR6hbgepz+HTG5lrFNn64p/BSW4CCvG/H5mPzplA51ljgcJsXprqHmvLqE2yWFf52MugRPr4mEs7KPYJNTrs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Change the creation of mkey to be performed in multiple steps: data allocation, DMA setup and actual call to HW to create that mkey. In this new flow, the whole input to MKEY command is saved to eliminate the need to keep array of pointers for DMA addresses for receive list and in the future patches for send list too. In addition to memory size reduce and elimination of unnecessary data movements to set MKEY input, the code is prepared for future reuse. Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 156 ++++++++++++++++++++---------------- drivers/vfio/pci/mlx5/cmd.h | 4 +- 2 files changed, 90 insertions(+), 70 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index fdc3e515741f..1832a6c1f35d 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -313,39 +313,21 @@ static int mlx5vf_cmd_get_vhca_id(struct mlx5_core_dev *mdev, u16 function_id, return ret; } -static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn, - struct mlx5_vhca_data_buffer *buf, - struct mlx5_vhca_recv_buf *recv_buf, - u32 *mkey) +static u32 *alloc_mkey_in(u32 npages, u32 pdn) { - size_t npages = buf ? buf->npages : recv_buf->npages; - int err = 0, inlen; - __be64 *mtt; + int inlen; void *mkc; u32 *in; inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + - sizeof(*mtt) * round_up(npages, 2); + sizeof(__be64) * round_up(npages, 2); - in = kvzalloc(inlen, GFP_KERNEL); + in = kvzalloc(inlen, GFP_KERNEL_ACCOUNT); if (!in) - return -ENOMEM; + return NULL; MLX5_SET(create_mkey_in, in, translations_octword_actual_size, DIV_ROUND_UP(npages, 2)); - mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, in, klm_pas_mtt); - - if (buf) { - struct sg_dma_page_iter dma_iter; - - for_each_sgtable_dma_page(&buf->table.sgt, &dma_iter, 0) - *mtt++ = cpu_to_be64(sg_page_iter_dma_address(&dma_iter)); - } else { - int i; - - for (i = 0; i < npages; i++) - *mtt++ = cpu_to_be64(recv_buf->dma_addrs[i]); - } mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry); MLX5_SET(mkc, mkc, access_mode_1_0, MLX5_MKC_ACCESS_MODE_MTT); @@ -359,9 +341,29 @@ static int _create_mkey(struct mlx5_core_dev *mdev, u32 pdn, MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT); MLX5_SET(mkc, mkc, translations_octword_size, DIV_ROUND_UP(npages, 2)); MLX5_SET64(mkc, mkc, len, npages * PAGE_SIZE); - err = mlx5_core_create_mkey(mdev, mkey, in, inlen); - kvfree(in); - return err; + + return in; +} + +static int create_mkey(struct mlx5_core_dev *mdev, u32 npages, + struct mlx5_vhca_data_buffer *buf, u32 *mkey_in, + u32 *mkey) +{ + __be64 *mtt; + int inlen; + + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); + if (buf) { + struct sg_dma_page_iter dma_iter; + + for_each_sgtable_dma_page(&buf->table.sgt, &dma_iter, 0) + *mtt++ = cpu_to_be64(sg_page_iter_dma_address(&dma_iter)); + } + + inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + + sizeof(__be64) * round_up(npages, 2); + + return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); } static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) @@ -374,20 +376,28 @@ static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) if (mvdev->mdev_detach) return -ENOTCONN; - if (buf->dmaed || !buf->npages) + if (buf->mkey_in || !buf->npages) return -EINVAL; ret = dma_map_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); if (ret) return ret; - ret = _create_mkey(mdev, buf->migf->pdn, buf, NULL, &buf->mkey); - if (ret) + buf->mkey_in = alloc_mkey_in(buf->npages, buf->migf->pdn); + if (!buf->mkey_in) { + ret = -ENOMEM; goto err; + } - buf->dmaed = true; + ret = create_mkey(mdev, buf->npages, buf, buf->mkey_in, &buf->mkey); + if (ret) + goto err_create_mkey; return 0; + +err_create_mkey: + kvfree(buf->mkey_in); + buf->mkey_in = NULL; err: dma_unmap_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); return ret; @@ -401,8 +411,9 @@ void mlx5vf_free_data_buffer(struct mlx5_vhca_data_buffer *buf) lockdep_assert_held(&migf->mvdev->state_mutex); WARN_ON(migf->mvdev->mdev_detach); - if (buf->dmaed) { + if (buf->mkey_in) { mlx5_core_destroy_mkey(migf->mvdev->mdev, buf->mkey); + kvfree(buf->mkey_in); dma_unmap_sgtable(migf->mvdev->mdev->device, &buf->table.sgt, buf->dma_dir, 0); } @@ -779,7 +790,7 @@ int mlx5vf_cmd_load_vhca_state(struct mlx5vf_pci_core_device *mvdev, if (mvdev->mdev_detach) return -ENOTCONN; - if (!buf->dmaed) { + if (!buf->mkey_in) { err = mlx5vf_dma_data_buffer(buf); if (err) return err; @@ -1380,56 +1391,54 @@ static int alloc_recv_pages(struct mlx5_vhca_recv_buf *recv_buf, kvfree(recv_buf->page_list); return -ENOMEM; } +static void unregister_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + u32 *mkey_in) +{ + dma_addr_t addr; + __be64 *mtt; + int i; + + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); + for (i = npages - 1; i >= 0; i--) { + addr = be64_to_cpu(mtt[i]); + dma_unmap_single(mdev->device, addr, PAGE_SIZE, + DMA_FROM_DEVICE); + } +} -static int register_dma_recv_pages(struct mlx5_core_dev *mdev, - struct mlx5_vhca_recv_buf *recv_buf) +static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + struct page **page_list, u32 *mkey_in) { - int i, j; + dma_addr_t addr; + __be64 *mtt; + int i; - recv_buf->dma_addrs = kvcalloc(recv_buf->npages, - sizeof(*recv_buf->dma_addrs), - GFP_KERNEL_ACCOUNT); - if (!recv_buf->dma_addrs) - return -ENOMEM; + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - for (i = 0; i < recv_buf->npages; i++) { - recv_buf->dma_addrs[i] = dma_map_page(mdev->device, - recv_buf->page_list[i], - 0, PAGE_SIZE, - DMA_FROM_DEVICE); - if (dma_mapping_error(mdev->device, recv_buf->dma_addrs[i])) + for (i = 0; i < npages; i++) { + addr = dma_map_page(mdev->device, page_list[i], 0, PAGE_SIZE, + DMA_FROM_DEVICE); + if (dma_mapping_error(mdev->device, addr)) goto error; + + *mtt++ = cpu_to_be64(addr); } + return 0; error: - for (j = 0; j < i; j++) - dma_unmap_single(mdev->device, recv_buf->dma_addrs[j], - PAGE_SIZE, DMA_FROM_DEVICE); - - kvfree(recv_buf->dma_addrs); + unregister_dma_pages(mdev, i, mkey_in); return -ENOMEM; } -static void unregister_dma_recv_pages(struct mlx5_core_dev *mdev, - struct mlx5_vhca_recv_buf *recv_buf) -{ - int i; - - for (i = 0; i < recv_buf->npages; i++) - dma_unmap_single(mdev->device, recv_buf->dma_addrs[i], - PAGE_SIZE, DMA_FROM_DEVICE); - - kvfree(recv_buf->dma_addrs); -} - static void mlx5vf_free_qp_recv_resources(struct mlx5_core_dev *mdev, struct mlx5_vhca_qp *qp) { struct mlx5_vhca_recv_buf *recv_buf = &qp->recv_buf; mlx5_core_destroy_mkey(mdev, recv_buf->mkey); - unregister_dma_recv_pages(mdev, recv_buf); + unregister_dma_pages(mdev, recv_buf->npages, recv_buf->mkey_in); + kvfree(recv_buf->mkey_in); free_recv_pages(&qp->recv_buf); } @@ -1445,18 +1454,29 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, if (err < 0) return err; - err = register_dma_recv_pages(mdev, recv_buf); - if (err) + recv_buf->mkey_in = alloc_mkey_in(npages, pdn); + if (!recv_buf->mkey_in) { + err = -ENOMEM; goto end; + } + + err = register_dma_pages(mdev, npages, recv_buf->page_list, + recv_buf->mkey_in); + if (err) + goto err_register_dma; - err = _create_mkey(mdev, pdn, NULL, recv_buf, &recv_buf->mkey); + err = create_mkey(mdev, npages, NULL, recv_buf->mkey_in, + &recv_buf->mkey); if (err) goto err_create_mkey; return 0; err_create_mkey: - unregister_dma_recv_pages(mdev, recv_buf); + unregister_dma_pages(mdev, npages, recv_buf->mkey_in); +err_register_dma: + kvfree(recv_buf->mkey_in); + recv_buf->mkey_in = NULL; end: free_recv_pages(recv_buf); return err; diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index 7d4a833b6900..25dd6ff54591 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -58,8 +58,8 @@ struct mlx5_vhca_data_buffer { u64 length; u32 npages; u32 mkey; + u32 *mkey_in; enum dma_data_direction dma_dir; - u8 dmaed:1; u8 stop_copy_chunk_num; struct list_head buf_elm; struct mlx5_vf_migration_file *migf; @@ -133,8 +133,8 @@ struct mlx5_vhca_cq { struct mlx5_vhca_recv_buf { u32 npages; struct page **page_list; - dma_addr_t *dma_addrs; u32 next_rq_offset; + u32 *mkey_in; u32 mkey; }; From patchwork Wed Oct 30 15:13:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13856690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31C93D5CCB3 for ; Wed, 30 Oct 2024 15:14:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B916B6B00B3; Wed, 30 Oct 2024 11:14:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B47226B00B4; Wed, 30 Oct 2024 11:14:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9945F6B00B5; Wed, 30 Oct 2024 11:14:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 722AF6B00B3 for ; Wed, 30 Oct 2024 11:14:22 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 29C0A120DCA for ; Wed, 30 Oct 2024 15:14:22 +0000 (UTC) X-FDA: 82730614008.24.1DAC220 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf15.hostedemail.com (Postfix) with ESMTP id 134DFA0029 for ; Wed, 30 Oct 2024 15:13:53 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ghYbMSWC; spf=pass (imf15.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730301180; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QYh9sFrXIKKYU6Uwp+BpQnZ3MIxMzmAe+9VeQG88/mk=; b=K4OJSzl94tqOLzEcUA3QBqdiefCOqg3zXFve0IV27wSSibmSRBKScCYu0teduNqc8XYHtk 3p+jY4TcptaQrocjf6bA/fA4ivNl7mwc9ShO0AyDFdb/LtsmXmyY9P04GvzkaWAADTLns6 K+XjoS3y0vb1b8w0c72c/Y6ikUy12z0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ghYbMSWC; spf=pass (imf15.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730301180; a=rsa-sha256; cv=none; b=64xy/CptxUPKucFyL/9erFC6RD91aGPk1gCtLJcycjaSEtHuqkhvVR7AvlVS84EC5X8+Kd 5OfWtuOcAYzlmDVtEdX+PYr3Q0O390tuI/IAI6s20iAUfz6MhwOLxSi1x4EWDO8xTrIooH eDn5hglQB8wyOSykQDu+tM6GQ4ZtkDM= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 7DC355C006D; Wed, 30 Oct 2024 15:13:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 61EF8C4CECE; Wed, 30 Oct 2024 15:14:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1730301259; bh=kZnELGk3J+NhjUahaVqpHHvafG1ZWZH/rQu8efz7iEA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ghYbMSWCqwZf897V5ved3sZCfe9m69BJiRyg1zgV2KWYGzIQ47ibpT5oVDvwfhbHC X96vAKovM72xOmtPDfvVqneVQpAHHXug3l748qG9qlJmRacPUipL4SKw/2NmXRMXze B19eeCSUASGJBp5V4fS5c5/JeIfMqDQqzBoclCRB6QzyxDXsFzFK8nDa7gXlWcg45u fwt+G6oIUdRGlKto+/aQF2yN/ZyVH9M1CPXhKvDBahLLb/Q3r+2LIQ8dS2icebgdJu HOOXDtMQs+i1VHr9W6VLTsiEHmlNFt1Qk7k19qmVVJjeh863bWT7AMnoKzeWTZ5is4 pQ634e4B/8zSw== From: Leon Romanovsky To: Jens Axboe , Jason Gunthorpe , Robin Murphy , Joerg Roedel , Will Deacon , Christoph Hellwig , Sagi Grimberg Cc: Leon Romanovsky , Keith Busch , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , Marek Szyprowski , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 17/17] vfio/mlx5: Convert vfio to use DMA link API Date: Wed, 30 Oct 2024 17:13:03 +0200 Message-ID: X-Mailer: git-send-email 2.46.2 In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 134DFA0029 X-Stat-Signature: 1mbwi6i8445q1b55e53pb3fxfgupoydo X-HE-Tag: 1730301233-782103 X-HE-Meta: U2FsdGVkX19jwgVsF7SlbitVcq2lcbmFnUGBtFypfdUGxYM7Amjg+MmpwJkI+Z17wBII6ZJ5utD13gRJz9FvnhhKBtThfc0tvziqPXj49tEfp5Xeb7dlV3loac6Peqhnsl1SsSYzB0FDJAhim7e6MbbYgudoWhkLWVHF81ZhT5pbsadH6KiibVCAXTDAFndJoaIsFhaGQnRFHagky3y8bJuC6UtswhFWOfstsunUPkS1S5xLahUMcFpHI2tIZXY9M1l6hHgRAHoY19NcuJZ7UIPgnmi8xU2i3ZY6DXgqbApibJYJgW4fG8suxKvChRKiSvkV2N9E2msmgE1TpK4kuaMlGii3Ut3EYSA0ro+skyB5ZOSPznYCMlP47ZGopU1urKeO9X+11DEPO8x6kYtxyk0QvIs49s9Kkk2AIgihz7FfcEV0v9FIQ1hG0BS4SPSx5CxtDiUNu06CsE+WsM/OcbFQA98sTs3fRcRvrA8Dz/2fOJGht6ngoEJZwi+rf3kTmlktIt5rtaQ18Y+5RIx4cf0Zj6e1MofaUuJsqypzB6jto6MJxEHEYBICInndwPOUlindG8/qgUW7Y+b7CQi4eOmq/uysT0bdA3wGiZTEIlYp3I02LxMP6ii6RjsM71m2LSbG7g+G8fzEpD2rLNjrizt1DyE9S7FhmxrMGAMGCf6h9HaL+MTqO7hsAKf2C2dDOo1v6oqi2nUwFbCh3o4hNWQpGW6QgUvgdC10K/gB5fZc+aRaYdNP5gSlH1kVzjubqKL/GAC7Zu4Y3fKAPFC39Hkw3so/adCqBPj2mco1zFGbsk04cYTxel4JkbAbKC36ITKqZ43JujjikGnW+y5s4uQkotm8cw8mTJUFZcYOrn6kb+Ob7veedrKPwxH1C52XrJhb16H9Xxaabflrw4xQud99GyO324i2LL8OYP4WJg3oE3Q362aAxzbz/wPueTpFUsnfU9T/jIV3nnSQIzz ceO9Hh5q 4GinQPvgDnb3/fylH2rJgJuLfZwfChcSoFIvIjm49sjhVVXX+lvsYcdpZjbKr32/EdENSWyRaU32tr1SpZr3rJqGg+/Gnv5fAVAlQXARQKG30EbBfRV/vGZ117YLU/LLSb4vD2uM8Sup8grJV23AXYEgCZUXv8dBw4oEBADHqgISvBpGeFpy6V+C7u3lYtNHxVLgm3Yot9BFoRRWBq5kMJ2ktf1w/QQ2qpe6kbTpjI7JYwTA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Remove intermediate scatter-gather table as it is not needed if DMA link API is used. This conversion reduces drastically the memory used to manage that table. Signed-off-by: Leon Romanovsky --- drivers/vfio/pci/mlx5/cmd.c | 295 ++++++++++++++++------------------- drivers/vfio/pci/mlx5/cmd.h | 21 ++- drivers/vfio/pci/mlx5/main.c | 31 ---- 3 files changed, 148 insertions(+), 199 deletions(-) diff --git a/drivers/vfio/pci/mlx5/cmd.c b/drivers/vfio/pci/mlx5/cmd.c index 1832a6c1f35d..cde1481ed23c 100644 --- a/drivers/vfio/pci/mlx5/cmd.c +++ b/drivers/vfio/pci/mlx5/cmd.c @@ -345,25 +345,81 @@ static u32 *alloc_mkey_in(u32 npages, u32 pdn) return in; } -static int create_mkey(struct mlx5_core_dev *mdev, u32 npages, - struct mlx5_vhca_data_buffer *buf, u32 *mkey_in, +static int create_mkey(struct mlx5_core_dev *mdev, u32 npages, u32 *mkey_in, u32 *mkey) { + int inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + + sizeof(__be64) * round_up(npages, 2); + + return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); +} + +static void unregister_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + u32 *mkey_in, struct dma_iova_state *state, + enum dma_data_direction dir) +{ + dma_addr_t addr; __be64 *mtt; - int inlen; + int i; - mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - if (buf) { - struct sg_dma_page_iter dma_iter; + WARN_ON_ONCE(dir == DMA_NONE); - for_each_sgtable_dma_page(&buf->table.sgt, &dma_iter, 0) - *mtt++ = cpu_to_be64(sg_page_iter_dma_address(&dma_iter)); + if (dma_use_iova(state)) { + dma_iova_destroy(mdev->device, state, npages * PAGE_SIZE, dir, 0); + } else { + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, + klm_pas_mtt); + for (i = npages - 1; i >= 0; i--) { + addr = be64_to_cpu(mtt[i]); + dma_unmap_page(mdev->device, addr, PAGE_SIZE, dir); + } } +} - inlen = MLX5_ST_SZ_BYTES(create_mkey_in) + - sizeof(__be64) * round_up(npages, 2); +static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, + struct page **page_list, u32 *mkey_in, + struct dma_iova_state *state, + enum dma_data_direction dir) +{ + dma_addr_t addr; + size_t mapped = 0; + __be64 *mtt; + int i, err; - return mlx5_core_create_mkey(mdev, mkey, mkey_in, inlen); + WARN_ON_ONCE(dir == DMA_NONE); + + mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); + + if (dma_iova_try_alloc(mdev->device, state, 0, npages * PAGE_SIZE)) { + addr = state->addr; + for (i = 0; i < npages; i++) { + err = dma_iova_link(mdev->device, state, + page_to_phys(page_list[i]), mapped, + PAGE_SIZE, dir, 0); + if (err) + goto error; + *mtt++ = cpu_to_be64(addr); + addr += PAGE_SIZE; + mapped += PAGE_SIZE; + } + err = dma_iova_sync(mdev->device, state, 0, mapped); + if (err) + goto error; + } else { + for (i = 0; i < npages; i++) { + addr = dma_map_page(mdev->device, page_list[i], 0, + PAGE_SIZE, dir); + err = dma_mapping_error(mdev->device, addr); + if (err) + goto error; + *mtt++ = cpu_to_be64(addr); + } + } + return 0; + +error: + unregister_dma_pages(mdev, i, mkey_in, state, dir); + return err; } static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) @@ -379,96 +435,93 @@ static int mlx5vf_dma_data_buffer(struct mlx5_vhca_data_buffer *buf) if (buf->mkey_in || !buf->npages) return -EINVAL; - ret = dma_map_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); - if (ret) - return ret; - buf->mkey_in = alloc_mkey_in(buf->npages, buf->migf->pdn); - if (!buf->mkey_in) { - ret = -ENOMEM; - goto err; - } + if (!buf->mkey_in) + return -ENOMEM; - ret = create_mkey(mdev, buf->npages, buf, buf->mkey_in, &buf->mkey); + ret = register_dma_pages(mdev, buf->npages, buf->page_list, + buf->mkey_in, &buf->state, buf->dma_dir); + if (ret) + goto err_register_dma; + + ret = create_mkey(mdev, buf->npages, buf->mkey_in, &buf->mkey); if (ret) goto err_create_mkey; return 0; err_create_mkey: + unregister_dma_pages(mdev, buf->npages, buf->mkey_in, &buf->state, + buf->dma_dir); +err_register_dma: kvfree(buf->mkey_in); buf->mkey_in = NULL; -err: - dma_unmap_sgtable(mdev->device, &buf->table.sgt, buf->dma_dir, 0); return ret; } +static void free_page_list(u32 npages, struct page **page_list) +{ + int i; + + /* Undo alloc_pages_bulk_array() */ + for (i = npages - 1; i >= 0; i--) + __free_page(page_list[i]); + + kvfree(page_list); +} + void mlx5vf_free_data_buffer(struct mlx5_vhca_data_buffer *buf) { - struct mlx5_vf_migration_file *migf = buf->migf; - struct sg_page_iter sg_iter; + struct mlx5vf_pci_core_device *mvdev = buf->migf->mvdev; + struct mlx5_core_dev *mdev = mvdev->mdev; - lockdep_assert_held(&migf->mvdev->state_mutex); - WARN_ON(migf->mvdev->mdev_detach); + lockdep_assert_held(&mvdev->state_mutex); + WARN_ON(mvdev->mdev_detach); if (buf->mkey_in) { - mlx5_core_destroy_mkey(migf->mvdev->mdev, buf->mkey); + mlx5_core_destroy_mkey(mdev, buf->mkey); + unregister_dma_pages(mdev, buf->npages, buf->mkey_in, + &buf->state, buf->dma_dir); kvfree(buf->mkey_in); - dma_unmap_sgtable(migf->mvdev->mdev->device, &buf->table.sgt, - buf->dma_dir, 0); } - /* Undo alloc_pages_bulk_array() */ - for_each_sgtable_page(&buf->table.sgt, &sg_iter, 0) - __free_page(sg_page_iter_page(&sg_iter)); - sg_free_append_table(&buf->table); + free_page_list(buf->npages, buf->page_list); kfree(buf); } -static int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, - unsigned int npages) +static int mlx5vf_add_pages(struct page ***page_list, unsigned int npages) { - unsigned int to_alloc = npages; - struct page **page_list; - unsigned long filled; - unsigned int to_fill; - int ret; + unsigned int filled = 0, done = 0; + int i; - to_fill = min_t(unsigned int, npages, PAGE_SIZE / sizeof(*page_list)); - page_list = kvzalloc(to_fill * sizeof(*page_list), GFP_KERNEL_ACCOUNT); - if (!page_list) + *page_list = kvcalloc(npages, sizeof(struct page *), GFP_KERNEL_ACCOUNT); + if (!*page_list) return -ENOMEM; - do { - filled = alloc_pages_bulk_array(GFP_KERNEL_ACCOUNT, to_fill, - page_list); - if (!filled) { - ret = -ENOMEM; + for (;;) { + filled = alloc_pages_bulk_array(GFP_KERNEL_ACCOUNT, + npages - done, + *page_list + done); + if (!filled) goto err; - } - to_alloc -= filled; - ret = sg_alloc_append_table_from_pages( - &buf->table, page_list, filled, 0, - filled << PAGE_SHIFT, UINT_MAX, SG_MAX_SINGLE_ALLOC, - GFP_KERNEL_ACCOUNT); - if (ret) - goto err; - buf->npages += filled; - /* clean input for another bulk allocation */ - memset(page_list, 0, filled * sizeof(*page_list)); - to_fill = min_t(unsigned int, to_alloc, - PAGE_SIZE / sizeof(*page_list)); - } while (to_alloc > 0); + done += filled; + if (done == npages) + break; + } - kvfree(page_list); return 0; err: - kvfree(page_list); - return ret; + for (i = 0; i < done; i++) + __free_page(*page_list[i]); + + kvfree(*page_list); + *page_list = NULL; + return -ENOMEM; } + struct mlx5_vhca_data_buffer * mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, enum dma_data_direction dma_dir) @@ -483,10 +536,12 @@ mlx5vf_alloc_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, buf->dma_dir = dma_dir; buf->migf = migf; if (npages) { - ret = mlx5vf_add_migration_pages(buf, npages); + ret = mlx5vf_add_pages(&buf->page_list, npages); if (ret) goto end; + buf->npages = npages; + if (dma_dir != DMA_NONE) { ret = mlx5vf_dma_data_buffer(buf); if (ret) @@ -1345,101 +1400,16 @@ static void mlx5vf_destroy_qp(struct mlx5_core_dev *mdev, kfree(qp); } -static void free_recv_pages(struct mlx5_vhca_recv_buf *recv_buf) -{ - int i; - - /* Undo alloc_pages_bulk_array() */ - for (i = 0; i < recv_buf->npages; i++) - __free_page(recv_buf->page_list[i]); - - kvfree(recv_buf->page_list); -} - -static int alloc_recv_pages(struct mlx5_vhca_recv_buf *recv_buf, - unsigned int npages) -{ - unsigned int filled = 0, done = 0; - int i; - - recv_buf->page_list = kvcalloc(npages, sizeof(*recv_buf->page_list), - GFP_KERNEL_ACCOUNT); - if (!recv_buf->page_list) - return -ENOMEM; - - for (;;) { - filled = alloc_pages_bulk_array(GFP_KERNEL_ACCOUNT, - npages - done, - recv_buf->page_list + done); - if (!filled) - goto err; - - done += filled; - if (done == npages) - break; - } - - recv_buf->npages = npages; - return 0; - -err: - for (i = 0; i < npages; i++) { - if (recv_buf->page_list[i]) - __free_page(recv_buf->page_list[i]); - } - - kvfree(recv_buf->page_list); - return -ENOMEM; -} -static void unregister_dma_pages(struct mlx5_core_dev *mdev, u32 npages, - u32 *mkey_in) -{ - dma_addr_t addr; - __be64 *mtt; - int i; - - mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - for (i = npages - 1; i >= 0; i--) { - addr = be64_to_cpu(mtt[i]); - dma_unmap_single(mdev->device, addr, PAGE_SIZE, - DMA_FROM_DEVICE); - } -} - -static int register_dma_pages(struct mlx5_core_dev *mdev, u32 npages, - struct page **page_list, u32 *mkey_in) -{ - dma_addr_t addr; - __be64 *mtt; - int i; - - mtt = (__be64 *)MLX5_ADDR_OF(create_mkey_in, mkey_in, klm_pas_mtt); - - for (i = 0; i < npages; i++) { - addr = dma_map_page(mdev->device, page_list[i], 0, PAGE_SIZE, - DMA_FROM_DEVICE); - if (dma_mapping_error(mdev->device, addr)) - goto error; - - *mtt++ = cpu_to_be64(addr); - } - - return 0; - -error: - unregister_dma_pages(mdev, i, mkey_in); - return -ENOMEM; -} - static void mlx5vf_free_qp_recv_resources(struct mlx5_core_dev *mdev, struct mlx5_vhca_qp *qp) { struct mlx5_vhca_recv_buf *recv_buf = &qp->recv_buf; mlx5_core_destroy_mkey(mdev, recv_buf->mkey); - unregister_dma_pages(mdev, recv_buf->npages, recv_buf->mkey_in); + unregister_dma_pages(mdev, recv_buf->npages, recv_buf->mkey_in, + &recv_buf->state, DMA_FROM_DEVICE); kvfree(recv_buf->mkey_in); - free_recv_pages(&qp->recv_buf); + free_page_list(recv_buf->npages, recv_buf->page_list); } static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, @@ -1450,10 +1420,12 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, struct mlx5_vhca_recv_buf *recv_buf = &qp->recv_buf; int err; - err = alloc_recv_pages(recv_buf, npages); - if (err < 0) + err = mlx5vf_add_pages(&recv_buf->page_list, npages); + if (err) return err; + recv_buf->npages = npages; + recv_buf->mkey_in = alloc_mkey_in(npages, pdn); if (!recv_buf->mkey_in) { err = -ENOMEM; @@ -1461,24 +1433,25 @@ static int mlx5vf_alloc_qp_recv_resources(struct mlx5_core_dev *mdev, } err = register_dma_pages(mdev, npages, recv_buf->page_list, - recv_buf->mkey_in); + recv_buf->mkey_in, &recv_buf->state, + DMA_FROM_DEVICE); if (err) goto err_register_dma; - err = create_mkey(mdev, npages, NULL, recv_buf->mkey_in, - &recv_buf->mkey); + err = create_mkey(mdev, npages, recv_buf->mkey_in, &recv_buf->mkey); if (err) goto err_create_mkey; return 0; err_create_mkey: - unregister_dma_pages(mdev, npages, recv_buf->mkey_in); + unregister_dma_pages(mdev, npages, recv_buf->mkey_in, &recv_buf->state, + DMA_FROM_DEVICE); err_register_dma: kvfree(recv_buf->mkey_in); recv_buf->mkey_in = NULL; end: - free_recv_pages(recv_buf); + free_page_list(npages, recv_buf->page_list); return err; } diff --git a/drivers/vfio/pci/mlx5/cmd.h b/drivers/vfio/pci/mlx5/cmd.h index 25dd6ff54591..d7821b5ca772 100644 --- a/drivers/vfio/pci/mlx5/cmd.h +++ b/drivers/vfio/pci/mlx5/cmd.h @@ -53,7 +53,8 @@ struct mlx5_vf_migration_header { }; struct mlx5_vhca_data_buffer { - struct sg_append_table table; + struct page **page_list; + struct dma_iova_state state; loff_t start_pos; u64 length; u32 npages; @@ -63,10 +64,6 @@ struct mlx5_vhca_data_buffer { u8 stop_copy_chunk_num; struct list_head buf_elm; struct mlx5_vf_migration_file *migf; - /* Optimize mlx5vf_get_migration_page() for sequential access */ - struct scatterlist *last_offset_sg; - unsigned int sg_last_entry; - unsigned long last_offset; }; struct mlx5vf_async_data { @@ -133,6 +130,7 @@ struct mlx5_vhca_cq { struct mlx5_vhca_recv_buf { u32 npages; struct page **page_list; + struct dma_iova_state state; u32 next_rq_offset; u32 *mkey_in; u32 mkey; @@ -224,8 +222,17 @@ struct mlx5_vhca_data_buffer * mlx5vf_get_data_buffer(struct mlx5_vf_migration_file *migf, u32 npages, enum dma_data_direction dma_dir); void mlx5vf_put_data_buffer(struct mlx5_vhca_data_buffer *buf); -struct page *mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, - unsigned long offset); +static inline struct page * +mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, + unsigned long offset) +{ + int page_entry = offset / PAGE_SIZE; + + if (page_entry >= buf->npages) + return NULL; + + return buf->page_list[page_entry]; +} void mlx5vf_state_mutex_unlock(struct mlx5vf_pci_core_device *mvdev); void mlx5vf_disable_fds(struct mlx5vf_pci_core_device *mvdev, enum mlx5_vf_migf_state *last_save_state); diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index a1dbee3be1e0..d6cf97101c41 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -34,37 +34,6 @@ static struct mlx5vf_pci_core_device *mlx5vf_drvdata(struct pci_dev *pdev) core_device); } -struct page * -mlx5vf_get_migration_page(struct mlx5_vhca_data_buffer *buf, - unsigned long offset) -{ - unsigned long cur_offset = 0; - struct scatterlist *sg; - unsigned int i; - - /* All accesses are sequential */ - if (offset < buf->last_offset || !buf->last_offset_sg) { - buf->last_offset = 0; - buf->last_offset_sg = buf->table.sgt.sgl; - buf->sg_last_entry = 0; - } - - cur_offset = buf->last_offset; - - for_each_sg(buf->last_offset_sg, sg, - buf->table.sgt.orig_nents - buf->sg_last_entry, i) { - if (offset < sg->length + cur_offset) { - buf->last_offset_sg = sg; - buf->sg_last_entry += i; - buf->last_offset = cur_offset; - return nth_page(sg_page(sg), - (offset - cur_offset) / PAGE_SIZE); - } - cur_offset += sg->length; - } - return NULL; -} - static void mlx5vf_disable_fd(struct mlx5_vf_migration_file *migf) { mutex_lock(&migf->lock);