From patchwork Wed Jun 15 16:12:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B7C8CCA480 for ; Wed, 15 Jun 2022 16:13:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0C836B0088; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B9EC16B0092; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F20346B008A; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 66A526B0085 for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 4027F12016D for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.08.1A61452 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf13.hostedemail.com (Postfix) with ESMTP id 61BE920088 for ; Wed, 15 Jun 2022 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=mtQBRhKAuvCwGThlB7Jvwgfm5H5uDyU50loVdVNL23w=; b=KVDw0upQ0XsQg5ld50QmMKqfuM zK5MPimorBAJ7iSWd598AK9ReJelJZFlnTySPkRItKhv7bP/HLKFr/t74fjhCUYg2gosfFc/HpmfB /uDGsntT1kxL72jImNb/UFODuVrXKC9lsUDvZ4UW5a1rUSzX/wh1e4Cuh51xRK/zxRoNtJ7Qs6Zqh IjXnp4C6Sg+k/XB8LDO0bepfPKYmoHBQPGetKwzkk3PCzYugXToX+2mSw7D2NB4FuIQBaIkTSVgJZ 3Zgh2D3yclSbrwsgN2LqB5pxvwNkF+q+9SbeJz1ou8WOXyUkPkBbBVzH357TzDa2nL3X4nlBJrqGE 5gcc3BzA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdN-0084iQ-L4; Wed, 15 Jun 2022 10:12:46 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdG-0004Zr-RR; Wed, 15 Jun 2022 10:12:38 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Chaitanya Kulkarni Date: Wed, 15 Jun 2022 10:12:13 -0600 Message-Id: <20220615161233.17527-2-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, kch@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 01/21] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=KVDw0upQ; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf13.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mtQBRhKAuvCwGThlB7Jvwgfm5H5uDyU50loVdVNL23w=; b=Ur6y4XwVK6ima0ehMWNas2HJGMEtJcm3p+Zkx7JlwWT6KEFZGuKynRmtdJzdQFPpCrXojq xz9/leM+rdE99JBluosluPYmL1nMA4tR7jp+VodTSR2V1aI8DaqXI+BXd9yfcjQE9MrUYQ seozGVfl2yNNhE6Udgna/n1C2NHdZ6s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=QMK0D/fD5IeUmfUjhFG/6J8TglwnsK5zVVtBxJQepnmrpt/1RdnYhK8vsyYjsmZHkA1/kO UIJqH4/1oJsnRwES+Mzs+6jqnuKYdFQe/lbkJAaRIIa7MAF5bvumHOU+/b6NN02mISBh/7 M/FdP3gGTtiJY2y82vunWj8MCKAMy1Y= X-Stat-Signature: 3bt9agahranagio7qszhf58m4x1we9wp X-Rspamd-Queue-Id: 61BE920088 X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=KVDw0upQ; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf13.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam10 X-HE-Tag: 1655309573-238696 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make use of the third free LSB in scatterlist's page_link on 64bit systems. The extra bit will be used by dma_[un]map_sg_p2pdma() to determine when a given SGL segments dma_address points to a PCI bus address. dma_unmap_sg_p2pdma() will need to perform different cleanup when a segment is marked as a bus address. The new bit will only be used when CONFIG_PCI_P2PDMA is set; this means PCI P2PDMA will require CONFIG_64BIT. This should be acceptable as the majority of P2PDMA use cases are restricted to newer root complexes and roughly require the extra address space for memory BARs used in the transactions. Signed-off-by: Logan Gunthorpe Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig --- drivers/pci/Kconfig | 5 +++++ include/linux/scatterlist.h | 44 ++++++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 1 deletion(-) diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 133c73207782..5cc7cba1941f 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -164,6 +164,11 @@ config PCI_PASID config PCI_P2PDMA bool "PCI peer-to-peer transfer support" depends on ZONE_DEVICE + # + # The need for the scatterlist DMA bus address flag means PCI P2PDMA + # requires 64bit + # + depends on 64BIT select GENERIC_ALLOCATOR help Enableѕ drivers to do PCI peer-to-peer transactions to and from diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index 7ff9d6386c12..6561ca8aead8 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -64,12 +64,24 @@ struct sg_append_table { #define SG_CHAIN 0x01UL #define SG_END 0x02UL +/* + * bit 2 is the third free bit in the page_link on 64bit systems which + * is used by dma_unmap_sg() to determine if the dma_address is a + * bus address when doing P2PDMA. + */ +#ifdef CONFIG_PCI_P2PDMA +#define SG_DMA_BUS_ADDRESS 0x04UL +static_assert(__alignof__(struct page) >= 8); +#else +#define SG_DMA_BUS_ADDRESS 0x00UL +#endif + /* * We overload the LSB of the page pointer to indicate whether it's * a valid sg entry, or whether it points to the start of a new scatterlist. * Those low bits are there for everyone! (thanks mason :-) */ -#define SG_PAGE_LINK_MASK (SG_CHAIN | SG_END) +#define SG_PAGE_LINK_MASK (SG_CHAIN | SG_END | SG_DMA_BUS_ADDRESS) static inline unsigned int __sg_flags(struct scatterlist *sg) { @@ -91,6 +103,11 @@ static inline bool sg_is_last(struct scatterlist *sg) return __sg_flags(sg) & SG_END; } +static inline bool sg_is_dma_bus_address(struct scatterlist *sg) +{ + return __sg_flags(sg) & SG_DMA_BUS_ADDRESS; +} + /** * sg_assign_page - Assign a given page to an SG entry * @sg: SG entry @@ -245,6 +262,31 @@ static inline void sg_unmark_end(struct scatterlist *sg) sg->page_link &= ~SG_END; } +/** + * sg_dma_mark_bus address - Mark the scatterlist entry as a bus address + * @sg: SG entryScatterlist + * + * Description: + * Marks the passed in sg entry to indicate that the dma_address is + * a bus address and doesn't need to be unmapped. + **/ +static inline void sg_dma_mark_bus_address(struct scatterlist *sg) +{ + sg->page_link |= SG_DMA_BUS_ADDRESS; +} + +/** + * sg_unmark_pci_p2pdma - Unmark the scatterlist entry as a bus address + * @sg: SG entryScatterlist + * + * Description: + * Clears the bus address mark. + **/ +static inline void sg_dma_unmark_bus_address(struct scatterlist *sg) +{ + sg->page_link &= ~SG_DMA_BUS_ADDRESS; +} + /** * sg_phys - Return physical address of an sg entry * @sg: SG entry From patchwork Wed Jun 15 16:12:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DC32CCA483 for ; Wed, 15 Jun 2022 16:13:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E582D6B007B; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AB2316B0080; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D2076B0088; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EF56F6B0085 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin31.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CBBBA205CA for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.31.222AB7C Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf20.hostedemail.com (Postfix) with ESMTP id 6902A1C0023 for ; Wed, 15 Jun 2022 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=dDcdCEaWGv/l2098gGlW8wOT/rK8fbKH3lNyefLy2uU=; b=jrRPehtQjnwFT9fc0jh95GHygA bdV8reEzaqp9WJZ335mwCqqfi7aKzHcfq44yZsd+aV9wUibpB3xBhAto5o+j6yvnirP4cGWR5S39p 0XfCIhvUgy9ESkAaN7jtBYdJQNfWBEHDDPdl/8tgme6cpP+30ZChoOIeNm3TjGjCxU6KnwuMf+AIW edsKz20JAl61eh+aPomqwRmCKt2oCpdHuAQot81N38CC2l2BwtrnjKP6eBsiZLBKhti56rrEZNJ9R v7voaboUX7kQV65eKr7anMUK+TplynbGqVV1EGTf3xAGIrx9t9dWVp/Gjh2UDf62QHfpf5SloTH4v ZZm2jk8w==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdN-0084iR-L8; Wed, 15 Jun 2022 10:12:47 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdG-0004Zu-Ul; Wed, 15 Jun 2022 10:12:38 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Bjorn Helgaas , Chaitanya Kulkarni Date: Wed, 15 Jun 2022 10:12:14 -0600 Message-Id: <20220615161233.17527-3-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, bhelgaas@google.com, jhubbard@nvidia.com, rcampbell@nvidia.com, kch@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 02/21] PCI/P2PDMA: Attempt to set map_type if it has not been set X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=OLg72u5TKNoZbt7eXv/dJSHqHFCIMdAblRdWqcZbjBO6S8vrV4CwV7Ow9iQa5R/w4JDJar MUFZZ0O3nXsxtQ868JKimdzSxJyTx3bSGmVHq/YgumupVVCiH74fMkZbV4s8YmylRs2jUE K64tLQY1BwcXUHdpsnFahVKh5mR2/gY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dDcdCEaWGv/l2098gGlW8wOT/rK8fbKH3lNyefLy2uU=; b=2XDLqMJMHUeQkso6ee+MeBkWRL6hYH4egoH+RuimJchrtBgOZxOJMvEPKpoC9OBV0N9okP +9hFz4u0QTSpR7gInS7yYhtLt42RI6HPYCnPWPgb7+jiLX10hRA20ohO1ACt4+aVj2oBfa 3/gkxsipZHynZFAc2mmj3r9Hbml+oW0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=jrRPehtQ; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf20.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=jrRPehtQ; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf20.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6902A1C0023 X-Stat-Signature: 5r4guacfysdpxmb4zqgqt11fqwcowebr X-HE-Tag: 1655309573-647040 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt to find the mapping type for P2PDMA pages on the first DMA map attempt if it has not been done ahead of time. Previously, the mapping type was expected to be calculated ahead of time, but if pages are to come from userspace then there's no way to ensure the path was checked ahead of time. This change will calculate the mapping type if it hasn't pre-calculated so it is no longer invalid to call pci_p2pdma_map_sg() before the mapping type is calculated, so drop the WARN_ON when that is the case. Signed-off-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig --- drivers/pci/p2pdma.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 462b429ad243..4e8bc457e29a 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -854,6 +854,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, struct pci_dev *provider = to_p2p_pgmap(pgmap)->provider; struct pci_dev *client; struct pci_p2pdma *p2pdma; + int dist; if (!provider->p2pdma) return PCI_P2PDMA_MAP_NOT_SUPPORTED; @@ -870,6 +871,10 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, type = xa_to_value(xa_load(&p2pdma->map_types, map_types_idx(client))); rcu_read_unlock(); + + if (type == PCI_P2PDMA_MAP_UNKNOWN) + return calc_map_type_and_dist(provider, client, &dist, true); + return type; } @@ -912,7 +917,7 @@ int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg, case PCI_P2PDMA_MAP_BUS_ADDR: return __pci_p2pdma_map_sg(p2p_pgmap, dev, sg, nents); default: - WARN_ON_ONCE(1); + /* Mapping is not Supported */ return 0; } } From patchwork Wed Jun 15 16:12:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882628 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A237CCA48B for ; Wed, 15 Jun 2022 16:13:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FEBC6B0082; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 13D276B0085; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9056E6B007E; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 476556B0080 for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 19504802D2 for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.01.878A4B1 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf24.hostedemail.com (Postfix) with ESMTP id D852B18008B for ; Wed, 15 Jun 2022 16:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=4HmuropDcIG289DLyabNfN+2ik7mrSydIe74mWngsTc=; b=BI+jXRl41cxcyPFcEEqTyg3c80 yY9fn5p7ZYssFU+KGqsAal8jqOVzMfxNHVWyLVfGRBEK4ImR8dMO0/6rGc/nHXZi1AH2Ev1HQdl11 D1/gRPthJvv0Te8Gpz9FehuM7HuvAu4ffi1w1TiLffn0T4dqlKZ68wiegqqqbXY5gYQ0bdUOe9voZ h+cLugNvATmvZdBUHVGCvDRlDpU8cIfebR5EsZQBFWse+LGBukPIGO+RVLClTNVIsg7wRhJjSBqVB Xa67sY4qC9hOrc1eIKNDIsx0BdhdIMKESeNmK1v1SchFbTkLy16qNup8GOKZhv/OIWzoLvajfWkLR XGLCMgfg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdN-0084iS-LB; Wed, 15 Jun 2022 10:12:47 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004Zx-23; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Bjorn Helgaas , Jason Gunthorpe , Chaitanya Kulkarni Date: Wed, 15 Jun 2022 10:12:15 -0600 Message-Id: <20220615161233.17527-4-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, bhelgaas@google.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com, kch@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 03/21] PCI/P2PDMA: Expose pci_p2pdma_map_type() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4HmuropDcIG289DLyabNfN+2ik7mrSydIe74mWngsTc=; b=vynGU3Ocogbh3G4c6CE5QAJEj1f0MzBVTcMI0IPdK6x8Vr0ZhIZoH/GRIwOpD2kUWvQYjE 957LBuuWa/f0ewQ42w/6r8w9juNjVBNBzXXdbA7ctxXwOI6uaePxAqSjt6+epmpumZ/VgZ TjCq1mIpG7iwOCqJ81lLLRaR6R700vs= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=BI+jXRl4; spf=pass (imf24.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=yRk3YjnJY2hWMajJttF1k+IR8WMU+rp4ZP3mbsreRm7KLOdqVUFSsWQCtHawZBBzAFemkn qYnhxZTIjj6SNUvE7iSnpEaphgfAojTsK+n4AtojQRAY1J7lkvmMRNaEaOkfgO+b4OIUaV 7PnQ+H8vni79ZWU0CJIb3kc7HLlpYXk= X-Stat-Signature: h57xgeq3yudmz4be5a36s8c541mb66yr X-Rspamd-Queue-Id: D852B18008B X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=BI+jXRl4; spf=pass (imf24.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1655309574-678585 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pci_p2pdma_map_type() will be needed by the dma-iommu map_sg implementation because it will need to determine the mapping type ahead of actually doing the mapping to create the actual IOMMU mapping. Prototypes for this helper are added to dma-map-ops.h as they are only useful to dma map implementations and don't need to pollute the public pci-p2pdma header Signed-off-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Reviewed-by: Jason Gunthorpe Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig --- drivers/pci/p2pdma.c | 25 +++++++++++++-------- include/linux/dma-map-ops.h | 45 +++++++++++++++++++++++++++++++++++++ 2 files changed, 61 insertions(+), 9 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 4e8bc457e29a..10b1d5c2b5de 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -10,6 +10,7 @@ #define pr_fmt(fmt) "pci-p2pdma: " fmt #include +#include #include #include #include @@ -20,13 +21,6 @@ #include #include -enum pci_p2pdma_map_type { - PCI_P2PDMA_MAP_UNKNOWN = 0, - PCI_P2PDMA_MAP_NOT_SUPPORTED, - PCI_P2PDMA_MAP_BUS_ADDR, - PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, -}; - struct pci_p2pdma { struct gen_pool *pool; bool p2pmem_published; @@ -847,8 +841,21 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) } EXPORT_SYMBOL_GPL(pci_p2pmem_publish); -static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, - struct device *dev) +/** + * pci_p2pdma_map_type - return the type of mapping that should be used for + * a given device and pgmap + * @pgmap: the pagemap of a page to determine the mapping type for + * @dev: device that is mapping the page + * + * Returns one of: + * PCI_P2PDMA_MAP_NOT_SUPPORTED - The mapping should not be done + * PCI_P2PDMA_MAP_BUS_ADDR - The mapping should use the PCI bus address + * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE - The mapping should be done normally + * using the CPU physical address (in dma-direct) or an IOVA + * mapping for the IOMMU. + */ +enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, + struct device *dev) { enum pci_p2pdma_map_type type = PCI_P2PDMA_MAP_NOT_SUPPORTED; struct pci_dev *provider = to_p2p_pgmap(pgmap)->provider; diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 0d5b06b3a4a6..d693a0e33bac 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -379,4 +379,49 @@ static inline void debug_dma_dump_mappings(struct device *dev) extern const struct dma_map_ops dma_dummy_ops; +enum pci_p2pdma_map_type { + /* + * PCI_P2PDMA_MAP_UNKNOWN: Used internally for indicating the mapping + * type hasn't been calculated yet. Functions that return this enum + * never return this value. + */ + PCI_P2PDMA_MAP_UNKNOWN = 0, + + /* + * PCI_P2PDMA_MAP_NOT_SUPPORTED: Indicates the transaction will + * traverse the host bridge and the host bridge is not in the + * allowlist. DMA Mapping routines should return an error when + * this is returned. + */ + PCI_P2PDMA_MAP_NOT_SUPPORTED, + + /* + * PCI_P2PDMA_BUS_ADDR: Indicates that two devices can talk to + * each other directly through a PCI switch and the transaction will + * not traverse the host bridge. Such a mapping should program + * the DMA engine with PCI bus addresses. + */ + PCI_P2PDMA_MAP_BUS_ADDR, + + /* + * PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: Indicates two devices can talk + * to each other, but the transaction traverses a host bridge on the + * allowlist. In this case, a normal mapping either with CPU physical + * addresses (in the case of dma-direct) or IOVA addresses (in the + * case of IOMMUs) should be used to program the DMA engine. + */ + PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, +}; + +#ifdef CONFIG_PCI_P2PDMA +enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, + struct device *dev); +#else /* CONFIG_PCI_P2PDMA */ +static inline enum pci_p2pdma_map_type +pci_p2pdma_map_type(struct dev_pagemap *pgmap, struct device *dev) +{ + return PCI_P2PDMA_MAP_NOT_SUPPORTED; +} +#endif /* CONFIG_PCI_P2PDMA */ + #endif /* _LINUX_DMA_MAP_OPS_H */ From patchwork Wed Jun 15 16:12:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75987CCA49B for ; Wed, 15 Jun 2022 16:13:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F7646B0075; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 078876B007B; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3D7B6B0075; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7C9026B0074 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 57417416 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.23.AABBD78 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf18.hostedemail.com (Postfix) with ESMTP id 9FB211C007E for ; Wed, 15 Jun 2022 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=h/JTrfXf6Jyg1W5dIU7TUpc6v7/794fb7gFuUxisHhk=; b=mTSbObFcFcLjAbcPCQ0vcxdWCT J3Znx7w4sCw8lrk9hAqz5j2XxM7S+9HG5FhOgUk9AXvzt8KJ0mvrjBcZwHyHC/NTTXMghMrAV+Oxn wH6Mi2fpuD2loMNvtJFQKZ8sLiE+oXzKDrm05++476ArYK2Rs5Pio5IRR9ZvjPYiuia04IlzYr2rz M2F99hsJ69BUsG5d6B9I7d2kEGyUdKaEDIWabcUMD1TrhuujmoTF9roi/q3uIFJzDEQGcpCJeqI7t iIX8rhU6eWp3O6xxaLYHuUsaMaqHcJ/0w0P8EgyCa8I0uIY69gbJRvo+wqbkXGAfwxMVaWJP8+EGB NKL4PKcg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdN-0084iT-LC; Wed, 15 Jun 2022 10:12:46 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004a0-5o; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Bjorn Helgaas Date: Wed, 15 Jun 2022 10:12:16 -0600 Message-Id: <20220615161233.17527-5-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com, bhelgaas@google.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 04/21] PCI/P2PDMA: Introduce helpers for dma_map_sg implementations X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h/JTrfXf6Jyg1W5dIU7TUpc6v7/794fb7gFuUxisHhk=; b=IYZiEwQYLX5a11ZB1p+bxz/dBhsRPE4pV9aH/JQKPe1uQ6tt8pjgnGbzIh7rYiRtzARM/2 IJeMCMXooHcC6G2lwjD7geI/aovdXgsTm6TXkRTVEOJeRMgfV8a0nM7oVKltUND7CRo9kk Rx9jsup9Aniyxf34BkdIRsXR9BTRusg= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mTSbObFc; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf18.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=rkf1ZIsHUB0OXZ+Zqqr2Cm5U/LgQGJb3H91eYkwmAS8SrP76+HnWLsKwGT+OA378qqGGPf 2CARpwTmcZ+NE02CaKNV0Ea06W7tsOaLWbi7rtt4F4ddVEO9JF01kf9hxB2lMlm7stTPP1 5MdJKiUn9UVeSoZJCHxC/oQa+xjfeGI= X-Stat-Signature: sdfwhjtz5mbciqi6h3sjkexx4fnfyc35 X-Rspamd-Queue-Id: 9FB211C007E X-Rspam-User: X-Rspamd-Server: rspam05 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mTSbObFc; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf18.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-HE-Tag: 1655309573-565961 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add pci_p2pdma_map_segment() as a helper for simple dma_map_sg() implementations. It takes an scatterlist segment that must point to a pci_p2pdma struct page and will map it if the mapping requires a bus address. The return value indicates whether the mapping required a bus address or whether the caller still needs to map the segment normally. If the segment should not be mapped, -EREMOTEIO is returned. This helper uses a state structure to track the changes to the pgmap across calls and avoid needing to lookup into the xarray for every page. Also add pci_p2pdma_map_bus_segment() which is useful for IOMMU dma_map_sg() implementations where the sg segment containing the page differs from the sg segment containing the DMA address. Prototypes for these helpers are added to dma-map-ops.h as they are only useful to dma map implementations and don't need to pollute the public pci-p2pdma header. Signed-off-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Reviewed-by: Christoph Hellwig --- drivers/pci/p2pdma.c | 59 +++++++++++++++++++++++++++++++++++++ include/linux/dma-map-ops.h | 21 +++++++++++++ 2 files changed, 80 insertions(+) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 10b1d5c2b5de..2fc0f4750a2e 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -951,6 +951,65 @@ void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, } EXPORT_SYMBOL_GPL(pci_p2pdma_unmap_sg_attrs); +/** + * pci_p2pdma_map_segment - map an sg segment determining the mapping type + * @state: State structure that should be declared outside of the for_each_sg() + * loop and initialized to zero. + * @dev: DMA device that's doing the mapping operation + * @sg: scatterlist segment to map + * + * This is a helper to be used by non-IOMMU dma_map_sg() implementations where + * the sg segment is the same for the page_link and the dma_address. + * + * Attempt to map a single segment in an SGL with the PCI bus address. + * The segment must point to a PCI P2PDMA page and thus must be + * wrapped in a is_pci_p2pdma_page(sg_page(sg)) check. + * + * Returns the type of mapping used and maps the page if the type is + * PCI_P2PDMA_MAP_BUS_ADDR. + */ +enum pci_p2pdma_map_type +pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, + struct scatterlist *sg) +{ + if (state->pgmap != sg_page(sg)->pgmap) { + state->pgmap = sg_page(sg)->pgmap; + state->map = pci_p2pdma_map_type(state->pgmap, dev); + state->bus_off = to_p2p_pgmap(state->pgmap)->bus_offset; + } + + if (state->map == PCI_P2PDMA_MAP_BUS_ADDR) { + sg->dma_address = sg_phys(sg) + state->bus_off; + sg_dma_len(sg) = sg->length; + sg_dma_mark_bus_address(sg); + } + + return state->map; +} + +/** + * pci_p2pdma_map_bus_segment - map an sg segment pre determined to + * be mapped with PCI_P2PDMA_MAP_BUS_ADDR + * @pg_sg: scatterlist segment with the page to map + * @dma_sg: scatterlist segment to assign a DMA address to + * + * This is a helper for iommu dma_map_sg() implementations when the + * segment for the DMA address differs from the segment containing the + * source page. + * + * pci_p2pdma_map_type() must have already been called on the pg_sg and + * returned PCI_P2PDMA_MAP_BUS_ADDR. + */ +void pci_p2pdma_map_bus_segment(struct scatterlist *pg_sg, + struct scatterlist *dma_sg) +{ + struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(sg_page(pg_sg)->pgmap); + + dma_sg->dma_address = sg_phys(pg_sg) + pgmap->bus_offset; + sg_dma_len(dma_sg) = pg_sg->length; + sg_dma_mark_bus_address(dma_sg); +} + /** * pci_p2pdma_enable_store - parse a configfs/sysfs attribute store * to enable p2pdma diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index d693a0e33bac..752f91e5eb5d 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -413,15 +413,36 @@ enum pci_p2pdma_map_type { PCI_P2PDMA_MAP_THRU_HOST_BRIDGE, }; +struct pci_p2pdma_map_state { + struct dev_pagemap *pgmap; + int map; + u64 bus_off; +}; + #ifdef CONFIG_PCI_P2PDMA enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, struct device *dev); +enum pci_p2pdma_map_type +pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, + struct scatterlist *sg); +void pci_p2pdma_map_bus_segment(struct scatterlist *pg_sg, + struct scatterlist *dma_sg); #else /* CONFIG_PCI_P2PDMA */ static inline enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, struct device *dev) { return PCI_P2PDMA_MAP_NOT_SUPPORTED; } +static inline enum pci_p2pdma_map_type +pci_p2pdma_map_segment(struct pci_p2pdma_map_state *state, struct device *dev, + struct scatterlist *sg) +{ + return PCI_P2PDMA_MAP_NOT_SUPPORTED; +} +static inline void pci_p2pdma_map_bus_segment(struct scatterlist *pg_sg, + struct scatterlist *dma_sg) +{ +} #endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_DMA_MAP_OPS_H */ From patchwork Wed Jun 15 16:12:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91DF4CCA47F for ; Wed, 15 Jun 2022 16:13:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B07616B0085; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E0316B0088; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD5086B008C; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 77EDB6B008A for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4742D21183 for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.28.7AC2F13 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf21.hostedemail.com (Postfix) with ESMTP id E37D21C0098 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=qwHhhptcjs675dAW5wV9iUf/ZWGXFcOtu4sOGzEzFjs=; b=YmHHnpHXs1wVWqcPzTDFNBWVuK Ufuk1/bP6jiv6cicN1+HWISek74RNoL6XDVXgD7MfiZHGSYeMFU1VMkaI3HdZvcSSTBlpEk975Xet lM/pDCalBifWIevWRcadscS84nE643nE1GCU6X2f1+CZXXJji+nzCGP+8WVgEXEXkQvOK3IIGXXtx 3weJ0ks+ru8GXZrthAQyc6RkGCl0P/w2PaCX0ZfOx+CCLOOuMC0RnH41/aJ+0ioIpS0vJoBDZxxr1 bWVy1Hbv4a7Yb3N5PcrE8KcO4tvJf4kt4PJqPiPl8lBLKgA/ER3s3R9j0Y92JXvdG9PRIgDrYaSu/ 0ccdiREQ==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdS-0084iR-Tu; Wed, 15 Jun 2022 10:12:52 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004a3-92; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Jason Gunthorpe Date: Wed, 15 Jun 2022 10:12:17 -0600 Message-Id: <20220615161233.17527-6-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 05/21] dma-mapping: allow EREMOTEIO return code for P2PDMA transfers X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309576; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qwHhhptcjs675dAW5wV9iUf/ZWGXFcOtu4sOGzEzFjs=; b=Qbkl5jmpdisSNx1h/UhknRTNazh5da0CNyCFAAyAwFRL+q/a69lseRW6LjOtV3vuJpEvpP c95nBg2FurX8LqyRl8YXNAm8psUk3eyZGXOOKihyYJNioOc5GTc2Sa8v0ts/iQgOuZ5j/f mBsyi6+1qOSfLyH3Ig/edNijwnG98dE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=YmHHnpHX; spf=pass (imf21.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309576; a=rsa-sha256; cv=none; b=3t4bCEYnrfkTAx7+neppWqGJj8ghyaVyFeplNXXtxvdMKiyCIy3xPURSMyQKlWwS/NoXTR vHJuwwzQ6eQNqEnWDTzYEbXtaWdQLlZ9F1JNFGxhMDJp25jDH9KVVLwAjdU1hRC/g/RbD6 H8ai38vqi8xw5xcgbm5b66LF2FgIhXQ= X-Stat-Signature: aour94m4wujr9s3a76w1keqirbbedq41 X-Rspamd-Queue-Id: E37D21C0098 X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=YmHHnpHX; spf=pass (imf21.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1655309575-908049 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add EREMOTEIO error return to dma_map_sgtable() which will be used by .map_sg() implementations that detect P2PDMA pages that the underlying DMA device cannot access. Signed-off-by: Logan Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Christoph Hellwig --- kernel/dma/mapping.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index db7244291b74..9f65d1041638 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -197,7 +197,7 @@ static int __dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, if (ents > 0) debug_dma_map_sg(dev, sg, nents, ents, dir, attrs); else if (WARN_ON_ONCE(ents != -EINVAL && ents != -ENOMEM && - ents != -EIO)) + ents != -EIO && ents != -EREMOTEIO)) return -EIO; return ents; @@ -255,6 +255,8 @@ EXPORT_SYMBOL(dma_map_sg_attrs); * complete the mapping. Should succeed if retried later. * -EIO Legacy error code with an unknown meaning. eg. this is * returned if a lower level call returned DMA_MAPPING_ERROR. + * -EREMOTEIO The DMA device cannot access P2PDMA memory specified in + * the sg_table. This will not succeed if retried. */ int dma_map_sgtable(struct device *dev, struct sg_table *sgt, enum dma_data_direction dir, unsigned long attrs) From patchwork Wed Jun 15 16:12:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB157CCA473 for ; Wed, 15 Jun 2022 16:13:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72F8C6B0089; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44B036B0080; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A64136B0082; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 52D106B0083 for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 34340205CA for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.15.1D17321 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf04.hostedemail.com (Postfix) with ESMTP id CFEEF40082 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=jCSZ8IiexiLg6xeZs8NoCeAkNyFMm5P24LUC66Ooo74=; b=m4IINGTPtkliYLz4Gks3cnB1BC biy1HwuRogvCGBxDB8WzbLKy7Tv6XHTwwuZjoTlU6uy7INqp6djC32zkMwE6o7FOCR9M/as+xhtMY RftFad8HVViYx193lifsQlULX/6w3qpMBUtP46fk0yrdwZG2CtG/6hhTERwv0YGk/5yuWX89lbFXL q8hjAOPKWOvSbzOapLr9M9/kYn3zwNhqnEuvhqBHHG/6MO37pzDLwD5obNAkrt7oqZC4J0Y45dyVG b9I9K2TKGsn8qzfoGTjbjW0ewRVz/NFzMsfLyQoBmQjP/1M17hie9ZE1xo6LWSHCMboNOi44r34qG KHX9qJIg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdS-0084iQ-GY; Wed, 15 Jun 2022 10:12:52 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004a6-CX; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:18 -0600 Message-Id: <20220615161233.17527-7-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 06/21] dma-direct: support PCI P2PDMA pages in dma-direct map_sg X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jCSZ8IiexiLg6xeZs8NoCeAkNyFMm5P24LUC66Ooo74=; b=TKFwWhVDKlASD6qLhvZBRol1tnxHammdpB8lVuVC0RBqdPa9D8qSk0Jj1MUoZJwfGRxs1i BEQgc1em+OuwTw7AQMFT0aV3ekOcIsVxakiDgeAW0JVn23hFkLlBS/va4YtnJnVSNNeSHV +k2yOjzXQapx3gZ96ukEbtE+Ba26KUw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=m4IINGTP; spf=pass (imf04.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=yPQqpQXPPOm+vYXnhMJ+w82CivRqC+87mdrhOBPRr9k3gZ7mtE4NOVsoUAMX9AFQRFgzsq +KcfQL4MwCvE2i+WloHm/eHJuTBt7gI3aOKWOBoBv6+2n2lFjSRt3yRuV8FrLI28ubgnZy Y/N3V5UxHqShpafh4sb+V7q5iNByCZk= X-Stat-Signature: jmrd1encbr837fxe6atic5zdoobn71hx X-Rspamd-Queue-Id: CFEEF40082 X-Rspamd-Server: rspam11 X-Rspam-User: Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=m4IINGTP; spf=pass (imf04.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-HE-Tag: 1655309575-921604 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add PCI P2PDMA support for dma_direct_map_sg() so that it can map PCI P2PDMA pages directly without a hack in the callers. This allows for heterogeneous SGLs that contain both P2PDMA and regular pages. A P2PDMA page may have three possible outcomes when being mapped: 1) If the data path between the two devices doesn't go through the root port, then it should be mapped with a PCI bus address 2) If the data path goes through the host bridge, it should be mapped normally, as though it were a CPU physical address 3) It is not possible for the two devices to communicate and thus the mapping operation should fail (and it will return -EREMOTEIO). SGL segments that contain PCI bus addresses are marked with sg_dma_mark_pci_p2pdma() and are ignored when unmapped. P2PDMA mappings are also failed if swiotlb needs to be used on the mapping. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- kernel/dma/direct.c | 43 +++++++++++++++++++++++++++++++++++++------ kernel/dma/direct.h | 8 +++++++- 2 files changed, 44 insertions(+), 7 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index e978f36e6be8..133a4be2d3e5 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -454,29 +454,60 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } +/* + * Unmaps segments, except for ones marked as pci_p2pdma which do not + * require any further action as they contain a bus address. + */ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { struct scatterlist *sg; int i; - for_each_sg(sgl, sg, nents, i) - dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, - attrs); + for_each_sg(sgl, sg, nents, i) { + if (sg_is_dma_bus_address(sg)) + sg_dma_unmark_bus_address(sg); + else + dma_direct_unmap_page(dev, sg->dma_address, + sg_dma_len(sg), dir, attrs); + } } #endif int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { - int i; + struct pci_p2pdma_map_state p2pdma_state = {}; + enum pci_p2pdma_map_type map; struct scatterlist *sg; + int i, ret; for_each_sg(sgl, sg, nents, i) { + if (is_pci_p2pdma_page(sg_page(sg))) { + map = pci_p2pdma_map_segment(&p2pdma_state, dev, sg); + switch (map) { + case PCI_P2PDMA_MAP_BUS_ADDR: + continue; + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Any P2P mapping that traverses the PCI + * host bridge must be mapped with CPU physical + * address and not PCI bus addresses. This is + * done with dma_direct_map_page() below. + */ + break; + default: + ret = -EREMOTEIO; + goto out_unmap; + } + } + sg->dma_address = dma_direct_map_page(dev, sg_page(sg), sg->offset, sg->length, dir, attrs); - if (sg->dma_address == DMA_MAPPING_ERROR) + if (sg->dma_address == DMA_MAPPING_ERROR) { + ret = -EIO; goto out_unmap; + } sg_dma_len(sg) = sg->length; } @@ -484,7 +515,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, out_unmap: dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); - return -EIO; + return ret; } dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index a78c0ba70645..e38ffc5e6bdd 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -8,6 +8,7 @@ #define _KERNEL_DMA_DIRECT_H #include +#include int dma_direct_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size, @@ -87,10 +88,15 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev, phys_addr_t phys = page_to_phys(page) + offset; dma_addr_t dma_addr = phys_to_dma(dev, phys); - if (is_swiotlb_force_bounce(dev)) + if (is_swiotlb_force_bounce(dev)) { + if (is_pci_p2pdma_page(page)) + return DMA_MAPPING_ERROR; return swiotlb_map(dev, phys, size, dir, attrs); + } if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (is_pci_p2pdma_page(page)) + return DMA_MAPPING_ERROR; if (is_swiotlb_active(dev)) return swiotlb_map(dev, phys, size, dir, attrs); From patchwork Wed Jun 15 16:12:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C1A6C433EF for ; Wed, 15 Jun 2022 16:54:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71C4C6B0072; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C9E36B0078; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54DE86B0075; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 353256B0074 for ; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0A9E833C98 for ; Wed, 15 Jun 2022 16:54:55 +0000 (UTC) X-FDA: 79581069750.28.3315DA6 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf31.hostedemail.com (Postfix) with ESMTP id 934B12009E for ; Wed, 15 Jun 2022 16:54:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=Qrm5texuQYIS48hhNa5mDas14Fbkzf7AsffQMeTIYTY=; b=J9pqrOPAXO3RIDDVYXSUwsCt7h WM4rlAEmLdBxNnGoRpXHAHe3cgbyQk7eDDoKP65uH509Rd5p+fIRCr0lJHLUZ/l+DPyuZO3oPcQ+O EPbzanBFdkU3kP1JjW6LwM0ZKUgA0KwUJfXv/W8z06gxgvkWB5u4c7g9iZBKeBHiIdysJUzeVJDyA qCvbYOFWjfWQry5RcJsvno+ISdHJYLhWU7MBpDfmq5fbtfobBvbeN+dpZ9nW6IXqWcFdGD8Y2UOmZ XKiyn9bwWdfM1lAMuW0Blj07bzOahCiYs+hNalxeOqd598zkH7HTwJoYEeAKcMQBzpUSH1p+RSIeV G108GVNA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdS-0084iS-7l; Wed, 15 Jun 2022 10:12:51 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004a9-Fo; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Jason Gunthorpe Date: Wed, 15 Jun 2022 10:12:19 -0600 Message-Id: <20220615161233.17527-8-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 07/21] dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655312094; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qrm5texuQYIS48hhNa5mDas14Fbkzf7AsffQMeTIYTY=; b=gMvXFheEg38PgVr1daVLjX2f39RiikSHsn3TRvWnIqrJ03HdTp0z7XDf1ItArhOMQgTp9K lZDSHK5RwuXTl4flZPwwiVUzTqCq5ljbiS++QekyAaFOZoHeC5/G0qnF5KTmyPwtnsQr0S B6cxn5ry1j8GtloUpcc8mF3b/NQH0lA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655312094; a=rsa-sha256; cv=none; b=gICQkTkBPimRFn/6hE4BoC6QBD8cZ/IZwEcqFiZZql8Lb/ztwwiYLaa22LV8d1trsuvWhk pqR8HGFsdzmpwH0/i0i8BEtRbY3MgpIfdJ8bY6NlLW6Of7K+uOpTfidxQkmST77q4otlCL nddM2i0Fx4Lm7EoBb0vtQEuZDGbOj3w= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=J9pqrOPA; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=J9pqrOPA; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: qzaxp6ifohcheh9smsbjmoz3x93b9h1o X-Rspamd-Queue-Id: 934B12009E X-HE-Tag: 1655312094-594095 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a flags member to the dma_map_ops structure with one flag to indicate support for PCI P2PDMA. Also, add a helper to check if a device supports PCI P2PDMA. Signed-off-by: Logan Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Christoph Hellwig --- include/linux/dma-map-ops.h | 10 ++++++++++ include/linux/dma-mapping.h | 5 +++++ kernel/dma/mapping.c | 18 ++++++++++++++++++ 3 files changed, 33 insertions(+) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 752f91e5eb5d..4d4161d58ce0 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -11,7 +11,17 @@ struct cma; +/* + * Values for struct dma_map_ops.flags: + * + * DMA_F_PCI_P2PDMA_SUPPORTED: Indicates the dma_map_ops implementation can + * handle PCI P2PDMA pages in the map_sg/unmap_sg operation. + */ +#define DMA_F_PCI_P2PDMA_SUPPORTED (1 << 0) + struct dma_map_ops { + unsigned int flags; + void *(*alloc)(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index dca2b1355bb1..f7c61b2b4b5e 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -140,6 +140,7 @@ int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma, unsigned long attrs); bool dma_can_mmap(struct device *dev); int dma_supported(struct device *dev, u64 mask); +bool dma_pci_p2pdma_supported(struct device *dev); int dma_set_mask(struct device *dev, u64 mask); int dma_set_coherent_mask(struct device *dev, u64 mask); u64 dma_get_required_mask(struct device *dev); @@ -250,6 +251,10 @@ static inline int dma_supported(struct device *dev, u64 mask) { return 0; } +static inline bool dma_pci_p2pdma_supported(struct device *dev) +{ + return false; +} static inline int dma_set_mask(struct device *dev, u64 mask) { return -EIO; diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 9f65d1041638..21793506fdb6 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -722,6 +722,24 @@ int dma_supported(struct device *dev, u64 mask) } EXPORT_SYMBOL(dma_supported); +bool dma_pci_p2pdma_supported(struct device *dev) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + /* if ops is not set, dma direct will be used which supports P2PDMA */ + if (!ops) + return true; + + /* + * Note: dma_ops_bypass is not checked here because P2PDMA should + * not be used with dma mapping ops that do not have support even + * if the specific device is bypassing them. + */ + + return ops->flags & DMA_F_PCI_P2PDMA_SUPPORTED; +} +EXPORT_SYMBOL_GPL(dma_pci_p2pdma_supported); + #ifdef CONFIG_ARCH_HAS_DMA_SET_MASK void arch_dma_set_mask(struct device *dev, u64 mask); #else From patchwork Wed Jun 15 16:12:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EBBCCCA48E for ; Wed, 15 Jun 2022 16:13:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6ABC6B0083; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A0A06B007B; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24FC26B0082; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E53146B0080 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BC0C73500D for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.28.0361BA3 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf12.hostedemail.com (Postfix) with ESMTP id 46D164009E for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=DdZTvd3yBCTOLfdH85j+eUYcuinHr6KLhT1SRgVQVD0=; b=HIHDMfL5brPKgsuHPyRaaOLIQG Re/GuSfaJP+xJ8dav+q+G6pnRKbJ3UUU56fV8zFqNSNQIf5mSZ5s3GNtGQKSZk0Es4MbL8pPjMTfi DBL5pXyKNE/AbsOV6KZB02bJay7RCSe8ImtccTN0aoZG0rYmSDxUZ2L0GzDQuizWJNbBBRiBCWSay 0Fb5+WuBsVf1w7qupINs5imSr+fBwB66tuFbwF/lhDT2zjFY7b+Nfo6CfbtJTjP4nuujpoP4fY6xG L0BIrMURi6lyD1m87JeM+HqTB1aazkonUzXWq92MsaAZ3cJ4kL/eLh22mHgCsADJ/Pgig5PIiAaNU jrcBTAIA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdR-0084iP-SZ; Wed, 15 Jun 2022 10:12:51 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004aC-K7; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Jason Gunthorpe Date: Wed, 15 Jun 2022 10:12:20 -0600 Message-Id: <20220615161233.17527-9-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 08/21] iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=NuZrDlvoxBVzk9Ab1fJy+qBa7+2wwzZRugeEYGtmTYNARgx/2MbW2E0YriniMDyZC56Jgt CgPJ4nA2zEDW/5l9rTMsaJy1c8oLfYCjMZg4RjEBHa/npst7IeiTfaKU0LjLF0Pu4/iKvo /K00+nI7fUrH1CdbYZQwu6tlMfu3DDU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=HIHDMfL5; spf=pass (imf12.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DdZTvd3yBCTOLfdH85j+eUYcuinHr6KLhT1SRgVQVD0=; b=SEjsnCZf0RMpQKcwc0N8EDiJoKoj36KUdYFQmUIA5LyrVkInGVfE8Sap/k4G/Y18Me/cp2 ihzAT+0m2ZDa+9bizUV13TfRYDw+miSJux/vetwsX3r3URSXSDywA4HEZAJtHxHSSOqF+p Pzzm+R5JrRqPX8DMacqCI49PUvKhfEY= X-Stat-Signature: 8ku4cghc4u8qibk3xyhx3k5cyopcu7xz X-Rspamd-Queue-Id: 46D164009E Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=HIHDMfL5; spf=pass (imf12.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1655309575-225829 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a PCI P2PDMA page is seen, set the IOVA length of the segment to zero so that it is not mapped into the IOVA. Then, in finalise_sg(), apply the appropriate bus address to the segment. The IOVA is not created if the scatterlist only consists of P2PDMA pages. A P2PDMA page may have three possible outcomes when being mapped: 1) If the data path between the two devices doesn't go through the root port, then it should be mapped with a PCI bus address 2) If the data path goes through the host bridge, it should be mapped normally with an IOMMU IOVA. 3) It is not possible for the two devices to communicate and thus the mapping operation should fail (and it will return -EREMOTEIO). Similar to dma-direct, the sg_dma_mark_pci_p2pdma() flag is used to indicate bus address segments. On unmap, P2PDMA segments are skipped over when determining the start and end IOVA addresses. With this change, the flags variable in the dma_map_ops is set to DMA_F_PCI_P2PDMA_SUPPORTED to indicate support for P2PDMA pages. Signed-off-by: Logan Gunthorpe Reviewed-by: Jason Gunthorpe --- drivers/iommu/dma-iommu.c | 68 +++++++++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index f90251572a5d..b01ca0c6a7ab 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -1062,6 +1063,16 @@ static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents, sg_dma_address(s) = DMA_MAPPING_ERROR; sg_dma_len(s) = 0; + if (is_pci_p2pdma_page(sg_page(s)) && !s_iova_len) { + if (i > 0) + cur = sg_next(cur); + + pci_p2pdma_map_bus_segment(s, cur); + count++; + cur_len = 0; + continue; + } + /* * Now fill in the real DMA data. If... * - there is a valid output segment to append to @@ -1158,6 +1169,8 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, struct iova_domain *iovad = &cookie->iovad; struct scatterlist *s, *prev = NULL; int prot = dma_info_to_prot(dir, dev_is_dma_coherent(dev), attrs); + struct dev_pagemap *pgmap = NULL; + enum pci_p2pdma_map_type map_type; dma_addr_t iova; size_t iova_len = 0; unsigned long mask = dma_get_seg_boundary(dev); @@ -1193,6 +1206,35 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, s_length = iova_align(iovad, s_length + s_iova_off); s->length = s_length; + if (is_pci_p2pdma_page(sg_page(s))) { + if (sg_page(s)->pgmap != pgmap) { + pgmap = sg_page(s)->pgmap; + map_type = pci_p2pdma_map_type(pgmap, dev); + } + + switch (map_type) { + case PCI_P2PDMA_MAP_BUS_ADDR: + /* + * A zero length will be ignored by + * iommu_map_sg() and then can be detected + * in __finalise_sg() to actually map the + * bus address. + */ + s->length = 0; + continue; + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Mapping through host bridge should be + * mapped with regular IOVAs, thus we + * do nothing here and continue below. + */ + break; + default: + ret = -EREMOTEIO; + goto out_restore_sg; + } + } + /* * Due to the alignment of our single IOVA allocation, we can * depend on these assumptions about the segment boundary mask: @@ -1215,6 +1257,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, prev = s; } + if (!iova_len) + return __finalise_sg(dev, sg, nents, 0); + iova = iommu_dma_alloc_iova(domain, iova_len, dma_get_mask(dev), dev); if (!iova) { ret = -ENOMEM; @@ -1236,7 +1281,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, out_restore_sg: __invalidate_sg(sg, nents); out: - if (ret != -ENOMEM) + if (ret != -ENOMEM && ret != -EREMOTEIO) return -EINVAL; return ret; } @@ -1244,7 +1289,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t start, end; + dma_addr_t end, start = DMA_MAPPING_ERROR; struct scatterlist *tmp; int i; @@ -1260,14 +1305,22 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, * The scatterlist segments are mapped into a single * contiguous IOVA allocation, so this is incredibly easy. */ - start = sg_dma_address(sg); - for_each_sg(sg_next(sg), tmp, nents - 1, i) { + for_each_sg(sg, tmp, nents, i) { + if (sg_is_dma_bus_address(tmp)) { + sg_dma_unmark_bus_address(tmp); + continue; + } if (sg_dma_len(tmp) == 0) break; - sg = tmp; + + if (start == DMA_MAPPING_ERROR) + start = sg_dma_address(tmp); + + end = sg_dma_address(tmp) + sg_dma_len(tmp); } - end = sg_dma_address(sg) + sg_dma_len(sg); - __iommu_dma_unmap(dev, start, end - start); + + if (start != DMA_MAPPING_ERROR) + __iommu_dma_unmap(dev, start, end - start); } static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, @@ -1460,6 +1513,7 @@ static unsigned long iommu_dma_get_merge_boundary(struct device *dev) } static const struct dma_map_ops iommu_dma_ops = { + .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, .alloc_pages = dma_common_alloc_pages, From patchwork Wed Jun 15 16:12:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882630 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8251C433EF for ; Wed, 15 Jun 2022 16:13:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 977A96B0080; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6982E6B0092; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B2B4C6B0089; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 870636B0092 for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 5B7F612018A for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.13.BE88E68 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf25.hostedemail.com (Postfix) with ESMTP id DC1B4A008F for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=DC/ETh8yOjqQaZexWWMF/9NUyC2tj6LUwuHIuAOiKYU=; b=EYS9ywhOZhK+SMZI06FJMJW86j 2/1pd41bY/z3VuuuJJcVsRNpKhOpgeqskrJgqKUUAYPZPVxO2Sv4QdEHVew4hSN5VQZ+Jbpo55xQU Z8+m5+5blzdgjyo1oiyrucw9tA1t+mKiljpdZHDK11NupAC+TFwmmCfDJ3hhrU66engeAKqz12vyc qFIhlzicini34cL+/BB5atURwqyrhf7Se1ZbXf25go4/PoQg0E8McB1vjoHzZ8BBK3DA4IJbQ9bSS IdThINpBJXiKkn3dDee2HC4NL8Mt7M/Ct1N3Vc8HsTw5c+BAw9a6KmzBcY6TkN0P6Nj3tVb1bOTCO gPlTVeUA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdR-0084iT-Qd; Wed, 15 Jun 2022 10:12:50 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdH-0004aH-Qt; Wed, 15 Jun 2022 10:12:39 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Chaitanya Kulkarni Date: Wed, 15 Jun 2022 10:12:21 -0600 Message-Id: <20220615161233.17527-10-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, kch@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 09/21] nvme-pci: check DMA ops when indicating support for PCI P2PDMA X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=sHsbt5JpK+J7RslmbXYfx/AascBAptSKabrp+DjnekTb5AswRSPVWhB/3426sE1Wvbue0D fhUZwCLp32l3HVzgG7UFDG24gChmO6jVEbpccryzZTwkFRuZjX/3NHDXgYu+qKBl1RCw4C V9hZkVarnP1q3ZVkP1DmY/4AwKhNBrM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DC/ETh8yOjqQaZexWWMF/9NUyC2tj6LUwuHIuAOiKYU=; b=fgu8LnulU6UtJAyEoIVLAZHOrnUqdlGYniUq+6WAgVf7GK6dWMXXG1PAJeRdcAM+52fu6l NQ8ILVVpTNUhFG3tr1DCTVBFuBs9gJO/rXyAH+3j2C4VgCzYhcoSOtmXL4YX3R81NBN/sm MVi5RZwMAXww/kKVHSKWNV+2GeVbLqU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=EYS9ywhO; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf25.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=EYS9ywhO; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf25.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: DC1B4A008F X-Stat-Signature: ptj51eckkqrd3b4tshgxggcqgr8xqky4 X-HE-Tag: 1655309575-875579 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a supports_pci_p2pdma() operation in nvme_ctrl_ops to replace the fixed NVME_F_PCI_P2PDMA flag such that the dma_map_ops flags can be checked for PCI P2PDMA support. Signed-off-by: Logan Gunthorpe Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig --- drivers/nvme/host/core.c | 3 ++- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 11 +++++++++-- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 24165daee3c8..d6e76f2dc293 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3981,7 +3981,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid, blk_queue_flag_set(QUEUE_FLAG_STABLE_WRITES, ns->queue); blk_queue_flag_set(QUEUE_FLAG_NONROT, ns->queue); - if (ctrl->ops->flags & NVME_F_PCI_P2PDMA) + if (ctrl->ops->supports_pci_p2pdma && + ctrl->ops->supports_pci_p2pdma(ctrl)) blk_queue_flag_set(QUEUE_FLAG_PCI_P2PDMA, ns->queue); ns->ctrl = ctrl; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 9b72b6ecf33c..957f79420cf3 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -495,7 +495,6 @@ struct nvme_ctrl_ops { unsigned int flags; #define NVME_F_FABRICS (1 << 0) #define NVME_F_METADATA_SUPPORTED (1 << 1) -#define NVME_F_PCI_P2PDMA (1 << 2) int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val); int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val); int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val); @@ -503,6 +502,7 @@ struct nvme_ctrl_ops { void (*submit_async_event)(struct nvme_ctrl *ctrl); void (*delete_ctrl)(struct nvme_ctrl *ctrl); int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size); + bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); }; /* diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 48f4f6eb877b..e5e032ab1c71 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2976,17 +2976,24 @@ static int nvme_pci_get_address(struct nvme_ctrl *ctrl, char *buf, int size) return snprintf(buf, size, "%s\n", dev_name(&pdev->dev)); } +static bool nvme_pci_supports_pci_p2pdma(struct nvme_ctrl *ctrl) +{ + struct nvme_dev *dev = to_nvme_dev(ctrl); + + return dma_pci_p2pdma_supported(dev->dev); +} + static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name = "pcie", .module = THIS_MODULE, - .flags = NVME_F_METADATA_SUPPORTED | - NVME_F_PCI_P2PDMA, + .flags = NVME_F_METADATA_SUPPORTED, .reg_read32 = nvme_pci_reg_read32, .reg_write32 = nvme_pci_reg_write32, .reg_read64 = nvme_pci_reg_read64, .free_ctrl = nvme_pci_free_ctrl, .submit_async_event = nvme_pci_submit_async_event, .get_address = nvme_pci_get_address, + .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma, }; static int nvme_dev_map(struct nvme_dev *dev) From patchwork Wed Jun 15 16:12:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E4B5C43334 for ; Wed, 15 Jun 2022 16:13:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 010816B0081; Wed, 15 Jun 2022 12:12:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5C556B008C; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07AB76B0081; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 881AC6B0093 for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 638E521190 for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.28.B8D4C80 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf31.hostedemail.com (Postfix) with ESMTP id 0E8A52009E for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=nxQs1u/iNMUOdED7LTA+Mhs47OupmoJLe3Pw4MQ7NVg=; b=ofFJ++YePU8wUbubeU2SOgtc5l Cn1NM36buqbw552sASlAb8j7NF8B46iDVp9ilVwMhP2D/f27bP0bBWu56PyzeA99gDajmLxDVqAXh n5Ss0dDddTX0L8hi3nmZChspP8/Yg7jnorUZ5TCvgAunW1/klbw3EcPtDhi+qPh0/BIHMdPP3FnAo hmYnSfrvUYwS9/1c+RytuBTOXMDCPs1RbQWt5Epz0Tz9QGOiVFFPEYq2aSZgPZH6NHxGXmtN1fmIK 1I3iuSnPaza02bdUhhe8Z3weC1X4mbVBwcjMR0b8Zj8QqD0I9Mh24DNkfwatPjdAl5X12+im5QnvC cPwtigpA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdR-0084iR-Jo; Wed, 15 Jun 2022 10:12:50 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004aM-1w; Wed, 15 Jun 2022 10:12:40 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Max Gurtovoy , Chaitanya Kulkarni Date: Wed, 15 Jun 2022 10:12:22 -0600 Message-Id: <20220615161233.17527-11-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, mgurtovoy@nvidia.com, kch@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 10/21] nvme-pci: convert to using dma_map_sgtable() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309576; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nxQs1u/iNMUOdED7LTA+Mhs47OupmoJLe3Pw4MQ7NVg=; b=jWDWtc6tKLFhPpbzeItS9fpW3ai/PFruyyrnbUogtmQkugRqAxqAk3siALO+52p+GQwLgL aJf1NcD1JOpqR1hMAyPcx2sDYTqSCRSoKqWpb/HjKjoI9s5s7vVvKxK7UteABy2CIffgRY fh23CEjh8Hs1OGy5aF7WyjnwU5ZEoCI= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=ofFJ++Ye; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309576; a=rsa-sha256; cv=none; b=3hfltjqTnMu6sB211vuqQ9PbRgYtHj0nAr3KYLkeMXZzFX/C0wreit69eR9vphRSts6yoJ QjUJPtjyS6ukoXtlqotPsjyHgsHOnunvunWLYzUV+IJYJES+QMIXbmvtj6Cn+/vaj8epZ1 E4oMSk0aZhi1fqRg7FEUA8hZiYQWJ80= X-Rspamd-Queue-Id: 0E8A52009E X-Rspam-User: X-Stat-Signature: 77wuqux6gfxu1cghpt13qenn5xwsgsh1 Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=ofFJ++Ye; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam04 X-HE-Tag: 1655309575-7467 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The dma_map operations now support P2PDMA pages directly. So remove the calls to pci_p2pdma_[un]map_sg_attrs() and replace them with calls to dma_map_sgtable(). dma_map_sgtable() returns more complete error codes than dma_map_sg() and allows differentiating EREMOTEIO errors in case an unsupported P2PDMA transfer is requested. When this happens, return BLK_STS_TARGET so the request isn't retried. Signed-off-by: Logan Gunthorpe Reviewed-by: Max Gurtovoy Reviewed-by: Chaitanya Kulkarni Reviewed-by: Christoph Hellwig --- drivers/nvme/host/pci.c | 69 +++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 40 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index e5e032ab1c71..52b52a7efa9a 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -230,11 +230,10 @@ struct nvme_iod { bool use_sgl; int aborted; int npages; /* In the PRP list. 0 means small pool in use */ - int nents; /* Used in scatterlist */ dma_addr_t first_dma; unsigned int dma_len; /* length of single DMA segment mapping */ dma_addr_t meta_dma; - struct scatterlist *sg; + struct sg_table sgt; }; static inline unsigned int nvme_dbbuf_size(struct nvme_dev *dev) @@ -524,7 +523,7 @@ static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx) static void **nvme_pci_iod_list(struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - return (void **)(iod->sg + blk_rq_nr_phys_segments(req)); + return (void **)(iod->sgt.sgl + blk_rq_nr_phys_segments(req)); } static inline bool nvme_pci_use_sgls(struct nvme_dev *dev, struct request *req) @@ -576,17 +575,6 @@ static void nvme_free_sgls(struct nvme_dev *dev, struct request *req) } } -static void nvme_unmap_sg(struct nvme_dev *dev, struct request *req) -{ - struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - - if (is_pci_p2pdma_page(sg_page(iod->sg))) - pci_p2pdma_unmap_sg(dev->dev, iod->sg, iod->nents, - rq_dma_dir(req)); - else - dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req)); -} - static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); @@ -597,9 +585,10 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) return; } - WARN_ON_ONCE(!iod->nents); + WARN_ON_ONCE(!iod->sgt.nents); + + dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); - nvme_unmap_sg(dev, req); if (iod->npages == 0) dma_pool_free(dev->prp_small_pool, nvme_pci_iod_list(req)[0], iod->first_dma); @@ -607,7 +596,7 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) nvme_free_sgls(dev, req); else nvme_free_prps(dev, req); - mempool_free(iod->sg, dev->iod_mempool); + mempool_free(iod->sgt.sgl, dev->iod_mempool); } static void nvme_print_sgl(struct scatterlist *sgl, int nents) @@ -630,7 +619,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, struct nvme_iod *iod = blk_mq_rq_to_pdu(req); struct dma_pool *pool; int length = blk_rq_payload_bytes(req); - struct scatterlist *sg = iod->sg; + struct scatterlist *sg = iod->sgt.sgl; int dma_len = sg_dma_len(sg); u64 dma_addr = sg_dma_address(sg); int offset = dma_addr & (NVME_CTRL_PAGE_SIZE - 1); @@ -703,16 +692,16 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, dma_len = sg_dma_len(sg); } done: - cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sg)); + cmnd->dptr.prp1 = cpu_to_le64(sg_dma_address(iod->sgt.sgl)); cmnd->dptr.prp2 = cpu_to_le64(iod->first_dma); return BLK_STS_OK; free_prps: nvme_free_prps(dev, req); return BLK_STS_RESOURCE; bad_sgl: - WARN(DO_ONCE(nvme_print_sgl, iod->sg, iod->nents), + WARN(DO_ONCE(nvme_print_sgl, iod->sgt.sgl, iod->sgt.nents), "Invalid SGL for payload:%d nents:%d\n", - blk_rq_payload_bytes(req), iod->nents); + blk_rq_payload_bytes(req), iod->sgt.nents); return BLK_STS_IOERR; } @@ -738,12 +727,13 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge, } static blk_status_t nvme_pci_setup_sgls(struct nvme_dev *dev, - struct request *req, struct nvme_rw_command *cmd, int entries) + struct request *req, struct nvme_rw_command *cmd) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); struct dma_pool *pool; struct nvme_sgl_desc *sg_list; - struct scatterlist *sg = iod->sg; + struct scatterlist *sg = iod->sgt.sgl; + unsigned int entries = iod->sgt.nents; dma_addr_t sgl_dma; int i = 0; @@ -841,7 +831,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); blk_status_t ret = BLK_STS_RESOURCE; - int nr_mapped; + int rc; if (blk_rq_nr_phys_segments(req) == 1) { struct bio_vec bv = req_bvec(req); @@ -859,26 +849,25 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, } iod->dma_len = 0; - iod->sg = mempool_alloc(dev->iod_mempool, GFP_ATOMIC); - if (!iod->sg) + iod->sgt.sgl = mempool_alloc(dev->iod_mempool, GFP_ATOMIC); + if (!iod->sgt.sgl) return BLK_STS_RESOURCE; - sg_init_table(iod->sg, blk_rq_nr_phys_segments(req)); - iod->nents = blk_rq_map_sg(req->q, req, iod->sg); - if (!iod->nents) + sg_init_table(iod->sgt.sgl, blk_rq_nr_phys_segments(req)); + iod->sgt.orig_nents = blk_rq_map_sg(req->q, req, iod->sgt.sgl); + if (!iod->sgt.orig_nents) goto out_free_sg; - if (is_pci_p2pdma_page(sg_page(iod->sg))) - nr_mapped = pci_p2pdma_map_sg_attrs(dev->dev, iod->sg, - iod->nents, rq_dma_dir(req), DMA_ATTR_NO_WARN); - else - nr_mapped = dma_map_sg_attrs(dev->dev, iod->sg, iod->nents, - rq_dma_dir(req), DMA_ATTR_NO_WARN); - if (!nr_mapped) + rc = dma_map_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), + DMA_ATTR_NO_WARN); + if (rc) { + if (rc == -EREMOTEIO) + ret = BLK_STS_TARGET; goto out_free_sg; + } iod->use_sgl = nvme_pci_use_sgls(dev, req); if (iod->use_sgl) - ret = nvme_pci_setup_sgls(dev, req, &cmnd->rw, nr_mapped); + ret = nvme_pci_setup_sgls(dev, req, &cmnd->rw); else ret = nvme_pci_setup_prps(dev, req, &cmnd->rw); if (ret != BLK_STS_OK) @@ -886,9 +875,9 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, return BLK_STS_OK; out_unmap_sg: - nvme_unmap_sg(dev, req); + dma_unmap_sgtable(dev->dev, &iod->sgt, rq_dma_dir(req), 0); out_free_sg: - mempool_free(iod->sg, dev->iod_mempool); + mempool_free(iod->sgt.sgl, dev->iod_mempool); return ret; } @@ -912,7 +901,7 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) iod->aborted = 0; iod->npages = -1; - iod->nents = 0; + iod->sgt.nents = 0; ret = nvme_setup_cmd(req->q->queuedata, req); if (ret) From patchwork Wed Jun 15 16:12:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28115CCA480 for ; Wed, 15 Jun 2022 16:13:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69A226B0087; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 66DF66B007B; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF8266B007E; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D68D86B007D for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AA3D360535 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.09.36D7262 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf19.hostedemail.com (Postfix) with ESMTP id 3F6531A0082 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=HPYoImFXHjbpSm9PaLmzjRXa3JPsRqW7zrVLQeixgD4=; b=r7bhAAOfkNY4F2njsaTodS3vVL ykLOAV5I8R0D9JqUk0j6HNto1uBerDF7Pvc4asv/N5vsKy0IHSg1kmRxo9s693lLiOvxZoXGxpPz2 bexVzgLsB793JhxTlK3QLOv4skBYFRBdSZlf/CVkA4itwt/g2UNOK80hXSAqPNIQzJDARjJpztmxV 8sGvIeKnV+vjTfhlaSFZvew69qs86S9JnsFbBp4hTj2cI6Sqp/pOLsMe+X2huw124J/G5XIIEn9EQ Q2f1RP9/WHgeN47lUwhHH1jLaqXOJR50y4me6KpUxSA+93qus7kTWlZAMfZuqhceqsO8elIp14JDN i/GIya3Q==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdR-0084iQ-HF; Wed, 15 Jun 2022 10:12:50 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004aQ-7K; Wed, 15 Jun 2022 10:12:40 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Jason Gunthorpe , Max Gurtovoy Date: Wed, 15 Jun 2022 10:12:23 -0600 Message-Id: <20220615161233.17527-12-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com, mgurtovoy@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 11/21] RDMA/core: introduce ib_dma_pci_p2p_dma_supported() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=Ga5r/bfGK9ADj68QhI+afjB2zmf9r/hwT+gL9+/PFx1wSzdKqcohnKuKR90cvtwgKCGEdD tZDyeEAJmJIU9HNE/03O1ACCVOEsurdvR/Iy0j7a+9KWWQgi9VnPwSzKT5x9qG7Ku0r756 9ZQh3eCxynpDHMlGQVKe09x6Ykw3pRs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HPYoImFXHjbpSm9PaLmzjRXa3JPsRqW7zrVLQeixgD4=; b=SzguWKBY67Mh3DBavYCY2Cz8x/xIesifZKdko8sjMO+5MpMv6/TxOInohOYuTtqIHtU6gv hG2n+OhlvqYhvuItOKzsSCOYDE11KmeKqu1lcozHO2qt4WCBKUZ/YzpIOSzvQOrGnkVwkx I7j9x3LXYv7SFOgupzu/ux1ldQKfWUI= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=r7bhAAOf; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf19.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=r7bhAAOf; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf19.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 3F6531A0082 X-Stat-Signature: s4u6p4ai5xq38z4soxnf1gmgrw8s9ziq X-HE-Tag: 1655309575-373368 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce the helper function ib_dma_pci_p2p_dma_supported() to check if a given ib_device can be used in P2PDMA transfers. This ensures the ib_device is not using virt_dma and also that the underlying dma_device supports P2PDMA. Use the new helper in nvme-rdma to replace the existing check for ib_uses_virt_dma(). Adding the dma_pci_p2pdma_supported() check allows switching away from pci_p2pdma_[un]map_sg(). Signed-off-by: Logan Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Max Gurtovoy Reviewed-by: Christoph Hellwig --- drivers/nvme/target/rdma.c | 2 +- include/rdma/ib_verbs.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index 09fdcac87d17..4597bca43a6d 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -415,7 +415,7 @@ static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev, if (ib_dma_mapping_error(ndev->device, r->send_sge.addr)) goto out_free_rsp; - if (!ib_uses_virt_dma(ndev->device)) + if (ib_dma_pci_p2p_dma_supported(ndev->device)) r->req.p2p_client = &ndev->device->dev; r->send_sge.length = sizeof(*r->req.cqe); r->send_sge.lkey = ndev->pd->local_dma_lkey; diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 9c6317cf80d5..523843d9ed6c 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -4013,6 +4013,17 @@ static inline bool ib_uses_virt_dma(struct ib_device *dev) return IS_ENABLED(CONFIG_INFINIBAND_VIRT_DMA) && !dev->dma_device; } +/* + * Check if a IB device's underlying DMA mapping supports P2PDMA transfers. + */ +static inline bool ib_dma_pci_p2p_dma_supported(struct ib_device *dev) +{ + if (ib_uses_virt_dma(dev)) + return false; + + return dma_pci_p2pdma_supported(dev->dma_device); +} + /** * ib_dma_mapping_error - check a DMA addr for error * @dev: The device for which the dma_addr was created From patchwork Wed Jun 15 16:12:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95D66C43334 for ; Wed, 15 Jun 2022 16:54:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 238506B0071; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C5236B0072; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0623C6B0074; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EAFA16B0071 for ; Wed, 15 Jun 2022 12:54:55 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AE030338 for ; Wed, 15 Jun 2022 16:54:55 +0000 (UTC) X-FDA: 79581069750.13.1F5FE50 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf31.hostedemail.com (Postfix) with ESMTP id 0D736200A9 for ; Wed, 15 Jun 2022 16:54:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=E3OuyFTlPOMWods4T8GcP7ZVTzRsCfvad0BDfewE94Y=; b=AJS/ErU6QFAY8jbAvFU18GiUtb FTte+z8bIp8Ybx0f1HkqqJWQLLEhJHQSkf9wFfqzXyXbKf5dwaPMl1LBviKkNt2olWFRI7oprrSGf gGYh46EAVGZ2nt+EIgAC+5okqC3czFB/dqIKdWW/Tt/gyfo6kwSyA6Fe29xl9WhYaNarU+3CWR2iA 0gDPP3xxY0uxB3FNnwpiVqBwKsruzoZJktRnXEAKtQfViLsl3OTrpfMz+ABNjywEA1Sz0mjLS6qAr QNiLeSTDIctcyMVy3fSRQG9Afb9tM/qbqK0fXNcvsOcWnLQcZ58rAKXF6O+k1LxW7iIIQU+NL+yBk Un9lmpdg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdR-0084iS-3j; Wed, 15 Jun 2022 10:12:50 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004aU-Bw; Wed, 15 Jun 2022 10:12:40 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Jason Gunthorpe Date: Wed, 15 Jun 2022 10:12:24 -0600 Message-Id: <20220615161233.17527-13-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 12/21] RDMA/rw: drop pci_p2pdma_[un]map_sg() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655312095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E3OuyFTlPOMWods4T8GcP7ZVTzRsCfvad0BDfewE94Y=; b=z4dLxwHFgO55fBSsO+N4R0RmHWGc4wz1cbcg+adFhbkvQIifKscI7rLEPAlMSXxDArzNdw JSeEeQyf7+j3G4fdNK+hMfQgTNHmw7dKGR07SIZsr33GS88vixwOUYUP0xKxTRt7OgcWJo ifalo7LwKhGFpQfv22zEby08Nmzu2Eo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655312095; a=rsa-sha256; cv=none; b=0bjNoBy7WkwfjyGEyPljZo57gns8DfXkgQgWUjqTNmxdTlRBZ3/N0xBsQyZxYfnn1wtWWH loAycPasm2D9idXimQkM60HTLmU/iPHInFHcXFwer5485D+8xj58pZ/Ooov+MD9TPfWukc /jLGpSfR2pjd/NlH2Zde7SKA+AknIks= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b="AJS/ErU6"; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b="AJS/ErU6"; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: ggjp3xhnn5qtx8t3oyku7k16bzbjdkb7 X-Rspamd-Queue-Id: 0D736200A9 X-HE-Tag: 1655312094-988570 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dma_map_sg() now supports the use of P2PDMA pages so pci_p2pdma_map_sg() is no longer necessary and may be dropped. This means the rdma_rw_[un]map_sg() helpers are no longer necessary. Remove it all. Signed-off-by: Logan Gunthorpe Reviewed-by: Jason Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/infiniband/core/rw.c | 45 ++++++++---------------------------- 1 file changed, 9 insertions(+), 36 deletions(-) diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c index 4d98f931a13d..8367974b7998 100644 --- a/drivers/infiniband/core/rw.c +++ b/drivers/infiniband/core/rw.c @@ -274,33 +274,6 @@ static int rdma_rw_init_single_wr(struct rdma_rw_ctx *ctx, struct ib_qp *qp, return 1; } -static void rdma_rw_unmap_sg(struct ib_device *dev, struct scatterlist *sg, - u32 sg_cnt, enum dma_data_direction dir) -{ - if (is_pci_p2pdma_page(sg_page(sg))) - pci_p2pdma_unmap_sg(dev->dma_device, sg, sg_cnt, dir); - else - ib_dma_unmap_sg(dev, sg, sg_cnt, dir); -} - -static int rdma_rw_map_sgtable(struct ib_device *dev, struct sg_table *sgt, - enum dma_data_direction dir) -{ - int nents; - - if (is_pci_p2pdma_page(sg_page(sgt->sgl))) { - if (WARN_ON_ONCE(ib_uses_virt_dma(dev))) - return 0; - nents = pci_p2pdma_map_sg(dev->dma_device, sgt->sgl, - sgt->orig_nents, dir); - if (!nents) - return -EIO; - sgt->nents = nents; - return 0; - } - return ib_dma_map_sgtable_attrs(dev, sgt, dir, 0); -} - /** * rdma_rw_ctx_init - initialize a RDMA READ/WRITE context * @ctx: context to initialize @@ -327,7 +300,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u32 port_num, }; int ret; - ret = rdma_rw_map_sgtable(dev, &sgt, dir); + ret = ib_dma_map_sgtable_attrs(dev, &sgt, dir, 0); if (ret) return ret; sg_cnt = sgt.nents; @@ -366,7 +339,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u32 port_num, return ret; out_unmap_sg: - rdma_rw_unmap_sg(dev, sgt.sgl, sgt.orig_nents, dir); + ib_dma_unmap_sgtable_attrs(dev, &sgt, dir, 0); return ret; } EXPORT_SYMBOL(rdma_rw_ctx_init); @@ -414,12 +387,12 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, return -EINVAL; } - ret = rdma_rw_map_sgtable(dev, &sgt, dir); + ret = ib_dma_map_sgtable_attrs(dev, &sgt, dir, 0); if (ret) return ret; if (prot_sg_cnt) { - ret = rdma_rw_map_sgtable(dev, &prot_sgt, dir); + ret = ib_dma_map_sgtable_attrs(dev, &prot_sgt, dir, 0); if (ret) goto out_unmap_sg; } @@ -486,9 +459,9 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, kfree(ctx->reg); out_unmap_prot_sg: if (prot_sgt.nents) - rdma_rw_unmap_sg(dev, prot_sgt.sgl, prot_sgt.orig_nents, dir); + ib_dma_unmap_sgtable_attrs(dev, &prot_sgt, dir, 0); out_unmap_sg: - rdma_rw_unmap_sg(dev, sgt.sgl, sgt.orig_nents, dir); + ib_dma_unmap_sgtable_attrs(dev, &sgt, dir, 0); return ret; } EXPORT_SYMBOL(rdma_rw_ctx_signature_init); @@ -621,7 +594,7 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, break; } - rdma_rw_unmap_sg(qp->pd->device, sg, sg_cnt, dir); + ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir); } EXPORT_SYMBOL(rdma_rw_ctx_destroy); @@ -649,8 +622,8 @@ void rdma_rw_ctx_destroy_signature(struct rdma_rw_ctx *ctx, struct ib_qp *qp, kfree(ctx->reg); if (prot_sg_cnt) - rdma_rw_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir); - rdma_rw_unmap_sg(qp->pd->device, sg, sg_cnt, dir); + ib_dma_unmap_sg(qp->pd->device, prot_sg, prot_sg_cnt, dir); + ib_dma_unmap_sg(qp->pd->device, sg, sg_cnt, dir); } EXPORT_SYMBOL(rdma_rw_ctx_destroy_signature); From patchwork Wed Jun 15 16:12:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD054CCA47F for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29C606B0071; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24B8C6B0072; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 113A76B0074; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 03B246B0071 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id D45D0601E6 for ; Wed, 15 Jun 2022 16:12:54 +0000 (UTC) X-FDA: 79580963868.04.F220AD1 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf02.hostedemail.com (Postfix) with ESMTP id 709C48000A for ; Wed, 15 Jun 2022 16:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=ZrjQH0Cup1rM4nd2GRv9TzKM0VooeR2ywa/jH43VO8E=; b=K8b3a3dLtKbLAQ6EXSvANzTKW7 3dU+pmoDHBQAn7sU5St+kCSqesBVWJ4lYut+OeGrHoX68Ll1zV5tc6HClutZ7TB3pNyFXAwO4hLg+ hQDPpja0a0r/3AvjLtBw98XXsTreXz5lLyXuffaL9rfY/dEiYcoNOnDfcBNwltLQbbOKxJdNxc0Rz 1uENncVKvga+5BagrJPsBwuwx9DzG9LxScBVTTgwXNzqKH90kRniGUiJ0rt9+CJ5ixFsT07fLMxpP M/Y0MUscOGcW2knHL7l3cDhTqon3LpK4J7xo7UtDulpaN2V3IiRSkGtg6pJR2wUzormLlbKk2xqxv jFTOI/Ug==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdQ-0084iP-Qs; Wed, 15 Jun 2022 10:12:49 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004aY-Gc; Wed, 15 Jun 2022 10:12:40 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Bjorn Helgaas , Jason Gunthorpe , Max Gurtovoy Date: Wed, 15 Jun 2022 10:12:25 -0600 Message-Id: <20220615161233.17527-14-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, logang@deltatee.com, bhelgaas@google.com, jhubbard@nvidia.com, rcampbell@nvidia.com, jgg@nvidia.com, mgurtovoy@nvidia.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 13/21] PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=u5JRaaxNLzg3f0z9VgCzRWyaa/CPrzShBM3Dq7df+8A8xK2ApG0wymFbc84GL/NDTGojxt tUBhSthZ4unIsyBiHgw3QtpXGJjpMhsY8U9t9gvDXjfsxYnRMsUNr6Focinu9YG58XZVzU duMEiTQNK+QNF2r9FNY6jH7IG6qaX4o= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=K8b3a3dL; spf=pass (imf02.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZrjQH0Cup1rM4nd2GRv9TzKM0VooeR2ywa/jH43VO8E=; b=pvJLKruXqV0GCwh82SIJ5QSmIUUzOOqNI3S7pGHhLNlnLipzaBIB2ZzN6Lt9a9t6STZv6g 6HP5mD0k6yQsMOCvIZ0ao+zvgDsm0ikMhXGTuHZ0Jv+PqFocRAve+pMMcMML3rTacOXQFD RnQDXDOyuUp7JQ8pfh3VN9+JBhnF4k4= X-Stat-Signature: j3joyyqfez4decqso6tmifrbrytnzoj1 X-Rspamd-Queue-Id: 709C48000A Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=K8b3a3dL; spf=pass (imf02.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1655309574-798249 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This interface is superseded by support in dma_map_sg() which now supports heterogeneous scatterlists. There are no longer any users, so remove it. Signed-off-by: Logan Gunthorpe Acked-by: Bjorn Helgaas Reviewed-by: Jason Gunthorpe Reviewed-by: Max Gurtovoy Reviewed-by: Christoph Hellwig --- drivers/pci/p2pdma.c | 66 -------------------------------------- include/linux/pci-p2pdma.h | 27 ---------------- 2 files changed, 93 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 2fc0f4750a2e..d4e635012ffe 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -885,72 +885,6 @@ enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, return type; } -static int __pci_p2pdma_map_sg(struct pci_p2pdma_pagemap *p2p_pgmap, - struct device *dev, struct scatterlist *sg, int nents) -{ - struct scatterlist *s; - int i; - - for_each_sg(sg, s, nents, i) { - s->dma_address = sg_phys(s) + p2p_pgmap->bus_offset; - sg_dma_len(s) = s->length; - } - - return nents; -} - -/** - * pci_p2pdma_map_sg_attrs - map a PCI peer-to-peer scatterlist for DMA - * @dev: device doing the DMA request - * @sg: scatter list to map - * @nents: elements in the scatterlist - * @dir: DMA direction - * @attrs: DMA attributes passed to dma_map_sg() (if called) - * - * Scatterlists mapped with this function should be unmapped using - * pci_p2pdma_unmap_sg_attrs(). - * - * Returns the number of SG entries mapped or 0 on error. - */ -int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - struct pci_p2pdma_pagemap *p2p_pgmap = - to_p2p_pgmap(sg_page(sg)->pgmap); - - switch (pci_p2pdma_map_type(sg_page(sg)->pgmap, dev)) { - case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: - return dma_map_sg_attrs(dev, sg, nents, dir, attrs); - case PCI_P2PDMA_MAP_BUS_ADDR: - return __pci_p2pdma_map_sg(p2p_pgmap, dev, sg, nents); - default: - /* Mapping is not Supported */ - return 0; - } -} -EXPORT_SYMBOL_GPL(pci_p2pdma_map_sg_attrs); - -/** - * pci_p2pdma_unmap_sg_attrs - unmap a PCI peer-to-peer scatterlist that was - * mapped with pci_p2pdma_map_sg() - * @dev: device doing the DMA request - * @sg: scatter list to map - * @nents: number of elements returned by pci_p2pdma_map_sg() - * @dir: DMA direction - * @attrs: DMA attributes passed to dma_unmap_sg() (if called) - */ -void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs) -{ - enum pci_p2pdma_map_type map_type; - - map_type = pci_p2pdma_map_type(sg_page(sg)->pgmap, dev); - - if (map_type == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE) - dma_unmap_sg_attrs(dev, sg, nents, dir, attrs); -} -EXPORT_SYMBOL_GPL(pci_p2pdma_unmap_sg_attrs); - /** * pci_p2pdma_map_segment - map an sg segment determining the mapping type * @state: State structure that should be declared outside of the for_each_sg() diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h index 8318a97c9c61..2c07aa6b7665 100644 --- a/include/linux/pci-p2pdma.h +++ b/include/linux/pci-p2pdma.h @@ -30,10 +30,6 @@ struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, unsigned int *nents, u32 length); void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl); void pci_p2pmem_publish(struct pci_dev *pdev, bool publish); -int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs); -void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, unsigned long attrs); int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev, bool *use_p2pdma); ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev, @@ -83,17 +79,6 @@ static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev, static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) { } -static inline int pci_p2pdma_map_sg_attrs(struct device *dev, - struct scatterlist *sg, int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - return 0; -} -static inline void pci_p2pdma_unmap_sg_attrs(struct device *dev, - struct scatterlist *sg, int nents, enum dma_data_direction dir, - unsigned long attrs) -{ -} static inline int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev, bool *use_p2pdma) { @@ -119,16 +104,4 @@ static inline struct pci_dev *pci_p2pmem_find(struct device *client) return pci_p2pmem_find_many(&client, 1); } -static inline int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir) -{ - return pci_p2pdma_map_sg_attrs(dev, sg, nents, dir, 0); -} - -static inline void pci_p2pdma_unmap_sg(struct device *dev, - struct scatterlist *sg, int nents, enum dma_data_direction dir) -{ - pci_p2pdma_unmap_sg_attrs(dev, sg, nents, dir, 0); -} - #endif /* _LINUX_PCI_P2P_H */ From patchwork Wed Jun 15 16:12:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98344C433EF for ; Wed, 15 Jun 2022 16:13:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 262656B008A; Wed, 15 Jun 2022 12:12:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1FD656B008C; Wed, 15 Jun 2022 12:12:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 428CC6B0095; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 081086B0092 for ; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D9C9435225 for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) X-FDA: 79580963952.18.E074356 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf11.hostedemail.com (Postfix) with ESMTP id 5055540082 for ; Wed, 15 Jun 2022 16:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=cy9XTpt2FU24C9e+HkFSboKSQbqr83R/V0hLrLGYdpk=; b=pd2HZB3bg2LSWZzj9EZpMOVi0s nG+iNfaXYF8eeuTEDUoZ0U8qSJTVi7vDWTdy/zmTG7/s7iE+8risFdqTCofKryO67JiCPS74QcN/D HXxUH4ZrFy8nPMUav1tnD1IuJGBOOuZbWG4zPMBXQZ2V7cHVbV5AqG0TMDbRmoolQZ69rWlMCuV15 T0Af4nUJnITfvcQU/YUlR3K/EW+r0rHA597WJEYgSPeOHyZlIYcSs4Bbe8tDPRZnOXOQJ6SUYYLig neTok2Gg7BE33rdf0OLJdVOtUp3HtDwt/bScL/0+Sc/BKuSD5YbYRxFJoenJ2Ctdtlk2YGexdDvOk vCLIivzA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdQ-0084iT-KT; Wed, 15 Jun 2022 10:12:49 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004ac-LN; Wed, 15 Jun 2022 10:12:40 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:26 -0600 Message-Id: <20220615161233.17527-15-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 14/21] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=pd2HZB3b; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf11.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309576; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cy9XTpt2FU24C9e+HkFSboKSQbqr83R/V0hLrLGYdpk=; b=REFTvGqmIa+XDl6hM+POdSlem4XVyKr8PE8HTow4FtZsLm6fWKW78Vog+PFR5yS2FqIh2Z p9ocpjLsXoOmo9pATky64rgUdNONQnGL/8rkVvcH3PYP9EL3fhjG3ATEWv4ylOTrNySfky Mmb5pW8XIW78MGmhuTgL2CAB+SnYjog= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309576; a=rsa-sha256; cv=none; b=DDyn4cC9dCta+p3bwiOvA2OuYYgLG3cs3ScwsslOYXB5PvGe9OYggQwKwFMpZ4I3YWEvst goAZ2toH9+FYlMx3fRuQvLquCqZ9mY0Tdny5Uvd6NH4tPaICX2HrEp5aetFALfLd9VAFhN x3X/NY2Q3ljBd59/R7g/boKvf0oPCVg= X-Stat-Signature: s5ihht6e9953o3azzbgd9pdrck9h57a8 X-Rspamd-Queue-Id: 5055540082 X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=pd2HZB3b; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf11.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam10 X-HE-Tag: 1655309574-353164 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: GUP Callers that expect PCI P2PDMA pages can now set FOLL_PCI_P2PDMA to allow obtaining P2PDMA pages. If GUP is called without the flag and a P2PDMA page is found, it will return an error. FOLL_PCI_P2PDMA cannot be set if FOLL_LONGTERM is set. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- include/linux/mm.h | 1 + mm/gup.c | 22 +++++++++++++++++++++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bc8f326be0ce..0bcb54ea503c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2941,6 +2941,7 @@ struct page *follow_page(struct vm_area_struct *vma, unsigned long address, #define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ #define FOLL_PIN 0x40000 /* pages must be released via unpin_user_page */ #define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup */ +#define FOLL_PCI_P2PDMA 0x100000 /* allow returning PCI P2PDMA pages */ /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each diff --git a/mm/gup.c b/mm/gup.c index 551264407624..f15f01d06a09 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -564,6 +564,12 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, goto out; } + if (unlikely(!(flags & FOLL_PCI_P2PDMA) && + is_pci_p2pdma_page(page))) { + page = ERR_PTR(-EREMOTEIO); + goto out; + } + VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && !PageAnonExclusive(page), page); @@ -994,6 +1000,9 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) if ((gup_flags & FOLL_LONGTERM) && vma_is_fsdax(vma)) return -EOPNOTSUPP; + if ((gup_flags & FOLL_LONGTERM) && (gup_flags & FOLL_PCI_P2PDMA)) + return -EOPNOTSUPP; + if (vma_is_secretmem(vma)) return -EFAULT; @@ -2289,6 +2298,10 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, VM_BUG_ON(!pfn_valid(pte_pfn(pte))); page = pte_page(pte); + if (unlikely(!(flags & FOLL_PCI_P2PDMA) && + is_pci_p2pdma_page(page))) + goto pte_unmap; + folio = try_grab_folio(page, 1, flags); if (!folio) goto pte_unmap; @@ -2368,6 +2381,12 @@ static int __gup_device_huge(unsigned long pfn, unsigned long addr, undo_dev_pagemap(nr, nr_start, flags, pages); break; } + + if (!(flags & FOLL_PCI_P2PDMA) && is_pci_p2pdma_page(page)) { + undo_dev_pagemap(nr, nr_start, flags, pages); + break; + } + SetPageReferenced(page); pages[*nr] = page; if (unlikely(!try_grab_page(page, flags))) { @@ -2856,7 +2875,8 @@ static int internal_get_user_pages_fast(unsigned long start, if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM | FOLL_FORCE | FOLL_PIN | FOLL_GET | - FOLL_FAST_ONLY | FOLL_NOFAULT))) + FOLL_FAST_ONLY | FOLL_NOFAULT | + FOLL_PCI_P2PDMA))) return -EINVAL; if (gup_flags & FOLL_PIN) From patchwork Wed Jun 15 16:12:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D453C43334 for ; Wed, 15 Jun 2022 16:13:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E11B6B007E; Wed, 15 Jun 2022 12:12:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CADB96B0088; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6935D6B0081; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 02BCA6B0087 for ; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D47E860726 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.20.412344E Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf14.hostedemail.com (Postfix) with ESMTP id F3D57100077 for ; Wed, 15 Jun 2022 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=NVweqEcjCecZ0bg5dlWjMJWWbRBpQR845+naXv01S1E=; b=mGlJB+Mc6r6QavYLUJHJ2f6Gcf 4KC3tU5HmwUMDyO6T64JnNBflaA1Oaq/889fD31ncukdVXKstY3aDuLUssracQXrTgPh503f7c3Gm 9oj+XtAemKDfgOgUrEkOhMLRx7YNuoTdepVnGBAGOZtn7In3FvSV64tzpxAi3mfRaDQNZByACspsC XfjfXCXxwzOBwEvK7qOAPCqBKHK80aqj8hMmZYlu9nFhBs4fE1kVcpjRmX6n3I+3TQ0fbrvvOg9Lw zNRi3B96PZzbaOrO/k/k6P23XfdlbDDDJPP/i6fydDuKVlrF5ptVyivsSZrkhvrBfWhPjYBHZ1EJC Ese8FOBw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdQ-0084iR-Hv; Wed, 15 Jun 2022 10:12:49 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004ag-Qo; Wed, 15 Jun 2022 10:12:40 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:27 -0600 Message-Id: <20220615161233.17527-16-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 15/21] iov_iter: introduce iov_iter_get_pages_[alloc_]flags() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NVweqEcjCecZ0bg5dlWjMJWWbRBpQR845+naXv01S1E=; b=4TpZ24ELJNGIuVhkSMJDB/RZSkKbk1E5b9hJofvjndZiz+dOpWvOAeFuK0NxwI7xJTjGWa s+n97hGqNUGIs2hfv0CnjF3Ae9p21wtWYpd4Bf0PlApHIhUl+i+YyZf30AdSeb6TFKrlfX jDW/6Q2O/R28jGx9bjeDbTtR+ByGHq4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309575; a=rsa-sha256; cv=none; b=Rd8yBeoRTgPNCTY/MOqY07DeXASd64V8xXp2hy3CzrcJ1JywCZYFxDOZeyk5L3MJOf8Gkj EnowQs+t1CDtytMie1KeW5YmuezPyt+5otS+Ce8Gs71sCMz/cJCLCStrRWaTpuG0edAzHM gMxJjV76xxBsJDgsuJ+Uw91gq0kPnMA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mGlJB+Mc; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf14.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mGlJB+Mc; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf14.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: i1rz4tp4ieijfbsnpf7yohbez5bonaxe X-Rspamd-Queue-Id: F3D57100077 X-HE-Tag: 1655309573-845046 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add iov_iter_get_pages_flags() and iov_iter_get_pages_alloc_flags() which take a flags argument that is passed to get_user_pages_fast(). This is so that FOLL_PCI_P2PDMA can be passed when appropriate. Signed-off-by: Logan Gunthorpe --- include/linux/uio.h | 6 ++++++ lib/iov_iter.c | 25 +++++++++++++++++++------ 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 739285fe5a2f..ddf9e4cf4a59 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -232,8 +232,14 @@ void iov_iter_pipe(struct iov_iter *i, unsigned int direction, struct pipe_inode void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xarray *xarray, loff_t start, size_t count); +ssize_t iov_iter_get_pages_flags(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned maxpages, size_t *start, + unsigned int gup_flags); ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); +ssize_t iov_iter_get_pages_alloc_flags(struct iov_iter *i, + struct page ***pages, size_t maxsize, size_t *start, + unsigned int gup_flags); ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, size_t maxsize, size_t *start); int iov_iter_npages(const struct iov_iter *i, int maxpages); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 6dd5330f7a99..9bf6e3af5120 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -1515,9 +1515,9 @@ static struct page *first_bvec_segment(const struct iov_iter *i, return page; } -ssize_t iov_iter_get_pages(struct iov_iter *i, +ssize_t iov_iter_get_pages_flags(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, - size_t *start) + size_t *start, unsigned int gup_flags) { size_t len; int n, res; @@ -1528,7 +1528,6 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, return 0; if (likely(iter_is_iovec(i))) { - unsigned int gup_flags = 0; unsigned long addr; if (iov_iter_rw(i) != WRITE) @@ -1558,6 +1557,13 @@ ssize_t iov_iter_get_pages(struct iov_iter *i, return iter_xarray_get_pages(i, pages, maxsize, maxpages, start); return -EFAULT; } +EXPORT_SYMBOL_GPL(iov_iter_get_pages_flags); + +ssize_t iov_iter_get_pages(struct iov_iter *i, struct page **pages, + size_t maxsize, unsigned maxpages, size_t *start) +{ + return iov_iter_get_pages_flags(i, pages, maxsize, maxpages, start, 0); +} EXPORT_SYMBOL(iov_iter_get_pages); static struct page **get_pages_array(size_t n) @@ -1640,9 +1646,9 @@ static ssize_t iter_xarray_get_pages_alloc(struct iov_iter *i, return actual; } -ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, +ssize_t iov_iter_get_pages_alloc_flags(struct iov_iter *i, struct page ***pages, size_t maxsize, - size_t *start) + size_t *start, unsigned int gup_flags) { struct page **p; size_t len; @@ -1654,7 +1660,6 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, return 0; if (likely(iter_is_iovec(i))) { - unsigned int gup_flags = 0; unsigned long addr; if (iov_iter_rw(i) != WRITE) @@ -1667,6 +1672,7 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, p = get_pages_array(n); if (!p) return -ENOMEM; + res = get_user_pages_fast(addr, n, gup_flags, p); if (unlikely(res <= 0)) { kvfree(p); @@ -1694,6 +1700,13 @@ ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, return iter_xarray_get_pages_alloc(i, pages, maxsize, start); return -EFAULT; } +EXPORT_SYMBOL_GPL(iov_iter_get_pages_alloc_flags); + +ssize_t iov_iter_get_pages_alloc(struct iov_iter *i, struct page ***pages, + size_t maxsize, size_t *start) +{ + return iov_iter_get_pages_alloc_flags(i, pages, maxsize, start, 0); +} EXPORT_SYMBOL(iov_iter_get_pages_alloc); size_t csum_and_copy_from_iter(void *addr, size_t bytes, __wsum *csum, From patchwork Wed Jun 15 16:12:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C308CCCA48C for ; Wed, 15 Jun 2022 16:12:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A63436B0074; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C6936B007E; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7082B6B0078; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 586556B0072 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2DF0960726 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.19.4909305 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf26.hostedemail.com (Postfix) with ESMTP id D6C0B140093 for ; Wed, 15 Jun 2022 16:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=CEsEo1LL0bapP/SgeXBMkHfYqc6Wv4poDHt3UT6FxEg=; b=Nn5f3rDAO5IH26Vm3tZC0/G00z 08nWaGa176sIR0AMP9ufpCFl8dqGInHVEIurBrhZxq8qyYDrYDC/FLwjHaP0xkmUhL0tsPpMFsnzC TrHv4QKvFeifgNV5SXw6bzeBiEo5BMK1Wn4z1hGAMBkF+nRrTHuJRTK9R1MaGtIlhbHed9Xofrf+/ /E6cLO35IfpR+c+O/fl3k708ke9cdQ/QBKGT2Ot+27a8SgRnqOwwPWqIzU5BypTjVOfuD2fhybceo a/BKPzddCviQTOWiZgVLC94Cl7+8/HNXYOTzmN6fEQW6XtzPv1NSvwCLPRKV2ZpBid/6o/AbJcz6B VA4O9vnw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdP-0084iP-Uw; Wed, 15 Jun 2022 10:12:48 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdI-0004ak-WE; Wed, 15 Jun 2022 10:12:41 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:28 -0600 Message-Id: <20220615161233.17527-17-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 16/21] block: add check when merging zone device pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=EAmTSnseZwOmKAAfhJzzILjNS30eGT9ZcYw0GToPcSZst2fTDTv0WXr/IKLN1Ut9CnqZnx QsL8NUB2tQf4tIHgPclndkLY481m6d+czvWlrIZayNe0Tn6dkorLTqym/sGInuiG94VdLq CWgE0dhOt5xl3eE5hmECVTijl5zRr1k= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=Nn5f3rDA; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf26.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CEsEo1LL0bapP/SgeXBMkHfYqc6Wv4poDHt3UT6FxEg=; b=W6no+fq1FDJ1x9fXtrx4D/mpFjwUUhJ0TMweh2FsWbEf6uxAFoThPZv1MhCa9okV2sLCyN Vy6UbuR6Q8fbZbV+M1O+elOE6EXBAz+NKkPQ0bltUSfijMSnWFzl8Y5ydub7l2TB1fhx// xmR8shJe9tk/Jt8TpPsIOUg/RM37ks8= Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=Nn5f3rDA; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf26.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspam-User: X-Stat-Signature: ceqm8ec14anok8sehmuzhhiucp7ouf1d X-Rspamd-Queue-Id: D6C0B140093 X-Rspamd-Server: rspam08 X-HE-Tag: 1655309574-711691 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Consecutive zone device pages should not be merged into the same sgl or bvec segment with other types of pages or if they belong to different pgmaps. Otherwise getting the pgmap of a given segment is not possible without scanning the entire segment. This helper returns true either if both pages are not zone device pages or both pages are zone device pages with the same pgmap. Add a helper to determine if zone device pages are mergeable and use this helper in page_is_mergeable(). Signed-off-by: Logan Gunthorpe --- block/bio.c | 2 ++ include/linux/mm.h | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/block/bio.c b/block/bio.c index f92d0223247b..a402a4760457 100644 --- a/block/bio.c +++ b/block/bio.c @@ -865,6 +865,8 @@ static inline bool page_is_mergeable(const struct bio_vec *bv, return false; if (xen_domain() && !xen_biovec_phys_mergeable(bv, page)) return false; + if (!zone_device_pages_have_same_pgmap(bv->bv_page, page)) + return false; *same_page = ((vec_end_addr & PAGE_MASK) == page_addr); if (*same_page) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0bcb54ea503c..33b2f4d9fd0a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1108,6 +1108,24 @@ static inline bool is_zone_device_page(const struct page *page) { return page_zonenum(page) == ZONE_DEVICE; } + +/* + * Consecutive zone device pages should not be merged into the same sgl + * or bvec segment with other types of pages or if they belong to different + * pgmaps. Otherwise getting the pgmap of a given segment is not possible + * without scanning the entire segment. This helper returns true either if + * both pages are not zone device pages or both pages are zone device pages + * with the same pgmap. + */ +static inline bool zone_device_pages_have_same_pgmap(const struct page *a, + const struct page *b) +{ + if (is_zone_device_page(a) != is_zone_device_page(b)) + return false; + if (!is_zone_device_page(a)) + return true; + return a->pgmap == b->pgmap; +} extern void memmap_init_zone_device(struct zone *, unsigned long, unsigned long, struct dev_pagemap *); #else @@ -1115,6 +1133,11 @@ static inline bool is_zone_device_page(const struct page *page) { return false; } +static inline bool zone_device_pages_have_same_pgmap(const struct page *a, + const struct page *b) +{ + return true; +} #endif static inline bool folio_is_zone_device(const struct folio *folio) From patchwork Wed Jun 15 16:12:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62A9ACCA489 for ; Wed, 15 Jun 2022 16:13:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07D886B007D; Wed, 15 Jun 2022 12:12:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0F3F6B0081; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92C796B0080; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6DB1F6B0075 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 445352054E for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.02.3F13D48 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf28.hostedemail.com (Postfix) with ESMTP id CA995C00A0 for ; Wed, 15 Jun 2022 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=UO35KE8YVx2v+yNTFMhmKCQE8ptUbM1lG6ODy8MFD4U=; b=TsE9W15o3EM/nOeQHIurouy/XA 2Y3+/BjdySgEMcfs9J9Xr3aFaFURXmsv3E6ylJDixGSN9s7g2SYY6k8/grdKUKTBmAaXttKXHKYId 8Tb8qzTHtupqDS54U8Ua402SNJp9CGGyjD4uUAGFAN4fPKqrYTUUiS57TgSmpYDH1NktW0RN014IT 1aacZVnPSkai9pTda7rEsqYo57R6/USD4Uj1WtQp3fMnHUD6mrnWJjmK6OMB8WlszyclZBg/9qMB0 U/N8B4+j8XWxbDNGEOQzbKXjwS85TBLNUeTFC5Ec5ZIE0ZIWESQ/C1jGVncnz+JbORAazZSyWmpZa xZImoPPw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdP-0084iQ-Un; Wed, 15 Jun 2022 10:12:49 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdJ-0004ao-5d; Wed, 15 Jun 2022 10:12:41 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:29 -0600 Message-Id: <20220615161233.17527-18-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 17/21] lib/scatterlist: add check when merging zone device pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=n3lUNPDtBtnIEkRi3dAFWjIfyphvILM/WjHHuX1+8D+446j3Tq9B6UVt7g3PwVfS+4rTVd QB6HS8Los1qX++p9irhbVD28wqm/Q3ZzrIFIgpBVcAsc18n5NETmDbGXTHUzfYoNvEYqab uq2SVSI6XrbcGOM5Wd7uFY+A9UPKODM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=TsE9W15o; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf28.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UO35KE8YVx2v+yNTFMhmKCQE8ptUbM1lG6ODy8MFD4U=; b=b/r2LKs40GyyEEYL6Sjgb6kks/n/XVZj9jSt+wT1mIEwl5+XIfm0livRscF/7XWt26VJj3 WOvsqqshB8WndWIIx3l/WI3nqUp0wMZlOnOfTin41KQd6ds0U15Og15mXE6VWe+6B2N/Yp mPxOlWK4qJmqdlFmyMKnxsUNYgJIcPU= Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=TsE9W15o; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf28.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspam-User: X-Stat-Signature: tqctb4w9o7cziam5o3zxj3p6buu7dpri X-Rspamd-Queue-Id: CA995C00A0 X-Rspamd-Server: rspam08 X-HE-Tag: 1655309573-107366 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Consecutive zone device pages should not be merged into the same sgl or bvec segment with other types of pages or if they belong to different pgmaps. Otherwise getting the pgmap of a given segment is not possible without scanning the entire segment. This helper returns true either if both pages are not zone device pages or both pages are zone device pages with the same pgmap. Factor out the check for page mergability into a pages_are_mergable() helper and add a check with zone_device_pages_are_mergeable(). Signed-off-by: Logan Gunthorpe --- lib/scatterlist.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/lib/scatterlist.c b/lib/scatterlist.c index d5e82e4a57ad..af53a0984f76 100644 --- a/lib/scatterlist.c +++ b/lib/scatterlist.c @@ -410,6 +410,15 @@ static struct scatterlist *get_next_sg(struct sg_append_table *table, return new_sg; } +static bool pages_are_mergeable(struct page *a, struct page *b) +{ + if (page_to_pfn(a) != page_to_pfn(b) + 1) + return false; + if (!zone_device_pages_have_same_pgmap(a, b)) + return false; + return true; +} + /** * sg_alloc_append_table_from_pages - Allocate and initialize an append sg * table from an array of pages @@ -447,6 +456,7 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, unsigned int chunks, cur_page, seg_len, i, prv_len = 0; unsigned int added_nents = 0; struct scatterlist *s = sgt_append->prv; + struct page *last_pg; /* * The algorithm below requires max_segment to be aligned to PAGE_SIZE @@ -460,21 +470,17 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, return -EOPNOTSUPP; if (sgt_append->prv) { - unsigned long paddr = - (page_to_pfn(sg_page(sgt_append->prv)) * PAGE_SIZE + - sgt_append->prv->offset + sgt_append->prv->length) / - PAGE_SIZE; - if (WARN_ON(offset)) return -EINVAL; /* Merge contiguous pages into the last SG */ prv_len = sgt_append->prv->length; - while (n_pages && page_to_pfn(pages[0]) == paddr) { + last_pg = sg_page(sgt_append->prv); + while (n_pages && pages_are_mergeable(last_pg, pages[0])) { if (sgt_append->prv->length + PAGE_SIZE > max_segment) break; sgt_append->prv->length += PAGE_SIZE; - paddr++; + last_pg = pages[0]; pages++; n_pages--; } @@ -488,7 +494,7 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, for (i = 1; i < n_pages; i++) { seg_len += PAGE_SIZE; if (seg_len >= max_segment || - page_to_pfn(pages[i]) != page_to_pfn(pages[i - 1]) + 1) { + !pages_are_mergeable(pages[i], pages[i - 1])) { chunks++; seg_len = 0; } @@ -504,8 +510,7 @@ int sg_alloc_append_table_from_pages(struct sg_append_table *sgt_append, for (j = cur_page + 1; j < n_pages; j++) { seg_len += PAGE_SIZE; if (seg_len >= max_segment || - page_to_pfn(pages[j]) != - page_to_pfn(pages[j - 1]) + 1) + !pages_are_mergeable(pages[j], pages[j - 1])) break; } From patchwork Wed Jun 15 16:12:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55B44C43334 for ; Wed, 15 Jun 2022 16:54:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 980676B0075; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 948326B0078; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 718A96B0074; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5EC846B0072 for ; Wed, 15 Jun 2022 12:54:56 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3030F33E0A for ; Wed, 15 Jun 2022 16:54:56 +0000 (UTC) X-FDA: 79581069792.14.D160232 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf31.hostedemail.com (Postfix) with ESMTP id B7513200AB for ; Wed, 15 Jun 2022 16:54:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=lkkVGV05lv0CRWPOx1aNSbQEHovTv1Jz5BwTja4gYMc=; b=UA65EMT8xNvd8UHWetmwqme2a8 mbh5ZAtLMo8oOkZni6a7e7xdFqgBaoKX6BW/nxEmRzBODrgtFlIhBvcgXCHi8DFCSj6ybFs8cFwFU JN5NcIKa2YwfaEFkGBKhq98tfYdc1ip6TDMNluisXCqwKBiArxGhT8ctmSh7yv0ONfMQKfAbFrlZ9 GPQ1EKHrP0E8uLkOsybyw5PjBkPkxm2wJNwPMXA7XshLSwQare9mNo0JzCLQGmtGTjS8GRORUOZTW 4uR6gNCfjlfo6idpW5EL2HFlkSZ0pa8rOW0YLmVjRT5GSymz5ydETTSteAPM/W/wXKf8A0ksSqUyH Qqr3rCNQ==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdP-0084iS-Qu; Wed, 15 Jun 2022 10:12:48 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdJ-0004as-C7; Wed, 15 Jun 2022 10:12:41 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:30 -0600 Message-Id: <20220615161233.17527-19-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 18/21] block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655312095; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lkkVGV05lv0CRWPOx1aNSbQEHovTv1Jz5BwTja4gYMc=; b=LnrNScsl2oHaAX1cW/kmcMN7FUDtNVbPRDHVmkxLa4bV5QW/2OO7GBLLfd6cOqxseCTZrF fWdxz6N354hyNr0bbiN2eZlb/EccMLx+oGeI3x3Ux1aba+qxnCvlVj28db93m57wLsS/A9 x+6/qqL6Sj5nhobYh3Q5Kh5YkYqf9n4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655312095; a=rsa-sha256; cv=none; b=shI/vMoVKA8oCmgBIl/1+ssVA49MqNYQzCa+Bt3wgnEq3VZKgoNMM2lW+v8csHcMQ9JaM6 p+uerBLI1rQoORT+in+i0l7n5Kb+2Y+Z3IX+csJV/azafhXhO4tSLweaT3jR1FnL7NaQej kgDjfYsdEL0k1M2TPI/kK8z+GOLWBfg= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=UA65EMT8; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=UA65EMT8; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: ma1cd8dmjo8xxysoyano99o5s9b5e6wm X-Rspamd-Queue-Id: B7513200AB X-HE-Tag: 1655312095-166500 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a bio's queue supports PCI P2PDMA, set FOLL_PCI_P2PDMA for iov_iter_get_pages_flags(). This allows PCI P2PDMA pages to be passed from userspace and enables the O_DIRECT path in iomap based filesystems and direct to block devices. Signed-off-by: Logan Gunthorpe --- block/bio.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/block/bio.c b/block/bio.c index a402a4760457..0d152da8938d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1180,6 +1180,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) struct bio_vec *bv = bio->bi_io_vec + bio->bi_vcnt; struct page **pages = (struct page **)bv; bool same_page = false; + unsigned int flags = 0; ssize_t size, left; unsigned len, i; size_t offset; @@ -1192,7 +1193,12 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) BUILD_BUG_ON(PAGE_PTRS_PER_BVEC < 2); pages += entries_left * (PAGE_PTRS_PER_BVEC - 1); - size = iov_iter_get_pages(iter, pages, LONG_MAX, nr_pages, &offset); + if (bio->bi_bdev && bio->bi_bdev->bd_disk && + blk_queue_pci_p2pdma(bio->bi_bdev->bd_disk->queue)) + flags |= FOLL_PCI_P2PDMA; + + size = iov_iter_get_pages_flags(iter, pages, LONG_MAX, nr_pages, + &offset, flags); if (unlikely(size <= 0)) return size ? size : -EFAULT; From patchwork Wed Jun 15 16:12:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA8F0CCA494 for ; Wed, 15 Jun 2022 16:13:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C80B36B0078; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BEF1C6B007B; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A3426B007D; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5E21E6B0074 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 3518C60284 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.20.C9220FA Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf29.hostedemail.com (Postfix) with ESMTP id BBEC6120089 for ; Wed, 15 Jun 2022 16:12:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=OqVljjCs7GJPfFJtCSMELP7hz7tpfnVYUqS/KYMT6cE=; b=D4xQJkI4XeiyHus7VGwGco9/eT CWBRfM3M9mPI7KhaQok5aaace5jsJ66GPHd0+bjL351TCnWoppTHSeVyW6ItHXgJGDaRJ7hd2IjiM F+kw1tcTRsPYrcDQzs2AykzCRimr2Zvwz2+TG3Yj8eCSUTt7BF3j2IR7Sq5QbQB0NBLtsSoXiMJpa dL7PRG248B15XgI06t6MosY+U92Q+v/iTv48RWM+JfmN+UMx2ilS1x342uwJrz5aoLn/s5TTg1Ggk H0HOUX/IOvALtFUR5mtAWvXlY6k08p6E637EzSv1gougvqbFBVszN6lMOnaLAfpvvZyeF3giITm3y oEzjQ+9g==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdP-0084iR-O3; Wed, 15 Jun 2022 10:12:48 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdJ-0004aw-Gi; Wed, 15 Jun 2022 10:12:41 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:31 -0600 Message-Id: <20220615161233.17527-20-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 19/21] block: set FOLL_PCI_P2PDMA in bio_map_user_iov() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OqVljjCs7GJPfFJtCSMELP7hz7tpfnVYUqS/KYMT6cE=; b=EFFSeX3a6wpWW9pRUX084bSrUmiejMvLs0fu/RM13kYUNMcuoN4hsdT4G9VvrbxxE2n52m PFXrRx62ERDPA3anzx0d+D/KukOCP/6VEJVPuE7mrz3kB1gPkEl6EW5louCFnASIh0JF4i mO+cuk8hNHcZArrbxRtZ/RnrN+fdtXE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=D4xQJkI4; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf29.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=h+OgQ/JuT2LNy4ogYpCtIVAzB6a/WaGNyQsilfqMI837V9kXkOvGRJKpiUmfYSPo6hPWx2 wE6/ZpGw0xJQLS6a/5LD0qsqg4oCpYetj/ah5AdAuMArU26UtwcdZglsr8Nz0J4LX4Nnsx vRJ3KcRZes+jQQkj/BFqLwOaVJjHX+I= X-Stat-Signature: eephoxe68k8w5g13c38s34e4iogn15ht X-Rspamd-Queue-Id: BBEC6120089 X-Rspam-User: X-Rspamd-Server: rspam05 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=D4xQJkI4; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf29.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-HE-Tag: 1655309574-782260 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a bio's queue supports PCI P2PDMA, set FOLL_PCI_P2PDMA for iov_iter_get_pages_flags(). This allows PCI P2PDMA pages to be passed from userspace and enables the NVMe passthru requests to use P2PDMA pages. Signed-off-by: Logan Gunthorpe --- block/blk-map.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/block/blk-map.c b/block/blk-map.c index df8b066cd548..1d6bcf193a42 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -236,6 +236,7 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, { unsigned int max_sectors = queue_max_hw_sectors(rq->q); unsigned int nr_vecs = iov_iter_npages(iter, BIO_MAX_VECS); + unsigned int flags = 0; struct bio *bio; int ret; int j; @@ -248,13 +249,17 @@ static int bio_map_user_iov(struct request *rq, struct iov_iter *iter, return -ENOMEM; bio_init(bio, NULL, bio->bi_inline_vecs, nr_vecs, req_op(rq)); + if (blk_queue_pci_p2pdma(rq->q)) + flags |= FOLL_PCI_P2PDMA; + while (iov_iter_count(iter)) { struct page **pages; ssize_t bytes; size_t offs, added = 0; int npages; - bytes = iov_iter_get_pages_alloc(iter, &pages, LONG_MAX, &offs); + bytes = iov_iter_get_pages_alloc_flags(iter, &pages, LONG_MAX, + &offs, flags); if (unlikely(bytes <= 0)) { ret = bytes ? bytes : -EFAULT; goto out_unmap; From patchwork Wed Jun 15 16:12:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FB05C43334 for ; Wed, 15 Jun 2022 16:55:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56ED36B0074; Wed, 15 Jun 2022 12:54:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A7516B0078; Wed, 15 Jun 2022 12:54:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D3AE6B007B; Wed, 15 Jun 2022 12:54:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0E1576B0074 for ; Wed, 15 Jun 2022 12:54:57 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id DF9E4601F3 for ; Wed, 15 Jun 2022 16:54:56 +0000 (UTC) X-FDA: 79581069792.02.3B3A61D Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf31.hostedemail.com (Postfix) with ESMTP id 43626200A2 for ; Wed, 15 Jun 2022 16:54:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=7T5Yp912TMrTnBD25fOSWhwseQ6HQoYmxMJPi7/7D1A=; b=t2aQN6Ic10SJRu095iqQv8oLQ8 1TpsurtP9/Obh/q9QIuVsiXHQ4/mc53d3/RYtRNEFh182mDWT2WQJMZpKMF3Hcwi/IBwwy6xzTxMo TcSn3wAD3e7NYVa/xAaIJvR9Xrypr8KAmZIrch7CjIh2JGeUbBOFo+23EyhclbndJfbX3tFtolwwG //hR0PBLlHfjVGwLVezcAhMLkGAQkJYQ8htIlK/t0Xy+h7nAcjjTo8KbJphBcoFpRXXI7EVlYTOGc doFiItWN75mJKpImuCAcJPMGkuIkYgsB+pBBV7mbe5WJ/vPPRrjk5fMcWQS/2/bDXYQgq6sYGnWq8 uHzFhUuw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdP-0084iT-5A; Wed, 15 Jun 2022 10:12:48 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdJ-0004b0-L7; Wed, 15 Jun 2022 10:12:41 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe , Bjorn Helgaas Date: Wed, 15 Jun 2022 10:12:32 -0600 Message-Id: <20220615161233.17527-21-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com, bhelgaas@google.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 20/21] PCI/P2PDMA: Introduce pci_mmap_p2pmem() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655312096; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7T5Yp912TMrTnBD25fOSWhwseQ6HQoYmxMJPi7/7D1A=; b=f490Ybh2sHjSTsvFgOeeXGmazRRAK/K+xIDknJZUGyvzsqKOfTpltfrag8VA620ks2lGEY ga43nsLNhG1c9DKasSr4+cxxDisj1wU3M34/UTnE73GyfuNxuwbnI+O0Qt9zBzcUbFYJVe fJDZmfDwptYKJl4SMUbhaLpKp8ClLPE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655312096; a=rsa-sha256; cv=none; b=AxBdA84sGwpKrpfGJKo9xa2YOIDedfHAn2EJC91Ew6sEtiDmJLqUrdXKvxUDgPzjyy0sbw tVsBWjeOm2v0z1vv1NTr/ARpivhQxC1PEG82tzB97DshRlRjwUN+cZRcQNBh9eoGQA6l2S IqsLaVmKTKyue76LZLKgAKavIACji4E= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=t2aQN6Ic; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=t2aQN6Ic; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf31.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam12 X-Rspam-User: X-Stat-Signature: kpsdwbrzqp8oxga7a9aoyx9gqafdgaad X-Rspamd-Queue-Id: 43626200A2 X-HE-Tag: 1655312096-314724 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce pci_mmap_p2pmem() which is a helper to allocate and mmap a hunk of p2pmem into userspace. Pages are allocated from the genalloc in bulk with their reference count set to one. They are returned to the genalloc when the page is put through p2pdma_page_free() (the reference count is once again set to 1 in free_zone_device_page()). The VMA does not take a reference to the pages when they are inserted with vmf_insert_mixed() (which is necessary for zone device pages) so the backing P2P memory is stored in a structures in vm_private_data. A pseudo mount is used to allocate an inode for each PCI device. The inode's address_space is used in the file doing the mmap so that all VMAs are collected and can be unmapped if the PCI device is unbound. After unmapping, the VMAs are iterated through and their pages are put so the device can continue to be unbound. An active flag is used to signal to VMAs not to allocate any further P2P memory once the removal process starts. The flag is synchronized with concurrent access with an RCU lock. The VMAs and inode will survive after the unbind of the device, but no pages will be present in the VMA and a subsequent access will result in a SIGBUS error. Signed-off-by: Logan Gunthorpe Acked-by: Bjorn Helgaas --- drivers/pci/p2pdma.c | 210 ++++++++++++++++++++++++++++++++++++- include/linux/pci-p2pdma.h | 16 +++ include/uapi/linux/magic.h | 1 + 3 files changed, 225 insertions(+), 2 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index d4e635012ffe..a6572069008b 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -17,14 +17,19 @@ #include #include #include +#include +#include #include #include #include +#include struct pci_p2pdma { struct gen_pool *pool; bool p2pmem_published; struct xarray map_types; + struct inode *inode; + bool active; }; struct pci_p2pdma_pagemap { @@ -101,6 +106,41 @@ static const struct attribute_group p2pmem_group = { .name = "p2pmem", }; +/* + * P2PDMA internal mount + * Fake an internal VFS mount-point in order to allocate struct address_space + * mappings to remove VMAs on unbind events. + */ +static int pci_p2pdma_fs_cnt; +static struct vfsmount *pci_p2pdma_fs_mnt; + +static int pci_p2pdma_fs_init_fs_context(struct fs_context *fc) +{ + return init_pseudo(fc, P2PDMA_MAGIC) ? 0 : -ENOMEM; +} + +static struct file_system_type pci_p2pdma_fs_type = { + .name = "p2dma", + .owner = THIS_MODULE, + .init_fs_context = pci_p2pdma_fs_init_fs_context, + .kill_sb = kill_anon_super, +}; + +static void p2pdma_page_free(struct page *page) +{ + struct pci_p2pdma_pagemap *pgmap = to_p2p_pgmap(page->pgmap); + struct percpu_ref *ref; + + gen_pool_free_owner(pgmap->provider->p2pdma->pool, + (uintptr_t)page_to_virt(page), PAGE_SIZE, + (void **)&ref); + percpu_ref_put(ref); +} + +static const struct dev_pagemap_ops p2pdma_pgmap_ops = { + .page_free = p2pdma_page_free, +}; + static void pci_p2pdma_release(void *data) { struct pci_dev *pdev = data; @@ -117,6 +157,9 @@ static void pci_p2pdma_release(void *data) gen_pool_destroy(p2pdma->pool); sysfs_remove_group(&pdev->dev.kobj, &p2pmem_group); xa_destroy(&p2pdma->map_types); + + iput(p2pdma->inode); + simple_release_fs(&pci_p2pdma_fs_mnt, &pci_p2pdma_fs_cnt); } static int pci_p2pdma_setup(struct pci_dev *pdev) @@ -134,17 +177,32 @@ static int pci_p2pdma_setup(struct pci_dev *pdev) if (!p2p->pool) goto out; - error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev); + error = simple_pin_fs(&pci_p2pdma_fs_type, &pci_p2pdma_fs_mnt, + &pci_p2pdma_fs_cnt); if (error) goto out_pool_destroy; + p2p->inode = alloc_anon_inode(pci_p2pdma_fs_mnt->mnt_sb); + if (IS_ERR(p2p->inode)) { + error = -ENOMEM; + goto out_unpin_fs; + } + + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_release, pdev); + if (error) + goto out_put_inode; + error = sysfs_create_group(&pdev->dev.kobj, &p2pmem_group); if (error) - goto out_pool_destroy; + goto out_put_inode; rcu_assign_pointer(pdev->p2pdma, p2p); return 0; +out_put_inode: + iput(p2p->inode); +out_unpin_fs: + simple_release_fs(&pci_p2pdma_fs_mnt, &pci_p2pdma_fs_cnt); out_pool_destroy: gen_pool_destroy(p2p->pool); out: @@ -152,6 +210,18 @@ static int pci_p2pdma_setup(struct pci_dev *pdev) return error; } +static void pci_p2pdma_unmap_mappings(void *data) +{ + struct pci_dev *pdev = data; + struct pci_p2pdma *p2pdma = rcu_dereference_protected(pdev->p2pdma, 1); + + /* Ensure no new pages can be allocated in mappings */ + p2pdma->active = false; + synchronize_rcu(); + + unmap_mapping_range(p2pdma->inode->i_mapping, 0, 0, 1); +} + /** * pci_p2pdma_add_resource - add memory for use as p2p memory * @pdev: the device to add the memory to @@ -198,6 +268,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->range.end = pgmap->range.start + size - 1; pgmap->nr_range = 1; pgmap->type = MEMORY_DEVICE_PCI_P2PDMA; + pgmap->ops = &p2pdma_pgmap_ops; p2p_pgmap->provider = pdev; p2p_pgmap->bus_offset = pci_bus_address(pdev, bar) - @@ -209,6 +280,11 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, goto pgmap_free; } + error = devm_add_action_or_reset(&pdev->dev, pci_p2pdma_unmap_mappings, + pdev); + if (error) + goto pages_free; + p2pdma = rcu_dereference_protected(pdev->p2pdma, 1); error = gen_pool_add_owner(p2pdma->pool, (unsigned long)addr, pci_bus_address(pdev, bar) + offset, @@ -217,6 +293,7 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, if (error) goto pages_free; + p2pdma->active = true; pci_info(pdev, "added peer-to-peer DMA memory %#llx-%#llx\n", pgmap->range.start, pgmap->range.end); @@ -1023,3 +1100,132 @@ ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev, return sprintf(page, "%s\n", pci_name(p2p_dev)); } EXPORT_SYMBOL_GPL(pci_p2pdma_enable_show); + +/** + * pci_p2pdma_file_open - setup file mapping to store P2PMEM VMAs + * @pdev: the device to allocate memory from + * @file: the file to open + * + * Set f_mapping of the file to the p2pdma inode so that mappings + * are can be torn down on device unbind. + */ +int pci_p2pdma_file_open(struct pci_dev *pdev, struct file *file) +{ + struct pci_p2pdma *p2pdma; + int ret; + + ret = simple_pin_fs(&pci_p2pdma_fs_type, &pci_p2pdma_fs_mnt, + &pci_p2pdma_fs_cnt); + if (ret) + return ret; + + rcu_read_lock(); + p2pdma = rcu_dereference(pdev->p2pdma); + if (p2pdma) { + ihold(p2pdma->inode); + file->f_mapping = p2pdma->inode->i_mapping; + rcu_read_unlock(); + } else { + rcu_read_unlock(); + simple_release_fs(&pci_p2pdma_fs_mnt, &pci_p2pdma_fs_cnt); + } + + return 0; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_file_open); + +/** + * pci_p2pdma_file_release - release a file opened with pci_p2pdma_file_open() + * @file: the userspace vma to map the memory to + * + * Release the reference to f_mapping set by pci_p2pdma_file_open() + */ +void pci_p2pdma_file_release(struct file *file) +{ + if (file->f_mapping->host != file->f_inode) { + iput(file->f_mapping->host); + simple_release_fs(&pci_p2pdma_fs_mnt, &pci_p2pdma_fs_cnt); + } +} +EXPORT_SYMBOL_GPL(pci_p2pdma_file_release); + +/** + * pci_mmap_p2pmem - setup an mmap region to be backed with P2PDMA memory + * that was registered with pci_p2pdma_add_resource() + * @pdev: the device to allocate memory from + * @vma: the userspace vma to map the memory to + * + * The file must call pci_p2pdma_mmap_file_open() in its open() operation. + * + * Returns 0 on success, or a negative error code on failure + */ +int pci_mmap_p2pmem(struct pci_dev *pdev, struct vm_area_struct *vma) +{ + size_t len = vma->vm_end - vma->vm_start; + struct pci_p2pdma *p2pdma; + struct percpu_ref *ref; + unsigned long vaddr; + void *kaddr; + int ret; + + /* prevent private mappings from being established */ + if ((vma->vm_flags & VM_MAYSHARE) != VM_MAYSHARE) { + pci_info_ratelimited(pdev, + "%s: fail, attempted private mapping\n", + current->comm); + return -EINVAL; + } + + if (vma->vm_pgoff) { + pci_info_ratelimited(pdev, + "%s: fail, attempted mapping with non-zero offset\n", + current->comm); + return -EINVAL; + } + + rcu_read_lock(); + p2pdma = rcu_dereference(pdev->p2pdma); + if (!p2pdma || !p2pdma->active) { + ret = -ENODEV; + goto out; + } + + kaddr = (void *)gen_pool_alloc_owner(p2pdma->pool, len, (void **)&ref); + if (!kaddr) { + ret = -ENOMEM; + goto out; + } + + /* + * vm_insert_page() can sleep, so a reference is taken to mapping + * such that rcu_read_unlock() can be done before inserting the + * pages + */ + if (unlikely(!percpu_ref_tryget_live_rcu(ref))) { + ret = -ENODEV; + goto out_free_mem; + } + rcu_read_unlock(); + + for (vaddr = vma->vm_start; vaddr < vma->vm_end; vaddr += PAGE_SIZE) { + ret = vm_insert_page(vma, vaddr, virt_to_page(kaddr)); + if (ret) { + gen_pool_free(p2pdma->pool, (uintptr_t)kaddr, len); + return ret; + } + percpu_ref_get(ref); + put_page(virt_to_page(kaddr)); + kaddr += PAGE_SIZE; + len -= PAGE_SIZE; + } + + percpu_ref_put(ref); + + return 0; +out_free_mem: + gen_pool_free(p2pdma->pool, (uintptr_t)kaddr, len); +out: + rcu_read_unlock(); + return ret; +} +EXPORT_SYMBOL_GPL(pci_mmap_p2pmem); diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h index 2c07aa6b7665..0ffe782940da 100644 --- a/include/linux/pci-p2pdma.h +++ b/include/linux/pci-p2pdma.h @@ -34,6 +34,9 @@ int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev, bool *use_p2pdma); ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev, bool use_p2pdma); +int pci_p2pdma_file_open(struct pci_dev *pdev, struct file *file); +void pci_p2pdma_file_release(struct file *file); +int pci_mmap_p2pmem(struct pci_dev *pdev, struct vm_area_struct *vma); #else /* CONFIG_PCI_P2PDMA */ static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, u64 offset) @@ -90,6 +93,19 @@ static inline ssize_t pci_p2pdma_enable_show(char *page, { return sprintf(page, "none\n"); } +static inline int pci_p2pdma_file_open(struct pci_dev *pdev, + struct file *file) +{ + return 0; +} +static inline void pci_p2pdma_file_release(struct file *file) +{ +} +static inline int pci_mmap_p2pmem(struct pci_dev *pdev, + struct vm_area_struct *vma) +{ + return -EOPNOTSUPP; +} #endif /* CONFIG_PCI_P2PDMA */ diff --git a/include/uapi/linux/magic.h b/include/uapi/linux/magic.h index f724129c0425..59ba2e60dc03 100644 --- a/include/uapi/linux/magic.h +++ b/include/uapi/linux/magic.h @@ -95,6 +95,7 @@ #define BPF_FS_MAGIC 0xcafe4a11 #define AAFS_MAGIC 0x5a3c69f0 #define ZONEFS_MAGIC 0x5a4f4653 +#define P2PDMA_MAGIC 0x70327064 /* Since UDF 2.01 is ISO 13346 based... */ #define UDF_SUPER_MAGIC 0x15013346 From patchwork Wed Jun 15 16:12:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12882619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2F13CCA487 for ; Wed, 15 Jun 2022 16:12:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 720036B0072; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6BA5C6B007B; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 532836B0075; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 404AE6B0074 for ; Wed, 15 Jun 2022 12:12:55 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 163F635151 for ; Wed, 15 Jun 2022 16:12:55 +0000 (UTC) X-FDA: 79580963910.13.3BE6D0C Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf16.hostedemail.com (Postfix) with ESMTP id ADDE2180052 for ; Wed, 15 Jun 2022 16:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=X1Y+xHoeQH3UW63rjymiD/emNoqwBnemSVoIxTF9EbY=; b=iKgX6plSnGYTUQgEz3h2eQ8oq4 FGlAIFJWTusbVwsUGEKQv5w3YldIr+Zb7y9e5Tci57QG4okfn/WNYoSJSPedAWtRsJ317QnqZ3tYN B1fywGUu5iwDLF9GPEgBC7jVnfl/bzFg+cc/N6mZ6JQngmxQwJYFT1EsGSRC6XIG1doRvnySpVRU/ tzfXsXC/u1Ur26GJQGnmxT26Dpy7yBQ61LA/ooaFX1WIMUL63WFzcYKEzecykYtWm5V8bVzy+p78C Z+iMAawOzUWIvBjOXPztbfswoP+njdZ1jiGYb3TqxWthTb6w8cx/ik5VmwwPqeARrQ2tc76XIBIjc 0eUneuvg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1o1VdO-0084iQ-Ox; Wed, 15 Jun 2022 10:12:47 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1o1VdJ-0004b4-QT; Wed, 15 Jun 2022 10:12:41 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Logan Gunthorpe Date: Wed, 15 Jun 2022 10:12:33 -0600 Message-Id: <20220615161233.17527-22-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220615161233.17527-1-logang@deltatee.com> References: <20220615161233.17527-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, helgaas@kernel.org, dan.j.williams@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, jhubbard@nvidia.com, rcampbell@nvidia.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v7 21/21] nvme-pci: allow mmaping the CMB in userspace X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655309574; a=rsa-sha256; cv=none; b=wN/7djMLFAgUwz+UCYMY754GWQY7GR5iKw2b2JivioMb129T6Mv9pQzTSWwC//I4fNp4HY VolwtsAK6yZIwrz5dnPz5/gEr4dzy86dolIjaJ0ZvOAyT/o2T6szYwfz+CKQ9l6htEWaMR TVwxZTSWpLocnAXa8Cz4zGPPXKpboyg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=iKgX6plS; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf16.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655309574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X1Y+xHoeQH3UW63rjymiD/emNoqwBnemSVoIxTF9EbY=; b=pCS9UGodPvJqNtB3VY+ouvPBjLWcmpfqJE3+dukXJsQTeol3AeYRmfDcGd15OW8cQ6ATYH hrZ4vtRPULybLkyUSUCyvgyASBzND2wdVxwlG9MS7vIMxjc7G9Z3Ov8B5YTY0CQ61XleRE fxfCmFFFPKwsiF7xZHQhfcJp8nzRmk0= X-Rspamd-Queue-Id: ADDE2180052 X-Rspam-User: Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=iKgX6plS; dmarc=pass (policy=none) header.from=deltatee.com; spf=pass (imf16.hostedemail.com: domain of gunthorp@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=gunthorp@deltatee.com X-Rspamd-Server: rspam06 X-Stat-Signature: cjgeftzdn3f37oafrht4aydhr66toi8z X-HE-Tag: 1655309573-971473 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow userspace to obtain CMB memory by mmaping the controller's char device. The mmap call allocates and returns a hunk of CMB memory, (the offset is ignored) so userspace does not have control over the address within the CMB. A VMA allocated in this way will only be usable by drivers that set FOLL_PCI_P2PDMA when calling GUP. And inter-device support will be checked the first time the pages are mapped for DMA. Currently this is only supported by O_DIRECT to an PCI NVMe device or through the NVMe passthrough IOCTL. Signed-off-by: Logan Gunthorpe --- drivers/nvme/host/core.c | 35 +++++++++++++++++++++++++++++++---- drivers/nvme/host/nvme.h | 3 +++ drivers/nvme/host/pci.c | 23 +++++++++++++++++++++++ 3 files changed, 57 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index d6e76f2dc293..23fe4b544bf1 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3166,6 +3166,7 @@ static int nvme_dev_open(struct inode *inode, struct file *file) { struct nvme_ctrl *ctrl = container_of(inode->i_cdev, struct nvme_ctrl, cdev); + int ret = -EINVAL; switch (ctrl->state) { case NVME_CTRL_LIVE: @@ -3175,13 +3176,25 @@ static int nvme_dev_open(struct inode *inode, struct file *file) } nvme_get_ctrl(ctrl); - if (!try_module_get(ctrl->ops->module)) { - nvme_put_ctrl(ctrl); - return -EINVAL; - } + if (!try_module_get(ctrl->ops->module)) + goto err_put_ctrl; file->private_data = ctrl; + + if (ctrl->ops->cdev_file_open) { + ret = ctrl->ops->cdev_file_open(ctrl, file); + if (ret) + goto err_put_mod; + } + return 0; + +err_put_mod: + module_put(ctrl->ops->module); +err_put_ctrl: + nvme_put_ctrl(ctrl); + return ret; + } static int nvme_dev_release(struct inode *inode, struct file *file) @@ -3189,11 +3202,24 @@ static int nvme_dev_release(struct inode *inode, struct file *file) struct nvme_ctrl *ctrl = container_of(inode->i_cdev, struct nvme_ctrl, cdev); + if (ctrl->ops->cdev_file_release) + ctrl->ops->cdev_file_release(file); + module_put(ctrl->ops->module); nvme_put_ctrl(ctrl); return 0; } +static int nvme_dev_mmap(struct file *file, struct vm_area_struct *vma) +{ + struct nvme_ctrl *ctrl = file->private_data; + + if (!ctrl->ops->mmap_cmb) + return -ENODEV; + + return ctrl->ops->mmap_cmb(ctrl, vma); +} + static const struct file_operations nvme_dev_fops = { .owner = THIS_MODULE, .open = nvme_dev_open, @@ -3201,6 +3227,7 @@ static const struct file_operations nvme_dev_fops = { .unlocked_ioctl = nvme_dev_ioctl, .compat_ioctl = compat_ptr_ioctl, .uring_cmd = nvme_dev_uring_cmd, + .mmap = nvme_dev_mmap, }; static ssize_t nvme_sysfs_reset(struct device *dev, diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 957f79420cf3..44ff05d8e24d 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -503,6 +503,9 @@ struct nvme_ctrl_ops { void (*delete_ctrl)(struct nvme_ctrl *ctrl); int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size); bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); + int (*cdev_file_open)(struct nvme_ctrl *ctrl, struct file *file); + void (*cdev_file_release)(struct file *file); + int (*mmap_cmb)(struct nvme_ctrl *ctrl, struct vm_area_struct *vma); }; /* diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 52b52a7efa9a..8ef3752b7ddb 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2972,6 +2972,26 @@ static bool nvme_pci_supports_pci_p2pdma(struct nvme_ctrl *ctrl) return dma_pci_p2pdma_supported(dev->dev); } +static int nvme_pci_cdev_file_open(struct nvme_ctrl *ctrl, struct file *file) +{ + struct pci_dev *pdev = to_pci_dev(to_nvme_dev(ctrl)->dev); + + return pci_p2pdma_file_open(pdev, file); +} + +static void nvme_pci_cdev_file_release(struct file *file) +{ + pci_p2pdma_file_release(file); +} + +static int nvme_pci_mmap_cmb(struct nvme_ctrl *ctrl, + struct vm_area_struct *vma) +{ + struct pci_dev *pdev = to_pci_dev(to_nvme_dev(ctrl)->dev); + + return pci_mmap_p2pmem(pdev, vma); +} + static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name = "pcie", .module = THIS_MODULE, @@ -2983,6 +3003,9 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .submit_async_event = nvme_pci_submit_async_event, .get_address = nvme_pci_get_address, .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma, + .cdev_file_open = nvme_pci_cdev_file_open, + .cdev_file_release = nvme_pci_cdev_file_release, + .mmap_cmb = nvme_pci_mmap_cmb, }; static int nvme_dev_map(struct nvme_dev *dev)