From patchwork Thu Mar 11 23:31:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F174C28E80 for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7B5B364F92 for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231390AbhCKXtD (ORCPT ); Thu, 11 Mar 2021 18:49:03 -0500 Received: from ale.deltatee.com ([204.191.154.188]:55760 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231278AbhCKXsx (ORCPT ); Thu, 11 Mar 2021 18:48:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=p6LSuxLxIXhW56xVX60ERoOIubTQKg0kFsUTFXBJO+A=; b=KvuTM+MB3gjvIolcULjFn23dNB 3UTcV/MdnfIXaNhLXS0a6rZH29zcNJFPH69erQKmEMF5Us9dBYKvKT+cwUnXcVVfkHmoSfRX8WPai K8M6TbmudsvF4nPi214vAODKBReiwl70HbEUINS//9CLpBrmWytPjko8FzX0jH0oKXnKz6JIpHCnG MDZ3uAQYYiD3i8bG4wDYOe1PrAaOL2kYOIuUOoONKVRQOznvCE2vkBlTujANvlJn5HAvzkEV7v7EY DAdA+NzNmhMoK6wFXyvdDV9raMZtrodSamhEKKTgZxJGwGLXqT+iYTFkagT3nR+ql1sjAqryz+oId 4O7xEzbA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmZ-0003er-Rv; Thu, 11 Mar 2021 16:32:15 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmV-00024M-T0; Thu, 11 Mar 2021 16:31:51 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:31 -0700 Message-Id: <20210311233142.7900-2-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 01/11] PCI/P2PDMA: Pass gfp_mask flags to upstream_bridge_distance_warn() X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org In order to call this function from a dma_map function, it must not sleep. The only reason it does sleep so to allocate the seqbuf to print which devices are within the ACS path. Switch the kmalloc call to use a passed in gfp_mask and don't print that message if the buffer fails to be allocated. Signed-off-by: Logan Gunthorpe Acked-by: Bjorn Helgaas --- drivers/pci/p2pdma.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 196382630363..bd89437faf06 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -267,7 +267,7 @@ static int pci_bridge_has_acs_redir(struct pci_dev *pdev) static void seq_buf_print_bus_devfn(struct seq_buf *buf, struct pci_dev *pdev) { - if (!buf) + if (!buf || !buf->buffer) return; seq_buf_printf(buf, "%s;", pci_name(pdev)); @@ -495,25 +495,26 @@ upstream_bridge_distance(struct pci_dev *provider, struct pci_dev *client, static enum pci_p2pdma_map_type upstream_bridge_distance_warn(struct pci_dev *provider, struct pci_dev *client, - int *dist) + int *dist, gfp_t gfp_mask) { struct seq_buf acs_list; bool acs_redirects; int ret; - seq_buf_init(&acs_list, kmalloc(PAGE_SIZE, GFP_KERNEL), PAGE_SIZE); - if (!acs_list.buffer) - return -ENOMEM; + seq_buf_init(&acs_list, kmalloc(PAGE_SIZE, gfp_mask), PAGE_SIZE); ret = upstream_bridge_distance(provider, client, dist, &acs_redirects, &acs_list); if (acs_redirects) { pci_warn(client, "ACS redirect is set between the client and provider (%s)\n", pci_name(provider)); - /* Drop final semicolon */ - acs_list.buffer[acs_list.len-1] = 0; - pci_warn(client, "to disable ACS redirect for this path, add the kernel parameter: pci=disable_acs_redir=%s\n", - acs_list.buffer); + + if (acs_list.buffer) { + /* Drop final semicolon */ + acs_list.buffer[acs_list.len - 1] = 0; + pci_warn(client, "to disable ACS redirect for this path, add the kernel parameter: pci=disable_acs_redir=%s\n", + acs_list.buffer); + } } if (ret == PCI_P2PDMA_MAP_NOT_SUPPORTED) { @@ -566,7 +567,7 @@ int pci_p2pdma_distance_many(struct pci_dev *provider, struct device **clients, if (verbose) ret = upstream_bridge_distance_warn(provider, - pci_client, &distance); + pci_client, &distance, GFP_KERNEL); else ret = upstream_bridge_distance(provider, pci_client, &distance, NULL, NULL); From patchwork Thu Mar 11 23:31:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C8F6C43333 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8368564FCE for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231320AbhCKXtA (ORCPT ); Thu, 11 Mar 2021 18:49:00 -0500 Received: from ale.deltatee.com ([204.191.154.188]:41534 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231241AbhCKXsn (ORCPT ); Thu, 11 Mar 2021 18:48:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=MTA2+M7vdLJxXxHYOK9U0k000IpPFkzl7XK6463+pFs=; b=FjJsKZ5LGW1FuIX+RDt4/0qpAp r2njv0oqYGqnhACdOut9mLWzC3ph6VGi+mM46gPp3XJoEYVdLJ+xFYFmdHFHmt3/n1ZJqq87nD+iv Re3mXFmoTD/PaEMAP4OsIFIkv1AB1OWVvx0hawBBsD+D/ZQotC7n7uA+1GSghCHFxmR9LL4H1XW8Q zvkIOQ1dewff1FGyOJXlLGEhjWgS5Bdf1w+Hjt/MnIZDP5t3o8OdgEn7vfrPocavrhMZOtqMvNoDn hj7CZv/guhyfLxb4HrGY+Kit8fwIbqYUz+WfycmHxVYSqMOw9C13RWshnkVth/atpXnCWFLnCYKn5 lde1K09w==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmZ-0003es-Rw; Thu, 11 Mar 2021 16:32:15 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024P-1b; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:32 -0700 Message-Id: <20210311233142.7900-3-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 02/11] PCI/P2PDMA: Avoid pci_get_slot() which sleeps X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org In order to use upstream_bridge_distance_warn() from a dma_map function, it must not sleep. However, pci_get_slot() takes the pci_bus_sem so it might sleep. In order to avoid this, try to get the host bridge's device from bus->self, and if that is not set just get the first element in the list. It should be impossible for the host bridges device to go away while references are held on child devices, so the first element should not change and this should be safe. Signed-off-by: Logan Gunthorpe --- drivers/pci/p2pdma.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index bd89437faf06..2135fe69bb07 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -311,11 +311,15 @@ static const struct pci_p2pdma_whitelist_entry { static bool __host_bridge_whitelist(struct pci_host_bridge *host, bool same_host_bridge) { - struct pci_dev *root = pci_get_slot(host->bus, PCI_DEVFN(0, 0)); const struct pci_p2pdma_whitelist_entry *entry; + struct pci_dev *root = host->bus->self; unsigned short vendor, device; if (!root) + root = list_first_entry_or_null(&host->bus->devices, + struct pci_dev, bus_list); + + if (!root || root->devfn) return false; vendor = root->vendor; From patchwork Thu Mar 11 23:31:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4EADC433E0 for ; Thu, 11 Mar 2021 23:50:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 949C964F7C for ; Thu, 11 Mar 2021 23:50:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231573AbhCKXtb (ORCPT ); Thu, 11 Mar 2021 18:49:31 -0500 Received: from ale.deltatee.com ([204.191.154.188]:36472 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229574AbhCKXs7 (ORCPT ); Thu, 11 Mar 2021 18:48:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=GuKrcY+MDyB48LoFpeM5eZewsFi6xn3yS3/qfyrxy0g=; b=bskuVVfDsBtrM8Ka5VeIt1rZSw KoTj2gXNVBq/5+o+ETpfcxb0hasISNtuCI0MrZDLAlWv61F8veluRVLJPFkvrSKjWNsJ/ETU6rBfo ApD3VqQNs/1jUtgTZIv0Uf87j5hDtS9AZYmU2DJWPORg7cMxKyR2RBqXETGwv9pqlf8EuFiFfxeMo w1oi0NlLPR7HHZ/82oxXWNp+7iyeiGYWJLHhC2+TI6/yjTwyE2p/VfADeYODAHg/j349bktyIEVHC EVGiU3gd0tTUxCYpRhmRifKUcErf5Zd6RBHTh39esGgR0ffbddaI0dCgfhcLc6rIJ01yGkmNF2DDH qROVkdWA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmZ-0003et-Ru; Thu, 11 Mar 2021 16:31:57 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024S-7F; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:33 -0700 Message-Id: <20210311233142.7900-4-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 03/11] PCI/P2PDMA: Attempt to set map_type if it has not been set X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Attempt to find the mapping type for P2PDMA pages on the first DMA map attempt if it has not been done ahead of time. Previously, the mapping type was expected to be calculated ahead of time, but if pages are to come from userspace then there's no way to ensure the path was checked ahead of time. Signed-off-by: Logan Gunthorpe --- drivers/pci/p2pdma.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 2135fe69bb07..7f6836732bce 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -819,11 +819,18 @@ EXPORT_SYMBOL_GPL(pci_p2pmem_publish); static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct pci_dev *provider, struct pci_dev *client) { + enum pci_p2pdma_map_type ret; + if (!provider->p2pdma) return PCI_P2PDMA_MAP_NOT_SUPPORTED; - return xa_to_value(xa_load(&provider->p2pdma->map_types, - map_types_idx(client))); + ret = xa_to_value(xa_load(&provider->p2pdma->map_types, + map_types_idx(client))); + if (ret != PCI_P2PDMA_MAP_UNKNOWN) + return ret; + + return upstream_bridge_distance_warn(provider, client, NULL, + GFP_ATOMIC); } static int __pci_p2pdma_map_sg(struct pci_p2pdma_pagemap *p2p_pgmap, @@ -871,7 +878,6 @@ int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg, case PCI_P2PDMA_MAP_BUS_ADDR: return __pci_p2pdma_map_sg(p2p_pgmap, dev, sg, nents); default: - WARN_ON_ONCE(1); return 0; } } From patchwork Thu Mar 11 23:31:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82D2DC43331 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 74D4364F70 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231309AbhCKXs7 (ORCPT ); Thu, 11 Mar 2021 18:48:59 -0500 Received: from ale.deltatee.com ([204.191.154.188]:37902 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231238AbhCKXsl (ORCPT ); Thu, 11 Mar 2021 18:48:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=SCChKxOLna0Q8gAs6qhk6ikz9L246Zfzrk9nkrpKYb4=; b=bU1GOcegswBCC8SOec1nyEkx0A VZQKHaKmcKUhgfpPzKGEvZZIe9kEAjBk6aTtm++lM9s5YEqcvVhRkHXu4Eg6j1kkWzDad+es9fcIN QPP80CO4PXoSi9GzW8aQa0VETX9vReZ6PLOr9FSqvphL6ToAedUIMVZeOCDdeM/0sObWdc7OLjSkO Ywi39f+d6/GdN6wHTVvUpHYyYqIi1sTw1HqLzgLn5+CqgZlxQB3jZThIH8z5PrWHOb/9iueBJi4r+ +uYu9XwpOBvwFnqHHB3tcvMM+kWZ8Qd+l9XxGCMttKba6dxUL/BkAoERcTXfDuijs6/MtAg862DnG w/IJk/9w==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmZ-0003ev-Rw; Thu, 11 Mar 2021 16:31:56 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024V-Ap; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:34 -0700 Message-Id: <20210311233142.7900-5-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 04/11] PCI/P2PDMA: Introduce pci_p2pdma_should_map_bus() and pci_p2pdma_bus_offset() X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Introduce pci_p2pdma_should_map_bus() which is meant to be called by DMA map functions to determine how to map a given p2pdma page. pci_p2pdma_bus_offset() is also added to allow callers to get the bus offset if they need to map the bus address. Signed-off-by: Logan Gunthorpe --- drivers/pci/p2pdma.c | 50 ++++++++++++++++++++++++++++++++++++++ include/linux/pci-p2pdma.h | 11 +++++++++ 2 files changed, 61 insertions(+) diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c index 7f6836732bce..66d16b7eb668 100644 --- a/drivers/pci/p2pdma.c +++ b/drivers/pci/p2pdma.c @@ -912,6 +912,56 @@ void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, } EXPORT_SYMBOL_GPL(pci_p2pdma_unmap_sg_attrs); +/** + * pci_p2pdma_bus_offset - returns the bus offset for a given page + * @page: page to get the offset for + * + * Must be passed a PCI p2pdma page. + */ +u64 pci_p2pdma_bus_offset(struct page *page) +{ + struct pci_p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(page->pgmap); + + WARN_ON(!is_pci_p2pdma_page(page)); + + return p2p_pgmap->bus_offset; +} +EXPORT_SYMBOL_GPL(pci_p2pdma_bus_offset); + +/** + * pci_p2pdma_dma_map_type - determine if a DMA mapping should use the + * bus address, be mapped normally or fail + * @dev: device doing the DMA request + * @pgmap: dev_pagemap structure for the mapping + * + * Returns: + * 1 - if the page should be mapped with a bus address, + * 0 - if the page should be mapped normally through an IOMMU mapping or + * physical address; or + * -1 - if the device should not map the pages and an error should be + * returned + */ +int pci_p2pdma_dma_map_type(struct device *dev, struct dev_pagemap *pgmap) +{ + struct pci_p2pdma_pagemap *p2p_pgmap = to_p2p_pgmap(pgmap); + struct pci_dev *client; + + if (!dev_is_pci(dev)) + return -1; + + client = to_pci_dev(dev); + + switch (pci_p2pdma_map_type(p2p_pgmap->provider, client)) { + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + return 0; + case PCI_P2PDMA_MAP_BUS_ADDR: + return 1; + default: + return -1; + } +} +EXPORT_SYMBOL_GPL(pci_p2pdma_dma_map_type); + /** * pci_p2pdma_enable_store - parse a configfs/sysfs attribute store * to enable p2pdma diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h index 8318a97c9c61..3d55ad19f54c 100644 --- a/include/linux/pci-p2pdma.h +++ b/include/linux/pci-p2pdma.h @@ -34,6 +34,8 @@ int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs); void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs); +u64 pci_p2pdma_bus_offset(struct page *page); +int pci_p2pdma_dma_map_type(struct device *dev, struct dev_pagemap *pgmap); int pci_p2pdma_enable_store(const char *page, struct pci_dev **p2p_dev, bool *use_p2pdma); ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev, @@ -83,6 +85,15 @@ static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev, static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) { } +static inline u64 pci_p2pdma_bus_offset(struct page *page) +{ + return -1; +} +static inline int pci_p2pdma_dma_map_type(struct device *dev, + struct dev_pagemap *pgmap) +{ + return -1; +} static inline int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) From patchwork Thu Mar 11 23:31:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75E7FC43332 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 627AD64F7C for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231189AbhCKXs7 (ORCPT ); Thu, 11 Mar 2021 18:48:59 -0500 Received: from ale.deltatee.com ([204.191.154.188]:34142 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231224AbhCKXsi (ORCPT ); Thu, 11 Mar 2021 18:48:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=SoJ7QXcdMSNUWNuLgYtfThq+6IS5T78F0fYqiwZNrwI=; b=QIMuZxlg4vewqu/4hC9UxXtgJ0 o3uE3b2Nt1fxjOPIpxtlRszTvMXmqezV8xlxuXlUdnBrVDqJJ6ywk6oHVWuB7N6QbwUyvrauZn5tR Czn6BJRJ54+4CiKJjqMB+AUUc+4SV8kD13q7+6L1bQgUZdOQyNhmHLnvU4UEMOFa6d0AYVSr/r3OH zm7L3fx3qibd57BVLO4u82KL7qds64KW2ceFG8fqLRl4U00vjfuFApPj+bO2wDyf+RKaYb9lcqf4j pRS9wveW5+nbQBmQ6dRKJKhwWNECTH0VurPYT1q2Ew+itwn4+g7k33ZjhZ+4NnAfSBX9PK8DH1wrT H9so8qRw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUms-0003ev-BW; Thu, 11 Mar 2021 16:32:15 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024Y-E1; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:35 -0700 Message-Id: <20210311233142.7900-6-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 05/11] lib/scatterlist: Add flag for indicating P2PDMA segments in an SGL X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Make use of the third free LSB in scatterlist's page_link on 64bit systems. The extra bit will be used by dma_[un]map_sg() to determine when a given SGL segments dma_address points to a PCI bus address. dma_unmap_sg() will need to perform different cleanup when this is the case. Using this bit requires adding an additional dependency on CONFIG_64BIT to CONFIG_PCI_P2PDMA. This should be acceptable as the majority of P2PDMA use cases are restricted to newer root complexes and roughly require the extra address space for memory BARs used in the transactions. Signed-off-by: Logan Gunthorpe --- drivers/pci/Kconfig | 2 +- include/linux/scatterlist.h | 49 ++++++++++++++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig index 0c473d75e625..90b4bddb3300 100644 --- a/drivers/pci/Kconfig +++ b/drivers/pci/Kconfig @@ -163,7 +163,7 @@ config PCI_PASID config PCI_P2PDMA bool "PCI peer-to-peer transfer support" - depends on ZONE_DEVICE + depends on ZONE_DEVICE && 64BIT select GENERIC_ALLOCATOR help Enableѕ drivers to do PCI peer-to-peer transactions to and from diff --git a/include/linux/scatterlist.h b/include/linux/scatterlist.h index 6f70572b2938..5525d3ebf36f 100644 --- a/include/linux/scatterlist.h +++ b/include/linux/scatterlist.h @@ -58,6 +58,21 @@ struct sg_table { #define SG_CHAIN 0x01UL #define SG_END 0x02UL +/* + * bit 2 is the third free bit in the page_link on 64bit systems which + * is used by dma_unmap_sg() to determine if the dma_address is a PCI + * bus address when doing P2PDMA. + * Note: CONFIG_PCI_P2PDMA depends on CONFIG_64BIT because of this. + */ + +#ifdef CONFIG_PCI_P2PDMA +#define SG_PCI_P2PDMA 0x04UL +#else +#define SG_PCI_P2PDMA 0x00UL +#endif + +#define SG_PAGE_LINK_MASK (SG_CHAIN | SG_END | SG_PCI_P2PDMA) + /* * We overload the LSB of the page pointer to indicate whether it's * a valid sg entry, or whether it points to the start of a new scatterlist. @@ -65,8 +80,9 @@ struct sg_table { */ #define sg_is_chain(sg) ((sg)->page_link & SG_CHAIN) #define sg_is_last(sg) ((sg)->page_link & SG_END) +#define sg_is_pci_p2pdma(sg) ((sg)->page_link & SG_PCI_P2PDMA) #define sg_chain_ptr(sg) \ - ((struct scatterlist *) ((sg)->page_link & ~(SG_CHAIN | SG_END))) + ((struct scatterlist *) ((sg)->page_link & ~SG_PAGE_LINK_MASK)) /** * sg_assign_page - Assign a given page to an SG entry @@ -80,13 +96,13 @@ struct sg_table { **/ static inline void sg_assign_page(struct scatterlist *sg, struct page *page) { - unsigned long page_link = sg->page_link & (SG_CHAIN | SG_END); + unsigned long page_link = sg->page_link & SG_PAGE_LINK_MASK; /* * In order for the low bit stealing approach to work, pages * must be aligned at a 32-bit boundary as a minimum. */ - BUG_ON((unsigned long) page & (SG_CHAIN | SG_END)); + BUG_ON((unsigned long) page & SG_PAGE_LINK_MASK); #ifdef CONFIG_DEBUG_SG BUG_ON(sg_is_chain(sg)); #endif @@ -120,7 +136,7 @@ static inline struct page *sg_page(struct scatterlist *sg) #ifdef CONFIG_DEBUG_SG BUG_ON(sg_is_chain(sg)); #endif - return (struct page *)((sg)->page_link & ~(SG_CHAIN | SG_END)); + return (struct page *)((sg)->page_link & ~SG_PAGE_LINK_MASK); } /** @@ -222,6 +238,31 @@ static inline void sg_unmark_end(struct scatterlist *sg) sg->page_link &= ~SG_END; } +/** + * sg_mark_pci_p2pdma - Mark the scatterlist entry for PCI p2pdma + * @sg: SG entryScatterlist + * + * Description: + * Marks the passed in sg entry to indicate that the dma_address is + * a PCI bus address. + **/ +static inline void sg_mark_pci_p2pdma(struct scatterlist *sg) +{ + sg->page_link |= SG_PCI_P2PDMA; +} + +/** + * sg_unmark_pci_p2pdma - Unmark the scatterlist entry for PCI p2pdma + * @sg: SG entryScatterlist + * + * Description: + * Clears the PCI P2PDMA mark + **/ +static inline void sg_unmark_pci_p2pdma(struct scatterlist *sg) +{ + sg->page_link &= ~SG_PCI_P2PDMA; +} + /** * sg_phys - Return physical address of an sg entry * @sg: SG entry From patchwork Thu Mar 11 23:31:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F04AC4360C for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F32C364F7C for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231328AbhCKXtA (ORCPT ); Thu, 11 Mar 2021 18:49:00 -0500 Received: from ale.deltatee.com ([204.191.154.188]:45208 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbhCKXsp (ORCPT ); Thu, 11 Mar 2021 18:48:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=5m+PzWqUWnOR3YxadyZCg0MfweU6UruMUlqYZWYNYZM=; b=HV/kvTIN+l3Mjh5BJT+0f0JjD8 P6NqH+K5UqG4RHUSkRXS55CWxhlfN8E2lcAnezcnbPCxPgqjSDCeptqojK/JrP65f6TtoiSv+NLzp i6CTfLrDVHkXIL8Jm++pnpKOzRx64R6plr4lmBgo2J86KWpIa5RUob7/lMB/i+2tMsw3mKATtPvU7 k5xJoWpB3bmKp16E0VOC+RoakdJYP+xRQUbJZopKLN8frZh+KI0iyApQnWSi0ByDGcKuFpu1/qnh7 5Pa7/Ot9X9cYqQbjRcHQ+GxkViOGRZhbHsZ8W+7xZ3B0JN6TLomzW1Z4Z5QoUjYegFZVnsHaWnImv uJfSt0Bw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmp-0003ev-A4; Thu, 11 Mar 2021 16:32:12 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024b-Hp; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:36 -0700 Message-Id: <20210311233142.7900-7-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 06/11] dma-direct: Support PCI P2PDMA pages in dma-direct map_sg X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add PCI P2PDMA support for dma_direct_map_sg() so that it can map PCI P2PDMA pages directly without a hack in the callers. This allows for heterogeneous SGLs that contain both P2PDMA and regular pages. SGL segments that contain PCI bus addresses are marked with sg_mark_pci_p2pdma() and are ignored when unmapped. Signed-off-by: Logan Gunthorpe --- kernel/dma/direct.c | 35 ++++++++++++++++++++++++++++++++--- kernel/dma/mapping.c | 13 ++++++++++--- 2 files changed, 42 insertions(+), 6 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 002268262c9a..f326d32062dd 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "direct.h" /* @@ -387,19 +388,47 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, struct scatterlist *sg; int i; - for_each_sg(sgl, sg, nents, i) + for_each_sg(sgl, sg, nents, i) { + if (sg_is_pci_p2pdma(sg)) + continue; + dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, attrs); + } } #endif int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { - int i; + struct dev_pagemap *pgmap = NULL; + int i, map = -1, ret = 0; struct scatterlist *sg; + u64 bus_off; for_each_sg(sgl, sg, nents, i) { + if (is_pci_p2pdma_page(sg_page(sg))) { + if (sg_page(sg)->pgmap != pgmap) { + pgmap = sg_page(sg)->pgmap; + map = pci_p2pdma_dma_map_type(dev, pgmap); + bus_off = pci_p2pdma_bus_offset(sg_page(sg)); + } + + if (map < 0) { + sg->dma_address = DMA_MAPPING_ERROR; + ret = -EREMOTEIO; + goto out_unmap; + } + + if (map) { + sg->dma_address = sg_phys(sg) + sg->offset - + bus_off; + sg_dma_len(sg) = sg->length; + sg_mark_pci_p2pdma(sg); + continue; + } + } + sg->dma_address = dma_direct_map_page(dev, sg_page(sg), sg->offset, sg->length, dir, attrs); if (sg->dma_address == DMA_MAPPING_ERROR) @@ -411,7 +440,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, out_unmap: dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); - return 0; + return ret; } dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index b6a633679933..adc1a83950be 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -178,8 +178,15 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, EXPORT_SYMBOL(dma_unmap_page_attrs); /* - * dma_maps_sg_attrs returns 0 on error and > 0 on success. - * It should never return a value < 0. + * dma_maps_sg_attrs returns 0 on any resource error and > 0 on success. + * + * If 0 is returned, the mapping can be retried and will succeed once + * sufficient resources are available. + * + * If there are P2PDMA pages in the scatterlist then this function may + * return -EREMOTEIO to indicate that the pages are not mappable by the + * device. In this case, an error should be returned for the IO as it + * will never be successfully retried. */ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -197,7 +204,7 @@ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); else ents = ops->map_sg(dev, sg, nents, dir, attrs); - BUG_ON(ents < 0); + debug_dma_map_sg(dev, sg, nents, ents, dir); return ents; From patchwork Thu Mar 11 23:31:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 653A8C4332D for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 52CE864F93 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231181AbhCKXs6 (ORCPT ); Thu, 11 Mar 2021 18:48:58 -0500 Received: from ale.deltatee.com ([204.191.154.188]:58788 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229606AbhCKXsg (ORCPT ); Thu, 11 Mar 2021 18:48:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=KOMMXadwrYwZEB6tWYvcMS2uWikbMkq234oR8x5l6uU=; b=NOigEiKa5QVhqkJx08sL74oozH IZ3CiU8+eeNi0GWrmCuxBvytyQml8bVvak4gI55pcE2jXr1VSHIoSsjrWGiFBqneKdlc/tREjEfQK vpBhsUQYflR8mj4rh+OgbyUcpWwHvWJro43CV02zDTEMELz51KCyW4YJ1HEQ5aT++Wx4fWd0e2UIz 1Fdm889jVbolaeVeZCD1GDvmPKyUJzpB3kfWPtbn3sTqKbzNUDsxzF+bTniILfxZ6v1Lj6Izgqeej bn38QX113rFGYQkNk83OpSUc66XivY5TKSYhkyr9Yr5OB+kx3ZkOj25E4nbNn9vcglbG8mk4sRFr+ 74XDjSSA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmm-0003ev-8M; Thu, 11 Mar 2021 16:32:09 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024e-Lc; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:37 -0700 Message-Id: <20210311233142.7900-8-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 07/11] dma-mapping: Add flags to dma_map_ops to indicate PCI P2PDMA support X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Add a flags member to the dma_map_ops structure with one flag to indicate support for PCI P2PDMA. Also, add a helper to check if a device supports PCI P2PDMA. Signed-off-by: Logan Gunthorpe --- include/linux/dma-map-ops.h | 3 +++ include/linux/dma-mapping.h | 5 +++++ kernel/dma/mapping.c | 8 ++++++++ 3 files changed, 16 insertions(+) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 51872e736e7b..481892822104 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -12,6 +12,9 @@ struct cma; struct dma_map_ops { + unsigned int flags; +#define DMA_F_PCI_P2PDMA_SUPPORTED (1 << 0) + void *(*alloc)(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs); diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 2a984cb4d1e0..a6bced004de8 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -138,6 +138,7 @@ int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma, unsigned long attrs); bool dma_can_mmap(struct device *dev); int dma_supported(struct device *dev, u64 mask); +int dma_pci_p2pdma_supported(struct device *dev); int dma_set_mask(struct device *dev, u64 mask); int dma_set_coherent_mask(struct device *dev, u64 mask); u64 dma_get_required_mask(struct device *dev); @@ -233,6 +234,10 @@ static inline int dma_supported(struct device *dev, u64 mask) { return 0; } +static inline int dma_pci_p2pdma_supported(struct device *dev) +{ + return 0; +} static inline int dma_set_mask(struct device *dev, u64 mask) { return -EIO; diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index adc1a83950be..cc1b97deb74d 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -540,6 +540,14 @@ int dma_supported(struct device *dev, u64 mask) } EXPORT_SYMBOL(dma_supported); +int dma_pci_p2pdma_supported(struct device *dev) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + return !ops || ops->flags & DMA_F_PCI_P2PDMA_SUPPORTED; +} +EXPORT_SYMBOL(dma_pci_p2pdma_supported); + #ifdef CONFIG_ARCH_HAS_DMA_SET_MASK void arch_dma_set_mask(struct device *dev, u64 mask); #else From patchwork Thu Mar 11 23:31:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43C7EC43381 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3618364FAD for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230510AbhCKXs6 (ORCPT ); Thu, 11 Mar 2021 18:48:58 -0500 Received: from ale.deltatee.com ([204.191.154.188]:54984 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231151AbhCKXse (ORCPT ); Thu, 11 Mar 2021 18:48:34 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=C2zq0RJZ8ThW76MS1rjASAKoSeACRy/BMOxwbAjwe44=; b=YPe2kj6qoSd9nU97o+42pKhQUP y2qAxVsNW2xg08F86j2k6fHSw0JIOBKt4vGOF1xJyulq1SMyEeFb60M/SBtozVsp6vNvoHttPK4Jw stSan4zrYF1Ii9fS92AnH6ab6+Zlti52V3HTZJq0iugJ7W0LlXxqsLKRqo/DgqwooMi3LeDt4gNY6 ufhUPy2A4fkCCcIY2K9L34KNFFSsp5ihRrCUTfpxCmBgTFAYdWT9zTcpAmR2+wb0ShSoEurwgjYDD X4yJDBXX6ammjq8UJF9nT3cxQtZ2UcZO6QhW8YJEIwejEaDqTvIAYlKrAR0740Gv2opckyqN6QAbg quJBdmpg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmi-0003ev-Lc; Thu, 11 Mar 2021 16:32:06 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmW-00024h-SB; Thu, 11 Mar 2021 16:31:52 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:38 -0700 Message-Id: <20210311233142.7900-9-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 08/11] iommu/dma: Support PCI P2PDMA pages in dma-iommu map_sg X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When a PCI P2PDMA page is seen, set the IOVA length of the segment to zero so that it is not mapped into the IOVA. Then, in finalise_sg(), apply the appropriate bus address to the segment. The IOVA is not created if the scatterlist only consists of P2PDMA pages. Similar to dma-direct, the sg_mark_pci_p2pdma() flag is used to indicate bus address segments. On unmap, P2PDMA segments are skipped over when determining the start and end IOVA addresses. With this change, the flags variable in the dma_map_ops is set to DMA_F_PCI_P2PDMA_SUPPORTED to indicate support for P2PDMA pages. Signed-off-by: Logan Gunthorpe --- drivers/iommu/dma-iommu.c | 63 ++++++++++++++++++++++++++++++++------- 1 file changed, 53 insertions(+), 10 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index af765c813cc8..c0821e9051a9 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -846,7 +847,7 @@ static void iommu_dma_unmap_page(struct device *dev, dma_addr_t dma_handle, * segment's start address to avoid concatenating across one. */ static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents, - dma_addr_t dma_addr) + dma_addr_t dma_addr, unsigned long attrs) { struct scatterlist *s, *cur = sg; unsigned long seg_mask = dma_get_seg_boundary(dev); @@ -864,6 +865,20 @@ static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents, sg_dma_address(s) = DMA_MAPPING_ERROR; sg_dma_len(s) = 0; + if (is_pci_p2pdma_page(sg_page(s)) && !s_iova_len) { + if (i > 0) + cur = sg_next(cur); + + sg_dma_address(cur) = sg_phys(s) + s->offset - + pci_p2pdma_bus_offset(sg_page(s)); + sg_dma_len(cur) = s->length; + sg_mark_pci_p2pdma(cur); + + count++; + cur_len = 0; + continue; + } + /* * Now fill in the real DMA data. If... * - there is a valid output segment to append to @@ -960,11 +975,12 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, struct iommu_dma_cookie *cookie = domain->iova_cookie; struct iova_domain *iovad = &cookie->iovad; struct scatterlist *s, *prev = NULL; + struct dev_pagemap *pgmap = NULL; int prot = dma_info_to_prot(dir, dev_is_dma_coherent(dev), attrs); dma_addr_t iova; size_t iova_len = 0; unsigned long mask = dma_get_seg_boundary(dev); - int i; + int i, map = -1, ret = 0; if (static_branch_unlikely(&iommu_deferred_attach_enabled) && iommu_deferred_attach(dev, domain)) @@ -993,6 +1009,23 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, s_length = iova_align(iovad, s_length + s_iova_off); s->length = s_length; + if (is_pci_p2pdma_page(sg_page(s))) { + if (sg_page(s)->pgmap != pgmap) { + pgmap = sg_page(s)->pgmap; + map = pci_p2pdma_dma_map_type(dev, pgmap); + } + + if (map < 0) { + ret = -EREMOTEIO; + goto out_restore_sg; + } + + if (map) { + s->length = 0; + continue; + } + } + /* * Due to the alignment of our single IOVA allocation, we can * depend on these assumptions about the segment boundary mask: @@ -1015,6 +1048,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, prev = s; } + if (!iova_len) + return __finalise_sg(dev, sg, nents, 0, attrs); + iova = iommu_dma_alloc_iova(domain, iova_len, dma_get_mask(dev), dev); if (!iova) goto out_restore_sg; @@ -1026,19 +1062,19 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) goto out_free_iova; - return __finalise_sg(dev, sg, nents, iova); + return __finalise_sg(dev, sg, nents, iova, attrs); out_free_iova: iommu_dma_free_iova(cookie, iova, iova_len, NULL); out_restore_sg: __invalidate_sg(sg, nents); - return 0; + return ret; } static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, enum dma_data_direction dir, unsigned long attrs) { - dma_addr_t start, end; + dma_addr_t end, start = DMA_MAPPING_ERROR; struct scatterlist *tmp; int i; @@ -1054,14 +1090,20 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, * The scatterlist segments are mapped into a single * contiguous IOVA allocation, so this is incredibly easy. */ - start = sg_dma_address(sg); - for_each_sg(sg_next(sg), tmp, nents - 1, i) { + for_each_sg(sg, tmp, nents, i) { + if (sg_is_pci_p2pdma(tmp)) + continue; if (sg_dma_len(tmp) == 0) break; - sg = tmp; + + if (start == DMA_MAPPING_ERROR) + start = sg_dma_address(tmp); + + end = sg_dma_address(tmp) + sg_dma_len(tmp); } - end = sg_dma_address(sg) + sg_dma_len(sg); - __iommu_dma_unmap(dev, start, end - start); + + if (start != DMA_MAPPING_ERROR) + __iommu_dma_unmap(dev, start, end - start); } static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, @@ -1254,6 +1296,7 @@ static unsigned long iommu_dma_get_merge_boundary(struct device *dev) } static const struct dma_map_ops iommu_dma_ops = { + .flags = DMA_F_PCI_P2PDMA_SUPPORTED, .alloc = iommu_dma_alloc, .free = iommu_dma_free, .alloc_pages = dma_common_alloc_pages, From patchwork Thu Mar 11 23:31:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C84AC43619 for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1BD5764FC0 for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231355AbhCKXtC (ORCPT ); Thu, 11 Mar 2021 18:49:02 -0500 Received: from ale.deltatee.com ([204.191.154.188]:49292 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231248AbhCKXss (ORCPT ); Thu, 11 Mar 2021 18:48:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=uc5bXdFjfzszbviuSRD0/nYF6yd83ZtY9+zFkxPv3to=; b=csakyDP9QELnHiJJt/bc3YHT5J MoHtqOwNirvsD4AKK/wAoHQc1OcPqyxDLj6E6XbXXUlFNW8j0LFHY9i4+0dHbeSg2ntbrL+pi1i1j eZgg3PZ5kKt2v2KRZww13squhr+psIVilPV8ssOeCYlImV6asqMxmsdVVlevyQ2g4RTUO3Gs5m962 z2F1pS21nxt6uZzoVlb+9gSM1CZY/0Xh44C+NtzUIfHZPsMv/U0L/phcr8dTwoMmZ7ngxqPQCkfhn I5tPDsWoFxgDPV9tg1liF/lEyAEWDpXgxSWtEnSpoGLd0s/0dx4NQ/GOD4GT+AT6+UVa8Dq2T5t5L 11sJaiZw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmf-0003ev-OY; Thu, 11 Mar 2021 16:32:02 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmX-00024k-0x; Thu, 11 Mar 2021 16:31:53 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:39 -0700 Message-Id: <20210311233142.7900-10-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 09/11] block: Add BLK_STS_P2PDMA X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Create a specific error code for when P2PDMA pages are passed to a block devices that cannot map them (due to no IOMMU support or ACS protections). This makes request errors in these cases more informative of as to what caused the error. Signed-off-by: Logan Gunthorpe --- block/blk-core.c | 2 ++ include/linux/blk_types.h | 7 +++++++ 2 files changed, 9 insertions(+) diff --git a/block/blk-core.c b/block/blk-core.c index fc60ff208497..2cc75b56ac43 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -192,6 +192,8 @@ static const struct { [BLK_STS_ZONE_OPEN_RESOURCE] = { -ETOOMANYREFS, "open zones exceeded" }, [BLK_STS_ZONE_ACTIVE_RESOURCE] = { -EOVERFLOW, "active zones exceeded" }, + [BLK_STS_P2PDMA] = { -EREMOTEIO, "P2PDMA to invalid device" }, + /* everything else not covered above: */ [BLK_STS_IOERR] = { -EIO, "I/O" }, }; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index db026b6ec15a..728a0898cb34 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -142,6 +142,13 @@ typedef u8 __bitwise blk_status_t; */ #define BLK_STS_ZONE_ACTIVE_RESOURCE ((__force blk_status_t)16) +/* + * BLK_STS_P2PDMA is returned from the driver if P2PDMA memory fails to be + * mapped to the target device. This is a permanent error and the request + * should not be retried. + */ +#define BLK_STS_P2PDMA ((__force blk_status_t)17) + /** * blk_path_error - returns true if error may be path related * @error: status the request was completed with From patchwork Thu Mar 11 23:31:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 076D5C433E6 for ; Thu, 11 Mar 2021 23:49:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CFC5864F92 for ; Thu, 11 Mar 2021 23:49:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231294AbhCKXs5 (ORCPT ); Thu, 11 Mar 2021 18:48:57 -0500 Received: from ale.deltatee.com ([204.191.154.188]:49304 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230431AbhCKXsa (ORCPT ); Thu, 11 Mar 2021 18:48:30 -0500 X-Greylist: delayed 993 seconds by postgrey-1.27 at vger.kernel.org; Thu, 11 Mar 2021 18:48:30 EST DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=f6LQJfU8fKCFP6wYnRzgrCuu/xZDkoPmYFwLlI2G6Nc=; b=UuacCDLP+aPTN2kI5xuW2pW3NX wRncdycKWjJFExz+mnxROhQcpVF1gXWZhV65ZpSskTow6KMIgw/J9qqiVNiXAjQdpltZMGKdnPe3H +KW8XVmsTYTyf3+Hj6JYjWScsspdcpnvZqyKFSJQvBI9tt8Z6SpVIiSexUipSeRh8wbTHlnzMTjtp vpuhjUwovWzMn0zaN27WBYnRJe/uB4je0UzXq07sgXaQZ5A2eYRLhIsAvyYUxtC1CQhuVffpwLNPR koYdqe1tUBRPQP259Myn8zm+sN7hQU7gFB42c4U/NZxiahyacl3fCm9wlOCdBL770WTxjTS/WYfXV Q0RBcAjg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmd-0003et-Gh; Thu, 11 Mar 2021 16:32:15 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmX-00024n-6O; Thu, 11 Mar 2021 16:31:53 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:40 -0700 Message-Id: <20210311233142.7900-11-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 10/11] nvme-pci: Check DMA ops when indicating support for PCI P2PDMA X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Introduce a supports_pci_p2pdma() operation in nvme_ctrl_ops to replace the fixed NVME_F_PCI_P2PDMA flag such that the dma_map_ops flags can be checked for PCI P2PDMA support. Signed-off-by: Logan Gunthorpe --- drivers/nvme/host/core.c | 3 ++- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 11 +++++++++-- 3 files changed, 12 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index e68a8c4ac5a6..501b4a979776 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -3928,7 +3928,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid, blk_queue_flag_set(QUEUE_FLAG_STABLE_WRITES, ns->queue); blk_queue_flag_set(QUEUE_FLAG_NONROT, ns->queue); - if (ctrl->ops->flags & NVME_F_PCI_P2PDMA) + if (ctrl->ops->supports_pci_p2pdma && + ctrl->ops->supports_pci_p2pdma(ctrl)) blk_queue_flag_set(QUEUE_FLAG_PCI_P2PDMA, ns->queue); ns->queue->queuedata = ns; diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 07b34175c6ce..9c04df982d2c 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -473,7 +473,6 @@ struct nvme_ctrl_ops { unsigned int flags; #define NVME_F_FABRICS (1 << 0) #define NVME_F_METADATA_SUPPORTED (1 << 1) -#define NVME_F_PCI_P2PDMA (1 << 2) int (*reg_read32)(struct nvme_ctrl *ctrl, u32 off, u32 *val); int (*reg_write32)(struct nvme_ctrl *ctrl, u32 off, u32 val); int (*reg_read64)(struct nvme_ctrl *ctrl, u32 off, u64 *val); @@ -481,6 +480,7 @@ struct nvme_ctrl_ops { void (*submit_async_event)(struct nvme_ctrl *ctrl); void (*delete_ctrl)(struct nvme_ctrl *ctrl); int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size); + bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); }; #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 17ab3320d28b..7d40c6a9e58e 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2759,17 +2759,24 @@ static int nvme_pci_get_address(struct nvme_ctrl *ctrl, char *buf, int size) return snprintf(buf, size, "%s\n", dev_name(&pdev->dev)); } +static bool nvme_pci_supports_pci_p2pdma(struct nvme_ctrl *ctrl) +{ + struct nvme_dev *dev = to_nvme_dev(ctrl); + + return dma_pci_p2pdma_supported(dev->dev); +} + static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name = "pcie", .module = THIS_MODULE, - .flags = NVME_F_METADATA_SUPPORTED | - NVME_F_PCI_P2PDMA, + .flags = NVME_F_METADATA_SUPPORTED, .reg_read32 = nvme_pci_reg_read32, .reg_write32 = nvme_pci_reg_write32, .reg_read64 = nvme_pci_reg_read64, .free_ctrl = nvme_pci_free_ctrl, .submit_async_event = nvme_pci_submit_async_event, .get_address = nvme_pci_get_address, + .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma, }; static int nvme_dev_map(struct nvme_dev *dev) From patchwork Thu Mar 11 23:31:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Logan Gunthorpe X-Patchwork-Id: 12133315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEB27C28E88 for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DEB9E64F92 for ; Thu, 11 Mar 2021 23:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231398AbhCKXtE (ORCPT ); Thu, 11 Mar 2021 18:49:04 -0500 Received: from ale.deltatee.com ([204.191.154.188]:59150 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231282AbhCKXs5 (ORCPT ); Thu, 11 Mar 2021 18:48:57 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=RWXwANJSi1YHE9SXh/vkrbb+LuWGD2o8fdtHIWP0im0=; b=Il/909HX1ReIjuH/DQQRUWCbGJ rwqplq8j7zCFjrGAFApqpwgRF0WhSKa9dp79xbHuG02pdcKARnuTLWP8goViDVydS9Ev8lyIlPv+v FWWJIDnnyUqnBSoUN04RvsjWs4OiwrwcMQ/jN0KFAnN4q4YB0a/DHhu/OFHMBrYafzzhePOaTJx8n 2ttD1xTgCwB4llOsN1PhW9D9Lo8U3/AVemwfu3FBTgqS+hyTD65gStZo+XmcrTIKv3Z7PLjhOCfAu sC6+8+PT545jn/sNo86PM5hWX1sYA+iztBB/qyFL2L83oj5ucCaODvzgnGwZIjG/qNX+9ZIi+hIoR ZYvSGKuw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKUmc-0003ev-TB; Thu, 11 Mar 2021 16:31:59 -0700 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.92) (envelope-from ) id 1lKUmX-00024q-BH; Thu, 11 Mar 2021 16:31:53 -0700 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: Stephen Bates , Christoph Hellwig , Dan Williams , Jason Gunthorpe , =?utf-8?q?Christian_K=C3=B6nig?= , Ira Weiny , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Logan Gunthorpe Date: Thu, 11 Mar 2021 16:31:41 -0700 Message-Id: <20210311233142.7900-12-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20210311233142.7900-1-logang@deltatee.com> References: <20210311233142.7900-1-logang@deltatee.com> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, sbates@raithlin.com, hch@lst.de, jgg@ziepe.ca, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, jason@jlekstrand.net, dave.hansen@linux.intel.com, dan.j.williams@intel.com, iweiny@intel.com, andrzej.jakowski@intel.com, dave.b.minturn@intel.com, jianxin.xiong@intel.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [RFC PATCH v2 11/11] nvme-pci: Convert to using dma_map_sg for p2pdma pages X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Convert to using dma_[un]map_sg() for PCI p2pdma pages. This should be equivalent, though support will be somewhat less (only dma-direct and dma-iommu are currently supported). Signed-off-by: Logan Gunthorpe --- drivers/nvme/host/pci.c | 27 +++++++-------------------- 1 file changed, 7 insertions(+), 20 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7d40c6a9e58e..89ca5acf7a62 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -577,17 +577,6 @@ static void nvme_free_sgls(struct nvme_dev *dev, struct request *req) } -static void nvme_unmap_sg(struct nvme_dev *dev, struct request *req) -{ - struct nvme_iod *iod = blk_mq_rq_to_pdu(req); - - if (is_pci_p2pdma_page(sg_page(iod->sg))) - pci_p2pdma_unmap_sg(dev->dev, iod->sg, iod->nents, - rq_dma_dir(req)); - else - dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req)); -} - static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); @@ -600,7 +589,7 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) WARN_ON_ONCE(!iod->nents); - nvme_unmap_sg(dev, req); + dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req)); if (iod->npages == 0) dma_pool_free(dev->prp_small_pool, nvme_pci_iod_list(req)[0], iod->first_dma); @@ -868,13 +857,11 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, if (!iod->nents) goto out_free_sg; - if (is_pci_p2pdma_page(sg_page(iod->sg))) - nr_mapped = pci_p2pdma_map_sg_attrs(dev->dev, iod->sg, - iod->nents, rq_dma_dir(req), DMA_ATTR_NO_WARN); - else - nr_mapped = dma_map_sg_attrs(dev->dev, iod->sg, iod->nents, - rq_dma_dir(req), DMA_ATTR_NO_WARN); - if (!nr_mapped) + nr_mapped = dma_map_sg_attrs(dev->dev, iod->sg, iod->nents, + rq_dma_dir(req), DMA_ATTR_NO_WARN); + if (nr_mapped < 0) + ret = BLK_STS_P2PDMA; + if (nr_mapped <= 0) goto out_free_sg; iod->use_sgl = nvme_pci_use_sgls(dev, req); @@ -887,7 +874,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, return BLK_STS_OK; out_unmap_sg: - nvme_unmap_sg(dev, req); + dma_unmap_sg(dev->dev, iod->sg, iod->nents, rq_dma_dir(req)); out_free_sg: mempool_free(iod->sg, dev->iod_mempool); return ret;