From patchwork Sun Dec 1 10:36:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13889462 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2057.outbound.protection.outlook.com [40.107.223.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60CED3F9FB; Sun, 1 Dec 2024 10:38:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.223.57 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049491; cv=fail; b=cUC4nfDzPP9EbK5DActppwowgEsLeMVgpcoxS2nFAQ9PcEQ4uKgoeAoaAYjryBS1JWylI3036sMGpR/NNmm/Lsge4SZXWssvGPS2qPAucNh8zXrcxFnIBwRM4TH9tCFkHuh9yFRoo8WTw5BVR3LtLA2lFGk5yJ5+1IhwhF4AzkA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049491; c=relaxed/simple; bh=ZfgNVsbYhloAnKXFz+ZaS8A+gn9i3mdbOafRFlloRIM=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Q/AJPmd4VBleMiZ/OAf07dG431BoBU1ukrMOxMFCtOxs8WLxTRAVZN+I2tklWyvR95CspDePYKejtm4yfOzcaQku4kxC8N5D+gZX6Wd0boY3I5nAk37aLgQ1FGj9IzxhBAQE5QBb96PadZccGEGRXl8J2k+5tXMbpIGqILnYEWA= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=N7SiPhOq; arc=fail smtp.client-ip=40.107.223.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="N7SiPhOq" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=mRMVJP0/YpmwIaGnF+JgB354+0zk8ywhrDB+Xn6HDZ6q7v8hGPr0WIssnOB1Z5DGqVMeRVOjetsyPHxybQuEkuHpEtiT5SoRcMW/H8qMUAdKJfszdGahWJ94sNzV051NY+B5PSzg3Uj8yP3oEQoyc/BD2t03PKsNdCsjE1VYPqAhpXUvDeVryE+7n7oWDWCf3+9x5OMc/rss/EvsPZ6wLWIzOJlYCztJFnNSyJmjKqaaUaz4ta9HjqEL3no5juKcYyHtHkhNzVZRN+ATfA0TOvIguf5sx9N4ir/e9Fs4Zv+8LVuKy/DKsJ3PJn0zy+v6VVM+FubYYWltCRX8CSAWFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BMhDRgrewIAka6a+gUfJ5hQiGVZI7odoh414sahOfLM=; b=Hnr/bRUMhZ0peeRG4FiFOpR64wpkmHNFkcdRg+B07M9l/H5Gm4KTZuVyma6EQQqhwRIblyVOjgefTTVPaUXBYWBgLB4SXEadQeN1pwKBiC8dEtKvpjxkRs/Os6V9ejz4YXvkHuzvKc1Ob9CMFtjKxDtT2N5oqwkVPkZl3u2h0d1pjJljiX23qOFIP9zLJ07qwW2Ql+LBFbEy9ybafUWJ7/SFWY3lAC1CHoLjlKFeO4LPShEeDGmf8BeRK2SLy/0X9oN/u2TVYrh7CN5YD4cCLd/cVl+0/lKr5ciIP/0uJWtj9MgvS7oQHUCcu9JGkzpVHVXTfikW/b6mWLE2Nih63g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BMhDRgrewIAka6a+gUfJ5hQiGVZI7odoh414sahOfLM=; b=N7SiPhOqMSEBmhkK/bSx+PA+x7Ldm3VhimuGgJrp7LsRlIOp+2yTFEhlHgRj/jPjz9RvbMDgH4JI2joclw6QSpAOn/RZT37NlSaPI7ZOHKbIZqKXk6T33HAHirjOBqgWkU0FtHG7orXznNYhiGX9EBrXbS6TM/e464Q4VUCtXDNo3LLu8EUH78UxKHkUj/BBkvnNDdelsHzgRZi6ZFqvqeaWsG6IQmxO9uSEd6hmMBfX3X6RUempFtMVkrWvAmxUBjxVCu5iIDYV9eB4dF0Zpj2rHszO/VIOF8Snl6FAeI0MZ7lQbbQSX7ExYBCdfriCwr3/Or93JIZpPeKyu7APMQ== Received: from CH0PR03CA0242.namprd03.prod.outlook.com (2603:10b6:610:e5::7) by PH7PR12MB5757.namprd12.prod.outlook.com (2603:10b6:510:1d0::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 10:38:02 +0000 Received: from DS3PEPF0000C37D.namprd04.prod.outlook.com (2603:10b6:610:e5:cafe::9d) by CH0PR03CA0242.outlook.office365.com (2603:10b6:610:e5::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.17 via Frontend Transport; Sun, 1 Dec 2024 10:38:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37D.mail.protection.outlook.com (10.167.23.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 10:38:01 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:37:53 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:37:52 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 1 Dec 2024 02:37:48 -0800 From: Yonatan Maman To: , , , , , , , , , , , , , , , , Subject: [RFC 1/5] mm/hmm: HMM API to enable P2P DMA for device private pages Date: Sun, 1 Dec 2024 12:36:55 +0200 Message-ID: <20241201103659.420677-2-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201103659.420677-1-ymaman@nvidia.com> References: <20241201103659.420677-1-ymaman@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37D:EE_|PH7PR12MB5757:EE_ X-MS-Office365-Filtering-Correlation-Id: 4cd84002-9361-4640-34df-08dd11f43b4c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|36860700013|82310400026|921020; X-Microsoft-Antispam-Message-Info: bG86dYyH+RIyb6NlGzVA1kni4MXz4cBhWM7ilkAVWA0DRuub6Pi9LF8iS8o1rzusBU0YBecO6ODgVrJufP1urSoamG6EuPOvelC18LOwBmfflcK93sPr+vOTFc9/Y/YHUKfYjf3MR74Ip1r/e+ecGkOR5ys65gCVinkHpSWn5pr379stLqB3rbY30BpXv20/JVzAqNK1ZOi70UZ/wn+DUvmQe2W55TaKBdvsRervjq6+ltACcEze5xk+ZMesxdV8BJkau6P9aibit60LeIGNCi2gdb7FIGIlaRMXvJo74V5XKFliLDLUo1yWsk6HdSVumZqA7p69F6nYv7Ncb5ZFcv2NB/gRblAL6qA57JkSmreMFiI0BMaSw0ms/F8x0rMymisGTU1qOPsB2DwN58N/LA6gjVgXFF7wsiozNFRClHVLOuMg9H1sNhR2t08Egv7I6FpcjHiHWYpMeDoW6OZxbgske+wAxppIdjWpgMaiIAGTmOPf1hYVq9JHK8ETAxWSfl2MayXa54qXi5NTycy+Rk9ZwMs/ZKc5cJ/w/GHANOIIDvzf4rIaKzUBukJ/H61Z79yjlszq3qfemm3bFQE3t6azT046Tc7b54uMC+fCtOTr8Cx+kLcTuu4a+80nzXFwwHkylShSWVuILhaELNHdz5ThNtNaA+UumhOgVyGeq1APw+v17o2fRpzwwts3dGOPcyJPbuUXiWJuX6VGMyHnZTZ//yp8CRuMv9EwlGsc5hhpqohLuQJPT/Ywc/IUTWjbiVeH1GghKZheC5V8ncmyr164x8jch5c0keQxop9nZHC/4tUW/eEkXWw2QoAT1LroTim2zHh6VW17RJBLArNOJUNtExT+m0zhk25pYKmhitTDJLbhouXyfwWMWVXkNLpr0J1ZKdW69FnioQt8kv/+m4lPhBEdCZqe81wtsmEQMp3hUJBTt0+rFbATek7vrb/vBQfuZ2a2EZzbbnPFxkEpUX/+pIjUtEd9cEOPVG3ErrGkLHzbQoQgci9bN4kM7/Px47lKlX2VG/880boI5rVdCGjveNZoj+87bBRIFywZFZGgOPrDgcjn1OnTjQmgS34TTBt4UwE1zJr6WQgA9tVkKQkdAr1y+xzADwVQRboY98/qL/+Kmo3w0My5RLn5jA4Cx3wUsu0Pa1BtAQ67ahkwlNz8P5BQrIVHm/4zVR69PkNu53ly4xU3Y4TajhOtpuZDP51rLnujBiQORclU+6VXT935XUM0bhx8ZH/VTnW7P2ezl7uJFmC67FcfbZM8B8Q7RUBTIIDDTc/a/7GxfE7PywvCkJrvT9lLsq1mfoWQcg14irf9ECft+snHqGh+XvhZu98wnLHVvrgX1d/t2DYQ6RMNl4f5HYhdjGYMLn4mVRCANLYBjpD6JV6zbRlUiM7/7aJxzETTVXCVPbBkA03yA4nQ8SMfCN/1xyMyjK7NsvW02NO9ZPL/NmtoUfZPXItgnB86CGeW4GE0crn8fxgXxg== X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(36860700013)(82310400026)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 10:38:01.9770 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4cd84002-9361-4640-34df-08dd11f43b4c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37D.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5757 From: Yonatan Maman hmm_range_fault() by default triggered a page fault on device private when HMM_PFN_REQ_FAULT flag was set. pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly degrades performance. Thus, enabling Peer-to-Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead. Introduced an API to support P2P DMA for device private pages,includes: - Leveraging the struct pagemap_ops for P2P Page Callbacks. This callback involves mapping the page for P2P DMA and returning the corresponding PCI_P2P page. - Utilizing hmm_range_fault for initializing P2P DMA. The API also adds the HMM_PFN_REQ_TRY_P2P flag option for the hmm_range_fault caller to initialize P2P. If set, hmm_range_fault attempts initializing the P2P connection first, if the owner device supports P2P, using p2p_page. In case of failure or lack of support, hmm_range_fault will continue with the regular flow of migrating the page to RAM. This change does not affect previous use-cases of hmm_range_fault, because both the caller and the page owner must explicitly request and support it to initialize P2P connection. Signed-off-by: Yonatan Maman Signed-off-by: Gal Shalom --- include/linux/hmm.h | 3 ++- include/linux/memremap.h | 8 ++++++ mm/hmm.c | 57 +++++++++++++++++++++++++++++++++------- 3 files changed, 57 insertions(+), 11 deletions(-) if (!pte_present(pte)) { swp_entry_t entry = pte_to_swp_entry(pte); - /* - * Don't fault in device private pages owned by the caller, - * just report the PFN. - */ if (is_device_private_entry(entry) && - pfn_swap_entry_to_page(entry)->pgmap->owner == - range->dev_private_owner) { - cpu_flags = HMM_PFN_VALID; - if (is_writable_device_private_entry(entry)) - cpu_flags |= HMM_PFN_WRITE; - *hmm_pfn = (*hmm_pfn & HMM_PFN_DMA_MAPPED) | swp_offset_pfn(entry) | cpu_flags; + hmm_handle_device_private(range, pfn_req_flags, entry, hmm_pfn)) { + *hmm_pfn = *hmm_pfn & HMM_PFN_DMA_MAPPED; return 0; } diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 62980ca8f3c5..017f22cef893 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -26,6 +26,7 @@ struct mmu_interval_notifier; * HMM_PFN_DMA_MAPPED - Flag preserved on input-to-output transformation * to mark that page is already DMA mapped + * HMM_PFN_ALLOW_P2P - Allow returning PCI P2PDMA page * * On input: * 0 - Return the current state of the page, do not fault it. @@ -41,7 +42,7 @@ enum hmm_pfn_flags { HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3), /* Sticky flag, carried from Input to Output */ + HMM_PFN_ALLOW_P2P = 1UL << (BITS_PER_LONG - 6), HMM_PFN_DMA_MAPPED = 1UL << (BITS_PER_LONG - 7), HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8), diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 3f7143ade32c..cdf5189be5e9 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -89,6 +89,14 @@ struct dev_pagemap_ops { */ vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf); + /* + * Used for private (un-addressable) device memory only. Return a + * corresponding PFN for a page that can be mapped to device + * (e.g using dma_map_page) + */ + int (*get_dma_pfn_for_device)(struct page *private_page, + unsigned long *dma_pfn); + /* * Handle the memory failure happens on a range of pfns. Notify the * processes who are using these pfns, and try to recover the data on diff --git a/mm/hmm.c b/mm/hmm.c index a852d8337c73..1c080bc00ee8 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -226,6 +226,51 @@ static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range, return pte_write(pte) ? (HMM_PFN_VALID | HMM_PFN_WRITE) : HMM_PFN_VALID; } +static bool hmm_handle_device_private(struct hmm_range *range, + unsigned long pfn_req_flags, + swp_entry_t entry, + unsigned long *hmm_pfn) +{ + struct page *page = pfn_swap_entry_to_page(entry); + struct dev_pagemap *pgmap = page->pgmap; + int ret; + pfn_req_flags &= range->pfn_flags_mask; + pfn_req_flags |= range->default_flags; + + /* + * Don't fault in device private pages owned by the caller, + * just report the PFN. + */ + if (pgmap->owner == range->dev_private_owner) { + *hmm_pfn = swp_offset_pfn(entry); + goto found; + } + + /* + * P2P for supported pages, and according to caller request + * translate the private page to the match P2P page if it fails + * continue with the regular flow + */ + if (pfn_req_flags & HMM_PFN_ALLOW_P2P && + pgmap->ops->get_dma_pfn_for_device) { + ret = pgmap->ops->get_dma_pfn_for_device(page, hmm_pfn); + if (!ret) { + *hmm_pfn |= HMM_PFN_ALLOW_P2P; + goto found; + } + } + + return false; + +found: + *hmm_pfn |= HMM_PFN_VALID; + if (is_writable_device_private_entry(entry)) + *hmm_pfn |= HMM_PFN_WRITE; + return true; +} + static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, unsigned long end, pmd_t *pmdp, pte_t *ptep, unsigned long *hmm_pfn) @@ -249,17 +294,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, From patchwork Sun Dec 1 10:36:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13889463 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2061.outbound.protection.outlook.com [40.107.244.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DCFF13AD06; Sun, 1 Dec 2024 10:38:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.61 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049500; cv=fail; b=JuxcRCQyfxeuGjtbiWrcAJhhEKQtVuolNlxqOmznLx8Gc3GbbaS8Aej+cxUgKC5pk1l6xGXmfoJgkFNfltQRYG5FwAfdR3JYL6iACP95HdMFfrFeEUTt3OwO3jXc37vE8LwZxbDjImuuQqvssOhTWktzQD14f11mBvnNgN2rKaI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049500; c=relaxed/simple; bh=kJzRUfcYOfIrg8+1v3kd4n0zY1QVeGhPR/eZUPN2Sak=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hlzsLG0XQbhOD8Np7VG2uB5C35z5wVDDWf7/Jrj4aOuJyjJJHMfsK/Vwjz4n8seUkQMoGTmWdUznN6ANoY0LsutcCRvcbFs7h+VdSRRrcTI6FiK2wSNvBK3cyGZiqi7QeLPwTQopawM5aop7SueaFy9DvQTWn4dDLAKWcZLyBEw= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=J7kLu88k; arc=fail smtp.client-ip=40.107.244.61 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="J7kLu88k" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=l2eEo3JjBbLTQYmp7JaASfZ+08c2xUiSPMRSCJxNZonGKUR9kKEUYOAiosWiBx0zxpD0ura8P3PlT0MA5ifyiS/SFWVU1BeR3zSvKpBpBo5pBRFRI6wKpPINOS9lhsnNFeUuwmfVKDgOD9cPBiIFrxWZUsbv8bSUGeCWd2/Tgj8pZhwfdPjXEpRZofMapyf1EGvygvYShOgX48sarHfTGyZdeYYfr9eRV1sqoefMxjVFggMOE9KNnuznetSWxp6+4jSUgIRrPaJdLsfsa5D9heZ1ap5kLiv1jW3ZzK56mVBX3hUBR1+ubSRDIlVlj2ARhLdSLRoxijFrY6caW5Fo/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UA/4IxtsjDCMtBq0SlhR9bcQvXIpvvmGq2lJk5dRHR4=; b=YFR0hPn4TAMd0ljg+5Rsn1f7SzbtmM0jbSrPybw1+kn40+58X19wXjZvr04BKFEceKISYuWCdHJ1Ig37846HqEH5I4aPfgFqFZZwoJSG2lLgTztunsgwyCdrhoZgON1D1OTCBE9eMNX1xh4niFlKpp1DT5TBJAHkodgtEqKJamnbPaQPpNpMwusfdFVCN4pKsKxSJy2uwYVaAYK3o3MZD3XG0mN7pSILY6Xb9k9iTJ/23RxIJ+XsM1HWxWc4bmlcU7EhENJOSYqd0wgg5/QOMRZ6gV98z6NWewsDpOfBnz9SjZCgK3aw0gDfu+hyR1aR6My3PSYWLcRtOhr7kyWydQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UA/4IxtsjDCMtBq0SlhR9bcQvXIpvvmGq2lJk5dRHR4=; b=J7kLu88kTOcBPx3vqLhtD5Q7ANddfM2NN5b9z+Jj6nJuh3bzGFMF+ssoaqsvz9B9zVOngS0xCcvy1ozISCWVXUU75aUECHNOQlxtlGkY047i7IgZQ6aUwUWzvLvN0FK63v2HEqlqZEGu/ArSljRbyXXe1Qzz5Lr468Zqutbz9Y2BIuDsMKO5XiynaYL8/3/l1UesxfL5XAXaBqfFZnC0/LXaxYQ6PLL3X8S8HRRTWNTg4eZ6COx/+qbeMK9BbnnF3Kwcm9Pos95lKnh7wkK9WMGeopXpK+e5i3P+yxfIXq2/euOSAMIurqddDT2siB1R5jlQbO0T6MbgI3SR4GEAYA== Received: from CH0PR03CA0258.namprd03.prod.outlook.com (2603:10b6:610:e5::23) by PH7PR12MB6657.namprd12.prod.outlook.com (2603:10b6:510:1fe::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 10:38:07 +0000 Received: from DS3PEPF0000C37D.namprd04.prod.outlook.com (2603:10b6:610:e5:cafe::7e) by CH0PR03CA0258.outlook.office365.com (2603:10b6:610:e5::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 10:38:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by DS3PEPF0000C37D.mail.protection.outlook.com (10.167.23.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 10:38:06 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:37:59 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:37:59 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 1 Dec 2024 02:37:55 -0800 From: Yonatan Maman To: , , , , , , , , , , , , , , , , Subject: [RFC 2/5] nouveau/dmem: HMM P2P DMA for private dev pages Date: Sun, 1 Dec 2024 12:36:56 +0200 Message-ID: <20241201103659.420677-3-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201103659.420677-1-ymaman@nvidia.com> References: <20241201103659.420677-1-ymaman@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS3PEPF0000C37D:EE_|PH7PR12MB6657:EE_ X-MS-Office365-Filtering-Correlation-Id: 124e31bd-e88b-4ca1-39ff-08dd11f43e3b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: JlS2ib7zLFQeVi6ncrc2kdUZJp2r0qik4WAftnohOi3NKtCjGgBHNVCHagKJVsVtDZLUGjbL0hXTaXVHlFhDYjmLET7+lsD1Ocj6Wi8/5NhRqBAHt8weob7rqsh7rIQyOn6D0C0lD8s30oVBrjQ1g87nHBqXDhDMkVVgjLrhrIrg7rFKf7J9WPppxEghzj/1NXOSxQ41IacPlMaR6zyu1qoMiw8DlIvZ/QyQqqKGgtvxwf63ga0TxfqoB3Jnylq5Va7atDb3F/9K9h0gpn5et5D58nx+J1Nw54YbavFq+bdSPWM5rSVSAp/fF8tFbh/AUcQkcs+Z9gPtrNKRkKKp3UKLdfJLxXcBHzRnu5rAeYwrfjOcMghYwxYuRt77TC/hlzdrODOABw7YyhiKWTS4KnzwoyKvmGoJ6M+fqOfXdG3J9ToXTVJv/hIx9EShZMCrAsbcqLnhC38pmbbAS/VcpejWft2aI25CFXkHKWQ7CTj57YNj6f+j3IunJIWTy0HLaEWitjA8vJqiZIZAr9LjYqNdD0T9PlCYtDuTYNJXewpkPlHPOFduxaZvQn5ZaV/5ddO28DJ2sthzuL/3BtPb4WfNKJIfXElem7lBOZKeutUgVBAtnbnec2nP3v9sTT2euD1OQD3VlFHuvExwFMGWnb9SInN31LApKows2OytZB5RKBBBR1QOThncUqHE3eoiynkOpwp3i/cVaLawnsBHyc3aGAcKgehPgw1ctrYmdGC7X6PnsdNWsXrhPT57afgb80s+mePrwiVJDpye96Rrlx+lY40PuGBdiOKA5+uOg++sxXTRb73LYt1P8Tc0a/OvIJInmn44oe8gflN9w7kAMWFXpqHJz5NSoLhw4AZrigCuVFHU3bpj/snpS+GzZWGHEOQefoVUw4luvt4lTb33UlAVpexBi/H7ltDAKewwfXZgx3oktCo5REFn7NVI5FS4HecdrOqw1cCaoUHtZWKzHTyuMAhSQFE3oUStkwtz7QhuNnsPVhqB6GUTrKoTv4M+eKD5je3Om74v3QNFs/j9H9Ezy1Y6lGF9JE/9OquA8V7fkqkkJDndNmW9zN/4jaszSGnVSZ/d4WYUk6MVC3QwcnqBUJ9i1Gk95EkiI6F6AFzqBWexKQDXgH5f9HFV2f8JfMJDVjTsSDG6MXUOm+zOdlLUsucl/FkFBjKPz3t4z5sNgYP8nxGluyzoVbCPSfOUiVaiNSexx/M7/Opz/Rm8+uWQsUGwaBZwrdzUPMtrRKNbpyto0ES7MR9IXWiRzU1w4BCVYuvcTDpXJfZ8R2tDPhwmMJVqLxtsw39EUHBFTgq3j7Yzn4rnIScVU0wUDoju9kJErnqA1q/cKrICzN19bHOsif0Bkdg4X2nlLeNTbsexiBf71RlZuS+XTedCM37Oz71H4dMHGw0YX2dHCK4t5UeVhUqyAQE5tPe2rdH0hECam+IxIUJ8Hz8ab3k/pT+WX876AtQpUNMdhClFKXyYIQ== X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 10:38:06.8833 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 124e31bd-e88b-4ca1-39ff-08dd11f43e3b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DS3PEPF0000C37D.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6657 From: Yonatan Maman Enabling Peer-to-Peer DMA (P2P DMA) access in GPU-centric applications is crucial for minimizing data transfer overhead (e.g., for RDMA use- case). This change aims to enable that capability for Nouveau over HMM device private pages. P2P DMA for private device pages allows the GPU to directly exchange data with other devices (e.g., NICs) without needing to traverse system RAM. To fully support Peer-to-Peer for device private pages, the following changes are made: - Introduce struct nouveau_dmem_hmm_p2p within struct nouveau_dmem to manage BAR1 PCI P2P memory. p2p_start_addr holds the virtual address allocated with pci_alloc_p2pmem(), and p2p_size represents the allocated size of the PCI P2P memory. - nouveau_dmem_init - Ensure BAR1 accessibility and assign struct pages (PCI_P2P_PAGE) for all BAR1 pages. Introduce nouveau_alloc_bar1_pci_p2p_mem in nouveau_dmem to expose BAR1 for use as P2P memory via pci_p2pdma_add_resource and implement static allocation and assignment of struct pages using pci_alloc_p2pmem. This function will be called from nouveau_dmem_init, and failure triggers a warning message instead of driver failure. - nouveau_dmem_fini - Ensure BAR1 PCI P2P memory is properly destroyed during driver cleanup. Introduce nouveau_destroy_bar1_pci_p2p_mem to handle freeing of PCI P2P memory associated with Nouveau BAR1. Modify nouveau_dmem_fini to call nouveau_destroy_bar1_pci_p2p_mem. - Implement Nouveau `p2p_page` callback function - Implement BAR1 mapping for the chunk using `io_mem_reserve` if no mapping exists. Retrieve the pre-allocated P2P virtual address and size from `hmm_p2p`. Calculate the page offset within BAR1 and return the corresponding P2P page. Signed-off-by: Yonatan Maman Reviewed-by: Gal Shalom --- drivers/gpu/drm/nouveau/nouveau_dmem.c | 110 +++++++++++++++++++++++++ 1 file changed, 110 insertions(+) struct nouveau_channel *chan; }; +struct nouveau_dmem_hmm_p2p { + size_t p2p_size; + void *p2p_start_addr; +}; + struct nouveau_dmem { struct nouveau_drm *drm; struct nouveau_dmem_migrate migrate; + struct nouveau_dmem_hmm_p2p hmm_p2p; struct list_head chunks; struct mutex mutex; struct page *free_pages; @@ -158,6 +167,60 @@ static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spage, return 0; } +static int nouveau_dmem_bar1_mapping(struct nouveau_bo *nvbo, + unsigned long long *bus_addr) +{ + int ret; + struct ttm_resource *mem = nvbo->bo.resource; + + if (mem->bus.offset) { + *bus_addr = mem->bus.offset; + return 0; + } + + if (PFN_UP(nvbo->bo.base.size) > PFN_UP(nvbo->bo.resource->size)) + return -EINVAL; + + ret = ttm_bo_reserve(&nvbo->bo, false, false, NULL); + if (ret) + return ret; + + ret = nvbo->bo.bdev->funcs->io_mem_reserve(nvbo->bo.bdev, mem); + *bus_addr = mem->bus.offset; + + ttm_bo_unreserve(&nvbo->bo); + return ret; +} + +static int nouveau_dmem_get_dma_pfn(struct page *private_page, + unsigned long *dma_pfn) +{ + int ret; + unsigned long long offset_in_chunk; + unsigned long long chunk_bus_addr; + unsigned long long bar1_base_addr; + struct nouveau_drm *drm = page_to_drm(private_page); + struct nouveau_bo *nvbo = nouveau_page_to_chunk(private_page)->bo; + struct nvkm_device *nv_device = nvxx_device(drm); + size_t p2p_size = drm->dmem->hmm_p2p.p2p_size; + + bar1_base_addr = nv_device->func->resource_addr(nv_device, 1); + offset_in_chunk = + (page_to_pfn(private_page) << PAGE_SHIFT) - + nouveau_page_to_chunk(private_page)->pagemap.range.start; + + ret = nouveau_dmem_bar1_mapping(nvbo, &chunk_bus_addr); + if (ret) + return ret; + + *dma_pfn = chunk_bus_addr + offset_in_chunk; + if (!p2p_size || *dma_pfn > bar1_base_addr + p2p_size || + *dma_pfn < bar1_base_addr) + return -ENOMEM; + + return 0; +} + static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) { struct nouveau_drm *drm = page_to_drm(vmf->page); @@ -221,6 +284,7 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops = { .page_free = nouveau_dmem_page_free, .migrate_to_ram = nouveau_dmem_migrate_to_ram, + .get_dma_pfn_for_device = nouveau_dmem_get_dma_pfn, }; static int @@ -413,14 +477,31 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) kvfree(dma_addrs); } +static void nouveau_destroy_bar1_pci_p2p_mem(struct nouveau_drm *drm, + struct pci_dev *pdev, + void *p2p_start_addr, + size_t p2p_size) +{ + if (p2p_size) + pci_free_p2pmem(pdev, p2p_start_addr, p2p_size); + + NV_INFO(drm, "PCI P2P memory freed(%p)\n", p2p_start_addr); +} + void nouveau_dmem_fini(struct nouveau_drm *drm) { struct nouveau_dmem_chunk *chunk, *tmp; + struct nvkm_device *nv_device = nvxx_device(drm); if (drm->dmem == NULL) return; + nouveau_destroy_bar1_pci_p2p_mem(drm, + nv_device->func->pci(nv_device)->pdev, + drm->dmem->hmm_p2p.p2p_start_addr, + drm->dmem->hmm_p2p.p2p_size); + mutex_lock(&drm->dmem->mutex); list_for_each_entry_safe(chunk, tmp, &drm->dmem->chunks, list) { @@ -586,10 +667,28 @@ nouveau_dmem_migrate_init(struct nouveau_drm *drm) return -ENODEV; } +static int nouveau_alloc_bar1_pci_p2p_mem(struct nouveau_drm *drm, + struct pci_dev *pdev, size_t size, + void **pp2p_start_addr) +{ + int ret; + + ret = pci_p2pdma_add_resource(pdev, 1, size, 0); + if (ret) + return ret; + + *pp2p_start_addr = pci_alloc_p2pmem(pdev, size); + + NV_INFO(drm, "PCI P2P memory allocated(%p)\n", *pp2p_start_addr); + return 0; +} + void nouveau_dmem_init(struct nouveau_drm *drm) { int ret; + struct nvkm_device *nv_device = nvxx_device(drm); + size_t bar1_size; /* This only make sense on PASCAL or newer */ if (drm->client.device.info.family < NV_DEVICE_INFO_V0_PASCAL) @@ -610,6 +709,17 @@ nouveau_dmem_init(struct nouveau_drm *drm) kfree(drm->dmem); drm->dmem = NULL; } + + /* Expose BAR1 for HMM P2P Memory */ + bar1_size = nv_device->func->resource_size(nv_device, 1); + ret = nouveau_alloc_bar1_pci_p2p_mem(drm, + nv_device->func->pci(nv_device)->pdev, + bar1_size, + &drm->dmem->hmm_p2p.p2p_start_addr); + drm->dmem->hmm_p2p.p2p_size = (ret) ? 0 : bar1_size; + if (ret) + NV_WARN(drm, + "PCI P2P memory allocation failed, HMM P2P won't be supported\n"); } static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c index 1a072568cef6..003e74895ff4 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -40,6 +40,9 @@ #include #include #include +#include +#include /* * FIXME: this is ugly right now we are using TTM to allocate vram and we pin @@ -77,9 +80,15 @@ struct nouveau_dmem_migrate { From patchwork Sun Dec 1 10:36:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13889464 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2066.outbound.protection.outlook.com [40.107.94.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B5A913959D; Sun, 1 Dec 2024 10:38:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.66 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049503; cv=fail; b=PvneYC8XOgA4vZb6ZuajVHZmexayuRNLx9aSxqAREDKZ1BZ3iOVWZnlvOYwbyjwyN+CqSyd8CfGC14AHcWx8WYhIxBv4WWLQcOMXwTN00s+fUgUKa/emGDInBydIY50np8wal+W4/NXQHWQhyl0Z0vgtB4akbSLJStSMVCsInkE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049503; c=relaxed/simple; bh=3XTZ5SvySxH6aoWIelzJlBi8JM+nIHk1UBWey9N3VIM=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=t8pLwjjcSvG11nQWBNOCUipUtVL9Xuzt5IfxC9VpRtvqfOroOeY8xbxrDZX5XeqGdg6iM2AMHWO+XZ3BU0fNN6xEdXkb9iUNyRN/z31gu0H6wGCKCfvyXg+B18G9TCyZhe2aV+aKuwyC/+tT1PZRRcuz3jVP3RaHpun3TE3WRiU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=NclyeTDx; arc=fail smtp.client-ip=40.107.94.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="NclyeTDx" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=coZf6enY2JBnvm4OLwRB7JgceBNOiI7U4xaMYBAHGjNDFlnLZlSCmoy+2dcG3QU6FudTnxDYKuWKufl75C9YfUFYgyS2flpFJVQXjNmadWff0d4ULmQcu+5uawAlMDL9WXLEkuD94UeMoDqmPwvPvbUyUR1WK7dv/tHePIKVL3x+bsjBNyfbqCQ0mXFgOcN39aA8TPydA7rCY7rb/h24Hg0+S5lohqqYrXby4rOUescTlkOeXXMC5ynkECPE86nrZAA96vQcmwjDCZyMVAUi+MEMiTq4pOwGQdDt4dWByBYSbIz/1mCRTwVtt+g31KpskCLuIJBCCJde8sSb4aWX/w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kbyg+LGFlJ8ejF7f5+c17QNDW/EF3rYADtqdJApzx3s=; b=e4gOfd70LeeZyPKSvZN9NvGgdfezc6iG51jm+X/e6bq3wz29elY0UaFJbJ4HXzn2UCkLjPKlRXfd61zV4V1OUrBTvp3rMVHi8hcG0Coxc7xjh0i7VXzjChYhyyycmWO2Ypg0d4yQ+FNnpqm9jmBhh6bQnp+Pw18T1dCS5bNA2KWEHK8bvMHS3omTCLLTYmQc8pQwkLLnZ4YAZVNPdPjVEadfFKO17c7xjCc1mvDMFwaOejxysxEwYoHtu3THPA3+O25nLJ00rNaQ7oS/b6BcIkOp4tm7g/4nlhd6QPW9bTflaLHfczUJWiSKYfgsHHR7iPLVwc0dAnjAvXsmjJMtcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kbyg+LGFlJ8ejF7f5+c17QNDW/EF3rYADtqdJApzx3s=; b=NclyeTDxNisNJmR9xwDac4qKn+fI9eC2DdkCOQpcty40fxUri1vd3LV1olE8n4QvKQanJVU/LDZ5CRH+sjbDc7uy1SvGSQPpQRf3dbTTTC2ZncgmTS6sF63Qptc0YPTQmjlR52aEcm+CguSZCVSUM9xJVEB3KX+7ltsgQWzWgB7ndk7C/cQVW091kw/qAO/CwABlFd2T9lW3c+KaxLOZdV4Nzh0wFt+gDLGuJZ4p8FAEV9HhBBDYBE1j6mxQU+53bjNw58htgjVBue/JgncyTi1JVhhYUjHOiIMuilcTSh39Lbazpl71/CMQEDVwFzLKB0kylRDgDcKRmKwNSgu+rg== Received: from BLAPR03CA0051.namprd03.prod.outlook.com (2603:10b6:208:32d::26) by IA1PR12MB7688.namprd12.prod.outlook.com (2603:10b6:208:420::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.15; Sun, 1 Dec 2024 10:38:16 +0000 Received: from BL02EPF0001A0FA.namprd03.prod.outlook.com (2603:10b6:208:32d:cafe::57) by BLAPR03CA0051.outlook.office365.com (2603:10b6:208:32d::26) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 10:38:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BL02EPF0001A0FA.mail.protection.outlook.com (10.167.242.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 10:38:16 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:38:06 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:38:06 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 1 Dec 2024 02:38:02 -0800 From: Yonatan Maman To: , , , , , , , , , , , , , , , , Subject: [RFC 3/5] IB/core: P2P DMA for device private pages Date: Sun, 1 Dec 2024 12:36:57 +0200 Message-ID: <20241201103659.420677-4-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201103659.420677-1-ymaman@nvidia.com> References: <20241201103659.420677-1-ymaman@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FA:EE_|IA1PR12MB7688:EE_ X-MS-Office365-Filtering-Correlation-Id: ad54d123-a9c9-4cc4-275a-08dd11f443c8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|1800799024|36860700013|921020; X-Microsoft-Antispam-Message-Info: KZmezP+Wq3JGWwEB6N2WfTCFWHXrLwv6y4ywRh7UDxP1riQyHowo0PqAEKLNgkvKmbu7GXRKLQOLbYdmny38oYZjVTmGWWfICCTb7xFQmtziftdTch5u9//dHUqYpFlG3Wmj2SKp2fabhu5Rf47KRSWos9iYi+QHlsPNSYrE/m2HtMhcjZVFlTkvbcEcZ49KhFZyC5u3blC3tQhHarQdfbArMDq2DN+XqUUyq8Ph6Jcr470VsxuiD3FB4J4TtFPNhCI56HyzBF52Ks+FpEREKeto+ItnWsA2Oa884y0d/zsNiejgvcgE8CAnVPIBEHRy2NICJ3EC1Z8aXHwktsHbYn2iMIg32NmYUR/xPjY9mbAePwr/WoHkZnU8y+odqIUsy/epypEPqh123pLnw4GmPZEbLIa+3jUe9GaJbUgfRBWxWCW9k8xmd660o+jt8yQJLAQmb/fxG7nEp7QQLpPCbmclIFEkIFXpJp/jMMFmmv63aHkjAACwtIrbGSDUpJFHCPWBe8eSr9qk7N2U6TB4MAGwqq4Z3BRCwQzZRdsVku+yvQ/ooqEa1ukT96ontW0U6ONKkkt41v8KbnEX5a9VsPHJTuNn2F4/T3sKRbbGimdGDHQAMmkvwtIDZGSBRSdgzbvNnlHMSVhteiQwcLYYBvWD9eNyI+a6IBf3jzl4DJNookWHeOOQQWWMgxeBZskaJjgqrd1IB+QB6P2XBWo1POzLuJ67VQMKp7pAwL/QLJ3GUB+5W4YWBytOMamvWPjamGL68mVzw0eBB8K6/G1GKhWQuOc0UXf47SfDN1XhWms3wGASdVzSXgqsWMYQ0mtHaL5XqY9xBXyyQI58le43hx+8XfWDSGJvAX47B6+DItbrjugC8KTKDpz7R4DdFnddtVBqHfgDLV2oG7WzCPDc1zuFvBdxJ7dk8IOX3rxKYdOHlbkOOS1cZXCr+AAuLnkeLToXDOwLuVASYPxmq6/5LYtpFirYtaHsXnx6nFEcobnel2JXnRv4zUyMDWFCEn2684DiFv40aSZ1ns71shiHfxQDWjvaLnrst1SobrkjUOeLRfnzW4wn5K8IjGtovmhS839NstxI0/gxlXjCHODaqblTBVKtXLJPSJB7oX19XUR+ebj9t4ycxqYBotFhKQ3LBifRuE71Cw2PVC7Bs4YF8jiIfIqqkb/WafICTe0HIco8D71BXms9vLPeC9qIwC0lUaAJ8TC1UQxAeQIFRvE//JcJUY8QBAdmF9LVMySen1fVcTWdLdzZAndu60Irp8XKfC841fsIKFjh5y3eyGVv9VKm2+PHRlQkWQmJyvkMrA8q3LzO9RSAj9wQqfO2Ptv339a4A1jJ0CNOuqDbQEoAgWaSoDdlsX9RhHb3tkhmvCTJOk9mIVQAePaID/mbywlkVM2ssdWRrmjeUroF+m9DZKRTxEw2uDhiIzdU9EUyJBFKIFt9YtQ1J7yquiKICbIwpyaO987Y7hwwoNWdaU6P4A== X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(1800799024)(36860700013)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 10:38:16.1486 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ad54d123-a9c9-4cc4-275a-08dd11f443c8 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FA.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB7688 From: Yonatan Maman Add Peer-to-Peer (P2P) DMA request for hmm_range_fault calling, utilizing capabilities introduced in mm/hmm. By setting range.default_flags to HMM_PFN_REQ_FAULT | HMM_PFN_REQ_TRY_P2P, HMM attempts to initiate P2P DMA connections for device private pages (instead of page fault handling). This enhancement utilizes P2P DMA to reduce performance overhead during data migration between devices (e.g., GPU) and system memory, providing performance benefits for GPU-centric applications that utilize RDMA and device private pages. Signed-off-by: Yonatan Maman Signed-off-by: Gal Shalom --- drivers/infiniband/core/umem_odp.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index 51d518989914..4c2465b9bdda 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -332,6 +332,10 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, range.default_flags |= HMM_PFN_REQ_WRITE; } + if (access_mask & HMM_PFN_ALLOW_P2P) + range.default_flags |= HMM_PFN_ALLOW_P2P; + + range.pfn_flags_mask = HMM_PFN_ALLOW_P2P; range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]); timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); From patchwork Sun Dec 1 10:36:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13889465 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2079.outbound.protection.outlook.com [40.107.94.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D713E13959D; Sun, 1 Dec 2024 10:38:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.79 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049510; cv=fail; b=mSRoeP8SNhk6gxZZqWeMCygkumvCEi0jxqJ9pd5SnFSRr5/KvHyOoOCLSSM7ij82YuKllsPX4+yaT+TkhA0bfetNoeJAhPJnnvKW1nCLMV9b8ISzGbYBk+JiulrLU1/zMvadmjWKU5ns0PkI/21x1wfWaHvlnL0Ckjenpt9V1NU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049510; c=relaxed/simple; bh=lWVtCoQ0kc8ESYbg8Ni7tONY0GPRduSl2cyqJ0wfWMg=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=BJ79mc6SeNfbk/zJ/NEvylvX8zm0n4KAXN2WvSYq07p8sZLXXjebX0CTRTvPsgn4bnpmUszHxvO1nRvfh9JWxU6zs4EM3XPAC/vcRzAmAozeAftX/rGuIhZUW5rUs+c+43Lk6rE8eNRoxxb8Rb8L0mK8e2w85NBFwBeGieD9gvQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=ju7jO10U; arc=fail smtp.client-ip=40.107.94.79 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="ju7jO10U" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=k9dML4BQOGhYdlULdXPLescpYREcpwyP+gI/Wqk0F6LKybELoTDPCeEDJlkUGFWj5+/IS4Xa9/yJtzYxjuGB4U+NaydsT2MbO25AwO9waKrdw9ug1/N2XHvaSt2hK1WKog9EvwYXZF/9taAIuCu+3n4EeK8VCzIxiqxWOvcxVsQ31TBEuUo1ahoCPokigqWIgeYzqta3Xkw9HPI2iK4q6Hf6AAcuZSz8Kuc8qnROywGUJlUJtdUX2Lnvc1ImmVQQEjRzmEWEJvrylnN90PtxNwi+euwSh6bVHdMmiK+8HbcLGDp8g2S3uifP5DgbKdPrZaLe6OWNoQ5LJxzyt0LDxA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G5lebes+3vCSQ2c73btKjDroQu3/xg9vnF+gZ5x1ies=; b=qcNa8aQa4F2oo5STzR1V7B/QrCRj5jkxPnY9cHCsXZIEo2Hz/zP8OwMJn6m/OVTlvHDuNkewhj/KzT0kc4njdyCJjNOceDhrIKwO+1Ndlg8olaSijZUX/I19kZR3apfQBWjKMeyUg6dGTb5cxYzgojahBbCPnTYD12yWZnyYMyHYe9jwhHzirYD+nzzo9vErE83xdJ8WkXMxZD2aSiaOOH7MnfSwT/G7vSlQAcF6wsyAy9m+Bf8k1vw8JVCgj4bG5vOfPqPaTKE6XDm5qqMWpxzNgRw8JPrnihov+ZlHHYvUx+ebvthWgB/cvl9BDWp2zYVpnQb4Ia4xbNhpBq3C8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G5lebes+3vCSQ2c73btKjDroQu3/xg9vnF+gZ5x1ies=; b=ju7jO10U7pE96ufbXoZ9BffptrFQJEnU/yIKGbzekfWQA7tUaOjvIzMzOd6n3MszQP4AZG3XqgFwqpusZKXuZQnRVV/qwygsNLb/rCz/eaYa5Jm/AYHHDH0dA28b2i2DrKQbJluSnkXAxE6D83WjoRcV10iaa0wILDKTMsWgVH1mm0wf31YWC28kSHdiekYheRv2+0/kIU8NBCQYq6UMFW4YMC+fbJ9Og/fSGugfw0hSCsoSXQ6kzNzwBXfu5P+xwABhtQeh1lSCvYQ1kZvDJb2itSJ04BEjEQhmIyEd7NvkgeVyXcVGjgTrz8DgmUyUw6BXJ2uXVf1mGqrsk4OW4g== Received: from BN1PR10CA0010.namprd10.prod.outlook.com (2603:10b6:408:e0::15) by IA1PR12MB9524.namprd12.prod.outlook.com (2603:10b6:208:596::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 10:38:24 +0000 Received: from BL02EPF0001A0FC.namprd03.prod.outlook.com (2603:10b6:408:e0:cafe::cf) by BN1PR10CA0010.outlook.office365.com (2603:10b6:408:e0::15) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.17 via Frontend Transport; Sun, 1 Dec 2024 10:38:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BL02EPF0001A0FC.mail.protection.outlook.com (10.167.242.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 10:38:23 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:38:11 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:38:11 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 1 Dec 2024 02:38:07 -0800 From: Yonatan Maman To: , , , , , , , , , , , , , , , , Subject: [RFC 4/5] RDMA/mlx5: Add fallback for P2P DMA errors Date: Sun, 1 Dec 2024 12:36:58 +0200 Message-ID: <20241201103659.420677-5-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201103659.420677-1-ymaman@nvidia.com> References: <20241201103659.420677-1-ymaman@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FC:EE_|IA1PR12MB9524:EE_ X-MS-Office365-Filtering-Correlation-Id: 1202fbe9-9a6b-47be-1980-08dd11f4485a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: lggH5GV1eeXpGR81zJS92Pi/2C8pUZROuRntiu1vQfJEBRAw3t/d7cOMJ0dFiaSe/wiolGsxLqWdglPM64sJb8lJafSxRJmuEHmAPqlgR/S8ii/nCy0xcRFN1M6w6QQTESz5EOuYW+N38n5l07E8kXcLzd0ZPA8RzSL4dVJLgKdJe/g6+GRomHajRSlDM32RmesXrsU+nsWQhrWCMIXIwzG7hTX+lSbSpgLebt5KhrFVgqPBJqswdCo73IRWb8bGb51nww/JrSmYrbgQOc4Z3uExNRxG+OqQb23jRDv4UypQG/19zZ8prTBhgcZPgQNKQKeyHNMGftOY/kFtg6eNzTEv229vSkH72fehR9EpB8OB9XfU0UyAq3BCvMVRN6g7RLLyAoHTjbjHT+NCNkj7nS8AwLg/z/t+je3cO3EXmjqI4/OSIhKkjlCpaTYY3wjLUmkhS9TLVTr2JTZxHPsX07jOhtXycUyCvT+GWso8NosLDyJFrEtMB5wXXO4ZKBirZqhgkFi6rkD2HdGipL3ZKo3WVKrx4t/8jjimHlNcYhapu9N8+Gldp/Z80B7HnXydjSKtSGwvSesQ2d4hRMrWvBpzA8ziv1xak0LmkmPHud8kaOo4cRtX6Gm3+HtWTB9mDvMz8+2JDdjOggeyHAJG65L3PAFCszfuOrj/8kTblGvzTUmFTD6wst5Iwd/5D7DFaTrLfl6RlvKcRzoV/xfOzZnx0XjBDLKQyT9wclE3InYCDYe0hP7W3qEhRi83uU3UbKVmOeyAIQ5amrjulxusEzdSqkiTzdaudxBxc2xfKpB8Gglc+1zyPwaglvUQ+iZWif1Lu+cJ7dp4DwUw3Ql2lr7QzRcSrzjBB3eLntCe/DcXE9luBYSgs1Hw89/peG0vyzdtk8YUQmz8RewPIje59VaTmGPeZ988Vr2c56UlwOq6FLn9VWNDdUa14zawVzQts/AHPW52nvTbhn6iM+0uSPTm6zTogPOm3vnsY0V38Ru/u9I7oVw7EWXp2wPf2i5plvYe5b+A2u69lDzB8NGufVMW2jQVO2FllhMlI8r34kSlCdj8hLE52Ve7b+XDp8EOFtzri3578SIXZkk/h7q7YjX5xSru3h2rUCVZVIh8K9Q0fTdP1LBbCxop+WQQlF8aM98kF9yzO7Udo8y5+pwirapeR6zgmKMFhyxCYVPAt3spcTEW0RTbFm3rjsFk655rqqeRizK7YfWTQeRUK10IIowqTteabWEDjRb6gFBSOZTMqfgMLbEJmoHPqlgMi0N7hhQAI3700M5/Not7Z4L/492r8pjKjgZ+x1TtGFquImlPsauUghTqzD2gUw1MqTY492ZbuGWEd6f0KejAIIhvnayXdt+XKIC+NoJxsImmyOvL9IxgeMcgEkrTAhCLBORCwmxLCwddg9ChHTuJaqcIo408nTo7luyInTwrj5YWhcY5A0NLm3ISxlXoZ26CXr3qoEpSOSNgxLTm+dcLImT/cA== X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 10:38:23.8003 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1202fbe9-9a6b-47be-1980-08dd11f4485a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FC.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB9524 From: Yonatan Maman Handle P2P DMA mapping errors when the transaction requires traversing an inaccessible host bridge that is not in the allowlist: - In `populate_mtt`, if a P2P mapping fails, the `HMM_PFN_ALLOW_P2P` flag is cleared only for the PFNs that returned a mapping error. - In `pagefault_real_mr`, if a P2P mapping error occurs, the mapping is retried with the `HMM_PFN_ALLOW_P2P` flag only for the PFNs that didn't fail, ensuring a fallback to standard DMA(host memory) for the rest, if possible. Signed-off-by: Yonatan Maman Signed-off-by: Gal Shalom --- drivers/infiniband/hw/mlx5/odp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index fbb2a5670c32..f7a1291ec7d1 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -169,6 +169,7 @@ static int populate_mtt(__be64 *pas, size_t start, size_t nentries, struct pci_p2pdma_map_state p2pdma_state = {}; struct ib_device *dev = odp->umem.ibdev; size_t i; + int ret = 0; if (flags & MLX5_IB_UPD_XLT_ZAP) return 0; @@ -184,8 +185,11 @@ static int populate_mtt(__be64 *pas, size_t start, size_t nentries, dma_addr = hmm_dma_map_pfn(dev->dma_device, &odp->map, start + i, &p2pdma_state); - if (ib_dma_mapping_error(dev, dma_addr)) - return -EFAULT; + if (ib_dma_mapping_error(dev, dma_addr)) { + odp->map.pfn_list[start + i] &= ~(HMM_PFN_ALLOW_P2P); + ret = -EFAULT; + continue; + } dma_addr |= MLX5_IB_MTT_READ; if ((pfn & HMM_PFN_WRITE) && !downgrade) @@ -194,7 +198,7 @@ static int populate_mtt(__be64 *pas, size_t start, size_t nentries, pas[i] = cpu_to_be64(dma_addr); odp->npages++; } - return 0; + return ret; } int mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries, @@ -696,6 +700,10 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp, if (odp->umem.writable && !downgrade) access_mask |= HMM_PFN_WRITE; + /* + * try fault with HMM_PFN_ALLOW_P2P flag + */ + access_mask |= HMM_PFN_ALLOW_P2P; np = ib_umem_odp_map_dma_and_lock(odp, user_va, bcnt, access_mask, fault); if (np < 0) return np; @@ -705,6 +713,16 @@ static int pagefault_real_mr(struct mlx5_ib_mr *mr, struct ib_umem_odp *odp, * ib_umem_odp_map_dma_and_lock already checks this. */ ret = mlx5r_umr_update_xlt(mr, start_idx, np, page_shift, xlt_flags); + if (ret == -EFAULT) { + /* + * Indicate P2P Mapping Error, retry with no HMM_PFN_ALLOW_P2P + */ + access_mask &= ~HMM_PFN_ALLOW_P2P; + np = ib_umem_odp_map_dma_and_lock(odp, user_va, bcnt, access_mask, fault); + if (np < 0) + return np; + ret = mlx5r_umr_update_xlt(mr, start_idx, np, page_shift, xlt_flags); + } mutex_unlock(&odp->umem_mutex); if (ret < 0) { From patchwork Sun Dec 1 10:36:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13889466 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2044.outbound.protection.outlook.com [40.107.92.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 211B11531E6; Sun, 1 Dec 2024 10:38:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.44 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049513; cv=fail; b=OlAnN4ioNNkz+UulrhGmWjySZANeuOJ1gDxWQ5RnNwzXzYKVtLPj8YxAMW4RaCs4uMftMvkJ7eTSuQ5bmw/PDLYVegvlFWxXdu/Rlvwq9SKfr0E39VMxqxeBea1jzMMfJNWswU8oRGzq3UFijvcanRq5jyQ4yx2fs/HyRUa09tA= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1733049513; c=relaxed/simple; bh=Rph06a55Qm07+kWBf5L0JI7fUcHFSZw5Dz6VotJ4DZo=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=gE5tFoEY0HPqisBmfrfT+JszreVZTxtFysquKnCtseXUdl0uzg1BSNkBog0XeID0pD4oY4ZhCOUzr5Rje7AMxRGpDyHAfiR6JzWOBLbgk8uBlTF++PjZvbkuAlPhE9hFDPt9WtakBqUFiJUtcvEzJpuTVc3KdslIXhRcDKcH+N8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=IDwFBfwO; arc=fail smtp.client-ip=40.107.92.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="IDwFBfwO" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BpK0LmS8UTBk7qNjxrWU92rXyT8/etkQJ0thpXKnTBfE93o8GUwWSVjDOIwzh1Aogu7KhlzOHmjtqvc5RyS1j6NiWWAdqasY9rJa2HIP2hNlbQvMpreinGcWq4/a7KJwhL7Vab4gWCms0KnQ0z8hjiONw8OtQGpbTVIrsAXmV3kbsmRZYsUN0mvxwF8QQE3B78l7sW0BcV7L8sN4apQmwkd6ni2SAcMqLuGXuy+QEjRf+qxluhv2UC99WKnDUjLLIrTwiJBTlbogEnM/j0YfF9/Ujt3qzO51ccOMAJQZNOW/FvM9ZjuA4HI8ZtpGQtmlwvPchZDgHaRFF2kqwVbQMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Om5/P0VucjwtVSyPAahMihMrFKhMEGTpXlaDQvTox2M=; b=u2Jm+zm6RVSCNbKPPGpGFeLjdWDPsif5wfY88usujP89st8yort8Ienfxi2SIUCP7q9/De1YeNc8UYcZfKswJRMhyzpO6z8vJ/nj5ZANjo6sxus3K3eBbEVxK4JvKDi3qSeWR7e7pkp2ukCy5zCSg5K+CRKYNC6v7LWXuJvLNrGGtMeKG5VML5665ACuxKweDSkCl7YPWGKvJ/IC9Pi5KBdE4acm6ml+TgEMs7nAA/9qSFJiPubyBjY2JtNeNKZ0Im1FcS6C/EsuwS3OH0Hb3J6Rb5ggRcIstx2O6c7ZMA4mdLh1nnYBo25tOvs4bTfjkj5OaCpixOzpzT9QLbyxxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Om5/P0VucjwtVSyPAahMihMrFKhMEGTpXlaDQvTox2M=; b=IDwFBfwO5qS9Tg6KYGwJCywwX5rEM15VOTszLcScVSysxrWyuhMYcpE33plE/Hj8y1CQ/AEY+e3GIYhPzTX6XvmzQoLySHveKcPRxeZOn8UAfmEf/zcIRYyTvlpedvTaCi/Glk1XnA5E3TFH6J+h6NLvCwLqKDvY8kP12u0bgNAYHUuvZTj0V1BhpaDjMOihzD1iy9LlpWVdc2oMcYDZUbH8MrYnMvYHmwWao7RE3e5OmwSXZS9JmdKR2XtlbE4uvA3LxmG0SmWyU9/I62MPoO0SB4GHsh3jEPd+lz7e0BKNN4837TpyP7fiNx19fKXU79jP6mDYL2V9U0tw8l89vg== Received: from BN8PR12CA0020.namprd12.prod.outlook.com (2603:10b6:408:60::33) by SN7PR12MB8790.namprd12.prod.outlook.com (2603:10b6:806:34b::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.16; Sun, 1 Dec 2024 10:38:28 +0000 Received: from BL02EPF0001A0FD.namprd03.prod.outlook.com (2603:10b6:408:60:cafe::dc) by BN8PR12CA0020.outlook.office365.com (2603:10b6:408:60::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.17 via Frontend Transport; Sun, 1 Dec 2024 10:38:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by BL02EPF0001A0FD.mail.protection.outlook.com (10.167.242.104) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8230.7 via Frontend Transport; Sun, 1 Dec 2024 10:38:27 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:38:17 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Sun, 1 Dec 2024 02:38:16 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Sun, 1 Dec 2024 02:38:13 -0800 From: Yonatan Maman To: , , , , , , , , , , , , , , , , Subject: [RFC 5/5] RDMA/mlx5: Enabling ATS for ODP memory Date: Sun, 1 Dec 2024 12:36:59 +0200 Message-ID: <20241201103659.420677-6-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201103659.420677-1-ymaman@nvidia.com> References: <20241201103659.420677-1-ymaman@nvidia.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF0001A0FD:EE_|SN7PR12MB8790:EE_ X-MS-Office365-Filtering-Correlation-Id: c96b6a3c-d68c-458e-1b18-08dd11f44ab7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|36860700013|82310400026|921020; X-Microsoft-Antispam-Message-Info: NqxX3b2JY16FbXvl8cD9+y1neSRMNNSXfyxI9S1O5WiQaeHcoeWxxhjiEcY5yjJYfY/6GsaaA0UTui1909CmrGNFQ6FOu5DAegxKJUgCKgJ82EH5XpjKoQhmfadMoxwrRHsDDRf1elq4Eur1I4MD5vMl5v3moVeM7l+VYUyxCFh0oX6GyiweSXdkB/K0KoSZv4qlKBEMMKchiiQCFCMYgTJSBfhx1NzKA3ZLYuCpUCOKDhz07+bgRkMg37DVR1Ve1ekLAW7GnnHTfmkKezXCqjbv7DLtIsJGrU6vHtQOx2vpvJ9Rx2tHmaahZp58b/SG7SeCzxP18jebJWoiYHEHYDd4X8HeI2sZmrbceSsWVg1F2HeF2YpaJTQXhp64O+RW9U88JvGalV2ykBt1kmyb4trMyVgfJZSSxMGA2bckimRnEJeTVAro873CyZT5SJoHPl6Q4oKhWl+4BrRIRchAzsCxu2GhvjUMDk+59eSdG9JUMKTtTRZHRYTas7MygxLL4awByg8KwTf3Lm5F4Ud7aL878PkAboaORVCwMXZHrKRvgJWkGsuv19AZC01TygPXvoSs3g6Xnb+rWByWYiBUqcckFzLODrspZbLZcQccO4NZANJ3gfdnHNAGm6aDDSn7Z+H7FDWcLhorr4IenjgsxOO+uCIrvOGk1ItH/GrNCIvoqg1WGTjQUAXAouhuL6P+wJEI+P7RAAMRrXaWzyvnih3DiaVgSvQ4XhSgzh1Y2mkaHzOH6JhT+ZbbfaFF39DkIK++ubmGmUl5NYxwvGn8lEh5Nl7/bWX348FDIRlE6tmb71sbTzPPzvBRZ7frt6WRjdCTFNKA87V4tH+1hdkAcK4JmemOYMW46RMMV9zNVmBHL4gjzlBJ+rBIKLFWleyE44/oRW3xvUsbFx0dLk8Xals33J1IbgAy71xTUSm7cCeED5TLIYMFCrgYHdKc3YLY3ZWF3b0XASg5JioPQK4fnx3Pm+ihQEB08Su4e6CNtqOba0NK8W2RW8hXKYSynr30LQM7cwYgp35E+MXnxRRenih4Ee/KcWQR5Tb3XwALiOR9jxuu3ALwDch+tpbMpO4GNodAXy6Lf58I6+kYZ+LHMlCbJg5mjb2I+G3miVGMU6jsh52nujR3qbLpC/C5qRuA1l7z+beSbEf0787Jn4kqfBrxjb6bw3cuHEPkzYFTypeiCFEgueJGUhtIxavHB8FNPw/npEAuTODFQ4EHNwAlrfDguTGFItpCB03yVGZaYYZajayqXLZ+BVatD9C/9y399bpCBiP6D6L9VSYttqEzBC0YXIkJYyC0V+gvBHPNRa73uSrHoCz72KPf0gBU111AkQuoX3p/q0LfTtOKvAwBtgNEXes84bK1cL5NiL+xDoJemmZb21ruycgPF/sLbTH9a8Bed7P36tNRFGSLxuenOqwwTmQII0scny5Wjl+wI0S9pqSPlYU0Qv3D4k1UQuSHFjdsl8OU1GiScPpk04FilA== X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(36860700013)(82310400026)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 10:38:27.7809 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c96b6a3c-d68c-458e-1b18-08dd11f44ab7 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF0001A0FD.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8790 From: Yonatan Maman ATS (Address Translation Services) mainly utilized to optimize PCI Peer-to-Peer transfers and prevent bus failures. This change employed ATS usage for ODP memory, to optimize DMA P2P for ODP memory. (e.g DMA P2P for private device pages - ODP memory). Signed-off-by: Yonatan Maman Signed-off-by: Gal Shalom --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 1bae5595c729..702d155f5048 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1705,9 +1705,9 @@ static inline bool rt_supported(int ts_cap) static inline bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev, struct ib_umem *umem, int access_flags) { - if (!MLX5_CAP_GEN(dev->mdev, ats) || !umem->is_dmabuf) - return false; - return access_flags & IB_ACCESS_RELAXED_ORDERING; + if (MLX5_CAP_GEN(dev->mdev, ats) && (umem->is_dmabuf || umem->is_odp)) + return access_flags & IB_ACCESS_RELAXED_ORDERING; + return false; } int set_roce_addr(struct mlx5_ib_dev *dev, u32 port_num,