From patchwork Tue Oct 15 15:23:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13836615 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80E78D1D878 for ; Tue, 15 Oct 2024 15:24:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AC966B0089; Tue, 15 Oct 2024 11:24:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15CD16B008A; Tue, 15 Oct 2024 11:24:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F18466B008C; Tue, 15 Oct 2024 11:24:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB07B6B0089 for ; Tue, 15 Oct 2024 11:24:22 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8C3AB1A16D7 for ; Tue, 15 Oct 2024 15:24:06 +0000 (UTC) X-FDA: 82676207586.09.573CD82 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2063.outbound.protection.outlook.com [40.107.100.63]) by imf27.hostedemail.com (Postfix) with ESMTP id 1955740006 for ; Tue, 15 Oct 2024 15:24:11 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=TeMzfZgG; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf27.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.100.63 as permitted sender) smtp.mailfrom=ymaman@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1729005827; a=rsa-sha256; cv=pass; b=T2U8h/3+CkJ0qxLWkEdp1s/uTNj5Z1ZRskhQSjAT8gCvVnRiEs0sBmBqbT8qTM+sho2NFs kGY+Dxc2fJMIc1WiXOsVSRH+JVch4wwRTJNz6raJzG7H6r8Oo4m5z8h1puJYZFHqn/1gb7 9auFWlesBGucRiKcFukAKrlJtKN2Lxw= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=TeMzfZgG; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf27.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.100.63 as permitted sender) smtp.mailfrom=ymaman@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729005827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vWcn9NVt1hK7DI5Z8sasmo2JCUcg6jAxYQ0go1j3Uyk=; b=MV/Pza0P0K0nlbWDYnV7Dc7FXJ7B0i4Khea2m/mOGoi33wOE2Z4rQotRo08X8yIvUTN4mA AUCD7geiK+LLSs033ME3cFb5OLnkD6pbpdUiifGLaUYv9fti+qyHTeh45KiTR1jHCRpuaX P1aXJkn+8b1JZWfwuZD+Y5aApN31+08= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OOIa4n2GeEtQdQ5mIrObPHNq2tbUyQH0yoRmpptZnFJqba4Lg563lSBhCaM067gFS3dS2mOCWCjJfgsagyXK9Fe3DXntcJQC56t0SzQLSwHx3PaCJmxNj5lwKca7KvHOW+/7t+nYPLUkjjlBp6LDetrKq6KJIZdjfU/P4Tk8KdJsay06aZlxBWTUwq5su6vm7dlyUk2OdAYaoffrDZQWXX0SHFgp6sZjMFQVYg66SsWyoPcM3dJWs61Sz1JX14ThxhwkZ8jVl4CLY7Cm5eIR3yndR6nG5qQNYEItaagbXGfXn4j8GTWm5fR1FWt0dPdo/6ZSf/G+OmiiixbgDQhhIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vWcn9NVt1hK7DI5Z8sasmo2JCUcg6jAxYQ0go1j3Uyk=; b=s/hUjmd3GWdOfX2YYvEHuOhin80QYC919HLnOyPbtqKHdEaXd9DPRZBLZDKEckfG02M/9GhzEp7Os/OcFdgQSU/ISFz4p76Qt/Ye4wOGJMxvcAc264M2+NlPwN0wvMWM/GvH2flKfltrtSCrVK7en8d4H09F3EpnsudQorgckr5vszit89oPBk6yszjiLe1gLuyvjrrxQkJ9AlFbTsAZrCuHJswWVci2w96PK3o72lBZfw2WwuI8sWH3G+0tkkNkm4GzszIFGEoOYwCfdAGK85O5SrVNB4mGq+7ty2tapoWUDEGitsANWazvgU9MsNgp6ysjJGFAgR0Eki5srDl/Qg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vWcn9NVt1hK7DI5Z8sasmo2JCUcg6jAxYQ0go1j3Uyk=; b=TeMzfZgGXRyEnUbvup3rg4dxNu88kGxtycFUWNDUnHWE0YoLMIdRoezFFfEhcch7toMybYecn87buCZtIXR3gyXf2XRltjTIkMaSNETTbkLXuJDaJnxzNuiw6wmJjvNFMA1AEWOq3tN9T0TdVccNzR6EHzUp2qg/G95jwQqgaAYjx/NCRRod6OGhlUrwa8NlT+Eu99nc27LPTJxq7WmYjCATJx/weMsVoXBEA4y7RvZwYiVZwOH51d9iJFNtHf58BUC4L97WrQDzKw0FdFtmuQP5IxderL0Ei5RgTWIi5T7I9RAf1RbYTbQU8NdDsywY2LRIOhphPYKQVjr8E39XVg== Received: from SJ0PR03CA0135.namprd03.prod.outlook.com (2603:10b6:a03:33c::20) by DS0PR12MB7996.namprd12.prod.outlook.com (2603:10b6:8:14f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.16; Tue, 15 Oct 2024 15:24:15 +0000 Received: from SJ5PEPF000001D1.namprd05.prod.outlook.com (2603:10b6:a03:33c:cafe::60) by SJ0PR03CA0135.outlook.office365.com (2603:10b6:a03:33c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Tue, 15 Oct 2024 15:24:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SJ5PEPF000001D1.mail.protection.outlook.com (10.167.242.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Tue, 15 Oct 2024 15:24:14 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:04 -0700 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:03 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 15 Oct 2024 08:23:59 -0700 From: Yonatan Maman To: , , , , , , , , , , , , , , , CC: Yonatan Maman , Gal Shalom Subject: [PATCH v1 1/4] mm/hmm: HMM API for P2P DMA to device zone pages Date: Tue, 15 Oct 2024 18:23:45 +0300 Message-ID: <20241015152348.3055360-2-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015152348.3055360-1-ymaman@nvidia.com> References: <20241015152348.3055360-1-ymaman@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001D1:EE_|DS0PR12MB7996:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c904050-d6f0-4e81-1d1c-08dced2d6daf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|36860700013|82310400026|921020; X-Microsoft-Antispam-Message-Info: DfnJDOvTskSHOrWolZIi/Fe9XUtlZpVqH6rLAkZNFCSfnSOYyTNOUgx+XtrISmwUY89Gc+zL21Vh7T3xjcR3ChSlJmAsg4zOKB02SEA77WSbtmw//RKQWyH8srpPFT3op2aJdwEEs2G4Ifs/VLnWDaSdACMsZMFOqtP50pPdQBy0r5BwdyrIt8VgOxyIag1VGiJPg+K6MiVKJAr70mErbkIjIZcEpNi/nPboO/yuxIizy/rFiDXAJXaF4silKSn7ApovMzRo5lZoYuZIKAkWLPMU6fB9owzP2wKs7QvSww2WCVvZBXD9n9l+NdLxUgWf5LNuPg/iVD4vwnyrVauJm1ayMlVI+v4c3Lod6vhGvi8vapM1cX+rodzD1WohhaPJ+t81xqj/9OJp6N7fvzVBd+hx+c+p4Mnaqy+yhEsNRI7LXkS7xWnbt0H1lV9Je2nkUA/dhrYFZ8FxJ0eSPKWl9WiAo7VEUow+05Sc7F5t4pC+janrlXDs7KCe14PuJyiRUCyYGizKN04K1XDSk6zfujiAKdX04Gq+xKhyzjAiMc1SyMtCsyNdsus17NqC6sWV1XeknlVOSybhZx4IGlZWaR1IlIW6qNVRYdgOcCX8sgNGmqEF2fzyGnfO26RwzAWP1gi/KD9PopE+9aFR4Ftrh1sfCKegC4PX09km0G8KQ1of5ELn1N4c08MgAk0zUCEkjFDnYzf4jmuK9WZC3DLLresibMLu3xZVBcwcvhxDDI/H/qMBUjsUxLt4BkgXjgUPI7CnX+cuKpTPUK7Ym7hzXqvFSr+5+nf1VJ60SG2cvTSIHOp746IPt/CywM8v5Q8z935cSBlsTo03yNdQkQgpQxlpDu6EdKUzvj0TFnU+nlaDBhV1sdFCYPkcH44opP7b49q/O1Zn8uAjCAmLnNiPhWVaWsGy7X97cNXMLrV6B5Lzdv0T8bF9N3Tx9n5w2g16aBlJRJP7vWWuDsJBa955lq6hm8hRKLgfsAoICFiFA+fztx/L+ACuAAe00UZUvee1yNk1SMbhzNLom8Qvl9Q0sE1SwDMz7Gn9eJ8PPN6jSkfss5RrWw0c5kBmRru6RISKCBw7P37ond3gLSXlJX7wSRlUu8X8+yaIaoaRiOgV8a0Lqy8b3rJqw4cyHcjFeCfk6r6aAwhGhFBkNVVY0Vqv/DfRipK1YWNoIAiXc0z/d7unK0XmTLhrGq7+0CJdl1FGrLTI4hEYAJbMw+Dqa9Q+3cLL/uIWDcqb0w6+xArwMVZ+Z0l28ANL0k4HdOjgW6MILiaUeYCt/u3R2wSP690X9yVeBqVmI3T4LpIGWQoZ07HN3nA0T9stDf1FaY++67Ucb4BK6fpmms9HSb07yC2R3ZlGt10pISSXgjobJLhn8ZwrGntjAVuK3VybTKNWWFbn X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(36860700013)(82310400026)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2024 15:24:14.9115 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c904050-d6f0-4e81-1d1c-08dced2d6daf X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001D1.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7996 X-Rspam-User: X-Stat-Signature: gnr8jfyeqtbdb3s8ngpn916itiusweop X-Rspamd-Queue-Id: 1955740006 X-Rspamd-Server: rspam02 X-HE-Tag: 1729005851-249677 X-HE-Meta: U2FsdGVkX1+oNghe4jF0suxtHl3lnuwRxuAqCVZqu87wfSpp/H5orhD3WkUEqZMMxUZKTTClf+qiD1kCc5gwx0jLrwtxHD7gxmUNDSh98XHwDxZLE0oSXDnLkZhrRe0SacJBSPyuzmikcXYloOmCVnTdTx6yFkGOoWRYOVf/3t8mZsN6FG7d87HN1My4N6XdHxpbuGBnC+iasoUIaqx5E8W5w0i73mZgfyJZXig94B6PlgvGfsjhLpOZgY0GSYMgzwo5WIN3lH2kl7PuAsYpM+cRjY1lm/RiEkPO2kctCDq5pgTWgfO3ntHvw/9JIbhUE/y06Bo0zANAzMMXUpx1x1z+VTtk08v96lmyszzh/eITvYTrRGMgv/SckeTrkT6KErVnMcxE0orP5N1oiEBjmPp4hJ5tcEOtunuz1JmD6Ax6yBCVjcmPkn/xKy3AuTuYvJVsn5XoKEijPpxKYb+c0AojGx1DnZp2S6Edih7h2cfhgvJ+ufXvQl7cIB8PLASiQk+6b/9An6BH9C9aA2tGuhJG6nu1FaNplp0EVYbJk+w8XLuE1ihNHWJoQf5puvhVleSXL1fpHdQla6gkl0eLeAuXD+voTd+8fNgVz5Du2AtSzi59pfD8zQZtEnITVad85MXAIs0BBP9ZtrtwITEGTuMH6UwU3/T6KGPZUg4A0qVY7AskL3dWuen2pvIAuSTWO8RWNxVP2/l+0uAh4S7KBfIWmP97rH2Do62zb0l1FR9di9eSYauvfm99overGY/b0b2XPNkuojp73W7WpDx4d5h8Et4rqd8TrSfGTkre0zIOVFOPFP0rNw0ek/DJiEF95X+1+qp1jYTXAYiHZSx25CPmLGizsnhzyLTdy9PwCQDzKOLd4k0VrPgtMOjXZx+E6LbdtAwJR4cMPfR6nAo62KLFtShwU/JRERmHA06u2+GLQapYDEqzqlQslz4Ok/xHwPYd+uXReoQcR6eM4n4 VfcnYD59 cwQURPxtSLoT8RTW1DJM6GbXod/zVOx1OaduTqOv6jI7dSMTZxNkS0LXB9mQbXBJOAAkYT2m+YAtFcbm5aprRo7iJNzQuyy0/gJhON45JJ8AK4dzAxDDQL/M30JcNPHKb8tdvvZjqJDuXG+y7yuIKUysJJZIKcIQI8PzDbiBJeYHVkrjA9I8HeP5L/u8JHEkpqmf6+kq93XVrQoRlqAjz8AMKnPcxmVhON68FHWr1zPc4BOOBPUE33IBOKGQ9o5V8Whj5QWWqDfqjLH7VReeAOYwkbYZ7DkVENfhK5ESyU71D7DQv552qxGyLfwahWv3SPo+nU9n7XtkKv0dA5XmE68HS0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yonatan Maman hmm_range_fault() natively triggers a page fault on device private pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly damages performance. Thus, enabling Peer-to- Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead. This change introduces an API to support P2P connections for device private pages by implementing the following: - Leveraging the struct pagemap_ops for P2P Page Callbacks. This callback involves mapping the page to MMIO and returning the corresponding PCI_P2P page. - Utilizing hmm_range_fault for Initializing P2P Connections. The API also adds the HMM_PFN_REQ_TRY_P2P flag option for the hmm_range_fault caller to initialize P2P. If set, hmm_range_fault attempts initializing the P2P connection first, if the owner device supports P2P, using p2p_page. In case of failure or lack of support, hmm_range_fault will continue with the regular flow of migrating the page to RAM. This change does not affect previous use-cases of hmm_range_fault, because both the caller and the page owner must explicitly request and support it to initialize P2P connection. Signed-off-by: Yonatan Maman Reviewed-by: Gal Shalom --- include/linux/hmm.h | 2 ++ include/linux/memremap.h | 7 +++++++ mm/hmm.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 37 insertions(+) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..7154f5ed73a1 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -41,6 +41,8 @@ enum hmm_pfn_flags { /* Input flags */ HMM_PFN_REQ_FAULT = HMM_PFN_VALID, HMM_PFN_REQ_WRITE = HMM_PFN_WRITE, + /* allow returning PCI P2PDMA pages */ + HMM_PFN_REQ_ALLOW_P2P = 1, HMM_PFN_FLAGS = 0xFFUL << HMM_PFN_ORDER_SHIFT, }; diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 3f7143ade32c..0ecfd3d191fa 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -89,6 +89,13 @@ struct dev_pagemap_ops { */ vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf); + /* + * Used for private (un-addressable) device memory only. Return a + * corresponding struct page, that can be mapped to device + * (e.g using dma_map_page) + */ + struct page *(*get_dma_page_for_device)(struct page *private_page); + /* * Handle the memory failure happens on a range of pfns. Notify the * processes who are using these pfns, and try to recover the data on diff --git a/mm/hmm.c b/mm/hmm.c index 7e0229ae4a5a..987dd143d697 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -230,6 +230,8 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, unsigned long cpu_flags; pte_t pte = ptep_get(ptep); uint64_t pfn_req_flags = *hmm_pfn; + struct page *(*get_dma_page_handler)(struct page *private_page); + struct page *dma_page; if (pte_none_mostly(pte)) { required_fault = @@ -257,6 +259,32 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, return 0; } + /* + * P2P for supported pages, and according to caller request + * translate the private page to the match P2P page if it fails + * continue with the regular flow + */ + if (is_device_private_entry(entry)) { + get_dma_page_handler = + pfn_swap_entry_to_page(entry) + ->pgmap->ops->get_dma_page_for_device; + if ((hmm_vma_walk->range->default_flags & + HMM_PFN_REQ_ALLOW_P2P) && + get_dma_page_handler) { + dma_page = get_dma_page_handler( + pfn_swap_entry_to_page(entry)); + if (!IS_ERR(dma_page)) { + cpu_flags = HMM_PFN_VALID; + if (is_writable_device_private_entry( + entry)) + cpu_flags |= HMM_PFN_WRITE; + *hmm_pfn = page_to_pfn(dma_page) | + cpu_flags; + return 0; + } + } + } + required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (!required_fault) { From patchwork Tue Oct 15 15:23:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13836616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1A8ECFC277 for ; Tue, 15 Oct 2024 15:24:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 493566B008C; Tue, 15 Oct 2024 11:24:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 442EC6B0092; Tue, 15 Oct 2024 11:24:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26E656B0093; Tue, 15 Oct 2024 11:24:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 05C256B008C for ; Tue, 15 Oct 2024 11:24:29 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C2FE41A16E8 for ; Tue, 15 Oct 2024 15:24:13 +0000 (UTC) X-FDA: 82676208132.04.8E9CF88 Received: from NAM02-BN1-obe.outbound.protection.outlook.com (mail-bn1nam02on2078.outbound.protection.outlook.com [40.107.212.78]) by imf25.hostedemail.com (Postfix) with ESMTP id 9021FA0015 for ; Tue, 15 Oct 2024 15:24:21 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="l/Ch0etB"; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf25.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.212.78 as permitted sender) smtp.mailfrom=ymaman@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729005709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UeFbwwYOoIblsKPl/s8N30cj3fK8KQ5mY3fN6Hyt39E=; b=DGM6LPtPwC1tvmIw3u6r+4CkXSBOTD2f1mo9R6dmRauJKfeva7w/hTXqlryJgWh1NhHerx 64P+b7+W/xnlwMHUwonwmvPMbnrGHG4EnI0kUzxWjMO9Gty+asn+OphdPEk/U7r06VvaRn 5p+8wEWsareZokt7bSeQKhGSz4ZZSlI= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1729005709; a=rsa-sha256; cv=pass; b=kSLgjboKO5rI5w0bKRzVZwi5anOlGOYYALdZBHEseiY5zJbjhzMyQI8yj5MfdibqOVs06L 9YqG8bcxhG9w9g7tFbFLLxSem4/s+3zWmzZPNln9Mzj8Ygq0A0wC5eh2gvlORKixwP1DOj zCHVOnRZ80lX2n4OQ83WBn+BhjrAXFM= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="l/Ch0etB"; dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf25.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.212.78 as permitted sender) smtp.mailfrom=ymaman@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Axo9k7lU/bIy9B4yThpUBVeCYJlMdDmw6/xJaWqL9w9fhCbH+W5nqJiIodarjUkLLPBKJFy6DJ2qp019YLp4atXwOdghij3CC7tG//kvVOZ5Upcq6tZN6RUZwn49xnKvzt6lHin9KtG2IW86ddvASnJjJNjYRTBe/1kNaenagQHoCIK+I4RLwfgKST4VSqJxvX77WtzQyYZJGkE3kPU6Qku/0PvxCJIjpNRhSIJj+onGGO3ZIN02YItuYgSNeF3pYuVPnlPenaxO5y41sUG0SuqMsyCjXoKFUOxsiyDBrAELobhB5pbkIFn8UpQwL17Dm++8xsV2ncYvwVOxSZrquA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UeFbwwYOoIblsKPl/s8N30cj3fK8KQ5mY3fN6Hyt39E=; b=rSiw0eGG444LRwFEcMfK9dmjnQ8+uSJrhaL81Ml6wI5mqqHtjaKWjA0weB5LFRGrS4UEPstcnJqN/3wTSkMo7NyMIjv/2zanhAK3y419uXeOz8DflVeZmQ53r1SUcMQKkqMmVRLGqOqwYPCV3aV+pb6Gqwo67MWhUNw7qVCtmAFWMsZGjb1xvUq454ZdLZWwR2uuwq6qfKKsWSteoFWJO36178d72nDbWiMC1PxdubgsRh5XebZ8e3Q7BNroz5e/2RZV4L7MiWnqUpxoCKAn7DG2g5v5Kq1BL+yMOgDpTk+9C+xzxBNEnyqtGSQ0mcORavf6v+4hcKjR7DONhJUpUw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UeFbwwYOoIblsKPl/s8N30cj3fK8KQ5mY3fN6Hyt39E=; b=l/Ch0etBOd4wF5ALD2AUZpKN4ptcOReI3rmcpUrvMbsGtpPyfq+A9ri4P+37lEFdXBRG0Zzjj1CeBlVS8PL/blgP56j06RTWUQASraeStsw8KDfRTYOU2/2/Gq8TxXgp2KgTphRF0rakGIfEtXGaiLpeqVl/Fu3vc7f8sh7xVnvxSauDBCEKfOG2Zu5zDHwpcJWRjBmP7nfh408VqoiSsuHmgrNHLjI17oEKkleHALYlmh/0CgtWoDfdTlRJImdRNOAPBZfUrF+dboP9adu5U+SxNm726udgBqu+GeyRoFHEMa0w2W9dCpz1JQ07YpHcrlaAu7gGBOTrIW50N5N2kA== Received: from SJ0PR03CA0142.namprd03.prod.outlook.com (2603:10b6:a03:33c::27) by MW5PR12MB5683.namprd12.prod.outlook.com (2603:10b6:303:1a0::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26; Tue, 15 Oct 2024 15:24:20 +0000 Received: from SJ5PEPF000001D1.namprd05.prod.outlook.com (2603:10b6:a03:33c:cafe::d5) by SJ0PR03CA0142.outlook.office365.com (2603:10b6:a03:33c::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.26 via Frontend Transport; Tue, 15 Oct 2024 15:24:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SJ5PEPF000001D1.mail.protection.outlook.com (10.167.242.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Tue, 15 Oct 2024 15:24:20 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:10 -0700 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:09 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 15 Oct 2024 08:24:05 -0700 From: Yonatan Maman To: , , , , , , , , , , , , , , , CC: Yonatan Maman , Gal Shalom Subject: [PATCH v1 2/4] nouveau/dmem: HMM P2P DMA for private dev pages Date: Tue, 15 Oct 2024 18:23:46 +0300 Message-ID: <20241015152348.3055360-3-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015152348.3055360-1-ymaman@nvidia.com> References: <20241015152348.3055360-1-ymaman@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001D1:EE_|MW5PR12MB5683:EE_ X-MS-Office365-Filtering-Correlation-Id: 86127026-a9c9-4bc0-e657-08dced2d70e0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024|921020; X-Microsoft-Antispam-Message-Info: PmqoIpocYN9j18sLReWwQo763iYXgOiMYJlI/7TqkZCVR6A2hgvNMmSti9sNYDPVwSXPyup/seLgoitXWYAxwPQCdFpNOAJZL5Z8FSzoupgfYtjk1lrHf/s1RZDWToz+Q0fF1MvF5GbJzX6eIeNSfuYbbbOGA26k2aqRrYm3lfbx32jYk+1IYSyg46RLRNu2mAXLBGXGHTb9Lj44f6tpg7+MjCG95qgWHcMbDfcDApcEg6Dtn/p47K9n/O/ySBQn9W/FxoVge4xy+OQkmHp3Dw5Rb5EXSP2GEgSP2jWb3pDQnTr2PRxNMPPiDxD6sTQF4R4rj4v3t5hPgXxVaJwbNSCqoq+Lky8JrU06jf2luStQz4zDfvwEbuj7408wLUv+GN9UfA2ex0kCkdLWhw3Pwa6QuVmVB8+Rs/JDyeSlLKXaR1oAWwL0VGfnYKR9roFpTdVzEdijoH412CtvBOE4CJCb9UHz3NisP6j0tAea+N+ZGKssJDkPyQdA+5aZ/gIvA83u5Ql0vPASoa6rD+Ywp2tLEX1kTy0IQzmF8KZGgD9/X9kS2XzN7YZNOrRlNuK1BdLZZ90hTmyuQtw67kaKTCaVG2rJQallL9mh9kJWGX665YD2KSSqAZ4wMQ03XCm/vwj1rnsrYUNmtIgCKi4mYrCIS+4CFXcaYpNQxlfbKIVt1MF3IAXrdhmPIihSUfkkTINryJoPxim2mqgsBPs6l5PDiD/8p0zxWwKU6UqGcTjBhFobCu/cpJNZ2UnlQrgbE3P5JpTPRzad5Bn8ciKyUFuPXd9P2sm8BBWMvNqaMIf0F3WAwvndfXQB5gpv6quojeJ/TgwT6WvlBnKurvnEElxgFLLOEL+K+fPCK47xbjIK7Wo2WT2Zw++gyscGBtg6Rl7UwgFL8+ICwwogoDYhrn+UDLi4RG2jamsHhWLLEdmDq1FQTL+2DR0rxwnE29OhghWOyKHhFUyM/9LRHpfJwn25MS4BZRB2NPEgGtAcirYduf6FwfswUCJiWKOBx/L9dhronOqBlYOVCSsA+uwgzotbrEqVKd7nVEyrJ/bnGK7lfe/d6DFhGUc4Z3PHBRVL0G4BZVHtuJzUCUxp1oVeNEcWPK23z/A0n7m0jipxXaZlMXJKNfjtInnAD0fkDnHoyehZJ3MN3xg+uNpA8Mf+m0i4o1UfjLNq1cod5W3dAXuwyPcN3TsDns8MHQXHAdQWiuKd+/C1GccbPdYlogbIbkudu83XJdi6l9kuRam+xpIW296ivkg7ofl+AF/IZowHm2gOfLI3KDwZlIVzthCI8sMh5GzixIFGemDs53gnWogop3MiI6O28R1aIcOVQ5xwiAArYuPDK/QDqweqWu37ZdGdGDWW6NWv0SsgY+wvH3644oc6z1kp/dQVeP6E6jJF X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2024 15:24:20.2710 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 86127026-a9c9-4bc0-e657-08dced2d70e0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001D1.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR12MB5683 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9021FA0015 X-Stat-Signature: jcxq9r5tfehu5865ecwkq1fw9c3syjid X-Rspam-User: X-HE-Tag: 1729005861-950450 X-HE-Meta: U2FsdGVkX18BZrsNjOeqrQRUJXnDuEegP59VcwWIMwtTuV4cJr51VyQoOm+e2uoOckg+d8TRO18N6KXX4CiFyosw3YZ+qZX7vrmF96ho4FQNQ6LzHK0AxgAe/MAY9lfAUny7wmn8KGYZKMyjpxxj79oIvfswMr3b/A5YzVXtq71nylEJESsAcrk8x8G9XmUXZk6zWaeBtsVd23LWo5JuvJ0gqAW/ZNE6N/Gg0ojGw5Mkr9hUEa5tO4TnSF+HyvcGgO0G14R1BumhzV5j/mmc8qVF/HiHKT/J92PNMV2IQ/jxpaW/RLjxUj3fqmuOoSGy8FI3LR3TEnLKMoZDgmetMG+hiqnn3vKad/y5XfZoWXvlLS4BBlxsJymlKOzaiuifXJrenk3CqInvn+iaENp8qsa1OuMy3DGag1K2HcBHwUPJouYawh1N+lT++cvq1ZiQL6ufxCIj8cwEO/IJP4IOZ0UhrUVYbdcWAzibM+ycop3q6eaA4IqP+CsJVlUrZdgdwgVWZN5Dh8okZWnyVHizDqlHIa7KgrMuYXfbgw+xr6WSCmn4c2Z0oKWcyL6DZnDqKGesDFnD7x6LVqXIui7nYfdHzMpoG06ShlrtuLssoJjNiwI1Oi8dlHBhpMI1uoV3GOv1MFkfeh4iJfe+raNchEitxvnD9+Rx8fh+/c+c3zsYr1PPx1woDN6HkQMjm6zgGPipjcGLUEd+bb+ll0LX4byPzH0v5/xEi+yHXlCI0J7Frqa8WZQWA9cFWRx43vC/XrB+bnw/R6ubmn5F3SKJwtlG1Qx+4o1aEmTl2+0F6SRE09KYdx/OTevuB9Aq4ct1Zoag4nrzNJV405bM15fgrNP+DqQ2k7KIykAkp8LWoqgSAPb+MWM/He41PxGSww+XnfYWUasiSoEqb0mg/Ilmd4TFpdo2hawqpiqPHZ2i2nS0kj2zdRwdn9Hv2cgyZW8PsQIRVzXEz9tufcEnRFw mUGx0pIj ImaAyK2dWsnBqmIgLcBTPkfnfqZsvf4zSyoW60ftYaJPS5QMnXpfaDliDzcj3TAEUWT2TBAr3BHKrUM3Cm6bgKGnr7dIWJeaUBh2zS3thHofTLKVqnGxjHGLUU5HQps734gyXE36F1TqVFebz1wVXWjgBkkkz6B3i139dRAHFNdNb6T7LLAig0dkXuZoGStPz2aemHtm2rPKSU7WdQHgvHTwbTjeiuMuc/IowIuftk6yMYw6R8xBnX9VWXvAZJvnW/w0z9gvULAgsgPhd4c7wrPPhRfgMTQJTx9Ge+VO0z7qNxW/ggBEnYMNmBQH7en8BNxdeGDAtKrtcKAHmTcAcTq02rSZEr3+1NWOdBvY+HXP1idTMpoWcLVkYjAGzZZpkHt3oSSnepErKXPc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yonatan Maman Enabling Peer-to-Peer DMA (P2P DMA) access in GPU-centric applications is crucial for minimizing data transfer overhead (e.g., for RDMA use- case). This change aims to enable that capability for Nouveau over HMM device private pages. P2P DMA for private device pages allows the GPU to directly exchange data with other devices (e.g., NICs) without needing to traverse system RAM. To fully support Peer-to-Peer for device private pages, the following changes are made: - Introduce struct nouveau_dmem_hmm_p2p within struct nouveau_dmem to manage BAR1 PCI P2P memory. p2p_start_addr holds the virtual address allocated with pci_alloc_p2pmem(), and p2p_size represents the allocated size of the PCI P2P memory. - nouveau_dmem_init - Ensure BAR1 accessibility and assign struct pages (PCI_P2P_PAGE) for all BAR1 pages. Introduce nouveau_alloc_bar1_pci_p2p_mem in nouveau_dmem to expose BAR1 for use as P2P memory via pci_p2pdma_add_resource and implement static allocation and assignment of struct pages using pci_alloc_p2pmem. This function will be called from nouveau_dmem_init, and failure triggers a warning message instead of driver failure. - nouveau_dmem_fini - Ensure BAR1 PCI P2P memory is properly destroyed during driver cleanup. Introduce nouveau_destroy_bar1_pci_p2p_mem to handle freeing of PCI P2P memory associated with Nouveau BAR1. Modify nouveau_dmem_fini to call nouveau_destroy_bar1_pci_p2p_mem. - Implement Nouveau `p2p_page` callback function - Implement BAR1 mapping for the chunk using `io_mem_reserve` if no mapping exists. Retrieve the pre-allocated P2P virtual address and size from `hmm_p2p`. Calculate the page offset within BAR1 and return the corresponding P2P page. Signed-off-by: Yonatan Maman Reviewed-by: Gal Shalom --- drivers/gpu/drm/nouveau/nouveau_dmem.c | 117 ++++++++++++++++++++++++- 1 file changed, 115 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/nouveau/nouveau_dmem.c b/drivers/gpu/drm/nouveau/nouveau_dmem.c index 1a072568cef6..13fb8671f212 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dmem.c +++ b/drivers/gpu/drm/nouveau/nouveau_dmem.c @@ -40,6 +40,9 @@ #include #include #include +#include +#include + /* * FIXME: this is ugly right now we are using TTM to allocate vram and we pin @@ -77,9 +80,15 @@ struct nouveau_dmem_migrate { struct nouveau_channel *chan; }; +struct nouveau_dmem_hmm_p2p { + size_t p2p_size; + void *p2p_start_addr; +}; + struct nouveau_dmem { struct nouveau_drm *drm; struct nouveau_dmem_migrate migrate; + struct nouveau_dmem_hmm_p2p hmm_p2p; struct list_head chunks; struct mutex mutex; struct page *free_pages; @@ -158,6 +167,61 @@ static int nouveau_dmem_copy_one(struct nouveau_drm *drm, struct page *spage, return 0; } +static int nouveau_dmem_bar1_mapping(struct nouveau_bo *nvbo, + unsigned long long *bus_addr) +{ + int ret; + struct ttm_resource *mem = nvbo->bo.resource; + + if (mem->bus.offset) { + *bus_addr = mem->bus.offset; + return 0; + } + + if (PFN_UP(nvbo->bo.base.size) > PFN_UP(nvbo->bo.resource->size)) + return -EINVAL; + + ret = ttm_bo_reserve(&nvbo->bo, false, false, NULL); + if (ret) + return ret; + + ret = nvbo->bo.bdev->funcs->io_mem_reserve(nvbo->bo.bdev, mem); + *bus_addr = mem->bus.offset; + + ttm_bo_unreserve(&nvbo->bo); + return ret; +} + +static struct page *nouveau_dmem_get_dma_page(struct page *private_page) +{ + int ret; + unsigned long long offset_in_chunk, offset_in_bar1; + unsigned long long chunk_bus_addr, page_bus_addr; + unsigned long long bar1_base_addr; + struct nouveau_drm *drm = page_to_drm(private_page); + struct nouveau_bo *nvbo = nouveau_page_to_chunk(private_page)->bo; + struct nvkm_device *nv_device = nvxx_device(drm); + void *p2p_start_addr = drm->dmem->hmm_p2p.p2p_start_addr; + size_t p2p_size = drm->dmem->hmm_p2p.p2p_size; + + bar1_base_addr = nv_device->func->resource_addr(nv_device, 1); + offset_in_chunk = + (page_to_pfn(private_page) << PAGE_SHIFT) - + nouveau_page_to_chunk(private_page)->pagemap.range.start; + + ret = nouveau_dmem_bar1_mapping(nvbo, &chunk_bus_addr); + if (ret) + return ERR_PTR(ret); + + page_bus_addr = chunk_bus_addr + offset_in_chunk; + if (!p2p_size || page_bus_addr > bar1_base_addr + p2p_size || + page_bus_addr < bar1_base_addr) + return ERR_PTR(-ENOMEM); + + offset_in_bar1 = page_bus_addr - bar1_base_addr; + return virt_to_page(p2p_start_addr + offset_in_bar1); +} + static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) { struct nouveau_drm *drm = page_to_drm(vmf->page); @@ -219,8 +283,9 @@ static vm_fault_t nouveau_dmem_migrate_to_ram(struct vm_fault *vmf) } static const struct dev_pagemap_ops nouveau_dmem_pagemap_ops = { - .page_free = nouveau_dmem_page_free, - .migrate_to_ram = nouveau_dmem_migrate_to_ram, + .page_free = nouveau_dmem_page_free, + .migrate_to_ram = nouveau_dmem_migrate_to_ram, + .get_dma_page_for_device = nouveau_dmem_get_dma_page, }; static int @@ -413,14 +478,31 @@ nouveau_dmem_evict_chunk(struct nouveau_dmem_chunk *chunk) kvfree(dma_addrs); } +static void nouveau_destroy_bar1_pci_p2p_mem(struct nouveau_drm *drm, + struct pci_dev *pdev, + void *p2p_start_addr, + size_t p2p_size) +{ + if (p2p_size) + pci_free_p2pmem(pdev, p2p_start_addr, p2p_size); + + NV_INFO(drm, "PCI P2P memory freed(%p)\n", p2p_start_addr); +} + void nouveau_dmem_fini(struct nouveau_drm *drm) { struct nouveau_dmem_chunk *chunk, *tmp; + struct nvkm_device *nv_device = nvxx_device(drm); if (drm->dmem == NULL) return; + nouveau_destroy_bar1_pci_p2p_mem(drm, + nv_device->func->pci(nv_device)->pdev, + drm->dmem->hmm_p2p.p2p_start_addr, + drm->dmem->hmm_p2p.p2p_size); + mutex_lock(&drm->dmem->mutex); list_for_each_entry_safe(chunk, tmp, &drm->dmem->chunks, list) { @@ -586,10 +668,30 @@ nouveau_dmem_migrate_init(struct nouveau_drm *drm) return -ENODEV; } +static int nouveau_alloc_bar1_pci_p2p_mem(struct nouveau_drm *drm, + struct pci_dev *pdev, size_t size, + void **pp2p_start_addr, + size_t *pp2p_size) +{ + int ret; + + ret = pci_p2pdma_add_resource(pdev, 1, size, 0); + if (ret) + return ret; + + *pp2p_start_addr = pci_alloc_p2pmem(pdev, size); + *pp2p_size = (*pp2p_start_addr) ? size : 0; + + NV_INFO(drm, "PCI P2P memory allocated(%p)\n", *pp2p_start_addr); + return 0; +} + void nouveau_dmem_init(struct nouveau_drm *drm) { int ret; + struct nvkm_device *nv_device = nvxx_device(drm); + size_t bar1_size; /* This only make sense on PASCAL or newer */ if (drm->client.device.info.family < NV_DEVICE_INFO_V0_PASCAL) @@ -610,6 +712,17 @@ nouveau_dmem_init(struct nouveau_drm *drm) kfree(drm->dmem); drm->dmem = NULL; } + + /* Expose BAR1 for HMM P2P Memory */ + bar1_size = nv_device->func->resource_size(nv_device, 1); + ret = nouveau_alloc_bar1_pci_p2p_mem(drm, + nv_device->func->pci(nv_device)->pdev, + bar1_size, + &drm->dmem->hmm_p2p.p2p_start_addr, + &drm->dmem->hmm_p2p.p2p_size); + if (ret) + NV_WARN(drm, + "PCI P2P memory allocation failed, HMM P2P won't be supported\n"); } static unsigned long nouveau_dmem_migrate_copy_one(struct nouveau_drm *drm, From patchwork Tue Oct 15 15:23:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13836617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA2F4D1D878 for ; Tue, 15 Oct 2024 15:24:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5CB2A6B0093; Tue, 15 Oct 2024 11:24:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57B876B0095; Tue, 15 Oct 2024 11:24:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F4B96B0096; Tue, 15 Oct 2024 11:24:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1DD016B0093 for ; Tue, 15 Oct 2024 11:24:40 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id ABAC91214A3 for ; Tue, 15 Oct 2024 15:24:31 +0000 (UTC) X-FDA: 82676208468.12.86F05B6 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2057.outbound.protection.outlook.com [40.107.243.57]) by imf14.hostedemail.com (Postfix) with ESMTP id 1440E10000C for ; Tue, 15 Oct 2024 15:24:27 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=LHMbqxL0; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf14.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.243.57 as permitted sender) smtp.mailfrom=ymaman@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729005734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=edElnGBClLAiAIzr2HyoPLhOZQadZKfGUs1gCkMRJUw=; b=1JwILcrM5eUZxGx2IhM5bjA8AZQnyb8K5tOG1RKadkXoHOi6XvkJzB9d1zN/Dfos/NP1ke EAVoR/9MAY088/B8laysGzKPhVKX2M0DKV8oiZ0Iz9qBbUXFvoe0C4kvtow8qt28hnrOLG 0+8cpIPOJEjvoMRZGAntLVVlYMCsivY= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1729005734; a=rsa-sha256; cv=pass; b=M54S+sKUHRbSTXZE+RmGnJXbwS+JYYoAvGLhAaB0SLkp0N/CpzYV2WxnBhjQ53kgbCcyZd ytOfVnjHVqZLP40vRAKUjlglf3/qUQRhS5m7qnDuPkiC4vyJSCZkveTlMV/HIpIju+XCHa rcXlVfHoDAKlOq1XzX+GOcbqPh3ElEg= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=LHMbqxL0; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf14.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.243.57 as permitted sender) smtp.mailfrom=ymaman@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BczGfVw2kiHqT6X4uX2ZvTRAN2vzGXTe2+5IPBG30OJ16PdJPVpTADLe+TpijcTjXA5ydE99zjrrqES8pMfKl/KgSmLIqPRnWl/yVM4WCnU2gkyVfCFe0P/SVFJ37Zj6YxWwUlw54QJfd0EpnEm/ieRdg1RVWEzpAVkJ+oSdFV3GtnqDT04r1Vw8cydMUAiNCWn1i7p4OHcMgyh51NRL+hCvZp7+FxNDx1Rvtj+ALjonYIhJ06dh2qZf1WG67gyQJE+WgmjFvLFuMUSfWK8HoY5LutpN5Q+B39xULHeIOy+oWO8Zhpnoyw01uxXJBteCo7Ge8Hicbi4iYRUgowh81g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=edElnGBClLAiAIzr2HyoPLhOZQadZKfGUs1gCkMRJUw=; b=KZOH1lJzBDItjPztpavhm18cE9S/7jgbhWjC0Sxni1NqXwhw3sx1ourLHhtClryGVN9sz/2yfV//5DA/ALWl+wyAaBXDCYOYfvMTfIOSzkHKJXD+3/+x313VRoBNKLDpL7V8/YNYQD8uYfQ6GaNnpAM0/XxH7kVwnjVks55GtjOaGsrEs1DHXjnkXSosgTTLYgUGxjGiYIDFvkm3Bm9aPnIIYuXssgT3jHm6uAzsiHGwo6M2aR/ZKcf521FMc7HM0nnFaO3Wwmrf8AZitYyh3NUZmBNMANMRr3x26OJWnyLjMNhSmD8ALzkWVISjJ3Ep8d0xtkNIxTyeTSzRHZlDEA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=edElnGBClLAiAIzr2HyoPLhOZQadZKfGUs1gCkMRJUw=; b=LHMbqxL0uTpeNiT6Y/2KX/xEVbGKUNzG1aqI0iMpcbvloHdzx2b+sdIz2XsBk/cLjZLE6guetblQskzqpPeJd3uW7QDSyOB6sd9tmNNOcKccFampyhtrpDlMprCmSRRq2JvNj3QUHoeM4ZMTmvLwjdJZTjRqRfFc2VxCsmlPoSqKB6HKahuRsGjt14iwxIkbwajThE8YAst5/EmuT/0A2CU43BPT44u4959oy2NzYJPLB2Kad96VmR9mUrbkV07pJdEIfnOz8pesVrwZpb2n+Tqj2qkP8QD1Tw5heV+Ejqmmut3gCqe6Nq0znjzgbnFKe/wexCfG+u9Dsa1pAq7Cuw== Received: from MW2PR16CA0034.namprd16.prod.outlook.com (2603:10b6:907::47) by PH8PR12MB6819.namprd12.prod.outlook.com (2603:10b6:510:1ca::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8026.16; Tue, 15 Oct 2024 15:24:31 +0000 Received: from SJ5PEPF000001D0.namprd05.prod.outlook.com (2603:10b6:907:0:cafe::55) by MW2PR16CA0034.outlook.office365.com (2603:10b6:907::47) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.27 via Frontend Transport; Tue, 15 Oct 2024 15:24:30 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SJ5PEPF000001D0.mail.protection.outlook.com (10.167.242.52) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Tue, 15 Oct 2024 15:24:30 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:15 -0700 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:15 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 15 Oct 2024 08:24:11 -0700 From: Yonatan Maman To: , , , , , , , , , , , , , , , CC: Yonatan Maman , Gal Shalom Subject: [PATCH v1 3/4] IB/core: P2P DMA for device private pages Date: Tue, 15 Oct 2024 18:23:47 +0300 Message-ID: <20241015152348.3055360-4-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015152348.3055360-1-ymaman@nvidia.com> References: <20241015152348.3055360-1-ymaman@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001D0:EE_|PH8PR12MB6819:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e92872e-9f3d-418a-f069-08dced2d7731 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|376014|82310400026|7416014|1800799024|921020; X-Microsoft-Antispam-Message-Info: 41tq66SQC9BvA+ZxClDXTtEFy2mvOJ0ssmBFrOYs1l89eCSNlpsFKp2jCJNsn0hP/BgSFrTPTFiqVg5HXJG0lYHl1bgIt+FB48h5mM9Vyookr3EjXC+hQcOJGS7j8fAtLKyFKwJf3N8nILhwtJtOSNGniWTsv5S8556K8DxF0fwS1auMhqcUIZz2x9DzL1QZkc5kueW24woCSFEG5iyHtjlIWGaGhLRO1QIroVIG1lkOkM6/8tQ8Xiu+Vls8yapMhtJ29Tf89HMr0qLXkpopwo+m6eNJl0pZkOHNlZahjfyRe3IiiNRoynSZWoOgPQUn4nj/nCmiwoQTTTjZb12bmegFtgeJNz0mU9i2neyO7BzR0BjPR85AmOMKYOHPE0tC09AHKcRLffQRUvFF4SQnT9NtDGCNlzMC9/ME4zyrug9M0dAlA1grjERgdAGSE804EE1LVXUVG/S5IyIY5V/HmT6wnyD7Pxvy3gzhvTWUH5q+O5kah163U3DlO6MvkpR3rFZEOdVdcgUHAf7uzEPNCuw/UwtfcTlcLOx6/jiI52W/QfYdS6KJNfC/TChQyjhRqygXNuluJQJf7E6plzm79aArOkL7zvRDlcIBiXTdc2PS1nPp9/X/XAWZRm90FPhTAF8MaTdt4FqgluQ+aF61Rv/Sh79JjjOaqXeaQ+Slw27uBJe4eJrxuLaaArFwfZg/A1Y6X6fG2c89Nob7qg9u0Iru15toAR4hLXiBI5LNjDCQ8oDhR6X+670D264LA2ObTJMtEtwWY8igldKlOicVfBZ8b1m4K34UR2gGVkn9RQagyIwsS3y+Qq2I4CSpJRd74bEwZkOGPWolFwlB4+8kBmofrGkovawhXJ5vh97zT4OisZDxhY0QhDOErv9OwOsH3lfEPbt8f6whfozVj62pxcsq01U0zJjVZR9X5OcwnwE1SwTuoP1S0ZEh9YEwpxv/Hea97QUh6LFr4z2fOZIyu1gR+5hGTjrPRyGNUpR8tqT6xjVSiyAM2dSIZFx6vM3NL9dR7iubQtySU0d6LKaTnUijeqeokB+SpcaihYn7rWFyKgMg/xs6m1/EmEeqGqjZ90G04hlxWLhOk9P5vp1KqjETdab4mFKHbUC1fkDcvIbHwOOtGEE9QCFQ/OiJfr88Op4LLEvxgNcyGDqvcLnN1RNXrKbqQCAWEd+CH+hmRqZyVg29VXQF/jnYoBIr2FcygahRh1+74XoqBy2xlsQxkQbsOf0oO3L8vYQNH1KHzVWHf83VIBxHEa9UHDPETouAAV/IMt4ZjCAIuwh9JjvFsUFx2YH9nzvBr7FViET4LAcV/kYH37FLPobzcjfeCvuQM+Ffy1I2w+dOpDvHTYjCifKuNv/ouqOp3z00lxNKZmZCbKdylMixQdGZplqvt/9u X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(36860700013)(376014)(82310400026)(7416014)(1800799024)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2024 15:24:30.8684 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3e92872e-9f3d-418a-f069-08dced2d7731 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001D0.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6819 X-Rspamd-Queue-Id: 1440E10000C X-Stat-Signature: wpr8g73a5ssb9zrw8rt9ywg4krebrm71 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1729005867-295373 X-HE-Meta: U2FsdGVkX1++IWgm/mvA55Gh/P8G5kIzJn9jTY8qxnD1SKwR2FL6vEaRGRLDHcuNeU056Z7oUgFa42AG05HZzK42bJo2YDl1F7C00TKvZePKlBCo7X7kbw59If5jGoXHmO11/IVLA/+s0i/Jw9F6T1MKBSrA3a8E+bOzjEIaPCcY1rb0xN9UefViSArKETXh3b2GlSvOpwVbliiAzzaAbGaCRGnPX4Jsi6Caj8jKwa5TEeBNoovXRP3FAbpW7yu88RzTJloXhparFQs1iIdYkE+XREv74Pt5qbdugHCRZOGUc8LLwW7FQd3kIhKWopDAMLpE87/Dk7uCv6XVTdeFTsIMlNlx+4ZXXZJwYGtusW1krHXE3fet/ZFCptLX2Rt9rYbgIDEsSYegh3seQjR+IIFWej1luOe6EcnLux5pTXDn5mX8dxknNbWq0TFKzjrYa0NOqqT3cjjZhsfRY7AXKWwjcqZ8WMRZQUeE6VjreOM6KeY5eEH4x4rGW6Bbm0b2SbtcJyyqLmdXq+VVoNMO3iOPS74RwUjlv5NkBvCLBjUgPkkyy3cRAawSXXRAYW8LhCw13MMS2g3OP/Dwlr+lXy2TlG5ALKpZrL2K8QrfbOfF4IPyIwyJY7nIUAtLG2sFk5beXYo7ypwscoZPRS4L+FTwTVWw0TztYMC4Kuh7dvnPyI/lmYkOHinfaLDq/CZkKw2fmbD6L+VkmoXDPK4SJx9ajRw7iooVlZYTGN9/haaA8opTyuT1sDyvMTDosB91VTmgaOW1rw++gdRfeJ6GXecGPedT28FCGwwYG6LfklhAY6+OGVr0MMO/502oszEFv8QJi7lHflUp9a85CMyMXizC8INcvvuortNwxUenXx8gFi+/TxnQGYzmIq8d++SHnDgRK6tthzDCdl8cLvGqU4h2S8yQMp7g6kH04ozBizHM+PCOQxiRSZ7uaC2TlSTPkQWd9CbUqXPvHBovoFG f7wmUFWK nweqHDvS17MrCp77GQWHF0naey1N741t5oY9fqBGHBnvEvtpZsF+tPgXPivkuVUS4HuIyUnuKSlcoq1GDF3ozy23zavzCUK48hE8dMs24JxhcDuqykVX6C2e4PO3hSRS44tycm3E768l0B0PEvk7BgVLj3pWoAabVe2nSeYzCIZEyaROhmdi2Eq6kwe/Pg4aqmORRWMw9tWT3gxZxvU2R2p50zKCsBsBqz3O9zrOjDOJxK4vsG8Oqwej8yOSWOhLU66+0lknzSvMRu2Ej2UMQ668yOrJRslF32v9l9arTX6fF4+oZcNnQkkIlJ18JpFS4PNga3zNmpztBZPnPm5ukx+zvKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yonatan Maman Add Peer-to-Peer (P2P) DMA request for hmm_range_fault calling, utilizing capabilities introduced in mm/hmm. By setting range.default_flags to HMM_PFN_REQ_FAULT | HMM_PFN_REQ_TRY_P2P, HMM attempts to initiate P2P DMA connections for device private pages (instead of page fault handling). This enhancement utilizes P2P DMA to reduce performance overhead during data migration between devices (e.g., GPU) and system memory, providing performance benefits for GPU-centric applications that utilize RDMA and device private pages. Signed-off-by: Yonatan Maman Reviewed-by: Gal Shalom --- drivers/infiniband/core/umem_odp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index e9fa22d31c23..1f6498d26df4 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -381,7 +381,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt, pfn_start_idx = (range.start - ib_umem_start(umem_odp)) >> PAGE_SHIFT; num_pfns = (range.end - range.start) >> PAGE_SHIFT; if (fault) { - range.default_flags = HMM_PFN_REQ_FAULT; + range.default_flags = HMM_PFN_REQ_FAULT | HMM_PFN_REQ_ALLOW_P2P; if (access_mask & ODP_WRITE_ALLOWED_BIT) range.default_flags |= HMM_PFN_REQ_WRITE; From patchwork Tue Oct 15 15:23:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yonatan Maman X-Patchwork-Id: 13836618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D16B6D1D878 for ; Tue, 15 Oct 2024 15:24:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6192D6B0096; Tue, 15 Oct 2024 11:24:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C9D76B0098; Tue, 15 Oct 2024 11:24:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 444426B0099; Tue, 15 Oct 2024 11:24:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 24C626B0096 for ; Tue, 15 Oct 2024 11:24:45 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B53991416B6 for ; Tue, 15 Oct 2024 15:24:35 +0000 (UTC) X-FDA: 82676208552.09.F58F972 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2045.outbound.protection.outlook.com [40.107.244.45]) by imf27.hostedemail.com (Postfix) with ESMTP id 642BD40012 for ; Tue, 15 Oct 2024 15:24:34 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=Lh6jSU5c; spf=pass (imf27.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.244.45 as permitted sender) smtp.mailfrom=ymaman@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729005834; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E/lrVl1VFYPdVp8Dnfq0tyenbmZBRBm95p50EAMlqC8=; b=TjYYNGFl3y0tHP32MnY2YA18PMXaFNOChEgPz/bSc155x1ZorFbUFrLrmPnEeg9GGOp24I pdgtdoSp057dtYiZqBW9VeopY24OX+EIYDTH2/dD16/TnV65Hte3codV2A8kQ1YgtzM4RQ yD7s0WyRRlBOLvULOQv/lTZBhwCgEgU= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=Lh6jSU5c; spf=pass (imf27.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.244.45 as permitted sender) smtp.mailfrom=ymaman@nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1729005834; a=rsa-sha256; cv=pass; b=IfK2WVcMVAzQw+HPBsH12rx6XJZX9deqmLi9mxobelX6SPUCSq/f2p37v6S+def0jfHLgE bqZd2DcjJuD4p3NRbILdNj2eUy6nGSbgHdp7On4my9F8e/FdybPQo4M8EkhQYNHnZ2bQFP rx0o2ozyshkLAXvEz1rP+Pn7aInf7Js= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=N9YJsP9XsJbz7uAP2K0dp6xNI/hDiO6qGTeK5Uw1Q0rOgi+uNQvSh3vIIFfk0a6cmqCMirB3A+OmDiUorbUGTZXxQBkkQR9Jy/U+JA/ESqvu4DTUk0f6oDqldadlTT2pKUB6v9gpH6W/94pp4PNOrPZ/FtuRxIPNyt3TriwomJcO9vAIKI60ojWMCF072FwSdkZVW2hG09NslJSr4AOOEBKnKaOCExtv+MfCPYYGmemjO+8vz6fj2Xs0mYkV8he/3rtxuZ474TzLi9c//rpZAN+waW8Jxdw3oHEWC2vT4R8TvO3HmBGg6H9a+Xhlff9Ntmp3PODNl9OpF0WQYUn84Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E/lrVl1VFYPdVp8Dnfq0tyenbmZBRBm95p50EAMlqC8=; b=H1+AM++YtgfGPwve3GXUHlwYEq71HTAIbZRXhwuoSmmhO01BhmDNcFzXcFg2jzi1Bq5AJlY8iVVatCDZ3e/cbZtXMWBv7ByijLKuJaJ1aI18cqz31C0staQxjr9wbScJ3YckPF14WWNqBfK1KnejStmq7VR2j33enI0ZUhNMxafVdIUshIAhM89E7BwZN2i/gwttJh1+iqTVzSnfcytCb+Yo6zkj8fFZWI4ldaZkxoY7R6okgNrDQF8hvBoFFHyS2f+5BoFm6oNMnrXa7q044t7JeF5d/Gijk3/2oIpqOKSRn/CP5p+XBexANTq9+Dc+JQEoRtve11HScZ2Ya/+s5g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E/lrVl1VFYPdVp8Dnfq0tyenbmZBRBm95p50EAMlqC8=; b=Lh6jSU5cq+AJm3eSwpR5vM3WKFGMiiI0ArrgHRcGy+G26xrIxj5Qm2jXmHwzrHFsLyUg+qfbp4/Ft9Nvp+xWOochIVPJW9EmpinCVjW+oW+d/uCloP9K0zYyFVl9TedC2EFBHJqwztMPmzpbWuWzjYnvtnXcGgbXvhDI0EwWH1Xro72YaVvhY0xuHNiiXilJxBfNnuncUzQ7swa0a4WgtwC2FRTRsmx69EdaBOwYejBhvTiPHKMbsR9hpWp0BQ7stxBAjRH4SlxmIycccI2fCxN1s959plO7279ZwxlXd2xlRVYmCnqrpkemIC+TgRjbuhoHB+kdXF1zXBk/LVmMGg== Received: from BN9PR03CA0263.namprd03.prod.outlook.com (2603:10b6:408:ff::28) by IA0PR12MB8864.namprd12.prod.outlook.com (2603:10b6:208:485::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.18; Tue, 15 Oct 2024 15:24:35 +0000 Received: from BN3PEPF0000B36D.namprd21.prod.outlook.com (2603:10b6:408:ff:cafe::1c) by BN9PR03CA0263.outlook.office365.com (2603:10b6:408:ff::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.18 via Frontend Transport; Tue, 15 Oct 2024 15:24:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.232) by BN3PEPF0000B36D.mail.protection.outlook.com (10.167.243.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.1 via Frontend Transport; Tue, 15 Oct 2024 15:24:34 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:21 -0700 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:20 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 15 Oct 2024 08:24:17 -0700 From: Yonatan Maman To: , , , , , , , , , , , , , , , CC: Yonatan Maman , Gal Shalom Subject: [PATCH v1 4/4] RDMA/mlx5: Enabling ATS for ODP memory Date: Tue, 15 Oct 2024 18:23:48 +0300 Message-ID: <20241015152348.3055360-5-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015152348.3055360-1-ymaman@nvidia.com> References: <20241015152348.3055360-1-ymaman@nvidia.com> MIME-Version: 1.0 X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B36D:EE_|IA0PR12MB8864:EE_ X-MS-Office365-Filtering-Correlation-Id: 831e4da0-c848-4740-6d30-08dced2d799a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|7416014|36860700013|82310400026|921020; X-Microsoft-Antispam-Message-Info: 4V6EdxkGyMmUbDNSLHuCOf4a7m//JN836m3OuI9qYscOVWt+U1KV+VogYCWs5hF6D44MvDWdOQ0C7eYTaZCOy8NbHzvaIrv0AssPtniQBp/a2EmvngZRJVVkErCMv42+1DZfO9m9GbLVNurLr7XL3vNp7FbzDuegoZZA0dtZGMnLCIIlc8QHFjvjcAjObQgwmGfD+kqpUlkAMXuM+Qbt0KUJQ2QDp6s4UUfQdRPb5YbtEOYb+BTjBzL7mHrPnmLxclt2sAkn5CSW8HwV/VzOOcf7GQ77hqWBrFuGvBS1NCFQFtL4Ld9+Wtv9ieBydDY59W9biykBJABEWo2iz7D0Ot2Wc0NtE2BJpOYxkZf9SVjH7KviVTnncewtNe0zatpdI9zTj5yq9Cs5QjymV7l/ev4AzLWjtsH6Uj3hNap8Fp7X0bPHseR+oqsS7b98JvRHZqaFk8zaSr4nqkGB12v5YzITQR1P8LhMQpax5+PIp3gKBbK5mTlbbL3JWl1S8rRaVHAgD4vAfO0fzEJS7EKZe5ZeGJpng8GOh6E2gS0Dl31nsQb5bVWOEFG7mJCUtlw0IOqbpffRECcanPk1zfVQ3x2C4s0ORa6CMT4mQ1DODPzLUPa8tqiUCiqVAIigjj0e8mSuP9JOWA1ym02pJkelDoCavGeYOtc3oOhjl/Z8fam6tAG+chcIxdRlfYSfsU4jTUax8anRN6QLr9qkB5J8hVHki1SUVwhigcQylrGIGC60E3OAPGC3sdyKYrU2/iugyiy1rpbuhzd8vMCvJoBh6x9Lzz0zxvxPLzBPFVhkujhPVTr/nfYRulw2jSchtWciOsDEVNwlsH+jLSWWnFELrn3gapfNepkjcn2y3mimX5UK/HfKn+AT1V5MXx3BeYq5hrrbMSbdFKmSDGNcgrT0HobxwO/avBZyed6hbPPin+LvVlaJK7rwSzk6lKkI2ON0jaEwbByphS/wZzQjz7eEZne7KCjv+FEU71nxJpSPJ+mdHYfoG4xJDnEEOy57HC9xYGblU8qKM1XVrDTMYdZL+HkZKuzkdKCVNz4whOhf8GaeH+RAeI6eEIYZm2Z/SiBAcn3BBxpfqslJLQsPTMLbZ1ZBTwliDBtBI0TA91CJ17TUCPx3yWecidpw3bFKgrobR8qNDK7Laz0uYv91rZENBXNikOv9/Cx8quSo1cyuGDlfisgXhs7X8Hoa6nkiPAQaHEc72YjH2X4ZM50ywA0t/NFBgke8porAC9bTCRY8YCZIRbs4JPttKHLk+RQnyQiUHlaxsCfK+bW2pJza7xAPOzv7Yg24cDFfjYUethjgIEMPi+uMxm3O30NKAcshzQAkL1bQY3TGAFmoRXC4Ndgs8Fi4oxmb3g0QVvz1FuHnOHHrD2G0M4UXMmNS255FALDz X-Forefront-Antispam-Report: CIP:216.228.118.232;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge1.nvidia.com;CAT:NONE;SFS:(13230040)(376014)(1800799024)(7416014)(36860700013)(82310400026)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2024 15:24:34.7211 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 831e4da0-c848-4740-6d30-08dced2d799a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B36D.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8864 X-Rspam-User: X-Stat-Signature: s6uqng456nn8xr3dhafzjj5xrifcnsdg X-Rspamd-Queue-Id: 642BD40012 X-Rspamd-Server: rspam11 X-HE-Tag: 1729005874-724489 X-HE-Meta: U2FsdGVkX1/2qEmm16WxdKsLt0QO7NzAyEIN4AsHwLQ1V/471pNWFpw0xuwzOzIySOrYyUgxSqILUFrK1K/0vHkek7iSNr1DP6hc3g12M/05g3sQ4f9MQJ21OJHiKpFXHummHKCOlWhGiJv9DcpZGV/JtGG1YQf/kZ1HHaK1Jjf0T/E3tSsbrJdKiYnYlMzw8ZRZ+9u7Er7KbOpRnYQZOzx0lcdnszqH0FcJhq54va0Ek3wpn5hDmsZSGVc7ca6uJbvfGW4kDyRkQUXcsFyXgAbDP8R1Vb8m9vc/RZDbSl2R4zBXdRrkG7+tC7mLlZ93ms2NyCCH/lquoAa7xQTtvJ/YwpYfiu/yYqIKTNsMEaCQLK2dUVuHQRn8CSb0bYd2KlsJQyB6gV4JbTCfa4/7JFoKPk2kCVTBZgp0Cu7PAnSPBRMhj3KReT9/PacSsiXXWlQvaCGUxB05jLciag/akrEbe4AQbMOAKFQ/dOb8NyRFnSVgFQ+RwzFFjRcAz+QA6knUzX3po9il7J5b29hesq6x0ofK9fOLNQcuE4NjlhstSD73szOcVFdg4tTnmFDdp3YByFq2gl5zUUBO2XE9XLrYYbeQP+TjvxIHuiaFRz9tzTVc9Xi0Vj3R/od3GSXSgcZZRImEG+Dmh4brkSHBMwAUvxGb6gYtRgMA2IWdSP9ve4GlCC2F8eFgVxcpEpdg2H2NXTNHxT3/9wkleRJiXheIn7RaSCTf8C65EOigH1nLHg8AElWkjOrrA5Ss1kNQXD9CFrIsfWbZe4RmFMvOwQg5vcHREF4V+/5n0UqPu/6D6cxGhfTLsyWDPynLV+ioj/i7uixt9gOegwXOZCB78wW1jU9+CQHLhL6UY0/Tji3Q2VI/vqLAWsGNe5MTYnN1cokiEY+a838VPlsRtF/CrD3RyxC82coEiomHHXAMT45KK8RD4qIBiVz1auDIkLW7Lkf0IvSSPuvXN7xFpxB yEv+JljD PrGXFtPzVgPSZKxT6eIaY9dLvZm1mYP+b4IXpljiKMPPaPZgLYBrRLNr/5e207B+uX1Qpv9BM9QrGSzkgJoe4iGAFA6e/HzEQS5abBFIFje9Ps2MaEkG5n4EKLRPkpRCSIrN4fFYpkD+pUB74r+wiRapBfbRLZf8d+L+bCDNIaoE1SIj1yU1sPUNE2HLdWrneDBB55svs8qoYRvfSMq5Dn+1BPqAbfg65ZA99AnHA+RRCu1mg+Qp5QOAwSA5lVx9K6fK6Z8VJVv5daKmqhkVuFv8v64xSHgi7X+/kAKIcxZm0TPZO+n2b4k+0fLQ7ePRkYGMVagYgPCqZMY6VftFsAyuM1w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yonatan Maman ATS (Address Translation Services) mainly utilized to optimize PCI Peer-to-Peer transfers and prevent bus failures. This change employed ATS usage for ODP memory, to optimize DMA P2P for ODP memory. (e.g DMA P2P for private device pages - ODP memory). Signed-off-by: Yonatan Maman Reviewed-by: Gal Shalom --- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 23fd72f7f63d..55ccbc7d9aa3 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -1701,9 +1701,9 @@ static inline bool rt_supported(int ts_cap) static inline bool mlx5_umem_needs_ats(struct mlx5_ib_dev *dev, struct ib_umem *umem, int access_flags) { - if (!MLX5_CAP_GEN(dev->mdev, ats) || !umem->is_dmabuf) - return false; - return access_flags & IB_ACCESS_RELAXED_ORDERING; + if (MLX5_CAP_GEN(dev->mdev, ats) && (umem->is_dmabuf || umem->is_odp)) + return access_flags & IB_ACCESS_RELAXED_ORDERING; + return false; } int set_roce_addr(struct mlx5_ib_dev *dev, u32 port_num,