From patchwork Tue Oct 15 11:13:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Thomas Hellstrom X-Patchwork-Id: 13836209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5872CFC28A for ; Tue, 15 Oct 2024 11:14:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3909F6B0089; Tue, 15 Oct 2024 07:14:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 341EA6B008A; Tue, 15 Oct 2024 07:14:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 208B66B008C; Tue, 15 Oct 2024 07:14:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0554D6B0089 for ; Tue, 15 Oct 2024 07:14:12 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C1DC812153B for ; Tue, 15 Oct 2024 11:14:04 +0000 (UTC) X-FDA: 82675577418.02.BF0E33D Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.18]) by imf15.hostedemail.com (Postfix) with ESMTP id C5D0BA0016 for ; Tue, 15 Oct 2024 11:14:02 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g2kcnMoC; spf=none (imf15.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 192.198.163.18) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728990778; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=ovUX9wDDtKwfKw6KRhu4JKYLHTZadz5mwudfDlmTLi8=; b=6GxZXQh8NUj3TxKFRIrPWHRp39FvDfK6iWz2+pLAR4bOL0VwV6idWYifRRs/4RDduQZ7qp SftqcVXT/dMNbb+Oo5dpLQoQO86/vTJDKQeswudt+Z5YObu0OzQaYATKziOW175SUDzeSJ ISSBR35hlFb9QOC64lBtQ3XrvzkYkfA= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g2kcnMoC; spf=none (imf15.hostedemail.com: domain of thomas.hellstrom@linux.intel.com has no SPF policy when checking 192.198.163.18) smtp.mailfrom=thomas.hellstrom@linux.intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728990778; a=rsa-sha256; cv=none; b=hLuMs05Ghue/QcqC84xmOR6b97rjzZLHrHMfH14/QIJCZPQs6ll5QmjT9qtUrDqJgAtJ7+ jOiM/Z2CEpXDI9tGyJ0CdTBAp5hd+o1z1W0zMH0+/Wpx19u+DuNxk2XqTqETO8g54tWGWH UewpaFHCPV0dN7ejS5FhGK3cewYcOeg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1728990850; x=1760526850; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=kFWXuUejgX7TsesV5BIyhjvVFVuTGp7EAJnDBugnccs=; b=g2kcnMoCSa29vNU+rIceNHMv9v+zS+0aGxBAnWScJiK0AGV+3sDezBp7 LF8t+NelfmQTbX9w6plw4oQ1NSLHvPM+Z/qgz+PtVm1nUeF28ZhvdRaC2 84OOHfg1N6VOxInfoShtK+RGKxWXFaYoY5GGrN2b+rqmhVIv/B7q1dxg8 o/TsT1yOMl7BV0GByCPlpWIvR0GurjxFthr+FSvURPY1jW8CC6tOnQfBJ AQxAI5KRuSoKFcb+xdkkq6YB8HUMujSL49PK9uKvXo/wlm07AD4jSJ3dk m6FlZPT5P82lcjgZl0Pzda3miwmQ7Yp4YIB22QPPevcEU9a2eUoAwIMxB A==; X-CSE-ConnectionGUID: xO/appAISxCxs6NPerqygQ== X-CSE-MsgGUID: Ws5uALZLT/idIG8gcom0aw== X-IronPort-AV: E=McAfee;i="6700,10204,11225"; a="27823413" X-IronPort-AV: E=Sophos;i="6.11,204,1725346800"; d="scan'208";a="27823413" Received: from fmviesa003.fm.intel.com ([10.60.135.143]) by fmvoesa112.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2024 04:14:10 -0700 X-CSE-ConnectionGUID: 6rh/y1YTTTGJQYJDzX7OCA== X-CSE-MsgGUID: JEFQCYmhToiybWHZDFjOEA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,204,1725346800"; d="scan'208";a="81840388" Received: from cpetruta-mobl1.ger.corp.intel.com (HELO fedora..) ([10.245.246.43]) by fmviesa003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 Oct 2024 04:14:07 -0700 From: =?utf-8?q?Thomas_Hellstr=C3=B6m?= To: intel-xe@lists.freedesktop.org Cc: =?utf-8?q?Thomas_Hellstr=C3=B6m?= , Matthew Brost , Jason Gunthorpe , Simona Vetter , DRI-devel , Linux Memory Management List , LKML Subject: [RFC PATCH] mm/hmm, mm/migrate_device: Allow p2p access and p2p migration Date: Tue, 15 Oct 2024 13:13:22 +0200 Message-ID: <20241015111322.97514-1-thomas.hellstrom@linux.intel.com> X-Mailer: git-send-email 2.46.0 MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: C5D0BA0016 X-Stat-Signature: xe69r14mki5kh86stoxkbxsggzshcdzf X-HE-Tag: 1728990842-307676 X-HE-Meta: U2FsdGVkX19REFMlVAOgs6WZrsbZ1Q4xFMsxauIDs/QTv89j2ORTfu7nfXlLRSofN2RqsA2EsJChnhVT730GI6WeLFtrIFBfZMA0v55T7HOK2TfdJo2DIwkrSHPTeN64/2fkTaUn/sSyrE/3/Td3/tPNnpvqbxOJTl9jnSq5sTG5Nt4cbLRD6nKX1hsmuFI7x5/uIyvBj8/Ju35aYmtHKCxNVE17u+nMdctZCSW8kbCsMefjlDnbvvyr8GC128NdBExOF8MbJoZQ+eEkxIzgQukLtWuaMYxFX4rzUZWL62kE0EJXrwOCEXJ43ZwL1oes9p6q9p37g6+gM9r90O3fC2PHe3n209ZJVa3thOmzOYMl86pK9FC1dD3RJWeIAOCZ5ZdYZeDHnpNLKhaRLipFWXJr7iUAXOLrt69toMjuU6s1fe6/EYv8g1rRC6j1rgeTbQcS0H0VmoPL929Y2kBV+Huw34b0XHWv65KtB/UWzMI3Iau04651YH48n4bUlmx2Yx/IQGmGwHXfAMy0rhoLIx6W/i5518AC1kUcL+YX7FpAW8QMTIdUAh49pzx7+/gd8eWmSoYVFdj+i6V37VVC5SyrsDJwmEnqLEhb65TNP/gQ96h10ivW/plkyc2kKx2fb81LDZ555tItUVhbJQd8HQEtMqcKDEf814T1Cz6cUpsLaQvIOrRUXshpmou+vWQJisHZbfXdbIeEcpjJTreeAU6A4j7nz6r5XQwzH61/vWo1lKONHULOEeUjHKCD4Oct+XrVm952QKKEe9axFQzDEuq6L6iWTPfdeQ9TljMLt3+jm5zhojgjV+3MrBl+7wrd3IooZP8QmzVCv55xSvfedKcUXsIA/VfFqRk1oTpgNW8uwVIQ/opdqgMy5VJtWVj9VKgkfzAwHk3eENaQDf6zQVX2VsWj0EIF+uteIs68TD1oaG657r8xkXfGImbdV24zemmixTXgCut7txt/8Ig NIlp0frD prqmrpeO+BLeWIhKuG3cy6X8LOfkwvX1BwWO36FfyKuUuStLmzFRNTeBspzeDp4h4c3QTu/fwMyZgt3qeprmCPaitgl7zRMmDh4AFn7RJWtfH1MKsfTpfSAK/Tiy4iEJnPGwE13Qu/PO3e5kIuJ/K30v2awrvKGJ8eMqGlxwaWeaJdv4PF6sb7JUty2MTIBK4d+SExnCgUkBL9mDMIVlUluXD9YbntWIkVHTlNAOB6/2mc5q3EbVwA3JsVFf8QoznC6ZWWA2PdnYwY9n+IOMgaZzfhkzS1b0N/VVWsj4dVVqxa7+2rA1SsDDgMEsOzwhOi7sPfaOGfy+oGOl8yxWij0lAyVpA/WShHbQIvP6sRrZIo4JDYLMkUKkhMok6GrBCTmJzNJhEpiW7YFDbGWJDOyoIWP7Crj4JteJC9fsYvHMbktDYDO7PB3EqC+3vwC0K3qMwL4Wv3EzBG/M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce a way for hmm_range_fault() and migrate_vma_setup() to identify foreign devices with fast interconnect and thereby allow both direct access over the interconnect and p2p migration. The need for a callback arises because without it, the p2p ability would need to be static and determined at dev_pagemap creation time. With a callback it can be determined dynamically, and in the migrate case the callback could separate out local device pages. The hmm_range_fault() change has been tested internally, the migrate_vma_setup() change hasn't yet. Seeking early feedback. Any suggestions appreciated. Cc: Matthew Brost Cc: Jason Gunthorpe Cc: Simona Vetter Cc: DRI-devel Cc: Linux Memory Management List Cc: LKML Signed-off-by: Thomas Hellström --- include/linux/hmm.h | 2 ++ include/linux/migrate.h | 29 +++++++++++++++++++++++++++++ mm/hmm.c | 13 +++++++++++-- mm/migrate_device.c | 12 ++++++++++++ 4 files changed, 54 insertions(+), 2 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..4de909a1e10a 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -12,6 +12,7 @@ #include struct mmu_interval_notifier; +struct p2p_allow; /* * On output: @@ -97,6 +98,7 @@ struct hmm_range { unsigned long default_flags; unsigned long pfn_flags_mask; void *dev_private_owner; + struct p2p_allow *p2p; }; /* diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 002e49b2ebd9..0ff085b633e3 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -183,10 +183,37 @@ static inline unsigned long migrate_pfn(unsigned long pfn) return (pfn << MIGRATE_PFN_SHIFT) | MIGRATE_PFN_VALID; } +struct p2p_allow; + +/** + * struct p2p_allow_ops - Functions for detailed cross-device access. + */ +struct p2p_allow_ops { + /** + * @p2p_allow: Whether to allow cross-device access to device_private pages. + * @allow: Pointer to a struct p2p_allow. Typically subclassed by the caller + * to provide needed information. + * @page: The page being queried. + */ + bool (*p2p_allow)(struct p2p_allow *allow, struct page *page); +}; + +/** + * struct p2p_allow - Information needed to allow cross-device access. + * @ops: Pointer to a struct p2p_allow_ops. + * + * This struct is intended to be embedded / subclassed to provide additional + * information needed by the @ops p2p_allow() callback. + */ +struct p2p_allow { + const struct p2p_allow_ops *ops; +}; + enum migrate_vma_direction { MIGRATE_VMA_SELECT_SYSTEM = 1 << 0, MIGRATE_VMA_SELECT_DEVICE_PRIVATE = 1 << 1, MIGRATE_VMA_SELECT_DEVICE_COHERENT = 1 << 2, + MIGRATE_VMA_SELECT_DEVICE_P2P = 1 << 3, }; struct migrate_vma { @@ -222,6 +249,8 @@ struct migrate_vma { * a migrate_to_ram() callback. */ struct page *fault_page; + /* Optional identification of devices for p2p migration */ + struct p2p_allow *p2p; }; int migrate_vma_setup(struct migrate_vma *args); diff --git a/mm/hmm.c b/mm/hmm.c index 7e0229ae4a5a..8c28f9b22ed2 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -220,6 +221,15 @@ static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range, return pte_write(pte) ? (HMM_PFN_VALID | HMM_PFN_WRITE) : HMM_PFN_VALID; } +static bool hmm_allow_devmem(struct hmm_range *range, struct page *page) +{ + if (likely(page->pgmap->owner == range->dev_private_owner)) + return true; + if (likely(!range->p2p)) + return false; + return range->p2p->ops->p2p_allow(range->p2p, page); +} + static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, unsigned long end, pmd_t *pmdp, pte_t *ptep, unsigned long *hmm_pfn) @@ -248,8 +258,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, * just report the PFN. */ if (is_device_private_entry(entry) && - pfn_swap_entry_to_page(entry)->pgmap->owner == - range->dev_private_owner) { + hmm_allow_devmem(range, pfn_swap_entry_to_page(entry))) { cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |= HMM_PFN_WRITE; diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 9cf26592ac93..8e643a3872c9 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -54,6 +54,13 @@ static int migrate_vma_collect_hole(unsigned long start, return 0; } +static bool migrate_vma_allow_p2p(struct migrate_vma *migrate, struct page *page) +{ + if (likely(!migrate->p2p)) + return false; + return migrate->p2p->ops->p2p_allow(migrate->p2p, page); +} + static int migrate_vma_collect_pmd(pmd_t *pmdp, unsigned long start, unsigned long end, @@ -138,6 +145,11 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, page->pgmap->owner != migrate->pgmap_owner) goto next; + if (!(migrate->flags & + MIGRATE_VMA_SELECT_DEVICE_P2P) || + !migrate_vma_allow_p2p(migrate, page)) + goto next; + mpfn = migrate_pfn(page_to_pfn(page)) | MIGRATE_PFN_MIGRATE; if (is_writable_device_private_entry(entry))