From patchwork Wed Oct 23 23:39:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13848096 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4748D0BB4A for ; Wed, 23 Oct 2024 23:39:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 594D36B009B; Wed, 23 Oct 2024 19:39:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 51E1C6B009D; Wed, 23 Oct 2024 19:39:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3984C6B009E; Wed, 23 Oct 2024 19:39:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1E4CC6B009B for ; Wed, 23 Oct 2024 19:39:25 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5154A80F3B for ; Wed, 23 Oct 2024 23:39:09 +0000 (UTC) X-FDA: 82706485344.23.5B1F8D8 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.12]) by imf29.hostedemail.com (Postfix) with ESMTP id B7AEF120008 for ; Wed, 23 Oct 2024 23:38:58 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=T6dDAnbu; spf=pass (imf29.hostedemail.com: domain of matthew.brost@intel.com designates 192.198.163.12 as permitted sender) smtp.mailfrom=matthew.brost@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729726685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IXcEVc03d7xZKypkofExdFzQX/fZm25Gd8Wo705AvoM=; b=mJXJIgtDK/h9zApx5YGg/7lvAo9RjpwOv1iaX5jkSwKFmqNoTcIhm2+38YkmZV9xnuOZYS LDgEtnDhIIqBEfCkZIxciuRXtBYdm0h/ykNKc/4uxaNstJ3zCCmi0E8M1bRQQJc8LhnpUw /sYB2H2VkDSL0ZYH+ybpDFljndFAzPI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=T6dDAnbu; spf=pass (imf29.hostedemail.com: domain of matthew.brost@intel.com designates 192.198.163.12 as permitted sender) smtp.mailfrom=matthew.brost@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729726685; a=rsa-sha256; cv=none; b=LgywEQc6qlO+trzGdVFP9unyIazUTr7khW6WJW7wOr0rkdYtkocUoCGcqM2hwOdFGeCyzc WjRAncqrifoyvNnXz7oMu+rISNqpP6yN/Ki2Ozg5Y/I4+BHcDXbRHyX7kjywLy25QsI4xO uiIt9uqULTUbqpuJe9v317kAILGN0Xw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1729726763; x=1761262763; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DnOcHz9KmFrXRT0ci9znLuaYw7w3MglgopwNNF2CSag=; b=T6dDAnbuqHcCrTvPLwlqqKxM+AZt2uORPxUBomYehQg7JRT8/99xXel9 qNg583YoVOxprDytd+5LiQIsJzn2hH/0zrqG8V5I8x80Tjxkq4jIpQxt0 FjfNaDzw1uD/p9HdWTAq1PM2MwVQyb9cJNvbW8LNs/0dPK8XmbHLu7PPV JiuppMDj/XbDpBvGmwPRiww8Jq8I3Spa3+FmLQJeJzXCFbEfVNpU6ZRa+ 0HL9I0SXE/yvrGdLMReD3hKaILDwOoxbabz3Mx2J8uwY1pkEImcj8cidY RKplQ3n7+6wP8aBzak/8/qXuKV8aX4VX/gRMBDDcNnrRqsnm5cDSaBn1v A==; X-CSE-ConnectionGUID: 8MgawR9oSyelcPf/UedhQA== X-CSE-MsgGUID: erExKMgbSqed0ext9k3u7w== X-IronPort-AV: E=McAfee;i="6700,10204,11234"; a="33251099" X-IronPort-AV: E=Sophos;i="6.11,227,1725346800"; d="scan'208";a="33251099" Received: from fmviesa009.fm.intel.com ([10.60.135.149]) by fmvoesa106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Oct 2024 16:39:19 -0700 X-CSE-ConnectionGUID: vHd3tA6QSpCOaheKbKXr4Q== X-CSE-MsgGUID: zKqmRqKLQUehodDvQOtCOQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,227,1725346800"; d="scan'208";a="80420924" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmviesa009-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Oct 2024 16:39:19 -0700 From: Matthew Brost To: intel-xe@lists.freedesktop.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, simona.vetter@ffwll.ch, akpm@linux-foundation.org, felix.kuehling@amd.com, apopple@nvidia.com Subject: [PATCH v3 1/1] mm/migrate: Add migrate_device_pfns Date: Wed, 23 Oct 2024 16:39:44 -0700 Message-Id: <20241023233944.1798835-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241023233944.1798835-1-matthew.brost@intel.com> References: <20241023233944.1798835-1-matthew.brost@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: B7AEF120008 X-Stat-Signature: tnm7owhb9roxg4t4xkguzubwy78safo7 X-HE-Tag: 1729726738-145064 X-HE-Meta: U2FsdGVkX1+YE+DZkjLv70EerZfKScENV9wAariDO6kRtNp4m+JL2DcNfl8p8zesbiMv7OEg+yqYp1fff0kZ7LrBgf6kheZJkrn76L4Q4jdoC0omkn3TaqtR6zu6Tb5frgO0lDnRCkqbaED9Q/iZ2Y3BG6T+XduHV0RpIx7p5jtFAUM5yfkUoKXEH8LMoh/2s32f7U4soSNV9sNxPYMzUJ/xc0wF8W12e9keZi1i1VyHItHEUAKrv/bDvTHOBCzulth/RAhrnK6qe4dQ+jF2LahZ9yY3LkIX5ZH4NlVOEsYkAtzz6UeuVTyE9iIYF1sMWyQCGZZ+drFd/TbJlb/Ffsw0qtAJryTAgfr/YVXu2O+/MLMzl77YiX0DEeKn163JjxGDZoq6poGeLJOW8N4XkGHCG4p6fvMxVF4cBVUkmQ2JbhN1ibAGIyOKR5fega2mqqb0cGw6/YAHbB7b19wRgGT9hAAQ/TT/M8KqJ3nS2iEMgvsU15AyO7SEEJv+JiENGmFZs6B87l0HRadWbTY36tWvOy7ddnTv8QrtPoc4OON4tElua63SSvXoWhrbdAOFJLJsrj8rhPh4/BUFAwtpZ6YRUmiyBBthCnsXzi0Zp6XaIr1wejJh+b/l4Z0q2RxFjFO7PofHp+WJ6Pp4ewG26XwfZlLEGEt/MwyehZC3JmBolLnCx34fa6js+ZmC+8roax2EWiE9z78iA1bX/m+9F3urAkYTk4zdh0+6mHp4puFaUngfSNV1MkT2U3mpFZK1l1aBVrbJsEDUsolvlpf8DOcvpsaC9Dx7j+TIb97kglyf0mcpf6xMSQ73k+ptQbGvrSBaCM2s0I53p0iBFRfE5SoLRvUC1Y02hvFnMl2SzveT8u2ilrofhs364IT5DAoAuYX4fyCTc3H8IJXVphOiXa9X9oPI9Fo6GN2rw+dRg18IydAvJNwauLJ/NU4dKuQiZj/MVxKj0hTFwTe5594 TJu7FCQR jqYaln12pBgFIqX4+JJyLB4kA2e/vtP6BttUf2ErK/Emye9FLZfocxp5TXLBIbDkMRey6xNshwLQZwZWgMZZWhpFu0NDelZv02MkGpleQvb/o9mUq3ZbTXY3qDgCw5PFx7DQhyzdt0aGeXYJtG8sUt+AvhirAeLC67x1zzmcHIW9AEj8lq/GePaOp+s1jYEg/qD18aW76pPHNbz3G+FKEl6NIDz3ys4SM/8GmRg8VWKrxM2qh5DlEiJaAkzoxQo5bAT+9cPk3iX80XCHU4uYDmHZxPbXZ9+btaxoHk6ZwmAXKh/xnv46D7CJNUFn8FNUeoS+FIV93uwwyB71bBVT9zlkIkXB6fukne8s1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement migrate_device_pfns to prepare an array of PFNs for migration. Handles non-contiguous ranges of device pages that require migration. v2: - s/migrate_device_vma_range/migrate_device_prepopulated_range - Drop extra mmu invalidation (Vetter) v3: - s/migrate_device_prepopulated_range/migrate_device_pfns (Alistair) - Use helper to lock device pages (Alistair) Cc: Cc: Alistair Popple Cc: Andrew Morton Signed-off-by: Matthew Brost --- include/linux/migrate.h | 1 + mm/migrate_device.c | 52 +++++++++++++++++++++++++++++------------ 2 files changed, 38 insertions(+), 15 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 002e49b2ebd9..6254746648cc 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -229,6 +229,7 @@ void migrate_vma_pages(struct migrate_vma *migrate); void migrate_vma_finalize(struct migrate_vma *migrate); int migrate_device_range(unsigned long *src_pfns, unsigned long start, unsigned long npages); +int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages); void migrate_device_pages(unsigned long *src_pfns, unsigned long *dst_pfns, unsigned long npages); void migrate_device_finalize(unsigned long *src_pfns, diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 9cf26592ac93..950229706485 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -876,6 +876,22 @@ void migrate_vma_finalize(struct migrate_vma *migrate) } EXPORT_SYMBOL(migrate_vma_finalize); +static unsigned long migrate_device_pfn_lock(unsigned long pfn) +{ + struct folio *folio; + + folio = folio_get_nontail_page(pfn_to_page(pfn)); + if (!folio) + return 0; + + if (!folio_trylock(folio)) { + folio_put(folio); + return 0; + } + + return migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; +} + /** * migrate_device_range() - migrate device private pfns to normal memory. * @src_pfns: array large enough to hold migrating source device private pfns. @@ -900,29 +916,35 @@ int migrate_device_range(unsigned long *src_pfns, unsigned long start, { unsigned long i, pfn; - for (pfn = start, i = 0; i < npages; pfn++, i++) { - struct folio *folio; + for (pfn = start, i = 0; i < npages; pfn++, i++) + src_pfns[i] = migrate_device_pfn_lock(pfn); - folio = folio_get_nontail_page(pfn_to_page(pfn)); - if (!folio) { - src_pfns[i] = 0; - continue; - } + migrate_device_unmap(src_pfns, npages, NULL); - if (!folio_trylock(folio)) { - src_pfns[i] = 0; - folio_put(folio); - continue; - } + return 0; +} +EXPORT_SYMBOL(migrate_device_range); - src_pfns[i] = migrate_pfn(pfn) | MIGRATE_PFN_MIGRATE; - } +/** + * migrate_device_pfns() - migrate device private pfns to normal memory. + * @src_pfns: pre-popluated array of source device private pfns to migrate. + * @npages: number of pages to migrate. + * + * Similar to migrate_device_range() but supports non-contiguous pre-popluated + * array of device pfns to migrate. + */ +int migrate_device_pfns(unsigned long *src_pfns, unsigned long npages) +{ + unsigned long i; + + for (i = 0; i < npages; i++) + src_pfns[i] = migrate_device_pfn_lock(src_pfns[i]); migrate_device_unmap(src_pfns, npages, NULL); return 0; } -EXPORT_SYMBOL(migrate_device_range); +EXPORT_SYMBOL(migrate_device_pfns); /* * Migrate a device coherent folio back to normal memory. The caller should have