From patchwork Wed Mar 19 19:22:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A4A1C35FFA for ; Wed, 19 Mar 2025 19:24:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29CDE28000A; Wed, 19 Mar 2025 15:23:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24ABA280001; Wed, 19 Mar 2025 15:23:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C5EC28000A; Wed, 19 Mar 2025 15:23:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DEC77280001 for ; Wed, 19 Mar 2025 15:23:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E1336C0406 for ; Wed, 19 Mar 2025 19:23:58 +0000 (UTC) X-FDA: 83239275756.18.50FDC57 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2088.outbound.protection.outlook.com [40.107.223.88]) by imf15.hostedemail.com (Postfix) with ESMTP id D7C13A001B for ; Wed, 19 Mar 2025 19:23:55 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QnipFg4+; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf15.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.88 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412236; a=rsa-sha256; cv=pass; b=GpvwFPQg4ymspzkVviSpBIEW3EcCvz6V3BY/H04ZJMr8r5d//dwIVto8MKm4Oyz3ztgaYG W7Q2q8SvuJwSIHXC9wrLRGlzeMWZTNJXHnLEvWXr1cMrMAKgFLSiu3Rgt3l/nXYXTbcNzZ GV3u92/497izRzknfwFSuoOPstZakaU= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QnipFg4+; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf15.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.88 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=C5RXBlzF3qevVTh3B9iP05ObnaGOmXJ4bdq6S2Czr/9OLtprXU3i0qmJDHYEXx62DvPyhM 1iF1tQhwZRSuuqcV0bLrg1n+AyBcTp9DlU2qaYNE5WhblZjXWSGXxcXbNYB+5bKQWhbLSb MOtWdPJefQQxtETEdnIcLgk+XazUJKU= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LfD8pgNR3dG18aXRfuhxSJKUITEQ1hQwT8uiTUZBZ+WrJm7pX150hn8/2h9Ja0jjdO6jru0oy1ZxcBhNyAY4D7fOnAlnAhw2prqlxJEqZa2kw6BTqL3/BMQxunZBTwM1rly3qtB8osLid5sjTvrbVMWDAO4h/yTsM5xh/ps33g+MVk/rsJevr9UAHY0p8OegqR44oXL1IbRIfFIFBO7PqMP96DO18A0YWESXFaquVM02e97GsYIJrZIgwkhaqzYr2nv7XygvYGAZ29S+lAzh1UZmPHYnlSrddLniMuaQB6rVFEFmKCI3uxdF1eyl5M7CxBmh/ZA8sz+w0TI9Do6waw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=ZOrqcAdPZ3A5grvaDQDzgx5rlI9dYEPbinShaB++dUcN/eyeiWFP2olhM8qWJhsQze//fVjMGJDQlSIuCc20woi8ELf1hg7Gy/02e04MHJE05Ce3YyXE4UTLbsA37DP/zDJFS75HDkj+jwCGnFBcplOUs50gbWBhZ024BNd0kUcoBmq88EETJzkqLetjj6ZUncYztdzxwh9GRyT4jAILE15U6rMgc6l6+9I/fryh7o95R2JH78XxLtxp5MXr0gY4IlzF/i1GXxtSJl8N9ngOcjYqueXaHK3C9bOGgOw9x2/XTpuYto3dPlOPqemOSBZnlK/EeyjR0/DkTz0gDJC2ZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=QnipFg4+uIIn811W7K3qKeZpmZZc56zTiARSfWh953l8yk/WyUoDUAYd1HEbg8kIWF28bSE9zQvOBCe8uxgLrNaLsQfxr/dPtNqrp3eU+ZbO+ywp55/JxnW/ndd1WzLo+uWRMqrtY1RYclCdA5WUuqhP0t3nSDCzb7Oq/qw2WmM= Received: from CH0PR03CA0303.namprd03.prod.outlook.com (2603:10b6:610:118::16) by MN2PR12MB4336.namprd12.prod.outlook.com (2603:10b6:208:1df::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:50 +0000 Received: from CH1PEPF0000AD7B.namprd04.prod.outlook.com (2603:10b6:610:118:cafe::9) by CH0PR03CA0303.outlook.office365.com (2603:10b6:610:118::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.35 via Frontend Transport; Wed, 19 Mar 2025 19:23:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD7B.mail.protection.outlook.com (10.167.244.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:50 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:42 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Mike Day Subject: [PATCH RFC V2 5/9] mm: add support for copy offload for folio Migration Date: Wed, 19 Mar 2025 19:22:08 +0000 Message-ID: <20250319192211.10092-6-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7B:EE_|MN2PR12MB4336:EE_ X-MS-Office365-Filtering-Correlation-Id: ab83a946-6cf6-4667-1e0b-08dd671b9450 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: gcPZODinQLlEs6YEZGp33h2jgJIRZ6JR5G/qgtIPPGZtG9ntNW9s52KrynBj1nCYzuO3q9Vv16DKoksgjhgDyUUE5ehyD29XpTKj8HRWOs5wX8kvLeclVO+yjmPo/SJTR6IIO6hiG8D/TPdTS3Ob//vIsmeeGsvUJB6FXI9FS2im8ZoR1TRr0UlNnNHf6R/Gr24HxdDNiHwevTZTsWCOcdfEkQ25/ESTvaE/RUiREHQa9BIyLCecPgoqLpx7l38+tBRTIZMrxBjrcddPIVsTux22R6a5YxgfXYKYzAKPgVRqiGF3eXf2wSy+AX/gc7tQJxr7SAYBrWP50pkAqTtgqYnUfhWx7NmUrEGdDZGyhkc/Q1EZMqTLnxHOlP1j4Dwdv16Y0Q3//MNpXH45ChXRvQfoa4OBy8+WhWaIhj26Imc/wVgLDdw4sfhIhVcQLyIapchYPLh5LbTlUP5Wjk1/7SVagkESUrmnr2Wn+vvxJvXPzdtLBWFASqxcyFqo38QeeacvWMjIvwTRBTNiNAQIi4NMoxGe5QGVS1ICjvCX4C2PriiDHdbK9s5hiIckMQNl1BaoyBdrd3zKtDKVQIvqzwDRypEmFYVQ2wttc6hnrRxOIXjpnwI8InWPXVAUzrEQ2AbSPZGraUwxMZgVJveuqjuGV809TpcvhsP1WQbbIM8JbYGhuFiO8DkPmXDP+JIIKSrk37fRc0irq/lj9oTdkpoZKStvvKxG5JTFkaBYXuvBfBjFGF9bDd6UdoV/2gglylIMOmFyCBjtmIRtlIBG2AAd2vi1Grw+9SQTcurHvpQJUofAa45BuDKh8Iio+3aM0U7RZweqoYOJqbFgdIShI3LOC21KUMxbsgf/lmYWfwR6Ts/PkVjH5jX0mWxoJbwqidvBR+5F6jVfp6Dn35K6yVjytTrwltFknokTK2ijC55MT7r1gIuis1u3kGVaE2S+/k+FAqsaLkzKNTvbuSXAhZyTxCX7KGVpt9yUriQ//UItx2rDWCj8mJutlcgaG5meT1iGkZfV7hDH+DlG+oSfW+KHTZ/YHx6Wp/ABHmOgCyefwXpVSbqZ68LAoLUrsmpE/ZhugAg7MlB0+a7XYmoS1L9pjBl24E/BGHqdrmdxHKFgSMUo0mEmsA+5oNOA5MMMA49LYHs8m9LaaKqADhDmfodgBAkWaQI+UleqgWahTi3GgWu6qq5cm4WaAmU6aEz2CropiwRDdFT3Tn81Mpvj48ByX7gZfgmQYPG1U+U35acqd1t7+pba/SpfSc4MDAGRqZTWU8fJOocwngxyL+F/26guxe/hU6NfOCeEMdU7cxr0OKWaj+kgFQ6fuvNozxFXZUhUEw2ytWg8wwyDYt3dPsJJs94xZcvqvvdp14d/cAccnOJp71Xyh7Q+kOIQ9JtI9KHBIcMpqWXmgijsI4F6XHMznRB9OYo796K5bib/gJdIn/AYuQV9XFDcwsiT8aNrk0RIOjDgG/FWrUV8ZRw5LLVe2w8YP9ftAhEFDaHu728= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:50.5909 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ab83a946-6cf6-4667-1e0b-08dd671b9450 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4336 X-Rspam-User: X-Rspamd-Queue-Id: D7C13A001B X-Rspamd-Server: rspam05 X-Stat-Signature: snhe1iazyagkcwi7irdmw55mw1by8zcg X-HE-Tag: 1742412235-707460 X-HE-Meta: U2FsdGVkX19zUAg0h8N6NIKVQIAhrQpfysmPki7HzH/UBscs8Fv0tezrA0rJh6TJhbQ/V4J6I+yogseSyH3BlLYKCKlU7e/y1ZukeUcsCbO1kfJdHsmaV1sZKhHI4w9LyvY6PC4mqrwbSv7ZwBb8Ce+cO4MTb0h1uVsM+8C0B46gkXwWED3C4MW4wvg7yzmeFULinfRU5ffCnpNKG925JaNGUeDy8nJT6uuYmCRe8Iub00LsW/SOhfE0IZppibHr/DWTj3qhqBAYp27P9VWkmY58MUZaFEsp28Mh2bJRYDSD1ZLpnNxxSfemcpqg6nUrQbEDpTdl+DVjEkdXWk+/PkPFpEm8Df6vwsK6zIW8fAhLWLAQDVSZAwDiPiMsgflRa1C7T3SWT5er7pBRm0kQCgdI7HLN4iAHX1mLkr8qTQEpR+FyOoogvZ2bHbWhGPEMU7q3UwT1XL85Bk/vUMg9l5JYOZwRRFEoD/Jag1pxZsq+Pk8XIa77Bbl9kdq9Bxg2a0ZoFLi+YTg3IJP1dMmQzipcx83oIlhQOA49Ua1fl3JhhZA26g6dXVUqh2klgLRzUPgwrhrWWgNx6lQsH93t9iMEJW3SRlGjXFfwELvibf34zW0qJiIQs5gTy3Y3m0fQ2wOp43jWUF/BsuvAebVOf0O9CKNuvYVP2l8fdiJBHxUjzCGBjFm5dHHJCxzYOPlubxEfV6ye7Oi6eHrdfU50GoonqVL42XikYzNVSGR7DvBH2unGjvAt73HyZC1y/K3EGPPY/B9qjavKcnlgKHgWySK7vQ2i4wXhPmXUyL2iOv1sfPyDb30ILp+a8LlPagLEe6THwWVeaPjTcg95BWu9jR8A9krk9x5DFfXP3IAjH/izVIw3VWUaAoO3WF5vhmLuh6kx7BmmnJs5GiCnA1zV8vtBN+WyBvMBsPz3IxWbwqpILquWhX0psfh9VY7N42MttcZp6D/i89aH+DmTvBT UyyxHOJ9 w37mDPlxkZqHLwCvqryyV09UtXwa7ImYJ7r+G3F0zPXeLdjhicuLf3kVgt+ZqzyCgnTFe/d45jzQML+cVN7G7C76PKZ8eNpttZwlJ/kvOdres3V1n0nC/tvQCpauNvcU+mLxLY14muLChNdBK2W6avDhdFj71b4+I/Cm/g//EDR3XG6rn8Bq3UsSI3w1oPXjZCqnHzV0Or4/b1kppJVJKt0VADAKH44U0LY/RSwGXzAcSmSorAVr7qkSH3aK5InQ1F3ZmtcGpkTRvtH4eBykFYprh7XJM0rFtGv2WvC+SsPv17eapItv+6MAVBN25s942R43pw1R+4DX9T6FkUSaIe63fLTrin3drP440fZu/B/PzPKd3JyIpxmOrcWpY3uXs5ojH3lMVHLCSbH1AO93P1gC85stjCfTrXXjcbViqHte2m6iKU5gt4IXYJizPYvjv7DljD0LLLgSwtdkYBPD3TIRwsmybtSiTDN9Zi0fWxlMXyRXDQkmNeJmf4qc3JUHaPLWbxmvOh9Hl/78iriVMy1/k3Gywau4kZYlqulLt+2BcGaQBudfjv8YDZxVS/6FclkvN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Mike Day Offload-Copy drivers should implement following functions to enable folio migration offloading: migrate_offc() - This function takes src and dst folios list undergoing migration. It is responsible for transfer of page content between the src and dst folios. can_migrate_offc() - It performs necessary checks if offload copying migration is supported for the give src and dst folios. Offload-Copy driver should include a mechanism to call start_offloading and stop_offloading for enabling and disabling migration offload respectively. [Shivank: Rename the APIs and files to generalize the original DMA-specific offload implementation to support various copy offloading mechanisms such as DMA engines, CPU multi-threading, or other hardware accelerators.] Signed-off-by: Mike Day Signed-off-by: Shivank Garg --- include/linux/migrate_offc.h | 36 +++++++++++++++++++++++++ mm/Kconfig | 8 ++++++ mm/Makefile | 1 + mm/migrate.c | 43 ++++++++++++++++++++++++++++-- mm/migrate_offc.c | 51 ++++++++++++++++++++++++++++++++++++ 5 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 include/linux/migrate_offc.h create mode 100644 mm/migrate_offc.c diff --git a/include/linux/migrate_offc.h b/include/linux/migrate_offc.h new file mode 100644 index 000000000000..908f81ebd621 --- /dev/null +++ b/include/linux/migrate_offc.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _MIGRATE_OFFC_H +#define _MIGRATE_OFFC_H +#include + +#define MIGRATOR_NAME_LEN 32 +struct migrator { + char name[MIGRATOR_NAME_LEN]; + int (*migrate_offc)(struct list_head *dst_list, struct list_head *src_list, int folio_cnt); + bool (*can_migrate_offc)(struct folio *dst, struct folio *src); + struct rcu_head srcu_head; + struct module *owner; +}; + +extern struct migrator migrator; +extern struct mutex migrator_mut; +extern struct srcu_struct mig_srcu; + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head); +void offc_update_migrator(struct migrator *mig); +unsigned char *get_active_migrator_name(void); +bool can_offc_migrate(struct folio *dst, struct folio *src); +void start_offloading(struct migrator *migrator); +void stop_offloading(void); +#else +static inline void srcu_mig_cb(struct rcu_head *head) { }; +static inline void offc_update_migrator(struct migrator *mig) { }; +static inline unsigned char *get_active_migrator_name(void) { return NULL; }; +static inline bool can_offc_migrate(struct folio *dst, struct folio *src) {return true; }; +static inline void start_offloading(struct migrator *migrator) { }; +static inline void stop_offloading(void) { }; +#endif /* CONFIG_OFFC_MIGRATION */ + +#endif /* _MIGRATE_OFFC_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..7a0693c3be4e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -722,6 +722,14 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE +config OFFC_MIGRATION + bool "Migrate Pages offloading copy" + def_bool n + depends on MIGRATION + help + An interface allowing external modules or driver to offload + page copying in page migration. + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..010142414176 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -93,6 +93,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_OFFC_MIGRATION) += migrate_offc.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o diff --git a/mm/migrate.c b/mm/migrate.c index 8b6cfb60087c..862a3d1eff60 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,7 @@ #include #include #include +#include #include @@ -743,6 +744,37 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) } EXPORT_SYMBOL(folio_migrate_flags); +DEFINE_STATIC_CALL(_folios_copy, folios_mc_copy); +DEFINE_STATIC_CALL(_can_offc_migrate, can_offc_migrate); + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head) +{ + static_call_query(_folios_copy); +} + +void offc_update_migrator(struct migrator *mig) +{ + int index; + + mutex_lock(&migrator_mut); + index = srcu_read_lock(&mig_srcu); + strscpy(migrator.name, mig ? mig->name : "kernel", MIGRATOR_NAME_LEN); + static_call_update(_folios_copy, mig ? mig->migrate_offc : folios_mc_copy); + static_call_update(_can_offc_migrate, mig ? mig->can_migrate_offc : can_offc_migrate); + if (READ_ONCE(migrator.owner)) + module_put(migrator.owner); + xchg(&migrator.owner, mig ? mig->owner : NULL); + if (READ_ONCE(migrator.owner)) + try_module_get(migrator.owner); + srcu_read_unlock(&mig_srcu, index); + mutex_unlock(&migrator_mut); + call_srcu(&mig_srcu, &migrator.srcu_head, srcu_mig_cb); + srcu_barrier(&mig_srcu); +} + +#endif /* CONFIG_OFFC_MIGRATION */ + /************************************************************ * Migration functions ***********************************************************/ @@ -1028,11 +1060,15 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, { int rc = -EAGAIN; bool is_lru = !__folio_test_movable(src); + bool can_migrate; VM_BUG_ON_FOLIO(!folio_test_locked(src), src); VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst); - if (likely(is_lru)) { + can_migrate = static_call(_can_offc_migrate)(dst, src); + if (unlikely(!can_migrate)) + rc = -EAGAIN; + else if (likely(is_lru)) { struct address_space *mapping = folio_mapping(src); if (!mapping) @@ -1868,7 +1904,10 @@ static void migrate_folios_batch_move(struct list_head *src_folios, goto out; /* Batch copy the folios */ - rc = folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + rc = static_call(_folios_copy)(dst_folios, src_folios, nr_batched_folios); + /* TODO: Is there a better way of handling the poison + * recover for batch copy and falling back to serial copy? + */ /* TODO: Is there a better way of handling the poison * recover for batch copy, instead of falling back to serial copy? diff --git a/mm/migrate_offc.c b/mm/migrate_offc.c new file mode 100644 index 000000000000..c632928a7c27 --- /dev/null +++ b/mm/migrate_offc.c @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +atomic_t dispatch_to_offc = ATOMIC_INIT(0); +EXPORT_SYMBOL_GPL(dispatch_to_offc); + +DEFINE_MUTEX(migrator_mut); +DEFINE_SRCU(mig_srcu); + +struct migrator migrator = { + .name = "kernel", + .migrate_offc = folios_mc_copy, + .can_migrate_offc = can_offc_migrate, + .srcu_head.func = srcu_mig_cb, + .owner = NULL, +}; + +bool can_offc_migrate(struct folio *dst, struct folio *src) +{ + return true; +} +EXPORT_SYMBOL_GPL(can_offc_migrate); + +void start_offloading(struct migrator *m) +{ + int offloading = 0; + + pr_info("starting migration offload by %s\n", m->name); + offc_update_migrator(m); + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 1); +} +EXPORT_SYMBOL_GPL(start_offloading); + +void stop_offloading(void) +{ + int offloading = 1; + + pr_info("stopping migration offload by %s\n", migrator.name); + offc_update_migrator(NULL); + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 0); +} +EXPORT_SYMBOL_GPL(stop_offloading); + +unsigned char *get_active_migrator_name(void) +{ + return migrator.name; +} +EXPORT_SYMBOL_GPL(get_active_migrator_name);