From patchwork Wed Mar 19 19:22:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFBBCC35FFA for ; Wed, 19 Mar 2025 19:23:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1547E280006; Wed, 19 Mar 2025 15:23:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 12C5B280001; Wed, 19 Mar 2025 15:23:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC212280006; Wed, 19 Mar 2025 15:23:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D0199280001 for ; Wed, 19 Mar 2025 15:23:26 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0289F14071F for ; Wed, 19 Mar 2025 19:23:26 +0000 (UTC) X-FDA: 83239274454.06.A390259 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2050.outbound.protection.outlook.com [40.107.244.50]) by imf10.hostedemail.com (Postfix) with ESMTP id 0D423C0013 for ; Wed, 19 Mar 2025 19:23:23 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=REd7jNs8; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf10.hostedemail.com: domain of shivankg@amd.com designates 40.107.244.50 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412204; a=rsa-sha256; cv=pass; b=RHpIEk0zfxJovoM5fbORS8bnw0d3copT1jjyKBTR8LhKeXxujRXm9YZYEUmY8/uT07LoqC 8DqEonxvxZfiRUFMn9F3FlSjKEL+42Df6IucChI2Bf2bNZcR00qMeiSbDQ3kMEM5X3qcH2 xmgcsV46K7DtpiL1XTAYiHo6lCd7Zbs= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=REd7jNs8; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf10.hostedemail.com: domain of shivankg@amd.com designates 40.107.244.50 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SZyoWMY8GPkC6c+QiCnwrogSvPL8DBaC1BvGNUgqyIg=; b=AwN7vu8jwKwTOqp86ZSk4CEikdvfUJadPW67eNLLWkGzMWQ8Ol5mpUL1zU/3AM4/7vH+y4 6qc53g02X2/qYs0+XzCfWSx3yIeeZYQ9/TkoluPcgFTULAC7Kt2wV8JwrvxID/jjJyYe4t YTmpjxpQrspHJylYBJryAZQcZN+/Q9A= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TBJqvH37che25yt2cYzIZGRPrKVkMD1fXZuQl8GsY04kK97ha0rgH0mSuGDAl7/RllrcuHawYedsbP/sMQfZYdqywovqqNYfRdqcSq3AThpLRuCoiKC/Y3+vDchjIJ1mOkBBjNdaVd8gHDObb5M26peo+xv4q2ffX2kkMG90E0WWuNnOMCkctxIIFYiiszeoZLjSNhehJRCyF6indTtWUbmKfgPUFPJ6o3d5yGPTMgFKyJcmM8e2qN6nYTA9950O6wAkf6iSirIOcmK3+wcvvZKSfowNl+lvaEN0ZSvjvJNFK0fEMOH7GHpSrdCzkm92u8lOOo1o9Vzyx81n1eNAHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SZyoWMY8GPkC6c+QiCnwrogSvPL8DBaC1BvGNUgqyIg=; b=DZ7ssxCoLNeb3/tiZQ3uD8g/PGjEbrOQZWrelCnLeGp4CI0d43aQNazRwJ1jLsZ3qScSEon7wKS+rq7+yymV0m55QJlfEbzrl0cJj7itt9lP+tBbIZew9Iw4Erb8EszE01ZY3mDbJGKmy7Shlg4lOtMu1oK+DH5v0ZjEmR88DCuRalsowOlyL2OMirxdcIyF37HHJ8WL7Hl48T9zqRUYkNY8vdEAEUWVSAaZyfHxR8LBiNS9rbtGrMmJrcJP094TAjqTCHbbgNSWJpa2dGHybZKLCR7+IdVXcybya1bx3wGhGx0fmLad/1RCQopRv6bIoTKsijx26fO76tVfTlwyxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SZyoWMY8GPkC6c+QiCnwrogSvPL8DBaC1BvGNUgqyIg=; b=REd7jNs8gHGP2mKaPvk8TYELAzgTvhFxbCAHUnwRWxPXWfx+JDYGIa4+Bh4ugqs6e6YvIl7esWQML58gL7fVcu7vvc9P42DIhc8IQ/2q0NDNwbPii5Xk01shvIvoXzM5NWjKSDvTfSmpOSGCeFZcHOpR/O6DHGLl2PgWl4TuItg= Received: from CYXPR02CA0026.namprd02.prod.outlook.com (2603:10b6:930:cc::9) by DS0PR12MB7993.namprd12.prod.outlook.com (2603:10b6:8:14b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:18 +0000 Received: from CY4PEPF0000EDD5.namprd03.prod.outlook.com (2603:10b6:930:cc:cafe::c) by CYXPR02CA0026.outlook.office365.com (2603:10b6:930:cc::9) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8511.26 via Frontend Transport; Wed, 19 Mar 2025 19:23:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD5.mail.protection.outlook.com (10.167.241.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:18 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:09 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 1/9] mm/migrate: factor out code in move_to_new_folio() and migrate_folio_move() Date: Wed, 19 Mar 2025 19:22:04 +0000 Message-ID: <20250319192211.10092-2-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD5:EE_|DS0PR12MB7993:EE_ X-MS-Office365-Filtering-Correlation-Id: 0dd7a6a9-10d7-4d68-f27a-08dd671b8106 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|1800799024|36860700013|7416014|82310400026; X-Microsoft-Antispam-Message-Info: dt0tl2cnY7vDBfpDdyeL2xwnBUyZT4Je/1hE2mS3nxOgahtC1f64Myv/XqTUtxL5ZhxVQrSdvu6i+KD2rsVI0FSrGiorB78wJocLI9i6P2r7RlhPPVOfibUuftRbYAkXiVQqx1XD0u5tjuR/P5SsZnDjLWVou7tPo+/Mn9luVlSDbzMcvp3Fag2x+Lc2H3CV3WXSfDwmYHEmPEC8T/qSpQnjh638rxvBMLENyW+jEryQIJ7jcwOawQ0BJafk9p8TpzbxH2iwEgxNSshf1sN6KC8xBU9zUmJLIbf3H8UlZOr0E4rU2AV+xy6LFhbKTpsqQTNqv2XARZ3iQMOZ44TfXFOfICS+XYei7zW4ErasgwNCwKZOkLH8EpZ9egV4tMQ4s3+ME8Dg2j/zDz4bmhqbu0xjaAJkyh7FVfhm7RlZAiaZI3aFP1eUwzuaW/3PliAPS0bswbZ0pQrYaJobsn460B9kH1bdnfFXPfr+SfewDL0UdOwpcPtz86Imu8EtZljDsZlU6070Mwq0YY/z6hUqtB7oh764Yc2x3FDSnYacerxHDL7XbH5bmA6b8Xdjqjzjuru8jkPGZb4uSoLfumYaGobMxo4QbFuEse3CA5LJ1O6KB/k8sWdxiA1JXb2ZCgr0sTG6r4sYCpMhsu6O7FsJDEXRGoGfQL7EiaUIXX65y3J+p7JwFPGdoZZCwgfca0sLG8Wul8YBb5S5vJYvUD28gTk6KHCQqUffQx/NhFVsT13swKG0FxHjaDdngXtdVbjyc392z7dgZoIH09GzL6tuTzcFMliyXaPYBE5bENRT8dCkk2oI0NbaYG/qh+EFcZfwi85a1GWvAh4PsftL5ybVEtoH0m/Fw0/1G+jJTIB8gVSGGSih7hs1N6Zd/rpSDhNgb7jRFodyasd8vpYkdW11s9rnHi0K2x/QusLg//xGecTOB4qKBoXWK1kIfuQBWoxzk1nQtuEvMv1HbX5UGmi705NLw39MNhrz7kMgzIqnkR/QJvyTc3kiL7oiEDXhtONcITOyIg4G/Y08iyL4GOire3cw9MHvEp3RfLKyh1av3MQU1G60uQO/xj75Rv4/W9jP8izvxZuVPRxJ1/WFwm64DIJ6dqeL5iPd05hvXzAaF6hWSUj4jkoy+SmRtlItJ/hQZAs49hWULIcxkgUowB5cFBr2S6w7nW2NBpMJ4owGVmWGGbu/d6iM291EGnRwvyjQSeTE6z3wsEXiI7st6/bP5QpS9AIX5M3y9XSQ35NPG6uLYkUYjzRTIYQcU+Kw5Gkj764R6ddZKnTDzMObCHgpHS0gvKhvmHIh00xCtlB35owofGBAoIXoydI+WWXESkY5qg5aTmJ1cHIL3HKl0FRJhKsuousg35JwYNSAQDBAyZistjtvZYTyH9jJGoFVEYTqmLLiNDwg6JEDBP/aOJ9bP9Mf6U2p4YO7hX9BApM9s1DCV9elrOrhrWmKgh5AstjAnRNSO6r6x2DGjJiK/JSM3JXSG+yOIoVxfdqY0Y17Ql8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(1800799024)(36860700013)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:18.1410 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0dd7a6a9-10d7-4d68-f27a-08dd671b8106 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD5.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7993 X-Stat-Signature: 9r83f94ho7d85ujrciaik4rykgmfc3ya X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0D423C0013 X-Rspam-User: X-HE-Tag: 1742412203-200420 X-HE-Meta: U2FsdGVkX183tD8B+88RilmVdUpeFNcx4ahOzHpcHmgKk2Or5mqeRts7Lg18NjTIi1tdyQAr0OYdnYR/gAW5KiiQGHxDaNJRwfK2ZiphlsE8QATbPCDV0AJCOS7+5dQRMYZSWjKF6c4v90NbfXuZ9/jcSZT/tdDjsDmFjgyc4aekUZlCQ2/YShC0Dod0KmES0uC6IfNWWdTS3jcS5ZzVrDv6xV/hMS+5AV9Q1SuXiVxtgYHxJCuYJBnPWaN6+m4wibn6mcVw41w3r05h5B9EA5wQFkx9Qu3GfuzWM+zaQyD4T0HH7rhvStRKWVEeBYSFdvgZBkFkud+mUaORlyrUhpwOaI7VDJkGmOu0J0BMA0vTD924JcGktakT4BWIDEjjkR0zyItyB/vOsdnLgs1nWMUPa2xjS1ppEO539Jzg+yg7WRdUALHL+0CTPTk7zfnq1EvDYBavwH9hbHCxQXwyvY7a17J+uyViTFAEdN/7UegSmb6EKTCpC9a+JBIL7bwMJ7FIJk+gwV7xYmf2XGFjw/Xn3MM/xK71RUi1APtdi5VEIFy2OB8EHfMmqN2e9QMYBEWjrz40EfHubw7PbGw6/Ez6lbn/DEz8kussINZFcwei3FMBVFTGNMV4nztux2KBDXSD8+0T9Eh6Xzb4Vxw/4NfoniNe8G/5H+LVeXvesUs6jlE/my9Y0a0e+t+o9HLbHuE1fvOLR3VubhtRv4/6LRPvfCED4kLhi0h7DeQtGKbXFN5LDX2dwvxi7VIcGn2Ayof+jvkfuQDE/qcmESMXUyw0cbx8S0KQeOkCwOQturicLGeuSNtAKakOOqizCC2BH7aTxU6cNQD1FfB9zNqRvBF0PLBnj2+sS00HdUAMXbj78BxsFzL/kAsw5BT8kg91NFfHXeZy3X00MZijs+MPsgFinn4UEtUfFg6oxattuVzBGitiMqNNvzwdWmcdtSNYDDPYaiOL/iZ1DqUqMVe boCbcfs6 kqYvSqK/TC1QoNBWeZe0WyOtGo9TlHMdF57ZWN9/r6jY1Wh9rG6S99yNpHdbG6DaHIDFW1AubXTAKvNPyc3txdZLY4XZGm0Bqa3bDXq4Uj/ILpRd8TI93tf61Jcyt/DO8sMWHBD5X/w4ARsS9P8l+SOmlXPu2RB/ELw7mYqr5hXCIG0/Icaqq9PaXIbVAZ+TbT9FGXRLlYl2F3xZH9tn8JWEajlc1mNtY24sXXFDC8nVLYGR0Q25vq+tnS7qtwcQT6MZ4ZdVwobsJyD1QSBcX8siiYyBN4nZUgrQrU3VGzAYWMGTGfiw1HzOwwHrA7S9bRLjxqC99FbvSKAbxNdPUMhu9ZLs5I7+d6EjNl9L0uE9Jo30f3CUeEw2t8swiEVmf8RuGT46tvSyUuylI2UhkM5bjKFzdblcjYGT7l0DViDktMRCVZE+ms/PzZqRJQQq8xUkzk2MF2CXiM7jFU+gVeb9f0zsJBofiEHDjdoF7ZY1P19NRaF1lYGrtMZAEtXT99UdoXPaaL4nUuS3t+AUgqzpjhk+Um/XVzzg0T/lvUxhTm4ud2yoqtqHsxo75oSI91Y1fmq9qzwo/IKibb+SmpYKYl0W3vUzVUItRy9c1zx69uXBQeTpowpzfMuVR35jvWPFQOoQCkybNjRc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan No function change is intended. The factored out code will be reused in an upcoming batched folio move function. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 101 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 65 insertions(+), 36 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index fb19a18892c8..ce7ddc56e878 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1014,18 +1014,7 @@ static int fallback_migrate_folio(struct address_space *mapping, return migrate_folio(mapping, dst, src, mode); } -/* - * Move a page to a newly allocated page - * The page is locked and all ptes have been successfully removed. - * - * The new page will have replaced the old page if this function - * is successful. - * - * Return value: - * < 0 - error code - * MIGRATEPAGE_SUCCESS - success - */ -static int move_to_new_folio(struct folio *dst, struct folio *src, +static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, enum migrate_mode mode) { int rc = -EAGAIN; @@ -1072,7 +1061,13 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); } +out: + return rc; +} +static void _move_to_new_folio_finalize(struct folio *dst, struct folio *src, + int rc) +{ /* * When successful, old pagecache src->mapping must be cleared before * src is freed; but stats require that PageAnon be left as PageAnon. @@ -1099,7 +1094,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, if (likely(!folio_is_zone_device(dst))) flush_dcache_folio(dst); } -out: +} + + +/* + * Move a page to a newly allocated page + * The page is locked and all ptes have been successfully removed. + * + * The new page will have replaced the old page if this function + * is successful. + * + * Return value: + * < 0 - error code + * MIGRATEPAGE_SUCCESS - success + */ +static int move_to_new_folio(struct folio *dst, struct folio *src, + enum migrate_mode mode) +{ + int rc; + + rc = _move_to_new_folio_prep(dst, src, mode); + + _move_to_new_folio_finalize(dst, src, rc); + return rc; } @@ -1341,29 +1358,9 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, return rc; } -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) +static void _migrate_folio_move_finalize1(struct folio *src, struct folio *dst, + int old_page_state) { - int rc; - int old_page_state = 0; - struct anon_vma *anon_vma = NULL; - bool is_lru = !__folio_test_movable(src); - struct list_head *prev; - - __migrate_folio_extract(dst, &old_page_state, &anon_vma); - prev = dst->lru.prev; - list_del(&dst->lru); - - rc = move_to_new_folio(dst, src, mode); - if (rc) - goto out; - - if (unlikely(!is_lru)) - goto out_unlock_both; - /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1379,8 +1376,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, 0); +} -out_unlock_both: +static void _migrate_folio_move_finalize2(struct folio *src, struct folio *dst, + enum migrate_reason reason, + struct anon_vma *anon_vma) +{ folio_unlock(dst); set_page_owner_migrate_reason(&dst->page, reason); /* @@ -1400,6 +1401,34 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, put_anon_vma(anon_vma); folio_unlock(src); migrate_folio_done(src, reason); +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru = !__folio_test_movable(src); + struct list_head *prev; + + __migrate_folio_extract(dst, &old_page_state, &anon_vma); + prev = dst->lru.prev; + list_del(&dst->lru); + + rc = move_to_new_folio(dst, src, mode); + if (rc) + goto out; + + if (unlikely(!is_lru)) + goto out_unlock_both; + + _migrate_folio_move_finalize1(src, dst, old_page_state); +out_unlock_both: + _migrate_folio_move_finalize2(src, dst, reason, anon_vma); return rc; out: From patchwork Wed Mar 19 19:22:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18F87C35FFF for ; Wed, 19 Mar 2025 19:23:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FFF8280007; Wed, 19 Mar 2025 15:23:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A9C3280001; Wed, 19 Mar 2025 15:23:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14A2E280007; Wed, 19 Mar 2025 15:23:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EC1C0280001 for ; Wed, 19 Mar 2025 15:23:33 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1AED0B753A for ; Wed, 19 Mar 2025 19:23:34 +0000 (UTC) X-FDA: 83239274748.18.29585C7 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2059.outbound.protection.outlook.com [40.107.93.59]) by imf18.hostedemail.com (Postfix) with ESMTP id 23F0E1C0003 for ; Wed, 19 Mar 2025 19:23:30 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=qeGOLX6i; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf18.hostedemail.com: domain of shivankg@amd.com designates 40.107.93.59 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412211; a=rsa-sha256; cv=pass; b=LFy4/udQe0kWMekYXTxpaU8FruCC42ldB2hpRpCAJB6//7+WUJvp1OUVGQrukxEOfGfMrX lKDhy51i5F2X1YVQVhhgqgPPQVZXhXsTcjvlMdPQ70O2pHWVmHXJ9EL0Z95ApE5Xecxxea voYe2Up3gt9pcdiIt09irfoadTmqEZI= ARC-Authentication-Results: i=2; imf18.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=qeGOLX6i; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf18.hostedemail.com: domain of shivankg@amd.com designates 40.107.93.59 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412211; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+H8IxYYIa9GO9xurZ2fEGRkp5bA92jEu5XUKX/h7S5M=; b=irAAdgrxUZY1OxBJQw/z3AfpNTfI6LgGKG15k3J3gDI9ylwf4FmS/AP+4h/EL7rMXB/i9a /DknezaBDr90OR/hactpuqjslgnHV9GTrTdhjmV9jZ/CwPB5oTCwfk63+oQxSbsxcyrDaA FFiYm3D5cfnlkW/DwoSt987gKzVPfV0= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=qWvpPFWSghZpx18cZKyIHWG1NgS9cWN6z3l+oQn0V1C/89RZh94bnmjvCrf5hg5RJm6IwlIeypVYYChFPgEc8PG2v7SfazhJAXKaZ2+HOahyymok6QY5XgVPLZfGvP8G+L25siTzlSzM1HcXmFuu3IB+bHFqO0Eum7mrkO8/N6ICqPUUtLuyKcv6uTQGa8ZwYnMUR5Amz071WZoAE2VOxv5/Son3l/67kI/nf+tS7Q4ZCHgDSrlhBZjKLoN0lhw/b3fseW5VJkDeZSFwA6Y2eCsMfzepEwjzZRwQmAPCpXW5evqdb72eCdumXj8PfmjWnWeh4WU0NOgNYT4dnsD9VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+H8IxYYIa9GO9xurZ2fEGRkp5bA92jEu5XUKX/h7S5M=; b=FO8fdlAbAx25OO/NgOlXFEilYXqm7J8V3mF13OmFVU5ttqq7JnlZK94FLOUlEAw61J3lme/tIRqAuRL2Sb+7bWKzwMV+hO9TxhWRlc7l5bMLusF09f+Py5K+QM+niQIlCKRVY54u7gFEOXK+SROWsQS1kzmlD1XS3b9oGkMlrWsvDmGs7/eTBpg2bSB+nw7oz/5mWWaxR4bV9PFUyRSuQFOloYoiYIaCWaUZ5JHWWFzvvyIIMnkbdsZJosoi8LG8ReCvAGtSx5iqGVfgyI5jA4qwFjhwWWtSYmw+1GVj3OO8w18vqPnWq37HbubmDUjpfJVMgMzDc0WMiVXat1Ygqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+H8IxYYIa9GO9xurZ2fEGRkp5bA92jEu5XUKX/h7S5M=; b=qeGOLX6i7A1PxDVw91iXZDbU2fZeeIa06xpcZckcsHd9x2LVAiOseRt5bMrxZZ6ZvRCAuBTywUVryvJ3IDuKhICbFqSMLB0LFGr5+jwBmRzS0/XQf9xDW2ZaIIGT3CzWRPqoGkklOXnAmNXGw0IvOFhdIzWsSFK85zRbiaADGSI= Received: from DM6PR02CA0129.namprd02.prod.outlook.com (2603:10b6:5:1b4::31) by IA1PR12MB6356.namprd12.prod.outlook.com (2603:10b6:208:3e0::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:26 +0000 Received: from CY4PEPF0000EDD7.namprd03.prod.outlook.com (2603:10b6:5:1b4:cafe::30) by DM6PR02CA0129.outlook.office365.com (2603:10b6:5:1b4::31) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:23:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD7.mail.protection.outlook.com (10.167.241.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:26 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:17 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 2/9] mm/migrate: revive MIGRATE_NO_COPY in migrate_mode. Date: Wed, 19 Mar 2025 19:22:05 +0000 Message-ID: <20250319192211.10092-3-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD7:EE_|IA1PR12MB6356:EE_ X-MS-Office365-Filtering-Correlation-Id: 300d8667-9e74-45dd-d7aa-08dd671b85cf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|36860700013|1800799024|82310400026; X-Microsoft-Antispam-Message-Info: kWnrc7ERbG4FxpzsQLmD8IqYETAHyIFAWNK2mLbzvqcLB+73Jj/iorg/oK1kzal4e4nth2qqUCbi4gyccdnMHoFpB2rjZ3OXDpvAwh4AxiLKiwJ5DjD7EwiL6YPS7eAXpcEE7jxAZDgZyQmggTplxGVwZXCgWuQABbapLj8vXdwtjpy4LT7qodl/aqIfnles9xIQVqVZjfXYlzcsQH7pzE0yvkinaTYNy1xrBkK/fvCgufvC92x/w128jlji+CmnE+5I4H7uAhFMNzeNzbRPI5O07vywB6/URGaEY2U+vE6Po4EJFfeB+Mdbd2/8JGZoK8Vsk9JavWPyrwJ9/t3kbfMLZFOE7gOe1Yq3/xErq7bGXYbO8ebWLav7upT4zrOmCTUluLAqXqHJpvWgEFwhK1PmMWz884eOOwfJ9MtZh+1HM2VJ+2UFfi7nByPFpNGXDfD/VXQGaOElYtvo2xKTL9EKt64c0aDOsNGPv0m2Wvijw0afKMV6Pm6PTmeyOy1PVDhXP5T1VEz2BMw4Cz0S+H7xk/z51ndKfXeb8n1FKJWi3564Zdn6CvplNuRwOz0MbiCU5rdTpdfYQ31wCKSMMGCKz4TDunMrQQYkX6sK8J16jhYgIEfa2/Hu0+QYieJbehAJxq/oSrb4oGvpp8K2u3Cxt4wLX2a17Ku4UxBb831Zqn/66C0DyMF6+IrwTCun4U3jVpZJNJ92+WrklNqBurB/RQ3g6zomdgqBGfL98HUTckSIH14bNNrnrEsOopb8VxEYAceKQ+9gU0OBmA7ei4ojxkEcbjrGoJ3E1vuzaUAR4GH6uYpBYAYnC15vhMTzobsxYUUzX4kjfZMvxLCqlDLvB1TUaJoN75fZ+G6dEkNb4Kjhp6VhL4IG93sNEuTY/L0DFA2TZ+wSkXxj0C9WoSd7VnwkiD8PxM0rczrQRL7nwlsRrFpuXzsuxwsVuxVL+GHveudqjhAPq9xZdx9Ba6GoJhzNoVozQV74SiUYCftChW+KoN4BOnYhBEYjMKpwQuxpP+DVEWSHRtv9SJ/9YpGeGv+j5o+oCFK4huhjJoiS15Jh6uoI6yR8ZpDxa58BxjOs9q/F7WebEyXcJmLD4IXu8hIpOEebZq+9YbBAUoa23MCvSTDwEfoIVARkF01IqXuJOdgQJWxoPPHq2bXYZb15U64bx1/m4A8oKX2LjbxuBr6DjEzNJZq0WIPhFOa51T8vENGLhEcpLjDZRQ63/mpKGOkt07jN8ljFRoZCR5sdBJLlZR0FckmnQSWjp5oKxpUkS9n283MOjf/lUw7X+NOjcCgehAxU5Hnb1Q8jsSexFLR/qB2+AfTnV1Z2v2NwrliCZgW6aUVZdx2l+TKWLwpRuomP8KD48WzpqizOYzxIW7qTdlFKSIoy1Il9S+DRSAjrxzlNkdJW508SdsSHDsUYKE0RsOLWao/wyO3Do7X/npGiCYA40kE2UAS3Fxjw/Z/iyV9yo7Q3KxwGt+A85kd/KDJXDUrp8fa231+y0+w= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(36860700013)(1800799024)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:26.1684 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 300d8667-9e74-45dd-d7aa-08dd671b85cf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD7.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6356 X-Stat-Signature: hkwdbupb6urxpq67bcb4fkbpdxkbgswj X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 23F0E1C0003 X-Rspam-User: X-HE-Tag: 1742412210-300419 X-HE-Meta: U2FsdGVkX1+v1AaMuYts9UA/bRj/I2kCGvt4vDDinBtnIxtwY0LTfRltnSaklZm+Tk8wB408V3NNWyEl79IECgUSvlhSpC4jMWOCoyDUagQQR+80ivm9fVbwUTInfAYW3VRX+x2/OSvXjwN+BjXGiXBXkN1W8qvbKpQlwUy1tKo77l3PgRA4nGH3Dlp7e/ujBJGHsriRRy2aJc7/10LJJkqh3hlWEeyS92bDBlXOwg/6mmQ1mXFHHGPhkeSeG0BgT80OY9rreXOzxS0FFBVlhEc1sRPAGXP8x5FYU1kwwZWZ2nS4U1ve5NuMtoUQZGrRoblfZlEMJIEWBbKuA6Tng5L38JYLeGALDrP0rwriFBg1od9HVhHzmufQGGLZLFf1vBNyf4xFs8deCQCyOPcCFEICohSlrXm3AAUvl4UnpU9u3bHPrnI/D8gpVVd1wJnVn5rwK8iwmulUglESmiMZhKtmO+ZAhIAOJNBkpv+v4f3cPG0l21y784vU62vOp5Onywr+MObaFWD9dAbCsrY79RmtD2F4Vj3DZQKL0Iuv3T68rDoLu+0UGwtOk1D1AvvlzhjNAo2SyqR+KPg2LkMdM0sY6IiMsmtViZW6hpzwO48qWPRhaPjIkb6zCxQnOj9nPfeBEkUQgDowexqc1XEIhUoQ97qNdhw+h3OobI7V0vLRFRSVpo59u1C1iP7y25yJ7l33L1D3t6F9rollu+WksuESwEQmsuPzHtO1kBdCux8LpAJtg1RVdZN84xroiQHdjz8GA5+nVW8F0Jnm2s2EXkRjpgaX+Wpu3/2WvtriJA5HNA1MrDfm7fc2xOoYIecH1kQEl6RuNOuWRCSi0s//B55I+O4FvOzr9blk/7JveJcQ0aEPwzTuUICyUSHkHwP73sspQGbOjeidtLs0hhMqklFWdKRhRGElPl7swtGIS+3Oyt8u9yUNWY2FVmB78u0ONcNmGsoIWTCGstFe80w eEwJPvQG Z838B8RupeKGFAQw+JXeIE027vdTSHUX48ZJPZkG97isJM95s2yTWllN9DOxOFqKFiHJ9Amha69yuFw0H0LbTZafnQW1dPG0U+4+BGGjSF3GsQw8H2yw85vzotCu1ofd8F9YTX/b+ghFkuII236qc6G76Iv+FFO1Xm32wPPScr3nleRGW0MGFC5SS9UFBmk9MbH1xm+xOVhoPJ5XFL0JcbW4X/Bzsi06mRBR/Wac83GSW4tpLgfAmbthBG1TUL1kFViBUCpIV9fIwgnCg3FIpNXkRybLMgur+fej+gSjPoIZDaWof9Y7v9pJNooIuUaOSPcZmY8q8NKBjDaWtdR3cXucH6shC+7Ar+Qi1sR/YxV3wjcdpgFR/9XLdGqT0Yb1eD9ZlJu3veHAQT7F5A/MlTJMwMJm75LCKf5Ax17et3bEQ8XAPWI+DVuG/M51MNC8+8PZsApJXuBECt7N/svu/LsSTMGvws5v/Cw08p6nNRGbHpC6l2gsKDztuEkyq8t15s9YSDt6/ZFaA5rFXy5KqX9b/evVCnFDavnrp0XGcifvNuFwR8V4B42Vr7gQty8vCcubVrIZwiYdUz5dijlrFPqUVffsnMuNeLdWL7oqjbaeiF/topUrUoGDbgfl0N74JFVkII7cgOKC5eEw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan It is a preparation patch. The added MIGRATE_NO_COPY will be used by the following patches to implement batched page copy functions by skipping folio copy process in __migrate_folio() and copying folios in one shot at the end. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- include/linux/migrate_mode.h | 2 ++ mm/migrate.c | 8 +++++--- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index 265c4328b36a..9af6c949a057 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -7,11 +7,13 @@ * on most operations but not ->writepage as the potential stall time * is too significant * MIGRATE_SYNC will block when migrating pages + * MIGRATE_NO_COPY will not copy page content */ enum migrate_mode { MIGRATE_ASYNC, MIGRATE_SYNC_LIGHT, MIGRATE_SYNC, + MIGRATE_NO_COPY, }; enum migrate_reason { diff --git a/mm/migrate.c b/mm/migrate.c index ce7ddc56e878..0d40ac069cea 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -757,9 +757,11 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, if (folio_ref_count(src) != expected_count) return -EAGAIN; - rc = folio_mc_copy(dst, src); - if (unlikely(rc)) - return rc; + if (mode != MIGRATE_NO_COPY) { + rc = folio_mc_copy(dst, src); + if (unlikely(rc)) + return rc; + } rc = __folio_migrate_mapping(mapping, dst, src, expected_count); if (rc != MIGRATEPAGE_SUCCESS) From patchwork Wed Mar 19 19:22:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA82AC35FFA for ; Wed, 19 Mar 2025 19:23:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06C8A280008; Wed, 19 Mar 2025 15:23:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 01BC0280001; Wed, 19 Mar 2025 15:23:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD9E3280008; Wed, 19 Mar 2025 15:23:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BBC6D280001 for ; Wed, 19 Mar 2025 15:23:43 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B8A6016074D for ; Wed, 19 Mar 2025 19:23:43 +0000 (UTC) X-FDA: 83239275126.04.573C71A Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2041.outbound.protection.outlook.com [40.107.93.41]) by imf26.hostedemail.com (Postfix) with ESMTP id 7AEFA14000D for ; Wed, 19 Mar 2025 19:23:39 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=WlhflszX; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf26.hostedemail.com: domain of shivankg@amd.com designates 40.107.93.41 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412220; a=rsa-sha256; cv=pass; b=upevnQhldKvhJmncxDjh+NOYHSl5qFQHPhT10b6/L1wyEOmhCCZSK6AzED7/gXMT9on9xj 6nutC1TiHqNXdHGys1F2dFD0NPj+sXur5rdOaezrBihXTWgtvwrhABeo+fKDnsbCmAL2qC d+NbXkOM35VwSuyWU06T2cLmar9hjKo= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=WlhflszX; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf26.hostedemail.com: domain of shivankg@amd.com designates 40.107.93.41 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R4pTY4qimGp+eX/PIhEn6Rb2JoweUqEplJQfgDoLjGM=; b=IT4YHdO6ny8ERNU0IqG4qdv0qER108BcyQbH0iLg/+pVXaqMtT7LaKRAvxk/Zf21BE+hUM hXE1enj3ncNG5UBznlrIk5oOiRNk8GkK6LjQzNt/Yn6CWHdx7mGk8yDBNKAG+gBIuQzA6u D92xE3tzf+fNohQxIvknUgsAv6TVcRI= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=m4dUnyTuEVtWO5ixBvw1Mlx/mUrHEoRGfYswiiTDW5igzi0YAA7J+qhTKyW1jhapGg7+e6JhwpIf/ZAL6c59IUe7EzCvJeAMFg2yiU3Vlm6EnLyia9MyJSsMfPRi3uiUxhYApcfL7z+jEbufFBTXyiraQRMxsUobD+0XTcvbEApHMTHTAFXywexbOHnRQZAUd0uvAwnKjJvaqoUrsqyoeF/WbrYknGBSVn0olGsReOnmUWZvtppbXhMOuUJtJ+7KWhfmJ9IcnyEDLtcfTUR8iUrXA/dGWM6DoSbbohrItnu8t42sUWMMnzqd0D3StPD4M+xw5gtcnoRj2uNciyPbnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=R4pTY4qimGp+eX/PIhEn6Rb2JoweUqEplJQfgDoLjGM=; b=kZQB2Dgm8XbVoZ0op+HJRnhpJALAg//kmIQrrn6AJijxM5d6iEFDzmrIzNn1WlFcBfJ3LKHl8svphY3/ZcBbdvjU6F6HPvhryJc2VK8BfywwabjBlxnOfqG8UNTkdtVIqTzODXPBzZ/09zOAY4IU00qQ/WpUdxxYM0jbXQAQ0VMFIg5JTig7mqzUBw/rzWmgwY/1/9AQvNsx8J51Nba3rlDGqclQkhgaiPGqkDRyMWbr3cfqSuL17z12KcuG3X79xk/8eHaNuqZx4AC82wziazCtTTTm6h92zFbA/z5FZwG4CSQD3hooxYVpnMB4sev3KbDeoV+fvNL1LZakKbZXlg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R4pTY4qimGp+eX/PIhEn6Rb2JoweUqEplJQfgDoLjGM=; b=WlhflszXByX/8Yyxr6+7//rUPB0Vwe1e31YteQOOvwUOGg2WE37BGVqiyHYpBtOmW7w4pcn3ppCd9SNP3Zz0Yvzp37V+vfOnWkjp8h8Zj+wfUrl6zdopUUS9fJNnL3/ELD+uFF//uvPyQG4/Z6oo6OsrQf72QQMGvLt35K76Wlc= Received: from DS7PR03CA0061.namprd03.prod.outlook.com (2603:10b6:5:3bb::6) by CY5PR12MB6480.namprd12.prod.outlook.com (2603:10b6:930:33::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:34 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:5:3bb:cafe::b7) by DS7PR03CA0061.outlook.office365.com (2603:10b6:5:3bb::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:23:34 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:34 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:25 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 3/9] mm: batch folio copying during migration Date: Wed, 19 Mar 2025 19:22:06 +0000 Message-ID: <20250319192211.10092-4-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|CY5PR12MB6480:EE_ X-MS-Office365-Filtering-Correlation-Id: 0eba36b8-6c5c-4688-3f01-08dd671b8a85 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: 5K5op3W+FPxeOfUYOGxG0vXfoJ+uNKB5RMlWqQIu6QsHSFIfHWlmjyJKGEF8C5hQw3aos53j5x0zVfoLuM8nLbR0CcNnEHlj7VqXqKhrhfTuPby1mBAl3y5WVizIqWGcYuJdb9ij4utyC18JyH5gBYcnROFfNleg4wn97khtluO7m/nVKCpYHGi/gT+fZ1Xv4Lbn9wHuW/xwfklOz58zW0cssENnvPoFkCTiyhE1RuUXDO9wONvBCLVLns2jB6P9GrKbxV+IJLn+V9YtA3zJnD+8SqQZ5vGPLYN2GxNHPCiz2F+ZMADtcuSXh2tKzwXgA9eytdOVNODXlMLDshytuy3fJ26vem4AXdnAdbiL2iQ6U2Yotll/zH5iLM3+I5CAzqkOdycmND4GHO/1mL3/uhuXenZqLH2OBhJUg1PnkpTjH3sIlwJJOH1dY2l5f/BORbqUi9dN/fxOHJXzRWceoEC3qC7Pzxa5K4Z2qtkUaDAqBYhdpYHLUp2rpVrhHOY4kIcEcZWpwDOaWuMBFs05HcHbXy1+rZ837z+MFm2tIXowTKVSeScxFzRo3FxCEAeqhQM4CQJ5DNXvXjPSjPlFzNjjvHc7Xsxb2hoYnRaUkqsxGQbGeIsPeabfz4xE2lHj3JfG53PW30Q3kmghjUamVDKB7ARqfThSa+OnNN+pmAKwB3M+xerW8igwzAqHI8r2EBjBB3k/+/xZ0ZEUzKLxF8PJlOhL0aOI2IWRVWtUeEDUSjvjbuJJ0+qnkEjySvdTDuxXrq57xG5lgnz0JjjTiZxAzxwqe1nItN8FkRQVJnJSNAreOeMQtUR79OWTjEDy+N/BblSTqkQ4IhpwJdundh9xCIzM4e0JsBtX/0+emORifrZc6McCClbpl4TmgeLSysToFwMSr1j8tXpe2BiRQN9B5UcBMpw39g1jbht5APM0NsdtcWv3htsnDzACw8yHjoleOq4lEgnuPbL1RL4TbTUydqHx51ebDZaJmg0kGgznbs5BW0VnzVa2mA1avNcouFCRxgNyfNN7amv4QIB9xwU/IhFKxYj6B5QYLdm6RwDTrBlsISd0dMSHOYLAgHv/d8iD4VTStY5qvtqzaWA+eTC4e1u/FaxzzV/9/CvbscdZLeLZTpuoyd/tG4PK40y4nsZZz2Q16YtSo3r278MsSbrOsNXP8e2xbbSsNF/VwUPBA9Nnz9kjPZYgS/umjdTVKWCyNCeaeb/Lb+1C9b28HfuTDuuZLalaHMg3VK+/mvA2mmu3g5lFVGBjjCbc2Hp5sq2SF1o8HRZuNMNyZBwX6UR1azYbm5y76NPIcCd6pyjowq+L3vUmt6ckT4b8R+ZrjN0vPW1WbKe8kWjo6/R6Bbxgz6tcQ48iVaF0egbwLlXYc8CuqfnoK3fxWXm69SBsdbDpNoYLnyWT3tT46aPd6UIY1nd/NA+sW8J9Nk62NmUNG/Ja8Y65DwWLBMdd5K1KIkVzqiAcoIzXEJJUTGJbNA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:34.0682 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0eba36b8-6c5c-4688-3f01-08dd671b8a85 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6480 X-Rspam-User: X-Rspamd-Queue-Id: 7AEFA14000D X-Stat-Signature: hufm4amxrjfnt9sa4ug9u6fzoof85c3h X-Rspamd-Server: rspam06 X-HE-Tag: 1742412219-115648 X-HE-Meta: U2FsdGVkX1+vH+cXxyq1aksQgNffsOh2E7ZeDB7Q0yQfQsiPbcem10iIYWsIGlmPi+b7ji1edVPUccWoujGYZw61fETEMxnpDiljsFcETGheJ/hjnTBY0fbNpDsD2o3Nmd1tJ83BRAPlFk7leoeUFt9esKVA9sjO/wxdBu2LyOL5kJmFgzYVOS3OMNyTxhnryblKidMh9NFOomC/SqXUpl6t1fVOp4eA+gXfcaOQA+omXIREXZl6aP3izkA2LMTUQNs6OcC5qENIW7I5kEpqVReCNq9+SSnfZav33XJX0YltiJsmCPUvhLzQHnaycadLaS8S7jGMIo2O+m7WfGEdo3VzNL3uFW9ZCoWWQQ1ryXsZ4bpoho/wSOwkwu0itkF98LxHlS2umOnclBbtSbl2Ra122PiBuvsf8vqMql7usssDi4vN5Vdjqu0uijQpBlgDr0FxJZnjkg3Wl26dMx5rpvacKfn4Ry8lAvGdv9V2nGGea1d8gNcY1uXXeUc8i8B4damCsl2+sViDOOnYYGhmGxYmuDIiibRK5nYq1iL6pf/ou6pXL8c3jIfUr9VnI06IDSg4Rf+k0nelKmwQIgTeLI+OM3GWjq3Gd0grk51FEGzEiFvykehCzXPG6kEapfiBL19ORBBc36SRxskj58lGWqTvW1VpIfJPTOX3K0YDXbSJozltLtPx85y+Uh2UqHn1Ord+TTLo81j6lGHKp3eX14F6T2qaVanFsLTWRK7UcWdsjo6d9qY8YSy3HaPrCGppr1oRhaBpb4p1TnyJBLV23wsXdPTSL/SlgTvDWvuu6oFXFIkTSuMqqrtcaC159xEX6N45kkc0YXF0m+kaLlkIlXbRYUWxIvrZjVecyALrK5qvbmElEWenDFHwBL7BZGfsIGVeq3CTZeMZJJUQWK+tLMP4kGM53LmS41wlaW8RErVDRzlBTT7a7rUTNpthR0td3WZFsc2nonrtv+7Vl/Y F+GY5qvC vsH3sK6W4wVbyxq3C9AbfkYf/+bE0pCPEjzKK7/kBp2c9n8/weVh3tix7snUJYU7c/ucuvqL7+o3j17Ud+gG0cIFDBjwzQ3AKPFxK33Mnv+aA9C+3npVtgs3TS4W00tOLaEd3HxKjIEbPs8jxF5fKMe+4L0H3++DjHL4nEl84EqOrMozVVzuwhU/MQa6BRvWXVQYL0Ggi0N0YR4U/iRkuOX5M425KHMEOoXcrExj0gPLYIFppZEmsoELOcYvlC+p1RKrVObdG4MZ8dupzc0VamEb/+iw6si2qQjhDOy8L48Lu1IpVnrCrj3CI7HSOwg7C301Z52sqolRxLEcd5405Pm/OmuU5Eos69z/dA7BPk72IjDPxScRYK4b/n7/wEoAqcpwsNUbZOORJdxFEhG3NRCxBdvigoGJFh6NxIItKV6k5tiQal0sSSjdSCILvkVtoRE3WQJJEmK1TKfwz0RAU+OFO6NylllwnXlkA9e4c1lLYuSl8bmpBCTIvNLF45Ek701gbT7PHc40uCumgvvuQSHAaYsbdySuxm280jNtrU8BWoodarrwehybLuvOhlwLn0hXB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000047, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce the folios_copy() and folios_mc_copy() to copy the folio content from the list of src folios to the list of dst folios. This is preparatory patch for batch page migration offloading. Signed-off-by: Shivank Garg --- include/linux/mm.h | 4 ++++ mm/util.c | 41 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8483e09aeb2c..612cba3d3dac 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1301,7 +1301,11 @@ void __folio_put(struct folio *folio); void split_page(struct page *page, unsigned int order); void folio_copy(struct folio *dst, struct folio *src); +void folios_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt); int folio_mc_copy(struct folio *dst, struct folio *src); +int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt); unsigned long nr_free_buffer_pages(void); diff --git a/mm/util.c b/mm/util.c index 8c965474d329..5d00d4c5b2dd 100644 --- a/mm/util.c +++ b/mm/util.c @@ -908,6 +908,47 @@ int folio_mc_copy(struct folio *dst, struct folio *src) } EXPORT_SYMBOL(folio_mc_copy); +/** + * folios_copy - Copy the contents of list of folios. + * @dst_list: Folios to copy to. + * @src_list: Folios to copy from. + * + * The folio contents are copied from @src_list to @dst_list. + * Assume the caller has validated that lists are not empty and both lists + * have equal number of folios. This may sleep. + */ +void folios_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt) +{ + struct folio *src, *dst; + + dst = list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + cond_resched(); + folio_copy(dst, src); + dst = list_next_entry(dst, lru); + } +} + +int folios_mc_copy(struct list_head *dst_list, struct list_head *src_list, + int __maybe_unused folios_cnt) +{ + struct folio *src, *dst; + int ret; + + dst = list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + cond_resched(); + ret = folio_mc_copy(dst, src); + if (ret) + return ret; + dst = list_next_entry(dst, lru); + } + + return 0; +} +EXPORT_SYMBOL(folios_mc_copy); + int sysctl_overcommit_memory __read_mostly = OVERCOMMIT_GUESS; int sysctl_overcommit_ratio __read_mostly = 50; unsigned long sysctl_overcommit_kbytes __read_mostly; From patchwork Wed Mar 19 19:22:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EB94C35FFA for ; Wed, 19 Mar 2025 19:23:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 78CA1280009; Wed, 19 Mar 2025 15:23:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 73A91280001; Wed, 19 Mar 2025 15:23:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B4EA280009; Wed, 19 Mar 2025 15:23:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3AC5A280001 for ; Wed, 19 Mar 2025 15:23:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 56278A8DEE for ; Wed, 19 Mar 2025 19:23:51 +0000 (UTC) X-FDA: 83239275462.27.66E5981 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2042.outbound.protection.outlook.com [40.107.223.42]) by imf12.hostedemail.com (Postfix) with ESMTP id 7072F40009 for ; Wed, 19 Mar 2025 19:23:48 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=Lk94Pf9C; spf=pass (imf12.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.42 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412228; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oi6OB1LzwfSy1hb0t8o9q2QfNMHz3zUKojtPDKuE4c8=; b=KpYpkClp2CHK12H1kXXqzHfw8K/+idClrjqJzhzocDPno82u/eScuYrqXp0pSqhgmt+DfF auW4fQ68Oulm+KrOp4jOADni91gDgoNsXTz8J8LfR5/TwyrPaLRFzp4nL32FVeCHSlB3IL uhO8Zdi1s5ouW/u8/vnIcC7jvKmiaWc= ARC-Authentication-Results: i=2; imf12.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=Lk94Pf9C; spf=pass (imf12.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.42 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412228; a=rsa-sha256; cv=pass; b=5JulCh37FtehK8XgikTLow86wt53+0TqizAW/9i4dW6XFSOG4Cn5bJRvzo8Hti0mWJAjST 4ueLIkEBwwLdACqSFV5ghRAemRGy5NYeQzjAcMCW4emUOLFmQy3XA8s0e1JNSPRRCDB2PH bOp/wkH9wAo3w25J/drWuWViqsK2iI4= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZHYBrowxLzylWRqCuSomJ3W2ojvElgZrcEw5b/AGhpwFcnoYwBIMIO7TFrEiraY2RjWRj/0V6jaQGzUjKEPhToSBqCEGxJWif7cH0ItH7HLeELYjc9TGqy9w/8QUf5i9nu+mRuTc1Hvc6xRNM0Xu92UMGec8QD00ZxvO44cqZEM8w5u7g4f3tLf9zmcOhPTr4CTG0eo4ip1Y4enXDazbVJcZol1074R7DDt9neRvW812P2jXsi495XMRRLKMy6MXAWOgC1aKDfGjXl7xDIvBP0dRRy3ZOP9xo9kY4Hs3FNdgDdsYyoNzdb/Pw06MJY2P140qxePxX8pMm97bth59Fw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oi6OB1LzwfSy1hb0t8o9q2QfNMHz3zUKojtPDKuE4c8=; b=D8uok86m0PB2jftkhjK09lAdgGZqpo0SfDjgX6jEEg7Mu7rUZirrzss/i+l/w8gAfjFnwVkS/H2zUDT6MlpgRZz+yGeNaBCEhAhuvth7K3iWSVb67iEbzaoYYZ+ASKyeCVajCVVugllED0xPcHbMCyc0AWOuxchJTSLggJ3y370StbFzsHl826RDKv3OsXDIVQxS9tEEAmqQaM71aggVkYmlrJQ247mq3mhzVkjlj8yT5qRO3vawVMnmgseCH5kJ+7yvcPMbiYIDsoA8dIT6F2NzPq8CU8G8Ix3H0HW+KWXfMzPhASD81VPPy85WjmTsFhOBK1YA8EbipWcNBa3WRA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oi6OB1LzwfSy1hb0t8o9q2QfNMHz3zUKojtPDKuE4c8=; b=Lk94Pf9CuwJPH0mwyc+h1TJjethlp3qZe+CXHRr0Hxq359V6dGtHZhHFGCZvfEDuuyE0JJYbJGboti+y56MmWiOruZWJq9FEWi9G9RZ9CWYdEOP/wRAHtqpkO+yP/SceC5WM/i5ypjvOkCjjZPjc58X3sRe4OmOuBDa/FB0Vx+A= Received: from DS7PR03CA0071.namprd03.prod.outlook.com (2603:10b6:5:3bb::16) by CY5PR12MB6156.namprd12.prod.outlook.com (2603:10b6:930:24::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:42 +0000 Received: from CY4PEPF0000EDD4.namprd03.prod.outlook.com (2603:10b6:5:3bb:cafe::49) by DS7PR03CA0071.outlook.office365.com (2603:10b6:5:3bb::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:23:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD4.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:42 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:33 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 4/9] mm/migrate: add migrate_folios_batch_move to batch the folio move operations Date: Wed, 19 Mar 2025 19:22:07 +0000 Message-ID: <20250319192211.10092-5-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD4:EE_|CY5PR12MB6156:EE_ X-MS-Office365-Filtering-Correlation-Id: be185686-a276-4bb1-03c1-08dd671b8f9b X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|82310400026|7416014|376014; X-Microsoft-Antispam-Message-Info: 4yG82fjcfSYrRunChtn23amHxAVznV+Gi2cCH9bNtqym/WYKDBhGecZeIn3LuUKGUW95Vukl99iE2rkNGvqGkhG5uRhHYTREfvqcQkK8wJfxazWxUoCV3XeHj4MzbFTovqN+A/mSgSz7dZzFKMgTopgOw5TUC2qqzm9Im3nb6ed+W2uszBOZWoVivX2v6SrHJJwB8/0BfV/EXyj7WyBs+dq/N1iv1Au7BwwSym/4Nvo0crPHR/IOpW6/D3ZFNyzM127E8QRSwo7ckJY6qA/LkIBYZevXDtM1HoFUF9lv5TIgIKvXBwFb9Pi4vkTCDdfPVXOLDbdXqG3Ude3GkYJA6cDbe9Ef3773ttLUuzD1zQxLuDckZuwH0l5opuG3nyugemkHg24aE0j2cGuA45HSZQFlalGk/xcSq7rIhYaGIYjNQOggi8Lh5EMhnQMMRCLwWdbtCE+QSc5f88LbkfTWEurIJRhO2xjI2rZXi/SqVFhLdyjmEjMNcA/Sivx+hEU4OzkQM5o8UA6oGObWJfnh4wqISmDP5300sbg9j9ZkkygtBqIJOumyiiuUdSAGzZbq+iCU5R/j+upGwF2AjQJ6V97kkx1jkntcZXwgfcH23xkLawI03nzx/nA8Gzn9Z72EdJ1L/FISFVRzOfe2bhkQzcNBzEj56bMjR5ABz73onoHVVSc+dZY8PTFd8he8EqNOkBTgy+E//ORqtjP+rl1fmEbR9PAKTxLWJxiqA4Xjl8Qmw/UaV6bv/GnenxOJYGt0qS48+GUAOPaE/P8TbMFSygcpCVpiJP1H52MUn38hZCs7AWtItZ91JyxgH18u1aHhDHaEF6SNpFtQt/jsxC7pJzLRlJGrbe+zr3gRwPC6yAymkBUebXW3Vlr56PzsVB6f1MVbvpVRR57jG6S3qae6m6TLmpRiDeAKLpDEExW+qmQhbcC5gE7U67ye0AcPDmtXVpVMhkGd6BHl8/pSkPfFXnNdZb/kEC1ega6mY7G5Y15EYraNzQPVBlY8KF/JGYVYLzKQyEcGD57fRf2VPhRDUPPYjChQht3ObRo7nKefCxqfhU0Y920I2omXTgVhdhUKXtMhv/uxbvHlJHgbuGbNs55Tv0Fuel7HzzSgKQC8YrT+Vpl1V8I4FEWJGDaAO6fNCfL5JIMk4r/dPqwMBc3holnGte4iMKbLP8nDOvSdsuOb9JZmi2RhIBwL7TA/dM+FBZbLnRWZniKyUC1rxLnylFnx9osdXovIK0SgQ14jJOq41rBz8qMpgzwBtR2UafTq0MLlMwOnPkxHp9gJFAgXIWAnwucsLoxQBCfJUiY7BpE5t4ar5dbQM4kDbVkF84SQw9R0339HOLS7DNvlinMsZBqCY09W4J3LmmhyRj/yLaFbOBtM+JkmNRmgBOypymYSb8aRhthhIdlRhsHtlrLk3U0b0t/cVw5Xi0wAjldmLxuNKbBQ60KFAljXNkTc+49lZBMKbZGPXg/QyUwGU5EJt7HmVGG0dtSib5lWoNIxpGU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(82310400026)(7416014)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:42.5994 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: be185686-a276-4bb1-03c1-08dd671b8f9b X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD4.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6156 X-Stat-Signature: 7miq35p6rfswx6zyunghxjw4s4eou849 X-Rspamd-Queue-Id: 7072F40009 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1742412228-179072 X-HE-Meta: U2FsdGVkX1+roUyhZqpLso2gpAxC1vL2UMe2tbW/enH/nPX7w71GYwH6bhjoR5ptV7ZBKYp4x68W2BGzd39mbpw2qJLvfPx6e2BAA97P+zhK79goaipGKQzucO8nvtr08qv4mKTWiUAa9m6EovIzpDo6jzOXnQHPGI3KSR5amq12mtBhqqdJ+KjFFYXoPtzT2RXfGmYYErMSwQFSYy1H5TgcnzVGSFqJHVZ0KLEyJQfD8+/zJ+Pnt/EKjL88MnehwAgy5JStT3aUVOrEs/2Obwk9cvfs9IwX0nM4S52xCeLAYPh0KvaV9ea3aMuGuQSSR0buIDfINMfpNUNZjQbRSJ/sbWLwiV6kk7HGTTXCaHdVjAJeSPEKnW1+qu9ydEOnE0mesQycvr3ul91V/pHnBe3OqKa5kRSJo6vYBUhIjdJKr20hVjXx3hzSWXyYHt+awSMGvP/ICA/n8w8OpzdUYs2Rpxu68mI17MNj5f+dc6AnTeKvNrfreH1XD+Q53ES+Yraj3+KbpevnY84rCxXKR19VaOtWUoNBLQvAhPUzrNr+qjeUdy2U103lPVr0qowriuRqt9gDGPMbNip1keIScl/QxV6RWJMWMESxmQe63iRAle4qXT5Uf/F38vSI4qwrvvKCyx5w+Ir5kwYuCTO9k7qyQLdNjRNBOin1Y+X+nIRtQP7HijIv5DwRdc1Rn4Cm71VU5ePE2OaXbynDLhK/xftWnf08ij2/b7wDt4hVG54rXMicZiYPnsXPjkthA7V7wbIk/6oER9BNTkyyc6UwxWPMgQ0+AMkdA3GUODaHwj3xU3KX0BIEt/0KwHBuy9L/5th0Y/kco8ofwoqfzqVS/foCDnPZ5fWAjVdPVtjN7uOEwZpEFlJ0DORt6MM+RPTjeyUI3s29owCWLKoV+6INbhLANAM6aQJzfvCI6yF+J0ma10DvNUOk3QIMsk+FWG+MJbQZee12Fr/3gOIM2cW FazKiqbl GXAZ5P49f6lbNyMZytmsmz+TqaCwtFlWapXOTLsUE4q8cNxE95qoBoGCwHWRMhtd7ngeQMsV1XTI6SXXOaAVA7GAc5doRa+JjJcy23lPH989vQEgVEZZYIr6bog9B4tbIB2JOcYqLhPAJ7yMOIgsSLVWQIThwORBzNEtzVbUg0QIF7nk3eMSpFSAeCS2U5EZISbPW5wcAHwjc5648MkhgY/UzFlgonZkSXGrzaH+sozRKiVNf8pEIMkBiqHwKklrxkIUXBxu3hAp6UgK2pkjtpLGSqODPaZORJTsOV1QbTa7dJkEGa1+IoU79BpdbM2505Q54tleRZsrBd7XiYVHPS14vFPEcG4DuwYeC0N/ixUzwk9/0Od42MhLqUqEqDUIKxIarb2TDXl5iM+vQuyE64y50hFBljPBvNuYCyeQl1H825b3CQBXln+uWQaIeUGk6h9RbX2n2sfRXxeM0A1bBJZbBfF5V16UTVQJwf1OKwMRijr5Jr9u0yom3HKW2Ls+AdLnUyXLCMaF9560SJ2O6WvDmehYnljxccDeyKGvFFTFQOlgz29Qltiv7/9cGKd0D3atZlFrfK3zcC5JNIJLIyxokRZz7qDH+sXW44Q55FrM/XqKvI2BKahTbABD9zaFlgl3TeG2du/JC1IY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparatory patch that enables batch copying for folios undergoing migration. By enabling batch copying the folio content, we can efficiently utilize the capabilities of DMA hardware or multi-threaded folio copy. It uses MIGRATE_NO_COPY to skip folio copy during metadata copy process and performed the copies in a batch later. Currently, the folio move operation is performed individually for each folio in sequential manner: for_each_folio() { Copy folio metadata like flags and mappings Copy the folio content from src to dst Update page tables with dst folio } With this patch, we transition to a batch processing approach as shown below: for_each_folio() { Copy folio metadata like flags and mappings } Batch copy all src folios to dst for_each_folio() { Update page tables with dst folios } dst->private is used to store page states and possible anon_vma value, thus needs to be cleared during metadata copy process. To avoid additional memory allocation to store the data during batch copy process, src->private is used to store the data after metadata copy process, since src is no longer used. [Zi Yan: Refactor the patch. Improved the original patch by removing the need for an extra mig_info allocation (for storing anon_vma and old page state). Instead, reuse src->private to store the data, making the implementation simpler and efficient.] Signed-off-by: Shivank Garg Signed-off-by: Zi Yan --- mm/migrate.c | 204 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 198 insertions(+), 6 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 0d40ac069cea..8b6cfb60087c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -752,12 +752,15 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, enum migrate_mode mode) { int rc, expected_count = folio_expected_refs(mapping, src); + unsigned long dst_private = (unsigned long)dst->private; /* Check whether src does not have extra refs before we do more work */ if (folio_ref_count(src) != expected_count) return -EAGAIN; - if (mode != MIGRATE_NO_COPY) { + if (mode == MIGRATE_NO_COPY) + dst->private = NULL; + else { rc = folio_mc_copy(dst, src); if (unlikely(rc)) return rc; @@ -771,6 +774,10 @@ static int __migrate_folio(struct address_space *mapping, struct folio *dst, folio_attach_private(dst, folio_detach_private(src)); folio_migrate_flags(dst, src); + + if (mode == MIGRATE_NO_COPY) + src->private = (void *)dst_private; + return MIGRATEPAGE_SUCCESS; } @@ -1044,7 +1051,7 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, mode); else rc = fallback_migrate_folio(mapping, dst, src, mode); - } else { + } else if (mode != MIGRATE_NO_COPY) { const struct movable_operations *mops; /* @@ -1062,7 +1069,8 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, rc = mops->migrate_page(&dst->page, &src->page, mode); WARN_ON_ONCE(rc == MIGRATEPAGE_SUCCESS && !folio_test_isolated(src)); - } + } else + rc = -EAGAIN; out: return rc; } @@ -1140,7 +1148,7 @@ static void __migrate_folio_record(struct folio *dst, dst->private = (void *)anon_vma + old_page_state; } -static void __migrate_folio_extract(struct folio *dst, +static void __migrate_folio_read(struct folio *dst, int *old_page_state, struct anon_vma **anon_vmap) { @@ -1148,6 +1156,13 @@ static void __migrate_folio_extract(struct folio *dst, *anon_vmap = (struct anon_vma *)(private & ~PAGE_OLD_STATES); *old_page_state = private & PAGE_OLD_STATES; +} + +static void __migrate_folio_extract(struct folio *dst, + int *old_page_state, + struct anon_vma **anon_vmap) +{ + __migrate_folio_read(dst, old_page_state, anon_vmap); dst->private = NULL; } @@ -1770,6 +1785,178 @@ static void migrate_folios_move(struct list_head *src_folios, } } +static void migrate_folios_batch_move(struct list_head *src_folios, + struct list_head *dst_folios, + free_folio_t put_new_folio, unsigned long private, + enum migrate_mode mode, int reason, + struct list_head *ret_folios, + struct migrate_pages_stats *stats, + int *retry, int *thp_retry, int *nr_failed, + int *nr_retry_pages) +{ + struct folio *folio, *folio2, *dst, *dst2; + int rc, nr_pages = 0, nr_batched_folios = 0; + int old_page_state = 0; + struct anon_vma *anon_vma = NULL; + int is_thp = 0; + LIST_HEAD(err_src); + LIST_HEAD(err_dst); + + if (mode != MIGRATE_ASYNC) { + *retry += 1; + return; + } + + /* + * Iterate over the list of locked src/dst folios to copy the metadata + */ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + /* + * dst->private is not cleared here. It is cleared and moved to + * src->private in __migrate_folio(). + */ + __migrate_folio_read(dst, &old_page_state, &anon_vma); + + /* + * Use MIGRATE_NO_COPY mode in migrate_folio family functions + * to copy the flags, mapping and some other ancillary information. + * This does everything except the page copy. The actual page copy + * is handled later in a batch manner. + */ + rc = _move_to_new_folio_prep(dst, folio, MIGRATE_NO_COPY); + + /* + * The rules are: + * Success: folio will be copied in batch + * -EAGAIN: move src/dst folios to tmp lists for + * non-batch retry + * Other errno: put src folio on ret_folios list, restore + * the dst folio + */ + if (rc == -EAGAIN) { + *retry += 1; + *thp_retry += is_thp; + *nr_retry_pages += nr_pages; + + list_move_tail(&folio->lru, &err_src); + list_move_tail(&dst->lru, &err_dst); + __migrate_folio_record(dst, old_page_state, anon_vma); + } else if (rc != MIGRATEPAGE_SUCCESS) { + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + + list_del(&dst->lru); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + migrate_folio_undo_dst(dst, true, put_new_folio, private); + } else /* MIGRATEPAGE_SUCCESS */ + nr_batched_folios++; + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + + /* Exit if folio list for batch migration is empty */ + if (!nr_batched_folios) + goto out; + + /* Batch copy the folios */ + rc = folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + + /* TODO: Is there a better way of handling the poison + * recover for batch copy, instead of falling back to serial copy? + */ + /* fallback to serial page copy if needed */ + if (rc) { + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && + folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + rc = folio_mc_copy(dst, folio); + + if (rc) { + /* + * dst->private is moved to src->private in + * __migrate_folio(), so page state and anon_vma + * values can be extracted from (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, + &anon_vma); + migrate_folio_undo_src(folio, + old_page_state & PAGE_WAS_MAPPED, + anon_vma, true, ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_folio, + private); + } + + switch (rc) { + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + *nr_failed += 1; + stats->nr_thp_failed += is_thp; + stats->nr_failed_pages += nr_pages; + break; + } + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + + /* + * Iterate the folio lists to remove migration pte and restore them + * as working pte. Unlock the folios, add/remove them to LRU lists (if + * applicable) and release the src folios. + */ + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, src_folios, lru) { + is_thp = folio_test_large(folio) && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + /* + * dst->private is moved to src->private in __migrate_folio(), + * so page state and anon_vma values can be extracted from + * (src) folio. + */ + __migrate_folio_extract(folio, &old_page_state, &anon_vma); + list_del(&dst->lru); + + _move_to_new_folio_finalize(dst, folio, MIGRATEPAGE_SUCCESS); + + /* + * Below few steps are only applicable for lru pages which is + * ensured as we have removed the non-lru pages from our list. + */ + _migrate_folio_move_finalize1(folio, dst, old_page_state); + + _migrate_folio_move_finalize2(folio, dst, reason, anon_vma); + + /* Page migration successful, increase stat counter */ + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + + dst = dst2; + dst2 = list_next_entry(dst, lru); + } +out: + /* Add tmp folios back to the list to re-attempt migration. */ + list_splice(&err_src, src_folios); + list_splice(&err_dst, dst_folios); +} + static void migrate_folios_undo(struct list_head *src_folios, struct list_head *dst_folios, free_folio_t put_new_folio, unsigned long private, @@ -1980,13 +2167,18 @@ static int migrate_pages_batch(struct list_head *from, /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); - retry = 1; + retry = 0; + /* Batch move the unmapped folios */ + migrate_folios_batch_move(&unmap_folios, &dst_folios, put_new_folio, + private, mode, reason, ret_folios, stats, &retry, + &thp_retry, &nr_failed, &nr_retry_pages); + for (pass = 0; pass < nr_pass && retry; pass++) { retry = 0; thp_retry = 0; nr_retry_pages = 0; - /* Move the unmapped folios */ + /* Move the remaining unmapped folios */ migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, From patchwork Wed Mar 19 19:22:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A4A1C35FFA for ; Wed, 19 Mar 2025 19:24:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29CDE28000A; Wed, 19 Mar 2025 15:23:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24ABA280001; Wed, 19 Mar 2025 15:23:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C5EC28000A; Wed, 19 Mar 2025 15:23:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DEC77280001 for ; Wed, 19 Mar 2025 15:23:58 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E1336C0406 for ; Wed, 19 Mar 2025 19:23:58 +0000 (UTC) X-FDA: 83239275756.18.50FDC57 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2088.outbound.protection.outlook.com [40.107.223.88]) by imf15.hostedemail.com (Postfix) with ESMTP id D7C13A001B for ; Wed, 19 Mar 2025 19:23:55 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QnipFg4+; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf15.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.88 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412236; a=rsa-sha256; cv=pass; b=GpvwFPQg4ymspzkVviSpBIEW3EcCvz6V3BY/H04ZJMr8r5d//dwIVto8MKm4Oyz3ztgaYG W7Q2q8SvuJwSIHXC9wrLRGlzeMWZTNJXHnLEvWXr1cMrMAKgFLSiu3Rgt3l/nXYXTbcNzZ GV3u92/497izRzknfwFSuoOPstZakaU= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QnipFg4+; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf15.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.88 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=C5RXBlzF3qevVTh3B9iP05ObnaGOmXJ4bdq6S2Czr/9OLtprXU3i0qmJDHYEXx62DvPyhM 1iF1tQhwZRSuuqcV0bLrg1n+AyBcTp9DlU2qaYNE5WhblZjXWSGXxcXbNYB+5bKQWhbLSb MOtWdPJefQQxtETEdnIcLgk+XazUJKU= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=LfD8pgNR3dG18aXRfuhxSJKUITEQ1hQwT8uiTUZBZ+WrJm7pX150hn8/2h9Ja0jjdO6jru0oy1ZxcBhNyAY4D7fOnAlnAhw2prqlxJEqZa2kw6BTqL3/BMQxunZBTwM1rly3qtB8osLid5sjTvrbVMWDAO4h/yTsM5xh/ps33g+MVk/rsJevr9UAHY0p8OegqR44oXL1IbRIfFIFBO7PqMP96DO18A0YWESXFaquVM02e97GsYIJrZIgwkhaqzYr2nv7XygvYGAZ29S+lAzh1UZmPHYnlSrddLniMuaQB6rVFEFmKCI3uxdF1eyl5M7CxBmh/ZA8sz+w0TI9Do6waw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=ZOrqcAdPZ3A5grvaDQDzgx5rlI9dYEPbinShaB++dUcN/eyeiWFP2olhM8qWJhsQze//fVjMGJDQlSIuCc20woi8ELf1hg7Gy/02e04MHJE05Ce3YyXE4UTLbsA37DP/zDJFS75HDkj+jwCGnFBcplOUs50gbWBhZ024BNd0kUcoBmq88EETJzkqLetjj6ZUncYztdzxwh9GRyT4jAILE15U6rMgc6l6+9I/fryh7o95R2JH78XxLtxp5MXr0gY4IlzF/i1GXxtSJl8N9ngOcjYqueXaHK3C9bOGgOw9x2/XTpuYto3dPlOPqemOSBZnlK/EeyjR0/DkTz0gDJC2ZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1Uk2NghjfYayPFwNKu3cg0qIrYSvpyALadQw52E+hQQ=; b=QnipFg4+uIIn811W7K3qKeZpmZZc56zTiARSfWh953l8yk/WyUoDUAYd1HEbg8kIWF28bSE9zQvOBCe8uxgLrNaLsQfxr/dPtNqrp3eU+ZbO+ywp55/JxnW/ndd1WzLo+uWRMqrtY1RYclCdA5WUuqhP0t3nSDCzb7Oq/qw2WmM= Received: from CH0PR03CA0303.namprd03.prod.outlook.com (2603:10b6:610:118::16) by MN2PR12MB4336.namprd12.prod.outlook.com (2603:10b6:208:1df::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:50 +0000 Received: from CH1PEPF0000AD7B.namprd04.prod.outlook.com (2603:10b6:610:118:cafe::9) by CH0PR03CA0303.outlook.office365.com (2603:10b6:610:118::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.35 via Frontend Transport; Wed, 19 Mar 2025 19:23:50 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD7B.mail.protection.outlook.com (10.167.244.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:50 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:42 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Mike Day Subject: [PATCH RFC V2 5/9] mm: add support for copy offload for folio Migration Date: Wed, 19 Mar 2025 19:22:08 +0000 Message-ID: <20250319192211.10092-6-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7B:EE_|MN2PR12MB4336:EE_ X-MS-Office365-Filtering-Correlation-Id: ab83a946-6cf6-4667-1e0b-08dd671b9450 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|1800799024|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: gcPZODinQLlEs6YEZGp33h2jgJIRZ6JR5G/qgtIPPGZtG9ntNW9s52KrynBj1nCYzuO3q9Vv16DKoksgjhgDyUUE5ehyD29XpTKj8HRWOs5wX8kvLeclVO+yjmPo/SJTR6IIO6hiG8D/TPdTS3Ob//vIsmeeGsvUJB6FXI9FS2im8ZoR1TRr0UlNnNHf6R/Gr24HxdDNiHwevTZTsWCOcdfEkQ25/ESTvaE/RUiREHQa9BIyLCecPgoqLpx7l38+tBRTIZMrxBjrcddPIVsTux22R6a5YxgfXYKYzAKPgVRqiGF3eXf2wSy+AX/gc7tQJxr7SAYBrWP50pkAqTtgqYnUfhWx7NmUrEGdDZGyhkc/Q1EZMqTLnxHOlP1j4Dwdv16Y0Q3//MNpXH45ChXRvQfoa4OBy8+WhWaIhj26Imc/wVgLDdw4sfhIhVcQLyIapchYPLh5LbTlUP5Wjk1/7SVagkESUrmnr2Wn+vvxJvXPzdtLBWFASqxcyFqo38QeeacvWMjIvwTRBTNiNAQIi4NMoxGe5QGVS1ICjvCX4C2PriiDHdbK9s5hiIckMQNl1BaoyBdrd3zKtDKVQIvqzwDRypEmFYVQ2wttc6hnrRxOIXjpnwI8InWPXVAUzrEQ2AbSPZGraUwxMZgVJveuqjuGV809TpcvhsP1WQbbIM8JbYGhuFiO8DkPmXDP+JIIKSrk37fRc0irq/lj9oTdkpoZKStvvKxG5JTFkaBYXuvBfBjFGF9bDd6UdoV/2gglylIMOmFyCBjtmIRtlIBG2AAd2vi1Grw+9SQTcurHvpQJUofAa45BuDKh8Iio+3aM0U7RZweqoYOJqbFgdIShI3LOC21KUMxbsgf/lmYWfwR6Ts/PkVjH5jX0mWxoJbwqidvBR+5F6jVfp6Dn35K6yVjytTrwltFknokTK2ijC55MT7r1gIuis1u3kGVaE2S+/k+FAqsaLkzKNTvbuSXAhZyTxCX7KGVpt9yUriQ//UItx2rDWCj8mJutlcgaG5meT1iGkZfV7hDH+DlG+oSfW+KHTZ/YHx6Wp/ABHmOgCyefwXpVSbqZ68LAoLUrsmpE/ZhugAg7MlB0+a7XYmoS1L9pjBl24E/BGHqdrmdxHKFgSMUo0mEmsA+5oNOA5MMMA49LYHs8m9LaaKqADhDmfodgBAkWaQI+UleqgWahTi3GgWu6qq5cm4WaAmU6aEz2CropiwRDdFT3Tn81Mpvj48ByX7gZfgmQYPG1U+U35acqd1t7+pba/SpfSc4MDAGRqZTWU8fJOocwngxyL+F/26guxe/hU6NfOCeEMdU7cxr0OKWaj+kgFQ6fuvNozxFXZUhUEw2ytWg8wwyDYt3dPsJJs94xZcvqvvdp14d/cAccnOJp71Xyh7Q+kOIQ9JtI9KHBIcMpqWXmgijsI4F6XHMznRB9OYo796K5bib/gJdIn/AYuQV9XFDcwsiT8aNrk0RIOjDgG/FWrUV8ZRw5LLVe2w8YP9ftAhEFDaHu728= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(1800799024)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:50.5909 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ab83a946-6cf6-4667-1e0b-08dd671b9450 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7B.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4336 X-Rspam-User: X-Rspamd-Queue-Id: D7C13A001B X-Rspamd-Server: rspam05 X-Stat-Signature: snhe1iazyagkcwi7irdmw55mw1by8zcg X-HE-Tag: 1742412235-707460 X-HE-Meta: U2FsdGVkX19zUAg0h8N6NIKVQIAhrQpfysmPki7HzH/UBscs8Fv0tezrA0rJh6TJhbQ/V4J6I+yogseSyH3BlLYKCKlU7e/y1ZukeUcsCbO1kfJdHsmaV1sZKhHI4w9LyvY6PC4mqrwbSv7ZwBb8Ce+cO4MTb0h1uVsM+8C0B46gkXwWED3C4MW4wvg7yzmeFULinfRU5ffCnpNKG925JaNGUeDy8nJT6uuYmCRe8Iub00LsW/SOhfE0IZppibHr/DWTj3qhqBAYp27P9VWkmY58MUZaFEsp28Mh2bJRYDSD1ZLpnNxxSfemcpqg6nUrQbEDpTdl+DVjEkdXWk+/PkPFpEm8Df6vwsK6zIW8fAhLWLAQDVSZAwDiPiMsgflRa1C7T3SWT5er7pBRm0kQCgdI7HLN4iAHX1mLkr8qTQEpR+FyOoogvZ2bHbWhGPEMU7q3UwT1XL85Bk/vUMg9l5JYOZwRRFEoD/Jag1pxZsq+Pk8XIa77Bbl9kdq9Bxg2a0ZoFLi+YTg3IJP1dMmQzipcx83oIlhQOA49Ua1fl3JhhZA26g6dXVUqh2klgLRzUPgwrhrWWgNx6lQsH93t9iMEJW3SRlGjXFfwELvibf34zW0qJiIQs5gTy3Y3m0fQ2wOp43jWUF/BsuvAebVOf0O9CKNuvYVP2l8fdiJBHxUjzCGBjFm5dHHJCxzYOPlubxEfV6ye7Oi6eHrdfU50GoonqVL42XikYzNVSGR7DvBH2unGjvAt73HyZC1y/K3EGPPY/B9qjavKcnlgKHgWySK7vQ2i4wXhPmXUyL2iOv1sfPyDb30ILp+a8LlPagLEe6THwWVeaPjTcg95BWu9jR8A9krk9x5DFfXP3IAjH/izVIw3VWUaAoO3WF5vhmLuh6kx7BmmnJs5GiCnA1zV8vtBN+WyBvMBsPz3IxWbwqpILquWhX0psfh9VY7N42MttcZp6D/i89aH+DmTvBT UyyxHOJ9 w37mDPlxkZqHLwCvqryyV09UtXwa7ImYJ7r+G3F0zPXeLdjhicuLf3kVgt+ZqzyCgnTFe/d45jzQML+cVN7G7C76PKZ8eNpttZwlJ/kvOdres3V1n0nC/tvQCpauNvcU+mLxLY14muLChNdBK2W6avDhdFj71b4+I/Cm/g//EDR3XG6rn8Bq3UsSI3w1oPXjZCqnHzV0Or4/b1kppJVJKt0VADAKH44U0LY/RSwGXzAcSmSorAVr7qkSH3aK5InQ1F3ZmtcGpkTRvtH4eBykFYprh7XJM0rFtGv2WvC+SsPv17eapItv+6MAVBN25s942R43pw1R+4DX9T6FkUSaIe63fLTrin3drP440fZu/B/PzPKd3JyIpxmOrcWpY3uXs5ojH3lMVHLCSbH1AO93P1gC85stjCfTrXXjcbViqHte2m6iKU5gt4IXYJizPYvjv7DljD0LLLgSwtdkYBPD3TIRwsmybtSiTDN9Zi0fWxlMXyRXDQkmNeJmf4qc3JUHaPLWbxmvOh9Hl/78iriVMy1/k3Gywau4kZYlqulLt+2BcGaQBudfjv8YDZxVS/6FclkvN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Mike Day Offload-Copy drivers should implement following functions to enable folio migration offloading: migrate_offc() - This function takes src and dst folios list undergoing migration. It is responsible for transfer of page content between the src and dst folios. can_migrate_offc() - It performs necessary checks if offload copying migration is supported for the give src and dst folios. Offload-Copy driver should include a mechanism to call start_offloading and stop_offloading for enabling and disabling migration offload respectively. [Shivank: Rename the APIs and files to generalize the original DMA-specific offload implementation to support various copy offloading mechanisms such as DMA engines, CPU multi-threading, or other hardware accelerators.] Signed-off-by: Mike Day Signed-off-by: Shivank Garg --- include/linux/migrate_offc.h | 36 +++++++++++++++++++++++++ mm/Kconfig | 8 ++++++ mm/Makefile | 1 + mm/migrate.c | 43 ++++++++++++++++++++++++++++-- mm/migrate_offc.c | 51 ++++++++++++++++++++++++++++++++++++ 5 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 include/linux/migrate_offc.h create mode 100644 mm/migrate_offc.c diff --git a/include/linux/migrate_offc.h b/include/linux/migrate_offc.h new file mode 100644 index 000000000000..908f81ebd621 --- /dev/null +++ b/include/linux/migrate_offc.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _MIGRATE_OFFC_H +#define _MIGRATE_OFFC_H +#include + +#define MIGRATOR_NAME_LEN 32 +struct migrator { + char name[MIGRATOR_NAME_LEN]; + int (*migrate_offc)(struct list_head *dst_list, struct list_head *src_list, int folio_cnt); + bool (*can_migrate_offc)(struct folio *dst, struct folio *src); + struct rcu_head srcu_head; + struct module *owner; +}; + +extern struct migrator migrator; +extern struct mutex migrator_mut; +extern struct srcu_struct mig_srcu; + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head); +void offc_update_migrator(struct migrator *mig); +unsigned char *get_active_migrator_name(void); +bool can_offc_migrate(struct folio *dst, struct folio *src); +void start_offloading(struct migrator *migrator); +void stop_offloading(void); +#else +static inline void srcu_mig_cb(struct rcu_head *head) { }; +static inline void offc_update_migrator(struct migrator *mig) { }; +static inline unsigned char *get_active_migrator_name(void) { return NULL; }; +static inline bool can_offc_migrate(struct folio *dst, struct folio *src) {return true; }; +static inline void start_offloading(struct migrator *migrator) { }; +static inline void stop_offloading(void) { }; +#endif /* CONFIG_OFFC_MIGRATION */ + +#endif /* _MIGRATE_OFFC_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..7a0693c3be4e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -722,6 +722,14 @@ config MIGRATION config DEVICE_MIGRATION def_bool MIGRATION && ZONE_DEVICE +config OFFC_MIGRATION + bool "Migrate Pages offloading copy" + def_bool n + depends on MIGRATION + help + An interface allowing external modules or driver to offload + page copying in page migration. + config ARCH_ENABLE_HUGEPAGE_MIGRATION bool diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..010142414176 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -93,6 +93,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o +obj-$(CONFIG_OFFC_MIGRATION) += migrate_offc.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o diff --git a/mm/migrate.c b/mm/migrate.c index 8b6cfb60087c..862a3d1eff60 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -44,6 +44,7 @@ #include #include #include +#include #include @@ -743,6 +744,37 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio) } EXPORT_SYMBOL(folio_migrate_flags); +DEFINE_STATIC_CALL(_folios_copy, folios_mc_copy); +DEFINE_STATIC_CALL(_can_offc_migrate, can_offc_migrate); + +#ifdef CONFIG_OFFC_MIGRATION +void srcu_mig_cb(struct rcu_head *head) +{ + static_call_query(_folios_copy); +} + +void offc_update_migrator(struct migrator *mig) +{ + int index; + + mutex_lock(&migrator_mut); + index = srcu_read_lock(&mig_srcu); + strscpy(migrator.name, mig ? mig->name : "kernel", MIGRATOR_NAME_LEN); + static_call_update(_folios_copy, mig ? mig->migrate_offc : folios_mc_copy); + static_call_update(_can_offc_migrate, mig ? mig->can_migrate_offc : can_offc_migrate); + if (READ_ONCE(migrator.owner)) + module_put(migrator.owner); + xchg(&migrator.owner, mig ? mig->owner : NULL); + if (READ_ONCE(migrator.owner)) + try_module_get(migrator.owner); + srcu_read_unlock(&mig_srcu, index); + mutex_unlock(&migrator_mut); + call_srcu(&mig_srcu, &migrator.srcu_head, srcu_mig_cb); + srcu_barrier(&mig_srcu); +} + +#endif /* CONFIG_OFFC_MIGRATION */ + /************************************************************ * Migration functions ***********************************************************/ @@ -1028,11 +1060,15 @@ static int _move_to_new_folio_prep(struct folio *dst, struct folio *src, { int rc = -EAGAIN; bool is_lru = !__folio_test_movable(src); + bool can_migrate; VM_BUG_ON_FOLIO(!folio_test_locked(src), src); VM_BUG_ON_FOLIO(!folio_test_locked(dst), dst); - if (likely(is_lru)) { + can_migrate = static_call(_can_offc_migrate)(dst, src); + if (unlikely(!can_migrate)) + rc = -EAGAIN; + else if (likely(is_lru)) { struct address_space *mapping = folio_mapping(src); if (!mapping) @@ -1868,7 +1904,10 @@ static void migrate_folios_batch_move(struct list_head *src_folios, goto out; /* Batch copy the folios */ - rc = folios_mc_copy(dst_folios, src_folios, nr_batched_folios); + rc = static_call(_folios_copy)(dst_folios, src_folios, nr_batched_folios); + /* TODO: Is there a better way of handling the poison + * recover for batch copy and falling back to serial copy? + */ /* TODO: Is there a better way of handling the poison * recover for batch copy, instead of falling back to serial copy? diff --git a/mm/migrate_offc.c b/mm/migrate_offc.c new file mode 100644 index 000000000000..c632928a7c27 --- /dev/null +++ b/mm/migrate_offc.c @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +atomic_t dispatch_to_offc = ATOMIC_INIT(0); +EXPORT_SYMBOL_GPL(dispatch_to_offc); + +DEFINE_MUTEX(migrator_mut); +DEFINE_SRCU(mig_srcu); + +struct migrator migrator = { + .name = "kernel", + .migrate_offc = folios_mc_copy, + .can_migrate_offc = can_offc_migrate, + .srcu_head.func = srcu_mig_cb, + .owner = NULL, +}; + +bool can_offc_migrate(struct folio *dst, struct folio *src) +{ + return true; +} +EXPORT_SYMBOL_GPL(can_offc_migrate); + +void start_offloading(struct migrator *m) +{ + int offloading = 0; + + pr_info("starting migration offload by %s\n", m->name); + offc_update_migrator(m); + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 1); +} +EXPORT_SYMBOL_GPL(start_offloading); + +void stop_offloading(void) +{ + int offloading = 1; + + pr_info("stopping migration offload by %s\n", migrator.name); + offc_update_migrator(NULL); + atomic_try_cmpxchg(&dispatch_to_offc, &offloading, 0); +} +EXPORT_SYMBOL_GPL(stop_offloading); + +unsigned char *get_active_migrator_name(void) +{ + return migrator.name; +} +EXPORT_SYMBOL_GPL(get_active_migrator_name); From patchwork Wed Mar 19 19:22:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90C72C36000 for ; Wed, 19 Mar 2025 19:24:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AE2128000B; Wed, 19 Mar 2025 15:24:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95D17280001; Wed, 19 Mar 2025 15:24:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B13D28000B; Wed, 19 Mar 2025 15:24:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 587A1280001 for ; Wed, 19 Mar 2025 15:24:08 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8CA2680744 for ; Wed, 19 Mar 2025 19:24:08 +0000 (UTC) X-FDA: 83239276176.06.DB68055 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2075.outbound.protection.outlook.com [40.107.223.75]) by imf21.hostedemail.com (Postfix) with ESMTP id 7CBAA1C001B for ; Wed, 19 Mar 2025 19:24:05 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=q1bOdpHS; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf21.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.75 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412245; a=rsa-sha256; cv=pass; b=huwQgOWyFUZeKOvx7KvqCNaB5SGsv0YttvkXp3PBGQGyPD7JGRc0nmSp4Yk+5jsp9Hr70H fvZRMEmhPqtVEsKQ3q57DVbTFXzZpYKJyb2R+qoIq4kgeLD/V/Ntw/WX4Er2TtX405iy11 lut+Wg8/sOcicUvvTY30KTpW6HFCDqM= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=q1bOdpHS; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf21.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.75 as permitted sender) smtp.mailfrom=shivankg@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=flOo3vLO2rJeiBptjrTWYFaJiojxSG0aBTYgf9G0Q9s=; b=CCf/ku9SX+oDmeQNyQlgz77Rsl+XZ/TxMDOxT6X9xV09YGnIicUxxgXfwD/Y2BW+hgvdKD Xy8lOgWH/7KnlGztdVdk6H/FAfL/fBedf/lrat/ms7iFC30j3FQpfxTE5xDLCSfHJZGu1y WgDJZHjpZPlGKz89qVMoQipdJQ7l/WA= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lUieM0gYppq/RbWGvc33RPxKs0d9/Gd1bcKiCxDAgTiKlaZPQELdyyclNAp0ItjQ45NFgkyN+feBUK4LEZFDJN8S5XEIOOnl5/TArtIxm46iUlEPtUj6m7rsOFoyHXShfmvSlc0iMymPZRNwTvT2Gy2Ej2jHKVN24HAqcVDInIjY3ukOgB5rddjb8JenkOEUVtxp93x68jciSr8IpfuQZpoEgNRj4U9Y9TT8JBUBKQtBRD/6U7xgi29tvJbFi2i0/6ILP94i9mvyL6k9LZxuCVxLajkDyh/fWz8H1aCRad3hCvcfMIxiEr9rh1RKnGLAR9IDLrgU8mQyuwA0NXWVVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=flOo3vLO2rJeiBptjrTWYFaJiojxSG0aBTYgf9G0Q9s=; b=XlNAhSEkmKEAm1qJSKRwbsv9LLdFPftwEJ6AWI5pFnixBALHFlzEVwDzWzFzN6k14cwXDsxJlpacYp6K7sRJhqI29311CwGtXJgh/92VWyNCJP6WTHWC8gp+drtBV+4TyqL9ukUxyABCqo4ez8FMVi+bN0h4BmXhQlYT1BbmzQuuU2MDbiLGuDaye02tg2YUqM7pHP4lEeUVlbsV6THHPFxV8QgShM5dOr59kRrjAm7nuCl8M1W1ltTy+hvgKuJYtLrI/oBDr76mnxnv7zLkQVDNdLn9I2iuAr/Bkxdxf1BXfQnjvsYvD7ocQcq/wnSSQvvsqDL21EGwi+4q/7/TQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=flOo3vLO2rJeiBptjrTWYFaJiojxSG0aBTYgf9G0Q9s=; b=q1bOdpHSzm+929plILyPP2MaFBZQkRuCDkqUtOBtd/Bpi7Vu6FEMrc+WnyCAuZY4ZKCrElbZAMUm4osSJA1ejWM/CxmmY3d+zqzI7rIReOgfbq3hBxs24gid7m3Gh5avDGmD6SGXIdfs++9xVjAVqCrM9pqyMpVLmrGMueYnUxE= Received: from CH0PR07CA0015.namprd07.prod.outlook.com (2603:10b6:610:32::20) by DS0PR12MB8563.namprd12.prod.outlook.com (2603:10b6:8:165::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:23:59 +0000 Received: from CH1PEPF0000AD76.namprd04.prod.outlook.com (2603:10b6:610:32:cafe::de) by CH0PR07CA0015.outlook.office365.com (2603:10b6:610:32::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:23:58 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD76.mail.protection.outlook.com (10.167.244.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:23:58 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:50 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 6/9] mm/migrate: introduce multi-threaded page copy routine Date: Wed, 19 Mar 2025 19:22:09 +0000 Message-ID: <20250319192211.10092-7-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD76:EE_|DS0PR12MB8563:EE_ X-MS-Office365-Filtering-Correlation-Id: 782e3ac6-43ea-462c-fa77-08dd671b9926 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: MzvXbJP6hhfW9pg5Pbpd3JDkG0nWx8nG9XvPpl717xPcaQv4uot6gkGyEiUVeZvXgeIxYEkOyIf4XHlhIh3wQnpxTgkY6XtZK8rVUhH0JcXam8ljtyE3IJUduhyUgQFxV/mXillyndg5fUo5CWpJvuvtEPwUiGbjZjRzLe0aWzcpRrnp6PqrK3y1KPqndhQ4/uJovRFRaVDKDlwAlszv0ifWXjdPIClrOuPhLV8T9BNb2gF/NpgMemk344gFgAOR2avQf/a0CfMJ2xRn7kxbC612au4spFSikE6xslp209HBGcF1CqA9ztLUOOZ1+WwACxDyhvQzEVq/Dxhed1qLurBSah7CfrNC9y52G4yPE+2H7LuiUWQxRNBcy5itCLxfBfftDXwlW+bkGpn74reEb6WVWAV/J6FKMz7McW8Xvn5mlmf6w2hzKAAGUXreKiqGWtTydGOauGzhIP4aE3ZRr3YFA2ICq5c/MKOQQQG1ZiOvVpmFfEzYu8NMjiDLqA7QBNHZGGCvDuzBw8CBXnbgZeYBgdpYcPdQk/One5B5yYt9JBUgKcBMS7HuCg7CsOwk0N+XZTM44tt3ENHTL9R6zyALqxwMEEZt/hVfqGFkE1vOT13RH3BRic/WM0370feNDrEyLXiNm2VAT9jLi4FqdhqmGx5gJwnuBY/o4UoWu0EzdYIeC5bSEXyBYmnm5hoEN8XaIvdzzNIsECJqLfZ9QMsNXtMJAYoSQeGRTg/iZ1SpMheHmORJgzzOPlq4oqqvPW2nqTbwjgrcjckZXS79SHd+m4EXSNCdHyEduJcKdNiXzf0De2ajW+SlJnrIDm7e5ScNAhO18M4cui4WZe0lNSPy7Kjfnkz2jl96jlcw7ek3ZPyG9VmpEzfrabUEJSGN6l2lExoQZHrWXemFsE8/ZuDyquK/BNV4OdANHKK2AnlODqSmmnVExA//i5xtU+5cyMUlH7dFn6vMYHm1e3IjNN4aPcNnrnrSiWZYlybhA3xrMOH3Hz8EFfdVODPuK2sYvPVurPVns/iAHeGSOgSfrwyll79zJqdElmbr/4KpEz1/wGrLP6L8zARSI9gPLF9ZBRgOBTO/M42LWKn0V5imSUrA2fdMNaCT3NVlM2pirGrBUungPEVQ2VbCSDumN43qdDUWzadvJYhlPTOvgSoR9Nj9AsogWb8yvMJIgc/KH5EVtqjJALOecFf3giskAXEx2j41PoFbgwmFadzCaub0pJZhkbMq+1O+CI6rdM51/GaxBX7wQPG/ssoqiOdycQG7EmMro9Rdfzdl3FzN5GhrIxbJU/nMjAc61CBF68IYUlQ+PgjdamyajH5idcbCL8akjXd+2+nxJNim98/YRsYdLBrJ2eDQiU5f7yoIZfiWLDyjCJOVJaT0SuNgQe1PbN9v6NqWFNWqxNc1EmDiwQahnvHbDXKY1UYAHI7V6xCiEd9BHsX4duz2Y+iv2zeD8Ny+F70sLS9sfC/0Z2o7SuWefqABm3otTe+5WRaZ7o1cqB8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:23:58.5849 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 782e3ac6-43ea-462c-fa77-08dd671b9926 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD76.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8563 X-Rspam-User: X-Rspamd-Queue-Id: 7CBAA1C001B X-Rspamd-Server: rspam05 X-Stat-Signature: 3ibwqw5ne679no5szyd5qszyjyooagab X-HE-Tag: 1742412245-309728 X-HE-Meta: U2FsdGVkX1+bUA85nP9GFSASNpzmRZL5dSbCKaLLwtATgMCFRD5N34gXVrGEaBcv62ZgpO2BfOd6Zz6Elbg0QY1LVcnOvIhF1W+AG3l3cObrVuKnXRSlP43pnHR3ptgEMK4N0su8Qnyzz/HWZ18mwTPXTrliU+hBcjgLbumJkqh8oaAp8cxHxLjd2276YuJprkru/yLHnDy+Hv7XMOQb22gyWrXkOig3/abHU7/Z8raJvE5VaMoe8qm5vyN6Uh7+jQq47/G6OXEPLA3magwZnEGmf9SnJil5QBmn9DP6FOL8m/rrlXEKYo8K//QkYVsyYfqpQcZufw0bO7NIObZJUi3KnOJu/ubEoVjRJWJFuF/uSSvWtINDxFqAXin2ZR5NereVzno0CwbxZST707Pw8kZYZ4kEZh7R1pCxXxDwGqkKfHSAB9MVAjDrKUW6a0NGncAAlSiyUgDOsmxtfeT0uAEyXXCHq0R7P3nHltH4Oy64aMZmvxV4UPEJ04Q27MmJB/iwFB12g8c2EAZhOrFBH2YKOJJzciBAXeR2f+/pNfrwQSJt/14ZguRjuaqDqNKjbb50HFdponugfoHXMpojvdz3eae29qdslbVBo5k3QnZfWFsfAIvSbQDIJAEjw0isxP+vV7OirwqJ8hgh7q+MUoxNyyNQrTbXAzMPHZg/QEr4bbnK2Wpr+UtBm7VPATuBXsJeBBlgjk4BIb5hDAYm9XL2/NYIZKU5ay0Oth0/PqMqBR69nxrZ4/9kYXP6Pb5O52RPoLtuBIX7k0XsJ7gMkPtSmqvf4zaOB2q6N6Y4gn5z4zchVWI3RWPXgo6lFWffeaQitqS+gpmifkQdQXOhvdcL+UymQ+4iA3nzHvKq5LfTwqs13nrj2r2ebY9vonioiNIcN2kibNZ1nE2sxJUGpgQhfAqh13Z2iwLnvXyhfsx0T5mt16LYtsEjEDfE/I+t25gs3geVC1BF80gZuIu soNlFh2R 3VK7Cb8bSSJ/kD87+D0qWO4rHNo+SgoYBCHlb9BvuS7FJMWcN2OWpR9qiKTteY4kpdXFIu2Xh4eoX3V2ndOdgAbUxbVpH0gC7yrSDwmHUj9ZjVbp6Pf96zgR0NVQYL+ft0ecGJpiyYyHMrOUiLRRx+F+JHyuijKc9+y9Y9WVRQpQ5tqprhyuKnFFbSPMoW1rdm/4NMIDOcdhuU4K74L5tVF18l+eLu++f5vzci2+L97roJlUeU7u9hCy/r9KdUbf30TyZ4IXLDaPR/QJcbLdGk9rssWZlYo0TtWESQgezKzmE/7/iMMvBUcVX8g/g+GrWxJNNhLS/HweqKQdXDvybuVLLdiVCZn56IzV/eHV0n7Q6oR0ir7BJY7HulFVR9Mrm2xeiXOn9b/CBV2PBmbEFqttmuotOPHkEDVN0dnNE9I05fchkdKP059WT0fvVi7S1jY5HmXQgr1MrVJl7MUfL624kncv2I0KW/cLGpVWQNxMw882N7VPEI373qwfGn+MtAX9YGFi5xJMX8Pgy+jBWVAVdqzruKua70pSMNVBZVRLwFQLeRg1SOcjRVfKvACF95WQJF4j0mZbcUgiCkcO7+hQDPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan Now page copies are batched, multi-threaded page copy can be used to increase page copy throughput. Enable using: echo 1 > /sys/kernel/cpu_mt/offloading echo NR_THREADS > /sys/kernel/cpu_mt/threads Disable: echo 0 > /sys/kernel/cpu_mt/offloading [Shivank: Convert the original MT copy_pages implementation into a module, leveraging migrate offload infrastructure and sysfs interface.] Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- drivers/Kconfig | 2 + drivers/Makefile | 3 + drivers/migoffcopy/Kconfig | 9 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/Makefile | 1 + drivers/migoffcopy/mtcopy/copy_pages.c | 337 +++++++++++++++++++++++++ mm/migrate.c | 11 +- 7 files changed, 357 insertions(+), 7 deletions(-) create mode 100644 drivers/migoffcopy/Kconfig create mode 100644 drivers/migoffcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/Makefile create mode 100644 drivers/migoffcopy/mtcopy/copy_pages.c diff --git a/drivers/Kconfig b/drivers/Kconfig index 7bdad836fc62..2e20eb83cd0b 100644 --- a/drivers/Kconfig +++ b/drivers/Kconfig @@ -245,4 +245,6 @@ source "drivers/cdx/Kconfig" source "drivers/dpll/Kconfig" +source "drivers/migoffcopy/Kconfig" + endmenu diff --git a/drivers/Makefile b/drivers/Makefile index 45d1c3e630f7..4df928a36ea3 100644 --- a/drivers/Makefile +++ b/drivers/Makefile @@ -42,6 +42,9 @@ obj-y += clk/ # really early. obj-$(CONFIG_DMADEVICES) += dma/ +# Migration copy Offload +obj-$(CONFIG_OFFC_MIGRATION) += migoffcopy/ + # SOC specific infrastructure drivers. obj-y += soc/ obj-$(CONFIG_PM_GENERIC_DOMAINS) += pmdomain/ diff --git a/drivers/migoffcopy/Kconfig b/drivers/migoffcopy/Kconfig new file mode 100644 index 000000000000..e73698af3e72 --- /dev/null +++ b/drivers/migoffcopy/Kconfig @@ -0,0 +1,9 @@ +config MTCOPY_CPU + bool "Multi-Threaded Copy with CPU" + depends on OFFC_MIGRATION + default n + help + Interface MT COPY CPU driver for batch page migration + offloading. Say Y if you want to try offloading with + MultiThreaded CPU copy APIs. + diff --git a/drivers/migoffcopy/Makefile b/drivers/migoffcopy/Makefile new file mode 100644 index 000000000000..0a3c356d67e6 --- /dev/null +++ b/drivers/migoffcopy/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) += mtcopy/ diff --git a/drivers/migoffcopy/mtcopy/Makefile b/drivers/migoffcopy/mtcopy/Makefile new file mode 100644 index 000000000000..b4d7da85eda9 --- /dev/null +++ b/drivers/migoffcopy/mtcopy/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_MTCOPY_CPU) += copy_pages.o diff --git a/drivers/migoffcopy/mtcopy/copy_pages.c b/drivers/migoffcopy/mtcopy/copy_pages.c new file mode 100644 index 000000000000..4c9c7d90c9fd --- /dev/null +++ b/drivers/migoffcopy/mtcopy/copy_pages.c @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Parallel page copy routine. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_NUM_COPY_THREADS 64 + +unsigned int limit_mt_num = 4; +static int is_dispatching; + +static int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items); +static bool can_migrate_mt(struct folio *dst, struct folio *src); + +static DEFINE_MUTEX(migratecfg_mutex); + +/* CPU Multithreaded Batch Migrator */ +struct migrator cpu_migrator = { + .name = "CPU_MT_COPY\0", + .migrate_offc = copy_page_lists_mt, + .can_migrate_offc = can_migrate_mt, + .owner = THIS_MODULE, +}; + +struct copy_item { + char *to; + char *from; + unsigned long chunk_size; +}; + +struct copy_page_info { + struct work_struct copy_page_work; + int ret; + unsigned long num_items; + struct copy_item item_list[]; +}; + +static unsigned long copy_page_routine(char *vto, char *vfrom, + unsigned long chunk_size) +{ + return copy_mc_to_kernel(vto, vfrom, chunk_size); +} + +static void copy_page_work_queue_thread(struct work_struct *work) +{ + struct copy_page_info *my_work = (struct copy_page_info *)work; + int i; + + my_work->ret = 0; + for (i = 0; i < my_work->num_items; ++i) + my_work->ret |= !!copy_page_routine(my_work->item_list[i].to, + my_work->item_list[i].from, + my_work->item_list[i].chunk_size); +} + +static ssize_t mt_offloading_set(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode = kstrtoint(buf, 0, &action); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + /* + * action is 0: User wants to disable MT offloading. + * action is 1: User wants to enable MT offloading. + */ + switch (action) { + case 0: + mutex_lock(&migratecfg_mutex); + if (is_dispatching == 1) { + stop_offloading(); + is_dispatching = 0; + } else + pr_debug("MT migration offloading is already OFF\n"); + mutex_unlock(&migratecfg_mutex); + break; + case 1: + mutex_lock(&migratecfg_mutex); + if (is_dispatching == 0) { + start_offloading(&cpu_migrator); + is_dispatching = 1; + } else + pr_debug("MT migration offloading is already ON\n"); + mutex_unlock(&migratecfg_mutex); + break; + default: + pr_debug("input should be zero or one, parsed as %d\n", action); + } + return sizeof(action); +} + +static ssize_t mt_offloading_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", is_dispatching); +} + +static ssize_t mt_threads_set(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int ccode; + unsigned int threads; + + ccode = kstrtouint(buf, 0, &threads); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + if (threads > 0 && threads <= MAX_NUM_COPY_THREADS) { + mutex_lock(&migratecfg_mutex); + limit_mt_num = threads; + mutex_unlock(&migratecfg_mutex); + pr_debug("MT threads set to %u\n", limit_mt_num); + } else { + pr_debug("Invalid thread count. Must be between 1 and %d\n",MAX_NUM_COPY_THREADS); + return -EINVAL; + } + + return count; +} + +static ssize_t mt_threads_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", limit_mt_num); +} + +static bool can_migrate_mt(struct folio *dst, struct folio *src) +{ + return true; +} + +int copy_page_lists_mt(struct list_head *dst_folios, + struct list_head *src_folios, int nr_items) +{ + struct copy_page_info *work_items[MAX_NUM_COPY_THREADS] = {0}; + unsigned int total_mt_num = limit_mt_num; + struct folio *src, *src2, *dst, *dst2; + int max_items_per_thread; + int item_idx; + int err = 0; + int cpu; + int i; + + if (IS_ENABLED(CONFIG_HIGHMEM)) + return -ENOTSUPP; + + if (total_mt_num > MAX_NUM_COPY_THREADS) + total_mt_num = MAX_NUM_COPY_THREADS; + + /* Each threads get part of each page, if nr_items < totla_mt_num */ + if (nr_items < total_mt_num) + max_items_per_thread = nr_items; + else + max_items_per_thread = (nr_items / total_mt_num) + + ((nr_items % total_mt_num) ? 1 : 0); + + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + work_items[cpu] = kzalloc(sizeof(struct copy_page_info) + + sizeof(struct copy_item) * + max_items_per_thread, + GFP_NOWAIT); + if (!work_items[cpu]) { + err = -ENOMEM; + goto free_work_items; + } + } + + if (nr_items < total_mt_num) { + for (cpu = 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + work_items[cpu]->num_items = max_items_per_thread; + } + + item_idx = 0; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + unsigned long chunk_size = PAGE_SIZE * folio_nr_pages(src) / total_mt_num; + char *vfrom = page_address(&src->page); + char *vto = page_address(&dst->page); + + VM_WARN_ON(PAGE_SIZE * folio_nr_pages(src) % total_mt_num); + VM_WARN_ON(folio_nr_pages(dst) != folio_nr_pages(src)); + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + work_items[cpu]->item_list[item_idx].to = + vto + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].from = + vfrom + chunk_size * cpu; + work_items[cpu]->item_list[item_idx].chunk_size = + chunk_size; + } + + item_idx++; + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + + for (cpu = 0; cpu < total_mt_num; ++cpu) + queue_work(system_unbound_wq, + (struct work_struct *)work_items[cpu]); + } else { + int num_xfer_per_thread = nr_items / total_mt_num; + int per_cpu_item_idx; + + + for (cpu = 0; cpu < total_mt_num; ++cpu) { + INIT_WORK((struct work_struct *)work_items[cpu], + copy_page_work_queue_thread); + + work_items[cpu]->num_items = num_xfer_per_thread + + (cpu < (nr_items % total_mt_num)); + } + + cpu = 0; + per_cpu_item_idx = 0; + item_idx = 0; + dst = list_first_entry(dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(src, src2, src_folios, lru) { + work_items[cpu]->item_list[per_cpu_item_idx].to = + page_address(&dst->page); + work_items[cpu]->item_list[per_cpu_item_idx].from = + page_address(&src->page); + work_items[cpu]->item_list[per_cpu_item_idx].chunk_size = + PAGE_SIZE * folio_nr_pages(src); + + VM_WARN_ON(folio_nr_pages(dst) != + folio_nr_pages(src)); + + per_cpu_item_idx++; + item_idx++; + dst = dst2; + dst2 = list_next_entry(dst, lru); + + if (per_cpu_item_idx == work_items[cpu]->num_items) { + queue_work(system_unbound_wq, + (struct work_struct *)work_items[cpu]); + per_cpu_item_idx = 0; + cpu++; + } + } + if (item_idx != nr_items) + pr_warn("%s: only %d out of %d pages are transferred\n", + __func__, item_idx - 1, nr_items); + } + + /* Wait until it finishes */ + for (i = 0; i < total_mt_num; ++i) { + flush_work((struct work_struct *)work_items[i]); + /* retry if any copy fails */ + if (work_items[i]->ret) + err = -EAGAIN; + } + +free_work_items: + for (cpu = 0; cpu < total_mt_num; ++cpu) + kfree(work_items[cpu]); + + return err; +} + +static struct kobject *mt_kobj_ref; +static struct kobj_attribute mt_offloading_attribute = __ATTR(offloading, 0664, + mt_offloading_show, mt_offloading_set); +static struct kobj_attribute mt_threads_attribute = __ATTR(threads, 0664, + mt_threads_show, mt_threads_set); + +static int __init cpu_mt_module_init(void) +{ + int ret = 0; + + mt_kobj_ref = kobject_create_and_add("cpu_mt", kernel_kobj); + if (!mt_kobj_ref) + return -ENOMEM; + + ret = sysfs_create_file(mt_kobj_ref, &mt_offloading_attribute.attr); + if (ret) + goto out_offloading; + + ret = sysfs_create_file(mt_kobj_ref, &mt_threads_attribute.attr); + if (ret) + goto out_threads; + + is_dispatching = 0; + + return 0; + +out_threads: + sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); +out_offloading: + kobject_put(mt_kobj_ref); + return ret; +} + +static void __exit cpu_mt_module_exit(void) +{ + /* Stop the MT offloading to unload the module */ + mutex_lock(&migratecfg_mutex); + if (is_dispatching == 1) { + stop_offloading(); + is_dispatching = 0; + } + mutex_unlock(&migratecfg_mutex); + + sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); + sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); + kobject_put(mt_kobj_ref); +} + +module_init(cpu_mt_module_init); +module_exit(cpu_mt_module_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Zi Yan"); +MODULE_DESCRIPTION("CPU_MT_COPY"); /* CPU Multithreaded Batch Migrator */ diff --git a/mm/migrate.c b/mm/migrate.c index 862a3d1eff60..e74dbc7a4758 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1831,18 +1831,13 @@ static void migrate_folios_batch_move(struct list_head *src_folios, int *nr_retry_pages) { struct folio *folio, *folio2, *dst, *dst2; - int rc, nr_pages = 0, nr_batched_folios = 0; + int rc, nr_pages = 0, total_nr_pages = 0, nr_batched_folios = 0; int old_page_state = 0; struct anon_vma *anon_vma = NULL; int is_thp = 0; LIST_HEAD(err_src); LIST_HEAD(err_dst); - if (mode != MIGRATE_ASYNC) { - *retry += 1; - return; - } - /* * Iterate over the list of locked src/dst folios to copy the metadata */ @@ -1892,8 +1887,10 @@ static void migrate_folios_batch_move(struct list_head *src_folios, old_page_state & PAGE_WAS_MAPPED, anon_vma, true, ret_folios); migrate_folio_undo_dst(dst, true, put_new_folio, private); - } else /* MIGRATEPAGE_SUCCESS */ + } else { /* MIGRATEPAGE_SUCCESS */ + total_nr_pages += nr_pages; nr_batched_folios++; + } dst = dst2; dst2 = list_next_entry(dst, lru); From patchwork Wed Mar 19 19:22:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 510CFC35FFF for ; Wed, 19 Mar 2025 19:24:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50C7528000C; Wed, 19 Mar 2025 15:24:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4BCC5280001; Wed, 19 Mar 2025 15:24:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30ED828000C; Wed, 19 Mar 2025 15:24:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0E6A4280001 for ; Wed, 19 Mar 2025 15:24:16 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 446E9A42FC for ; Wed, 19 Mar 2025 19:24:16 +0000 (UTC) X-FDA: 83239276512.19.4E561D4 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2043.outbound.protection.outlook.com [40.107.95.43]) by imf03.hostedemail.com (Postfix) with ESMTP id 4B2512000F for ; Wed, 19 Mar 2025 19:24:13 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=hca4S8Vb; spf=pass (imf03.hostedemail.com: domain of shivankg@amd.com designates 40.107.95.43 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LpXcxvR43WjPNZga97MpcBVDDzX5U8hqC4yBCUzSERo=; b=B9+NOnK6k5ZBMDfvoJ+PkaJYjladzqfjdnrD6sM8xkMOI0n9IsIvdrWLDg+pSmXZiK9lN+ /HtPKtEY5DkGBjBfgGZ8M2phn2+3uKsr+bzxEFqX1rXPimEgNzp13dgTEgM9dWhdSao/w9 aiUFhv2j85FDY5B8hFv/GIzF1RNlug0= ARC-Authentication-Results: i=2; imf03.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=hca4S8Vb; spf=pass (imf03.hostedemail.com: domain of shivankg@amd.com designates 40.107.95.43 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412253; a=rsa-sha256; cv=pass; b=i0Tky/+ptY7cdDxwnHv17m0P1lYy1mdhkEPd2cOORIfw3SQC1laCYZdZ4NdxjufYGIJ7/P JQnZgjDYKM5/qsOiu2b/UP4zid/KERevP2dFGyBlnz7p1B7pOM65xGcP99pH/ze7GfyoBY 7ojw54mvE1x+HKtAyxWy3Ux9UeY8pFM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TvfQ621e1doNOrcBS1py9X0CzzNjVIY/YO1wRKFxunsk0WuwNc/foKf5vWEM++fHD7cqofmaxJmDdj0BbC8OsBPmWaPhbrKZpoFfCfPeZCCDv33AgCdUGBd2SOKlY/BJPqV4zJEyT/ZfG1SfgsrIDn8eT/yQh3fFdXG70iQL69E5Z6qjXBqzfRQsGmq3xNWwNy1/cQt61LRYqAWoBIRhSRJoPKfHvl0qnn5Y4JmXfmiPzLNk2ipT3BQ7NZMWOsakVW73Hta8hJP/lXty6DdxbDtqcLVmp57f0Qk4Qv5Fbh4bXXdv0nBbxQEe3E7fbq1+XXVx90A5jR01mBvbAdvodQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LpXcxvR43WjPNZga97MpcBVDDzX5U8hqC4yBCUzSERo=; b=zG0dbuRmRnykwBuMMrJrwMX2j+IwEeCKmmmJryqh6zFpVSeqh+LdZpJsZ92+cMaByba5M+LKWIJkqIkEY7fYdPDztcJR22iLvpzajm08cdy04YNbWewNC4fAYvSLNgFuLADZTPIt6hK+9rF9NSWR0ZerpJO8i16JX6H617TuEi8SMi5ePGPcOva/HJGulAFf8ew3rE03oWGOkD6ST915ohumsuWtPyc4zE6gd3wKQOe9VS42ylbo0ykMLPpoMnPCoGjTGPfpqPwuiqQatvs5vuFoY7MIO/qQCC3tSIigDhGbpmPZn3ZiDXVmSS/tbB6/Q8oi2hLGAuEH9605hMShrQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LpXcxvR43WjPNZga97MpcBVDDzX5U8hqC4yBCUzSERo=; b=hca4S8VbCULl5rMm26fdTMYeH9s1LyBi50x0JiCk0gcNHn1UKAbHOViGmVAqcWhNOEJIxCmjHuueJ0z4XKRV3G6coc5eKz8r2/BJ8HgpWc1rgYJLcc1x8bSc5eFNav3AQE8XuNLAjRbY/RXkbcFHpnoOsqw/6SOK2qUpZjpkQDg= Received: from CH2PR14CA0036.namprd14.prod.outlook.com (2603:10b6:610:56::16) by LV2PR12MB5846.namprd12.prod.outlook.com (2603:10b6:408:175::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:24:06 +0000 Received: from CH1PEPF0000AD7A.namprd04.prod.outlook.com (2603:10b6:610:56:cafe::b3) by CH2PR14CA0036.outlook.office365.com (2603:10b6:610:56::16) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:24:06 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD7A.mail.protection.outlook.com (10.167.244.59) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:24:06 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:23:58 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 7/9] dcbm: add dma core batch migrator for batch page offloading Date: Wed, 19 Mar 2025 19:22:10 +0000 Message-ID: <20250319192211.10092-8-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD7A:EE_|LV2PR12MB5846:EE_ X-MS-Office365-Filtering-Correlation-Id: 2d478642-28ce-4af5-b404-08dd671b9dcf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: +xQDoEqiktFI/GctzlhTV4b01Jap78eIsK+aQeK8LsVChK7eoGmEprWSrQa2hCVr9wAYyNSu3UkSsoqfbYby0C96ZHqFDKyCvUgb86UJoeLowlLdbaU+sDtuX4c6nET7I7bYzzYi/qPnip1xpT9xAigd0j+tvg2SLxW9wTKlt2u1RsPpd+ESm+5RVI2yCITfHhxkU9PmSmn1n+LGyE3KwvThHLC8acf9TX5VJmzo0c5ojmKZGNlzg6gX3hlPlAMg3fhlry8SiWWf8NrywCNehRuXn3fcovRpMN+xFbZHWQa5+u/AAlSYfukQAF43tiKQakrfcL9r3fE+VLZTIVct1U0MCWZpt+DW7y+lpdnBKqC6nXk3DBLw13h6NY3FcA2JqevLOIeoX8WnHoZRkz6pzKjAUrfG31QbZ631LrGR8ysRgDZxj6ckTp6/KJVbpytD+n33+b9iSCPA5JpSnmO1YtDErZtj0IKl7j/yJLs0bzBlqf8v+S1jwGKv9y22ZqhqhAetz/lBHXCl1wEmgyLrh28BieVVO9N6bx2UBbTfiK7I+0ma+ERrj3nhS6x6w3si94oUZ25/GVncv1+fwnLU5bgdTvPLL1kuLoQPmeFkPlGglF3QyFxoqIRZq1oKZEVRuslndLBK5ZrToe4k6fqpkQalbgfa8sniTYZlFBOKfcQhf7IJ1pWJPIcdE1JNh4k6xz/AKPfX7iRt70e0XRnMzlB/TjKs1iAK/Qv9KS4QQVpaB/Y0E5eiPQRvkDjk8rQauhnAbSoWHc07UtL1w5XqX8mxSXV1KpbC1UwsYHTVZnXwjuqu154lZpe9boAl+gjYI+OC9T2rkv2X/DNSZHosLNeczQQHnWl5+FuKmlMqChFyHv8VmPO0B1BLB/NYsHbLBgw/s/ayXNMZDNRdp7zVzBVFHDFvQxskAfhD0cbY5AUuNEz+Vt0Fnf74bjGPjL8GTtzU5GvDuewQkI14WJXlaQUCypQ5MghooIGppzXfkTXufhGJdaHrT5SrM1IwhcuP0+HN6bF9zJXGzPDFcjGKLfYJTmNkpDlGSUXU/66dyqTg2pH5+gOs9eicKM3k6y/NslIdvRA/e+R4MG6YrjigV1FlKZloSzvLrO5HBqMKaVWB0CKrOGjBsARg9lRWsdRF38fFdIDO/KbqUJ6rYl+kHJ2ijBz8cwLasCq2FMdtDTXCXazl1nS3lCjtJeKNSjgCCC7TVPxUSRIhmlFpkPBHX6xDRzwCH99mqF0Dyd01gXiSLoEoR52wgNmHHhz2lkIkZb9kb95D/ukF7bDcv21wqQMkfsDzPqz/UnxDiu1v5ELs8DmWjj15V14AsAIAsbyH1Wdj5GUD+qQiFd24+0Ucd3qZuHX5OO2UGN72WZkn0rVUPu8jUCCK6cJwThy6Ks+cz3kwt7HgcvzJKTuP+J2rDys9VYoKxjxI344mgNw/4IOSQ3FDsgRZXoPj4/OulCrJIFAiQIadoIVbc5vw2znKtoHarMQA4aMzQh4FdBflFdU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(376014)(7416014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:24:06.5266 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2d478642-28ce-4af5-b404-08dd671b9dcf X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD7A.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5846 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4B2512000F X-Stat-Signature: 1qcrikerg3eeoodskw1xey9zpdznk47u X-HE-Tag: 1742412253-659538 X-HE-Meta: U2FsdGVkX18oRTw4czFh0SzFxAKRo8woAy8ZuQwPifdhqXZx5PLIjUCrk6q2eMgduZn3bf/0WDA2QaLnGUEclcEUm/BMkt3cBDpdsdR2UpbVo1QCDUZqcrth6rVj1E3G3/aFBpx4eurcXzG+X5qWN2aMdlxrUrgnWcNq75BR/woDGrZHCw4xo5rGo76+DGT/dPYsddKCT4+IpbyKKILWeZOdczPJayWiTfi3Tl1SCIEMEStEctBOe881FsfFMIjm5oR9i0x7r7Ce+chVlOmiPK80SaYuyW2L72Dgn8l0PMIIOxt1AxO/v7hmf3aQVcaTzhawdWMGkQxHLf3XTA5KmHJ6QHWRkZt1uJsMhF81m2uE6KJ25thZbqiutGCSVp19IfNXOAxFToq8c2UrLsuPy71NuXg9GlyhwIIwu83KK9KPn4h9HZLxAHepSAM0ONWiANjclyBLevnvMbl2ECq/iD1+5q8DUNsNfZDZAkGzUCdFE/8HSc4jycRxYZbNKm1Crm6GmfRQQU9lqnHgxroC+b0odYfjG3kKfKfxb7TKaRa0sQL8LC6PrT7/SeanQFtCVnEJ05RQBjR5RxtyQu59VHUO8RzrMYMMmaLYJ+aP4IQTX/MFlJqu6u3T9Nx9U2XipJy6LJSJx1LkCDnMmOhsNmYrXO24u2a+IWPuJMVx/egTJkihNV6OzJUSJJvYkWXnphiTOvSAyidGtNUGleAQ/GYCFVfGXU7P/KngxhtA2KmWbpPe9RSgaGO10/h7rsZg8HGwaO9ubEr1MI5D1UtjwH946PEilUOEwvx/cwUTiCJqrl+XumxyD+pJPN38BPdfRi3gbb3BAihYIyRnCApJflt9u5sxJ5Q6sOWTXwYWcdbVegHkVrswo8Dk+wwOId6xpSg+08A2ElSfDQyYzb5jjsUufodMbk1yf44IOCtKB1x2JYSBWwDZXai3VhL5cpvhNyUFc+CHt57G8t2zXhm 86f2kH/v xDuv+yQFbdgLraMybCcyH0HeNnHEC2Iy9NbTU9D7cl1ZdjIqSFxrfOJL5rEN5YUQEcG4qmgEiDsz4dKK3u+P5wUr86EIHLts5jo6qaMZjfia06ZP2gdpJ31pkiGA13l9IM9eqiU633fsPqqIkrjZz+dTSmaNtSufATlQiIC+O3yCN1nbqwpn/gPXTGmXdbjwFf/0NMNBUp1CHjzrlxmaSiYdYT1EsaNnmN543aoQU6knQl8zq/j1Qk4W3yEYmm9ARZeriOXByv7gEwObXsNz5c8hOrpC1QluK+SEO1+ETBOv4Gygv0OCl39LfuKXkH9sGnPaU/envw/L9bQ9cVNNU8GOEcBnHHiKWS4zEsOcKjN5r61SLT/KWfvLjkhVbFwtXBC48297/OGIXQr9GLKXJhqC3navUJyAi+LPKTqTgOQw/rArBBbU4u9V/3sYsaO9XzM9L2ryylBQHX6/ayq9uJTvWk4Q/LZMnBQXVcxJCbW8KOa9cqFOdei2gjn2XN0elk3u8jIs6dDeQeTQLhnmlogprFH0yhA9F3BiR38eMSbqOI//IUdTTxWz5S/wt+r9+vk7v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The dcbm (DMA core batch migrator) provides a generic interface using DMAEngine for end-to-end testing of the batch page migration offload feature. Enable DCBM offload: echo 1 > /sys/kernel/dcbm/offloading echo NR_DMA_CHAN_TO_USE > /sys/kernel/dcbm/nr_dma_chan Disable DCBM offload: echo 0 > /sys/kernel/dcbm/offloading Signed-off-by: Shivank Garg --- drivers/migoffcopy/Kconfig | 8 + drivers/migoffcopy/Makefile | 1 + drivers/migoffcopy/dcbm/Makefile | 1 + drivers/migoffcopy/dcbm/dcbm.c | 393 +++++++++++++++++++++++++++++++ 4 files changed, 403 insertions(+) create mode 100644 drivers/migoffcopy/dcbm/Makefile create mode 100644 drivers/migoffcopy/dcbm/dcbm.c diff --git a/drivers/migoffcopy/Kconfig b/drivers/migoffcopy/Kconfig index e73698af3e72..c1b2eff7650d 100644 --- a/drivers/migoffcopy/Kconfig +++ b/drivers/migoffcopy/Kconfig @@ -6,4 +6,12 @@ config MTCOPY_CPU Interface MT COPY CPU driver for batch page migration offloading. Say Y if you want to try offloading with MultiThreaded CPU copy APIs. +config DCBM_DMA + bool "DMA Core Batch Migrator" + depends on OFFC_MIGRATION && DMA_ENGINE + default n + help + Interface DMA driver for batch page migration offloading. + Say Y if you want to try offloading with DMAEngine APIs + based driver. diff --git a/drivers/migoffcopy/Makefile b/drivers/migoffcopy/Makefile index 0a3c356d67e6..dedc86ff54c1 100644 --- a/drivers/migoffcopy/Makefile +++ b/drivers/migoffcopy/Makefile @@ -1 +1,2 @@ obj-$(CONFIG_MTCOPY_CPU) += mtcopy/ +obj-$(CONFIG_DCBM_DMA) += dcbm/ diff --git a/drivers/migoffcopy/dcbm/Makefile b/drivers/migoffcopy/dcbm/Makefile new file mode 100644 index 000000000000..56ba47cce0f1 --- /dev/null +++ b/drivers/migoffcopy/dcbm/Makefile @@ -0,0 +1 @@ +obj-$(CONFIG_DCBM_DMA) += dcbm.o diff --git a/drivers/migoffcopy/dcbm/dcbm.c b/drivers/migoffcopy/dcbm/dcbm.c new file mode 100644 index 000000000000..185d8d2502fd --- /dev/null +++ b/drivers/migoffcopy/dcbm/dcbm.c @@ -0,0 +1,393 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * + * DMA batch-offlading interface driver + * + * Copyright (C) 2024 Advanced Micro Devices, Inc. + */ + +/* + * This code exemplifies how to leverage mm layer's migration offload support + * for batch page offloading using DMA Engine APIs. + * Developers can use this template to write interface for custom hardware + * accelerators with specialized capabilities for batch page migration. + * This interface driver is end-to-end working and can be used for testing the + * patch series without special hardware given DMAEngine support is available. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define MAX_DMA_CHANNELS 16 + +static int is_dispatching; +static int nr_dma_chan; + +static int folios_copy_dma(struct list_head *dst_list, struct list_head *src_list, int folios_cnt); +static int folios_copy_dma_parallel(struct list_head *dst_list, struct list_head *src_list, int folios_cnt, int thread_count); +static bool can_migrate_dma(struct folio *dst, struct folio *src); + +static DEFINE_MUTEX(migratecfg_mutex); + +/* DMA Core Batch Migrator */ +struct migrator dmigrator = { + .name = "DCBM\0", + .migrate_offc = folios_copy_dma, + .can_migrate_offc = can_migrate_dma, + .owner = THIS_MODULE, +}; + +static ssize_t offloading_set(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode = kstrtoint(buf, 0, &action); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + /* + * action is 0: User wants to disable DMA offloading. + * action is 1: User wants to enable DMA offloading. + */ + switch (action) { + case 0: + mutex_lock(&migratecfg_mutex); + if (is_dispatching == 1) { + stop_offloading(); + is_dispatching = 0; + } else + pr_debug("migration offloading is already OFF\n"); + mutex_unlock(&migratecfg_mutex); + break; + case 1: + mutex_lock(&migratecfg_mutex); + if (is_dispatching == 0) { + start_offloading(&dmigrator); + is_dispatching = 1; + } else + pr_debug("migration offloading is already ON\n"); + mutex_unlock(&migratecfg_mutex); + break; + default: + pr_debug("input should be zero or one, parsed as %d\n", action); + } + return sizeof(action); +} + +static ssize_t offloading_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", is_dispatching); +} + +static ssize_t nr_dma_chan_set(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int ccode; + int action; + + ccode = kstrtoint(buf, 0, &action); + if (ccode) { + pr_err("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + + if (action < 1) { + pr_err("%s: invalid value, at least 1 channel\n",__func__); + return -EINVAL; + } + if (action >= MAX_DMA_CHANNELS) + action = MAX_DMA_CHANNELS; + + mutex_lock(&migratecfg_mutex); + nr_dma_chan = action; + mutex_unlock(&migratecfg_mutex); + + return sizeof(action); +} + +static ssize_t nr_dma_chan_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%d\n", nr_dma_chan); +} + +static bool can_migrate_dma(struct folio *dst, struct folio *src) +{ + +// printk("folio_size %d\n",folio_size(src)); + if (folio_test_hugetlb(src) || folio_test_hugetlb(dst) || + folio_has_private(src) || folio_has_private(dst) || + (folio_nr_pages(src) != folio_nr_pages(dst))) { + pr_err("can NOT DMA migrate this folio %p\n",src); + return false; + } + return true; +} + +/** + * DMA channel and track its transfers + */ +struct dma_channel_work { + struct dma_chan *chan; + struct completion done; + int active_transfers; + spinlock_t lock; +}; + +/** + * Callback for DMA completion + */ +static void folios_dma_completion_callback(void *param) +{ + struct dma_channel_work *chan_work = param; + + spin_lock(&chan_work->lock); + chan_work->active_transfers--; + if (chan_work->active_transfers == 0) + complete(&chan_work->done); + spin_unlock(&chan_work->lock); +} + +/** + * process dma transfer: preparation part: map, prep_memcpy + */ +static int process_folio_dma_transfer(struct dma_channel_work *chan_work, + struct folio *src, struct folio *dst) +{ + struct dma_chan *chan = chan_work->chan; + struct dma_device *dev = chan->device; + struct device *dma_dev = dmaengine_get_dma_device(chan); + dma_cookie_t cookie; + struct dma_async_tx_descriptor *tx; + enum dma_ctrl_flags flags = DMA_CTRL_ACK | DMA_PREP_INTERRUPT; + dma_addr_t srcdma_handle, dstdma_handle; + size_t data_size = folio_size(src); + + /* Map source and destination pages */ + srcdma_handle = dma_map_page(dma_dev, &src->page, 0, data_size, DMA_TO_DEVICE); + if (dma_mapping_error(dma_dev, srcdma_handle)) { + pr_err("src mapping error\n"); + return -ENOMEM; + } + + dstdma_handle = dma_map_page(dma_dev, &dst->page, 0, data_size, DMA_FROM_DEVICE); + if (dma_mapping_error(dma_dev, dstdma_handle)) { + pr_err("dst mapping error\n"); + dma_unmap_page(dma_dev, srcdma_handle, data_size, DMA_TO_DEVICE); + return -ENOMEM; + } + + /* Prepare DMA descriptor */ + tx = dev->device_prep_dma_memcpy(chan, dstdma_handle, srcdma_handle, + data_size, flags); + if (unlikely(!tx)) { + pr_err("prep_dma_memcpy error\n"); + dma_unmap_page(dma_dev, dstdma_handle, data_size, DMA_FROM_DEVICE); + dma_unmap_page(dma_dev, srcdma_handle, data_size, DMA_TO_DEVICE); + return -EBUSY; + } + + /* Set up completion callback */ + tx->callback = folios_dma_completion_callback; + tx->callback_param = chan_work; + + /* Submit DMA transaction */ + spin_lock(&chan_work->lock); + chan_work->active_transfers++; + spin_unlock(&chan_work->lock); + + cookie = tx->tx_submit(tx); + if (dma_submit_error(cookie)) { + pr_err("dma_submit_error\n"); + spin_lock(&chan_work->lock); + chan_work->active_transfers--; + spin_unlock(&chan_work->lock); + dma_unmap_page(dma_dev, dstdma_handle, data_size, DMA_FROM_DEVICE); + dma_unmap_page(dma_dev, srcdma_handle, data_size, DMA_TO_DEVICE); + return -EINVAL; + } + + return 0; +} + +/** + * Copy folios using DMA in parallel. + * Divide into chunks, submit to DMA channels. + * if error, falls back to CPU + * Note: return 0 for all cases as error is taken care. + * TODO: Add poison recovery support. + */ +int folios_copy_dma_parallel(struct list_head *dst_list, + struct list_head *src_list, + int folios_cnt_total, int thread_count) +{ + struct dma_channel_work *chan_works; + struct dma_chan **channels; + int i, actual_channels = 0; + struct folio *src, *dst; + dma_cap_mask_t mask; + int channel_idx = 0; + int failed = 0; + int ret; + + /* TODO: optimise actual number of channels needed + at what point DMA set-up overheads < mig cost for N folio*/ + thread_count = min(thread_count, folios_cnt_total); + + /* Allocate memory for channels */ + channels = kmalloc_array(thread_count, sizeof(struct dma_chan *), GFP_KERNEL); + if (unlikely(!channels)) { + pr_err("failed to allocate memory for channels\n"); + folios_copy(dst_list, src_list, folios_cnt_total); + return 0; + } + + /* Request DMA channels */ + dma_cap_zero(mask); + dma_cap_set(DMA_MEMCPY, mask); + for (i = 0; i < thread_count; i++) { + channels[i] = dma_request_channel(mask, NULL, NULL); + if (!channels[i]) { + pr_err("could only allocate %d DMA channels\n", i); + break; + } + actual_channels++; + } + + if (unlikely(actual_channels == 0)) { + pr_err("couldn't allocate any DMA channels, falling back to CPU copy\n"); + kfree(channels); + folios_copy(dst_list, src_list, folios_cnt_total); + return 0; + } + + /* Allocate work structures */ + chan_works = kmalloc_array(actual_channels, sizeof(*chan_works), GFP_KERNEL); + if (unlikely(!chan_works)) { + pr_err("failed to allocate memory for work structures\n"); + for (i = 0; i < actual_channels; i++) + dma_release_channel(channels[i]); + kfree(channels); + folios_copy(dst_list, src_list, folios_cnt_total); + return 0; + } + + /* Initialize work structures */ + for (i = 0; i < actual_channels; i++) { + chan_works[i].chan = channels[i]; + init_completion(&chan_works[i].done); + chan_works[i].active_transfers = 0; + spin_lock_init(&chan_works[i].lock); + } + + /* STEP 1: Submit all DMA transfers across all channels */ + dst = list_first_entry(dst_list, struct folio, lru); + list_for_each_entry(src, src_list, lru) { + ret = process_folio_dma_transfer(&chan_works[channel_idx], src, dst); + if (unlikely(ret)) { + /* Fallback to CPU */ + folio_copy(dst, src); + failed++; + } + + channel_idx = (channel_idx + 1) % actual_channels; + + dst = list_next_entry(dst, lru); + } + + /* STEP 2: Issue all pending DMA requests */ + for (i = 0; i < actual_channels; i++) { + dma_async_issue_pending(chan_works[i].chan); + } + + /* STEP 3: Wait for all DMA operations to complete */ + for (i = 0; i < actual_channels; i++) { + wait_for_completion(&chan_works[i].done); + } + + if (failed) + pr_err("processed %d fallback with CPU\n", failed); + + /* Release all resources */ + for (i = 0; i < actual_channels; i++) { + dma_release_channel(channels[i]); + } + + kfree(chan_works); + kfree(channels); + + return 0; +} + +/** + * Similar to folios_copy but use dma. + */ +static int folios_copy_dma(struct list_head *dst_list, + struct list_head *src_list, + int folios_cnt) +{ + return folios_copy_dma_parallel(dst_list, src_list, folios_cnt, nr_dma_chan); +} + +static struct kobject *kobj_ref; +static struct kobj_attribute offloading_attribute = __ATTR(offloading, 0664, + offloading_show, offloading_set); +static struct kobj_attribute nr_dma_chan_attribute = __ATTR(nr_dma_chan, 0664, + nr_dma_chan_show, nr_dma_chan_set); + +static int __init dma_module_init(void) +{ + int ret = 0; + + kobj_ref = kobject_create_and_add("dcbm", kernel_kobj); + if (!kobj_ref) + return -ENOMEM; + + ret = sysfs_create_file(kobj_ref, &offloading_attribute.attr); + if (ret) + goto out; + + ret = sysfs_create_file(kobj_ref, &nr_dma_chan_attribute.attr); + if (ret) + goto out; + + is_dispatching = 0; + nr_dma_chan = 1; + + return 0; +out: + kobject_put(kobj_ref); + return ret; +} + +static void __exit dma_module_exit(void) +{ + /* Stop the DMA offloading to unload the module */ + sysfs_remove_file(kobj_ref, &offloading_attribute.attr); + sysfs_remove_file(kobj_ref, &nr_dma_chan_attribute.attr); + kobject_put(kobj_ref); +} + +module_init(dma_module_init); +module_exit(dma_module_exit); + +/* DMA Core Batch Migrator */ +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Shivank Garg"); +MODULE_DESCRIPTION("DCBM"); From patchwork Wed Mar 19 19:22:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAB6EC35FFF for ; Wed, 19 Mar 2025 19:24:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBEE328000D; Wed, 19 Mar 2025 15:24:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D6E3E280001; Wed, 19 Mar 2025 15:24:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9D9528000D; Wed, 19 Mar 2025 15:24:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9B09A280001 for ; Wed, 19 Mar 2025 15:24:21 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E3E84C0458 for ; Wed, 19 Mar 2025 19:24:21 +0000 (UTC) X-FDA: 83239276722.08.0F4E2E6 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2047.outbound.protection.outlook.com [40.107.223.47]) by imf16.hostedemail.com (Postfix) with ESMTP id 3B0D8180009 for ; Wed, 19 Mar 2025 19:24:19 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QiPbmOhV; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf16.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.47 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZaI3tqmugpKqCcloLD1mp5kwytZpVfANTVAtyTPxnVQ=; b=hbr/lR6VGAjgHp+hDIcwcUiY8O7Gf4oRfr6Mp4EXFEE+aJefWeTMop4FA0KlUL6Mnp+j/w iqgKFyfJ39wC9oR1+ZIEGK1JXbp6v0g3Ktuzi7W0u1y2vWtWp1IZ5UqU/tS0RICYT/V8oH s14x9Or+WTx12RHKr9NUNHQsXh7nmN4= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412259; a=rsa-sha256; cv=pass; b=wNpCRctjmGmJ2qvO2gxn8orIOXBDGmjORieEGIIJsPFqIovI4KvBkqSTws+JHt11MUAi0t SrvcRop6VsV12WSjIPBw/1hTHxO/IN/Ja3pCin4X6+uNc7kuZlRq/I69KFACj3yAZU9eMT ozo1jmZZYW/JYQlYSy5QfBt9Dofp72Q= ARC-Authentication-Results: i=2; imf16.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QiPbmOhV; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf16.hostedemail.com: domain of shivankg@amd.com designates 40.107.223.47 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KOCcuIf3vvCy8AdALaVQQ+6KxS9tLGdGp8OCnhrHzmdlYg7OfJNBlNQoEWhCif8TWC5qTVndN20QXmr1rGvG6h1oFExJsjWBOh7598GFRGoJ6Qr8RV5CuonPq8M6PS+UB5tlm0eB/onjp8el1Zho4NAW5QULVHsc1TzFfx6H1e6V7AvE3DnRrFqfs9UT8RaT8xsDip/qCgXrsR8brDsTYHcJWHzqtxhnuK1xqHRmZ8w/8tFg/aO4VJ+xb6vMMvRn+Iq5p2zgIgm6OSLcu4amN7ikuYcmQErKFI4UdStYWwEIQ10LZ7AfJ+vfK8XuN7ip2Ku4gegOxZj6Lla4/62YPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZaI3tqmugpKqCcloLD1mp5kwytZpVfANTVAtyTPxnVQ=; b=OPQppiH2T3dOBzraecc1tQmj7so62ucYz8tFcQX11ZhyRUObDxoRhmmTZq0IFlKCgw549M9UFaXnJJpfE+rSYiqQVb7x5SvySZmK4YxS13LvFRRhvaTiIoP4TwokpkUW6569MM2lsL0wtCf+w5ZoitDvluAbWq8EUBLcfxRpY4DwJQG+bNJbzdW6//tY2HDCznmXjW51kZ/8RRhVjhmr1NGXyeWstfSrwdS2L+yVpjz8bo5cv/I0fKvnSiJwyylEysdTcknq797RJj4Y+F4ssUlPPdwdHCM7U7oqr/0czLy/ZJR2D+SVlMEsATCMsxmE3tygSI5shNOz9COe/CD4DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZaI3tqmugpKqCcloLD1mp5kwytZpVfANTVAtyTPxnVQ=; b=QiPbmOhVcZjr5zNIHlHKgkC3f2n86ldOQskpll2clBYusFDyk6a7ZsbVH/mFyBUAo7CVb9OnAmuwFsY9mNB3Gyi2qJ4CfSFm4UuEO3VTNUmTknfZ80/7Zd3aGxP3HCoPRFpC/GpUGFCDbczYYAxPJgQJKdD8smiFRgo+CNKMMDA= Received: from CH2PR10CA0013.namprd10.prod.outlook.com (2603:10b6:610:4c::23) by SN7PR12MB8601.namprd12.prod.outlook.com (2603:10b6:806:26e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:24:16 +0000 Received: from CH1PEPF0000AD78.namprd04.prod.outlook.com (2603:10b6:610:4c:cafe::8a) by CH2PR10CA0013.outlook.office365.com (2603:10b6:610:4c::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:24:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD78.mail.protection.outlook.com (10.167.244.56) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:24:14 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:24:06 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 8/9] adjust NR_MAX_BATCHED_MIGRATION for testing Date: Wed, 19 Mar 2025 19:22:11 +0000 Message-ID: <20250319192211.10092-9-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD78:EE_|SN7PR12MB8601:EE_ X-MS-Office365-Filtering-Correlation-Id: c373366d-91f9-4806-12bb-08dd671ba2a0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|376014|1800799024|7416014; X-Microsoft-Antispam-Message-Info: K1fQyQGAe8p6vVUZFiVp36ZxQyIGAjDDmr/rfLUlFlOXld4x7strnOzoIVlOU5YQmN8ZhyOiOK/sMBc9KUKgfuxaTuij3ZGPbK4oOnWIo6qfeNHpVTqZJ0mv+PmCa9BUcRwWX+dwEeMLCT28BZEtA8Hr9z7ZFdJXJLNyR1JivaR2xFKL/ER6Q2VKcMyUBY7aXuy++6MpLVDmwmXNLJ+LIF2eV5Kbuh2LCGR66E/+MBUE3RgVcrxtK1zgxmFiAEDY3v2MWgtY6k3rb9KPugtSwVmzJ74uAWPDG1XQsMSJsn2v3CSSG0MlFESDGDKj7VlJ5Bjo8Fw9K1R2VdO17BT+SrXvjYsVxUZt5MEHYn2KS3ZN1AO048TMZKu4pyinMKwG/P9FHA7wnHEjQKxOhuQdQdUK8rxHIrDZ5P4seDTjVYCjuHU4bhJ2Q9woDgGpVpexKHMb1JGS1rZ45ZfAHJmRrygQrMk1YbipWlxenNhN+NtONkCubv2i3KSZ/WMFO0sAx3ot8q6lF5FA7XaA28mEV3zczVj64y8gwQMmdg4/mVoSfmJ6Zd995L/9kXpNLF3eKZwvMNqr7+HI9Q9XxP64bELBG5ip+li1CUA4op7FItT7ECIOH+mZtAMrn+K/Sqj8JqzAgVyKbjghspptzYgi+eKUQjQF3F8/73KjMHNtthpQrW5VYCi7ImEzYL2qiUr64haV4p4ILZym/yxYUaftqOAMNtz/7V7YxKd5qaNSIYmLkJIwLSzXTAPD/vZdbBW2ETH6P1+r57ITCGZAJGK8bXGcHfYw0thYmpSS4ea05Qa/kol7zg6xPtJcFRy2NFbojCyDjarbuYklaXR7BzqIr8dEnYSwPe7QsVvAT/KaklZJvf38vS4Q8Ss/EvMv6B/UKW2NPN2dXSlCwu86diYoZp2sT2nRqvStow8B+6GGUPTKppf+IVJg2TT9BZnAXosDQHhc2/RVGHx0YDQyS286O4mFSUdIhJzqvxhmWEcyFh7MSpF2EUmBGRwmCmn+4nGb3hQP5d4zC88WONbl4K08N3eHvF2cK9zWN+4PzsUMZ9trVxNb7Y7bBfm2i1GFxXx4X1fN6XgkapGqGpFaKKuPZ6c5VHX73iUiNsQzMvHcZ9LwwS9XdUzHrk1zrRc3pnRg5siBJTq3HLBCxJma5PFsOBqiqGa3yPvgoYz1+wvxGxrX68evrrdVWrk2Cy2Tno5Cqb4mh73P2qnt8y+FOy14ucxSekyEP0BMTIHQKmm/9Azec4RCiOkCQfRyNLbwipJZbG4tth/pgJTBvoVGMx0oGPqWuLHNZt2TDP5djPBpVbG3codCW+V8ObOZklT33o9odKAujJg/lGprDwbErcJrQZ47w8squQvCyrAH9bkk4ujpKVvnIj/HLngYBJr1FkHmTD4WkQtYDSMKAw6ysNH+nXebksZ/PpN19uL4HvEsdZtf7cjqps6Fy3tvI6k2IU4yFBLEnqYefky7A6E7AR0o42EYnFa151NsujRXp59Th9M= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(376014)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:24:14.5562 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: c373366d-91f9-4806-12bb-08dd671ba2a0 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD78.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8601 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3B0D8180009 X-Stat-Signature: kytwtz7xikzhoshidsqzrj5cwd9x3n6x X-HE-Tag: 1742412259-297538 X-HE-Meta: U2FsdGVkX19CAHzaXDTiPA5Z3Yr22/alHuZS5FziFv327Ig8bR5qRHHAbdXHOytWm8oSZkUVY4i42zA5hCmclKazImJAL27rbSCvEOAld47SeBa4VW4Nkd7NM2ZhRocWMCUrLLT8vbVzaSKN5riT8H2vGQQ/Pe/zuTseXhUfmSKiy04ejLa9rYDwGXImqKK+Z2zBzgOja1XXRWTrYbIxwAM1vCE9i2YUWItUAq7lN3xL3ZvnBC55n0ipKceBqgn74gjIEvLyQrOeIcIQGQGdGdrsV1oerZXSYl5cA9n3qVLgfbPnbXACq+MfQYMOn0VqDIUfXVajcTNYG8tvAz6LDe672YpY1IXAC/wAaYiCuYOi6VTm95w4P5x5MAc1rBcEXaVlh8RoKcKeYc3AhaJ7dDxJhzL/A4G1htFDj0vicmBuizdMTa5NGwgmvmh0QFOhNza0/W3TShYjHuu8lmrJSmhjdbhqx7V07/8sUZ8iK16H/WcAmk5Xwr7ZSPe6jz9ZjeZGWd43e8mv79ujuPvsSKktqJGGfr7bK314YtKRyT4bWmKTwrfDRao8lfj+XJfgCaiQxQuucRvKEkY+MPvJrcmOAdCpIeyBI8+vbXsqXJOFN33dBQ75/Bgg12jfwoMZV19qNKD1kziIw+KnnOykzUSKgRLwHrR7rbzETzz7AUZ7jLUZd1KNXgm4EGHhCLIwimcHPcNFYWlPuPN/gebT0/P008vIMl/u1ZBmGzE374mDAVp85IgS8O1ggWX7T0hD9LPOVQhps31HFRhMwrA1JbX9/TBsNjVkq7SX2dth6lKxQThro4auUSkJPqp0Ygf7MTRK6XujlX8I8+PywK+f4jQdW9KNvlYQvdYeBuJDq8deCjriDgX5G2fS6HBhIebrEfohEzRhapfDknQ+QrIMbjjya2/rBNdxuPQqFe9zsx3P/G4A1DYG1KPQteRXiT/PstG43Hlh0G9sW0HmnHY F2C+0NsC ljmh3aFpVLIQW2GceWbcuCpdhd5sYfOam4oNZ1aVgmIOVwQMGbXq0lRcQNQwSAuFpkOcpzohcNSK7M9gBWIYLBcv6erAevgFXulvm/7i2sttgEQw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan change NR_MAX_BATCHED_MIGRATION to HPAGE_PUD_NR to allow batching THP copies. These are for testing purpose only. Signed-off-by: Zi Yan Signed-off-by: Shivank Garg --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index e74dbc7a4758..f9eea16a975d 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1653,7 +1653,7 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f } #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#define NR_MAX_BATCHED_MIGRATION HPAGE_PUD_NR #else #define NR_MAX_BATCHED_MIGRATION 512 #endif From patchwork Wed Mar 19 19:22:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shivank Garg X-Patchwork-Id: 14023043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA824C35FFA for ; Wed, 19 Mar 2025 19:24:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F009828000E; Wed, 19 Mar 2025 15:24:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ED78C280001; Wed, 19 Mar 2025 15:24:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D535528000E; Wed, 19 Mar 2025 15:24:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B836C280001 for ; Wed, 19 Mar 2025 15:24:31 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0126D14072A for ; Wed, 19 Mar 2025 19:24:31 +0000 (UTC) X-FDA: 83239277184.26.32EB004 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2077.outbound.protection.outlook.com [40.107.94.77]) by imf07.hostedemail.com (Postfix) with ESMTP id 10A944000F for ; Wed, 19 Mar 2025 19:24:28 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=pSL+Qafx; spf=pass (imf07.hostedemail.com: domain of shivankg@amd.com designates 40.107.94.77 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c8P+xOLNadTEZ5L2TBrKUW55HXRM6KmjsOy9cLgaKdE=; b=pRpsHs36ca8wkk7cWx7t9crZ8lKUEqM6kVWJoyDrNnFrR6Z0K/DpkSykEJmei7bN9IXu/B 07KnGFed0gP5Y9XaOQwOYahUhKiotzbGPrFMqNLv68gEhT/kLB/TJICJUGcQJGZs4iVM1N 8NBucNqhLXyfMrN3Dq1m5+aq3URp9d0= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=pSL+Qafx; spf=pass (imf07.hostedemail.com: domain of shivankg@amd.com designates 40.107.94.77 as permitted sender) smtp.mailfrom=shivankg@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412269; a=rsa-sha256; cv=pass; b=5neLQkrsKFqKSOM/fi6QHSuAw75+BzUWcx6zvDbmbEEMAEv+Xk/GF0ayZfsKdIGpOHNRix OowcLzLkgaQz3PelBtvykwX2061QhmFuLU2w8j1oBzsTpjxB0HfYXOkeMVv7MYmylsQ3aJ ODRwhcZMYus5f9tZMG8T6R/4h9GXqGc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nQriWq3ZXlUSxGN5mClOuyTuKJs+tFeNJP1FAOXrL9IIdT8K7Ll/vgxUqNq8K+T4NtNZO7EFFvTuZNQspcFg71iw4q6mu/IYDwrr/65BLHTOVHI6dchw2UxP0QTIzqHOCgVoL654h8D0zAcSXBy2G3b5y0HtajDUEChiZY/qsUVogpf3wlevtD3xMcYD8cNXGBnYNfH5PZbWN6wnYexsa02b7HJe3tm8uotzK8uRNF/uLLxvNl5HkryL2janmh+6TlxmfkqI0YTB0N20dvQPtFTaStT18oMWC0QNYTz7+BFyuMkVkF8fIHWDMGj/nCHvdngjr2buk2OxuSX0/xdCyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=c8P+xOLNadTEZ5L2TBrKUW55HXRM6KmjsOy9cLgaKdE=; b=IPYtHdfe3hSXcZ6uJU7Ga5DgIB8zFXak0nffgG60ApzPCN/P17O4LkwjOzIBYlj5wfePX+imc0GxgGLx4oSmGWecIbKs8S5esI3tjePRQey1hR+Sl2Y6/JGrHinMbG7KJlzPiPqB1E1xbkFuiV1e+9M7eejnDICYwwkCCrynzzK65rJE3cDjmMPlfIRYO5PUFq0EGn6K+8Knl1OIzyv0/3pS0aFCzycC2yqqdyRxvNwmY5j+HUX01EQlqmmCAlPDXLKYTMQdXgnjWcH3PxhiMnUNk1+m1Kt7eeQeh2ljxji68h5XaIX0PLZOmv7fGGX+CJoA1UqVex963lbMLI3/Tw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c8P+xOLNadTEZ5L2TBrKUW55HXRM6KmjsOy9cLgaKdE=; b=pSL+QafxoE77gAYvKVjOIo28l49ybnj3kTIfqekB0W6EODp2JmbGnDdPW5pOPjTWnEfviVSh1h4psoSGTSU2NZEydTea8nbeBgHPz+nKsnCVhkzPMESunmXwt3xu9r46BKEWwxh+iX3pbbf9n17+W04F/yP+jSoNxdht4Ke3Za0= Received: from CH2PR10CA0015.namprd10.prod.outlook.com (2603:10b6:610:4c::25) by CH3PR12MB8331.namprd12.prod.outlook.com (2603:10b6:610:12f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:24:25 +0000 Received: from CH1PEPF0000AD78.namprd04.prod.outlook.com (2603:10b6:610:4c:cafe::f9) by CH2PR10CA0015.outlook.office365.com (2603:10b6:610:4c::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:24:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CH1PEPF0000AD78.mail.protection.outlook.com (10.167.244.56) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:24:25 +0000 Received: from kaveri.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:24:14 -0500 From: Shivank Garg To: , , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH RFC V2 9/9] mtcopy: spread threads across die for testing Date: Wed, 19 Mar 2025 19:22:12 +0000 Message-ID: <20250319192211.10092-10-shivankg@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319192211.10092-1-shivankg@amd.com> References: <20250319192211.10092-1-shivankg@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000AD78:EE_|CH3PR12MB8331:EE_ X-MS-Office365-Filtering-Correlation-Id: 79fb7930-f6ad-44f3-293e-08dd671ba8ee X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|376014|36860700013; X-Microsoft-Antispam-Message-Info: V6niP53U8rM7xPccJMkBaqVBh8LIChEV0FEUf1bYsp+egNW7sxbH7/GapdCyTmBgzqxOS210W+IYR5LkQHSp+EDHDqae/WD4MpZ+4eZOY08VlA3R1GPOOc2+bWUmozKS2KAjflFTRvW8D532aTEH4OYysjWDd5t8429QKl5XPRZUlbPTkWWRWwnvzxs+GRDgrBZAmB+hp73EDEDrVhNYajX7b3FGnWYHLfLF/kW1AftFaDp88E7L+q7mrdkj7u4Uw/mc/aKBBQGiEPlI53CUNdV4VmX5jRttjDvR7YqztNtF3vq48wG5n6lUXT7xO2eK/ddIhPnn8L+SxA7oqeJPHfcG7Sl3e3TzMEpLfe1xjZE2K9rL6KCeiEYGNfI6OGI5WfjeVwgL/K8eSvUkuVbU3qHk6TERQOX0ZKda/AmzRj5v0N63gAdf4IgaJJEBwvqeZ3IbL+yFrlfPd66oHf8b3T0Jv3KDpBdIL+f1AtZcohLHRVAKOi0Ud0orsWIOY8XHOjKlnQHRG9E5ZUa8R06MHpF163Zon/LoJvJZAb2euBxVJihiOyh6sKu381Z3KMR8sQMql7AT2IPzyDqxAizoO45q64g0avJzXu+maDHIZ+M42RZ3HlpuvyVeQ7kwsGU06wLsqp93LyoDxJgT153pM2LjUJGBGlXk30okcNTkeRB0glo4SBkgG6nbcLYyO9hUlF3e4/SF2La+4tNm64LWElBCBgPdbSpY+gMDyfRXwuXi5l6BkHQ577MVQUplVfX9tvt4XEkEQ8E16y2SX/bzdDW+eq48mZ95TioyLvfaiE+ay8KHq6fGynhjm05Odub9o7Z/gAJrz4f0vbubfVQvyVkujrSijNQCFunj53uaR2bUiaMeB/Q2wkOiRidUtx2LKmKAKeA87ljO7pLgpz/JKMSAJ4Urqw8DAeRdSI5eTH/kQb5d/DUeyhmOdvEhoVp5TWNGl5cmpLHcPjRCK4VhuJfqm3Yf6URIrYiQHnAbDmLGzE7nAHSOnVEkc4GkbWeKgfDLZ3mSmSTXMlf17jMpzbzpKwkknsoVuX/5c4g0BFLUfQI4LPqrsznXqWCxcJoJPSgPAMvqv0W74dLHvi36QI5Q6EZJgv77cCH1W0axJrrzQs8jtK0RYhXZUe3ao6EKaGHJ0L9E2IF2fxY7tr+CpMWp+xNcTOf7DFbrvHBojm9aE9ZzxEUqnbUtg06cc6xYG8bdsi3FMR1EIuSLBdHKflYUMKCZf8hv68mKk/1QV7rj07AAGJWP1O0O9GFRjQnPfeLzbDL87yJbRUN94mCE5xGHODHwmC8NwSpnZq69yEd7lsOE5q3wPIXPDwHLPdtz+do/UHAcWAM1B+ZBQ4M2iSUvBJm99esk6Bs1Cg2aFWAECwexaBreZs6M+EqIZ22K6hwfGKAiR/l0661YiEKx0tnXh+NpFOE3z6OZ8iIc0lQm6QnazT8PnLjsoGSJkK3gdPhToiTichdLD2Wyq4mIsXE899/Z0t3rgnbAlG7uoXs= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:24:25.1655 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 79fb7930-f6ad-44f3-293e-08dd671ba8ee X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000AD78.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8331 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 10A944000F X-Stat-Signature: ghii56rtfk4xj9qt4ry3bzmshrsoyot7 X-HE-Tag: 1742412268-16050 X-HE-Meta: U2FsdGVkX19dVrer1CLFpeVI49QvA0Cn4IiJ2DoN2GakESGwsqVUAXF06CelbmLqrKUTnN9xiPpzHovj3ZRJ2S0Ppq1DqwgjNuLoYnAYxRnyxPG80aO9baT2RC7oqAOy8YYMMzbCBex0Tsrdt+XQS0WclShBL6CSHPKUNNn1X0jkhIALAk1yoKB+k1z+buUTu6My5iLlqr5S7c7shzm6Pc6AMLFauUzjMQsfbXTbRWNdE5fbarqlt+TzHlM2MWaafwMCc0MBPMwaaLY8o9htdGWDR9r/+xHfPLVnX+bqiBKoISSbAno3nPNGh0x9SfUjPvB6hiWmyqkovYs31Zct8y0HD5KlRWBfBwcK4DpEViZ1pF+VTOqlZghx2lWNJwCgpE3BGWL46QZsLMkBmhk9fgYGLCtWDGSUuSnCLFKn3TA7ErC6e8JpFaC3PO0eyOOz8SBXBwyTOOjnL3gFBOgTpLDkO1pUmdG9stDg7KUWmBZRF5GPE/x0psjUYv6yZ796L+Bp7rDh3nMT8qNIyrOu784kGwuJLiOfloIiLRJxJ8KrzvwKqF9kA3lpFvQ9nV6at5H4tUjWw28aXliXFPNNJkVokwIIBmKP7Tjtq727lYIMBK84+iuL1HXkj7y3B+dxvTvwoOnOQ7rsbeKHMnI1tWjB3e4HkPrPSG8fa4EnLM66SI4uiN6Zk+JWpSd2Ybo7L9svLOCy4rCnsw4NIHSB2lPent/ErtnthCYrfGGA36XtLsQF74nSMfoyKTAlgJsjqx6KSuTyjT59F9zBfsFCFD/0qSS+ZQm3kgi68zJKEmVVg0TBnf7UJKhOSprVlDUfKit+F0W33XmTb0QR9TdHETw/hGlZkHJKlrUK49S26qvGt4DlfTqCnv6hza+8KLzymWaErSwuYt0Qqkue15kvVayuxKSgc8cov30Un067WHDoPJ2FaG78R8Es0tJv61imHwCYVgOXtcp2Zzmsm0a HRdOZoWp 6vx0Exh/8f6/Z5HR/X2iWVAtvSXosfpSIhawcmcnVS3Eyj7gWbCTCfb2LJDK+JbVUmttnraCXsYtm5iQ1mtA8I9dGEBy4Faay3VVp0L332rouIVo6pBLI4IJNvOtPiwl17mi+TTJwr1rpICqr172hzc42TeifEk3RqMkhRfofWHbaoP5CbttGFPmkgWnvc107KhOEpNG9HPpl2roagiZIRNYZyNY3h7Jz6qHhB5lw+MwP8lJJ4jUisM3dLazeqr0Lqyo4hN8Evd0BKSDHRbruq+f8GI0CWQAcJnVHcRQVQTG2y3YsTAEXifkg/YpbqAG6p03firRmYZ1oJGTygBSJO6v2uL7kVUnknLMXBAAqcxGNdWJ4j4Nz75dTmJGP61hKSPNImZO7ATx8k+lm0Q8v9jev2+k2+noU2DUkLqNMMrPpyeLBbIc2YP+m0LKtO20M1LvpTX1+qv68uypzMwAkfwzlwSdGTQSSbbBZq9VDhohzHFUnZoexBfeXnkZxwMOx4ZwGQns7xRVIblnzq0J70meXTHin9A1YUdp2RUcuCpo5UK4T/xSb8RNCRbiDJ28JjTE0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Select CPUs using sysfs For testing purpose only. Signed-off-by: Shivank Garg --- drivers/migoffcopy/mtcopy/copy_pages.c | 77 +++++++++++++++++++++++++- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/drivers/migoffcopy/mtcopy/copy_pages.c b/drivers/migoffcopy/mtcopy/copy_pages.c index 4c9c7d90c9fd..5178e6846890 100644 --- a/drivers/migoffcopy/mtcopy/copy_pages.c +++ b/drivers/migoffcopy/mtcopy/copy_pages.c @@ -15,11 +15,37 @@ #include #include -#define MAX_NUM_COPY_THREADS 64 +#define MAX_NUM_COPY_THREADS 32 unsigned int limit_mt_num = 4; static int is_dispatching; +static int cpuselect = 0; + +// spread across die +static const int cpu_id_list_0[] = + {0, 8, 16, 24, + 32, 40, 48, 56, + 64, 72, 80, 88, + 96, 104, 112, 120, + 128, 136, 144, 152, + 160, 168, 176, 184, + 192, 200, 208, 216, + 224, 232, 240, 248}; + +// don't spread, fill the die +static const int cpu_id_list_1[] = + {0, 1, 2, 3, + 4, 5, 6, 7, + 8, 9, 10, 11, + 12, 13, 14, 15, + 16, 17, 18, 19, + 20, 21, 22, 23, + 24, 25, 26, 27, + 28, 29, 30, 31}; + +int cpu_id_list[32] = {0}; + static int copy_page_lists_mt(struct list_head *dst_folios, struct list_head *src_folios, int nr_items); static bool can_migrate_mt(struct folio *dst, struct folio *src); @@ -143,6 +169,40 @@ static ssize_t mt_threads_show(struct kobject *kobj, return sysfs_emit(buf, "%u\n", limit_mt_num); } +static ssize_t mt_cpuselect_set(struct kobject *kobj, struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int ccode; + unsigned int cpuconfig; + + ccode = kstrtouint(buf, 0, &cpuconfig); + if (ccode) { + pr_debug("(%s:) error parsing input %s\n", __func__, buf); + return ccode; + } + mutex_lock(&migratecfg_mutex); + cpuselect = cpuconfig; + switch (cpuselect) { + case 1: + memcpy(cpu_id_list, cpu_id_list_1, MAX_NUM_COPY_THREADS*sizeof(int)); + break; + default: + memcpy(cpu_id_list, cpu_id_list_0, MAX_NUM_COPY_THREADS*sizeof(int)); + break; + } + + mutex_unlock(&migratecfg_mutex); + + return count; +} + + +static ssize_t mt_cpuselect_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + return sysfs_emit(buf, "%u\n", cpuselect); +} + static bool can_migrate_mt(struct folio *dst, struct folio *src) { return true; @@ -218,7 +278,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, } for (cpu = 0; cpu < total_mt_num; ++cpu) - queue_work(system_unbound_wq, + queue_work_on(cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); } else { int num_xfer_per_thread = nr_items / total_mt_num; @@ -255,7 +315,7 @@ int copy_page_lists_mt(struct list_head *dst_folios, dst2 = list_next_entry(dst, lru); if (per_cpu_item_idx == work_items[cpu]->num_items) { - queue_work(system_unbound_wq, + queue_work_on(cpu_id_list[cpu], system_unbound_wq, (struct work_struct *)work_items[cpu]); per_cpu_item_idx = 0; cpu++; @@ -286,6 +346,8 @@ static struct kobj_attribute mt_offloading_attribute = __ATTR(offloading, 0664, mt_offloading_show, mt_offloading_set); static struct kobj_attribute mt_threads_attribute = __ATTR(threads, 0664, mt_threads_show, mt_threads_set); +static struct kobj_attribute mt_cpuselect_attribute = __ATTR(cpuselect, 0664, + mt_cpuselect_show, mt_cpuselect_set); static int __init cpu_mt_module_init(void) { @@ -303,10 +365,18 @@ static int __init cpu_mt_module_init(void) if (ret) goto out_threads; + ret = sysfs_create_file(mt_kobj_ref, &mt_cpuselect_attribute.attr); + if (ret) + goto out_cpuselect; + + memcpy(cpu_id_list, cpu_id_list_0, MAX_NUM_COPY_THREADS*sizeof(int)); + is_dispatching = 0; return 0; +out_cpuselect: + sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); out_threads: sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); out_offloading: @@ -324,6 +394,7 @@ static void __exit cpu_mt_module_exit(void) } mutex_unlock(&migratecfg_mutex); + sysfs_remove_file(mt_kobj_ref, &mt_cpuselect_attribute.attr); sysfs_remove_file(mt_kobj_ref, &mt_threads_attribute.attr); sysfs_remove_file(mt_kobj_ref, &mt_offloading_attribute.attr); kobject_put(mt_kobj_ref);