From patchwork Wed Mar 19 19:30:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE459C35FFB for ; Wed, 19 Mar 2025 19:31:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 296E7280003; Wed, 19 Mar 2025 15:31:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 245BF280001; Wed, 19 Mar 2025 15:31:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 071C8280003; Wed, 19 Mar 2025 15:31:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D8274280001 for ; Wed, 19 Mar 2025 15:31:21 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D691DC036E for ; Wed, 19 Mar 2025 19:31:22 +0000 (UTC) X-FDA: 83239294404.14.3153ADE Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2084.outbound.protection.outlook.com [40.107.244.84]) by imf14.hostedemail.com (Postfix) with ESMTP id E0763100015 for ; Wed, 19 Mar 2025 19:31:19 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=NHUsEDeZ; spf=pass (imf14.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.244.84 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412680; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n1YlgkAYpPVRIE0Exq0WZ36EVvjUeFBDZwrxw1q80i8=; b=fByfScT4adppYj8drUZxak5v2CKlidpWzS6+Z5ewxq+wmI8/6fNQSs8WUIe60L9gfaVq5A y4Z/zqS15f1EEJTli81oqdghF0eVkyCleHDM4/oNIe2St7oXre5oAiLZi4yA7B5OJJOJGe 1JjFfIrFjO2L3W9igtabflwgmY9DsPA= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=NHUsEDeZ; spf=pass (imf14.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.244.84 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412680; a=rsa-sha256; cv=pass; b=ILr53RTM84C5IRGpfbTnaGgxfJ7hmgsy8VYACTiaiZQ1RzxIE0dP1dwUmdiWIYf++CZVQC OCz4Kpid//mBg38fnzzElJPrl2Tx/De34glRaWesc7dRGgCwwRsTbuUZWeTl/5pmQFHb4c lJBr9/BbWYqJVBLcu5LvDxvSwWfpSig= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xk1QqGh6HBCRoIKnxFcZ2n//DJ1VX8R/RLfqy9TQGGdFkDbXRw/+sKX/VlQmhz/rrPlkXFxoyWh6wBfHRYAPKWPTrmHd9wC0GebVv6helMsYqo2rOYs4ldGG6N3hsm56XSo+O9tptPQsB7LgkKAEvdcPO/jIbC4Kp5dqMDDLwiy6aDmXMH/5Axcl3tQaCaaQ2Iw54x1mfQKsNIwzCssf29hU2hr52n9wcOQOe1ZwwT9sTMMrgOG+lekLfCeIqYbB4Dwe1+qtabYpufZiCFbtH04XUQv79GhHbBWOCIH8NYVFeEYw6ILk7erpfU0wKHC/N0tA7qwB+GUOUJOyhUMtIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=n1YlgkAYpPVRIE0Exq0WZ36EVvjUeFBDZwrxw1q80i8=; b=wRirfef74xUPsuzFwZM5FpQyP0iCAv48ehirJYkgPSZXPBgBKKY7eJGTvDGaAuRnITiQpTPw44BCYkVSoijeJDJA5wkuCdz2ZUrWZoOk1iXRIw7hcEfbwZJt4S9vTJOQEruYk8OTd+qhRyQXXuLhBCDKMEvmpkW84rPJmMhKCADN5rXb4oyR1DXEpILyAKwONkfrPm0EMTKNmMNh2+IWNNJjf0moWKROfNLsgAkmG+LBx5XFpLRIDn/BHopTxnSLTVxuyyhsKJhLduTaiBnXG2x+sjDda6uVFb+0c6ML+j8iZx9D+C5FLhIG3xLQ9csNTcKrbYZB4bGs8/G6Hv/jaw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n1YlgkAYpPVRIE0Exq0WZ36EVvjUeFBDZwrxw1q80i8=; b=NHUsEDeZnSYaavYG0VPsefHnUOziaVjhbm8JatK9VWDk0dT9AKULm6iR8rVOuBQSRH/ZKRafcKqJ3wRP08euY47XsJ2UmLPoOfH+GsD8WGi/w580nHMQCNTW1y9LpB5JKhshK+JlMNz5AX1gNQCfZbW+ZckAKI4ekiMlzOLRYNM= Received: from SA1P222CA0187.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:3c4::14) by PH7PR12MB7258.namprd12.prod.outlook.com (2603:10b6:510:206::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:13 +0000 Received: from SN1PEPF00026369.namprd02.prod.outlook.com (2603:10b6:806:3c4:cafe::c5) by SA1P222CA0187.outlook.office365.com (2603:10b6:806:3c4::14) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026369.mail.protection.outlook.com (10.167.241.134) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:13 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:02 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 01/13] mm: Add kmmscand kernel daemon Date: Wed, 19 Mar 2025 19:30:16 +0000 Message-ID: <20250319193028.29514-2-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026369:EE_|PH7PR12MB7258:EE_ X-MS-Office365-Filtering-Correlation-Id: 8966c4a0-bc7f-44a7-2b6a-08dd671c9c2e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|376014|1800799024|7416014; X-Microsoft-Antispam-Message-Info: uOsbzxn9Ra/71b3QdilmFLcwsYqgpUH8zMdp5Ax2j3KUkZTt1blWU399UecFQwEHWrSbSx1RCwQ1lKT3aRgkytNVHhitIxH0J62qi3Oyk8pBqRc0364V5YagqGlzJu4x7qNo16eBdtWcm22bD+3AWUoXCrpv6yMvCXaSVomIFS380EWlUJhDc7mRi7HL4i+b2+tkkIBi72i1Nphw5FqKhPAEwf8Rc+JvMpx0yBZ9Ema3en6/OhVeDqJSwATPY39rxxNx4AZu6+i4mKRMGUX9snlUL7rMrR5bUf5OODCGrdmEvbJ+fSHDw98bPapVLq2Y/Dx+3j4QjJMx2Pzi0WzxOwStTuT/iqlX6dQPZmSUGH/ukniK0yNXF8jWNh2C7uWlVDzeRP7QTaA/pZheFcFhwWwA97ioTaV6OzeEZGDs/tgrT4JBKa2DmPK31J5dYccFy1e8i9xE81ln6ROZ5fdOPOtKEbD+q4QPpDIo2eqnHQ4pKEThhtbv2DLi63iCg+7Ahjwzdv5t1cWGG3/u65aCyICGHzcCeE/ulAR2cGcFRDnQsUC+hHO7tuXXgJ6+m9SdcA3ed9TtwJU9GtZZUWwTNMXeF3lBnZhRpx755OR8nFEZbTsB6ymA+ySJzXrUf3Fq9Qi14Eu9rl8BGj2oD/HSWZ2r40OXANzApJnmz5SXNUpUFyOp9MDmp24ROyFaSfp+/jNdoYYdY01RilVwUokQhG/DuKdqb0QpFY/TfIo97Hx+H1jw2kGolN9VyfDQA+X5z81lobaHL4gv3aYvsiI8xWuPGmcMbIsIWNSapSc2AwLWEQ4WM2tM+VlbZ3f2bnzuhZpUz2hNllNCWF1tKncjMWZPDQOCrYlRpRfx4Jyiemye+4AyawaMjplZgXfEf5qrMqfN08Y8ZXXD3vC6sgYICsJ92HcyHQuObrWtyLfL4iYWB+Q85vj3C/+rrEvpd8z81TM45KDg5UZo506zDgAmNFcaUQnDktniFWdKWl8haOpNG0kVPJStTU90w8c3iFAzznfMx0I072IuGpw/5gJmiHhPBz25EZjXC1Eg4hommyAnoZHjOrgtiLTazkpD+cgNahu8sufAPoJrmMThPKF27UghP4Ltc0C8jdqg2ixyoULboNA92oievqC2/SxKuU7SbA5ZH8hv4xUb45f1ZQZVfhMwS+OQWIL2f7DAi3jpVTNeWSp9qvsV0m4Hsceb+dqiXkUEZzA80xmKBZUaKwCb+Zav1E9aemWnUf6kOSgYNx4xdQGohjJ0Xir7Ajva61uScEoIIrqGIs0KyzAmioFvHmLILJpLg8er3PzgtGKU9DTsEme197NdthIoMGLFDVEvw5do4Ahp/G0IMUrwc/ZTnH3PuwfLkW1FXRUwRqprU7lvm75q3nUpFMqaIGBBk5ZZxpnFzpi8CQtVSm/Lms7XgkBBvjyDtoGWR1bE6yzdmcGMZuth5X5k3w8oBAsEotNcTjwxc1KY0hPc0nseeDCuJjLwiD2HTR8z1FeUhtpnR7A= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(376014)(1800799024)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:13.1620 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8966c4a0-bc7f-44a7-2b6a-08dd671c9c2e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026369.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7258 X-Stat-Signature: n7ajjhcuxbgnazyib54d85ws3ogu1mpc X-Rspamd-Queue-Id: E0763100015 X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1742412679-814603 X-HE-Meta: U2FsdGVkX1/JI/F8hbjg8jlKgHdToG+6Yy1TlV3UzJGUoli3MoU4g4Yr7usWX7a6n8P5XqE5Bud/fynGfS+0NIlSNkOii3DPJzOpZ42xmtaDyHiL/Qglkj4/PETz35p/JQyI/FgVP0XrFHpMjlDn/znPAWv0Yn72mg2kwRG3X/J+dte25XDYRmBLcKEoj3+YJ87Qwacq/ISc4TfPrkD8HtajLdXUWei+dipEshrcgoKHxSsKlX/GXq4RTkzDG9I1NmYdyQ3WxPQB3v4wSEQpHRg0/rccSG28Yn2Y/VKzlOpFQ13dwt2cpfwTawPb8UbgtTnlsSDUs37QrXeTJMnY7s+y6vGPSyZhVQCRZYI50ApDHb4vwHsb61zysbVXvFNh52kOBSqcaaHHQ6Ph0zZ2l9BM+VH3J0RAFaeiLy9DbbwSXTvVqOKrl4TendUzmSM0U/CXJCxVZkk/C6UQ+qmejdYDUK0bcVVSGa9CLZoBaC+2K1gzpDo9M0v5TgK3pxNCp7V7OBrFqZrIhjMBMK6AxIFttUUZ0mm1nK+CIhogrvIwT0zHbswsFeLYuCItL1SoeayVmSCJCSaezPYfJ3ZWsv/vU3UgquQFYmGxXYCBQIiD3NrqjQWXlJRuRbEg4jow79iqjSjZSBHu6xJXYpnHw1oW3LJioM3EHhe9V2gB1BZkWMly1Z5EZohxLy5fN8QoZruVg9rX26rrvwdqtmyCpBUv6ZHL9I/xn4vwpAaxQcESui6TCbzIbVjhdErAgT2eHhqQxGTJZns9tk5nphmkhzVX7H7ONcaUaENA9JXQm/RDNbYa5Ow1R8dSg2rwqffHkNnqIN2utErRA6erIlUWaQ2tVOnkEqNpMTlzm2klY9Ustv49BA235KFvgma3UhCxFgVaiLQtf3SXG+j9bpvByMBB0vs2Ts42QEIBhCHjFyjVVdbFP3ZFuqRKFFEjZ11ZrOH3OoW5cZ2HOx+49JG B+BSQrhX 5NcyfsW6DDs2u/m5y7yJjQfRNKtD5cZam5lLVgPpbD06aAZIUXv8ReSsJOtaumVOcLbNGxZ9G7hLkVN2WgooPK30W0Z1T9IA1Di8YERNmDq49WMTDW8V5eLmm3GtXCL8X1SgHiSBsMyMA+QBXf++FM7TX5R2u4hr+ZtEgugPypZFnQ/OS10EYwRQVmnXvbA4fJ69OuNjcBtNXi+c+T8177wHj0Iz/a+JtoiCFdQp71we0kf7io8+wUbHa2/s1F8ZmJpkJKWZ5ShPB+fOpGtBinAoroEolL+75AtzqE2IMYIydGWRwwMdcIIXamj6FhVXSQRmo76JGkDUuDDvOLCHAqyMDfTBdURjvP/YfH+gGB1fvrzQLqETd/mUtob7fjcEc459gn5qsanlFqXc2Eyo0+1z0yA+e91vDiwGkv9snj7naZ/jC3gyJfX9E05LCDTLaDEtrUCdbeYsIvCUG7T1Mka+hTkBcmejpWXneeD2ypZ/z60yew1cGJVNNS2rxgyhVuZQLNFCXkQ3kw19Lq/dMx/prt+i+cFSHJUygHis3Eu5YEfHojnevC5BwS2nv9Q+faPCcgywzCld0zSpAroeijtLi8QqExcdGJSbELashfNlap6Y9UgMDN9J5q6yVwt6egABRxjTpQ/b98aOd0RUQ97t1DgPnf5muTVDnSej/pIJPlA4YW6PXuQqs+kKHJjzCPwcYaRZ2O9pG1r88K4tjmdokNUxvkOvIC36Lz8xUGdYBDELrprCRofNOQdElIt8OrFKr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a skeleton to support scanning and migration. Also add a config option for the same. High level design: While (1): scan the slowtier pages belonging to VMAs of a task. Add to migation list Separate thread: migrate scanned pages to a toptier node based on heuristics The overall code is heavily influenced by khugepaged design. Signed-off-by: Raghavendra K T --- mm/Kconfig | 8 +++ mm/Makefile | 1 + mm/kmmscand.c | 176 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 185 insertions(+) create mode 100644 mm/kmmscand.c diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..5a4931633e15 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -783,6 +783,14 @@ config KSM until a program has madvised that an area is MADV_MERGEABLE, and root has set /sys/kernel/mm/ksm/run to 1 (if CONFIG_SYSFS is set). +config KMMSCAND + bool "Enable PTE A bit scanning and Migration" + depends on NUMA_BALANCING + help + Enable PTE A bit scanning of page. CXL pages accessed are migrated to + a regular NUMA node. The option creates a separate kthread for + scanning and migration. + config DEFAULT_MMAP_MIN_ADDR int "Low address space to protect from user allocation" depends on MMU diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..45e2f8cc8fd6 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -94,6 +94,7 @@ obj-$(CONFIG_FAIL_PAGE_ALLOC) += fail_page_alloc.o obj-$(CONFIG_MEMTEST) += memtest.o obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o +obj-$(CONFIG_KMMSCAND) += kmmscand.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o diff --git a/mm/kmmscand.c b/mm/kmmscand.c new file mode 100644 index 000000000000..6c55250b5cfb --- /dev/null +++ b/mm/kmmscand.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include "internal.h" + + +static struct task_struct *kmmscand_thread __read_mostly; +static DEFINE_MUTEX(kmmscand_mutex); + +/* How long to pause between two scan and migration cycle */ +static unsigned int kmmscand_scan_sleep_ms __read_mostly = 16; + +/* Max number of mms to scan in one scan and migration cycle */ +#define KMMSCAND_MMS_TO_SCAN (4 * 1024UL) +static unsigned long kmmscand_mms_to_scan __read_mostly = KMMSCAND_MMS_TO_SCAN; + +bool kmmscand_scan_enabled = true; +static bool need_wakeup; + +static unsigned long kmmscand_sleep_expire; + +static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); + +struct kmmscand_scan { + struct list_head mm_head; +}; + +struct kmmscand_scan kmmscand_scan = { + .mm_head = LIST_HEAD_INIT(kmmscand_scan.mm_head), +}; + +static int kmmscand_has_work(void) +{ + return !list_empty(&kmmscand_scan.mm_head); +} + +static bool kmmscand_should_wakeup(void) +{ + bool wakeup = kthread_should_stop() || need_wakeup || + time_after_eq(jiffies, kmmscand_sleep_expire); + if (need_wakeup) + need_wakeup = false; + + return wakeup; +} + +static void kmmscand_wait_work(void) +{ + const unsigned long scan_sleep_jiffies = + msecs_to_jiffies(kmmscand_scan_sleep_ms); + + if (!scan_sleep_jiffies) + return; + + kmmscand_sleep_expire = jiffies + scan_sleep_jiffies; + wait_event_timeout(kmmscand_wait, + kmmscand_should_wakeup(), + scan_sleep_jiffies); + return; +} + +static unsigned long kmmscand_scan_mm_slot(void) +{ + /* placeholder for scanning */ + msleep(100); + return 0; +} + +static void kmmscand_do_scan(void) +{ + unsigned long iter = 0, mms_to_scan; + + mms_to_scan = READ_ONCE(kmmscand_mms_to_scan); + + while (true) { + cond_resched(); + + if (unlikely(kthread_should_stop()) || + !READ_ONCE(kmmscand_scan_enabled)) + break; + + if (kmmscand_has_work()) + kmmscand_scan_mm_slot(); + + iter++; + if (iter >= mms_to_scan) + break; + } +} + +static int kmmscand(void *none) +{ + for (;;) { + if (unlikely(kthread_should_stop())) + break; + + kmmscand_do_scan(); + + while (!READ_ONCE(kmmscand_scan_enabled)) { + cpu_relax(); + kmmscand_wait_work(); + } + + kmmscand_wait_work(); + } + return 0; +} + +static int start_kmmscand(void) +{ + int err = 0; + + guard(mutex)(&kmmscand_mutex); + + /* Some one already succeeded in starting daemon */ + if (kmmscand_thread) + goto end; + + kmmscand_thread = kthread_run(kmmscand, NULL, "kmmscand"); + if (IS_ERR(kmmscand_thread)) { + pr_err("kmmscand: kthread_run(kmmscand) failed\n"); + err = PTR_ERR(kmmscand_thread); + kmmscand_thread = NULL; + goto end; + } else { + pr_info("kmmscand: Successfully started kmmscand"); + } + + if (!list_empty(&kmmscand_scan.mm_head)) + wake_up_interruptible(&kmmscand_wait); + +end: + return err; +} + +static int stop_kmmscand(void) +{ + int err = 0; + + guard(mutex)(&kmmscand_mutex); + + if (kmmscand_thread) { + kthread_stop(kmmscand_thread); + kmmscand_thread = NULL; + } + + return err; +} + +static int __init kmmscand_init(void) +{ + int err; + + err = start_kmmscand(); + if (err) + goto err_kmmscand; + + return 0; + +err_kmmscand: + stop_kmmscand(); + + return err; +} +subsys_initcall(kmmscand_init); From patchwork Wed Mar 19 19:30:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2735CC35FFB for ; Wed, 19 Mar 2025 19:31:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 404D0280005; Wed, 19 Mar 2025 15:31:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 38CAF280001; Wed, 19 Mar 2025 15:31:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 192B7280005; Wed, 19 Mar 2025 15:31:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E74F3280001 for ; Wed, 19 Mar 2025 15:31:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DBE121C81DA for ; Wed, 19 Mar 2025 19:31:31 +0000 (UTC) X-FDA: 83239294782.16.ECDDED2 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2040.outbound.protection.outlook.com [40.107.223.40]) by imf23.hostedemail.com (Postfix) with ESMTP id D8A7114001A for ; Wed, 19 Mar 2025 19:31:28 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=rcFspfru; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf23.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.40 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412689; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=IVMKreYDz0DrkhOmW3SSkO0yClBTLsz+JeiJd6tjfc3b9kDM20h4rJtIhV6d8j2YDcFP5Z YrzJ6yXMTxQNHe3bUAruBg/TQVwJ24xyvbw2cBzevyvhX4HD/edxzhMtnuBmXD5FTljt/z w91xshkaVH1U2Ch0ixINRKdGpb5V2sU= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412689; a=rsa-sha256; cv=pass; b=NopzLBtOsdl0e85lDtCk7SmpVY4YdlSZskJ/+Uh1l+NiWqJFLXhQ0heyW7DTJvmGACwoPq HHbwJxTN8pAo2WNqDVrXpVoaJeeC24y+y7znupwApKp5CZ6CH4X4PV7U7DCc2hoF0XU/dL lS5nYkRTy2gW6EfWfZIjQ8Lnwu++WTo= ARC-Authentication-Results: i=2; imf23.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=rcFspfru; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf23.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.40 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kibCpY+AWIHnpIBsHswAK7OW9k8W0vfpD4oZeaGnpRyaTaoPs1NooAsWgc/bp1oep/N5HQAWXVxmKNId6bVE8MYl53R1DZZMKHZSJ1KBQJijIQTOs90osHu4Ki+9Rj9gHjQPKta91IeDE7T7nQZNRAjdKgLByMg18MVfGTbpKUvezQ7T7cGkGR1Z20w9+cQberO9lGagvw+sp+1f53LN3bd/yaoXCq4XrvpqRlfHvTJMBpCNaNDcoH78ZwNKMc7p+rG3wy95MtZcbA8fW9MRWXWoOJp97RtfgYY6ULUYs3Rb3gysWK3HvHv6chGhCVfH2VqK1pb9ETyAbVpa13nmLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=BQxN1+zoYrzxYDKXDthRpMaUaTcmQV4i80r86u5rVouBhOGvaasLvkBGiwlM/LwHMuxCVyC6pIvykXJex3cLoO3+/BDa6P+t48PSu5TfUp/akkmkum0gByFvYszDcCwARkQ7UQmhZe7IEj/UEocRq7jIDRRYsQsmFSmWiyiflnPtgTybTR2eIRPQeuwduIA35tgd4JvYoY5gESSNeeSaiDphW5/EcifTxsUPqGsX1J2cD357Jgujuy04YlMtqWdVGjh+eJ/0lQbh/xmvuKMoz/HE45kPo4Y3ZIZMt+y2ZiRBTYKa2NJ3DhiyUaFN28iGtV/Viu3qrJk7W3S3xCZjwg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4zAsfk1Gi/fQowp0bKiSXni4gGILQhtwytmUrsvnI2k=; b=rcFspfruOE/sAd/tYVASi3Ib8RxA4n+CSPANb6WncngYZwng7fYy3Xg0hB62tpvH1wDEMiZrnOZhmLpB0fzsVVp1Zm19X0WGugptC4J8Miqe4xZhQnL495gjjpphGLEfTxm6199RsWG4herI3gEh4ptSl2gc84tItucGYC/xpYE= Received: from SA9PR10CA0015.namprd10.prod.outlook.com (2603:10b6:806:a7::20) by IA0PR12MB7675.namprd12.prod.outlook.com (2603:10b6:208:433::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:23 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::85) by SA9PR10CA0015.outlook.office365.com (2603:10b6:806:a7::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:22 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:12 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 02/13] mm: Maintain mm_struct list in the system Date: Wed, 19 Mar 2025 19:30:17 +0000 Message-ID: <20250319193028.29514-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|IA0PR12MB7675:EE_ X-MS-Office365-Filtering-Correlation-Id: 1e91f9a3-0125-4018-4357-08dd671ca186 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|7416014|376014|82310400026; X-Microsoft-Antispam-Message-Info: dexsYIeqHGJqJtQI2EXSAibliOi3YzKMApOlwl2TqowRFfLgxr1MOTyuSD7PvOFhWadrbBOa5bdaxUiHpruKEBwTJt0W0epoh1bzTuQrIoMERDwMLoWdPgYRjrjO8E8CMLSR+WadaTCmJX6XkMXE13JehXlw5vXEdy39m6i5Mk78wwlnExHh/8671u9iqowCP0chs7KdC2dgQo9gpwtxlBnHnfGtkqy+AUPzky1W0zK/WVPJ6/WBijiJbV5/MVGTMf2xPJGz5nny2x9HDTkW1CvCm/dpvQhuiKUgRtZ5pXu9gjwdK1WJy8FBv2nRh5xET80sws4Zht1wdTkl69PEi0rYzsH70/415UHc6rZKXk5+yEFs2H0cbJXTXRZ79r8ldgKRQtVtj0vIPh6u8Xtkx2ulBbb1okXiWiIuf2S+YNf7INExZvbSrVEInhsuvt0SmjMlah2Y+kYTRjKZ6BPfwrXaDMDJwWmtmMftKTAX3AF24p95kE7PV3qLKlr7bib3QXUShKRb0KM1Em2U93ekvn+5GdQHo4FZjy4l9IzX85HHoE97IoLMIEoE/pEPHp5owgn9Fffpo2SBSdcJPej40OAZUp6TTW0+p2YZ5m7Xr4fMdw0wQY08Hsh2GwjCFOvvqRFh/7VdGvY3enIdVEiwn744sPTQ99LyZAfnuAWeCg3UZso/bIJpttI6tlKceoXJwUoJitDIML62JaFcx4rAuYS9VONsmtqK2c5GhLwg1vVLly9oqsUhOW3V3eWpNl/snk6+B1JcRJwtvdlatmQE6xZCtETCkgtdosMx45M/3jagSSgabe9kCVWywMdR1Tj3vPfhmPVlt3iKzCoCbTEaZnamQ4x/TlsGeFeuCA6LIZyOzZWEkJuBjXv4cCeDIleBB85vK+lqqCG9w/eR5LtK5sWiDlNOMhTzgP9KDTtm2jcPQZ17WLqM1zVqJ2OZYEwyWI17xeVHrh/3QjCBDO7r6+Xmim5FpTEuzafLDKX1EMqxTZyIL/fls5+8/gRuL8zM2/TCdSJzr8Px5zx78OWJjjXeEzzdTZIxYWSHMSI7o9//y6nKxlu5xLugGWywB2OwtgDpZD81RysEfb8B1pJHYB8JSbl+UNgmvq+XDfC9HuX+VLXp4f954X2gzzoKrKZ3LcO+02ZHpT97WR2usYIDfq3ssGmv3POznNnIh2mzkGL1wFCXMvbpcs9hBT87di6FGNiMyqaPvtIpLv6rbhrSAbSnZY8flN3PEUZVtJVNyM+yWX8YEia30F45p9oL8zzdW2BkpdWbNOb2XC6EAO/Nci+oTEE8DgDnTfuMD30Tokg268RRAYxtzGdpSo/GsmFKCh1rj6QxSMc/CPqY9Zv7Ig4epp4VIhkOJ1VXMzCwCk5V41htmvybGZ+8iA0nVdvNCdUJRxDoFvtVou67+k2C55Uzy1yk296UKUUdowAO3BQH3UWkvN+vE2t+2h1xN3jFRsLJ0e75aVHBvIuqhD8NLmr8Xa/AtcujGCjIiJXwT/A= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(7416014)(376014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:22.2373 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1e91f9a3-0125-4018-4357-08dd671ca186 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7675 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D8A7114001A X-Stat-Signature: datza4xdywug6sudsotqtridcw39czft X-HE-Tag: 1742412688-186618 X-HE-Meta: U2FsdGVkX1+jLxlmuq/QdJVbc+b5ys1kI+2PPd5ed9WZ9NL9kde2A4xgpNtTX4dJyeoYfMYfCWaPu7dhBKwo4j0Bx7sZGaD96TobLGp7lh3X2c6OJhvT182SIk/hiz6FLNAHkSIVnZYMpD6U2D6etndZUKxaqr4fsv5WIa9y+SysY2BgWwAZSnAE11a9roji7mNv/KG5ijzvokWDukNV9h5WPuYB8buLPT+Kpx0G9fDs5uamdzvotOM8wAjK5wNPXTR0fVlvDU6tBshHORzGuJZesAYLZPmCFGdKJ2kt82zyRAxr/q53dcDn8J9QumzirPIAqLfZfu7HDmHgMISK74ln59qUyHFLdVn2eq+0TfOKiL6qEQWgS/0O4e6TJMxnIkRS+E7zEt6sH2mqRlqf6fnDWpuHDL94mZFD/6COasUjZAghVk5BHSdQIVr6PhlItMeV2UcHe99wmEQDXu3jnIIwxjjHWldz/5gEVUsMGzxVCfIq/RA+3TelpuMz7BlyQpEUU6ke6CODHc+/oiT0dyth+TliPX9kqqs+kFjTL5VREtBEyqBHQezZT6g4jjYA7jltjHFm2cPI1yaFzE/gM8Fl5zoqj7R41O4qwhgwsJbaXq7TDYi5M5TvN3HKQuoy90OYeXff/vmNoAX1JzUw1NUyGEpAAwSnVOHdIKVtar/Wc1CTfS/ZIA4vzub8ochCtjli5WPDMvCj19ax9e/g0PlTo28CFHB0woiZhI/ZunEEob9+/rgucWl8nlGDReebkxJV/T3rsrdBMshoag/2+bbFvplNAsGFJcHRynGZeWxfP9yrDtGGs6f8YSv3tYxC+Ax4edwZW1F60Ytp9x5Xagvbl7Ew/KT/yyKo4Sr95rIAnbr8dfR65e3Bxlp/Wb580iFNFiEm5lmWjIqm04QwGLMG9KpwDVoRt6y/7JKSOz4qcZywCpylU3vdEiXhHdJs58cBjOHoQCsj2/b9hfd n35gN6/A HOXa2H2ygM1h4iDrn/tOLnAjeDrUFJYGoIcCEK7aJjIbwoKB4i47EgpkM7uwUm95r03oEhEIKBGkkl+hDVSP60xje28VQNS2F1tVmuBM4gAMROaOKtEzvcPc3Z5VDpou6MjuMC4JivMjEl9676qGfVsQ/yfgziMgMoNsMXO/UoaTapFOINBBsInmePieHJbUr8T6muRCSkebbYqOlEEhoVfRCfwdjRgpm/r6d2BtNXbf06l0ceUTeDbXl0/mQRd+keUDW5JBl4G44eu8wovcK+ii4AWy3FEJufqonUGrzbZGfigPotvfJaGZDorvkZOIWzEzttNWqvSM81Ea6oD/TrnQp1U/tJz789C+9zhTbBMNJOuxYK45H5Y3zddG3mfkY3eCEm9FTMtV9gzsybHdpuT9AooWYEIYB43gJvpKMujmHRqq3ttP63sEjrtvpnsxmbcIozbr+iINAMQ7OpXa+61oUmSwRdFNXOkoJBZifKqEOdxurxlvlNP0xRp7+/ZLRtM9ab6aF3gw02pvJYpXHUjsNmugFmjk9+hATiUULIjCy2nAo8JIwnFyz9prSxp41E+i+rRoHOfkK1Tgm2RZxx12YCVF0BPl6kMTb/2xb3kPrgpMccBwyePQLjUujDPP9AXloAxQtUZZnnvNjLY9xYbC4Q4cp8tU2UkB34ff2XRPmUbp+Z1ePs8zEILGI1Me32/dL+JzjbhQD5qZ8YigYcEIRY1AJpAWumu2BqoVG4eMuqO1a5n85Ki6RAPjUTMJ/VM+dV6p0YFIBRn2/1MMifNr0753Os1DqiWaTR0O1K/73LhPW+CsMvgzNaSLrQpHJvIgUbYjxhTLORng= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a hook in the fork and exec path to link mm_struct. Reuse the mm_slot infrastructure to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- fs/exec.c | 4 ++ include/linux/kmmscand.h | 30 ++++++++++++++ kernel/fork.c | 4 ++ mm/kmmscand.c | 86 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 include/linux/kmmscand.h diff --git a/fs/exec.c b/fs/exec.c index 506cd411f4ac..e76285e4bc73 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -68,6 +68,7 @@ #include #include #include +#include #include #include @@ -266,6 +267,8 @@ static int __bprm_mm_init(struct linux_binprm *bprm) if (err) goto err_ksm; + kmmscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -288,6 +291,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) return 0; err: ksm_exit(mm); + kmmscand_exit(mm); err_ksm: mmap_write_unlock(mm); err_free: diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h new file mode 100644 index 000000000000..b120c65ee8c6 --- /dev/null +++ b/include/linux/kmmscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KMMSCAND_H_ +#define _LINUX_KMMSCAND_H_ + +#ifdef CONFIG_KMMSCAND +extern void __kmmscand_enter(struct mm_struct *mm); +extern void __kmmscand_exit(struct mm_struct *mm); + +static inline void kmmscand_execve(struct mm_struct *mm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_exit(struct mm_struct *mm) +{ + __kmmscand_exit(mm); +} +#else /* !CONFIG_KMMSCAND */ +static inline void __kmmscand_enter(struct mm_struct *mm) {} +static inline void __kmmscand_exit(struct mm_struct *mm) {} +static inline void kmmscand_execve(struct mm_struct *mm) {} +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) {} +static inline void kmmscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KMMSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 735405a9c5f3..f61c55cf33c2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -656,6 +657,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, mm->exec_vm = oldmm->exec_vm; mm->stack_vm = oldmm->stack_vm; + kmmscand_fork(mm, oldmm); + /* Use __mt_dup() to efficiently build an identical maple tree. */ retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) @@ -1353,6 +1356,7 @@ static inline void __mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ + kmmscand_exit(mm); exit_mmap(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 6c55250b5cfb..36d0fea31dea 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -7,13 +7,14 @@ #include #include #include +#include #include #include #include #include #include "internal.h" - +#include "mm_slot.h" static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); @@ -30,10 +31,21 @@ static bool need_wakeup; static unsigned long kmmscand_sleep_expire; +static DEFINE_SPINLOCK(kmmscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); +#define KMMSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HASH_BITS); + +static struct kmem_cache *kmmscand_slot_cache __read_mostly; + +struct kmmscand_mm_slot { + struct mm_slot slot; +}; + struct kmmscand_scan { struct list_head mm_head; + struct kmmscand_mm_slot *mm_slot; }; struct kmmscand_scan kmmscand_scan = { @@ -70,6 +82,11 @@ static void kmmscand_wait_work(void) return; } +static inline int kmmscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) == 0; +} + static unsigned long kmmscand_scan_mm_slot(void) { /* placeholder for scanning */ @@ -117,6 +134,65 @@ static int kmmscand(void *none) return 0; } +static inline void kmmscand_destroy(void) +{ + kmem_cache_destroy(kmmscand_slot_cache); +} + +void __kmmscand_enter(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *kmmscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kmmscand_exit() must not run from under us */ + VM_BUG_ON_MM(kmmscand_test_exit(mm), mm); + + kmmscand_slot = mm_slot_alloc(kmmscand_slot_cache); + + if (!kmmscand_slot) + return; + + slot = &kmmscand_slot->slot; + + spin_lock(&kmmscand_mm_lock); + mm_slot_insert(kmmscand_slots_hash, mm, slot); + + wakeup = list_empty(&kmmscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kmmscand_scan.mm_head); + spin_unlock(&kmmscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kmmscand_wait); +} + +void __kmmscand_exit(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free = 0; + + spin_lock(&kmmscand_mm_lock); + slot = mm_slot_lookup(kmmscand_slots_hash, mm); + mm_slot = mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + if (mm_slot && kmmscand_scan.mm_slot != mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free = 1; + } + + spin_unlock(&kmmscand_mm_lock); + + if (free) { + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kmmscand(void) { int err = 0; @@ -162,6 +238,13 @@ static int __init kmmscand_init(void) { int err; + kmmscand_slot_cache = KMEM_CACHE(kmmscand_mm_slot, 0); + + if (!kmmscand_slot_cache) { + pr_err("kmmscand: kmem_cache error"); + return -ENOMEM; + } + err = start_kmmscand(); if (err) goto err_kmmscand; @@ -170,6 +253,7 @@ static int __init kmmscand_init(void) err_kmmscand: stop_kmmscand(); + kmmscand_destroy(); return err; } From patchwork Wed Mar 19 19:30:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023063 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C6A0C35FFA for ; Wed, 19 Mar 2025 19:31:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 466F7280006; Wed, 19 Mar 2025 15:31:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F0F6280004; Wed, 19 Mar 2025 15:31:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F51D280006; Wed, 19 Mar 2025 15:31:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EDCFB280004 for ; Wed, 19 Mar 2025 15:31:44 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1618BA9A65 for ; Wed, 19 Mar 2025 19:31:46 +0000 (UTC) X-FDA: 83239295412.23.BE3AD38 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2054.outbound.protection.outlook.com [40.107.244.54]) by imf14.hostedemail.com (Postfix) with ESMTP id D1C0C100011 for ; Wed, 19 Mar 2025 19:31:41 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=FVBN95Of; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf14.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.244.54 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412702; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cx9mqxLbBvGY1Qdr7POiHWnrJdk6S1CQ2w02ch47VKs=; b=yeEPmPbgvCgHJF+O8dBnTEe16f3cCp9nB8kOdgQ8ZUPqkJykkNfcHcQxRuW3Ro/s/nca8C LTxuCdDYId/3bJ9Jr/rrqp7GvrPjeEjZm9sh9YA3sHuNyLrXhnx2AYiqFtdFoLEREEZP2I Afd5iRFVkNi9HJ0FhOGzHRX0QZh/OKU= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412702; a=rsa-sha256; cv=pass; b=8KfOYmepjonaxLyAh81lNwDpucXHQBAsJQ8De6zm6d2Kh1A29mqvpoWp0IeJZJYGUBTKCg mYTPzeJNfIaXHCqa4J6NgKyYjxNerDvKx8OcGtrwNuDOx/QfqkNXF7SBsTWu921otb2tB3 2f80JfsVod131g/NWI34Zxcyi8Vu3GU= ARC-Authentication-Results: i=2; imf14.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=FVBN95Of; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf14.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.244.54 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EnrfdaYEkoqgwUJbkZKpDtm81CIdd3DCcFJsc4OyQ9hXOfMYMgayiYdvcy74Eb8WBt/KHDe4KC8297sTNWliY4HV1b3jqSaxXcFumY9cMq9yBwhtgz0oM7ncutuNYcDDAAH4q7C9reAPDwPILMUIesFGkIbOLqq/I3tRXvDPhvnr48j80WIui7cTHSql6u/qx7sIvRF8yE2zn9oAxH/3Oqk3AyVkSSNJFgD9usG3EKHmg5qjFwNRy5QkJ+Ztxlv6AT/TbHTpXRPUpKHab4MWoQQ5sg5Ijkcgkaw80IsEcWBVn7TnMSpWh0WGD2BqwtBBr7FK+WoLpWxn6EGznJkWHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=cx9mqxLbBvGY1Qdr7POiHWnrJdk6S1CQ2w02ch47VKs=; b=Pr4585eyMNXyJgLLKLkLf8qhGTZBqRuHBHk911iNN2VjPIWgE6tAu0FD2fBnqfweIWNcD4vhUqxL/xWf6Hu4n/ZI0bl9uxlAx+sA34/2ygC3VLaLb9PLWvsvOeV8xhMa/nQM9yjjYV5RkgcbNg+D/RzJr9wP3wSjMHPxpHZxhuUunuoDd6eyt+h0PYUBm+d8O0d/udOc7y3rWj2jvgyEDjBuC8QvMQfbZqOEc8pK3uDIQcFDFlM7FF8yIl1PCKokr9OaHl+WksRnZ0/pwGAjs1/N3nAvLMrjp5nUnhUU3XGV+8m11Iba3lLbBu48fZddey3py+OUDbqySwn7XdF+1A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cx9mqxLbBvGY1Qdr7POiHWnrJdk6S1CQ2w02ch47VKs=; b=FVBN95Of5poV8druQhY+53PKt2RzSYkW9rha8O7wtd7L1HRrkT8tHel/r4zmASi/Gh4ne4OhYkEk5kYRO0+htAZMGfzpuJLS2RzJOI6yeFuvIZuN4uVAMbmNBQy0iAMkTUKkUQF99mLVGnfQsPDeptmOlnnKLsOU98J46Ivqje4= Received: from SA0PR11CA0165.namprd11.prod.outlook.com (2603:10b6:806:1bb::20) by SJ1PR12MB6338.namprd12.prod.outlook.com (2603:10b6:a03:455::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:31:33 +0000 Received: from SN1PEPF00026367.namprd02.prod.outlook.com (2603:10b6:806:1bb:cafe::e4) by SA0PR11CA0165.outlook.office365.com (2603:10b6:806:1bb::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.35 via Frontend Transport; Wed, 19 Mar 2025 19:31:32 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026367.mail.protection.outlook.com (10.167.241.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:32 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:22 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 03/13] mm: Scan the mm and create a migration list Date: Wed, 19 Mar 2025 19:30:18 +0000 Message-ID: <20250319193028.29514-4-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026367:EE_|SJ1PR12MB6338:EE_ X-MS-Office365-Filtering-Correlation-Id: 5feff88a-149c-4187-afa1-08dd671ca78a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: NTMEQ56E2JF6KYEc9gw2QsQreOr597HyqufvmoNP16bDmzEPj/QIC0li9G9E2yswVh6KBA7wUU3vX016XL2X675QUZDZIeWO/JrTF+ojXQydJd2qSE/w8FQ9OYDIXQKYhwuEC4kaz7cHtoFhE8lN4+fZxGk+5C4JmEtOX1nISqfoYIokylwhvODH+3/aVSk5p8Q/ayzL1rtQ0QakDhhlVclV7Bq72QyoWlrsQilelkyIhoc/inUDSwAWXyQX1hE6p5tD7R0IXjhI04bT9SJcR4I8NTssdtvvMigTRiVC8XpvFVnS+JnPFzahSx75PeoXgV9BQKRWeOFgFHwsX+EQ2fxro6Xr6q2in8hQdEYGuOsTLnMF78YSJIc8SL6nQp0vpAFQjIysXB0Tc+HiUi0rxjyLCfOcydld6u1GKDzrwC7uwac+cLV1CrIj76Cmu63j3KyxBFA5RfbouYFEj7huS2lTFLGZJYiy9Uibkd/mmZBBIr/6TRchZLKBYk3NYP50o8SOG8n7GPMQF9bsMx3fNxJF9ux7pbM+PVOBmLuOrd+cIeFl6q9bgAsbSh0P89xd275/p+qQ2VlKv1U4nWF2M6lItjBa0aefc7szodiWGcbO8z45bskN+e62169pYxnWjafkHQ273QDH+/tWj7sQ5ymDyHY8kVqymvQbNkhEc2tb3ael+gPjzzsZzNZmbyUNyWbdP2m9gYnCbgLsitWLkCgBZfqS2x8OmOuCV7V9x8ER/aNo8l9oqQR77BW446LEIABZvr3/LCGE9D49jRIp0XlEJyZZrXACh03ItoDUv5pJRn83C9ECjGNXTaefpKOL4kSUUMQ/dwbfhWc1bSD/qziStqlMSIuk3Ta2J+nhy2GjjouQySnO60B7VFL5Qp2u72ogOO8BXCA3lWYKoNgdBW4mr4ZKBK/7yE1dJL4D4aNLeGlq2IyWXVfuf9JbjPW9B/z3luK5O7vlU5mNTxLv/SwOXItj5dvNqKRtlt9t+XkwGEyzyOJiOOZQRJECw4WraMt47euEU/pXRpQZPcCmvLqEajsrtryan3WMDTPjVub+Zn+25ZN1gsbnju+/rHU5JQnEjtbsOohW3g6vlqonOMP2fbOdQHtSpxZ69s06vM+3mDELBuJIYya8k1wW446bcs51fBaivAvV22/++c8PQ7Nphewdk2qhtBz3Sm4eKSAbh2i3kjAUq9WUaEwucSiTiTfZ/a7NPs67VQFOCvIwEpiB6Trcm+7NYSfU3RJFZziQpC3o7v5WVK8xIsXcOCqNM+g+XAfh2uIbk78bhNEI63QCDgtK+VJ1RCIVbw0rx2XqXx2qocWUVk/69oylzK36Nx3Jk6sSVcTBAHCrcGTm2SvYJpyVzMdgQIWSSegQamIF6qvD6f8ty3T0eJAzEhHsrjWHkYb7xcdqAsyMb6oWfnsvxm+3gDIHnu2FJmi3Gsb5F5ToT32W5XHgJlA9E57YZ6s9DUaXyj4wKGvF+lFWkij7Pp01EBCc4QngPf42MTk= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:32.3482 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 5feff88a-149c-4187-afa1-08dd671ca78a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026367.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6338 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: D1C0C100011 X-Stat-Signature: sqdztb9gyorcuakjj5m3uqw3q5uy6bax X-HE-Tag: 1742412701-807094 X-HE-Meta: U2FsdGVkX18HtUSdbvbuMLBmDUWC4De9qUcKpumMBncwqefx47YfgcahSlb5Ccaw2P56oWmhqAuMltOtzy4JjTstT9pwGdJ7hOFXU6Glt+2wlRbZK7C8Xj/vo1x5ceu+32R74Fo48VE+NDzV0l7h1RJ5aWA56aXHwuYUsNjJTZDiClkSsf4vuf0VFJzc91xTXvDqYSJl4vIretXxoprtHksmvTxxXbxyWOG0V1eqb/3xjnz+3twbqMNB4sS80LjoFikL2Twj2EN8ZeOP6acn8i/GEP+IHhj/JHJh75yh6ZvTx6uU65rrbhg95ynLVqsGM074b4IXmqzgGl1fgIy0pIfSVy616JtfgWP+zZ0o2dq9Td771G8U/7wZqms+Pgsn9hebLcMFslaTdOvzzXn1UTDYrDMUWTT5zsyNDOZoQQ08nhbApKKAG/6w5IQW0pW1+2mQF4joYyueZBW6lA38XzSB4m7/plbv0WqqBlhZnmW1x2WeFNU/JO7HMsx4mXVWSL1G6q8rT8k92KflXaW1VEXP9eDnOnI7Aqu8gWZN27j3RUC3t4O7taOASq280uEYQsfCr+8cGP3erZDVS2p0QceUmOS54klBkLKKNKXg1+kXdVtE1i+qh4CEt8WAaMmhD8HiceZMlTsQ6918cDiy6GuqdunFCdmRdFbP8ZXhDHlZv7/CM71qjGMEFd31IL3fbRsq6qsIEdKMFKhaPNX0t92lxINbX29/mh1aUggt+HHAY25cqwU1gjxMX2bfxaAtcQV837VAmbEmE/K6B7C4WCZ67lXwtRW4GuCiVFvukcSPOQaBr/Ygby52n6u9SGdEg4tQXjK9JS/0ucqz16xq8QOk4TU4S1JWUDoLKaSDl/IlVyGMA7sd8Flp0olZhgHtAxWjm3pUK6SZLXSMfbRf1jy6MRcr7Re3UIUY5LD/QEomD/jZs5m8Cw/8t37a3ZcXLt0K5Iv+KJP12k8w+F7 bI67t/rx 8VGqe1KfGA7PHHvnXdG2HHy9McR6xu16pxV0wSk6Tms/YNuMiowoK/UZIPTocI7+TKHLmozyVXfWpOlTVT5EUiyp/5dhIT2jrkDZ6vVhBQVUH1na1qBIEQQRBWL2ut9BGRilPAEeSN5RtIJXyLNkYPrdiQDdQ+OwM7PN9HcJQO+/JUKAD7FU/NlZaMSAju/H2DDPhwks+bKEwURwmgiY5PGo6qLANkjsY1Y0ElC5pIZflhbrvtk3zE+8Zoad5jigM9LV1kSRPmbusbyKIYxFPURODEBGv+3g6ty3Ohh7F1r9AH5/ilRQVmjMa2kEZkr4Ztt2QfFbCnr793+oWtjrQkGR6wDC7brpBLekWzG+gfrvKThPelKdgiC5tq9CZiQhYPpPu/EZboDdobBCqcoK7jKLJdAoE8O3CCovo+hQkHHqEuewxpCV8nF3+HXOLq0JrC2PaMC5V67HeYC4w73Cg/sakRofwRFP7qREIDyNlDk65tF+yFBEuvhRRguugMtw5uqwYJTqTAjGEvPYQOcJnM4oCM+xeeq6D8KDKCPoNycxihagC6U2ONLsdkSdt+lb5/8Qf1AxnwFEl8DXGLBHK3bKFIiOGDYB36UG/FNNFOa8eqt2dFVzmjrhkD5Ijbknjb1kfcHNQg8rxfPtIrtijmswOfy2KdZlxRhZk71xkh5uuva+vpngyMBKN1bcdpDsOffNh9XRxs02dDbWEhSP1/dhPHAq4Zs5vCxUVZl6qysHcizWN3VHupDDC0XwMoGLPFR+1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since we already have the list of mm_struct in the system, add a module to scan each mm that walks VMAs of each mm_struct and scan all the pages associated with that. In the scan path: Check for the recently acccessed pages (folios) belonging to slowtier nodes. Add all those folios to a migration list. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 323 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 321 insertions(+), 2 deletions(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 36d0fea31dea..a76a58bf37b2 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,10 +4,18 @@ #include #include #include +#include +#include +#include +#include +#include +#include #include #include #include #include +#include +#include #include #include #include @@ -18,6 +26,11 @@ static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); +/* + * Total VMA size to cover during scan. + */ +#define KMMSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) +static unsigned long kmmscand_scan_size __read_mostly = KMMSCAND_SCAN_SIZE; /* How long to pause between two scan and migration cycle */ static unsigned int kmmscand_scan_sleep_ms __read_mostly = 16; @@ -39,10 +52,14 @@ static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HASH_BITS static struct kmem_cache *kmmscand_slot_cache __read_mostly; +/* Per mm information collected to control VMA scanning */ struct kmmscand_mm_slot { struct mm_slot slot; + long address; + bool is_scanned; }; +/* Data structure to keep track of current mm under scan */ struct kmmscand_scan { struct list_head mm_head; struct kmmscand_mm_slot *mm_slot; @@ -52,6 +69,33 @@ struct kmmscand_scan kmmscand_scan = { .mm_head = LIST_HEAD_INIT(kmmscand_scan.mm_head), }; +/* + * Data structure passed to control scanning and also collect + * per memory node information + */ +struct kmmscand_scanctrl { + struct list_head scan_list; + unsigned long address; +}; + +struct kmmscand_scanctrl kmmscand_scanctrl; + +/* Per folio information used for migration */ +struct kmmscand_migrate_info { + struct list_head migrate_node; + struct mm_struct *mm; + struct folio *folio; + unsigned long address; +}; + +static bool kmmscand_eligible_srcnid(int nid) +{ + if (!node_is_toptier(nid)) + return true; + + return false; +} + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -82,15 +126,277 @@ static void kmmscand_wait_work(void) return; } + +static inline bool is_valid_folio(struct folio *folio) +{ + if (!folio || folio_test_unevictable(folio) || !folio_mapped(folio) || + folio_is_zone_device(folio) || folio_likely_mapped_shared(folio)) + return false; + + return true; +} + +static bool folio_idle_clear_pte_refs_one(struct folio *folio, + struct vm_area_struct *vma, + unsigned long addr, + pte_t *ptep) +{ + bool referenced = false; + struct mm_struct *mm = vma->vm_mm; + pmd_t *pmd = pmd_off(mm, addr); + + if (ptep) { + if (ptep_clear_young_notify(vma, addr, ptep)) + referenced = true; + } else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (!pmd_present(*pmd)) + WARN_ON_ONCE(1); + if (pmdp_clear_young_notify(vma, addr, pmd)) + referenced = true; + } else { + WARN_ON_ONCE(1); + } + + if (referenced) { + folio_clear_idle(folio); + folio_set_young(folio); + } + + return true; +} + +static void page_idle_clear_pte_refs(struct page *page, pte_t *pte, struct mm_walk *walk) +{ + bool need_lock; + struct folio *folio = page_folio(page); + unsigned long address; + + if (!folio_mapped(folio) || !folio_raw_mapping(folio)) + return; + + need_lock = !folio_test_anon(folio) || folio_test_ksm(folio); + if (need_lock && !folio_trylock(folio)) + return; + address = vma_address(walk->vma, page_pgoff(folio, page), compound_nr(page)); + VM_BUG_ON_VMA(address == -EFAULT, walk->vma); + folio_idle_clear_pte_refs_one(folio, walk->vma, address, pte); + + if (need_lock) + folio_unlock(folio); +} + +static int hot_vma_idle_pte_entry(pte_t *pte, + unsigned long addr, + unsigned long next, + struct mm_walk *walk) +{ + struct page *page; + struct folio *folio; + struct mm_struct *mm; + struct vm_area_struct *vma; + struct kmmscand_migrate_info *info; + struct kmmscand_scanctrl *scanctrl = walk->private; + int srcnid; + + scanctrl->address = addr; + pte_t pteval = ptep_get(pte); + + if (!pte_present(pteval)) + return 0; + + if (pte_none(pteval)) + return 0; + + vma = walk->vma; + mm = vma->vm_mm; + + page = pte_page(*pte); + + page_idle_clear_pte_refs(page, pte, walk); + + folio = page_folio(page); + folio_get(folio); + + if (!is_valid_folio(folio)) { + folio_put(folio); + return 0; + } + srcnid = folio_nid(folio); + + + if (!folio_test_lru(folio)) { + folio_put(folio); + return 0; + } + + if (!folio_test_idle(folio) || folio_test_young(folio) || + mmu_notifier_test_young(mm, addr) || + folio_test_referenced(folio) || pte_young(pteval)) { + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return 0; + } + /* XXX: Leaking memory. TBD: consume info */ + info = kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); + if (info && scanctrl) { + + info->mm = mm; + info->address = addr; + info->folio = folio; + + /* No need of lock now */ + list_add_tail(&info->migrate_node, &scanctrl->scan_list); + } + } + + folio_set_idle(folio); + folio_put(folio); + return 0; +} + +static const struct mm_walk_ops hot_vma_set_idle_ops = { + .pte_entry = hot_vma_idle_pte_entry, + .walk_lock = PGWALK_RDLOCK, +}; + +static void kmmscand_walk_page_vma(struct vm_area_struct *vma, struct kmmscand_scanctrl *scanctrl) +{ + if (!vma_migratable(vma) || !vma_policy_mof(vma) || + is_vm_hugetlb_page(vma) || (vma->vm_flags & VM_MIXEDMAP)) { + return; + } + if (!vma->vm_mm || + (vma->vm_file && (vma->vm_flags & (VM_READ|VM_WRITE)) == (VM_READ))) + return; + + if (!vma_is_accessible(vma)) + return; + + walk_page_vma(vma, &hot_vma_set_idle_ops, scanctrl); +} + static inline int kmmscand_test_exit(struct mm_struct *mm) { return atomic_read(&mm->mm_users) == 0; } +static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) +{ + struct mm_slot *slot = &mm_slot->slot; + struct mm_struct *mm = slot->mm; + + lockdep_assert_held(&kmmscand_mm_lock); + + if (kmmscand_test_exit(mm)) { + /* free mm_slot */ + hash_del(&slot->hash); + list_del(&slot->mm_node); + + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } +} + static unsigned long kmmscand_scan_mm_slot(void) { - /* placeholder for scanning */ - msleep(100); + bool next_mm = false; + bool update_mmslot_info = false; + + unsigned long vma_scanned_size = 0; + unsigned long address; + + struct mm_slot *slot; + struct mm_struct *mm; + struct vm_area_struct *vma = NULL; + struct kmmscand_mm_slot *mm_slot; + + /* Retrieve mm */ + spin_lock(&kmmscand_mm_lock); + + if (kmmscand_scan.mm_slot) { + mm_slot = kmmscand_scan.mm_slot; + slot = &mm_slot->slot; + address = mm_slot->address; + } else { + slot = list_entry(kmmscand_scan.mm_head.next, + struct mm_slot, mm_node); + mm_slot = mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + address = mm_slot->address; + kmmscand_scan.mm_slot = mm_slot; + } + + mm = slot->mm; + mm_slot->is_scanned = true; + spin_unlock(&kmmscand_mm_lock); + + if (unlikely(!mmap_read_trylock(mm))) + goto outerloop_mmap_lock; + + if (unlikely(kmmscand_test_exit(mm))) { + next_mm = true; + goto outerloop; + } + + VMA_ITERATOR(vmi, mm, address); + + for_each_vma(vmi, vma) { + kmmscand_walk_page_vma(vma, &kmmscand_scanctrl); + vma_scanned_size += vma->vm_end - vma->vm_start; + + if (vma_scanned_size >= kmmscand_scan_size) { + next_mm = true; + /* TBD: Add scanned folios to migration list */ + break; + } + } + + if (!vma) + address = 0; + else + address = kmmscand_scanctrl.address + PAGE_SIZE; + + update_mmslot_info = true; + + if (update_mmslot_info) + mm_slot->address = address; + +outerloop: + /* exit_mmap will destroy ptes after this */ + mmap_read_unlock(mm); + +outerloop_mmap_lock: + spin_lock(&kmmscand_mm_lock); + WARN_ON(kmmscand_scan.mm_slot != mm_slot); + + /* + * Release the current mm_slot if this mm is about to die, or + * if we scanned all vmas of this mm. + */ + if (unlikely(kmmscand_test_exit(mm)) || !vma || next_mm) { + /* + * Make sure that if mm_users is reaching zero while + * kmmscand runs here, kmmscand_exit will find + * mm_slot not pointing to the exiting mm. + */ + if (slot->mm_node.next != &kmmscand_scan.mm_head) { + slot = list_entry(slot->mm_node.next, + struct mm_slot, mm_node); + kmmscand_scan.mm_slot = + mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + + } else + kmmscand_scan.mm_slot = NULL; + + if (kmmscand_test_exit(mm)) { + kmmscand_collect_mm_slot(mm_slot); + goto end; + } + } + mm_slot->is_scanned = false; +end: + spin_unlock(&kmmscand_mm_lock); return 0; } @@ -153,6 +459,7 @@ void __kmmscand_enter(struct mm_struct *mm) if (!kmmscand_slot) return; + kmmscand_slot->address = 0; slot = &kmmscand_slot->slot; spin_lock(&kmmscand_mm_lock); @@ -180,6 +487,12 @@ void __kmmscand_exit(struct mm_struct *mm) hash_del(&slot->hash); list_del(&slot->mm_node); free = 1; + } else if (mm_slot && kmmscand_scan.mm_slot == mm_slot && !mm_slot->is_scanned) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free = 1; + /* TBD: Set the actual next slot */ + kmmscand_scan.mm_slot = NULL; } spin_unlock(&kmmscand_mm_lock); @@ -233,6 +546,11 @@ static int stop_kmmscand(void) return err; } +static void init_list(void) +{ + INIT_LIST_HEAD(&kmmscand_scanctrl.scan_list); + init_waitqueue_head(&kmmscand_wait); +} static int __init kmmscand_init(void) { @@ -245,6 +563,7 @@ static int __init kmmscand_init(void) return -ENOMEM; } + init_list(); err = start_kmmscand(); if (err) goto err_kmmscand; From patchwork Wed Mar 19 19:30:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023064 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0792BC35FFA for ; Wed, 19 Mar 2025 19:31:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37B72280011; Wed, 19 Mar 2025 15:31:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 302CE280010; Wed, 19 Mar 2025 15:31:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13059280011; Wed, 19 Mar 2025 15:31:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E2E06280010 for ; Wed, 19 Mar 2025 15:31:50 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CAAFFB7449 for ; Wed, 19 Mar 2025 19:31:51 +0000 (UTC) X-FDA: 83239295622.16.5F239AB Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2068.outbound.protection.outlook.com [40.107.94.68]) by imf22.hostedemail.com (Postfix) with ESMTP id 74D38C0004 for ; Wed, 19 Mar 2025 19:31:48 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=4wesvsZf; spf=pass (imf22.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.94.68 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PtVePGSIcc5EsBIhdgHjgHCwmTFiPZreK0dOtW/EQes=; b=qroT6V4tPahS4iWs6vHfPEK9WlMhJ7XIMk7cEB2YjDlMMTq9irSj76UiZq5FeIEpM1B5Ib K3APuyttTPksuG0Cy6mBoN99huAjly3uO7ovzxhTPYMSwJf32/GuWqEAW6IKPKINEfePfl K8I6XPUfPUXijyyaTQBjgMijBgrYB5M= ARC-Authentication-Results: i=2; imf22.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=4wesvsZf; spf=pass (imf22.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.94.68 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412708; a=rsa-sha256; cv=pass; b=ffCKzqrKl2VjCxzFPuwcuUF7hgdl6qoxQzHdO18GWas+8/aTntdF4AAPP9wFBn7ZFwhWvM SOGs18cv5v1MsRpbb6GltNsBDGS+HPPG/0RK6/oqLjyH3wMisOMUG+uWjP35439DvvbQ0I oq1KkuTkMFnuQupIYfR4TAkC9Wvzexw= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iNV/yleleGLbHG0RUop+A6UkniDrrRy4IYpeAMMS1V26DgPmancCSx1fIGiFlpLCwq0wDW4/nZ589zth+1/6zLdCGVAaiNvNVgD2Ct/bjiJBDxOITTtbluRfQguIYq+Vn6i8mEjjXnsLiJL9hopeY9/t3a+69XPmJE2CNOIGUdRk0tRD6yzDt3SXpXd3wp+/3gEq37aq7yjSCXieVoDCq9ojR1scYxpKC64OM2ZRgo3UmuKn/k3AI/g4ty/56taYMmmLfWQPmrLWkZRqQB/EvMu588XKAUfVW0IaJl79y5Mxfk0IuRcC9Q/ggLIe6XQhOAb2UQFP9nFhPFU7ewtzcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PtVePGSIcc5EsBIhdgHjgHCwmTFiPZreK0dOtW/EQes=; b=HMvq2dy2xMZCHrc+bBpn6UEUdOENKD2M6KFTMLrNOErKVW2Fwg8RksAWVO5xwCVKApbmUHX26RcDsU7aMoFgNTsVSjw6LcdmfB9DB3TE8YpyKfJisWttbENZGH+TBe5sN98z9cov79OCzunvlp5hqnuCTJoCk8EibQm+KebhmLvpuxU9Lh4RSMvk+IGIEQJPjDir7VtePflvctiAKcKLWI1WxiAnElV454tjrDhwRbkOex5jvlq6gR1Qt9WXMsHPZMI5iefcqDcHMo/Y0lY0BSDHNpk/i7/VeiSsB7fsVhX4bSFEGPalwgx/NZspa9mZXY12GQc/puyfHT6wz5nVIg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PtVePGSIcc5EsBIhdgHjgHCwmTFiPZreK0dOtW/EQes=; b=4wesvsZfUuHPArobaYmr9axYEevL+wv5xx5S0HYFwRo7xevuL69iBJsp4IzaXSk4tmvBSBwlhyGzIfqWxwtTD37Kyuy2fSvQMap3CJWl/6cJAu11V3j/EnAhnm1q4dnxmgWep6iUEiSb2j8cLT2XlVTqsPoLuPEgbA4ol7bwZls= Received: from SA9PR10CA0006.namprd10.prod.outlook.com (2603:10b6:806:a7::11) by SA0PR12MB7091.namprd12.prod.outlook.com (2603:10b6:806:2d5::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:31:43 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::25) by SA9PR10CA0006.outlook.office365.com (2603:10b6:806:a7::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:43 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:42 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:32 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 04/13] mm: Create a separate kernel thread for migration Date: Wed, 19 Mar 2025 19:30:19 +0000 Message-ID: <20250319193028.29514-5-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|SA0PR12MB7091:EE_ X-MS-Office365-Filtering-Correlation-Id: 03dff38e-7350-4ae2-5724-08dd671caddd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|82310400026|7416014; X-Microsoft-Antispam-Message-Info: sF/pEHEPvPGRjshqRhWXUqdKI7+oW+GK9tWDSUXoqK4GeHHdDo3fs41nG5pGha3aua8nSmYi6OPLyk9yV965JdaVIpfe7wImESSkxK/eZFr9HQ89FJP3cNtdXktD0GpVFcX4qcLfwgDTI6LR/UK6PE1SOJxf2qAU1DgsBnHrnK/r4Gt2u/B+uM0NqPF9l9BUadoqM0IGSQACUjX9OP7MbtgW5+zlMFOXp5jviFl0KY7i7MzxmDmVh+LSr29nEaDOMsMb900FDIBYO4RQMcqyrYVJl/AQnZcasqxjsrDf0oQue1xO3B8xwGt+GHq5vowy8fSlYPtYi3o44N5HjZptHTkqu9sDwzTwtJfpXycki2rXJC17KRmJkf16HLABgtb/Jw6MghAcdIjfuGHK29HPOVVzaXWXtI1/8OANuTv/BY5Ra6xEYZ7HEtvgik0qfSvwPlalush4CkkLfVPVkZQSO38ElxEcOzuWSmb2sBLsTRHu+39WZj89gs+J9mwnMWUUg0uWUNEDIXq//3zF1x67pojvQUY0xmcU2AHZs3t3f4ZAzvPyVNsG+66PUJ2Jn4JGa5QhScbdhUG3e07LtWMj29NbpH5IWTMX7M8qVvAHb/BtzUE+0DA4w8DhmUGiyQ8qFMcxYzZKBDTjE+BfSMxkqXZDRkVaEZg294uANsfECkfZzEEu/Ndbq1lh9rXprjSvBQPnFZfOrfR3fvj3vdLQP7650+xPUGyojGpxJ5LkiPDagCvjkp8L+l3ywbnvs/NOixLPgFQN0I/lE/OcLzvKvfjihv/3F/N1ewMZq3Bx2MSjZR8vHxuvd1esqVlC6bnI9bpuuQci4PmRzm2AFTEbZ/JuyyXQKcnYWU2uyWtgbSTcD9mtQkbALM4pOot/mQ6W/fYp9f89ARQnfGgwy4J87vHyPFdJH5Dqs3ZY9TDdGmcEj6KXeGad16RxDE10artQeNZ5e565XFTgTU1WvEgKWsdquSIIq56PsUbKDsR/jVN0GrVkrx0nEP8ao4c6j9+W5rgAmBIkIB2Rp3PaXFqmdhWrwNtgGc/Cmv9Mn4i5ySGsWj2tA+vrBRO/Tk+xU/BTpc/CCmzzhpBYIvsgzVTsrtlF2lCFP5eMnKVAUHTRifePZQv007rBYMugAu9cBbM8JoXbxvT2d1z9eOtlgJPVsxlQDcbZs7fufZt8nwoupQ9/cQ+O61VyEaGRnRJ7eU9MfwbfXPn9Q2kUYF+pPjrXMq7OYzNMl6H7LO2uflNkTNOBGpLh7o9wp9jv65GYqWOJePcAPCEShZA6hI2cyil3N/WiE5fFhFl6xatsKBhvy1EuEsXKVNk0simDenz8ttaNS9MGDJ1gjM/J0lwu74vsHivRLTDpOq2F1lkVczJflnwf9o8Tm2VG7ePNOsIYdpKJq2SyLPo1P16ou0Jw01cdZW+UztqLdAUAUxnwuWmIuZ0ztW0d2Oaq/lXx5RZSD1HcvrWr+1TGsyiUSXu7M3QQYUrI8q76xDDWdnW289XYLVU= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(82310400026)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:42.9406 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 03dff38e-7350-4ae2-5724-08dd671caddd X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7091 X-Rspam-User: X-Rspamd-Queue-Id: 74D38C0004 X-Rspamd-Server: rspam03 X-Stat-Signature: 4gwa77oa8aokrgam1mpgajczpzr8gyae X-HE-Tag: 1742412708-271511 X-HE-Meta: U2FsdGVkX1/3BjwVQj7FZs3epTXQd4WaN0M+Bld5G2rPaC5lbWMoLYQGHRM2+/+FVFMCYtUmsXgp0TBAnB8pgsphojtTAxAPcnXiar7/gsKpGyJIIm8j9pNfdj1268JFWPmRkNavkD4grSYXXHuZ/s4oRFUIbTviZPyXbcRX5cVvaRu4oIMAW4CspoOEpeClUb+1gGT58zBLeF9UT6Ww1QxlDB/Wt5KCHXUyElmyE7xg/NnzpQxgGG8Gwra5xGBbvQC2TnmbAL5JckF2PkQH9gWom3x+nPkiOtXfBsfw+NWWeS/NcPht1tyjOEWpX7i4tfLyd4lhPV2L6yWU8kOyKvonjAy/gQkR8j0vMvnmvuytLdMxYVbetA1DNNW9+1wjng3gwUww+mvcNONg6iMpqpNPRoSLJAOCAucgyH+BrpLb1dyuon/8znP3GFVLTkAPYab1XmLuALzlBwWXPSpME49boQT1Sc9Y80aBxN3UIHFY5yKmNatIvgFeRDsWg6QBB/SERp3L9P2RVj9dSzYch3q3ovfoqexe3BFG3SRANYU054ZPC81cKrxzs+Nqj7Wxc13J2GUQk+lhU2zmhB2OMAUHgcZEy8hXwEUv4+o4+ANmcVZ2Rh5dHzFu4vG368mXQuJx04LlraGLxRhAeLXWlMfq/9vpmDOpO0wPcxU/VFaamw8hEQIp0jfEYi7YJ1z5713j9SlqGod+KKiPdoOIGUupT0zBCyYnpumD8JYTqdyVlTR/91xZybzXzPhglW7/5NaROVYldBsW/WPqXi8Hdc2ww2oyyD+xWAv3+oS12Y/HeMICbOJom3CDB66mbG6oBs7bsPitJCdD0v3BZGa94PRF8HhdM/axoWFvZC2rEF3MnGyS2kBpt1HrjmRf/1FM3wdmVM/gD4IEcwNrbXilFs1VpUQLqP2rcR7ikExq6JvQ+mpdnTyplrxA/AhmkOjoBGn4JPmijkFBknn4mMJ jlQWYfH9 CUOPJ3SCqy/vp3KSuxcZpLIiENxcXns5VzFJc8fiLKtySbD4RwZRxKStG0lfUHe6BWLpkmVJjcmGKGoorfZ1Vrj45wMTUHfPm/L362P/Iwtvz+9izoBQi8ovqLcVxOBBtg8Glk1MeME9uenYzxTMu9hwRM0aMj7JCGra8O4PfQXLu1Isc8Q33fwXuzTXR5zDqhFJWWBs8ZGJoOcjYsEL+w3aVOwXMfxhaF5KcBC7lMQ6p7cDLk7xmc/KRtZG0b++UyAex5hTOT+1kx01oEzF9WRjAQDf+7BnbG+tgP2+rxDOpNYmCLnQiegmT4H4/kTTviQURiqauIngV7qbuXrg+cdAKuKdDuwX5Ip3zYibewFQ4Jj/cqeHaEun+ChougbFSWp2WbI1fkU2PA2hcZu/M2EQUoaqZGKVoLxIVgKojDqMH9ogNjmabF0969u/ua7SBM/S6Ut24C7i4/Nacv+ETK+xrpacArkosIP5RQ3fCf7BT3bi8Huj1Pcc0Zk3hQDySifUccoINupsL+ZGFiwdjYw+2k7YJ4/QpO/ZUUW2X4xuZ92uNNGIf+yyP3p5ADGBP/Vr8/pe4EtWTgcYGKgSYN1iwlDqpWsoYpt4BSEOvBoKyq5mZ+Kiu2NhkPefp3a/vL0yY9kMKDzur5Gd8UUWm/+N6/aSYUFP3SCFiUSzRYWhZCa3VW3ltXZhZR+r8j7DvHnfdC2CQbpzyPHaEaizsl9kS1XuVLLSO3Ei9wgPyEk5DvR5M1tgs38rqBURmUO7q/msE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Having independent thread helps in: - Alleviating the need for multiple scanning threads - Aids to control batch migration (TBD) - Migration throttling (TBD) Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 157 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 154 insertions(+), 3 deletions(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index a76a58bf37b2..6e96cfab5b85 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -41,10 +42,26 @@ static unsigned long kmmscand_mms_to_scan __read_mostly = KMMSCAND_MMS_TO_SCAN; bool kmmscand_scan_enabled = true; static bool need_wakeup; +static bool migrated_need_wakeup; + +/* How long to pause between two migration cycles */ +static unsigned int kmmmigrate_sleep_ms __read_mostly = 20; + +static struct task_struct *kmmmigrated_thread __read_mostly; +static DEFINE_MUTEX(kmmmigrated_mutex); +static DECLARE_WAIT_QUEUE_HEAD(kmmmigrated_wait); +static unsigned long kmmmigrated_sleep_expire; + +/* mm of the migrating folio entry */ +static struct mm_struct *kmmscand_cur_migrate_mm; + +/* Migration list is manipulated underneath because of mm_exit */ +static bool kmmscand_migration_list_dirty; static unsigned long kmmscand_sleep_expire; static DEFINE_SPINLOCK(kmmscand_mm_lock); +static DEFINE_SPINLOCK(kmmscand_migrate_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); #define KMMSCAND_SLOT_HASH_BITS 10 @@ -80,6 +97,14 @@ struct kmmscand_scanctrl { struct kmmscand_scanctrl kmmscand_scanctrl; +struct kmmscand_migrate_list { + struct list_head migrate_head; +}; + +struct kmmscand_migrate_list kmmscand_migrate_list = { + .migrate_head = LIST_HEAD_INIT(kmmscand_migrate_list.migrate_head), +}; + /* Per folio information used for migration */ struct kmmscand_migrate_info { struct list_head migrate_node; @@ -101,6 +126,13 @@ static int kmmscand_has_work(void) return !list_empty(&kmmscand_scan.mm_head); } +static int kmmmigrated_has_work(void) +{ + if (!list_empty(&kmmscand_migrate_list.migrate_head)) + return true; + return false; +} + static bool kmmscand_should_wakeup(void) { bool wakeup = kthread_should_stop() || need_wakeup || @@ -111,6 +143,16 @@ static bool kmmscand_should_wakeup(void) return wakeup; } +static bool kmmmigrated_should_wakeup(void) +{ + bool wakeup = kthread_should_stop() || migrated_need_wakeup || + time_after_eq(jiffies, kmmmigrated_sleep_expire); + if (migrated_need_wakeup) + migrated_need_wakeup = false; + + return wakeup; +} + static void kmmscand_wait_work(void) { const unsigned long scan_sleep_jiffies = @@ -126,6 +168,19 @@ static void kmmscand_wait_work(void) return; } +static void kmmmigrated_wait_work(void) +{ + const unsigned long migrate_sleep_jiffies = + msecs_to_jiffies(kmmmigrate_sleep_ms); + + if (!migrate_sleep_jiffies) + return; + + kmmmigrated_sleep_expire = jiffies + migrate_sleep_jiffies; + wait_event_timeout(kmmmigrated_wait, + kmmmigrated_should_wakeup(), + migrate_sleep_jiffies); +} static inline bool is_valid_folio(struct folio *folio) { @@ -238,7 +293,6 @@ static int hot_vma_idle_pte_entry(pte_t *pte, folio_put(folio); return 0; } - /* XXX: Leaking memory. TBD: consume info */ info = kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { @@ -282,6 +336,28 @@ static inline int kmmscand_test_exit(struct mm_struct *mm) return atomic_read(&mm->mm_users) == 0; } +static void kmmscand_cleanup_migration_list(struct mm_struct *mm) +{ + struct kmmscand_migrate_info *info, *tmp; + + spin_lock(&kmmscand_migrate_lock); + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + if (mm == READ_ONCE(kmmscand_cur_migrate_mm)) { + /* A folio in this mm is being migrated. wait */ + WRITE_ONCE(kmmscand_migration_list_dirty, true); + } + + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (info && (info->mm == mm)) { + info->mm = NULL; + WRITE_ONCE(kmmscand_migration_list_dirty, true); + } + } + } + spin_unlock(&kmmscand_migrate_lock); +} + static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) { struct mm_slot *slot = &mm_slot->slot; @@ -294,11 +370,17 @@ static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) hash_del(&slot->hash); list_del(&slot->mm_node); + kmmscand_cleanup_migration_list(mm); + mm_slot_free(kmmscand_slot_cache, mm_slot); mmdrop(mm); } } +static void kmmscand_migrate_folio(void) +{ +} + static unsigned long kmmscand_scan_mm_slot(void) { bool next_mm = false; @@ -347,9 +429,17 @@ static unsigned long kmmscand_scan_mm_slot(void) if (vma_scanned_size >= kmmscand_scan_size) { next_mm = true; - /* TBD: Add scanned folios to migration list */ + /* Add scanned folios to migration list */ + spin_lock(&kmmscand_migrate_lock); + list_splice_tail_init(&kmmscand_scanctrl.scan_list, + &kmmscand_migrate_list.migrate_head); + spin_unlock(&kmmscand_migrate_lock); break; } + spin_lock(&kmmscand_migrate_lock); + list_splice_tail_init(&kmmscand_scanctrl.scan_list, + &kmmscand_migrate_list.migrate_head); + spin_unlock(&kmmscand_migrate_lock); } if (!vma) @@ -478,7 +568,7 @@ void __kmmscand_exit(struct mm_struct *mm) { struct kmmscand_mm_slot *mm_slot; struct mm_slot *slot; - int free = 0; + int free = 0, serialize = 1; spin_lock(&kmmscand_mm_lock); slot = mm_slot_lookup(kmmscand_slots_hash, mm); @@ -493,10 +583,15 @@ void __kmmscand_exit(struct mm_struct *mm) free = 1; /* TBD: Set the actual next slot */ kmmscand_scan.mm_slot = NULL; + } else if (mm_slot && kmmscand_scan.mm_slot == mm_slot && mm_slot->is_scanned) { + serialize = 0; } spin_unlock(&kmmscand_mm_lock); + if (serialize) + kmmscand_cleanup_migration_list(mm); + if (free) { mm_slot_free(kmmscand_slot_cache, mm_slot); mmdrop(mm); @@ -546,10 +641,59 @@ static int stop_kmmscand(void) return err; } +static int kmmmigrated(void *arg) +{ + for (;;) { + WRITE_ONCE(migrated_need_wakeup, false); + if (unlikely(kthread_should_stop())) + break; + if (kmmmigrated_has_work()) + kmmscand_migrate_folio(); + msleep(20); + kmmmigrated_wait_work(); + } + return 0; +} + +static int start_kmmmigrated(void) +{ + int err = 0; + + guard(mutex)(&kmmmigrated_mutex); + + /* Someone already succeeded in starting daemon */ + if (kmmmigrated_thread) + goto end; + + kmmmigrated_thread = kthread_run(kmmmigrated, NULL, "kmmmigrated"); + if (IS_ERR(kmmmigrated_thread)) { + pr_err("kmmmigrated: kthread_run(kmmmigrated) failed\n"); + err = PTR_ERR(kmmmigrated_thread); + kmmmigrated_thread = NULL; + goto end; + } else { + pr_info("kmmmigrated: Successfully started kmmmigrated"); + } + + wake_up_interruptible(&kmmmigrated_wait); +end: + return err; +} + +static int stop_kmmmigrated(void) +{ + guard(mutex)(&kmmmigrated_mutex); + kthread_stop(kmmmigrated_thread); + return 0; +} + static void init_list(void) { + INIT_LIST_HEAD(&kmmscand_migrate_list.migrate_head); INIT_LIST_HEAD(&kmmscand_scanctrl.scan_list); + spin_lock_init(&kmmscand_migrate_lock); init_waitqueue_head(&kmmscand_wait); + init_waitqueue_head(&kmmmigrated_wait); } static int __init kmmscand_init(void) @@ -568,8 +712,15 @@ static int __init kmmscand_init(void) if (err) goto err_kmmscand; + err = start_kmmmigrated(); + if (err) + goto err_kmmmigrated; + return 0; +err_kmmmigrated: + stop_kmmmigrated(); + err_kmmscand: stop_kmmscand(); kmmscand_destroy(); From patchwork Wed Mar 19 19:30:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023073 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 722B5C35FFB for ; Wed, 19 Mar 2025 19:32:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A295A280012; Wed, 19 Mar 2025 15:32:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B242280010; Wed, 19 Mar 2025 15:32:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DF67280012; Wed, 19 Mar 2025 15:32:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 57B39280010 for ; Wed, 19 Mar 2025 15:32:03 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6878814075E for ; Wed, 19 Mar 2025 19:32:04 +0000 (UTC) X-FDA: 83239296168.26.428D928 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2050.outbound.protection.outlook.com [40.107.93.50]) by imf28.hostedemail.com (Postfix) with ESMTP id 48DBCC001D for ; Wed, 19 Mar 2025 19:32:01 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=GN6uGhX0; spf=pass (imf28.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.93.50 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hsLJE/h2f3iBqJCSh/p9SqyDaJnSWoeY2MYMRp9SgIE=; b=x49HzOtgG6ys/YqgiPJzOkd4IEw+cKMOhlu5zJNaijpwRRiAvxAqXBSyVZFUnKmFmBqTYy rIi4H1IuVNFylK1oA9XvMzTHrC+omtKxy5+vaUeaX1jwxaJExSWbTP2BZBWNG12dkO4L0Z uijPzNalCtkftsAKUHBBLAj1EKFNnpc= ARC-Authentication-Results: i=2; imf28.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=GN6uGhX0; spf=pass (imf28.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.93.50 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412721; a=rsa-sha256; cv=pass; b=QGPqy+uTIAuyTnNK4aSFx83pMi60dYzJBeLIGg1KzKNlUn3ZAdCbtJVTAbKtN2mUUQYcL/ ZkY4CFu5EBHojEFDyiV53U8NxjIphJOHhdayzejUjfeExlTt4/EKAIjUjHpJQ417H15dp4 JoaN1Z+pJIlukr3NMKsv4kgBgraF+m4= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ppJDLvpTagOkHNCEnoyM3sxIhBOiWvGelk244CtIPpPkQfYgXNqAMyMORKnNsIfTKwN/85jkQce5NO0VL2dKM/T4naJeRCVs74MVcrZRpCVujBxpbUsr+cHfcke5FtJHedZM8sNliJ9e9eGZko8LJK/cQbTmiL15VFkGqo6mqGEzdGr1ooKe6+a5N3P02abp70iagIyNqoFdx6XLmJggBe4hDrCf9li7xn90YAAuD7Ui14QuqYVjCLJdk60BfImO6p3kuOLNZKqoU5SgXUq8ZI5NlvKCRZQ2IlqEDKjnMV3x5C8f6t+4vqSsgR4DmFBeHXIJ49lyL0ipJTwGt6obBg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hsLJE/h2f3iBqJCSh/p9SqyDaJnSWoeY2MYMRp9SgIE=; b=dMtZIz2+VzWo0AZ/R1zgTPHh3Nne99eI4Pp137HEAi3I54/CSMU/0m0Sv0HD4xqSRXFk28/4GEiF48tYeyF90O0zJwO8jj0QZwW5fvZ2Nwh30dJFXzWDCU5cSzloWAwvloEzOlYCOogdTUB59Lti1tJ8TnI7dNMtEaAxoQrG2CcMUcqQzLYi41Q84k64vMQRlPwzWilcFuQaFP29giGBupRKum0tA3/QhkpfEVJ1sFnBOBBI1Se48w2tf4t+F/ZvvDkHkcRgaUfxEC2my8bvPZ5Q8rvVebhVo/+EoaLlN9+1Bk8879jelRjy13KnkivD11yqnm/g3VmEadZd3f9aqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hsLJE/h2f3iBqJCSh/p9SqyDaJnSWoeY2MYMRp9SgIE=; b=GN6uGhX0nDlSZjalEwMOu0CfPljZOg/pns9KC6kD1YebA7RJS87FAL3RTa0VPRR8mONbABnMl50az+FyYaw8rowVxjz+sPo4Ujc1iHtJjHHxKND9W1lXbm9d3IwWFVZOEdX4oL0/lZDojuD4FaWl4eQa39QYZc3Klkst42kFhKI= Received: from SN6PR05CA0020.namprd05.prod.outlook.com (2603:10b6:805:de::33) by SJ1PR12MB6290.namprd12.prod.outlook.com (2603:10b6:a03:457::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:31:54 +0000 Received: from SN1PEPF0002636C.namprd02.prod.outlook.com (2603:10b6:805:de:cafe::e3) by SN6PR05CA0020.outlook.office365.com (2603:10b6:805:de::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:31:54 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636C.mail.protection.outlook.com (10.167.241.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:31:53 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:42 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 05/13] mm/migration: Migrate accessed folios to toptier node Date: Wed, 19 Mar 2025 19:30:20 +0000 Message-ID: <20250319193028.29514-6-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636C:EE_|SJ1PR12MB6290:EE_ X-MS-Office365-Filtering-Correlation-Id: 7f498ec8-74af-44c1-3a40-08dd671cb460 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|36860700013|376014; X-Microsoft-Antispam-Message-Info: 9SwE2bFnse8n/39XTWVI1UMO2afQRlhIF8aCdF5uxb/WfCFINloTJRFd2dpON/Pnm32NEMNWd0XwaAEdpe2ReO5rgUb2eNhJ8zt7W3w03P7eAOYza+Pf1adpui2Kp5BtHYXpvi0tIYP5SIJLFv5p2CGp3dri6bWmrLecHARzhXFc0ayCICsFRkHWSYjsELwo8m6J5JoM2sFDoLfJ3RevXdA1NCq57uIYV271IoWa9UL77fD1w+rJij7FG6fx+jHFoyOKejtF80SBC4ZIgHKU6lqIsi2RL4x6+MyyJ34OgsrlTUv6ZyGSyF0b74b0OoJyh5ta46VR0tx8uee1i6pSf/wbozoPIQnO96hObqAHp5aFebuS0l5PrOdnKPW4zyDW1EZ12yks0CAiVS1VT5Fn79zu4axuKQHisG2o0jChdLhzK3T44RMTepeP2iiJdzz2DqH27/yZsGtbSX/pLnUc7zbgwN9mNTf1gds0gnehGeFZkYiEBAC1RbOZrfWHjRkX4odPYJpHSlzYRYtwZxMYkZ+Ln/mLrKyP1RbiBHM/uJma8Y1b239WRAwzs+eDs+/fNiTiDMrQZG8fHNciezaLvRaPYUGvXxup3bApBgyK39zX3XYg/PVQS2ok8j9qkgaAg/UM/kvjLmTh6AVMgmlpazHo7MQ0sBzGbV2sZbCGusaJwPCHJoXDZui0HIOGLNxE5i+eFntn2ijMP1JCp+d0mJ46Grjzwr4AhhbbBpnJtUjlgycHTSjfTqn8dslsTFunuk6SfkAoHMfg5Z0XKlcQmS8Zy8dV2+2BHkZj1USdulusZpGZm5tECrJMtTsTCIEKOKhcliw4610N70Yfxmsz5ZaPmGwPX+RmycQqATZsYrycJzqFWel31bxPgLLihF8SWr/3OWKxiAjF41pbQkXYZvs7QnOICs6ZVvJhIhKPrvxSXBGUTnwjxBV56D5gB8YoLc/yDwdro0yl46zxKnooyrOstJTvVok863/OKVbOIrHRi/B9zHO0WsZhUlXPDz6OxM1T77ht0FWMO3I0kKpKgHvVQ/7LxfZHfDSx+mO36iNnptkeY8WM1gYSGTwkms5G/hN+YwuK0iFoIcrkLbELnQtxeuLgqd9g+emPzJ8vvYMyPOgPpIwVcFmGFybBT6GuE2RWh1pVNz1BpO69d3lCWMrrK61T2Lo4bqn3A1JxKed8ObtvdcDY1QVyXNxJVaTgjX9UL/Eg1Ryg2Ncf5JVd5VPx9A3PCrISJBrtbNRuO/uwBa7l2q34mpyC+P5jdh9qCeX+r6PqqkPzoDtBSwz2dC5NPhsJXOfbC5VmRSgN4se0UYagG8kjwWtPUWbLcGdR+zYxa9SXbwTISPGfs8Buc4UfK1b5G6oz9+Adn4yp2+C6ejm5EhL/Hyt6PEen1WYH6ZIAs3D5/9dR3aF+NiA+krFbum/mO0oGLwZsHtsvj/jRLk3s8TVmwLDxj43aeoG825qGP+xzzpP9Spab8pgy8w== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(36860700013)(376014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:31:53.8803 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7f498ec8-74af-44c1-3a40-08dd671cb460 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636C.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ1PR12MB6290 X-Rspam-User: X-Rspamd-Queue-Id: 48DBCC001D X-Rspamd-Server: rspam03 X-Stat-Signature: matw3kd7gwxpweuznry5o7g17eqkno7y X-HE-Tag: 1742412721-208909 X-HE-Meta: U2FsdGVkX1/NbZWVnL1T2VuMD4Zl9GPg6pkp0L++kL0t31IcTD1Y1JPyZXCbr7MWqrPjatSbLhvvuZKkHGVvYMrWGvFrSM836dk6K5TwWfsPUI7H3OnkRCduLG6okX4RKEdiL2h1nX19S5C3wioIrocJQ3R9stzBk1d9c5DCeVaCpBOiU4Q2uaZZYXRT2kvVNBwbuIcDtchFtddlUH6shuyQ0QlCfXnZw7HuA37Faq6YrhMMbfticP1muvOfTFZuSNqDYchzc6ueLdK4epFEGzAeC/EPL2rflbzqHNldm4Y2pAtCVi4bnCDyIhSPo+G4B7+S5ts4vIE77h/BLH0cK+QbT2AtaCB+/ZrVCVIKIN1ktDFvUZKHBiZIUsCimOvHa0XU5Xx4eBSScGGNBYxxlfnlM2rxDRQ78adxOlPnvjfCbKB19SeCcXxVF9EHIPLXhimrtIBsyqP8gM14raaCHXvm4AYgBjTGQdNGT/Go3NSboyiQLyH+AhZT3vefa4BOdAqozmSrcYEIOqP/nBDnQS1i9EnJKRoB8NtRDa793VgNWZrm9wjy+H/SOVVfhQnCN+T/ga/TiP03l54bmmT9t8zBgFD874r9eu/pwgnH/QFnuu8xtgwG4meAKkHi86SlqDKJroU7gHeCAn3R4sm5CEskahbl+LtOYFPICgBV1SmMXofOZ4Ggjp8/6Iks62VAs6FQBT15c9yifmGH2/VJEcsd3vPxsXu/1Yvu9I2Ffnx5otBrOZILN7PV6Jpfw2NZXchidjEv3lUMrN1YNfdriR7FqFlUmUh3uETfr4ZxxidIyvqmnU4VFL9DG+IMgkQuUezCUWrVmcRjWz6JmgzQ8NM4AzglH+Kd5vpTOTLOTQnTIVP8Rlxfl08vDEkN7N/mEL5VtlSTOOAWm7Ww9k/Mlq6TN4pChVI/kGJKIxN297KoW8fLSuPHwWhargZDV31lVvKaW7GbkDxJjRGZOcD mhCpXLp3 iWJAFlA/t4VDpQV32yiQug6mzuU+Jj3NQGGZYoTV2bebqQqKM8d9vgtnsqmSVRtY/5roBeHW2V471j/pIlJGmS1KAXzWwVpl6dVxqxxsJ9deeNt2PEkAM9Qtq4X69TKD3UHYMgZRkxBIwrwGyK3VZQaI0qxd5/4QFw7Lu9JM9HskcL00LM6zbLzzZ8HObcJXiyrmDS0EMXj21q/78dd1jS6WeeJUYXzdzb5fEVVuND0kP4j6lmTe5jLHzLnayJ1QcAIzsjjv2coaLiUL+3F1k9dh8fnWig89Qo3j2dEMC4BueVmL1V/l6I3bCMYbRNbjZ52AVNpLrIFakDJrDQM7eUyMnmR8QxjUbqnPKg5RgNDZTSFHNAIpMsVZvK1DanDG/KZ7hhDjsXHHpvMCIdw39KEv+7e597Zp6GCtkCSU75EACiCeFTTQcrrRneqntvgmP8R7w5q2VzGgPR9nqBlydYpTgXu18Bnq82+0qRNlPwhJ0WJNmw3d8i5S85CEORRXPu/MeOCDUQsANw8ndOAVZ4VThATNi7Om296ZWj37VQtZpilOJMndvivn3Y4p7AH0WEeyf8vQ0wcgHa/zS+zgvVR51KFsKnKgmx+2lWRsGvYztGE6j/5d46/2FHo2DeiapyBJxG8OXaWgEqKG9PBSxWbS16po+mqmCcVdM02Pa3mfTUkcYyFhqjr6i9r4o4bkyAg5wMiMRCcPdsj7AeUIIoQLAcwWYd8hTcKTm0OGUwqVqu4b2O8sJ0tvqFZ5Yxvg5Fhfo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For each recently accessed slowtier folio in the migration list: - Isolate LRU pages - Migrate to a regular node. The rationale behind whole migration is to speedup the access to recently accessed pages. Currently, PTE A bit scanning approach lacks information about exact destination node to migrate to. Reason: PROT_NONE hint fault based scanning is done in a process context. Here when the fault occurs, source CPU of the fault associated task is known. Time of page access is also accurate. With the lack of above information, migration is done to node 0 by default. Signed-off-by: Raghavendra K T --- include/linux/migrate.h | 2 + mm/kmmscand.c | 209 ++++++++++++++++++++++++++++++++++++++++ mm/migrate.c | 2 +- 3 files changed, 212 insertions(+), 1 deletion(-) PS: Later in the patch a simple heuristic is used to find the target node diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 29919faea2f1..22abae80cbb7 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -142,6 +142,8 @@ const struct movable_operations *page_movable_ops(struct page *page) } #ifdef CONFIG_NUMA_BALANCING +bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); int migrate_misplaced_folio_prepare(struct folio *folio, struct vm_area_struct *vma, int node); int migrate_misplaced_folio(struct folio *folio, int node); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 6e96cfab5b85..feca775d0191 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -59,6 +59,8 @@ static struct mm_struct *kmmscand_cur_migrate_mm; static bool kmmscand_migration_list_dirty; static unsigned long kmmscand_sleep_expire; +#define KMMSCAND_DEFAULT_TARGET_NODE (0) +static int kmmscand_target_node = KMMSCAND_DEFAULT_TARGET_NODE; static DEFINE_SPINLOCK(kmmscand_mm_lock); static DEFINE_SPINLOCK(kmmscand_migrate_lock); @@ -182,6 +184,76 @@ static void kmmmigrated_wait_work(void) migrate_sleep_jiffies); } +/* + * Do not know what info to pass in the future to make + * decision on taget node. Keep it void * now. + */ +static int kmmscand_get_target_node(void *data) +{ + return kmmscand_target_node; +} + +extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, + unsigned long nr_migrate_pages); + +/*XXX: Taken from migrate.c to avoid NUMAB mode=2 and NULL vma checks*/ +static int kmmscand_migrate_misplaced_folio_prepare(struct folio *folio, + struct vm_area_struct *vma, int node) +{ + int nr_pages = folio_nr_pages(folio); + pg_data_t *pgdat = NODE_DATA(node); + + if (folio_is_file_lru(folio)) { + /* + * Do not migrate file folios that are mapped in multiple + * processes with execute permissions as they are probably + * shared libraries. + * + * See folio_likely_mapped_shared() on possible imprecision + * when we cannot easily detect if a folio is shared. + */ + if (vma && (vma->vm_flags & VM_EXEC) && + folio_likely_mapped_shared(folio)) + return -EACCES; + /* + * Do not migrate dirty folios as not all filesystems can move + * dirty folios in MIGRATE_ASYNC mode which is a waste of + * cycles. + */ + if (folio_test_dirty(folio)) + return -EAGAIN; + } + + /* Avoid migrating to a node that is nearly full */ + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + for (z = pgdat->nr_zones - 1; z >= 0; z--) { + if (managed_zone(pgdat->node_zones + z)) + break; + } + + /* + * If there are no managed zones, it should not proceed + * further. + */ + if (z < 0) + return -EAGAIN; + + wakeup_kswapd(pgdat->node_zones + z, 0, + folio_order(folio), ZONE_MOVABLE); + return -EAGAIN; + } + + if (!folio_isolate_lru(folio)) + return -EAGAIN; + + node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio), + nr_pages); + + return 0; +} + static inline bool is_valid_folio(struct folio *folio) { if (!folio || folio_test_unevictable(folio) || !folio_mapped(folio) || @@ -191,6 +263,101 @@ static inline bool is_valid_folio(struct folio *folio) return true; } +enum kmmscand_migration_err { + KMMSCAND_NULL_MM = 1, + KMMSCAND_EXITING_MM, + KMMSCAND_INVALID_FOLIO, + KMMSCAND_NONLRU_FOLIO, + KMMSCAND_INELIGIBLE_SRC_NODE, + KMMSCAND_SAME_SRC_DEST_NODE, + KMMSCAND_PTE_NOT_PRESENT, + KMMSCAND_PMD_NOT_PRESENT, + KMMSCAND_NO_PTE_OFFSET_MAP_LOCK, + KMMSCAND_LRU_ISOLATION_ERR, +}; + +static int kmmscand_promote_folio(struct kmmscand_migrate_info *info, int destnid) +{ + unsigned long pfn; + unsigned long address; + struct page *page; + struct folio *folio; + int ret; + struct mm_struct *mm; + pmd_t *pmd; + pte_t *pte; + spinlock_t *ptl; + pmd_t pmde; + int srcnid; + + if (info->mm == NULL) + return KMMSCAND_NULL_MM; + + if (info->mm == READ_ONCE(kmmscand_cur_migrate_mm) && + READ_ONCE(kmmscand_migration_list_dirty)) { + WARN_ON_ONCE(mm); + return KMMSCAND_EXITING_MM; + } + + mm = info->mm; + folio = info->folio; + + /* Check again if the folio is really valid now */ + if (folio) { + pfn = folio_pfn(folio); + page = pfn_to_online_page(pfn); + } + + if (!page || PageTail(page) || !is_valid_folio(folio)) + return KMMSCAND_INVALID_FOLIO; + + if (!folio_test_lru(folio)) + return KMMSCAND_NONLRU_FOLIO; + + folio_get(folio); + + srcnid = folio_nid(folio); + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) { + folio_put(folio); + return KMMSCAND_INELIGIBLE_SRC_NODE; + } + + /* Also happen when it is already migrated */ + if (srcnid == destnid) { + folio_put(folio); + return KMMSCAND_SAME_SRC_DEST_NODE; + } + address = info->address; + pmd = pmd_off(mm, address); + pmde = pmdp_get(pmd); + + if (!pmd_present(pmde)) { + folio_put(folio); + return KMMSCAND_PMD_NOT_PRESENT; + } + + pte = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!pte) { + folio_put(folio); + WARN_ON_ONCE(!pte); + return KMMSCAND_NO_PTE_OFFSET_MAP_LOCK; + } + + ret = kmmscand_migrate_misplaced_folio_prepare(folio, NULL, destnid); + if (ret) { + folio_put(folio); + pte_unmap_unlock(pte, ptl); + return KMMSCAND_LRU_ISOLATION_ERR; + } + + folio_put(folio); + pte_unmap_unlock(pte, ptl); + + return migrate_misplaced_folio(folio, destnid); +} + static bool folio_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, @@ -379,6 +546,48 @@ static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) static void kmmscand_migrate_folio(void) { + int ret = 0, dest = -1; + struct kmmscand_migrate_info *info, *tmp; + + spin_lock(&kmmscand_migrate_lock); + + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (READ_ONCE(kmmscand_migration_list_dirty)) { + kmmscand_migration_list_dirty = false; + list_del(&info->migrate_node); + /* + * Do not try to migrate this entry because mm might have + * vanished underneath. + */ + kfree(info); + spin_unlock(&kmmscand_migrate_lock); + goto dirty_list_handled; + } + + list_del(&info->migrate_node); + /* Note down the mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, info->mm); + spin_unlock(&kmmscand_migrate_lock); + + if (info->mm) { + dest = kmmscand_get_target_node(NULL); + ret = kmmscand_promote_folio(info, dest); + } + + kfree(info); + + spin_lock(&kmmscand_migrate_lock); + /* Reset mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, NULL); + spin_unlock(&kmmscand_migrate_lock); +dirty_list_handled: + cond_resched(); + spin_lock(&kmmscand_migrate_lock); + } + } + spin_unlock(&kmmscand_migrate_lock); } static unsigned long kmmscand_scan_mm_slot(void) diff --git a/mm/migrate.c b/mm/migrate.c index fb19a18892c8..a073eb6c5009 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2598,7 +2598,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages, * Returns true if this is a safe migration target node for misplaced NUMA * pages. Currently it only checks the watermarks which is crude. */ -static bool migrate_balanced_pgdat(struct pglist_data *pgdat, +bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages) { int z; From patchwork Wed Mar 19 19:30:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023074 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83BB5C35FFB for ; Wed, 19 Mar 2025 19:32:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A40B3280013; Wed, 19 Mar 2025 15:32:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C951280010; Wed, 19 Mar 2025 15:32:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F59B280013; Wed, 19 Mar 2025 15:32:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 61374280010 for ; Wed, 19 Mar 2025 15:32:11 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 396191C88F5 for ; Wed, 19 Mar 2025 19:32:12 +0000 (UTC) X-FDA: 83239296504.29.60E0597 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2077.outbound.protection.outlook.com [40.107.220.77]) by imf18.hostedemail.com (Postfix) with ESMTP id E8FF31C001D for ; Wed, 19 Mar 2025 19:32:08 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=sa7knc7K; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf18.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.77 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4wGhb7LYtxUiwXUG0HCYGUwtr/i49pZpHiAdkz0j43g=; b=hsjPSQMQaoXF7jjmg96VceC3uFH/oFzm617ZbjhK5YJQmmV+13d5haf19CdRMWSBj7GkKb Z/fQFZuN8wUVX6ZDrW5ndJCBivzUddv8ojUSLp2+192RiZ/UdER43DZ7MgDjSKKa4jOOU6 6XgmDTQEjFaW862zhPPzWDPLNFz6wEI= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412729; a=rsa-sha256; cv=pass; b=6MC3585pBYJoE9JP2pG3Lp4SqpZhCyF0WWB+HvAnL2RyRK7CZug6AH78307mdZDMRQVSZ0 62hODGVar6FWYFbi6cHTAqk7AATizOJuaVJ3FKjxSqV3pnXZy3pd2GGAq1EAvJxZCy8ccU fOvjaveoKIyfu9jE2qwwj5Otp1xOZjo= ARC-Authentication-Results: i=2; imf18.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=sa7knc7K; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf18.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.77 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ec0JQ5oTyYwgXEw+UGkjBJk8LeoTOfpWlqoggoTT9BJdYZyUleAU5GOsjqlnq+q29BGAW7be9IsxfcwsvAzT30rbrasSungoUvFusfzZB3OwQdjJX11cBGuvRUafJ1r19O93eOcflrWqpWNOCcAQSGvrAbz1zrY+qBZ3KQ19hvlTrE1lBMTHBly7ADA+gu/9NRfO7BJkgKCGP/o0W05ymuPPIyKxJers+GynWtNr+iKGKiTpGEedgu/W865YET31Pz+kQkjn+obxkbDBXkIfpsPXjthRD0HsV/LL33CpI8B+0vqIgIYjju/Te2Jud3C49SpBOvlsXoyaF1CGIqtRMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4wGhb7LYtxUiwXUG0HCYGUwtr/i49pZpHiAdkz0j43g=; b=LKqQSltTEfcAvDtNKc2gmntGYa/Z8RWPKgrEkNhw8ye0SLOjFW7IIORd4n3dPF2xHpfukvH+5OlR5l4DtxUTzISiVSZkarMQ1n8MY3i8Qlsw7uyZKsULUTEXY89DpmcIeEerWJfS/H4C8u8WK7Q2tSADVwij47zoAxhbHcof6talKy+PYbxjgiX4GRmuXebWhkFy4fCfDcQ9OIel1wOwStA1k1wg/AO3lGeNPIcIvAK8UPnTSZ/cBryvJ9parg4rSQNZJALxbnS8dANxFBrz3UGqUJuFWIVfIi/qR6IMjDTBDcYgVltWAbAWK5fZ+9+XKnfL4URteT6JgKX52bOBLA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4wGhb7LYtxUiwXUG0HCYGUwtr/i49pZpHiAdkz0j43g=; b=sa7knc7KDjngonBEJXlqBh6UWjTsT87vDOGx9x/GK9LEtwCW3XN4QoGNKjmW062H0cbNtA2WLNFHx0l5CpE783xKbngoVJOIx2H6/KtP5lZDhJWtL+qJGLYO3sB7qUPEEOeIRmqOmPdPXW6wXq/zxBlXYQhMmAJw5Mj9Ex+UEBs= Received: from SA9PR13CA0118.namprd13.prod.outlook.com (2603:10b6:806:24::33) by SN7PR12MB7370.namprd12.prod.outlook.com (2603:10b6:806:299::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:32:02 +0000 Received: from SN1PEPF0002636D.namprd02.prod.outlook.com (2603:10b6:806:24:cafe::21) by SA9PR13CA0118.outlook.office365.com (2603:10b6:806:24::33) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:32:02 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636D.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:01 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:31:52 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 06/13] mm: Add throttling of mm scanning using scan_period Date: Wed, 19 Mar 2025 19:30:21 +0000 Message-ID: <20250319193028.29514-7-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636D:EE_|SN7PR12MB7370:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d38bf76-7eaa-4fd3-927a-08dd671cb923 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: EZMw0dXOIE1FJAwUi3ajEr3mFpLDxIn4VgVVCm5R4kyu/DrBTEZyjMekcC2bhQAEauWrNTLX2108tSbLMDDAaMD/AcjFCB2hmRdnctwE9UeyAsaUgOHxq8OG8NAEkJ2UlSbmU3lVFKzD0hv9WqBDX4KRg+VA5LfYiVV3HL5saMlDUHW6+Ir8/boPiGN6NI92qriPAEavNWrYnvZrb6rlSXw86cABtkmxaNKcdugbvcGtuah8GNKz54tyUNvtOcRNk9ha5WgcTc9f1A0Zy2U6Z+CQq+Sm98V5Ks+CQmyRQ9BGZFWUEAiiYSit2oYBJEbQMqH+O3RV0XZVE/pW+B9pPTD9nF15wNodahVpbDAFxEDFG4Be29ir0NJ9VYHgPcS4BaXCxEOWgVAS2zw5wVTsVk2tq6pjALwd7Qpnf8NQVQ41kITpMm5Rd0jx4bG87lo1QToTvR56b82DOxFFhNDCe9WrCbN8qH2D9CuTifXTglOr2/cPalvYswuybkevKY4EoL2U22YWgnmsRxOVj4fSXxPtBDzZaGg5M3GAEimkSwRQCNmui6UlvyFrZ1yU+fGbdAPQl8Grh0iJtwzoE4xKcxmQ0IhMXlMhdC0RMBLLVuiANTvd3i/JncY/8dfS+lwjKgcc0N5aKur5fh3czrxraHN7zE3G+KVFoeI5QH3UltTDd1Zuha1bnziDN/3UAWabbX2fkf/W7/MGRVMf8z/RxPA0s9Faas5+fYyFzY8xhUhsC5mLHg+fZIUIVlpYSyzlrExj3v5SYQqOktpruj7RXclvFfmzmifmiko1mnONk4DSJSOOYtu46jOlYP8ZtXL5Mh60R7dQyqhDLjP9YbUvLzalXv1VjTwuSeuxpGfe62Q+b5pIN7zsLOjosUwEui65ThZaxcXvQY9AZohdePwr71eJL1O1iWOTDYVUb6p5y2Y8i47CcFmq/bUcimXfX9db14aoHo1ByqeSBX32NTh9rTqFO5x8anVp00+heLvh2nSHBP2F/JlBzQc0d9f3cN8tgfxGC37nMIUXrwegakiGRU6ljaBkmBAr5sAyYP5UvxWDQIKLKfauHgyoZXrw6L6rw7W2PygQrQ6fG4ovFqhsnKHtZ3uVXljIfoL+q8/P0SSBZ+cfKM3rIUja3uJtpXoMCzkL3mE67a6blvFu3JwttdMBJs6taw5ki8cVgsdtQMESUVcB82OPvr2VrnbVfwag5y4KqvJ2pxgHkfzjxdPeWWu+AfaII19kkTO/K4LwCgxZgvfD7GYwvAyNF7s23Z/wDzqxRkO+DXRijRiGjikOcE/QgnXifpDHJivBhp2p1TLIq4I2tWXAgoxReMC6C0KqfytzWxMKawRYTll/hcN44oavhwSr3osQXv37mEyI81ZNmq2fQYZuj53zJF7Hnb54d7BSnnwrXTBwTGXZmBEH2r4GiD1Vc++lLTu0MHHFao4aH58AqrBtkZy4IJiPke5NdGGXNTdNiuh2Z2LplMjk+GzjM4GgsrAFvy8KElPsfms= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:01.8714 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d38bf76-7eaa-4fd3-927a-08dd671cb923 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7370 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E8FF31C001D X-Stat-Signature: no83bp3h9oifqdw74zi3gni74fjk9zww X-HE-Tag: 1742412728-709658 X-HE-Meta: U2FsdGVkX1+8QqMedhbh6o/IU0adOkGV9C3W+aj+M7rmrQl8GcpubSlKrYJdObZtKSptxGk0TnEqwIbZIBtNCvsSDEOBpufK7AHnQVhQpvwYNhgQhV8Lc75a9pXZfaLhXZGDauSYj883KL4cgvf2A518qljZUUsggR08uiqxz2hH5q/ggSj2AX653KIQJGdI8sb0aKGoahYFYf66dOZe0qKbfl8bA7EiVCiqWfyminpzD2DiW+BpBI16Ji1opCwrieSq2d7IH4UcW3YG47uiU5n0i/cDzZOevCuA2CgedGMN80Il4UE/l/PxzvRKFLgwgAjlsLXd/gRiK+YomqaH5G51K2sIFJBRdCatPKGCVffF8XJNCFrE+j+Qw+vkd0TEPKz3ux+cv7AgUgJyRg0CRKhSV98C6yAYjLDCWOM/HNgA89EVoGo8usYGaItaULOUuzfLOKn6N4jQ7dXYNyZ+AmBu+Fs1/j1jwGFYeC4pp1qQKBF+DK/M9pm82jH7vB8tKDav+5c7VtDa4rnyXkMFgWv7wWZFQzIF6mk8mJ6xdE2WVrkOe72c7fWZOwS4Lu/LIy9MQCOV8LHj3yAwXn0DbY2e2NEnglVedRSrkl4LPOSQapfBDOUHT8vJz9Ra5olUHE+WrR881V+s8s4cjPFzaiaBBFjTlyZycsrpJY6BzP3XeY6Myh5tD3qIkJJwzL3tmm1UZe3FyKE978UNj4QHIPN7YwDXuWqAZAgD4IHRb7R2ejb8VN4sC93rQQc9DRW8tDaLhui2dlg+Olt6hBW/U6/GpiAbczlbfdgS9DAcc0bJvAl8sSNVRsO1w+dwTrOpLp9nMN9ny+lm2orA8u+uVI0uMc4CGjDmy9Z6nptGHI4/bsIiyYzPOC2XwhUfblQhdzmHkTQdiYtB9Vb+FM9WiPvAE6EVY+9nctcpDs1K2o7x85nsKPSi9OOBjq0e51j8u7oR57ZfpNgOCiy5e/T uV5Aam9x gfruM+L+Hf2kHEZotFwPS4q7jIrtB6BXx1eUnI3w6YWmu64svf/uBO7/VjMgVQ/AOxh7gqH0ZOHOHwN4/60fpfD/fOjYD+5Ntn9kd12JPXqY+OBPC0uEK3K1M1a7aYN/PspZEquCjlWAQkn6A0OgkImgvDLVKT69h47A4+wINUrFD1hPHFPopF9/2OAU92isfjQ9evweZiQpjD8Dza+6x0o1G7s4qBiRK0AFmAXIw4VB0gJXaOsrybbO/+Nh9Q4uGggqQpP+9Pjc+I2/Xdn3xCqpkSakKovpgzqvbIsvb3a6U8uiOQVNHkbUH6JRyIBSX+tfhMd5WAAq/daVuzAxkab4N3fkExw8X47F7DSS2nkXCeTfsLfWVuIcYMA6x6e+iibqUhRR7P/WCzCih4p5cNkO0S8sl5eHikxRlXpjg+8CWQbgc1U5bJFPwhoOJQ3I3t0Dol2ypBjg2ssieCW9coQcOT3S+R1uB5U0kf2qJTc0uHnoAbZNO8wmgD3t9Ug1l3+48t1x/vUCNMaTKzbkMl3zc7Zby+wyNSFLnFIyxn133RPMtAD7kLwHh5191VWXBpou+LAd6JiwRguOKV5OJEiVsHc+ZYLm2JODW1//9OopAFdGu2AGw2KPbvAjLXZI1uLD67piooipnqonahLmjmkTTKd4O6ifB4QEkL+Z6ApkQ+LTaI8xkYdlvozlYH/jUl/KZBZY4vn2JNwIQUQ055txD5uG6bBM3WaA0YEnyuvDDRsaFavK9HCB2YfFs7bJg2OIKczrsKCHcRZjz1ahNROrKtkNSb4e1hTB+slCFjS4LQ1rJ2Vg8SiVHkg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Before this patch, scanning of tasks' mm is done continuously and also at the same rate. Improve that by adding a throttling logic: 1) If there were useful pages found during last scan and current scan, decrease the scan_period (to increase scan rate) by TUNE_PERCENT (15%). 2) If there were no useful pages found in last scan, and there are candidate migration pages in the current scan decrease the scan_period aggressively by 2 power SCAN_CHANGE_SCALE (2^3 = 8 now). Vice versa is done for the reverse case. Scan period is clamped between MIN (500ms) and MAX (5sec). Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 1 deletion(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index feca775d0191..cd2215f2e00e 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include "internal.h" @@ -33,6 +34,16 @@ static DEFINE_MUTEX(kmmscand_mutex); #define KMMSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) static unsigned long kmmscand_scan_size __read_mostly = KMMSCAND_SCAN_SIZE; +/* + * Scan period for each mm. + * Min: 500ms default: 2sec Max: 5sec + */ +#define KMMSCAND_SCAN_PERIOD_MAX 5000U +#define KMMSCAND_SCAN_PERIOD_MIN 500U +#define KMMSCAND_SCAN_PERIOD 2000U + +static unsigned int kmmscand_mm_scan_period_ms __read_mostly = KMMSCAND_SCAN_PERIOD; + /* How long to pause between two scan and migration cycle */ static unsigned int kmmscand_scan_sleep_ms __read_mostly = 16; @@ -74,6 +85,11 @@ static struct kmem_cache *kmmscand_slot_cache __read_mostly; /* Per mm information collected to control VMA scanning */ struct kmmscand_mm_slot { struct mm_slot slot; + /* Unit: ms. Determines how aften mm scan should happen. */ + unsigned int scan_period; + unsigned long next_scan; + /* Tracks how many useful pages obtained for migration in the last scan */ + unsigned long scan_delta; long address; bool is_scanned; }; @@ -590,13 +606,92 @@ static void kmmscand_migrate_folio(void) spin_unlock(&kmmscand_migrate_lock); } +/* + * This is the normal change percentage when old and new delta remain same. + * i.e., either both positive or both zero. + */ +#define SCAN_PERIOD_TUNE_PERCENT 15 + +/* This is to change the scan_period aggressively when deltas are different */ +#define SCAN_PERIOD_CHANGE_SCALE 3 +/* + * XXX: Hack to prevent unmigrated pages coming again and again while scanning. + * Actual fix needs to identify the type of unmigrated pages OR consider migration + * failures in next scan. + */ +#define KMMSCAND_IGNORE_SCAN_THR 256 + +/* Maintains stability of scan_period by decaying last time accessed pages */ +#define SCAN_DECAY_SHIFT 4 +/* + * X : Number of useful pages in the last scan. + * Y : Number of useful pages found in current scan. + * Tuning scan_period: + * Initial scan_period is 2s. + * case 1: (X = 0, Y = 0) + * Increase scan_period by SCAN_PERIOD_TUNE_PERCENT. + * case 2: (X = 0, Y > 0) + * Decrease scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 3: (X > 0, Y = 0 ) + * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). + * case 4: (X > 0, Y > 0) + * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + */ +static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm_slot, + unsigned long total) +{ + unsigned int scan_period; + unsigned long now; + unsigned long old_scan_delta; + + scan_period = mm_slot->scan_period; + old_scan_delta = mm_slot->scan_delta; + + /* decay old value */ + total = (old_scan_delta >> SCAN_DECAY_SHIFT) + total; + + /* XXX: Hack to get rid of continuously failing/unmigrateable pages */ + if (total < KMMSCAND_IGNORE_SCAN_THR) + total = 0; + + /* + * case 1: old_scan_delta and new delta are similar, (slow) TUNE_PERCENT used. + * case 2: old_scan_delta and new delta are different. (fast) CHANGE_SCALE used. + * TBD: + * 1. Further tune scan_period based on delta between last and current scan delta. + * 2. Optimize calculation + */ + if (!old_scan_delta && !total) { + scan_period = (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /= 100; + } else if (old_scan_delta && total) { + scan_period = (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; + scan_period /= 100; + } else if (old_scan_delta && !total) { + scan_period = scan_period << SCAN_PERIOD_CHANGE_SCALE; + } else { + scan_period = scan_period >> SCAN_PERIOD_CHANGE_SCALE; + } + + scan_period = clamp(scan_period, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCAN_PERIOD_MAX); + + now = jiffies; + mm_slot->next_scan = now + msecs_to_jiffies(scan_period); + mm_slot->scan_period = scan_period; + mm_slot->scan_delta = total; +} + static unsigned long kmmscand_scan_mm_slot(void) { bool next_mm = false; bool update_mmslot_info = false; + unsigned int mm_slot_scan_period; + unsigned long now; + unsigned long mm_slot_next_scan; unsigned long vma_scanned_size = 0; unsigned long address; + unsigned long total = 0; struct mm_slot *slot; struct mm_struct *mm; @@ -620,6 +715,8 @@ static unsigned long kmmscand_scan_mm_slot(void) mm = slot->mm; mm_slot->is_scanned = true; + mm_slot_next_scan = mm_slot->next_scan; + mm_slot_scan_period = mm_slot->scan_period; spin_unlock(&kmmscand_mm_lock); if (unlikely(!mmap_read_trylock(mm))) @@ -630,6 +727,11 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } + now = jiffies; + + if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) + goto outerloop; + VMA_ITERATOR(vmi, mm, address); for_each_vma(vmi, vma) { @@ -658,8 +760,10 @@ static unsigned long kmmscand_scan_mm_slot(void) update_mmslot_info = true; - if (update_mmslot_info) + if (update_mmslot_info) { mm_slot->address = address; + kmmscand_update_mmslot_info(mm_slot, total); + } outerloop: /* exit_mmap will destroy ptes after this */ @@ -759,6 +863,10 @@ void __kmmscand_enter(struct mm_struct *mm) return; kmmscand_slot->address = 0; + kmmscand_slot->scan_period = kmmscand_mm_scan_period_ms; + kmmscand_slot->next_scan = 0; + kmmscand_slot->scan_delta = 0; + slot = &kmmscand_slot->slot; spin_lock(&kmmscand_mm_lock); From patchwork Wed Mar 19 19:30:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39321C35FFA for ; Wed, 19 Mar 2025 19:32:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64E02280014; Wed, 19 Mar 2025 15:32:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D848280010; Wed, 19 Mar 2025 15:32:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42879280014; Wed, 19 Mar 2025 15:32:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 22C93280010 for ; Wed, 19 Mar 2025 15:32:22 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 33B82B7449 for ; Wed, 19 Mar 2025 19:32:23 +0000 (UTC) X-FDA: 83239296966.27.7075C1B Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2047.outbound.protection.outlook.com [40.107.92.47]) by imf15.hostedemail.com (Postfix) with ESMTP id 36ACBA000C for ; Wed, 19 Mar 2025 19:32:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=HjF3FpjC; spf=pass (imf15.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.92.47 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=y8Xj39u2uR4MqPXlurtQNALFnpUi3B6OTCfu53Izm5A=; b=EAOUmhrwK0dpaXMX13r9QiK8/OPdPmrZuUMdW9IyLL9v+7Oi1vSJOuoMJiuN7N+4XoqPK9 AuwgODk0GlJ4WBofL74ALAav9Dxbb+sgcHSu6Eqj60bd4M9kq6/2LfvuDwlIrVilIamysq YNWWiamkPbYVg1tNGteD1AGBAo9HaKE= ARC-Authentication-Results: i=2; imf15.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=HjF3FpjC; spf=pass (imf15.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.92.47 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412740; a=rsa-sha256; cv=pass; b=NzkGySlTF9kWAVKPx95visDpdGWBkQJuwFCMwr8w6XsR+fhGxWEzLKN4Mu4//KpAEp0J6r rfJLxzvPJyY7HaEWJV9LX6UQaMJ2HWJtg+FhvLNOXrA6OkP/LstUGdnFi87/O2+Rzg4r05 x+qgbV0uugyhTyMI5gWtklVhZhKjVDE= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=jaeMGZIzpn/7NaOLFXmIqLlnj30KRtggkgoqEGWkbp7KD6g8h8hqfMUTn1rdbgmkOTr56/O2HJ5gqIOhZdwM3g5nElHPIHBKvoB+pES9zRpyro0864q3ORf4fQ+Aymf8sAV365D11B8P+whJR57nGQhWOSYTJDeo04SsQrKyDlK2oFI1E6wPwdBYvebwb5BG/L6MwL79yt2JlQEwOBPvdTlpLnQm1vRiX+VxRW/EAq9SEHXVkLIAQKktofk95BHczXTT0ZC7RkzMFTy8BNIFPfAdGiHY0muaDqzozGpKbHlZOZMwf1Ucg9MLbn0DlezuQZ8zXOGxM8DV7kN3PH8djA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=y8Xj39u2uR4MqPXlurtQNALFnpUi3B6OTCfu53Izm5A=; b=VhPuzF8/nMphf4cUbXtwTEdhhkYcZrWifpoXINhkMmW2I32nNZMK+P1h0lvPI4Ba2QGuJv5Q+eO0soT2/kfiNLi0y5tbXgSuNUmo85RTdAdty67QEBoG/WqcR8+E9hz42ZMC28gPPAi+wQGTxCGjPVjUmtxAwkD8ezGXTOB3t4ZE3cyal0Bl5ZzjNB2ijl3zGuJ4OaT0GasO+KQqx+e8EXi/r9v1sXAJDwc5IOi6ndUasBda/SITr+kz4NskDyGgZEjuhPPIa5MLWHWJcWK5ZsbNQOcJRw7wNLohbSUTMw8SHgoh0hP8eZeo5bF83kMNWdNGpfCNDJ/OSvSdbBjHIQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=y8Xj39u2uR4MqPXlurtQNALFnpUi3B6OTCfu53Izm5A=; b=HjF3FpjCdG2sJ45ngskjt6Apo3L+DGDIei8LJ2LHRqQXtiPNsFBVvJv3vma38Wiv8EvOI0pbP1GuxxnfwOkMDhRGQMvrBlxb1rD8ukRTTzC1LRp46xJuATaeioq/02JcItv8b+McwhguuP442Jv5YJn8QBq0cmeJuRuzelLXoCY= Received: from SA1P222CA0080.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:35e::17) by SJ0PR12MB6805.namprd12.prod.outlook.com (2603:10b6:a03:44f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:12 +0000 Received: from SN1PEPF00026368.namprd02.prod.outlook.com (2603:10b6:806:35e:cafe::9f) by SA1P222CA0080.outlook.office365.com (2603:10b6:806:35e::17) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.25 via Frontend Transport; Wed, 19 Mar 2025 19:32:12 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026368.mail.protection.outlook.com (10.167.241.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:11 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:02 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 07/13] mm: Add throttling of mm scanning using scan_size Date: Wed, 19 Mar 2025 19:30:22 +0000 Message-ID: <20250319193028.29514-8-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026368:EE_|SJ0PR12MB6805:EE_ X-MS-Office365-Filtering-Correlation-Id: 32265f0b-0c30-4522-e16b-08dd671cbf18 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|82310400026|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: Em3rmXGsOsYvfggcI43NOGCql+BeQdIqHmAARBsb3q/umZaXNEy0RE5QY7JzisqVLi487UNMc+lTORKpjGQRuZfM1+lyfEPkgMQmI73N31nc3+gsb9X/k9dctLYN5/eqy11WaQtHBOuHBo6YzbUs6nqtXf7bh2ICiTmfONwAvzrXFol9oc4qtEitkKAx1O9vZptgNuI2qW502hyt9yY7A8NwKAN8Q5fYPqjfXRxPzI44zU+s7YAXhuk3E9RMSEpZzMwPXiZzfCyXcXQmr/apMS0HYrK0uLoM7TeZ5hGGmFBa72grW7GNtJ1up5gxCzuTTRAjp4jMC7hT/ZeIDuoHnR8DbpIu6VFs5HuY+oSJ1SmZIz32S0cfQt1TJgXPt/l8p/Hm4FzLh/PZEGaNH95+mOfZqHHXbh1QZckuenm5zu7EKPY0RiLJUNMISt08dw/JY6GUVapW2aj9VEDZZnvWLdWpZO2Ek22mXbK0hgvvtOzOxYTL8S05qrGEaKOPFbELn1OE2ZAc/831P94O8ngNRzmuYXPDKg4h2+9cPwFXGce7SaIbum8CYKEc9p4oJv90KqLxdbrrB+hNdmPXqMCB2dPDJezkjAJn5y/Q393QMFNzOk9TGO9lqnpJi9fg5XXdb9cryXkoaxbMnrIVGTuo49wibWqSXLw2Pj5b2Kwg8D0MIiYXCh+9prJXqgdziEv2PXSbRSJMmpR+mdfWuXF2vkA2gomM4HbeSVWwzZ69e07cS8Q03G6lhOptV0Rt84NluS4lFcMu4Fb+cwQiqBXkHcLMu5FBelhainN7fPvdbLzasDFKgMRD77Bt58PWHBDCv07ZpYHLULUOwLIEOQiYP8xpgFb/BwVsYQAx9Y6eLBtNPX9sn1+EI+PkB1IH4cnVAZBJ/80ut0QuHGJOTGuIiZgGiUh+9KfzfT0e8fCkRFiewMKORQ8rtCNOTWLbaeSVXVoTJ7+56SUQP0y1QOn5U5/z468ZB+SKKsBQOE6dp6lmWl5fJqgttrVrXhmCbTD6O4ElfaGbQfjVq/RvbKp+9xzwdo/qTIGCAUx9FcQOHC8pDnI7pb9Lzfl+eBkK6Fg7iRcPVzn9iFXTZNX3SZES0xXFG8Tc9RAQdcTfodP1/SRbc3cPQaTYC2wrDj0DkBefy+553CkxN48Nax2+TmaB1k1Ghf4qjpdw4lBX4reuqiae3wqIoqVHOJ+wS2wDXUe3TdNixqku443wrvt7bMUdqx94U7aMBghCXIoDXDrm2MKLMx7v+dyCA2hlktSQ/HxQFGYLqhefXZkT5/YYWoLRZ99o9wTeHpbX+RYvnFkQ8KMBmarpWw0CQg+B93crsnfNbNsOguOnSVBiKguBGibjI3ezrqiy4v734OKAHZ02jhUH6aOy3XXMTxd0XPkAvtJ0VysVtuPoVKDxeumnLM2uWMR2c4ASUy2wsnxChhq1zdJoQPX58wytkRHTOORvxwlBECJ/m4y6/a0XXl5bgZwfRCD6lKnQnrw8SoHVl71quE0= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(376014)(82310400026)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:11.8677 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32265f0b-0c30-4522-e16b-08dd671cbf18 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026368.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB6805 X-Rspam-User: X-Rspamd-Queue-Id: 36ACBA000C X-Rspamd-Server: rspam08 X-Stat-Signature: y6u53k49wekhesk7334yb598c1hq1gig X-HE-Tag: 1742412740-134687 X-HE-Meta: U2FsdGVkX19DUcn3SM4iFcTVRCZgHZEkh6WqL+ClAOIPrngfMMM/BUePJsmp9E52uQhGkuQoQispVHYbpsJDWiYy9skcoxKO+MNg+HQ1fEh2Gzc9upV6BbxWXulJYW9sePtvWCAoKUZAR02JIx2ZM/mz5HqA2F9qQb9jPVIRIhoEiR5AKvCdqflRNKpVH3eaOsQKrFohC3+waq/uelGHhi3enfZ95ZWqFQ1ya+154rGXjOFpsSEKE5Rw6F90umu1y10VwL0YXE0OiuZ8VbQI5ra4BpBJN0oSAkvcot0kabU3Mka4lEX/04lj6ubxPXV3T5UgDtEhVTfEdlbsVoWz7/EVLFgPfiZN7pgvCxBWYTAnQ/9hxORwPZm4DWvnPgOrkN+kbFNFvTNX9n1KT9UC8IdgZSnO9cE0iHURntY9JeL8bOFIVdCQkDm2wfwa6iqR22hmtpUHojGMIJK7i1XETLkL8N6PwT34L7JWBhjGRZcXuxaBa2YTuT/rPXKcN/bYemdfV2zdKz4gty80ehF1B+a1UQwOjK9mAizCxCmZN+21qHEuSJCGfWA81wYjrUaKQ2cWFoFZ/FpBRtBquGkUcGFXJtn2F3w4C5TibHae06Rx6xiTROSn5LSMa4U2YF4VFwe3m74Rqifdtnemcdz/41fGU9LfMSMN1QGGor7fRk+l+qu7JdF3y6pAiAk5O6N/XLSJWBgpznR2BHRX1UTgcvGYM0WUugnggtPIZTt4g2xlIFfqv+fAhd5VOBB7rW68nQfbwhWKmz8htomx/DnEwfAdPt9WyV5tjVB0Yk4urQm/yDp/fHJB7/kWqbXResEPGcRVIE9QwVSq53zK77k6/LxvopfqeU3KeqrqZGqYM4gkzIfxbecDFZXqtVn7GKGQIu+bNG54picOjv9fjHA6n5l3jKURWEVx98CXwMGKNnHYlMt1xzRNtnS3JtgxXkkh9h8j2BqeEshF8BDERPz nOWYFm7l cnQjE0rxot7DbBEsscEHbJKsmVUzrd8s24vzBFeoDz631h1cDxKV2S0D9a8tQvQc5m+xrKOrK9nACajUIPJ9yKS+1S4zjr661kIwWcK9hVeJ2IrAnrvJvORGWSXhjAVH8K5fGhb7PsTqG0hT2mGy+DackZ5Uk8q5ftjEUZIA3tERgq016798oQJVBgdIEC5vw+FnpXtbbHPqg06WZDFlv2TIhTjJIhmtRlAoV91W2qLY9I5cWGLXJ/rzWjGNJraaqJk2310JNFSrHbaoHDL99cSxTBEB9Gmi60wi8TVwbLnSxJ6dnLmDwNeMHxqex8hgR5rRZ59qPgfkwNhTqdtY6QTZGkYfOVFLweWo5MfSjO0spS2DnRdQvRQwucdVgfYrt2NiEEnlstJLOPVv/gjGEiG71bZzdDdahmUc3DnuG5rGiFW/Lo5r69A1bM1FDYN9B/tVjwn9U85cl+cN4j8bbz9j3/XfvX0mkqqEUzUMV48DtZRcRznUxmoUvbpWoGdT7u/M3AQ7c0aF12mJJMRVGhM80frsOlIbSI/jlDD0wzP+97zj8CQFRLWGHxj73PM8wkFHfcKK5wEUcRPyYNW2uqUbo2i2wlQrE2MeaTcDcyP3blK2E+ntXqBSKWYifTuJ50sNlTYfPfF+YoJ8on9GhPYHhqtWhj2P2xMLMuRLlgIt0RYAY0EqwaQ4bogA7AuUL3dlj5ZPL0HmjUjY4tAuLkJ6AlLSgzihZ3PB9Ww5i6qPJ6d6Isfi1Zclhx5e8/x/HqWhGas1TiiXAXmIsSU+7C6qWq3kVneb+62X3TG2o8m1xLdzeJ0vXtepYfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Before this patch, scanning is done on entire virtual address space of all the tasks. Now the scan size is shrunk or expanded based on the useful pages found in the last scan. This helps to quickly get out of unnecessary scanning thus burning lesser CPU. Drawback: If a useful chunk is at the other end of the VMA space, it will delay scanning and migration. Shrink/expand algorithm for scan_size: X : Number of useful pages in the last scan. Y : Number of useful pages found in current scan. Initial scan_size is 1GB case 1: (X = 0, Y = 0) Decrease scan_size by 2 case 2: (X = 0, Y > 0) Aggressively change to MAX (4GB) case 3: (X > 0, Y = 0 ) No change case 4: (X > 0, Y > 0) Increase scan_size by 2 Scan size is clamped between MIN (256MB) and MAX (4GB)). TBD: Tuning this based on real workload Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index cd2215f2e00e..a19b1f31271d 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -28,10 +28,15 @@ static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); + /* * Total VMA size to cover during scan. + * Min: 256MB default: 1GB max: 4GB */ +#define KMMSCAND_SCAN_SIZE_MIN (256 * 1024 * 1024UL) +#define KMMSCAND_SCAN_SIZE_MAX (4 * 1024 * 1024 * 1024UL) #define KMMSCAND_SCAN_SIZE (1 * 1024 * 1024 * 1024UL) + static unsigned long kmmscand_scan_size __read_mostly = KMMSCAND_SCAN_SIZE; /* @@ -90,6 +95,8 @@ struct kmmscand_mm_slot { unsigned long next_scan; /* Tracks how many useful pages obtained for migration in the last scan */ unsigned long scan_delta; + /* Determines how much VMA address space to be covered in the scanning */ + unsigned long scan_size; long address; bool is_scanned; }; @@ -621,6 +628,8 @@ static void kmmscand_migrate_folio(void) */ #define KMMSCAND_IGNORE_SCAN_THR 256 +#define SCAN_SIZE_CHANGE_SHIFT 1 + /* Maintains stability of scan_period by decaying last time accessed pages */ #define SCAN_DECAY_SHIFT 4 /* @@ -636,14 +645,26 @@ static void kmmscand_migrate_folio(void) * Increase scan_period by (2 << SCAN_PERIOD_CHANGE_SCALE). * case 4: (X > 0, Y > 0) * Decrease scan_period by SCAN_PERIOD_TUNE_PERCENT. + * Tuning scan_size: + * Initial scan_size is 4GB + * case 1: (X = 0, Y = 0) + * Decrease scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). + * case 2: (X = 0, Y > 0) + * scan_size = KMMSCAND_SCAN_SIZE_MAX + * case 3: (X > 0, Y = 0 ) + * No change + * case 4: (X > 0, Y > 0) + * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm_slot, unsigned long total) { unsigned int scan_period; unsigned long now; + unsigned long scan_size; unsigned long old_scan_delta; + scan_size = mm_slot->scan_size; scan_period = mm_slot->scan_period; old_scan_delta = mm_slot->scan_delta; @@ -664,20 +685,25 @@ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm_slot, if (!old_scan_delta && !total) { scan_period = (100 + SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /= 100; + scan_size = scan_size >> SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && total) { scan_period = (100 - SCAN_PERIOD_TUNE_PERCENT) * scan_period; scan_period /= 100; + scan_size = scan_size << SCAN_SIZE_CHANGE_SHIFT; } else if (old_scan_delta && !total) { scan_period = scan_period << SCAN_PERIOD_CHANGE_SCALE; } else { scan_period = scan_period >> SCAN_PERIOD_CHANGE_SCALE; + scan_size = KMMSCAND_SCAN_SIZE_MAX; } scan_period = clamp(scan_period, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCAN_PERIOD_MAX); + scan_size = clamp(scan_size, KMMSCAND_SCAN_SIZE_MIN, KMMSCAND_SCAN_SIZE_MAX); now = jiffies; mm_slot->next_scan = now + msecs_to_jiffies(scan_period); mm_slot->scan_period = scan_period; + mm_slot->scan_size = scan_size; mm_slot->scan_delta = total; } @@ -689,6 +715,7 @@ static unsigned long kmmscand_scan_mm_slot(void) unsigned int mm_slot_scan_period; unsigned long now; unsigned long mm_slot_next_scan; + unsigned long mm_slot_scan_size; unsigned long vma_scanned_size = 0; unsigned long address; unsigned long total = 0; @@ -717,6 +744,7 @@ static unsigned long kmmscand_scan_mm_slot(void) mm_slot->is_scanned = true; mm_slot_next_scan = mm_slot->next_scan; mm_slot_scan_period = mm_slot->scan_period; + mm_slot_scan_size = mm_slot->scan_size; spin_unlock(&kmmscand_mm_lock); if (unlikely(!mmap_read_trylock(mm))) @@ -864,6 +892,7 @@ void __kmmscand_enter(struct mm_struct *mm) kmmscand_slot->address = 0; kmmscand_slot->scan_period = kmmscand_mm_scan_period_ms; + kmmscand_slot->scan_size = kmmscand_scan_size; kmmscand_slot->next_scan = 0; kmmscand_slot->scan_delta = 0; From patchwork Wed Mar 19 19:30:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F395C35FFB for ; Wed, 19 Mar 2025 19:32:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C202280015; Wed, 19 Mar 2025 15:32:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24B41280010; Wed, 19 Mar 2025 15:32:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09D9D280015; Wed, 19 Mar 2025 15:32:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D7E6F280010 for ; Wed, 19 Mar 2025 15:32:34 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0E84755F55 for ; Wed, 19 Mar 2025 19:32:36 +0000 (UTC) X-FDA: 83239297512.22.BE61E4B Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2069.outbound.protection.outlook.com [40.107.220.69]) by imf03.hostedemail.com (Postfix) with ESMTP id A92FC20021 for ; Wed, 19 Mar 2025 19:32:27 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=kvsoUwai; spf=pass (imf03.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.69 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OWt1GZDA3EZY+FP4C7YhcSYsQNWOvQYIMshl1kgSG1I=; b=OcHoExzdicT3YdJvrpf1+w4WN6V6Q67FHxVDpnQwqfqGnHqMHjLbWBPnM2Oq0nARiAMd0P PQf3psa9mEJPJZM1CX2MNT45vWu/cZTj477gxv1+k8jsDW+BDrSD6pXrCtwgD/yCoHVnvS iYoK7lWr7Dm+nvdSwydba4R7kmnt+yE= ARC-Authentication-Results: i=2; imf03.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=kvsoUwai; spf=pass (imf03.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.69 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412747; a=rsa-sha256; cv=pass; b=hW0M+eTxxEmsHOq9b/WIzqCnLlbRQaomoW2BHJ+M2LhceD9o7BqOb62tgscowBAY12MqYY i/shQl+5WAA/s/j4TzHv6pClTbqFb+rIn+KZQ7yrLc8vl4hGKB2MPNLc9p7PDRZJRAH84S kk5hOXE8RVmQJ6lHmAmftOxJildnan8= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=zTMHDZR2EEMCdZs3UWwMQndK8OkpnKimRntopJahrXuFZ9kbYPkfgmfLikaZcxBUyN6e4iW+IhYhLafnE/mjxIirvtZ2ebDW2dcWmbgHqgl0ybuayEuh5/3ZHdfjGI7JrnfcryNmCQCthPVTN9BR/rcwLXgirUtXMjXLQyANy56rWKgKD40wlBMP3ggQBlj2DRrs0FjQUpUIRzgzrB7S5BtDC48DtgGlAF9VzuypLmlfutLwjVr5BhM5xwNi90arGI6ptu85+IXNkLutWiQ0kLDTgItgicQKXGWF1aFZ2u9g+Y6j3FERNzDm3HajwtTXDYZWzZkXoKLSU0XpsJ3KAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OWt1GZDA3EZY+FP4C7YhcSYsQNWOvQYIMshl1kgSG1I=; b=o62dgaE19l4LOptxjkvP+XjZ0lG30h1xwGc/4g7n+T+d3EdrQbfUCayQFA9ERB8aj3MVaKDjrg8ucnPrFgWEEimVDsTFtGaN/MdXZkNZpBDDaYsg7/dymSWI1zZ42MkLxD/izsykI6vMllfbi2x+T3V5+PKni+8Lcwtt4jsxqqTs3uL8TARYV1TCwo/nL6g6qMPqlppg4YnjkTQhs3z2bjpuOf95i8y8VmGXc4EBXqTVnFDJ1cyu1d8B4pKApHvZIk6crDHMLPfJbyPKhmxQ4b6CgZJOx6t/x75QZESBuI49IgbLM1OXgqdj8Eb0blok2ipKOX7fy55NUHblKmhxOA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OWt1GZDA3EZY+FP4C7YhcSYsQNWOvQYIMshl1kgSG1I=; b=kvsoUwaiYrAJ1PZq9XnhDKRKfvY20q2UzvMZzRGqeZQEJQ5KQnLCsDueb00ml0POvaK8hTRuL4F6JQ50aD8UjuqBIgdq9YvPuPeDrwRUTvHG8xILDFV+ZuTR38FOi4Hd8EbPyDHUPv3gCL38pLtatQ1xjLe1tR9LtLoTH2IfOx8= Received: from SN7PR04CA0005.namprd04.prod.outlook.com (2603:10b6:806:f2::10) by MW6PR12MB8835.namprd12.prod.outlook.com (2603:10b6:303:240::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:22 +0000 Received: from SN1PEPF0002636B.namprd02.prod.outlook.com (2603:10b6:806:f2:cafe::6e) by SN7PR04CA0005.outlook.office365.com (2603:10b6:806:f2::10) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:32:22 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636B.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:21 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:11 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 08/13] mm: Add initial scan delay Date: Wed, 19 Mar 2025 19:30:23 +0000 Message-ID: <20250319193028.29514-9-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636B:EE_|MW6PR12MB8835:EE_ X-MS-Office365-Filtering-Correlation-Id: 4eebad5a-3e68-4ae3-2f75-08dd671cc510 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: Ok7TasDbDnnHhl32YkpffKvFAVstQVfvDxyTfIRD1RdHJZ3ti1gvSNPBEDPWsN1fVWvcigqPcOWPmIPYELNylgi63+TXbAairLoptFGJYgOUa4XVqwOX1nUyqPqImScE61rlLT+7VB/H/ulIEJynayR/aSstXP7O6LZOXGJy2y8fm7LvxAFtH7lY0Auu5UxztfRNm9N3rT3AYlUzjDpPkyk14fGVlniQWmMxZxIlyDFjaG6x3EAf4/lWeXNEEgHsTP8dqiG5Htj1/Ard9fX/ZQHkvozKp27KWH2nLgi/V6rdc66JTjkeczlpiw9+gydWmA6WC+vU6Mm/SBVWNFXFqmjGl2dFBOVojp7H2xuVdHUo3J+OQUwmEncpvOnv1J0sg6clnw+n58NX9h6q0ppf6OVqbXZWu0NZzCl3/z5/rB7joDrWbiq3jXB0hpz4K9rH3oCWL6BZda0XgWYyvkKWete2FkQZysyN4cYdz5rE61KgZ7rP50nACBMlmK/R1p3z3utvQJWOGoE82/Sz+gGawovWWy31K0UEYO6pVCpGd3c490g9AGvO56e/FrrM2anDjAEJIqqNowmaDL8c4452hY15cyQciR2Cwi1UfF9oC7RDRxgu8dP1JwA3cWPaGPFQKl19GMDLdbMS9p0XoagceSJ9LMbd32DEjU3f+U9soU9/xFgZoi3L/R78vkBqrG/fRziW6XmzI/Wbp6DKMGqw/+A7V+y9bgcXCY4zulxjDnvi8FPMZwutksC4CUBRCtsUl1jy0/dZRKSr5Y5V92qiA56aM/BlDwdowGuQd8ekUUIHWNzGsIk4MLGyeE1McoCd1sUc1Rmz4Ecq2ImplJbRk1cSsjP9/q8LcyKbHJ99SEpCbManKluizDJml3C43Nt+BUvXU2BYEYsrSQebKWsEayWoXBKpdOUtN+6VBPUciZmxqd+F75FvaJoJXNKe7Od5YpDEoZ6s4YgetfA987E7mArZK66CGweH6OyQQ8eX4AZoZq84Ufc157EMyw12q9xnSegHZfu6vO0BQNjy+x3C67yp2WFo1/5XET8A9bEdTvx89n5uMV/DX2rPVZcqcAVoDB2gRjyYynA0PILW1vhrjRzu/26j7OMFR/RL48g3SP2vG9VfF+7M8zm+WZgVsxYpDgez9GvUG30N87GHrLaMC+yQRVry7BSuD+Ra6B+w0ELdp+2kToVHdH2wOqv9no9OM3uWrarSN6frojAmsos5n4bWyI+JIE6B/IrXvQqlxO/8spPD+ViNX6YioHyqYp+nThxBOP7ovoe4wP1LKIt6ojWRhvKWs8YGy3vg/sK7c2+lfNwRL6h+nkulDGY/qGFtxELrUry20xslvDvble6qZAT3Vo62JHkFm6vlOZycWbUkyGg0sTg51aoK4ojqbF/iJY4S8sarn13Zs2XA6/jSauW7MR0v4pvwBUlI+x4mdRGfTWHgokwNqSs8VmGPR+bo62g1FB1RaDVM7qbCmxug7lvvCfAczNsTARlCwJhWBGE= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:21.8770 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4eebad5a-3e68-4ae3-2f75-08dd671cc510 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8835 X-Rspam-User: X-Rspamd-Queue-Id: A92FC20021 X-Rspamd-Server: rspam03 X-Stat-Signature: idauftocoi4x7i6yf3oepw5siyykbxqu X-HE-Tag: 1742412747-208112 X-HE-Meta: U2FsdGVkX197Laa4sKIHZHxs0YOyeQjUAz0abBcE0ffk0W/6GYDzStxqWOHS+0cR21uzA2U5p/ObtCKjYbquZOKNxu4mvVF0BJGODeoBPRmELi7tZgaFJgwpSOwqjATnPszpFdYBC9vpcGx+TNeqqcz/q6xj/orY3UBvwMtUSq1/M7QNLorZkEbzQPyhCMV5ZXj/4kOiVpriXMa6PCPQGA8nxOCrw5Gy5mkR2KoUd83EzdUNi2nN3DWMBKg4bVQ1pTBqFnfObDumqogO9qFYP/UAGrVIz+nZKJWohRYNEpUDbBwIWR4p5+H/1vlIM88edb+yZXwog/lA5s+257EScun65Nlkk1XY7Sct82dFn9Wh0V5AUEWHB36/B3j0ndDPoGknKLYfhCQ8HIqDG2oWe30ju2xV7hat4ptRiDVKHIAyfKqP9KOOvuxrOC6//jBGyNzYg16c9ThhRvPBsrcj2WJWLvFT5EqMC5KGemSooSbHTujSjR/HtnGlaGtMPQ/RNGoZT8u28+pGtQO85Oa+2LYoGlD/jJZUdcFp5XWelB9f/XZ8urKL21yhMf5kX02QfoHP8ZE9sI5cLuCQdwBvBPDvfXgJvuoZ7YQuYKL6JjKXCpqoG0GOuizubUC0+3ULHzaZnWkRboBZErTdtGBxZFgcUQEMkfHE76cHjZf5XgXSsWVtcMQRqKSVYx/BOfhgiRxpLmnLqhLKBA40DINkXbe7Abaj/oQ0fI3Kk/L404lF8RTag1rbS4AY8xXHthlUkBuDkqybuzHMnpIym0tTYIgnZ2DwtflFZD3ECjZeHk66gkw6J+rgHdZgIsWWTX7+vC+MLA16huEB8Bw9dsUTPqRxbjo7cuUPmp30OVgC3VBCSN8skuPh3R5E/ps6wr5l8sfYMkKmydDf2IcSt2CVby0tkjciTosIjkUbjMxbFNnYCi2pTJUhyhHYcjFNr0hpikuepBMlz9GLAUxPjYZ f/ukMWhL M1q21j4AxZ+jSyU2QPEfev3EXQ64idV5KP+SXZ6RlwZw38M6Uvg1foVN+BNVp31cm7G2Vc6D+ys5LcCBYxsMYMRQNK1lLS4iLVGZtKfvCyDQe6siu3Zjbumk/XHxLCzNCn6gwONBw6LGeeHblSj0eWnkrJaByJcQtMCLrtKDdZitUef525jYmZOizcMFYuSvpFQS/Tn3wG0cr80TIJpJ0F0G5jN1lZ/su8Y+BqWtsenm7SccqOu7V8pGbXfgVq6NAn9Hx14TONIZKjQVTBZFoOxQyIbjGQnRXE+QHR5h4gyA5dMIUynY9ccT6syugR+VScGJpUGT/E+hmXeW8UkxJv4WbzbRe4I5neYhYn7XATwvMYE6n0I+WDTxDCLCfPbUMYrE2ZoKMKgvYdFVB9MWFggE+uJESzPCAHIQjrzw7IuPZOKy2MsACTsjidMGpWPkrcz+QWjl9YQSS+PXOr5M6iJ8p2jkerAI9jisEUXxvYEi32to15+Y2+fa90oDiV9Sad544268X/GX/aZYcI8P5Kj/6IbtOFVBqysUom55vjHnBuh0tDAM0t8dLz2+82rjdGibVyuTGuv+GnsPUdueQdDCheiC0diUUcV8aKUhvGSzANGUGv0bR6Pf2g4feCeta3F9+LWl6H9fdlizDvxHLPNCxqPC1hQXk9Eyvr+lWY5EIXO6VssGwOUQfgIwwvRs6yttdGgsVIc8PqAPnXHhAb26A+xmH2QnpgZ3Zrijrp3Zxz1JmiNq+isn22U4yL8EQZafF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is to prevent unnecessary scanning of short lived tasks spending in scanning. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index a19b1f31271d..84140b9e8ce2 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -28,6 +28,7 @@ static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); +extern unsigned int sysctl_numa_balancing_scan_delay; /* * Total VMA size to cover during scan. @@ -880,6 +881,7 @@ void __kmmscand_enter(struct mm_struct *mm) { struct kmmscand_mm_slot *kmmscand_slot; struct mm_slot *slot; + unsigned long now; int wakeup; /* __kmmscand_exit() must not run from under us */ @@ -890,10 +892,12 @@ void __kmmscand_enter(struct mm_struct *mm) if (!kmmscand_slot) return; + now = jiffies; kmmscand_slot->address = 0; kmmscand_slot->scan_period = kmmscand_mm_scan_period_ms; kmmscand_slot->scan_size = kmmscand_scan_size; - kmmscand_slot->next_scan = 0; + kmmscand_slot->next_scan = now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); kmmscand_slot->scan_delta = 0; slot = &kmmscand_slot->slot; From patchwork Wed Mar 19 19:30:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CEF3C35FFB for ; Wed, 19 Mar 2025 19:32:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A49F280016; Wed, 19 Mar 2025 15:32:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32B68280010; Wed, 19 Mar 2025 15:32:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17D32280016; Wed, 19 Mar 2025 15:32:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E7D5B280010 for ; Wed, 19 Mar 2025 15:32:41 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F25FAC0719 for ; Wed, 19 Mar 2025 19:32:42 +0000 (UTC) X-FDA: 83239297806.21.798C5F3 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2046.outbound.protection.outlook.com [40.107.220.46]) by imf13.hostedemail.com (Postfix) with ESMTP id 8A1BF20013 for ; Wed, 19 Mar 2025 19:32:39 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=L2lXQwqg; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf13.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.46 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412759; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3JobkGtP3Dd44LrrvsSjv2D3CsygQXYXyfCaumfB9Hk=; b=zubrqOQnhSSJdejO0VfGgLLxza7xhlUNVhHbwfhNim4WuOTocFTjIWnUfqUvzXIEQ9RUNn c905ppVVUHWVy3RvxHtY7AFS/vNvfLYBG3Ef7xF/jv/iLVuO2MSFbiCOoXrydI0RGSA+jY keWRsqwZ0ndtvWUcaArSmOkfxQ4ho/A= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412759; a=rsa-sha256; cv=pass; b=QZ+fpR/QkpV2BM0LrRtBCqd2tcuuh50bIKR9QKuWPzB+YpblUwZi65lBZ3JJTghbrgjyhc /tWg3SSyWli9pRRpXKPv3vVVrj5aKkKU89OJANdPmSZcSSI8v3pT3dGyNj1as3WkGibawi nWVsGMmkGd37N9urv4cX8WHYrdj4OyA= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=L2lXQwqg; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf13.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.46 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MqGRPycymWQayi5YBWyQAECzfD0Igb5HuJ1v8IUah1ESpXcTRjPE/1a6KfmWVGaXbpfEcr6DaRor8rRB9eNoaO3X/cYF1lQXZ/4GYLPqmLlf2tTQys5S1YysfYvxO769C9iA7fpKjqGEyZiGbwGtXVbZ3Mzi/12uOYBjYZGyQ3+u6rCUZm/y2K+rEuH4PBIDpioYawKic8xPVTTi5uZvsiU6yN0a1MMqnqv++a2cI51N5feHP7Fyca9W/dbbGDYhVeY9CYixxJxW38dlbq11FLxU7zRoYUaW2Ord3Zl0M/apxxasCgAwn1d798JHCNBRO8HkGe61490+lTzVbvspzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3JobkGtP3Dd44LrrvsSjv2D3CsygQXYXyfCaumfB9Hk=; b=lfisZdkpKVMdNpvdLTaxf4Z70bEXrcXO7ygHjYpIOiYPlHGB5syIa5rB5IPdn9ItR8Ca2hORVmKDtoF6M0nTUVd4Kh8uHf3E135cB+Li5HLYmCgGXDjZAJ0Bcs6treIpRCvsc9AzvjhS2TfsCU5gbsTRtSu476/ka0MN5f48nM3Xg7PPdySze+KO7rR4/IXP7zeFTKuPHp6yABBZ8BERaqzPbK9el4vslcAKKAN8uUA1EfMxqmSACtQ7554FgA4GA0WZrhM4VbyhorobjIlylBVQg8E7+N1gMP4TG19rfH99OtSP/tM0l+JmKecUIDUMutMzieVBYA+7IZ4T+1csxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3JobkGtP3Dd44LrrvsSjv2D3CsygQXYXyfCaumfB9Hk=; b=L2lXQwqgg3ziT7XgoNHqKtaFbmXyDj68hmkdNeNou/eETuOQJf0V74A7tEFeVFgrnLsnbnJ/6nWv8nrt4Ia9B2LI3I5wcER8omV+r3ybIe5zDheECHdN1Zs/Mq9NU3Yz2W4fs4LSKveNLY6G1YyYwJ4zrK5g6WLm0MVY0cvj8wM= Received: from SA9PR13CA0096.namprd13.prod.outlook.com (2603:10b6:806:24::11) by MN2PR12MB4110.namprd12.prod.outlook.com (2603:10b6:208:1dd::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:32 +0000 Received: from SN1PEPF0002636D.namprd02.prod.outlook.com (2603:10b6:806:24:cafe::be) by SA9PR13CA0096.outlook.office365.com (2603:10b6:806:24::11) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:32:31 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636D.mail.protection.outlook.com (10.167.241.138) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:31 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:21 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 09/13] mm: Add heuristic to calculate target node Date: Wed, 19 Mar 2025 19:30:24 +0000 Message-ID: <20250319193028.29514-10-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636D:EE_|MN2PR12MB4110:EE_ X-MS-Office365-Filtering-Correlation-Id: 89c7117f-eac1-4ed3-7784-08dd671ccae1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|1800799024|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: epnti3YhX8xkc7q4GWx1wq3kfO81RLoY3glgj+4tJV44spY3HwAKppiQO7/KYM4+dk+jZFH/PCtXjS+byH884CxZqpGzwFpck+TfRfycWqA7MOZypkWVVbT9mCC7dyldfY76kBFsuMH8B767I1+6PFjieR8O7zwa4PmrBnLkjPJUcakzWFtclBorCyknsiPqN0KwsCIpattrMcLmLXbcWtwAby6gNWetKHT3OugXD4kICgZG47COWjBnSezE7evblWVscKkpC29e4TpOH7W9pu8RFMDbwytF0ygm/O//zc0mVK7iNtttWKXZWUKz/0+7BvzRlRo7/0IcShIpnw0hNCm5I7G241CYjyrGMWl2fQ5Y2eTUYEZKdQMWaAFzhe1+Wgo9SVY7vTGGFfpXCZB8kpEMVreLv6WjRx5K/HD/JuIOy6/1bK+IH4Y5covYUmwkVBfeiZVoRypSxRyfn1kr/soQ+UhpgoglrstWV7KRfMjBVTVnJXgLHw49ObNqeWN3AE6J4IJUzVJFpemgVvfidyMZ1wA0FF5UPO2SM/0xoTlJp6KTh/fTNqqQ5foGx+SPFi+Cs93S6kCs3nfU47fZIcu44K7LOU1y7o+6p76G5MVZJdOBmEMLgSUmWnYYBUZ4/Qz2Wa6AfMG3nbMgNJjHAKOEKiDh6tJoqx1S0EqFJ6gEe+rd8JcnI9fxHbvU0lDGE/qSmtze370PLpNIqjzrJXudETSVEeB+jHchxI2JmgIzMjGC4gmTqF9Oq2TyCd8wX3UFmcZE/jbrS90Z3JBT3TyN8hu5D+zc4ihSOLX56L8IkVfcgTdDNOWbo9K/qisH3E3g61ZO29bcSTJlCTMr1iEm1JjTuX3TGD6AUfbDDw3aqPCem1GgtJjyY7U5nNqaacMb4VSIwS9WMfbtQYI1DT8UaxBOwgAbMT1Qs2P3qE4Rs8QUg7+/VC0y/uaJdlH4EOg6YOqHowsW4h6hq+Ia5ojcZewUewjwbk9m6URCEkhgdBYeZoKAHkwLsZIsVuEIIj6NI1wR8iB3iCLLDC5vPVIhgPsywHTPsFxFrRd9znhPAqwRqdOvlu/rrJ7I2FmCq0LV7nYemNWFBbZRdw3P1mnfwg7fRdrCDiHwAkXnAXTe/R/Q34M8GXkiWyvEgOnlAiCZkOEX4BYs24sDHpNtKgIgm0VeelvcxnCZ/Tq3HgVsO7koNIENn9xs7/Is1p8Vke6ec7YtSw2nKWKkFgVS6fdozwDGSayM7DYvy7JZp8ikgM7xicWvE5dW/sLeJR1wX8YBv5/cDdSxzWxf5Xh6HMfvQdY6BgGp3SfsrAwrKCUX4eNiQnDKsnImyu7hiZQb9p5NH2GVCi9d8E1wXuADrnOXneHTa7/Y/JCh4dR7CMkJmpdqFm7mMmUHgZtYwPGs5/Lg5NuSlEb3h1eFvG20FI5cOw5frGj/cRbVqwq5MzOIo1SIlnK+nSV7UgMMhXzwQfMJeUFinMbo8G6TD9/JX2A0vijtBt9XdIekMLBNeok= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(1800799024)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:31.5747 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 89c7117f-eac1-4ed3-7784-08dd671ccae1 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4110 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8A1BF20013 X-Stat-Signature: 3193sbham3thj8bws34agbsshjgbfeyg X-HE-Tag: 1742412759-454891 X-HE-Meta: U2FsdGVkX1/N38xU4tkaYushsi0P3rsgQatXVwmwpUiWOeqDodhYMdH2cNpDl/FvZw2OwZ+T67McJNm3pd4dScf8BqllLma1PgkgrPl+kRp05K7gggOX1eKA1kaT/xshhIszVF0TD4JuN1qte12si8gZ18lXMjFlELv5CWiG5EWjrNoJLQx9VMiEbq5Ycl3UouCvj91KlvI9BJgpKMOYmveNisVbfqvaUqrk+gwZJiQsRuBaA/W/fZDu3DuBNB6AnoP2kUF9HpxRSmud52wtTe1EcKxhund+monlVHWHCk2Nvzoux0rqOeNAjH/Il3luHIxwtfFJesSnNrGx7nWQYaJV9oBhq2Fy7a5sMi4ZUiD5XO13jxGFaOHVfWlJFCft7heEgxJ1saxoltmAnz3M6mCiSYlb+YXaVBIyJDpKtVLHy0ZE2yN3kF64gGM+Ld3lrrsz2x3nBlqytrm3XsEwViCl1SzUxDP6r2bKlkKQ+6m6ux1uO7GsT/ZvSefR0Gwvj/6viNN7UzLODOjRPzog5r1qYJZr1gz/4Mkush/cplzlFz9YSrRwlY4+o/r3m/lrXuBSwu62EAMm3YZWfpgozpdKrMHLmEwv0kaEWrjl/69O0R4vr+Iue7tu0WK3NxPs+wJsjARrx0xXoNcK+177WCAAgeeHV6vhrNJZWVR04tAGJhgR8fmjtYG+lQXmqnhMQOzT+hju/LwIOOaqH07PFnaM9bqWOOwD130l72XSmfQDXnuQLMAGF50eGlQ5z6/kiHkxkPtFjFlQoR/pZnRrV15mQYW7aHMn8fbK/axJpHHJZgmZMP0P18x8fuuSdx9KcRVZLe7EEsRPAYJdvqYX6AaJkkfPPTKN01c4nInjgDFsLwrgoM0RGTIajeY0j2GiSMBD6RzvCkgI7hhscCLep35HWdKt7CyLy/8RYEudJsDgBgAq0khXkL/8kIXvMy5y9P4ZBe47WvmFa0K+ji+ dS+B6jtN iDYohohvflQlX41OMrTHxetBkPOiQMU5Kj1f4xkykLGFsq/4P6XTnG+u7IwJVuWtDJ5MCEUNzjQiUyYbqRA7RdQLzMKVLmE6Ze5RGhazR4XeVAkHcYyU0Vm3gAlQvSNlm2RQ9fawNxVqyPYSKq478eRRDw/MDrN3nmzckgPnGPhHqH1VRoElb+hoyHCITPNETay7ZmKJtGN0hBap664VPPLQ1BjsLBq4XPmYgJWScN2nHXt6i1RENbcluVMwj70KB+nGrMfv553fUC4COjz2xDPzPi8cLIxhOwH9yNRIlbI7wRyik2CrPN6tpTk3hXomw4e9F38TcNJE5cv5lkS20fZdRiwDOtZF3W+7tyPhhLIaDFIMgqMdkYn6Wa2Alp8N4tkPCvtLZaWBogOGCmQIvOSOE07Sw8HZwkOuZL1ASZyaAqPnPCX58xIKn/240/UPKob3oEnZy7MbRDj1UlRSSZiR4GGN+YnoxJc9hMn89RFRxAARdLKaUiybkgBIFnCS7hDEXqh2UmCvWMw7WBYNGp7OwlB3J7DS3hhmlurhCjUBihI60yU8g7I7+ngQNlt3J+0hUvpxanFugLOdXJfwrgxVP/zrY3uBopcNpPcAYzxRs6nGg8kh1BH+HfMuos3zlkuptyW79mYt6M4DEsYDtpdWC13QwbwsMxmygzSk4Pv+h5xjV/nEjDyAn5NADgvkZZKiShDIuLPdFwapIZm1FpZQTsFszo3p3QbQOwR5ysRYjtkG7dGVHeDnMdULJGAs2DmrX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: One of the key challenges in PTE A bit based scanning is to find right target node to promote to. Here is a simple heuristic based approach: While scanning pages of any mm we also scan toptier pages that belong to that mm. We get an insight on the distribution of pages that potentially belonging to particular toptier node and also its recent access. Current logic walks all the toptier node, and picks the one with highest accesses. Signed-off-by: Raghavendra K T --- PS: There are many potential idea possible here. 1. we can do a quick sort on toptier nodes scan and access info and maintain the list of preferred nodes/fallback nodes in case of current target_node is getting filled up 2. We can also keep the history of access/scan information from last scan used its decayed value to get a stable view etc etc. include/linux/mm_types.h | 4 + mm/kmmscand.c | 174 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 174 insertions(+), 4 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0234f14f2aa6..eeaedc7473b1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1015,6 +1015,10 @@ struct mm_struct { /* numa_scan_seq prevents two threads remapping PTEs. */ int numa_scan_seq; #endif +#ifdef CONFIG_KMMSCAND + /* Tracks promotion node. XXX: use nodemask */ + int target_node; + #endif /* * An operation with batched TLB flushing is going on. Anything * that can move process memory needs to flush the TLB when diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 84140b9e8ce2..c2924b2e8a6d 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -88,6 +88,14 @@ static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HASH_BITS static struct kmem_cache *kmmscand_slot_cache __read_mostly; +/* Per memory node information used to caclulate target_node for migration */ +struct kmmscand_nodeinfo { + unsigned long nr_scanned; + unsigned long nr_accessed; + int node; + bool is_toptier; +}; + /* Per mm information collected to control VMA scanning */ struct kmmscand_mm_slot { struct mm_slot slot; @@ -100,6 +108,7 @@ struct kmmscand_mm_slot { unsigned long scan_size; long address; bool is_scanned; + int target_node; }; /* Data structure to keep track of current mm under scan */ @@ -118,7 +127,9 @@ struct kmmscand_scan kmmscand_scan = { */ struct kmmscand_scanctrl { struct list_head scan_list; + struct kmmscand_nodeinfo *nodeinfo[MAX_NUMNODES]; unsigned long address; + unsigned long nr_to_scan; }; struct kmmscand_scanctrl kmmscand_scanctrl; @@ -208,6 +219,98 @@ static void kmmmigrated_wait_work(void) migrate_sleep_jiffies); } +static unsigned long get_slowtier_accesed(struct kmmscand_scanctrl *scanctrl) +{ + int node; + unsigned long accessed = 0; + + for_each_node_state(node, N_MEMORY) { + if (!node_is_toptier(node) && scanctrl->nodeinfo[node]) + accessed += scanctrl->nodeinfo[node]->nr_accessed; + } + return accessed; +} + +static inline void set_nodeinfo_nr_accessed(struct kmmscand_nodeinfo *ni, unsigned long val) +{ + ni->nr_accessed = val; +} +static inline unsigned long get_nodeinfo_nr_scanned(struct kmmscand_nodeinfo *ni) +{ + return ni->nr_scanned; +} + +static inline void set_nodeinfo_nr_scanned(struct kmmscand_nodeinfo *ni, unsigned long val) +{ + ni->nr_scanned = val; +} + +static inline void reset_nodeinfo_nr_scanned(struct kmmscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); +} + +static inline void reset_nodeinfo(struct kmmscand_nodeinfo *ni) +{ + set_nodeinfo_nr_scanned(ni, 0); + set_nodeinfo_nr_accessed(ni, 0); +} + +static void init_one_nodeinfo(struct kmmscand_nodeinfo *ni, int node) +{ + ni->nr_scanned = 0; + ni->nr_accessed = 0; + ni->node = node; + ni->is_toptier = node_is_toptier(node) ? true : false; +} + +static struct kmmscand_nodeinfo *alloc_one_nodeinfo(int node) +{ + struct kmmscand_nodeinfo *ni; + + ni = kzalloc(sizeof(*ni), GFP_KERNEL); + + if (!ni) + return NULL; + + init_one_nodeinfo(ni, node); + + return ni; +} + +/* TBD: Handle errors */ +static void init_scanctrl(struct kmmscand_scanctrl *scanctrl) +{ + struct kmmscand_nodeinfo *ni; + int node; + + for_each_node(node) { + ni = alloc_one_nodeinfo(node); + if (!ni) + WARN_ON_ONCE(ni); + scanctrl->nodeinfo[node] = ni; + } +} + +static void reset_scanctrl(struct kmmscand_scanctrl *scanctrl) +{ + int node; + + for_each_node_state(node, N_MEMORY) + reset_nodeinfo(scanctrl->nodeinfo[node]); + + /* XXX: Not rellay required? */ + scanctrl->nr_to_scan = kmmscand_scan_size; +} + +static void free_scanctrl(struct kmmscand_scanctrl *scanctrl) +{ + int node; + + for_each_node(node) + kfree(scanctrl->nodeinfo[node]); +} + /* * Do not know what info to pass in the future to make * decision on taget node. Keep it void * now. @@ -217,6 +320,24 @@ static int kmmscand_get_target_node(void *data) return kmmscand_target_node; } +static int get_target_node(struct kmmscand_scanctrl *scanctrl) +{ + int node, target_node = NUMA_NO_NODE; + unsigned long prev = 0; + + for_each_node(node) { + if (node_is_toptier(node) && scanctrl->nodeinfo[node] && + get_nodeinfo_nr_scanned(scanctrl->nodeinfo[node]) > prev) { + prev = get_nodeinfo_nr_scanned(scanctrl->nodeinfo[node]); + target_node = node; + } + } + if (target_node == NUMA_NO_NODE) + target_node = kmmscand_get_target_node(NULL); + + return target_node; +} + extern bool migrate_balanced_pgdat(struct pglist_data *pgdat, unsigned long nr_migrate_pages); @@ -469,6 +590,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, } srcnid = folio_nid(folio); + scanctrl->nodeinfo[srcnid]->nr_scanned++; + if (scanctrl->nr_to_scan) + scanctrl->nr_to_scan--; + + if (!scanctrl->nr_to_scan) { + folio_put(folio); + return 1; + } if (!folio_test_lru(folio)) { folio_put(folio); @@ -479,11 +608,14 @@ static int hot_vma_idle_pte_entry(pte_t *pte, mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { + scanctrl->nodeinfo[srcnid]->nr_accessed++; + /* Do not try to promote pages from regular nodes */ if (!kmmscand_eligible_srcnid(srcnid)) { folio_put(folio); return 0; } + info = kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { @@ -571,6 +703,7 @@ static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) static void kmmscand_migrate_folio(void) { int ret = 0, dest = -1; + struct mm_struct *oldmm = NULL; struct kmmscand_migrate_info *info, *tmp; spin_lock(&kmmscand_migrate_lock); @@ -596,7 +729,16 @@ static void kmmscand_migrate_folio(void) spin_unlock(&kmmscand_migrate_lock); if (info->mm) { - dest = kmmscand_get_target_node(NULL); + if (oldmm != info->mm) { + if (!mmap_read_trylock(info->mm)) { + dest = kmmscand_get_target_node(NULL); + } else { + dest = READ_ONCE(info->mm->target_node); + mmap_read_unlock(info->mm); + } + oldmm = info->mm; + } + ret = kmmscand_promote_folio(info, dest); } @@ -658,7 +800,7 @@ static void kmmscand_migrate_folio(void) * Increase scan_size by (1 << SCAN_SIZE_CHANGE_SHIFT). */ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm_slot, - unsigned long total) + unsigned long total, int target_node) { unsigned int scan_period; unsigned long now; @@ -706,6 +848,7 @@ static inline void kmmscand_update_mmslot_info(struct kmmscand_mm_slot *mm_slot, mm_slot->scan_period = scan_period; mm_slot->scan_size = scan_size; mm_slot->scan_delta = total; + mm_slot->target_node = target_node; } static unsigned long kmmscand_scan_mm_slot(void) @@ -714,6 +857,7 @@ static unsigned long kmmscand_scan_mm_slot(void) bool update_mmslot_info = false; unsigned int mm_slot_scan_period; + int target_node, mm_slot_target_node, mm_target_node; unsigned long now; unsigned long mm_slot_next_scan; unsigned long mm_slot_scan_size; @@ -746,6 +890,7 @@ static unsigned long kmmscand_scan_mm_slot(void) mm_slot_next_scan = mm_slot->next_scan; mm_slot_scan_period = mm_slot->scan_period; mm_slot_scan_size = mm_slot->scan_size; + mm_slot_target_node = mm_slot->target_node; spin_unlock(&kmmscand_mm_lock); if (unlikely(!mmap_read_trylock(mm))) @@ -756,6 +901,9 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } + mm_target_node = READ_ONCE(mm->target_node); + if (mm_target_node != mm_slot_target_node) + WRITE_ONCE(mm->target_node, mm_slot_target_node); now = jiffies; if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -763,11 +911,17 @@ static unsigned long kmmscand_scan_mm_slot(void) VMA_ITERATOR(vmi, mm, address); + /* Either Scan 25% of scan_size or cover vma size of scan_size */ + kmmscand_scanctrl.nr_to_scan = mm_slot_scan_size >> PAGE_SHIFT; + /* Reduce actual amount of pages scanned */ + kmmscand_scanctrl.nr_to_scan = mm_slot_scan_size >> 1; + for_each_vma(vmi, vma) { kmmscand_walk_page_vma(vma, &kmmscand_scanctrl); vma_scanned_size += vma->vm_end - vma->vm_start; - if (vma_scanned_size >= kmmscand_scan_size) { + if (vma_scanned_size >= mm_slot_scan_size || + !kmmscand_scanctrl.nr_to_scan) { next_mm = true; /* Add scanned folios to migration list */ spin_lock(&kmmscand_migrate_lock); @@ -789,9 +943,19 @@ static unsigned long kmmscand_scan_mm_slot(void) update_mmslot_info = true; + total = get_slowtier_accesed(&kmmscand_scanctrl); + target_node = get_target_node(&kmmscand_scanctrl); + + mm_target_node = READ_ONCE(mm->target_node); + + /* XXX: Do we need write lock? */ + if (mm_target_node != target_node) + WRITE_ONCE(mm->target_node, target_node); + reset_scanctrl(&kmmscand_scanctrl); + if (update_mmslot_info) { mm_slot->address = address; - kmmscand_update_mmslot_info(mm_slot, total); + kmmscand_update_mmslot_info(mm_slot, total, target_node); } outerloop: @@ -988,6 +1152,7 @@ static int stop_kmmscand(void) kthread_stop(kmmscand_thread); kmmscand_thread = NULL; } + free_scanctrl(&kmmscand_scanctrl); return err; } @@ -1044,6 +1209,7 @@ static void init_list(void) spin_lock_init(&kmmscand_migrate_lock); init_waitqueue_head(&kmmscand_wait); init_waitqueue_head(&kmmmigrated_wait); + init_scanctrl(&kmmscand_scanctrl); } static int __init kmmscand_init(void) From patchwork Wed Mar 19 19:30:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C98FDC35FFA for ; Wed, 19 Mar 2025 19:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1C4C280017; Wed, 19 Mar 2025 15:32:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA472280010; Wed, 19 Mar 2025 15:32:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAB41280017; Wed, 19 Mar 2025 15:32:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9842A280010 for ; Wed, 19 Mar 2025 15:32:49 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A450B1A04F2 for ; Wed, 19 Mar 2025 19:32:50 +0000 (UTC) X-FDA: 83239298100.30.7823841 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2067.outbound.protection.outlook.com [40.107.223.67]) by imf23.hostedemail.com (Postfix) with ESMTP id A4FB0140007 for ; Wed, 19 Mar 2025 19:32:47 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=228ZUA6S; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf23.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.67 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412767; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yMVd8QbmmNVnERYroypm9kZTSC/0smb5CRgcSN7QW8I=; b=ZWIjFuIGoZRYem3a/ywiywb8oi2CbP0wqJWD290Hi9ddofnw5j/7xvZ6sJhxvddmQjPUjX Y/jAThvMqs75G0cFaAqILu/OsSrNwCDV+zZnqN4xcLNUGtCnXhDj2P6OHc1mrRdDzuds4Q 0HTP1NesWKK02yChw5yr1gZc/w2KTfs= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412767; a=rsa-sha256; cv=pass; b=EwnjPRVXG/X1oBa27xSjYr7aDEfYvp5AC/UY7hM7euqRqcIrypXV8NB2APYB3U07jOrkI9 8scaDs86EJhd65shQkLr84A9eY6Ynfuivj3q2zPnJk6La0vOEJlh228XNq1VGgrb45MNHn e+QwFLKf0Vvs1Jr+8rhz8cG6E8+t7vQ= ARC-Authentication-Results: i=2; imf23.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=228ZUA6S; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf23.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.67 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tYjFmh+W2io2rjBHEVxDI9GaaUKZOfDf5MVJ6g/AcjxXkmJ6xKEV05al1ShoP8XYOuHv6t1atVPULsXy50t0RL7ZWpwZmyAi+XrlH+xkd4yVZXOHYF9RzdjMyqTjfG4GrN1ra4Df36Nyhi0k/TYVQAL8GxC46j97IB0NELdxpIMD9JHD7VWYn2jGJEsnuIjcb2BSJ01AzfLWlDBNoDGnM+VgDXUnG4M5AxB7/nKb2ODYBEEpQR9CB7uc3aWTWDw7OJoQjBH3lXR34llP1krMm2kfxq/c3Wm+vPyBWjeS5s7+mW5m7SkqCdPMvNdKtHWPcbKWGmylKaXOe1nJ2pZQLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yMVd8QbmmNVnERYroypm9kZTSC/0smb5CRgcSN7QW8I=; b=gnNADV4GF1XTZncC167hF97yk+H3xBi8RipuLK3gJIReuRJ2+BmiewoFT09r92uAJB+VTR2rWArbOJJ/kFnt/SZhsv3P290XZ3nvpQ+igHYznKvL/eeQQDXWN6p6TIrJfgYzVaKHKM2C/0hscytaTuRs9D4TyhlW3XdE2Suw9qXnxZNR1zH3ZKRx5lnHaXzErFG9twqeDGcYdI4jI+Y5R/7fiyziCdC/4kJNW+eXamfqUKSJtehlJd3Zc2v3/mF0t73yzGdNNrId7AbEGOH8n1iWzQlqFEZLo7bcrE54SHrSwO01Zk7wcO2dv+iFEZ114jHKgigSf7i9mHKNQBhTNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yMVd8QbmmNVnERYroypm9kZTSC/0smb5CRgcSN7QW8I=; b=228ZUA6SiY0sEEmsrKpVoGHg1E07YX7wiZa+7UpI2obR3Buce0snpVjCd3v84qJ6VtklMC/BT8jokyFhM7uV/nvv5hgM97XdFciEa3cxSpXSf22L3MIaQ6miIDmidmgzoAC/W8GP2uxgFTmNLW2JqTnNoEtVi4kynjR3uCuAbIE= Received: from SA9PR10CA0001.namprd10.prod.outlook.com (2603:10b6:806:a7::6) by LV8PR12MB9229.namprd12.prod.outlook.com (2603:10b6:408:191::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.33; Wed, 19 Mar 2025 19:32:42 +0000 Received: from SN1PEPF0002636E.namprd02.prod.outlook.com (2603:10b6:806:a7:cafe::96) by SA9PR10CA0001.outlook.office365.com (2603:10b6:806:a7::6) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:32:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636E.mail.protection.outlook.com (10.167.241.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:41 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:31 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 10/13] sysfs: Add sysfs support to tune scanning Date: Wed, 19 Mar 2025 19:30:25 +0000 Message-ID: <20250319193028.29514-11-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636E:EE_|LV8PR12MB9229:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c35ec17-c9d3-4ec8-20f0-08dd671cd0ab X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|82310400026|1800799024|376014|36860700013; X-Microsoft-Antispam-Message-Info: d9gCgkgz0Dqnk0NRtvaDMsDCHBlj6IEcm7cax/yWMTCEsoCu86Vq787p/SyYH9CAFSe5TGdhbkVGkygkaDjqmRgVUq6r0DxQtR3wRziWS9tLzBwn5IMJtch6AL2gWrRkFBY+isXKvoIS1uLLzHU3lz53KAs1ShGMCiiHnr6g+Z2JL3dLm0HmRNrHqcaH94C1nnizTPb+icT8kXE/xg1VQjWiJpDQBHTnF8zEag2kJaw4zfN5nUsTpmziatkA/+a/F5STen8AmqSSMtJuH5Nbku0oipS5ojGcc6F+jeHXd+tZy323i0zPBIzGBMz7Ojaf+ViCa7OV4Y9pN4CJG3JWk0SyLB1bqDdV8XhTm1WsG6C7wIGoJlvYIcyIhLPjZZB8erl/g3P888/gi4nprRnJRmVpzqpaNsBB+DbnaxPjAyOoTcr3s7fjmlXO0eHZHRFqRzVCw4qle4MVVaDKEnY3S68ZhFs9irmp0Uhq6btt+pccqSpqPTaFuNHppEIBKdHckgHZdLPimGUO2sfdGunX5IgZVRqVL0XlPqcdGwkXV2tKbvL+MEkdWscgPXfRlEX4IH1n+xV/xpY5m26UAsyUnn8pz187+pmSTKb5T6uCcbcHes2Ly4hzhxva4vwbfbP3gMo1rEKaMVz1L2dRFjZuqX/x6a+RVCqvmQlQK1gIjckjkt5w5XjGIEFJSSioLA75ucFTGKgpwwgpCCuOMFhGUjATw6vpUPgsfoKTVz6ea7xR6v4eBGb7njXiKX4wYlMlU+Gk/COBRUfOMEvk/faY4gC2MHPzfXzuAT6RTrQQzNxAQ+vYyoV45CezyEtYRtwhfckgOr/Q9pAypYJOYdYZdiW6wRVDDUrh/B6F9a4BpUso+sTwWpFXU1RQFQg7SOJD6htqxEfcmwllM3DqaWsTiRfwArNnUNV00VuD5Szzzwlkoo9lIRv2FCAtxgiz76PabMmvUxt8Ed6cX1eDX6c9Ic9L0B0QAbo7z8cRvtpykXlPYIGif4cOZZjgiXuud4y07eTxcjHbkz7+4soKAyiQjQXYjbjZUvJdx/zqotc1RtHIYVDDjnAc5vH/Cau842QSHVcnDhEvRJD+OexsjKv10DpU0bDBzA9miK4tKFlbZu2YFWHVPu+RUpgOLySBOdpVBlk5BCZIeCVaDwv8Rsmzt3dSy5HSgDlHXBzj6GLoa8ln9hZpdlKNX3eVvj18dnfIDbc+eIj7HIeXujRpJaaXd8iEs+FxibM+XWTMe7ZQkIZjFvNt0Kn18eougjJH0gkMh6ebIkDf97wssyhibTdcBJ8w75s1wlqp93PorKToyZQKqtpPlMSgv1A7pxYXx9XD8nybc96J+MkdkJtOSHxYnR4FfzBuFUMLgseufochu8clfuqOBtMHNqPRAGJbkEZDgT4ELy6BcYa5Rfgq2VHvIqdvNxIKenmZFratRa/lJyx8mUqYi16mnwe48ExXM5Qi64FVy3GquY+rMcHg6/urkA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(7416014)(82310400026)(1800799024)(376014)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:41.3162 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c35ec17-c9d3-4ec8-20f0-08dd671cd0ab X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636E.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR12MB9229 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A4FB0140007 X-Stat-Signature: pw7scz9jc473zkq3zfrq9qj4xe9etuf4 X-HE-Tag: 1742412767-907123 X-HE-Meta: U2FsdGVkX1/FvA/7lhIyz7O/Xk0nN8NJrx2q+OSJ/P9qB7psUY4FMiOFp0zyVNJVQIeqOruNKWhf15Vy4XKY2j5umZf8ZaR4pFeJJT64ZvJ58N06HEDQIpvbpAWpoMYyMtd3KHHTTVSdn+r4PBCg1SQj5PfLxS6dZM1iVdlZUyx4lVVaWX6VkfWKX21B9BpzfisRCd6O/7JIfL9B3WhKMU667XDyB4uwBXEwuHoO/T29xghFfwm9k95qkjfHF1TAauD6BGiibVCgNDZtFcCIW8Rw4DePVstFyu0vdryZjtswo2eXgrOxpUT9z+Vmkd/fHO8XBfhJESid4iQuD9CTVwYV7/eL+O0eP35Cvn4p/VeOFBuUi/tGoB4bTPUrhCaO0JEY3KAFohutfqi1ztgPLIrSKFltgKre8QT+iF6Vw2y/ycPSgEn77kzsZ7UbGHnMRZDGqhR8nSoba+bxSPO0CLs/mm2jdQ62uk3CZ+TpJ0ZknmuixEvAwh/RsvlY2PIsyWyM1Xv4zxHTgEeIye25D/7yNSFsFCuykQaKVrB00dHmTz4zKXhSj+VO7HiwBJ3eVczlLMR9PC0PDx9aihM4/0OIIcszgFlGZYXteWUoqEVmt+l2WZYY2+13OlgA0fF1R1o14pFtxV8Lnzmz6qfzJlwh2VE7e+968cNg+J+1vW9UKe+K+UpbvHo1bMW15/Nx4Wcuu+4mmF0zqnev1ZsNGhzSKm9jOfsXElas1dYBa/NCuhLsEMtD1LU71WWHtAn+UTDskC+uO7rI3ecKJstx3fKd45OV4uSBHK2Jj0qIW5AHsK4VF2ORDsyY196jJ82aWGlOUhoZTQBnQpF5oqX5eE2emZ69f7Iy5wZ25GhS86d4A84PsMJNSdpHRToay02lC8fOtTZJy7KxsTK0nPyHHI2ySzi3NpO1BCJqc/abnPHqwqh18qfwiAATSO+Y5eoMuZ9U6KEFUxvoMe17GDy ZVUJZkTh JDl7n60A95fcs1VLgCUzI5USFNvI3S0S+GOAPdtz/fbZ7IVJ0wcsv4oA67Ho5jNoNcceoXLBaBMQ4MjI1yldGmkXik/0eivcXQA2w1BUwKaO/kX33DLyAIE8wuUqS9/gQJ6FWt3CkmBmpD5GlTrA2y+pCME/c+GOzh5E51DwWxbVK9SGrGa2jOaS+S7UJouVdvf9ghaw+3UCa5DsLpfXMdrrbXuFcDuMu+jaAysznUOsxXsWaKgmfDL/NLrzyVprndH1mX+IWajN9W37YtA3+RJwcoSiKcgBokgrBnEAlqd1BQHlfu4HIvp8M21t6eCcP2dgVHzl6kHGXGZcFsXjGOlVMxnPJ4OFPm51tv0UDwQc3+l4nNSvrIzpg8Nhg0koBKFenfb8SJY8Y+UlFx02mcUCp3Y8bnSPWbLLIAXe6uizuONBBGSUAoHtik0g3GzUKbJzaOBnmStaBxYK7oUmazz1mydkVvhynG8L2q8TJUi+h7hHZvon79wTuazlA8td0nPia2Mc6otJHjzE6pGDs58yeVY0rRFCUoyKqgf3+8lC4EB18wSyKkJhVXtTkmoH51OGI1LUJ6tj5/tKLEb4TSYI0gQV7nixuc69Pc79F68twhzTqwl3iB02D96aXXbY/S77RTGpPPan9IWzGgivp86CHr5gOpERfiYBoi6iAux3GdmYWbe6+6Ot741tX0fmc2J1s10zI1zZ/SQsHVR6CHqPg2CWBQWc/gppNoMJQxFkCGD4kOQTbTqkhxXytb5mVKzkU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Support below tunables: scan_enable: turn on or turn off mm_struct scanning scan_period: initial scan_period (default: 2sec) scan_sleep_ms: sleep time between two successive round of scanning and migration. mms_to_scan: total mm_struct to scan before taking a pause. target_node: default regular node to which migration of accessed pages is done (this is only fall back mechnism, otherwise target_node heuristic is used). Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 206 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 206 insertions(+) diff --git a/mm/kmmscand.c b/mm/kmmscand.c index c2924b2e8a6d..618594d7c148 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include "internal.h" @@ -158,6 +159,170 @@ static bool kmmscand_eligible_srcnid(int nid) return false; } +#ifdef CONFIG_SYSFS +static ssize_t scan_sleep_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_scan_sleep_ms); +} + +static ssize_t scan_sleep_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs; + int err; + + err = kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + kmmscand_scan_sleep_ms = msecs; + kmmscand_sleep_expire = 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} +static struct kobj_attribute scan_sleep_ms_attr = + __ATTR_RW(scan_sleep_ms); + +static ssize_t mm_scan_period_ms_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_mm_scan_period_ms); +} + +/* If a value less than MIN or greater than MAX asked for store value is clamped */ +static ssize_t mm_scan_period_ms_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int msecs, stored_msecs; + int err; + + err = kstrtouint(buf, 10, &msecs); + if (err) + return -EINVAL; + + stored_msecs = clamp(msecs, KMMSCAND_SCAN_PERIOD_MIN, KMMSCAND_SCAN_PERIOD_MAX); + + kmmscand_mm_scan_period_ms = stored_msecs; + kmmscand_sleep_expire = 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute mm_scan_period_ms_attr = + __ATTR_RW(mm_scan_period_ms); + +static ssize_t mms_to_scan_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%lu\n", kmmscand_mms_to_scan); +} + +static ssize_t mms_to_scan_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned long val; + int err; + + err = kstrtoul(buf, 10, &val); + if (err) + return -EINVAL; + + kmmscand_mms_to_scan = val; + kmmscand_sleep_expire = 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute mms_to_scan_attr = + __ATTR_RW(mms_to_scan); + +static ssize_t scan_enabled_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_scan_enabled ? 1 : 0); +} + +static ssize_t scan_enabled_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + unsigned int val; + int err; + + err = kstrtouint(buf, 10, &val); + if (err || val > 1) + return -EINVAL; + + if (val) { + kmmscand_scan_enabled = true; + need_wakeup = true; + } else + kmmscand_scan_enabled = false; + + kmmscand_sleep_expire = 0; + wake_up_interruptible(&kmmscand_wait); + + return count; +} + +static struct kobj_attribute scan_enabled_attr = + __ATTR_RW(scan_enabled); + +static ssize_t target_node_show(struct kobject *kobj, + struct kobj_attribute *attr, + char *buf) +{ + return sysfs_emit(buf, "%u\n", kmmscand_target_node); +} + +static ssize_t target_node_store(struct kobject *kobj, + struct kobj_attribute *attr, + const char *buf, size_t count) +{ + int err, node; + + err = kstrtoint(buf, 10, &node); + if (err) + return -EINVAL; + + kmmscand_sleep_expire = 0; + if (!node_is_toptier(node)) + return -EINVAL; + + kmmscand_target_node = node; + wake_up_interruptible(&kmmscand_wait); + + return count; +} +static struct kobj_attribute target_node_attr = + __ATTR_RW(target_node); + +static struct attribute *kmmscand_attr[] = { + &scan_sleep_ms_attr.attr, + &mm_scan_period_ms_attr.attr, + &mms_to_scan_attr.attr, + &scan_enabled_attr.attr, + &target_node_attr.attr, + NULL, +}; + +struct attribute_group kmmscand_attr_group = { + .attrs = kmmscand_attr, + .name = "kmmscand", +}; +#endif + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -1036,9 +1201,43 @@ static int kmmscand(void *none) return 0; } +#ifdef CONFIG_SYSFS +extern struct kobject *mm_kobj; +static int __init kmmscand_init_sysfs(struct kobject **kobj) +{ + int err; + + err = sysfs_create_group(*kobj, &kmmscand_attr_group); + if (err) { + pr_err("failed to register kmmscand group\n"); + goto err_kmmscand_attr; + } + + return 0; + +err_kmmscand_attr: + sysfs_remove_group(*kobj, &kmmscand_attr_group); + return err; +} + +static void __init kmmscand_exit_sysfs(struct kobject *kobj) +{ + sysfs_remove_group(kobj, &kmmscand_attr_group); +} +#else +static inline int __init kmmscand_init_sysfs(struct kobject **kobj) +{ + return 0; +} +static inline void __init kmmscand_exit_sysfs(struct kobject *kobj) +{ +} +#endif + static inline void kmmscand_destroy(void) { kmem_cache_destroy(kmmscand_slot_cache); + kmmscand_exit_sysfs(mm_kobj); } void __kmmscand_enter(struct mm_struct *mm) @@ -1223,7 +1422,13 @@ static int __init kmmscand_init(void) return -ENOMEM; } + err = kmmscand_init_sysfs(&mm_kobj); + + if (err) + goto err_init_sysfs; + init_list(); + err = start_kmmscand(); if (err) goto err_kmmscand; @@ -1239,6 +1444,7 @@ static int __init kmmscand_init(void) err_kmmscand: stop_kmmscand(); +err_init_sysfs: kmmscand_destroy(); return err; From patchwork Wed Mar 19 19:30:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02013C35FFB for ; Wed, 19 Mar 2025 19:33:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BC90280018; Wed, 19 Mar 2025 15:32:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 045B3280010; Wed, 19 Mar 2025 15:32:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB44A280018; Wed, 19 Mar 2025 15:32:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B8FEF280010 for ; Wed, 19 Mar 2025 15:32:58 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D93A0C0728 for ; Wed, 19 Mar 2025 19:32:59 +0000 (UTC) X-FDA: 83239298478.02.20C1B28 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2086.outbound.protection.outlook.com [40.107.223.86]) by imf18.hostedemail.com (Postfix) with ESMTP id F31441C0006 for ; Wed, 19 Mar 2025 19:32:56 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=ZiIXnefF; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf18.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.86 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412777; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sMsrH2gwsvfzOvnhHaikDdLZ0RBu7hhS898jqAkzLyg=; b=tfNNReV2iD3RgkDVYvZVqLjtTqsgsCuUucTN8qDBMiip4j6DHvQ27LLcuh9m3rZvhwl0VB scECUWUATmPQv8poVlFkkcja04kngG78NQL0UmAd5VW2xCHsyZrkt6lRvShMkxa9UBbx9V 9T8dcZfR+7/tdpd57/797c6kHgrFXrY= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412777; a=rsa-sha256; cv=pass; b=wpe7WhtluELPlYv8RG7AJQS8iI5LHEzcTK7ShA4YaK6zlVqJVZOpdCTDmA0GMmsgDNN160 sHM8k0q1P/hYK2uMO5m9mlhsdjifgRTeX0HxNe3cKbcs7tM068YciXvUGJck4bXpnLvSBH OGCd/+meL3jGseaOxquut9BHOoiQbg0= ARC-Authentication-Results: i=2; imf18.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=ZiIXnefF; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf18.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.86 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tcrRp8ibA1L8lZ/lJnU4UdcQBLzd/+9sMG+ccYx7+EaPlqpbtKCJICX46n2lR04M9nnZ3vf7CrY9mPh/gunz+BCmqJliF5PGmivbUDU2XTd1v6giuRWaTQalSKjMA4CLeg+lpOJKradrjhQTfnhPT9aHJzbsyL2Zb/bZn031Zh4LaRC+SnUk9dNxl/iH0yq5kfrFfFwERNcPFvojEgFbs5n+1HYf2Pwugm8226fGLOYHXTUEgJ/T5H4UxA2yUW/8klPVgmd8nplT4/P+QZLOih2tOBSSUu+lPxfFdz8/Eozu+TTzHMK3FZPeqYDlH2gUQd34f6YY/advwepsO1cceg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sMsrH2gwsvfzOvnhHaikDdLZ0RBu7hhS898jqAkzLyg=; b=NrJGMPqUjR8GANHmbt7fv8A8LFVqXVUbhOO34OKSPX41wf7xZcpjQms2qOl+gH8VCgsV8rduWI3ApVygX2qO03vDqg+kx+x3jW5ot+ezAi+PiqUYE5mj+LAgyHlVacqk0VunZMEtXew+FXm6dHIVeVOalXLstbR0xKZYvjgV2xTRy2RE4WNrrMmmnI7HUWLEu6sLZOOiJYjpiaWT429ROKLH6rNlFmzSguJgsZBDGL4hIUsapsFxk3BVVxuesec7QHYgtSeiqc0QfjNxVbT5YvHCC0/5pT5z2aPKtSQzrAfOmkdWjy1sQCLb361WPx+0Zecyr6IJRsN6x/cx66lWnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sMsrH2gwsvfzOvnhHaikDdLZ0RBu7hhS898jqAkzLyg=; b=ZiIXnefF1S/ioqWWZlOnbYguX0sEyJgPjFz3aZWjvwM7Wy5HE//UoVmGR2sKYnyvWuLAs4YtmIbXnADJT4A5rhexLE9gbmmWEu3jwnyJ3UaBcnLdWztiSMnJQvPoZHrcxLnnCuqSFq4+tdmLoAUMfO18sAeFpNLUuLEg00VpePs= Received: from SN7PR04CA0024.namprd04.prod.outlook.com (2603:10b6:806:f2::29) by CH2PR12MB4102.namprd12.prod.outlook.com (2603:10b6:610:a9::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:32:51 +0000 Received: from SN1PEPF0002636B.namprd02.prod.outlook.com (2603:10b6:806:f2:cafe::2f) by SN7PR04CA0024.outlook.office365.com (2603:10b6:806:f2::29) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.33 via Frontend Transport; Wed, 19 Mar 2025 19:32:51 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636B.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:32:51 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:41 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 11/13] vmstat: Add vmstat counters Date: Wed, 19 Mar 2025 19:30:26 +0000 Message-ID: <20250319193028.29514-12-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636B:EE_|CH2PR12MB4102:EE_ X-MS-Office365-Filtering-Correlation-Id: 187ab32e-35b4-463c-0133-08dd671cd69c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|7416014|376014|1800799024; X-Microsoft-Antispam-Message-Info: 388euDnwXyJ8MZTQe0dJ4kEIxumWHeEgVXY4HIQTOOofsXvWc6We2QmB0VvyH07J2lUgKahX1szjHK4TErKl3Mx6UfEeprEdcCJLkIqcu0QHMBjo2Hbqd0fqxULwriLDR38n2JU5sJSacVQhW2It7nrzRgFDS++UCnPxMmEisDXRfnxYkAMwFGl77Nx30wj1Ls767ZMYhLnRY9ElzxGjnkEJIn0CR/tYQB8LHKSlOl0A6FHTTibU1qRtoPAS49sMqHPNAjk8+Uo97Yjd0jpU8a5zSfyBRAVixs5g0k53DI8Ue3OcD/AtwEZKLNXtPk4MtTDcp6W+yysj4z3AQ4jUGP03J/C9cR1C40dc6+4b1kv8sXrnDQR66josTGcSLJumFuVN9tmm/Dbg7YTTWv3afP1gMGmPXonMXDBmwosibZRuCRKpQ3qxU07bUV8a/ikJsTcwczVENdr5NzZRZbjpWxSkhWeAyzHz/qVawpa9QXhSK9eZ2Mot9sVXI1iP2Ha9jXlBkAsyRGZEsGT4MrgIK0MtMA4+inktlHgCmvJbv2QkVr+6h6Pd7IxnAviuK2KAJD6odTP7JTs35ouU0RGKUh4ZlhoOmkS19GR4iAPP4ZXDej8bvdZ7ik92XrbboibiDyG1oR7XM4cbpRYnc04EwAvXyEAbHZYq4e6hOD/MAbMKKSPtqIoPz+2b3KhPRfNjg+2fM3/clWltBUw4FsrHgR+/2PqSiKp/DRmcFo0O2wYKHPtbobAZwQhzrBSfZySlcANOI1nPeYLKSwpbpRYpCzITbIhXUybeAXMjtWcKdjlDJ+ExlGD9qyauhrtTu+LFZ0Aq1wLf/2Rz3RPKgOgyMUiC7rnwgFXnklQLZisd5DgPRgjBSckq4YQrcOSSACW/G9cPpLnYEdHZL4wyZ5W2BfKYYfMWsUp5ij9cYDb4T8nC0GN9PXFyqumYkDBi6KjUatNmWChBShsFyFRmcVXfg83P0GrIGLhH/Lf2H5qOPIbxXey0LpwWxMSLdnms40tBGOabFRNrbLJM8OkZiXfO6q2WjCCrwj0U2VW+Fr/oaxm7XB1qJT575ybgcsZ9KaL5YtU35yeFi3a9nisgsl3pFIjijSvGcx/IeLFup5ez3lsHE02V/ZGB54pmqIF/II1xr3TOPX7akjlwsUdglLFUPqwYFTI9TPBBZDod8J2PTeBG/Z2ATvdPh7ktUyAqdq2jrvmowdcOotOCP6synpx/rciBBg5o3AT3g3zMmVWntniMGpQa214qQ5PGYPzpuXJPxWlAirBYTZNZMdXH52h4+gOHDzEq3L+kCEpGhs9/Yko6Ac2zK6Ik5TU8Gs65HOAzhbrPsRazz13Ud2dKbwyTpcWtax+DCqsABQ9kKKpgGj0ztja9TqisYXMolo1ytl6KlJtvIO2Ydk9sgowvq+X8zZ+UvfuwI8eJpVMhHu2Zs9cXdQZtMO9yJNp4WhRe1deO2Te+id6IOnLo4GZxAidDLNTNOZBMbNmjE3s/ySHpwOE= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(7416014)(376014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:32:51.3146 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 187ab32e-35b4-463c-0133-08dd671cd69c X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4102 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: F31441C0006 X-Stat-Signature: n7gd7fhixniuqs7acy13yhbyqqz8hueb X-HE-Tag: 1742412776-201590 X-HE-Meta: U2FsdGVkX1+0Cp3vJKhKv77buLzwHXziZgM9qJfPcoD2+1Wz8jy9QPOJZ3NAJf7G1/40GBvSs4nhFJsHVba7f1GktVZK3PtNrBRsh/kX3dp6PM03ZmzyPKGH+jOo2Ch2SDJjnDG4Bazujrvv8SOy+RZVRlxbEEcEFTMfCS1Da7IV5flDmHLHwptlf2TUCRXB2pMcQftJFiuxLyoGsu7ctQghf7zeur2Lab6GjgQiiTGsTlwMmNiruYK4JfFVyRmuFl7pWukuLXWxfAqGDsU/7l/9BDbOA+NMSg4lIzkDWZro6CJ1z6EX9+RL/JbP4H6YtedWth4EBr6vbwUetZb3Lwwvwg8Wtnp3H3ab1RsuV5sjPsfGquZspmHvbyELXaatPXje4yd2rIxh043r5zAMdzniN+XLR3OI/+h/eHWmPvrpbLr8hHPUzAJvLjjrTBTVRVVeF1jAOfrkkK2UYSBzqnw2NBkeaNe24B4C8YBcXyT+Px4LtBNV/9fVFobiUrMBWL5HHYIs1tCBQpt24tHa12xkcCDYxGHuTqn3NDhsfrff+pioZ6GvR0UKudjqqAKgwviIFAloEBYRAxaIHiDdZR3L7/JL5vsa1giTgwOaHiO+bFtzxYSbpR26X9LnwDc8thtXIJvG6byc8rPyd+wun4ZoWGAxu1/E0G/hjvY3JCU6pUnBZxlwaRinVYA/omzFJkrEp5GpjWcXwZI06wTuCdA1admkol3ywfbaRUmwUrbD9RfGJGlUZH1o3nwqAhAJg7QI58uGURq7TiqbYZ/ldsBZNQVGgxdBCHm8B3jVSRt9243oY2cAzhUN9MgLhf84JK21gflD7eyD7R8BwLrKabAb03sqLpWn70C5ZbddPfS7L1tat6/NzWzBKGoWMTqfgOW1Ps2rrqOdnmHzvTgUBNIauhT6Rd5Ei0uRDZY0024aKVMNfQ9BL0kr9Zr/X08zqdL2Akgd1L1cri6Xtly hiHI5Xop GjEsgiXaitdKv7fob8cqO0p/6F8VlUB8FpiXKjdSWDGW33LiyiMhc/fBIgBX1ZxxVrIdLUe/BOdFQbEU94aBG7KlaoKj47IKfhRzXLEG9Hz5O5OEPerYJ9ZMuXvAM8T64FAfN5ycKTUgus7wN8DAQG4Qz8ibCZGMdj8TX0aXmA5oDGF7Du2FCBOoAuYzyD/mMzIhZJELuw5J+DOh+XcUccIlpm65HSrKNIzyCNbw9h+VEt/g9LQ1Pzr/HyX4/ZvXlRcM9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add vmstat counter to track scanning, migration and type of pages. Signed-off-by: Raghavendra K T --- include/linux/mm.h | 11 ++++++++ include/linux/vm_event_item.h | 10 +++++++ mm/kmmscand.c | 52 ++++++++++++++++++++++++++++++++++- mm/vmstat.c | 10 +++++++ 4 files changed, 82 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b1068ddcbb7..e40a38c28a63 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -682,6 +682,17 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_KMMSCAND +void count_kmmscand_mm_scans(void); +void count_kmmscand_vma_scans(void); +void count_kmmscand_migadded(void); +void count_kmmscand_migrated(void); +void count_kmmscand_migrate_failed(void); +void count_kmmscand_slowtier(void); +void count_kmmscand_toptier(void); +void count_kmmscand_idlepage(void); +#endif + #ifdef CONFIG_NUMA_BALANCING static inline void vma_numab_state_init(struct vm_area_struct *vma) { diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index f70d0958095c..b2ccd4f665aa 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -65,6 +65,16 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, NUMA_HINT_FAULTS_LOCAL, NUMA_PAGE_MIGRATE, #endif +#ifdef CONFIG_KMMSCAND + KMMSCAND_MM_SCANS, + KMMSCAND_VMA_SCANS, + KMMSCAND_MIGADDED, + KMMSCAND_MIGRATED, + KMMSCAND_MIGRATE_FAILED, + KMMSCAND_SLOWTIER, + KMMSCAND_TOPTIER, + KMMSCAND_IDLEPAGE, +#endif #ifdef CONFIG_MIGRATION PGMIGRATE_SUCCESS, PGMIGRATE_FAIL, THP_MIGRATION_SUCCESS, diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 618594d7c148..c88b30e0fc7d 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -323,6 +323,39 @@ struct attribute_group kmmscand_attr_group = { }; #endif +void count_kmmscand_mm_scans(void) +{ + count_vm_numa_event(KMMSCAND_MM_SCANS); +} +void count_kmmscand_vma_scans(void) +{ + count_vm_numa_event(KMMSCAND_VMA_SCANS); +} +void count_kmmscand_migadded(void) +{ + count_vm_numa_event(KMMSCAND_MIGADDED); +} +void count_kmmscand_migrated(void) +{ + count_vm_numa_event(KMMSCAND_MIGRATED); +} +void count_kmmscand_migrate_failed(void) +{ + count_vm_numa_event(KMMSCAND_MIGRATE_FAILED); +} +void count_kmmscand_slowtier(void) +{ + count_vm_numa_event(KMMSCAND_SLOWTIER); +} +void count_kmmscand_toptier(void) +{ + count_vm_numa_event(KMMSCAND_TOPTIER); +} +void count_kmmscand_idlepage(void) +{ + count_vm_numa_event(KMMSCAND_IDLEPAGE); +} + static int kmmscand_has_work(void) { return !list_empty(&kmmscand_scan.mm_head); @@ -769,6 +802,9 @@ static int hot_vma_idle_pte_entry(pte_t *pte, return 0; } + if (node_is_toptier(srcnid)) + count_kmmscand_toptier(); + if (!folio_test_idle(folio) || folio_test_young(folio) || mmu_notifier_test_young(mm, addr) || folio_test_referenced(folio) || pte_young(pteval)) { @@ -784,14 +820,18 @@ static int hot_vma_idle_pte_entry(pte_t *pte, info = kzalloc(sizeof(struct kmmscand_migrate_info), GFP_NOWAIT); if (info && scanctrl) { + count_kmmscand_slowtier(); info->mm = mm; info->address = addr; info->folio = folio; /* No need of lock now */ list_add_tail(&info->migrate_node, &scanctrl->scan_list); + + count_kmmscand_migadded(); } - } + } else + count_kmmscand_idlepage(); folio_set_idle(folio); folio_put(folio); @@ -907,6 +947,12 @@ static void kmmscand_migrate_folio(void) ret = kmmscand_promote_folio(info, dest); } + /* TBD: encode migrated count here, currently assume folio_nr_pages */ + if (!ret) + count_kmmscand_migrated(); + else + count_kmmscand_migrate_failed(); + kfree(info); spin_lock(&kmmscand_migrate_lock); @@ -1083,6 +1129,7 @@ static unsigned long kmmscand_scan_mm_slot(void) for_each_vma(vmi, vma) { kmmscand_walk_page_vma(vma, &kmmscand_scanctrl); + count_kmmscand_vma_scans(); vma_scanned_size += vma->vm_end - vma->vm_start; if (vma_scanned_size >= mm_slot_scan_size || @@ -1108,6 +1155,8 @@ static unsigned long kmmscand_scan_mm_slot(void) update_mmslot_info = true; + count_kmmscand_mm_scans(); + total = get_slowtier_accesed(&kmmscand_scanctrl); target_node = get_target_node(&kmmscand_scanctrl); @@ -1123,6 +1172,7 @@ static unsigned long kmmscand_scan_mm_slot(void) kmmscand_update_mmslot_info(mm_slot, total, target_node); } + outerloop: /* exit_mmap will destroy ptes after this */ mmap_read_unlock(mm); diff --git a/mm/vmstat.c b/mm/vmstat.c index 16bfe1c694dd..3a6fa834ebe0 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1340,6 +1340,16 @@ const char * const vmstat_text[] = { "numa_hint_faults_local", "numa_pages_migrated", #endif +#ifdef CONFIG_KMMSCAND + "nr_kmmscand_mm_scans", + "nr_kmmscand_vma_scans", + "nr_kmmscand_migadded", + "nr_kmmscand_migrated", + "nr_kmmscand_migrate_failed", + "nr_kmmscand_slowtier", + "nr_kmmscand_toptier", + "nr_kmmscand_idlepage", +#endif #ifdef CONFIG_MIGRATION "pgmigrate_success", "pgmigrate_fail", From patchwork Wed Mar 19 19:30:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5825AC35FFB for ; Wed, 19 Mar 2025 19:33:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65045280019; Wed, 19 Mar 2025 15:33:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FEC2280010; Wed, 19 Mar 2025 15:33:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45393280019; Wed, 19 Mar 2025 15:33:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 24FCE280010 for ; Wed, 19 Mar 2025 15:33:24 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C1B59A9AD6 for ; Wed, 19 Mar 2025 19:33:24 +0000 (UTC) X-FDA: 83239299528.10.86929B6 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2041.outbound.protection.outlook.com [40.107.220.41]) by imf29.hostedemail.com (Postfix) with ESMTP id 03327120017 for ; Wed, 19 Mar 2025 19:33:07 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=EmXCUdck; spf=pass (imf29.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.41 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412788; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oYUB9Y5oL/vnLd9PbKivHKRUxW1ZVjk0QHYKJpkjL8Y=; b=3S323AzMfNRUJh2AmDpTtTYDOvI1VqxNC5bYwl0v1UIFvPS+D38kuA6mNkSrqOL/J/xd5L DRDdTL4A7UTOn0kX5AiS5xRYZuyJxFRPWD7zUVVlGJ8jLw2cfuvzigE1LigiL+uyzAuwpp yNuQ5/dDu1FFmf+fmHcdFQ4rHD9ojEQ= ARC-Authentication-Results: i=2; imf29.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=EmXCUdck; spf=pass (imf29.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.41 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412788; a=rsa-sha256; cv=pass; b=TObPjaEcsLcVvZ31V4PzAwfaJ+OZJNv2h/od62kAMzlfmY1zXZ0egbKpotmAr4lwxWv51Y am+JGguPLFA2fAUCfs/kBOyfWaECDraDSgHUPfPthVcAxwmGVI6xuCZL/xvDKMhp81quB4 iIRJLTT0ciwbLgd3tcM9OP8Bxz1Ewss= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HbThxnNT3sDyTD+HFT8KeE/eohImjEKvRngChsKKkTiOtbkfhnCsOUhyfoHG5flfIeMJ9GJ3kev9wIp2MV6li87aQ5WdPwiH0APTpVN/bqjvJkIWF/NPhVZPEQiAD8aG6YJhNL+lAtd+DMnCMPFiv6E8oWpMOAhDSy4t27U4viaovusMxhQQl8pw/pnx6RHO5svpnvaMpS4PyCAtbBJHPflllzol/aFDi8rp9ZUX4ocZYFRV1Pkr9YKmKf3Gp8wIYuJnPjxijP9njQjpyprmS+HViIjhC8YOaBFVm6XTgM/CGpargSihJgE/t11rxRDWmeU7LQoBfM1UA52Mw8PkYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=oYUB9Y5oL/vnLd9PbKivHKRUxW1ZVjk0QHYKJpkjL8Y=; b=d+Z5ZmzGtq2Zq6ArdCC2/XgRAbBUugeKcm4QVHW0muQxgrVac4WzVvzUtRGkTzglIVsGE2RTinA9Ft4acsyrT2U913MpBMM84r31RcAwUdT5MR8B5c3/z/KuIdUP1S995RQl/ezcFxxLr0hpsNm4D1ZPjWEbrz8Yxi/+x3NcLbc1Qf1iyLk0uMtnJGxxgfN6zZltc8Pnm+9t85fB9ctXDeCKQC1LfTc/HfS87YFPrSLMIKxF3ESIloze/eG3cyN+ePWscAcKpEF6e+Z1Y9gNljiSfXrwub/uFVtIG/sQ0gSv82yIVf40m38dm41ePpbmxyIpUofedKPezq1VW/etYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oYUB9Y5oL/vnLd9PbKivHKRUxW1ZVjk0QHYKJpkjL8Y=; b=EmXCUdckkV9nMB7LnWlYx3I9uMajNBcX/RNm0oTUmK9BnLf5uKbp/MEyk8vd9vjpGvB82yNLWFDmJSQVhOM90DuocRFGE7kPrAd69iDlH4/a1AUL/EGgg09NMvmBEP5ssxUOf4VgzMASJgdj+mGVe39yRDaVP0nonGqdAOy7VoQ= Received: from SA0PR11CA0169.namprd11.prod.outlook.com (2603:10b6:806:1bb::24) by PH7PR12MB5925.namprd12.prod.outlook.com (2603:10b6:510:1d8::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:33:02 +0000 Received: from SN1PEPF00026367.namprd02.prod.outlook.com (2603:10b6:806:1bb:cafe::1e) by SA0PR11CA0169.outlook.office365.com (2603:10b6:806:1bb::24) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:33:01 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF00026367.mail.protection.outlook.com (10.167.241.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:33:01 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:32:51 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 12/13] trace/kmmscand: Add tracing of scanning and migration Date: Wed, 19 Mar 2025 19:30:27 +0000 Message-ID: <20250319193028.29514-13-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026367:EE_|PH7PR12MB5925:EE_ X-MS-Office365-Filtering-Correlation-Id: 868d5814-7e1f-409a-7c98-08dd671cdc6e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700013|82310400026|1800799024|376014|7416014; X-Microsoft-Antispam-Message-Info: NzZ3jZA9+096bwOrh/i5UbiNtTMFuIT766qHEUdYOmOvR4TiAetx8Mlu3EKdUhdDMc0RVMbHvtNrv08eGYJQF+24uL/n3D1hdgahLYysmOYw7IoCv95z4hbt7KvmpwQr+4pBZrOHs8qtVa2pLjXX6hzWzaNZK1t3drwEXFHKdc3SV0UoXSejkgC90qBVm+npYY3tqGK4ANsMkcX5TuffSPGqyhjezZiXN78YZdCREpya+0uZ+OfWmuv9BBdy3JLRviNZMXnVBfikXvsjUCV2srPGA0TUNsQ6u/HUFTGbMZGz30lek8Oprc4ktjb++9Oao6IJMTlLvuzNE1qTIDb1Jk92jNE7imGv4vEAAbOas5zU+0jaDbkA0vbwq/UJgKcZcnx7X59G+L+fdoViDJZgPOFrFpaY3lYXF9BGLUKZRv65xxrv6X62ijXKoHyWai5TQwpEGhSRrZy5T+vIHlbnIwi5LaHBlTuHVRS/8KrYrCjHkXdwM487lnpcXeZhHPsJ4lYybw5O/KbX6nTK8U5fQl0sGbF1lu9qKupdlwaEgfAqAdZTWibhzykoriPJEpNvxAT1h/vBB/Y0xqFLUiqtUBIBawE+8QwR3s4hd+uH4W3G/+mUYgmZOU/zc/5hkaT5aSGSeXOitNsETPdm/8KDeMs2SSGbjiffR1nmiqZ3ii9hU8SKdwf+mu4mpIQMk+Xgjycgd6G1KY1MCCTsFt3OKqy7od50zlw6pVzk3W3VUBgpaITpLs3o04RmLqSPFyOQ69xrTP0us6AWWJr4eA8bnCoy0TkreVPL82SLRXW0niyTlBWTp4+/9vdYXHGhep3R7fKKYsrd0DYf6HIghTIBHoXuooEDQHLH5O9Po5GaeaS1zXr2MznQEEXKJboP2pDoI539HIRK27uwzxybBP6wfGnSyRyxx3vUQsb6g83r8s3ctUwJxBAME7J3BjXUISImZrJINnIFQV2q4hBCHG0D8uL6JWo9H8SI3QrMvudne70xX12dGFcmvtpU6IeJTaYixhoIRjQ6nT0ew1yO2d5D8W0mSX1AqljvpSZWTUssPcW5HazaNbYPtLWot4F4swUMtJCcoliFtdONyae3TcJE4SutHclnJDUGyki7JY+3+Krm7v2f2sXM6bg6RWNnXErgneRcCjipBOpU5jmyEQo2axjybSrb//M0ZZRmvuYmm2NeEqKvOWmriHj81LODuV/GrYb5UFYZz3UKvRuU4WRKb9HJfq285t2lwR5gAEXyV2ESBnGEmD0G3fCfjy6vDgY7KTqGHmfYFD4Yof80SGPX3wOMx+kDVtLRoBbk5XeMncVqH1noXshHKJ2v1DxabaT8Zjd1NYD336V3ToFTsP9uZa4JGpAEy/U0E15N0NK9easAccvRAs+QB8QjKCrxsyOoRfoloEyvbo3uKR5fNvBA6m+t44o46KrUEdycAqcoXEUzBbqLadsWw343ar8f8fcSyiZ75ixBCOo5sP/bOPhH2lPvR+cNOQedkDWCFTTHUnQ= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700013)(82310400026)(1800799024)(376014)(7416014);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:33:01.0827 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 868d5814-7e1f-409a-7c98-08dd671cdc6e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026367.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5925 X-Rspam-User: X-Rspamd-Queue-Id: 03327120017 X-Rspamd-Server: rspam08 X-Stat-Signature: cbgze4i9ibgqdbkj61db9x7w8wwqxb65 X-HE-Tag: 1742412787-41018 X-HE-Meta: U2FsdGVkX1+X5N+31ybc1k5uWU7kyOVq+o7+gQdYlWHddv8b10gV8MG9Fm1T+QtS618td7vM8pbl28/njfPRrqqP3NVZwCXeQEQBCX8wZk1YVfRSDk4fg93OcjS22X6Qx5gDb5si5N5g1JROfzHtJKM6iSR0AXZ/3cgG5xlHykoAmx+NuqFsZUOIAi2D3FkwdS9aT4H77OxHxmdJCqUQCs/4U4Qsw4vs8pvwD4RPl9YCXjpKXp4MLWmM6wmS7fw1BGbvkDQTzVwP2pqjmPb/ruCLYBkW1ncr/T3eOXEIKnCjuMFuA8rLMzdMe55axVQ2hH/Q2RnWzWLU90dxTPA2l1psqR2yWYHabvCIPsA4AfRw5qK/mqYPV3CpyAGZQsnZdGJUwYhaxk3KN7QnMwxi55oF4YCNKR/3qZXPSgT9OLWQ8ohEcQbZNspDwI2xp1+SWgh09mY4qQXhiKTe0hPoZqo5HHhHPZRmTtr+w648Hb/GcGK8FQw/cV1ZtvcubTD49e69lKLc+p6yh69c4UNZ1Pxz7xi4SoLK9ve3Ew7s0F6YJAnwf6J++zy++5J2dB2qGdkpk6BTLUzjldco1BDI/bSZVLFETHU76AI5o9dFqCcGNdBNngk+F2VjIi6RJpX9FNUQG9MnuHpBWne2hk5uhIO4bsjFXtfUlgjdklxPwHB7AxNgzst8BPgowrnTs2vV2TcN5us+hn8ORTJZK9qiO7jqYLz8ROkZYjw0l2ULOnl6bs1a0Iimpo1bflLcW9GPyouHjKIDXsdecAFo+QBxS6mkBfzdK2vqC1bLdbKHMnmxuGuUk5vSRcZIFbfqpYm164ho7nM4gYa/ecBzgTyHnJB5pkt8qPQhYLFemxt5s+gfOrSin5/q9zCxk1I7LxtV5zYUHVheaJPiGwAafkDYA1fZkWPjVHYwrtBwO27+/NEO1QPM8Ev2UQaKFQIYR+rmmOkiWHhkPP1pu5R5zoj 740ETwFP SQakyLRPPAZ0w9AYYkc+8Sr3K0scTSoHx6oQmZiWuXMIRP9JdlsCSVZowh9XYsS6W4eoNGM53YkF0/Qxpg+UwbV3/4ryS9OIcuEK1nnIVn4IqdemiAHvNkta7eaUfKb5OMDaVLGhNHSHVY1ciSp7nj623jLlfQGGzcULTHTC0C3RBfXNOFPbAytmjAB4Rg2P9cdqkmcDWAtib3zDoRAkxBPYcNpJG9AdRe6BugQjMRtYZB8op+Vu0NuRaqO/gIHtQynWkJfgw6S68oBhc8fAu2V8vb6m0lM7fnl8x1c1jS0JXhczhqtVLD4KyMPUuYKTyYz2rWFjWpqqJ+jjWoc39kpCZItmVOPVfe3YGCuKqkFrjIOuSLv5mhj+DZumq8mP+7xvpWSJ7Wx910ugYbL49Jx2amjJWocnkyDeKfCyPcMYa7MDWVzwyD9LP0Ya6kjEldOVXThzNM5BWWmGbBFwysfoDrazoaOcxKk8dKt11YakPBtg8Gi8t1YSGDPeIkYKOs6QcocT8xHaSTL4Q4t1Wd5ISD6oGlhraho3oNyH3OB9yD8PmZvryhtAvGsSpP9cWpZUalkX3R/detaqfpN6h0jt0UEF0ahRLBrmToI8cuiWygj+Y7ujFxzuE90jBnzrpPpJ44oa8VULejUw/PxFF6jG6KlqVbV4sKEGg90FzVOAskgHpUj/RZGaPWtyGj51GNZ83n1j3eeAMg1Tih4Cyc7mkeknXA++ZIsuwwIugRTaehflScRhfvViyx2yo9wlKtnUZVPxEzuyDa365QKA8ntGSdeJfsrOt34Lnxx6HSwKlzUfZbrQiqxebVqKglHtlob1+oO+kEwlnDrsD7UoRQB9ZUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add tracing support to track - start and end of scanning. - migration. CC: Steven Rostedt CC: Masami Hiramatsu CC: linux-trace-kernel@vger.kernel.org Signed-off-by: Raghavendra K T --- Changes done based Steves feedback: 1) Using EVENT class for similar traces 2) Dropping task_comm 3) remove unnecessary module name in print include/trace/events/kmem.h | 90 +++++++++++++++++++++++++++++++++++++ mm/kmmscand.c | 8 ++++ 2 files changed, 98 insertions(+) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index b37eb0a7060f..cef527ef9d79 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -9,6 +9,96 @@ #include #include +DECLARE_EVENT_CLASS(kmem_mm_class, + + TP_PROTO(struct mm_struct *mm), + + TP_ARGS(mm), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + ), + + TP_fast_assign( + __entry->mm = mm; + ), + + TP_printk("mm = %p", __entry->mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_enter, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_mm_exit, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +DEFINE_EVENT(kmem_mm_class, kmem_scan_mm_start, + TP_PROTO(struct mm_struct *mm), + TP_ARGS(mm) +); + +TRACE_EVENT(kmem_scan_mm_end, + + TP_PROTO( struct mm_struct *mm, + unsigned long start, + unsigned long total, + unsigned long scan_period, + unsigned long scan_size, + int target_node), + + TP_ARGS(mm, start, total, scan_period, scan_size, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( unsigned long, start ) + __field( unsigned long, total ) + __field( unsigned long, scan_period ) + __field( unsigned long, scan_size ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm = mm; + __entry->start = start; + __entry->total = total; + __entry->scan_period = scan_period; + __entry->scan_size = scan_size; + __entry->target_node = target_node; + ), + + TP_printk("mm=%p, start = %ld, total = %ld, scan_period = %ld, scan_size = %ld node = %d", + __entry->mm, __entry->start, __entry->total, __entry->scan_period, + __entry->scan_size, __entry->target_node) +); + +TRACE_EVENT(kmem_scan_mm_migrate, + + TP_PROTO(struct mm_struct *mm, + int rc, + int target_node), + + TP_ARGS(mm, rc, target_node), + + TP_STRUCT__entry( + __field( struct mm_struct *, mm ) + __field( int, rc ) + __field( int, target_node ) + ), + + TP_fast_assign( + __entry->mm = mm; + __entry->rc = rc; + __entry->target_node = target_node; + ), + + TP_printk("mm = %p rc = %d node = %d", + __entry->mm, __entry->rc, __entry->target_node) +); + TRACE_EVENT(kmem_cache_alloc, TP_PROTO(unsigned long call_site, diff --git a/mm/kmmscand.c b/mm/kmmscand.c index c88b30e0fc7d..38d7825c0d62 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -945,6 +945,7 @@ static void kmmscand_migrate_folio(void) } ret = kmmscand_promote_folio(info, dest); + trace_kmem_scan_mm_migrate(info->mm, ret, dest); } /* TBD: encode migrated count here, currently assume folio_nr_pages */ @@ -1115,6 +1116,9 @@ static unsigned long kmmscand_scan_mm_slot(void) mm_target_node = READ_ONCE(mm->target_node); if (mm_target_node != mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node); + + trace_kmem_scan_mm_start(mm); + now = jiffies; if (mm_slot_next_scan && time_before(now, mm_slot_next_scan)) @@ -1172,6 +1176,8 @@ static unsigned long kmmscand_scan_mm_slot(void) kmmscand_update_mmslot_info(mm_slot, total, target_node); } + trace_kmem_scan_mm_end(mm, address, total, mm_slot_scan_period, + mm_slot_scan_size, target_node); outerloop: /* exit_mmap will destroy ptes after this */ @@ -1323,6 +1329,7 @@ void __kmmscand_enter(struct mm_struct *mm) spin_unlock(&kmmscand_mm_lock); mmgrab(mm); + trace_kmem_mm_enter(mm); if (wakeup) wake_up_interruptible(&kmmscand_wait); } @@ -1333,6 +1340,7 @@ void __kmmscand_exit(struct mm_struct *mm) struct mm_slot *slot; int free = 0, serialize = 1; + trace_kmem_mm_exit(mm); spin_lock(&kmmscand_mm_lock); slot = mm_slot_lookup(kmmscand_slots_hash, mm); mm_slot = mm_slot_entry(slot, struct kmmscand_mm_slot, slot); From patchwork Wed Mar 19 19:30:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 14023081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 492FFC35FFA for ; Wed, 19 Mar 2025 19:34:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31C1D28001B; Wed, 19 Mar 2025 15:34:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A4A0280010; Wed, 19 Mar 2025 15:34:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1211C28001B; Wed, 19 Mar 2025 15:34:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E841A280010 for ; Wed, 19 Mar 2025 15:34:48 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3C455804AF for ; Wed, 19 Mar 2025 19:34:50 +0000 (UTC) X-FDA: 83239303140.29.6A07841 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2048.outbound.protection.outlook.com [40.107.243.48]) by imf07.hostedemail.com (Postfix) with ESMTP id 444DF40007 for ; Wed, 19 Mar 2025 19:33:16 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QFBJeEhH; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf07.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.243.48 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1742412796; a=rsa-sha256; cv=pass; b=qwBbf0QE5gg7OXQL/93+rZCKY0oTB3dXwcCGwlfMT/htqf5/mhFeUS2RwVxiPF5kXZzGCU dbUb8C4haW2UnL95ZmuIOrhrQa8AjJxh75trDXo+dCFuyqzBISvS/85V6rEswWQczpikyN 5wJ3Coj1EBfqDG7fCmsPHRqA7mfVF5Y= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=QFBJeEhH; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf07.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.243.48 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742412796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=krPBi94jIB/Baadfzed9TBHF0/sXB2vjDdvQbhvxMPQ=; b=qI6CseFy44UZb9nFr05blVLkdcoY3kEdaR5IbNc0om/UkyaexkA1gt2OMAwr2r+nPH8UUO VycGIIXAJQ/rqofi07lN4CLQ3MYPK6Ah1/8UKY2O6i9JcZJN+RjCqZiZTGfAHuYjk1nO+J 3EtBLJT3J39aGM56LQRKeiBxfiuZ9hQ= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=TN3lCGAF5u8M5d6j/wPcIe6UjgTbf7gbVirJ3yFuPlpTvr37+duBAwsAhOy0A00oSigtCjG1PdT7VyH/0cBYRSaEdKjk8exwQpCBJJFxKKeWdUduIuNE4t2wJUZhb70C6tOtXUk+kx/8OSmWQt5rMfXZv6Xseo9pK6VVeNeSg0318QnNRPi4wxxw32jDBGtNvPNrde1gF+yPau0kQlLd+p01s3P3X9wuQ27yhtvtvnQZ2VbzSZ1fl+/7wxEdKQnFWkZT/7Bhwgdw3tcARlKlP9Jlb5HitiZ0IVejIw41dQrzl2by3f/BFnSQUzoajBkJISKDKbqHGX8aIrBqI49m/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=krPBi94jIB/Baadfzed9TBHF0/sXB2vjDdvQbhvxMPQ=; b=jngXRrtDe8q9SGFkMpeyt0zkbPP7kzd0tMEMlyUunPcdyBJZcBrk8xMNXXMLLAF7CSEl2OJWviz5iVXIwik+7zJcZbC1v9HjZuwEYtU5QK0NzIo11OIUuWovEfajYqQR8upVtjFcwvDU8grTMEmF4SfOlWYln/IBNtEvgUF7bcvqFeklPNCUOVlN3nBvqnmH1XCjKRVXkRRvFycq3T1hj8I5B9b4SF+nmLorzEDKLHzi9FvvnFZnPryyrm8Un2YCd3CLDFsKayFSqOLQk7dMtHHuXYDf2vHbx+AwtOv2d3rTzEQ97+WTcYI5aOPT5IS7w8hp0Ggzbc6o2UAgs66FYA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=arm.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=krPBi94jIB/Baadfzed9TBHF0/sXB2vjDdvQbhvxMPQ=; b=QFBJeEhHLhhXvWKPNvlPnlh6M71JbF76bfBmp1k45jZzMn2x5SOqHdkTxr39C3poXbTuOqE0gUqePuaJg3yAXsDLJS/1F+8dZdjeQz7RP99+ztdjA3SJwWqcJKrVJ5hNOgnn9pUg7K8AydC0W4027/KbfQzZTI4HoyF9fv0+rKw= Received: from SN7PR04CA0018.namprd04.prod.outlook.com (2603:10b6:806:f2::23) by SA0PR12MB4383.namprd12.prod.outlook.com (2603:10b6:806:94::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8534.34; Wed, 19 Mar 2025 19:33:11 +0000 Received: from SN1PEPF0002636B.namprd02.prod.outlook.com (2603:10b6:806:f2:cafe::22) by SN7PR04CA0018.outlook.office365.com (2603:10b6:806:f2::23) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8534.34 via Frontend Transport; Wed, 19 Mar 2025 19:33:10 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by SN1PEPF0002636B.mail.protection.outlook.com (10.167.241.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8534.20 via Frontend Transport; Wed, 19 Mar 2025 19:33:10 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Wed, 19 Mar 2025 14:33:01 -0500 From: Raghavendra K T To: CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [RFC PATCH V1 13/13] prctl: Introduce new prctl to control scanning Date: Wed, 19 Mar 2025 19:30:28 +0000 Message-ID: <20250319193028.29514-14-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250319193028.29514-1-raghavendra.kt@amd.com> References: <20250319193028.29514-1-raghavendra.kt@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF0002636B:EE_|SA0PR12MB4383:EE_ X-MS-Office365-Filtering-Correlation-Id: 9bbe0152-2683-452a-ef4b-08dd671ce250 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|36860700013|376014|7416014|82310400026; X-Microsoft-Antispam-Message-Info: SX9G7uYor+8ZG/8teWgp8/APefddEfHP2qYFZRz/ZQ/7AlQvE2pt2uXS4aD/4+F7a26mAcztlNXxfyKd6/UTY1YSJsNRyZ+HMZRBDB2YiyBGVH8VXo6KUSvq2YPowAeK5jn9QG4jAz752YBYz3Oyrbysl5EeYOQY0f4AltB3mVNDeUGvDRawaVOAmSu81naqE4/ThJSw0wTOrEcO2PGRZ91SODg6Rrjo0zViom2te/F7us51+tCJSdWlBNMhDOw+zKm9MvaAtgtDp/KAwAwiG2jZI0jLphXvb4ptdvuQIdwz3cD++R9VLlXWpvbcFt+Bvfz4/N6Bbmkv4ji7/n7+xrfz3RTkjVng4eyN2AmcvdXgGacy89uOOX5PPZ4w0L7ubROVKVVZl2k/U/7Zr8OjCBlnh5inyoFI/Di0nwI7c0tzwH3zI0DAFocWJwPY0wzFieSAkrODMWFfp6DhRMS5OI8YsfC+smPMHgmRhtU4/xomLtAYBmqxX976AGM/y9qj7uYW16sAbC7zby/zHIiK0RluzhA0p7QLT/7wIPVWn4K50VXw2xopP3noPnyFdhKpxY9bPtx/odUKCJJ4Mp3oCWJOXhZA0aIjkXaaWRDKB5lUoKqtin8i3F4IpJsrFlqDRCSP6Gv8h1EJRVNCghB14IStbbolPHn9ebuLDI+hp56c/ifrXRfgvHCQApPmq2bDYSfpih5VGm3qrnl1h+X3M4F3olD9UWv3zCDLOJksydmo7THiKy+rnkQegI27+WABQDbtb5/zo3Ey1eyj53C+ImlcURKJtDTQDNQ5V6OlpE0Y7WNNNVPbcTRwJHM47+uxL8TmygMqLl2eQLaxpD5/dtNxAqfIM9ElPw/79Dn+NCN02BTDDrAto+RRvbXxfi8D8EVgcrfPyPBlymkxChwrgjJgxTwHAm2TdleOMuWsbh2rrkQQhTWmVoBHNSu7Z0fRRKmo0CzKdmNTjKbUPf7R3Dp7L3npAZzFTfcKMI69H48FztAcDt66J/sT6YouDoln5ZjGpKBffw7sEubPMuRqXOSefTBL4F+kFsfzUCFGPPYrV5L1q1gT0yiAyAw++cMd9p58DENZ+rVxC0Gpjp96QT1ZOD1tg1LIrPMgah/hv6f/JntKSXBl55TnMGRXVGoi8xgO90cH8DK9W73KqwaClA8hoYLS0TcmJhK946VWpJeCjGl9AxExVl7yZiyqydy0diCCjiDoxJkvw8rjONsMg6Di9KlLH68T8EzuCCXB3w2V62D+X8fnxUKCniSM1FztJRdk8wBtwTSz1Xb9onIGXv/ri/Wblz6GsGGGVg8jzwb9zK2b14q2KvA+ntU0K6paSnxoFTAoXn2ze1EpCpXB2cOheo/vqYZ8nb5fKTuG6cgqnutf+wX93cMrsDzG8FUvm8dEfmz8rY6ZScxhKZqSJ8vs8E3H/Cj/C63+qQDsgiGFZC439GsXkZchi/gsn3A6OQwAwtA9FoSD3qn+qV1sXw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(36860700013)(376014)(7416014)(82310400026);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Mar 2025 19:33:10.9241 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9bbe0152-2683-452a-ef4b-08dd671ce250 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF0002636B.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4383 X-Stat-Signature: k8crdntfjo6jtfaf4kzmphnuxozh6imy X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 444DF40007 X-Rspam-User: X-HE-Tag: 1742412796-413729 X-HE-Meta: U2FsdGVkX1/l/WJrw+XQErb+xqwgQ2NVE9J7SPhFflTxribo7MzPNQlj6xJNQ+2FwLWweB80ZAWIOH5RfjdctMJM3G/1RhheXAak8A1Wm4nLY3NqKg/rW6+ekOX4ym6vQSwXOQM2a4a7HnjfO76Ons8xUGMOIUUy5VR2K8o8RCAjNiQOMGHE9m+6TJ0Qd6attXc5mPdt4XbQfNlpOKokejlMcI9HhCp1Yg3aOkSIy9a5kKI8nStBMYZzCCG2aiRm82+c/STM5dK7OlNAFz70qqmg/eUFLv101kBP3DQRJDMKdL2yxWkNS65tDI5xPgvD1DA80F/NxpsgdTbhhx22pRu7CGDfi7arjSNFAmm6U4rv/3Dl5huEdZP31kVTNQgiFDZRcRv7+Dv9noJ/shygDJsohUqy5D4IzmJU12ewlb0aOnwbKScu1GcY1gjYo1LS4nvQQ6h5xkDPTk6Lq8FWjU9St/KV7NPZUvIVxiEH1qqYlnS24kFeuvpDGRRtKj2w5Iv+V/9jM8BnZlK3odBt7A0augNeLiCSx/ew/LnPs9m6P0GG0O5UMSEgPDQL6Vm46gcgDYY0+aigj2XqUeC/Wea6sXGwx0gP/qmkR+WbomZORwF9eD0Dgbq1/P/UIgWtmSzOSZ7fEBQJfgjK/s0STluUblt1kVCgW0gXRxbdItIFxUda+40ZyWwmQK9RNun72XzpjcC/rt1tvaORNy+uXhA+lRXj/is26uIogsOdBLCx4k16AEH3LvsOfd4zfKKv4J7nV5WKqimLpcEhX+3QLIkUKtpERvUr1ANg34ORTxD+Mq8wlSnpSWJzGA8od3G+MH7kyPPn34Gx1CXzcqWJawfg5CLPnvi4zJ9nPyJ7sMuQjiukATJlNuBpLBNdCsS0xxoggwIVoSuge4W9Jfae/eAREOW9ZDnd2cGCoKm4ZW1uXbhamJBsI7BWnLzZv+AGTiDnLljabzD5p2LXiwS YwS9xC76 LmDji2mPspv5Ekmb4gOXRHk4RAIQPoxd4FwZiJDXnC0mdyGYxv639ll0xNuf2NV+XmRkNedVcqrXsjJVmSmkvEYLMv7VRz5FjbTgzR17d+Kcv/3gJ/vKLHNoLQDAHb9TfuP386BQjYvvWuxheW/3E0GZBzJM1ZOvf4b6zEJOafMnqygDd5cNQHYV7v1Cu7FJZYkO5D8Pc/9h0f4xyq316YHcRQm5kuucIuoHv0SqjgPLMGmxeseqyMs3gc5D0KtUbKtvHqO9hFvYYO7D/1e4YBYc7Po5XHaGdVr4u5F2a5fnTqDgZaVttT7OEQJGAV5AVUm4LDe35aW6g/9wknobiiVcEEYrdJslydE2boXOiNTROKzKW6NEqmwbFgMw3rdt3bjvVH6bfCVZcDlMh+XPMomPwCni9r/klLOmWysganleHRqWCMrA1XYWPq0CF8wgMGOThrp/vA1TJveuSNKurlzs6p9NgZbtJfqO7SDo2puAM5TT0CpCiZH74csOUal1HKZtfU+q/L//kq+azEQeGVvu23QdS0CdrX1fS6vOFlxP96h3XqbsoYqiBZlIc/KqLnwdx03A+LkyL6upVSgwZsCOMY2Ybpl4RnM0ksHc1VS1qTTcKfZNc9nqyn5KS2s8D7xaTyIq2GJZd9RPdH9mApl972d0OUuz/As1BR7nMNK+3ZdoEuJA5JODCwQZB1zIycHIecZuzfQZ2D4EE6pQplCUK8dlX95sOsj0GCkligJu95mpu7xZy9MTi/7rUJcAAQY/W1SuJSmoGX+5qUf10rImlsga7luP3ESoCFx6ieVQFs2ZONPxDCm3gNfldVIQLl4UQd4Bi3KWQhqavrcXT7ye3TpKbgpCdocUKMvmVK2O5Npk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new scalar value (PTEAScanScale) to control per task PTE A bit scanning is introduced. 0 : scanning disabled 1-10 : scanning enabled. In future PTEAScanScale could be used to control aggressiveness of scanning. CC: linux-doc@vger.kernel.org CC: Jonathan Corbet CC: linux-fsdevel@vger.kernel.org Suggested-by: David Rientjes Signed-off-by: Raghavendra K T --- Documentation/filesystems/proc.rst | 2 ++ fs/proc/task_mmu.c | 4 ++++ include/linux/kmmscand.h | 1 + include/linux/mm_types.h | 3 +++ include/uapi/linux/prctl.h | 7 +++++++ kernel/fork.c | 4 ++++ kernel/sys.c | 25 +++++++++++++++++++++++++ mm/kmmscand.c | 5 +++++ 8 files changed, 51 insertions(+) diff --git a/Documentation/filesystems/proc.rst b/Documentation/filesystems/proc.rst index 09f0aed5a08b..78633cab3f1a 100644 --- a/Documentation/filesystems/proc.rst +++ b/Documentation/filesystems/proc.rst @@ -195,6 +195,7 @@ read the file /proc/PID/status:: VmLib: 1412 kB VmPTE: 20 kb VmSwap: 0 kB + PTEAScanScale: 0 HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 @@ -278,6 +279,7 @@ It's slow but very precise. VmPTE size of page table entries VmSwap amount of swap used by anonymous private data (shmem swap usage is not included) + PTEAScanScale Integer representing async PTE A bit scan agrression HugetlbPages size of hugetlb memory portions CoreDumping process's memory is currently being dumped (killing the process may lead to a corrupted core) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index f02cd362309a..55620a5178fb 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -79,6 +79,10 @@ void task_mem(struct seq_file *m, struct mm_struct *mm) " kB\nVmPTE:\t", mm_pgtables_bytes(mm) >> 10, 8); SEQ_PUT_DEC(" kB\nVmSwap:\t", swap); seq_puts(m, " kB\n"); +#ifdef CONFIG_KMMSCAND + seq_put_decimal_ull_width(m, "PTEAScanScale:\t", mm->pte_scan_scale, 8); + seq_puts(m, "\n"); +#endif hugetlb_report_usage(m, mm); } #undef SEQ_PUT_DEC diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h index b120c65ee8c6..7021f7d979a6 100644 --- a/include/linux/kmmscand.h +++ b/include/linux/kmmscand.h @@ -13,6 +13,7 @@ static inline void kmmscand_execve(struct mm_struct *mm) static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) { + mm->pte_scan_scale = oldmm->pte_scan_scale; __kmmscand_enter(mm); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index eeaedc7473b1..12184e8ebc58 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1018,6 +1018,9 @@ struct mm_struct { #ifdef CONFIG_KMMSCAND /* Tracks promotion node. XXX: use nodemask */ int target_node; + + /* Integer representing PTE A bit scan aggression (0-10) */ + unsigned int pte_scan_scale; #endif /* * An operation with batched TLB flushing is going on. Anything diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index 5c6080680cb2..18face11440a 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -353,4 +353,11 @@ struct prctl_mm_map { */ #define PR_LOCK_SHADOW_STACK_STATUS 76 +/* Set/get PTE A bit scan scale */ +#define PR_SET_PTE_A_SCAN_SCALE 77 +#define PR_GET_PTE_A_SCAN_SCALE 78 +# define PR_PTE_A_SCAN_SCALE_MIN 0 +# define PR_PTE_A_SCAN_SCALE_MAX 10 +# define PR_PTE_A_SCAN_SCALE_DEFAULT 1 + #endif /* _LINUX_PRCTL_H */ diff --git a/kernel/fork.c b/kernel/fork.c index f61c55cf33c2..bfbbacb8ec36 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -106,6 +106,7 @@ #include #include #include +#include #include #include @@ -1292,6 +1293,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, init_tlb_flush_pending(mm); #if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !defined(CONFIG_SPLIT_PMD_PTLOCKS) mm->pmd_huge_pte = NULL; +#endif +#ifdef CONFIG_KMMSCAND + mm->pte_scan_scale = PR_PTE_A_SCAN_SCALE_DEFAULT; #endif mm_init_uprobes_state(mm); hugetlb_count_init(mm); diff --git a/kernel/sys.c b/kernel/sys.c index cb366ff8703a..0518480d8f78 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2142,6 +2142,19 @@ static int prctl_set_auxv(struct mm_struct *mm, unsigned long addr, return 0; } +#ifdef CONFIG_KMMSCAND +static int prctl_pte_scan_scale_write(unsigned int scale) +{ + scale = clamp(scale, PR_PTE_A_SCAN_SCALE_MIN, PR_PTE_A_SCAN_SCALE_MAX); + current->mm->pte_scan_scale = scale; + return 0; +} + +static unsigned int prctl_pte_scan_scale_read(void) +{ + return current->mm->pte_scan_scale; +} +#endif static int prctl_set_mm(int opt, unsigned long addr, unsigned long arg4, unsigned long arg5) @@ -2811,6 +2824,18 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3, return -EINVAL; error = arch_lock_shadow_stack_status(me, arg2); break; +#ifdef CONFIG_KMMSCAND + case PR_SET_PTE_A_SCAN_SCALE: + if (arg3 || arg4 || arg5) + return -EINVAL; + error = prctl_pte_scan_scale_write((unsigned int) arg2); + break; + case PR_GET_PTE_A_SCAN_SCALE: + if (arg2 || arg3 || arg4 || arg5) + return -EINVAL; + error = prctl_pte_scan_scale_read(); + break; +#endif default: trace_task_prctl_unknown(option, arg2, arg3, arg4, arg5); error = -EINVAL; diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 38d7825c0d62..68ef2141c349 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -1113,6 +1113,11 @@ static unsigned long kmmscand_scan_mm_slot(void) goto outerloop; } + if (!mm->pte_scan_scale) { + next_mm = true; + goto outerloop; + } + mm_target_node = READ_ONCE(mm->target_node); if (mm_target_node != mm_slot_target_node) WRITE_ONCE(mm->target_node, mm_slot_target_node);