From patchwork Wed Mar 1 12:19:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 13155894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79948C64ED6 for ; Wed, 1 Mar 2023 12:20:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E0B76B0072; Wed, 1 Mar 2023 07:20:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 090586B0073; Wed, 1 Mar 2023 07:20:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4B366B0074; Wed, 1 Mar 2023 07:20:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D1E4B6B0072 for ; Wed, 1 Mar 2023 07:20:23 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 936801C6145 for ; Wed, 1 Mar 2023 12:20:23 +0000 (UTC) X-FDA: 80520237126.06.F5826B7 Received: from NAM04-MW2-obe.outbound.protection.outlook.com (mail-mw2nam04on2055.outbound.protection.outlook.com [40.107.101.55]) by imf26.hostedemail.com (Postfix) with ESMTP id 94A0E14001A for ; Wed, 1 Mar 2023 12:20:20 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=DorsG1lS; spf=pass (imf26.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.101.55 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677673220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FK+nIUdqOR9KEr8+cJsJk2w1QgWI/Up4YGCy/MVVpRs=; b=QQpGpzf8Dyaf6/wmE64sA+0FECOZq6whrIa3hJDni8ki/njauaYQ32G85jkEuwdqehQpJs kYHkAFbULBwWxnEqeXrh7Dw1WoOXkehsKGhBwSl70ckRjbdy/j4h3qmjLUJewp2Rd0j7qh Y9iYUBRF/adBg/LizfUCIV+20xI9jPo= ARC-Authentication-Results: i=2; imf26.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=DorsG1lS; spf=pass (imf26.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.101.55 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1677673220; a=rsa-sha256; cv=pass; b=nxisFgkaj5xaHKx2ZQhv9kYKjqg/jcFRuJrFh8oBlihUpddX155+gTg2jPM8jhqY85k6+x UKOlBWp6noMeqeqOM8vrJTFspRyxy2MKqKR/nW9AoPQjbsTlKKf7ZTQsU4sTIy1P3wZukR euTmlJ0LfZWETrdIajF7eVzZTmXKnDg= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JMjTV+TFmvtj2TTkF5kFKMCp2CLRJXO6xI+DJZyAxoOoPf+D4YPTzj+K9Fi+pvEPt38LD9N/u9Avl4/LFd67XW7r1BGYKyKseaSBxE5WQvUFm5sPSbk55rZmQ/d3xBVULY9FB3Tzh0QSz8yJYnDvG1DGlEQ0lx/obMYo56pleAccASa0SxH98cOx/iJg160Gp9ZKt90avyZclMdqEZdHkEJQytt3gAOVPc6QwHosXU0JwdArB3AG0mygYJ4+cFYCppZlTa13aYSVv61we0R3MptToTXtFRtevHhRCVwNmVm6ht+klLrVi+tSLYmM0NNisA6dGuWARqeGuljfVhhGXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FK+nIUdqOR9KEr8+cJsJk2w1QgWI/Up4YGCy/MVVpRs=; b=lCKaZu1onHhJc0OyStm1wfCYqz/k9I3a49UiN78wYm4Ho0tYZqSwEZIGpmxJG2hq8IcHfNAO0QoFxLladWer/W/tPSQQpQ6UQaX4Dh7gaXppqHJguyNb/jrRVa1IG1vt6H0LNr044odLiyAzsSqeFdvYf23QebXsYeRuV+wenZkfdNnMzP/z8SB0A2wShD31k/IlENxVN2tg0X1FJV5c1pf7+q3uzCndzjQvY3DAxdvdUS7sU5N6MdP3qpffBmmGGUqZWHzgWaAffiHNtDQK6akPJoupQ+NsYIyUNhmU3GXv3OybnFUAk7JpdsgomzfvhIygFNclZlwdiV4mnlWc0w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FK+nIUdqOR9KEr8+cJsJk2w1QgWI/Up4YGCy/MVVpRs=; b=DorsG1lS6n3Vzi/nbCIk5QBI+CtmmHJlvkOV1pNGEk7WpenhLjnwDg83iAvoAZlenhEAChef8oqMleWWydaOZ5Lv8VdKiIQoQKQn1vNCU5CmXVg+WWyIOgWOU4KEbXNMvBB4zap+dPYJyRaVXEI9Vr+CO8bz6xJcx/GtZiugW/A= Received: from DM6PR12CA0033.namprd12.prod.outlook.com (2603:10b6:5:1c0::46) by CH2PR12MB4037.namprd12.prod.outlook.com (2603:10b6:610:7a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.30; Wed, 1 Mar 2023 12:20:17 +0000 Received: from DS1PEPF0000E638.namprd02.prod.outlook.com (2603:10b6:5:1c0:cafe::84) by DM6PR12CA0033.outlook.office365.com (2603:10b6:5:1c0::46) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.18 via Frontend Transport; Wed, 1 Mar 2023 12:20:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DS1PEPF0000E638.mail.protection.outlook.com (10.167.17.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6156.12 via Frontend Transport; Wed, 1 Mar 2023 12:20:17 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Mar 2023 06:20:13 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Mel Gorman , Raghavendra K T Subject: [PATCH REBASE V3 1/4] sched/numa: Apply the scan delay to every new vma Date: Wed, 1 Mar 2023 17:49:00 +0530 Message-ID: <7a6fbba87c8b51e67efd3e74285bb4cb311a16ca.1677672277.git.raghavendra.kt@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS1PEPF0000E638:EE_|CH2PR12MB4037:EE_ X-MS-Office365-Filtering-Correlation-Id: 06abf500-d09c-41ef-e142-08db1a4f51b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iCupr+U8pqM+HeB3EUDY8aiPF6whw0HnzU8F27CU2ulVAlOYYITuzkjld6vRGzY3Il4+sTK9J1kiGU/WniRBRczzLw38vqSfi2zAqTPmtD0fMEb8vCdvO8zY4lq4yXfHLBi8XIqtGhJPZ6P0ts2o67QgHx9YI8/9y2rEVIK/p+00Lrdl46feVc3ckEFPkInLWixYZMDh+Mkd8S+AYXmMH4Baq0J3WEdqTG+p5jqDZJS/r2UUYbJ/GBPXhszzT4KgAHsynFlO5ItZwrXYGDwjLXm1Yf3cgUDxWQ6BIj0oRpM8MMPk+S0sy8tflCpI81qfvD1lHDBkhStmDnojzulO1gZbUV0yJN0ubEusMqOLHq1F4Ixx3xY8ygnXMtMci6HVfGIA5UQV9j5t4/SeVCI5ryyohL56S9R9i0cy1w5PlxzpCLB7bq2mUeN8tkLe8hiwxyhfLFTZFZMG1ix2LTXli3TPtIbfLxRrNiPeMHuV3F0QSRIw3pTbwkKellR7NQp4MISsYoN614MH7VaG11LqoMZ7ufCCZGYRJgo4tL0YSzdWdIkpfBjv7Qa/dXJGZ/Q1KKD2ONAvkGktgl/EdYiBqWIzCdOQObEZeDos8JLevk5Tvb7keXvxwbObWM8IthUnI7pNfkkzh7RO8xhA5R3U20iGgIXtWz9kg8nOq3iFSH7glqo35CmWHo1OghHNajRxRw9pk0A/ASgoKDbSkpZum2+vpyScX8laZ5td22HwPm75l/jf+iL+zW0m1/JUunxL X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(396003)(376002)(136003)(39860400002)(346002)(451199018)(40470700004)(36840700001)(46966006)(356005)(36860700001)(81166007)(82740400003)(36756003)(7696005)(41300700001)(2906002)(40480700001)(5660300002)(8676002)(4326008)(8936002)(82310400005)(70206006)(40460700003)(70586007)(2616005)(16526019)(336012)(186003)(47076005)(83380400001)(26005)(426003)(110136005)(478600001)(316002)(54906003)(6666004)(36900700001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Mar 2023 12:20:17.7888 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 06abf500-d09c-41ef-e142-08db1a4f51b5 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DS1PEPF0000E638.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4037 X-Rspamd-Queue-Id: 94A0E14001A X-Stat-Signature: 34tgb6d9xo717mjrx88ymhdbuuhcyszt X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677673220-241331 X-HE-Meta: U2FsdGVkX19L69aPu+45AodEQbvTzF7GTCO5g7vES06OALqyhvcNPJcQss+jFbn42BDNXDDHSdFBFntVHx+o8hSlKhZKgDYyt3tSmw2ypPfYMMpMsi80sbZJpRmOohd1uL1RnWGiihftlqGbtTA9z6Dq25q6/FfRTYqRL2vUYcGtDSYywYsZtQPqJe3SEZRWNy/VDdvM7gvAAo9FwLmB55k0SgVicUvtjjsJrJ4NdVvCupro/HNUThIjyfzW2NLm22AHuk3WZqSpyK3cTvFtaSUaI/dbYwX7PYD6QTOsP2l/q9uA+DG/ORjMAROqk70Zjv+xygcXg6Syv/wztB7er+j+HTM98d7vZrge/X5MLuOgUWa6b6r1714p7Bn9vU6bWaj67A0pYYBNFKOw2Sk3S/GwFPdY372aXqR8HxSyoxTo0SJ/KrL5Q7uUAfqv0pq0lR0C1Fv8TnPrVJmiDYiDaC+6fiY0nOoSj9iJn2eBjxjFgvOA1ACcdXl2gP9lq8DWNFxIEQIXy/P0ZgdEgsL52KQ9S8OtDhsj66YfIA2Kx/TaX7Kyfxa6+Mt9xir+y/NY9J99Ig4PCFdTMbJ8SbrY3+WV0bZhGjCk15AGXg/hoLfK6awSLwLRBr0ak+yCelOCp920RGKEB+7HJwEw37b59y2eSx72Yy9v7Y12UOF+dxTasG1ktRmwYlVmYyW47MgwrLwhQPR/3V6/SG0pw/3Y4bqqCjEInyjHcpaOKA6D7/gB7Bptnl/SIVZz6+L2X3nityo7NRa4fuk58WLBWJu+N2zpU6zcsQ5eakxWQk2eqGkuDN+rielQvkaB3qLMHRsGJiHs2HA864vgDYxTFR2RTjxc2W2J8B2q2CZpkroW3RRXPjyqUQ7SAHkizH26nJOH+RgfF0Pwzkm+4GwECOLeaoqn1+P0SUHhSRGXgNmRWe55p5KDsv9ELtrRJLmEqUgCZ+doVlme12p0m3aGsr4 PI9EKfrh Is34OXSrMW2UCyHZA+CYH5jZMVEHXfFWAS5JR+N6OldI+u55845INUPjbHTUqtTcjCOE4YsVjOZooA3chedVEHvIgZ7njpo/wPMOC2ONtLHG8YYGJrIzh2yyjyQch7HRE1R7sb4koH89iGtHake5nR3AHVIOdNDwTXqFegJccdU0fCMbFbDiMAu0JA3J3NJI8GhqsoxNzE4BnL9YSEyZFbGTij0N87SZz6byoNYUmwpYEIJlBN6aYe19xvRSae5Awv8WV2etaisCbYEYQr9qh4+yW1MU2+oEw3yK5pv7pGDKoFmsXGWI33z1phdSf1A4Akvfq53PA7pMhglDLz4QGZMaK6LFz/7EU7MFz5iYfHIFLT2xSXzUlZj0DT+Ulgh6JH5opHkyepUUDSH7BwpIepxcTvs5xRvwt+4qor1ToTwPGn/iB4BlYlw0Z5VkMvLhYwA68XNemOl+TMjO0zHIpbOF7SDVxLkFzbA5U6yA7RI1t2aFUftZq0d3SORTuokK1mSi+qf5MWlFT8QHNZkxC+7KI5a+4A7izWtzkmqnYLK9a9xfzG5xjwb6nmgNtjLttcf6RiAD+teYTBBc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mel Gorman Currently whenever a new task is created we wait for sysctl_numa_balancing_scan_delay to avoid unnessary scanning overhead. Extend the same logic to new or very short-lived VMAs. (Raghavendra: Add initialization in vm_area_dup()) Signed-off-by: Mel Gorman Signed-off-by: Raghavendra K T --- include/linux/mm.h | 16 ++++++++++++++++ include/linux/mm_types.h | 7 +++++++ kernel/fork.c | 2 ++ kernel/sched/fair.c | 19 +++++++++++++++++++ 4 files changed, 44 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3d4bb18dfcb7..2cce434a5e55 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -29,6 +29,7 @@ #include #include #include +#include struct mempolicy; struct anon_vma; @@ -626,6 +627,20 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_NUMA_BALANCING +static inline void vma_numab_state_init(struct vm_area_struct *vma) +{ + vma->numab_state = NULL; +} +static inline void vma_numab_state_free(struct vm_area_struct *vma) +{ + kfree(vma->numab_state); +} +#else +static inline void vma_numab_state_init(struct vm_area_struct *vma) {} +static inline void vma_numab_state_free(struct vm_area_struct *vma) {} +#endif /* CONFIG_NUMA_BALANCING */ + #ifdef CONFIG_PER_VMA_LOCK /* * Try to read-lock a vma. The function is allowed to occasionally yield false @@ -727,6 +742,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); vma_mark_detached(vma, false); + vma_numab_state_init(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 89bbf7d8a312..1cea78f60011 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -475,6 +475,10 @@ struct vma_lock { struct rw_semaphore lock; }; +struct vma_numab_state { + unsigned long next_scan; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -565,6 +569,9 @@ struct vm_area_struct { #endif #ifdef CONFIG_NUMA struct mempolicy *vm_policy; /* NUMA policy for the VMA */ +#endif +#ifdef CONFIG_NUMA_BALANCING + struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout; diff --git a/kernel/fork.c b/kernel/fork.c index 75792157f51a..305f963359dc 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -516,6 +516,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return NULL; } INIT_LIST_HEAD(&new->anon_vma_chain); + vma_numab_state_init(new); dup_anon_vma_name(orig, new); return new; @@ -523,6 +524,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) void __vm_area_free(struct vm_area_struct *vma) { + vma_numab_state_free(vma); free_anon_vma_name(vma); vma_lock_free(vma); kmem_cache_free(vm_area_cachep, vma); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 7a1b1f855b96..7c2bbc8d618b 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3027,6 +3027,25 @@ static void task_numa_work(struct callback_head *work) if (!vma_is_accessible(vma)) continue; + /* Initialise new per-VMA NUMAB state. */ + if (!vma->numab_state) { + vma->numab_state = kzalloc(sizeof(struct vma_numab_state), + GFP_KERNEL); + if (!vma->numab_state) + continue; + + vma->numab_state->next_scan = now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); + } + + /* + * Scanning the VMA's of short lived tasks add more overhead. So + * delay the scan for new VMAs. + */ + if (mm->numa_scan_seq && time_before(jiffies, + vma->numab_state->next_scan)) + continue; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);