From patchwork Wed Feb 1 08:02:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 13123893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F09BC38142 for ; Wed, 1 Feb 2023 08:03:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C576E6B0072; Wed, 1 Feb 2023 03:03:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C06556B0074; Wed, 1 Feb 2023 03:03:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA7B46B0075; Wed, 1 Feb 2023 03:03:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9BE436B0072 for ; Wed, 1 Feb 2023 03:03:32 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7D2FB8091B for ; Wed, 1 Feb 2023 08:03:32 +0000 (UTC) X-FDA: 80417983464.29.8B5D20B Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2079.outbound.protection.outlook.com [40.107.243.79]) by imf24.hostedemail.com (Postfix) with ESMTP id 9E9B2180002 for ; Wed, 1 Feb 2023 08:03:28 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=RxcmXlie; spf=pass (imf24.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.243.79 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1675238608; a=rsa-sha256; cv=pass; b=vtyJ3imBgTVdYwP1a8SSCtTOtgHxdONYsBwYh6FfHpnanLjiND654imfTqHg72wVlZnT7E jlKjHj+YvhIaY0uIVYWokdhhNEMh/t/rqTUn3E5EPxhOa5gBkRyPu9ROnDQzRTzmPVS5Lx QrhsJ/JvqdHZVbV105Yo7EJZe1VQR54= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=RxcmXlie; spf=pass (imf24.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.243.79 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675238608; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AhcuMGmyMR6wFmrQd9SM/5hyh6AFl2dn/jX8xfA+fTk=; b=u6QdX65XlyqZzdX6tl1jolIc1EdqXBdmU1xa6kdEnZqJ4zhXIOtpCFNRXB557QmgbCOU94 G6fXxJkrZkDhcZ5j7LReaOauhRWYwQZWl7Ccx91DnE3iJlCC/V7Ey6MnpfAXRfz073ZBoB rR0Zr5aJAZTI38k3lLObSSK6ugMnGZc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mvsd5PIG9G1vJY0uDpjV3o1hPM7aBxusFhRgoWan/BGg8YWM8Ho4fhUvoz470GV/sZ23w1BumzF1K0jYVFYteHuyEpa61GM2pReYrjpNXpcGm3ua0pYjtgfKW90wvYek6rt1eqWhkg+7CvLTC8HXSObkyvkXy3GbCU0tgjJTXufCOmJhllazG6PAjgB62t34dko59qxLJQlBaYWwvw4XZyvFcGchSfAJSYAzsZ8Fhqmv3aARpdvCON2rHvFZEymhtBO6h3Nol7NZI1ug4xCtWPrQF38ncNkLukx1Sbj8barPrvB1g/I5MIr3qe21pQ/VckFauVy9cJ4ClhhgNg+Q4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AhcuMGmyMR6wFmrQd9SM/5hyh6AFl2dn/jX8xfA+fTk=; b=etD+lxv6XdLA/T2Myd7/YFDB4Z1HxELSMpa+/0tLmP2jADZDSYoYqrHC9NwySwXNlr67+gaqCyAO7rOY+OuALoIb8CbdSdi8bebXEnIq/7j5yio6KLMqiYkyBb5unqlc6Zah4yTPTZwpHmJo0HOvSmzPdkVI1E7Hy6oCAYwFz0Fg9W+vna+19KxVzRx5N1xK2BiNdH/4dM+fGCm40d2rIvTAAXR8j1uSqe6T8YaDJzUOpN1bmGg1ISwHGGALZtfSh3a3CH5l6mK/KkXPL/8asck/ieQFJ1Q57OGbkNePVivRN4yDSQ64cpRI0/fnyiwb1l8DbPRhBQmGIIjEwKXBtA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AhcuMGmyMR6wFmrQd9SM/5hyh6AFl2dn/jX8xfA+fTk=; b=RxcmXlieyT4aNlBaMdoa68K87VGD0TBpMjYRJVq2wmpnq7veMTzPq50p82cjoPt10eJ7t+ivKbziPkcANnrfBwUfkXJRpg6uXcFiyju7PAaJxYQvWkbNHxMqvZMSio+04WwOaRbPKHP+aZCbJ+J9x85Cyiua6he5W47yaOAetkY= Received: from BN9P223CA0019.NAMP223.PROD.OUTLOOK.COM (2603:10b6:408:10b::24) by CH3PR12MB8511.namprd12.prod.outlook.com (2603:10b6:610:15c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38; Wed, 1 Feb 2023 08:03:26 +0000 Received: from BN8NAM11FT082.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10b:cafe::36) by BN9P223CA0019.outlook.office365.com (2603:10b6:408:10b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38 via Frontend Transport; Wed, 1 Feb 2023 08:03:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT082.mail.protection.outlook.com (10.13.176.94) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6064.22 via Frontend Transport; Wed, 1 Feb 2023 08:03:25 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Feb 2023 02:03:22 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Mel Gorman , Raghavendra K T Subject: [PATCH V2 1/3] sched/numa: Apply the scan delay to every vma instead of tasks Date: Wed, 1 Feb 2023 13:32:20 +0530 Message-ID: <1aebc55030925998a3df3cafb79c5cd28b199ea8.1675159422.git.raghavendra.kt@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT082:EE_|CH3PR12MB8511:EE_ X-MS-Office365-Filtering-Correlation-Id: 510611f5-71e3-49c2-9d40-08db042acbef X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /SRV4htTNbr7JEQTc+CKomAgA6e1uQtQT7+e74B3FoFmk66xOXpU5iqvOZ4kljvy6KwcH/DsdEX0hUnJlt/DgtUq1ZGjWh90uqDpL/BzgD5wjwX1jkwqGfBCa7pf1C30QN7RwAUEytf04cnHYb59cPTS4Revv6OOY1OD8Pb/4dV6oinvQ3TnYWcGC37eIoyHV2FpmYnRd0MaJgFgA3gRNX2GE1h84X0D0IggUOGZSCvMgI92UZVdc+X+D7L0un7r6Y1ggh8NRJ1ZYCJa7xuBT/MeXYvTPCysXLDyCD+EZH0cKZomBK+hT9h68Uyge2d1pPcKtj4jPJhPQfCfM7XqFfgQ9uKA25zH04fScUXebJla/L1F5gbD+BreG1CCburGOKUvHXj0D8p6LrEwHWzu9B01I1D9kJf8TE9EMLjetS1vroXTi9zgxhbhz+OB5WUqawekepNfg0/16czmXpmFmZn5m6TuxIuCSfAHILG32p2VpkRbsVIfRDFhYiVzfi9qaoyfw+NuA6BhRX9Eft6I2D3fAQSN8D54Av3mYkXLuaMK2jPlZ6/9kdVovXq6u49q+p4h8P8XZZPeM8P3BVSn3J1OZhZ82uACNFqMCQdsCKD5C/acJfcahgwprlvkFYxlJpxDzewXUk1rIOJ5OF9deeozAuIcUTKaybDBq928Zps7kdNudPiNLFS3PhTa/34JAIP3NgfCjNqB8SE/ErhPKO1HrJdMijTaUH09qFQ8LTQjkdjUp6o0hhWA/cxZfzmg X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(346002)(39860400002)(451199018)(46966006)(36840700001)(40470700004)(54906003)(316002)(8936002)(110136005)(4326008)(70586007)(8676002)(41300700001)(70206006)(356005)(40480700001)(82740400003)(36756003)(36860700001)(81166007)(40460700003)(186003)(26005)(2906002)(82310400005)(7696005)(47076005)(478600001)(426003)(83380400001)(2616005)(5660300002)(16526019)(336012)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 08:03:25.9204 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 510611f5-71e3-49c2-9d40-08db042acbef X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT082.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8511 X-Rspam-User: X-Rspamd-Queue-Id: 9E9B2180002 X-Rspamd-Server: rspam01 X-Stat-Signature: 68bxgoj9qouftkc7q5nhh38frebusko4 X-HE-Tag: 1675238608-155286 X-HE-Meta: U2FsdGVkX1/W1yZuppUVugxBCw6mtr2vR+T13bGKFKTYL33fBJjTA11G940+xsLHzjuF7dHbANTe2xhD39jAMmmZam6NgwwhlPFHYW4AAt4qS9X8hN+S0aeewnJqY2yPR7BYzo34nPoLcyJA14t+S6hFwPGTZsI4eI349obFbAmMgpqsLrzaIYecChgUT2MZRmZEh52jh2dbQRsHMVPtsJSWDGadlWCKSbLDIeFiAOATdQRu7IiE4ZhMjsPiNbGx5AT8Q0u4Aeuq4BRohUp7/Di66SkRGdN96+9GYkRUae7neDMcC8v8IZgytX1Kiow0HtWy32UjWkjMWImeTBQqXVLpgJWvY95lhhpjPZOS53mNkMlBuGNIslgr4aggrqtvKyFY5YR2hbE/SUSLaG+eZNi5gFWWqVK60uXnvMAwsb+tHaNFBhkmqPJdqV7cQiHHiNplXzC3SNrz219VMUVtO6k/wbSnqRLfUCHYsscop68gxUBuDD6kjK81GagZ4w62dFd2JyhQFIlL0u9BT30HOCm8J0swAIEqTSUV6jOSf9aHyGhsAt1Tk7rgsYD9dOIHQO2GW3f6otUkr3mukg6tAzHEGy6BsZDSDWDahbZOvBSprE3ddU20iVdbbT+iXsNiptO2tCBcp6Gy+RhocRjmJcajqLQHI9JyuV6fy/bbFv5JLni4ffbZrgUUSTBb4ldT73EdI+kJ0CvvtXgNzEyige0KLAnBIIXbzvlKA7xrC55kzpNFTjQ7BtQCwnvboZK55o4Wcb4DGoK88j9YuCThIjSWLtvw2cTiZHNpg1PWNAzgGBvrtkN/Pc8gNKB08A1uBd0/g+M4fIVPSvQr2Fd0/rlX93enA4IipPCj0dZIGmdeTudPV8+zW9rYB2MaMtyKwPv8PBNSmAY4bi4BKepN6VSTftW9vD9L58PYv7jua1ERMmJgrFPhX6CBLftSegyqXslwvF1EHKFX14VPWaW CsPxPh8c D5gnAuqsg9MjI5bQFwvATUe/7wKGJ91CBGGJXFnucm/aShZDDYttnza1nWpIUmQziqJ4y17GNHTNuA7TbbJWbRsaNAts7ui6sUaeDgNyc5gxCnI8onxLuVt3H22hRTVzVXLhBwMpQBut6Hty1Ho2iRHG7lkRoxdSRiIqGsLby81QP3SJn4blG1FIArPIK2KThAMfK0H3LF1c04MZnxHrCQzV0YEsYCCUPIZWL5UHHhsY1AFytb2+KHorZ7AXZzb+74qpQaksOeN1tcH9fdrSxxlVyH0roEZGPEA5fEPfww+i8H4kTEfi39XlxyLiqZuT6IrXOoHHLFJSFAwgxqohXDBNVGgeLnfpfIU5hkCsVDz2t0+T4rImJ++DXGuSIUvQ2JtiJC+zw2MhYJxmRsxfeYjxvroy9C2zxdQ8Gry1aD3URqxTwNOhcRVTCLRgsGspzZ7NaVB9PLkTifxwYZcGqo5H0DIDNkqHluTI6gB8HheLFSjFnJKOesdj4x4kRZBTCxhJAOnVt1tGVKVaut1q7vcQM7/C4X5YCzzHPsGWI+6+uIqAQWOJX86NPi0OKV4lip3d/4wCFEsAWuQcnzxjyVI0ELSL9YSzQ22eW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mel Gorman Avoid scanning new or very short-lived VMAs. (Raghavendra: Add initialization in vm_area_dup()) Signed-off-by: Mel Gorman Signed-off-by: Raghavendra K T --- include/linux/mm.h | 9 +++++++++ include/linux/mm_types.h | 7 +++++++ kernel/fork.c | 2 ++ kernel/sched/fair.c | 17 +++++++++++++++++ 4 files changed, 35 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 974ccca609d2..74d9df1d8982 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,14 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_NUMA_BALANCING +#define vma_numab_init(vma) do { (vma)->numab = NULL; } while (0) +#define vma_numab_free(vma) do { kfree((vma)->numab); } while (0) +#else +static inline void vma_numab_init(struct vm_area_struct *vma) {} +static inline void vma_numab_free(struct vm_area_struct *vma) {} +#endif /* CONFIG_NUMA_BALANCING */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +627,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_numab_init(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 500e536796ca..e84f95a77321 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -435,6 +435,10 @@ struct anon_vma_name { char name[]; }; +struct vma_numab { + unsigned long next_scan; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -504,6 +508,9 @@ struct vm_area_struct { #endif #ifdef CONFIG_NUMA struct mempolicy *vm_policy; /* NUMA policy for the VMA */ +#endif +#ifdef CONFIG_NUMA_BALANCING + struct vma_numab *numab; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout; diff --git a/kernel/fork.c b/kernel/fork.c index 08969f5aa38d..ac6f0477cf6e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_numab_init(new); dup_anon_vma_name(orig, new); } return new; @@ -481,6 +482,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) void vm_area_free(struct vm_area_struct *vma) { + vma_numab_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e4a0b8bd941c..060b241ce3c5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3015,6 +3015,23 @@ static void task_numa_work(struct callback_head *work) if (!vma_is_accessible(vma)) continue; + /* Initialise new per-VMA NUMAB state. */ + if (!vma->numab) { + vma->numab = kzalloc(sizeof(struct vma_numab), GFP_KERNEL); + if (!vma->numab) + continue; + + vma->numab->next_scan = now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); + } + + /* + * After the first scan is complete, delay the balancing scan + * for new VMAs. + */ + if (mm->numa_scan_seq && time_before(jiffies, vma->numab->next_scan)) + continue; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE); From patchwork Wed Feb 1 08:02:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 13123894 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29034C38142 for ; Wed, 1 Feb 2023 08:03:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B93496B0074; Wed, 1 Feb 2023 03:03:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B42C16B0075; Wed, 1 Feb 2023 03:03:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BD876B0078; Wed, 1 Feb 2023 03:03:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8C5336B0074 for ; Wed, 1 Feb 2023 03:03:37 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 36C9C120D62 for ; Wed, 1 Feb 2023 08:03:37 +0000 (UTC) X-FDA: 80417983674.07.0ECC773 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2051.outbound.protection.outlook.com [40.107.237.51]) by imf10.hostedemail.com (Postfix) with ESMTP id 51914C001E for ; Wed, 1 Feb 2023 08:03:34 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=TtZGqDs1; spf=pass (imf10.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.237.51 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675238614; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1R9KAiSIKM0EdbYofwwJVHMq44auVW63YveLhtMSQ3A=; b=qBJMyt9wonsBqYVeKmz/ldvnYfvhKIPkTxRTZ3P1SSCASb6u8RYjrgEcVMh85DQQhf5iwh 5Ri6G9uBZMNhKRYUf/fRLqteIe2fPPVSjazI4CbnudxNuImOMsKOqkhTGaO6NNuUh2LonC dGVo9OEJwnPrLDOdFLuN5bMRO9Skbd0= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=TtZGqDs1; spf=pass (imf10.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.237.51 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1675238614; a=rsa-sha256; cv=pass; b=PFwUpVG5spD9b3pzdS5jTdXJKo0ikpxwGHrxzdRdpcWHaOI2lvtpMrP06SGY/D5XCUCPHY Q9VUG5GanR7R7KxGx3KO7qsJtZ5Qw6VkD2JY2ifUtu87I9khnBiPAHqLNRJAUg9uJxcp2A Cj0SM3be9zevgaNJfrCW/Gulh52MjoM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EaisipyW1DNecXe/OL8nAK6OSSIwpNOx0jfBHm7SEMEY9I3la64UaqB+2679vkJ1I/X6Wm4OAnFV4eja279oqg6F71z4lcTfsmmmmAw3OP50+z9u3Fx9U1mB1nopLbhKC2NAvit5ul0zHffO5q7uycBk2GQUq4Ph0TysYd8rt50jnLj5PMcdr7GF3GNXWKQs4/Q9Os5gXdSg5tBP2HHLmgYwNHkrwoX6wmpEV0mMNb/oXiOPnvhIDlXM56IoQReG2fHo3N5XANltCgrSVzBuGVLv1w/XcK5uyWEP3KxVdSzU/gHz9F5CpTxesA0BVm1Q2udIiEMrAmFePDglZVppUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1R9KAiSIKM0EdbYofwwJVHMq44auVW63YveLhtMSQ3A=; b=fEpiMe2C2Xd93Q4f0+OwDRVDogTu5jIwL8ROm7N68EYy1wuKsgiNSzWyAVwG4AWQp6bYv/XauLL9YnHuug3R+ZkPhur3htRYUTNLWzOC6Cz9Jr6sFJUfkM3+Gx1vIRE918iBZABglicdB3jBWluxcCzWIPJZYObwj9smbMFztqPC/BCFxHi9SPxw/SBthQNYrAp3yxdCIaVEzHdDgRKA+NBeiCwvwO2mBKhlXY6ILIrGTOmKgBtQcgg/mt7DOe1pwlnaXcKyUPktVndFjtrEKkYvU7n66ZTbfX4tFg/eT6XV363RWzYKra89OPZ4LbQzr80+/AkQhfwEiq/DRl4YwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1R9KAiSIKM0EdbYofwwJVHMq44auVW63YveLhtMSQ3A=; b=TtZGqDs1IjOIhDWRY8F4GtSlIuDzjxWQq9V96wz5C88B+zMctmpv9PpPg0Y8kZxdeJ/t5aflF15zs/cJo8QLyhAwvlgbepVLWBQr6Tfewa5bC6/XppcyiEuHaerWVi72GxA2nDnMbtXCB9ghkg67RNeudzwJ8Vt55Br0d2YLs8s= Received: from BN1PR12CA0029.namprd12.prod.outlook.com (2603:10b6:408:e1::34) by CH3PR12MB7689.namprd12.prod.outlook.com (2603:10b6:610:14d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.36; Wed, 1 Feb 2023 08:03:29 +0000 Received: from BN8NAM11FT096.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e1:cafe::ee) by BN1PR12CA0029.outlook.office365.com (2603:10b6:408:e1::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.24 via Frontend Transport; Wed, 1 Feb 2023 08:03:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT096.mail.protection.outlook.com (10.13.177.195) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6064.24 via Frontend Transport; Wed, 1 Feb 2023 08:03:29 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Feb 2023 02:03:25 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Raghavendra K T Subject: [PATCH V2 2/3] sched/numa: Enhance vma scanning logic Date: Wed, 1 Feb 2023 13:32:21 +0530 Message-ID: <5f0872657ddb164aa047a2231f8dc1086fe6adf6.1675159422.git.raghavendra.kt@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT096:EE_|CH3PR12MB7689:EE_ X-MS-Office365-Filtering-Correlation-Id: 84e0c807-f189-4def-00f7-08db042ace20 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1x5Eufv38AZdDH+X32cME7v+8pJxC7VkM+4RaTgfaFTEGV+5Hmh59KNIC6uH7TaaloDWI+XpkEslufPnVS9HVJouLjrm5i2diGoO6gSmRky022Yl94Q+pZkUaCAnbJmll0nAfxlT66qcNYrivpR8bZbYYDd772Kdip0gEbLbtczngrLLYs+T+XRt5Uhvf7dbPcobVbx2iZzBosMCNEO7PqEyTATMotHLYcFyG/nNUmVXdtOGBS8QaSusXbdW0iDR8RURSV2edkvkYpYLNjKgmYhQtR1HoAdlk7QZ785P3txKCOzkUO/kOAXdffwc9u3/t05JpV7EuljVjtu/wyxXdAqlnccRwaX0++BIJ36ZeQQFj45frhMSpkOBXTRWjItTknNrnoKbQ7r8RarJvcigu3Ot/WVd5Ar4zt+g1goWhfvF/FvjVC1JlImGsbeUK/5raAGeqKgeytcjbnynmX6ht6wMF89EvWiY0ZUM73Ry1/pwyinvN11Vib8/b4dFUBkb1mjl+HV2vmPYlSxOZSun4yhEIjorMw9VPcEMzfTJ3kYXJ/3bbgAENyi5rfZGtgfHeCuDVuLrhI0YSKJfwc/4z3YRQnz+XF1GgzJuYFFoRBOmmVTpxD5foh2HpG67YyucvrF3kwA0NxilSEkeoqntFoASK9jmvCO1RnFbj/VwSUSrhAUS5nbO4+oqe2sQ0iK/SNYTAOfGslx6GxG/PDMl36ER4BX9hON07cRL/FZCjKOuKQluvoRoJfAsIE5ex0Ux X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(396003)(346002)(376002)(136003)(39860400002)(451199018)(40470700004)(36840700001)(46966006)(8676002)(81166007)(36860700001)(356005)(82740400003)(70206006)(316002)(36756003)(82310400005)(8936002)(5660300002)(4326008)(110136005)(41300700001)(70586007)(2906002)(40460700003)(2616005)(336012)(40480700001)(83380400001)(426003)(7696005)(478600001)(47076005)(26005)(186003)(16526019)(6666004)(54906003)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 08:03:29.5929 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 84e0c807-f189-4def-00f7-08db042ace20 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT096.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7689 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 51914C001E X-Stat-Signature: 8bf386i6h8zg5czc41mwuzn3njzi5n6y X-HE-Tag: 1675238614-498498 X-HE-Meta: U2FsdGVkX18waI9HFbH5ECgl4H4ly8UFkkb+3YIj99cxRyhu+6ri88/afU29oubZh8Jp1/MFERqF3Dt1WT8IdV0Ya8wMnoslbSKfquyvYSvFeiVCSVnQCc6m8TRUtWaijBKvxIX2bBtWKoHK6Ttu+UY9KUq5Fy0Rv8cPIeW5PYT+mHqp3CCwTO3g821ODp+3ptNnf4sAbMZ11PMddswSGSy5lx80cb2UL1a4NwZmmZtUmxD6gNswDk927Y6JgaeAa8RmvuADLgBLBqIimXzuTMQXwfsNCnacRHJcy8gQvs5U9OEE//dboJ9xYcnutaB6XdE9/QEAQGLl4Cvfah01Lseiybd4pEHN64xSoI7GwunAY9Nz3b121dGpW3HYW1SWqrlgKewJYKfgNs6Ls0wfsVLeiiOT3ocf6DmWoZuGZsdZTvuFOpLIY2yebajvL16rrxo979yObIbRnIzjUw8jJBapPIa+Ak9sIeJVpqvsyDSPom1bIiNOAXFh6QzkI9CFLdYknpBAWIKHOhKCx75pst1cAzVBTrq3ROtkzfCAoPwqdXGiG8VtS9Ro13brkj0TIw/7EDOLdZXvMbGzLnkDM5i8KhBhkDqB635lvXjSpQqW/vTfSOk0Ed8xBhIgxLbaiY6lQCDa+QjyWsR0UuiM9yKG6SuzIMUbvLRYO63JXHKI6aTBA9SUJLgrrDn7nVcNSy7QJ8CO4BEXXlBRnj0ci9TbwGpQWIjA0HshGsUhXwaaCXwtC51wgvt4pkTwDrjHhdIhEqgIZUcMkhcH+TcU1EE40y5zHaTS10V4P8qxPHJdL4Vx9Bifui0j1KuOmcBRUQ4leE7yOoGQXqQnbV4HfHsJQ1U1dcHn6KFPxCa6YmL9Knc1MU3tPd97M4w5ArhRiHS4SgcK4vx78PwRKXQlsl5wwew8WRplLRCSbVIX/oA37/K259TYvhsaPvW3TX/4C3fM9Tsxz0cvIL4UaRA woQnJgU5 VcDEwvbYhcbinDu3zAWHrwxoj0LqKoOisuGA9qtCbtG6aaM/GNEPO8K+0Hqkc8G3t8e3/akJAdPWvTdl9phz79ne0ae6JdCKeMHlh6EuN6W3XaUi/r3Xm2Iz+x+Uu6AwWNulXg2Jng6axgpYEVM697nSfWjge4amE+wtwObFS7uCEmW8pu5/nAbAT0Lpz6yIwWS7ONbN6P4zb/DsIvfHWbnpbnTvOYTFCsXCBg1CJA5OWeTwiFBM7qJ8QkyzaIW5Xldzh2qLir9+ni7m2q7eMWXFFWV06W36qk2bk4dRYceroHQGOi9fjq91IPQPs5J/Ey8RcwaZnjXIkJef/VnTaR4c+wa6j2Xtywmg4tfkIO9yZR56swzus+hPl4ip4m6btx9wo7W+KKIvTI8iJJB+YMOWFWVg6XpEBUOCzb/jOxHATVeKbG2t7a3D273Khl7O2DgX62BBiwnRD7jeZ3sc61qJohrbjxDVv2EqgxHAzcmXyaB+TQPCNHDROucp0AUyH9v6ylTaTAD0/FinO0hhJ/0lCaEEKsL9pIxe9luPh8j64IwO+RhKPwgVkUvsWyUmB7rS8b/Jy+vgozjTC5VtarVsjuGuoSgjjqb4g X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: During the Numa scanning make sure only relevant vmas of the tasks are scanned. Before: All the tasks of a process participate in scanning the vma even if they do not access vma in it's lifespan. Now: Except cases of first few unconditional scans, if a process do not touch vma (exluding false positive cases of PID collisions) tasks no longer scan all vma. Logic used: 1) 6 bits of PID used to mark active bit in vma numab status during fault to remember PIDs accessing vma. (Thanks Mel) 2) Subsequently in scan path, vma scanning is skipped if current PID had not accessed vma. 3) First two times we do allow unconditional scan to preserve earlier behaviour of scanning. Acknowledgement to Bharata B Rao for initial patch to store pid information. Suggested-by: Mel Gorman Signed-off-by: Raghavendra K T --- include/linux/mm.h | 14 ++++++++++++++ include/linux/mm_types.h | 1 + kernel/sched/fair.c | 15 +++++++++++++++ mm/huge_memory.c | 1 + mm/memory.c | 1 + 5 files changed, 32 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 74d9df1d8982..489422942482 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1381,6 +1381,16 @@ static inline int xchg_page_access_time(struct page *page, int time) last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS); return last_time << PAGE_ACCESS_TIME_BUCKETS; } + +static inline void vma_set_active_pid_bit(struct vm_area_struct *vma) +{ + unsigned int active_pid_bit; + + if (vma->numab) { + active_pid_bit = current->pid % BITS_PER_LONG; + vma->numab->accessing_pids |= 1UL << active_pid_bit; + } +} #else /* !CONFIG_NUMA_BALANCING */ static inline int page_cpupid_xchg_last(struct page *page, int cpupid) { @@ -1430,6 +1440,10 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid) { return false; } + +static inline void vma_set_active_pid_bit(struct vm_area_struct *vma) +{ +} #endif /* CONFIG_NUMA_BALANCING */ #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e84f95a77321..980a6a4308b6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -437,6 +437,7 @@ struct anon_vma_name { struct vma_numab { unsigned long next_scan; + unsigned long accessing_pids; }; /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 060b241ce3c5..3505ae57c07c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2916,6 +2916,18 @@ static void reset_ptenuma_scan(struct task_struct *p) p->mm->numa_scan_offset = 0; } +static bool vma_is_accessed(struct vm_area_struct *vma) +{ + unsigned int active_pid_bit; + + if (READ_ONCE(current->mm->numa_scan_seq) < 2) + return true; + + active_pid_bit = current->pid % BITS_PER_LONG; + + return vma->numab->accessing_pids & (1UL << active_pid_bit); +} + /* * The expensive part of numa migration is done from task_work context. * Triggered from task_tick_numa(). @@ -3032,6 +3044,9 @@ static void task_numa_work(struct callback_head *work) if (mm->numa_scan_seq && time_before(jiffies, vma->numab->next_scan)) continue; + if (!vma_is_accessed(vma)) + continue; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 811d19b5c4f6..d908aa95f3c3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1485,6 +1485,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) bool was_writable = pmd_savedwrite(oldpmd); int flags = 0; + vma_set_active_pid_bit(vma); vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { spin_unlock(vmf->ptl); diff --git a/mm/memory.c b/mm/memory.c index 8c8420934d60..2ec3045cb8b3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4718,6 +4718,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) bool was_writable = pte_savedwrite(vmf->orig_pte); int flags = 0; + vma_set_active_pid_bit(vma); /* * The "pte" at this point cannot be used safely without * validation through pte_unmap_same(). It's of NUMA type but From patchwork Wed Feb 1 08:02:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 13123895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5278C38142 for ; Wed, 1 Feb 2023 08:03:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F2076B0078; Wed, 1 Feb 2023 03:03:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A2506B007B; Wed, 1 Feb 2023 03:03:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2428E6B007D; Wed, 1 Feb 2023 03:03:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 13FC76B0078 for ; Wed, 1 Feb 2023 03:03:47 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id E56D81A0A73 for ; Wed, 1 Feb 2023 08:03:46 +0000 (UTC) X-FDA: 80417984052.29.D5F5145 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2042.outbound.protection.outlook.com [40.107.223.42]) by imf21.hostedemail.com (Postfix) with ESMTP id EFC061C0003 for ; Wed, 1 Feb 2023 08:03:43 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=qirlsF8c; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf21.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.42 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675238624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zBy421iEXPoI1DU2317/qdpzKhPgWFnxaUG/6sW3lQg=; b=c+8hOoZiIvEeqWfVZv5MczQ8SdugW9qkv/uoZo26SAgUSTW1O0r97z/s9kePdV6rc3LeV2 u55jzW10LdIw9pWtIKhX51yWc+TTkIMcy/9BoIqYaTM66KHoT7ih8MSi5inXuQPbJ6Anxf vxyk0jCQ8nwxUgB9oVPrJsibk2T5FgY= ARC-Authentication-Results: i=2; imf21.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=qirlsF8c; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf21.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.223.42 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1675238624; a=rsa-sha256; cv=pass; b=LhKPyPUhp/pYAv12B2HunhNR3G0oFJPQKKZ9NWRqK14LDfgrYr5wznaQPKkWQ667XR2t/5 Y0MfszY4CReppjL/zuhCXxpSKM09231boooNGWuGOt0DKAyPY3PUJ6Rel7xouUQpdhi3Zl d1t9pus4fHcSTh3/2A4wb6W1jOtdvbQ= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QutHDQm4lZeAIrwxYF/HoNq/20PqkqrHp+UedaZawYjNZfMVLUGMahVGSeIwQkpgGs0mpXNZRqCh/WlcqU3UdfJePrtjOJgb+AV+NDyq19nYXlU0v35hQNjDpfzcreIRrU3h5yVjz1UObAz+jds8X7Zpt7df4TZv2hBVgC4A7yJI0dHMjnXm9HBJ0mjaeS0UXY0gy4Ib5mVbYrGNocplsUBu3kHnsvGZKtb+D+GhVT/knKyeRkSYCdd/VqIUlzOaJU3pV7vpd4IlNpn8y06YtSo95Qj2eNvsDJ4IHIA7Gt11isAdUW9FamhAZaOrSpWnTbrRF4EwoasZUbzdqF8M2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zBy421iEXPoI1DU2317/qdpzKhPgWFnxaUG/6sW3lQg=; b=EVxB//Q5DkKMXQ+ICFbhLggcbgDfTtefYMDneP9KRL360VFdla2R3Uf4Vm8sVVx8gebO+PoTLQN8RmW1ETaKAV7qbCI7uSXivdGubmjpUdnbt96PkBQat+aiAHegartWGH/bpEV1jf8U6ILc3IUOa7deaCx/nJtUjgQ9/cWxDlxj+tY7btoPH/nx+aMh+/6lqB8nWA4C7K/2mtMWoK9NVvfzdA2lqoasdHiJbeitutXFPZ5oNNSDVlAaljxKwA5bamCQIPbfCe2md+mtKOUwwJa70ROfMf7GhNwDQHRxQKG5vpiyar1oXHIker9wiBIXX3LvwfAfXSEKwjwnMXiNgw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zBy421iEXPoI1DU2317/qdpzKhPgWFnxaUG/6sW3lQg=; b=qirlsF8c/7kHm5TSItSbvRRqT6avHT5eRJREx0qNRCRuqsknuODdeUwxfE79Kx/tRwUrv6CpkRz2EHutgBB80NaBe6hUzYBAUMPyOXZ84sceCFCz2NiDP6yOsu7dZMKC6VE6qbSPxRHVuhKakwG4KRIwJsUtBZStMk+pMd4g7hE= Received: from BN1PR14CA0014.namprd14.prod.outlook.com (2603:10b6:408:e3::19) by MN0PR12MB6272.namprd12.prod.outlook.com (2603:10b6:208:3c0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38; Wed, 1 Feb 2023 08:03:41 +0000 Received: from BN8NAM11FT086.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e3:cafe::fd) by BN1PR14CA0014.outlook.office365.com (2603:10b6:408:e3::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38 via Frontend Transport; Wed, 1 Feb 2023 08:03:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT086.mail.protection.outlook.com (10.13.176.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6064.22 via Frontend Transport; Wed, 1 Feb 2023 08:03:41 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Feb 2023 02:03:29 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Raghavendra K T Subject: [PATCH V2 3/3] sched/numa: Reset the accessing PID information periodically Date: Wed, 1 Feb 2023 13:32:22 +0530 Message-ID: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT086:EE_|MN0PR12MB6272:EE_ X-MS-Office365-Filtering-Correlation-Id: 753a8f99-9e26-4f55-bbcb-08db042ad528 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oLTh+cidC2YJSLzJPpFtDSWZ3aa4tbq+BHJD3eBC6XHLkMqyP9hHdLxDGiybeu5SHHs/kpch6WSBZhoXfUFK5xYGVaPOfwpkY/94OX9RiWssyqd9f9gb+TwEmz2ZLq58Ef2blCJxcQN3jO5l3XXwNQgyGike9bC8tmDCDzIdf8ScSMObuW/nq69xTH0h0PInmlUjH3SwHLXDaquVLALw23NPkhKM2hoVrmpNpQm5pfEUprNrM2+ddxBzwU7wcELSL/pzWU4MP6TxkQ+wXtyP+9Ed3papyn+MhAH6RpVvbjfa+dfdaBa/TSu5UBUbi9pDa987wufya1V50YG7e91qHLbq/snFoqj0m6hf7YRUnGqhQ4WoHlaDjMcP/4ycYQtOCCyJnI5tMLHiTH6P1t0sA8rGYqRreldX6xuMWlVaZ9BRtENHtqTO6/gfkZg0P+I9NIq7Jbxu/ziJnU3DlJJXbpnF8HH8hTlWlWfMuv3tEnIz2DOodjXTQyNtvH5HTTxjGUPmbnF9aLHwQP2IuWRZLTtd4BWX5+FqM+BL15qADgK1iv/iYaJCCDfleHHx6586LSrf/2s7NfR5hThK/bX2e8AlB3ii+6CkqQtrb4WrzqkeNEhRuOGDuF01hw4XksVZUvAbaGtFC1eqjZijvxWVZsoLsUrFz8b1y2VQ0U5/DCEidpIovQekiK+STLU+kpSuX1r7+tfitM1G1MG0Q8A+2LYBij/w1RoOuDUE3ikwaJJCjc5pWqLnWxJ6Rw15ws5Q X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(346002)(39860400002)(376002)(396003)(136003)(451199018)(46966006)(40470700004)(36840700001)(356005)(81166007)(82740400003)(40480700001)(82310400005)(36756003)(40460700003)(336012)(110136005)(426003)(7696005)(316002)(54906003)(47076005)(2616005)(6666004)(478600001)(26005)(16526019)(186003)(41300700001)(36860700001)(70206006)(70586007)(5660300002)(4326008)(83380400001)(2906002)(8676002)(8936002)(36900700001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 08:03:41.3942 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 753a8f99-9e26-4f55-bbcb-08db042ad528 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT086.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6272 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EFC061C0003 X-Rspam-User: X-Stat-Signature: xpbtnza1hzc5e8nc9b6xcd1mffd6imne X-HE-Tag: 1675238623-402735 X-HE-Meta: U2FsdGVkX1+24I6PBrfgU9r2M2w4gyLAfSy5EIBoo0nXbOQodApgIC7PD7vfJVKVosVeFPt3E8jD5ZVcMaR4goSpVvWJfwsP8FMthJ3mYaDlS+8Tfj4Hp5RJquPISFtu3BvAPE1Kvd5dDYUelaSAWud7cOd4VikhyKsJ7CtH9IG9LQRzV2LvoU89sGC0wkzahEsBAnfAV0C/p41GUpST4FTZ/KNGERiQni/jfm+wpT7pbZgffBogvLiY1hLjhBbsSB4ChIhyq73NnqMSnKMAXybKQPzpwv7+wYYcCa3MobGAF/VVX7EnHW1okENyj9NaooxCCNkMjGlT89V8QWWOORVDpY1ghq5RTE6TgU/ljyA2BaapOAU6brs7kXMnvvbHcfWKMySiqchFwqpyKq9oSPHW+0vf+8xjtZePJNQIk1Ulyco+fqDApzkkHvrrTUMy7JMxT0Pr6UA7Tn9ZlzK0jqoxK+q7YfMIrjxDNrVCV8/cRrC8J20jHQo0/Hn7hiWIFAMi+T15UAxCthv1YC/djty2rp9iYgQDWmUDON/kz+Z81PMseEbpgwl35fnm4FPtkHNbdb8WZJDLX5FSyRgqDP6QdYVYQMP4H4Tr+9rZ+2k/2uxL5D9trjx0c6EQ/6acWz06s3AVKakcDIG9pSeigD7p0b8yIwQpna+EG7J9gCcNFfEvjr5odtgVhxHsBibigF43Bz3syuhMyiCeuE0nl3Xq299afBnNvTMV02fTpwmDIWSsisEwSI/r013clkKjaPL1GD2TmyJ2B3hmtlKEfhMDRYw8S529bdDVZFJWeQQHKRt3IUyiCZZrKFWjgN7IdWbHHj0nNSCLn9ix0z3NWZHUbfE6JNGrXBjAj1FsVlUVemR78LiFsU0IfIpcku5yFRLEL0U85sqIxRQJtgUcTug2ACW7i0okSQhrRGj5TGyvLfXgj0R8vbD9lFUX4lEcOHoeoLsdvyt51Q9agh4 Z+iUDTcL 7XJzvFomrSEkmePlwuq34N7DseC9+zil751Nda86o9wfyYXRhjXLiOKjCh3Dr7zZX+ZFpyrPA8Qc/Qq59mU1xJC11qD/IcS5KV1CnhsCL3pRIFF8mizZpulpAbvi17MDythA0GFT+U0gMt6q37jca+sLbLRD2wKI/WkPP+zeaWyimylZ6emgsIZIRu1vzEgG9Ntuer4zLDwhZLWle0yk+DGo31jnYmQFhmCdQ+PjgOzqR/xulOzJ6/eLSlxzDGD14inNByjRUeykapZ64p8zjN4x1pJu4CUOqm82XQ1ZtEpPF7cDPL6T8eDQvl2O8OGweVqUfFl5BhdIL4xAKiMjy5/Bcxd5HX4l4SSIfHA4l8pIWhIhSK0aR3+0sHPFW029rp+1NlhT7mHfo0hCpszAj2BLE1yckg71jN0fqC3Jjx613eFOaXmYsYBUZJJyRw4XWtA1QyGQpCCZaE44pDAZX2afqNoJGw625SEwNzZu6wl7KrmZ1jyNTkZxIiLjX8/wx3gcgsy/K+FFbudOK1peJBkPyhpKS6VbaM3UeK3NQ0q7Zopa9uSstPiBFooBT5uLLc6hzjnLqBAXyhIOkysnB/S+Yt5sVVL68pC0v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helps to ensure, only recently accessed PIDs scan the VMAs. Current implementation: Reset accessing PIDs every (4 * sysctl_numa_balancing_scan_delay) interval after initial scan delay period expires. The reset logic is implemented in scan path Suggested-by: Mel Gorman Signed-off-by: Raghavendra K T --- Some of the potential ideas for clearing the accessing PIDs 1) Flag to indicate phase in life cycle of vma and tie with timestamp (reuse next_scan or so) VMA life cycle t1 t2 t3 t4 t5 t6 |<- DS ->|<- US ->|<- CS ->|<- US ->|<- CS ->| flags used to indicate whether we are in DS/CS/US phase DS (delay scan): Initial phase where scan is avoided for new VMA US (unconditional scan): Brief period where scanning is allowed irrespective of task faulting the VMA CS (conditional scan) : Longer conditiona scanning phase where task scanning is allowed only for VMA of interest 2) Maintain duplicate list of accessing PIDs to keep track of history of access. and switch/reset. use OR operation during iteration Two lists of PIDs maintained. At regular interval old list is reset and we make current list as old list At any point of time tracking of PIDs accessing VMA is determined by ORing list1 and list2 accessing_pids_list1 <- current list accessing_pids_list2 <- old list 3) Maintain per vma numa_seq also Currently numa_seq (how many times we are scanning entire set of VMAs) is maintained at mm level. Having numa_seq (almost like how many times the current VMA considered for scanning) per VMA may be helpful in some context (for e.g., whether we need to allow VMA scanning unconditionally for a newly created VMA). 4) Reset accessing PIDs at regular intervals (current implementation) t1 t2 t3 t4 t5 t6 |<- DS ->|<- CS ->|<- CS ->|<- CS ->|<- CS ->| The current implementation resets accessing PIDs every 4*scan_delay intervals after initial scan delay time expires. The reset logic is implemented in scan path include/linux/mm_types.h | 1 + kernel/sched/fair.c | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 980a6a4308b6..08a007744ea1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -437,6 +437,7 @@ struct anon_vma_name { struct vma_numab { unsigned long next_scan; + unsigned long next_pid_reset; unsigned long accessing_pids; }; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3505ae57c07c..14db6d8a5090 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2928,6 +2928,8 @@ static bool vma_is_accessed(struct vm_area_struct *vma) return vma->numab->accessing_pids & (1UL << active_pid_bit); } +#define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay) + /* * The expensive part of numa migration is done from task_work context. * Triggered from task_tick_numa(). @@ -3035,6 +3037,10 @@ static void task_numa_work(struct callback_head *work) vma->numab->next_scan = now + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); + + /* Reset happens after 4 times scan delay of scan start */ + vma->numab->next_pid_reset = vma->numab->next_scan + + msecs_to_jiffies(VMA_PID_RESET_PERIOD); } /* @@ -3047,6 +3053,17 @@ static void task_numa_work(struct callback_head *work) if (!vma_is_accessed(vma)) continue; + /* + * RESET accessing PIDs regularly for old VMAs. Resetting after checking + * vma for recent access to avoid clearing PID info before access.. + */ + if (mm->numa_scan_seq && + time_after(jiffies, vma->numab->next_pid_reset)) { + vma->numab->next_pid_reset = vma->numab->next_pid_reset + + msecs_to_jiffies(VMA_PID_RESET_PERIOD); + vma->numab->accessing_pids = 0; + } + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);