From patchwork Tue Feb 28 04:50:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 13154412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2658CC64ED6 for ; Tue, 28 Feb 2023 04:51:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 870756B0075; Mon, 27 Feb 2023 23:51:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 801956B0078; Mon, 27 Feb 2023 23:51:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 671756B007B; Mon, 27 Feb 2023 23:51:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 51AEA6B0075 for ; Mon, 27 Feb 2023 23:51:44 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 20EF2C1038 for ; Tue, 28 Feb 2023 04:51:44 +0000 (UTC) X-FDA: 80515477728.06.F759F85 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2082.outbound.protection.outlook.com [40.107.244.82]) by imf05.hostedemail.com (Postfix) with ESMTP id 132C7100009 for ; Tue, 28 Feb 2023 04:51:40 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=fedlaKWw; spf=pass (imf05.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.244.82 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677559901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q6pRaeAm2S0aSVYFRoe0W0nrXqMgPX0/7nNuAYxeMTY=; b=K0ttIorNgRj8DtbqJ8m4K/aZVDG3kagbNm8QgwA1h/pAtrgc8Yctc2U0WshpUv/knIsPIg TBbfNuwbSsxYqbj32aULz2ra9BCETUse4C3BFK66b5SC2ft3glH/R6eA25AfV6NwlojoHR olHFflZsyax5MEmCtNbyIyawjJMnDQM= ARC-Authentication-Results: i=2; imf05.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=fedlaKWw; spf=pass (imf05.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.244.82 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1677559901; a=rsa-sha256; cv=pass; b=ifuVD0W+/IK2veHTHsCA8uEvOU/2MAB+s2Ev/YoS4g5dhFSGh7s4DzrfloUFYgEmOATHKx jxTGgGFOKX8lr8GQBIDlPynSeuSoU22gJNqXPxy/1rbXTMUqALNs+KS5lozLJC0RkBTvyS pVwnBhDBDb0tb3ZHH/+wKTpe8Vv6A10= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nZShD4lpYv+i0nK6bWP4Ouj/0xHZwo7ROlnxQNmDlVhr0Ra+SDIG9RH2mNr9s95r7BkAHcOflFXDpZUiGB/Kla/97Sh06CuRGTkDBkGv7R5JVP0+AUEkQXGdFAXXVdg2h1kjFP27wb06a6AEgoVDI//A7OBx0g4sPkvIDuZ8CJkwhLWym/DvzadW7nvTa3eXvY4IkGB9bMZj9NCYkTATu2DvjHTIxhdrBCtAW64hOpjHf06AHKhnhzoQ8LpSUXD2667qrDbe0U3cm4sTlGHQVqV8wmmXmdccFACMnG5CCT/RkQziZxtZwHpOELgPFajudcM7HAzhnIb+J5v/agyp1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=q6pRaeAm2S0aSVYFRoe0W0nrXqMgPX0/7nNuAYxeMTY=; b=ImwK5qFbqn//e9r+1Gdi04FB5wWuxD6FIrvBM8sPXD+YrIGj2ywQ7ueZQMoY2gpZ+LLubyKd/1npWBo8ysHWtOuoya2WcVxzw7TBDs2R8WZ8pSpeGtntn5ljcfst2WhHzQQfCGSwKEl2lDbBimEvyRwwwRFyJljQhymoWuXF8l53c1By6e1pRB19IKaVtfm9FhKCBNMBqYHmzdxCcYFLv8bC0ZPn48aconEZtOlMqq/ksxET0TOkqKPY7MwbB2qfg81oofU8OIQ6ndKlJLC8MxzoD/6L92/A1qRLq6zsghn/7jR7A/cYaPjIAYSTXIGIPOt9VOgicBm0HFime6FPsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=q6pRaeAm2S0aSVYFRoe0W0nrXqMgPX0/7nNuAYxeMTY=; b=fedlaKWwRNK/HB4XfJ0pgVUC1wmXCTxAVSK+aAA3ZsMNF5+s3dwtR223ihxAxSvt6lXTdANSDyZ0FuCoLLupd+rQvq334rK/hIbyMfLurZ9w1QjBe8yw/v0sS1SX99po5z7N8T5MjcajNMr8SIQIDlxngCvAbene1e2lutESl18= Received: from MW4PR03CA0328.namprd03.prod.outlook.com (2603:10b6:303:dd::33) by SA1PR12MB5660.namprd12.prod.outlook.com (2603:10b6:806:238::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.27; Tue, 28 Feb 2023 04:51:36 +0000 Received: from CO1NAM11FT040.eop-nam11.prod.protection.outlook.com (2603:10b6:303:dd:cafe::b1) by MW4PR03CA0328.outlook.office365.com (2603:10b6:303:dd::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6134.29 via Frontend Transport; Tue, 28 Feb 2023 04:51:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1NAM11FT040.mail.protection.outlook.com (10.13.174.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6156.17 via Frontend Transport; Tue, 28 Feb 2023 04:51:35 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Mon, 27 Feb 2023 22:51:29 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Mel Gorman , Raghavendra K T Subject: [PATCH V3 1/4] sched/numa: Apply the scan delay to every new vma Date: Tue, 28 Feb 2023 10:20:19 +0530 Message-ID: <85e573d315109507fccf8e1b0906d90cb2039cc8.1677557481.git.raghavendra.kt@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT040:EE_|SA1PR12MB5660:EE_ X-MS-Office365-Filtering-Correlation-Id: 02e3038c-2630-4515-c1c6-08db194778a2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4x14G/ew6qo47jZ0L8C9jCWb3wfSsBKO9vcz4k6DNaFSxjLL3lw/B0+edEe1dnATi0lgxvHD5NmVjogZFzacp2AKtQ/vgdr03IXtgHlbDhi1XGxk/eH+JWC+RVObAWQTzPrqoYO8XB9ARO6E30rOsFip7tDcFfPHATnRKhirn+pFwUu9ePLo8yVznsqAh1kdaKHMbjMbI8bD4AyD9QLyl7VSxfPF5zzZsEChe4jN6Rc3f6Qa7hvPjVQkbIprVdaIGUUdRWCDH98sOI1tOK5DDlD0T9FZKqLtnZsNIcpQqqFO6X9vQKop156mdVrW1v4AoBg0424qK9M2yZmxuR2vhrEK9JWUdeFf8DCrSt252TdCiWUcab4IL/r2eeHmHXvugXM4/9/VeM6Ch4Nor+Zd9BE6nlJVA3NwjkjXJYZvz6pwHcoPv41fm7nSMZdhfO350vYvrog/LxYduSqmhYckRyu9olyMPVv51iUS242sdwwxjCXY3kz6RUj4onCE5AfFRmd1ZPkzPtgkwycMEvgqBSWpFyUq4YZjHuui+H9/ZFj9kVDL6EowSkk57bXnioPfNcIKWTSG1/Ox3CZPJTMsD7KlyIhKk/CZyQZ1CnUPmwaxoQpWR9ww3TYdem4Vm9zeipiYp2IF8f9pR+Fm2HRi7terUtP7dtzG6oT/II4VHKRjckRsTOqGR54YRSZndUsHBNSb61P45aM5BxbfddJfLfn6rht2I0wR2w1zItecvx5yUknFp/SbN1RIca3SPHpP X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(39860400002)(346002)(136003)(451199018)(40470700004)(46966006)(36840700001)(8936002)(5660300002)(41300700001)(70586007)(8676002)(4326008)(70206006)(2906002)(110136005)(316002)(54906003)(478600001)(7696005)(6666004)(36860700001)(426003)(47076005)(82310400005)(26005)(186003)(16526019)(2616005)(336012)(36756003)(83380400001)(356005)(82740400003)(81166007)(40460700003)(40480700001)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Feb 2023 04:51:35.8955 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 02e3038c-2630-4515-c1c6-08db194778a2 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT040.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB5660 X-Rspamd-Queue-Id: 132C7100009 X-Stat-Signature: ofs4z6jmbkehrouqx3nrrs6quodnwx7d X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677559900-255639 X-HE-Meta: U2FsdGVkX19WhSSxRkQiRteMa+cobpf2KyiK0aznMyIg+RtuX85EwyiokBPyIbbDInQojp/fLX4Yc2jbZ/TM8XyfZ+83BXgnDiIBgvbpBHkjajcTCJbXTkiCJhTaZkZc9A043yP7xRKXqJvC67LF0Lfd/uU+k6YQP4XFHsFacpl9uMwMUp4KSZmo8t2CHLCYBdheMhYrVDgIs0R+hUh94Cu7cQ7ZSWibUtbm7a+h5ZEtfY+Iun1rHPOqBjoRybzgz1V02/aoziT7FUovUOK6Tp71X3S6A/cnGRwLSFrO4g0NKtiaxB/WY6ayCNDqF8Iwr25NhftBatyNEh4tlOuFheEmDP1ibxp8iHHLOpxVfjOOo2T3h/1+Lr2hqMw1uyro7fsFjEHevw4gj1KO6+9g8Tc2sPsO7iPinCtdWJd9y172RGxj6fL3VxaTf87jHiRkTLcAEdzQ0mOigjTxQFnFlqw8D0o7cQJWsXOJOj+TqAji3YlmBsG4MIPOO2C3PDwPJ5ZmGe78CSA19TyA5BXd4oTBZFF2DaS6bYFZmY4AWrbAKFNKJWB5VeGUoHy5tHEhPqyWfTC60C7zJn1tsCRm7Tm30eeDgkRRAKzjXFtMNSUE3GRYUQsQpd3RyXfT93dPM+4lvMMrCZ8dqBOSuKm7apAp3sXRyKxShmaGJI2I4vnLyqHntFKbIjVY51NIktKTB3NHMYhYTcu3wuWqHNry5q2jMOPZpRG2+WI9ljQfeoMkSMze7AI50mB7J4ypqlQbXQ0G59WWhOlrW9Z64u9WrhT4bU/8kKLda4RZgvVCJBPRrGgPr98i3WE9pQx6+XG3KXN2NJdxjDhpmdxwnKVgyNwdRY8Tu3MFsu98y0ISFoX+aZPVSbhSYGUosZeNqExe8/napiJbeYAAHprnsKsa1B12OA+zHG3CtGAKrpPZiRfB22eP71S3mQGPTYx57aD21M/pnMF9pKvNVK9naZ/ CmYLl3XH F127nFoRg8/27vwJl5dBOXYiBWs7BW1eFSUTWihDBoBJTBwspvEXfco3DxCWPtj2CuayKUT7pyot9zX0NtWe+SE7dItA1Zr66HKiYmLvs+16UuLbIFZmUuYPBoVHwJ5+/PuZjClq2slCMRME8pFN2MuQSU6o01L6AI4uZJSBIxNzRZmsVIdgOYD7UAtny46OEaLkV+9UXFhq+LVmh9qiZolLYWYVtPjyYGLkbKyDP7PxIWhQ8ngXLkNCqE7mhByxfqLDdvdbsIglPwmpVqiQnT65Vh+VnxnFonI5ybu7TW71EdOpf1LMzwj37vLx23VeSP8DFwCIhPLc63TnExtz1zceU9Cc357FdiN6ssjQnAxzl+fp9B0GPdhvB6AfQXwQb9i+k1xBeRwKkqaLoYCQ0zMOhSd0RtTK8wbpXw9fn+B0QPn6WPbF/Bds4C1dCIEfUGRxq8nb1Kx+DfW8wmNpz3OiNn9D2nlidxEoj6ippj/w7cHTWs6pP3G0Erv/EbHuRi/urhLRfJAgU+8H9bz24l1LgG+Rz6O4CDbEt1t7pd337hy3EP6wx4/f6i6pEwlWms9J96Ac5bMZB/qs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mel Gorman Currently whenever a new task is created we wait for sysctl_numa_balancing_scan_delay to avoid unnessary scanning overhead. Extend the same logic to new or very short-lived VMAs. (Raghavendra: Add initialization in vm_area_dup()) Signed-off-by: Mel Gorman Signed-off-by: Raghavendra K T --- include/linux/mm.h | 16 ++++++++++++++++ include/linux/mm_types.h | 7 +++++++ kernel/fork.c | 2 ++ kernel/sched/fair.c | 19 +++++++++++++++++++ 4 files changed, 44 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 974ccca609d2..41cc8997d4e5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -29,6 +29,7 @@ #include #include #include +#include struct mempolicy; struct anon_vma; @@ -611,6 +612,20 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_NUMA_BALANCING +static inline void vma_numab_state_init(struct vm_area_struct *vma) +{ + vma->numab_state = NULL; +} +static inline void vma_numab_state_free(struct vm_area_struct *vma) +{ + kfree(vma->numab_state); +} +#else +static inline void vma_numab_state_init(struct vm_area_struct *vma) {} +static inline void vma_numab_state_free(struct vm_area_struct *vma) {} +#endif /* CONFIG_NUMA_BALANCING */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +634,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_numab_state_init(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 500e536796ca..a4a1093870d3 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -435,6 +435,10 @@ struct anon_vma_name { char name[]; }; +struct vma_numab_state { + unsigned long next_scan; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -504,6 +508,9 @@ struct vm_area_struct { #endif #ifdef CONFIG_NUMA struct mempolicy *vm_policy; /* NUMA policy for the VMA */ +#endif +#ifdef CONFIG_NUMA_BALANCING + struct vma_numab_state *numab_state; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout; diff --git a/kernel/fork.c b/kernel/fork.c index 08969f5aa38d..6c19a3305990 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_numab_state_init(new); dup_anon_vma_name(orig, new); } return new; @@ -481,6 +482,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) void vm_area_free(struct vm_area_struct *vma) { + vma_numab_state_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e4a0b8bd941c..e39c36e71cec 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3015,6 +3015,25 @@ static void task_numa_work(struct callback_head *work) if (!vma_is_accessible(vma)) continue; + /* Initialise new per-VMA NUMAB state. */ + if (!vma->numab_state) { + vma->numab_state = kzalloc(sizeof(struct vma_numab_state), + GFP_KERNEL); + if (!vma->numab_state) + continue; + + vma->numab_state->next_scan = now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); + } + + /* + * Scanning the VMA's of short lived tasks add more overhead. So + * delay the scan for new VMAs. + */ + if (mm->numa_scan_seq && time_before(jiffies, + vma->numab_state->next_scan)) + continue; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);