From patchwork Fri Feb 14 19:40:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975562 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2070.outbound.protection.outlook.com [40.107.244.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FC5F1C84AC; Fri, 14 Feb 2025 19:41:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562114; cv=fail; b=h+eTrhYukyQBRTdrldkVs/FY/4mpOxvQGGw3sXA26h5cQb+nDgz2HMj853MN6zWTEXX0bvFc79TDYVQG8thgBuRKerlGPznigBM10ujuNlDT4D9AJE3mowb5TwntgAW2W5qL1h2EWYOOReuw/BAENJ/iTPY/gwPu6xamrAcdYfU= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562114; c=relaxed/simple; bh=pQnE39Visvyu7+I/GD8/epZ6FIoOTGs0HAakpOTc+Fo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=u4pU3zKNJzECnmPJdSJ42mTqFDtv/0Les4jDbTZWtU2nVJWPhX5yAInctmdWbF9/H6lQfMIZsDg5Cgd3GXAEqZsLI53s6IM+MSUpJ049+Pdvlcq/iCq7Oe3NGWlnWtyLPvMgz3Va8GNfTP6PJGg0ZtZIE+PCsR673M4dSW0suJI= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=r3do4LjP; arc=fail smtp.client-ip=40.107.244.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="r3do4LjP" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FSgQ+QuXvOZQZTlnWASDgLmWA0KN6uKphklPYb1nZh7NRVtuTM1q9kv/RlHrHKNkVtYzrXjuSxFVZya/PFCz8BnxC3DbaXuLbT+FvkJV0ofp/zBdjHnFdasdTzrtvXkrcDcHiVtapYHgphIdcpKrSrD1Jg7w7X5QoBdzKGONZE1S2N3LGhbyRbz+chUYsIU4uvMON3W2IPI+psTMmdW3HZzwd0r5JpN0LH6+QjdzCRuN+skCIB3nmHlgQobK/lqS/9+s6tFaxVVQir8QxqDo7EptW7O6OHpDAOGi3EAVzCzUCrgtxYIujED10YXMCP0V+hUXWpxvnXY5fq/Q/Ree/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3D88WsA80FCFADNIXKmRk2q4zXQNOlHuP7WNgsVUjAc=; b=QtpOnxvmN6ik0daTZfaca4CvVmmSail/u3iQV4QOIGiUc56KZFAHp/fuhwC78FpneX7iqokt6IOX7C2l1jybs+a4Fu+migYdYf6p/yiOQwBLZLYcJG3dFIrs3npaNSoBHES5NsGDwhT9jC6U6Pz2RmqlUssYKcdc4rTsLNLDPFwVeT0KkVB7L1nuzrdktp5QkV4u04LFaF9hVevOICdua6yRQChvNh7ergc38R7o2cgPPJcKtA7FCo07wbpaWZpfCTJWTOl010ckvqlNd4vN5TD4IP4GO2ar8FdbpeDpMaZjdRdZmWuJgi4V2LVItCf4V0EJLqQWELOEcO36shxHcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3D88WsA80FCFADNIXKmRk2q4zXQNOlHuP7WNgsVUjAc=; b=r3do4LjPC3sglzvozDhgw/o5x/o/h80vPCZh7av8sKj4wUyysCdHRSzHNMrW7lj8UDzzworhynyjTod6p0RBPl//aUkgjm6AX4vR2kQ9Mj+PcaaOyckK6E2T+Oj+YOn6SD5szPdwFljaMPC6AwqOvkXo330KNix6W06HSMOQTwbqcXRgoPu1eSOVrvkbtyL4crLj/zcQrk//0WfdiWToJwclOMP51/XDoFUwcuhVt6QB6sQ0EShWJ9UbttEQYu6uEgj4gZvj2FiUj9P85Gs90zceQhgTOOZi3s1Pv7qAUYOHha+jwFNOAWC1I2z1gOSUQnou5/u+Z9VmNyUgGh41tA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SN7PR12MB7809.namprd12.prod.outlook.com (2603:10b6:806:34e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.16; Fri, 14 Feb 2025 19:41:50 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:41:50 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/8] nodemask: add nodes_copy() Date: Fri, 14 Feb 2025 20:40:00 +0100 Message-ID: <20250214194134.658939-2-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0139.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b8::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SN7PR12MB7809:EE_ X-MS-Office365-Filtering-Correlation-Id: daaea6f5-7129-4e1c-f8e8-08dd4d2fa03e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: JJ/9KhFibKPf1VwQ3H3eUVMsO+4J41IrkwyMI7uxg/P7Tf4AZCp754WsCwENylWEyscTqmAjx2ACCK4R8yg3ikxh4QZBIOInziOblLOyrxZQ5hwYxn/E9QmSWh4qWc/NbycfqijSwiZ4S8iRLxkVL/Y09E6lP5cF6Ys0aPIQBZf7Byi+vikOKqPehIgdtdkLkQMknnU89Wc8lZgDRhApkz/1AAD/IGzOxOIUV7fozgmkX453tWJPWKUtBvTrCh/9/vUSsuKPL7b+hRqJGFErDv3ZToxnEyPWd3vwUMF+jme0KltWPWAYqJDeEEP8uh9asFD5gIQ+M+1feJCx3TfFjWStaWLC5sRsv30IU2Fs/OcXW79IIkw0wXiGY/Qk03/xr9JjOf2yaNrZEtWk62J657EU3WVIck5ht3QFw/SARibm40EXFGBw5quY4U16ch84XcwjY7oBcd88Y22HYEjfPry4pnHRMda6zvtSRkYKtxJRWQ+dzpFmB5aj1oMBQnMMyjfRZ/o1WBzcdd4V8DiBrugkmz+cEmdCwFW+Eb7wQPZuNacwrPt1CTIg1ODnwm2Cy+j6DYSfCVPN0hYDoWwTtTYV40F4FUasRPbmUmwyaYeaUKNt4sySQbOaLTwIrzQ6uwEMUTRqI8597leuVKmexvSuePjkOVp681s8UOVhLEELEYruMFC4mfDIWpRO0WWGNswy94C6XsfAbedXRtrhZyBFBvXEGhCokk6KmniBsPnqeLNTwx4LqYX+t65DJ7aglUXJGcGbx3OXqv8qigXqxu4an2+eqm1+i7/lm3OlqXvH93zQQ/Dxrp/RAfiAZ9jYVd/wx4EA9zwm8h52CpKfGftKhi+yPz8+TV9jvSeEZX1NKAKp9L3aN2SCCvaIEyvPsBTpk3yDGjq8LiQhIDh2r3kDZrlnytXeLycQP9nS8kYWmr6SohkIjhAMVgikHLhLDv6Rp3chIz9PN90YV0LoxJhSyHiMw4NRTY0CD70JY4hD8C8KstxTxOISqWPu3RY9E3Gjdx+9WfuZDlr/wNuUsInbe0UhiTbIW6ObWYswxx+r5UmJcKMJ5mMpBSZg1Ov6Pq4GgOD4zYJTD3yntzC6EPYiE9jj52rwAysPfFumj8OaY+GC0VuoOlWg0fnWfoW2fP2201iQNSbs4YntlgaeuVxTGKzeiU98PgzmEfZGaGX1NIIZtaoUgHOAZ34UWQ2+qvrey4p4/7UsUKYQjrRVUIKX48OH6HL/5Q2uTAkn+GzMhBeKA8Dlagi0wapDAweDbCWRx/L/1gEP5Yw59C8U7hUFYEC+PHQOgKz6RHv+vEUS1ejExE5xdTIhVyHAjLdhuYO5aqiH4bTwKbG5lCXgUq0YhQiXloBKO9ArIOFCewV/UDz8+khabTLjzCLzb+yQ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: pEPGzztu0ZTkBLVkLEGIEa+5Tj9oVUwR/uHyAy/iszJrAJmZg+Txn1aoQgb44KxI3NUJQzNQLbZ6lisUQ+ERH34hTRSjGx8dyPrZpEyOtYSC5OjrSbbuGiKCXGn7KyWdgb4Pkbupqo8KAf1fM5CoXEad/VI5OBn7kthiF9cphjvpmuPvDOaGt4sX77HYgR7fcxcGx/iTUxZpJwZqnUnqZuE214w9dSeXd+MQ9AkON6QGKz8fdKs9dtpLqPeBFVleEMjJ/uTjA4C4RTReOHEGnjO0yChcw4jxPO3zGjTEpSkjxfLo/vDR237e8u2YKo5Ot9bJVl+v6Dmp9a9YRbvWtpbasTIZX2NKCDgTP1HJjFxNHdCqzPTMMOI97wkmpZKtyIhojI+k3iyN3LbNVfPVTZxJihSVdWv48YNcj4dwUBNx/bVQQREoe7OBYPuRvXD/zhMcoQxx7swQj4BznP58zlCj+mjcLpGCGPl67/RIzvO1yBalgzcpziKtOLVkKEUkBP63z70H0mbuzDqUEleEIegeAHiCvg4F8cGonpUrxKcDVLwbjSAs9SEylSy0dUtl7n40IYOCmYu3Ta4vZ7VsLtThv07cCVpPBJFEbg5RgQXM6vNRHZVQ9ba/F6y5BcqkKGNoury1lJjjLStkNXy3McahXKcIOUKF2lQqVYwz8cPe8cV8wkW5i4vDS/Uk7yfDIqMndG1CSdOAsU0EGu4Wn1x7OuQzQuxIH7eEQ3QiMGHV6/XiYCgXoNtkT6OlXXs4yyKSD6HirzqskWqscpczV1LGtsRuMXMaySAbKv3RuyKoQ1qcL4dUThKeTmIBNlRfrKUUbuaNcZ/YGQZOQcohHxi+AHto0emyNh6faw7OAFwWMrcNHISho8jYpPUMIu7kEU8uITJhxN1UZnNKQIKg9WaKVhOQizgkuRf0vp2s72lMxLvJH7o9NPY73phKMLXASeOe2lSBHRot8qpo7g+FxyuczdcaW8M7+yF2eEOsOIAPHxMNHQmy5KTcAJcd2cbg/0Ae6a8TGSXyDihYvKDQt2X5L/4UEvyv9ZXNeOJq5qp2IuB7gA7QbK1vxiDz4/0VMn7wflQCXNl618CdpFKKfSIkzYa+6abCij36sQv5uqkaYSRSd+3rc6s8yyqSNinxMJGrJ4TmVZtKYokUzF1mkCPjm4ALBxc3P7woqkOE6Rnf9sy7ZnlPXLSd76A5G+oLy7r7zrBabnRY270meAbYRiyc8iK+MMAh5cavogDYYYMnuEoyLtk9gozyITMzG0s3RQ697KI4N63pGIGFrVeQpIRWJVV8sZqaDzByZq63B59jiWOHmqV6OblOJZELaNzauKQRQzTXS84Ejrc16Reqpaq486e2/SWuZKwsG6NKWYOXcUt8RDoB7ySL84CQyBQdDpg4CrEaQyJISWLCx0XKGSho/GjtGR98ct/8ORGWG90L2RUtKRslgBKQ6Jkrom8ph0srpMz8dKapPm01In/uQvNtn7CGJRuDOkASTHwgQcaoGvywyCQ2Z522iu0DJlYw+Ocb9VF6ZgOJ1S8YcRMNvVqII2Exl+lZVlOpd0zx+1usuyYPjfpr9trsLsOszQIp X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: daaea6f5-7129-4e1c-f8e8-08dd4d2fa03e X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:41:50.6057 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uSpdfXCUQ+l0STH94iIJzRY8ICzzRFz36Rctt24jZRwXSHtS5U2ci3mcFejLFPMGjbh5BA+rpH/bhcGRIry32A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7809 From: Yury Norov Nodemasks API misses the plain nodes_copy() which is required in this series. Signed-off-by: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi --- include/linux/nodemask.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h index 9fd7a0ce9c1a7..41cf43c4e70f2 100644 --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -191,6 +191,13 @@ static __always_inline void __nodes_andnot(nodemask_t *dstp, const nodemask_t *s bitmap_andnot(dstp->bits, src1p->bits, src2p->bits, nbits); } +#define nodes_copy(dst, src) __nodes_copy(&(dst), &(src), MAX_NUMNODES) +static __always_inline void __nodes_copy(nodemask_t *dstp, + const nodemask_t *srcp, unsigned int nbits) +{ + bitmap_copy(dstp->bits, srcp->bits, nbits); +} + #define nodes_complement(dst, src) \ __nodes_complement(&(dst), &(src), MAX_NUMNODES) static __always_inline void __nodes_complement(nodemask_t *dstp, From patchwork Fri Feb 14 19:40:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975563 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2070.outbound.protection.outlook.com [40.107.243.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C374A1C861A; Fri, 14 Feb 2025 19:41:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.243.70 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562119; cv=fail; b=iSAIJ/6HBoDQuaBXlWg+rOA46nWgUlh2DFjFSeTvXvkHdBgpfG5q0f8AKNQcRKWUxhQO5mHGpgXsL8xe/0KulJptgJEAmEP2b3BazmGyXYfEK1J4Y3jRlf4VfTRMKhPYBLInkZQjcn4aO/CmD+0XEYsPBATfFoBJLctA1iTZlPg= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562119; c=relaxed/simple; bh=abpva88A29q1hGjevOu7GiccEPzOSauBGOJYgyNfJRg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=GSvzgeDhzoix4Dd39KUxUm0kIi0fQliBz+5C+n47plpawAia5bgp8sgEGe8LPBBKc4laVnyIWSfoLB/3k7ehkJf0S3K3l8PAQm2PCO4ORuWSBA1JIZVbJwUKXDYA2x7oPYqLmIWY7aU+aqyR1TalzNK2Zc6/+Fo/8PVjnTvGrP4= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=S3k+uz/D; arc=fail smtp.client-ip=40.107.243.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="S3k+uz/D" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xerXBTrWWY7O26lfyhkoJvI21AW/+bS+JRfxRdqvwSBBs3al3OUeEottNllrLh3MYHgxHByBP8VMhGdBPXBL/vUPANUlC0avhHm2GByu156xrfhMf8Hy4ktW0tL7SZ+lAGWl2MxPmWm5BuxNoiE9olmVI1cm/Mm9zU1HljPMKlUEVERuRiI+eVU587z6TNibU3AF81k0uE8XPDrlVo0L0AB9yd6rdnBtuS+bX2jDnzrW3LwDVtVsy741zFWu5R0ywykvEAxhV2SlpJIifSbMorg0y4qwPPmt8CmnAvRzg2LiTF/nsGjLFV+mXOMn/6KOeYsaOvnS7yOJbCo3G9YfNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QcDORQczBswtZLWD4PdRBfrw6VmL3jJuFgNG2lDku38=; b=sfSUCYHaUugV+RnIQny6xosM7i1N8AeykStdlCBJD7ESeYQgcNUFmpepxUy43T6pCrMI4UGJPnZ/dhf1e7IqvHs+IMRkNIb5osCt6sBk9msCphFzEwH7QVrpxxIdODNsTCfw8kmrLaxgMV3Or1NbPRyjbbORGkz/RQrHNPuckTjZiv40eV/rwf7lBOm5C1VSApWEcb9eLLS87i5kH1J2uPmDyNVm5uAQgEWix607XqPIjrlo2JgE1fgW62LqwCmlCe9lsQBr3azb8a7fYLeIv16Xt9dHRQMFL7QGQ+G+lD0egG2CyUDWwUjOQOBLbuXu28k+rCWRuOJDljj59EbiXw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QcDORQczBswtZLWD4PdRBfrw6VmL3jJuFgNG2lDku38=; b=S3k+uz/D+0IrcPwnxrZPEmqRgje11412mxNUvgcSCPXNW60nqEqS3YEmSxENZYTCcYipgg/rExc0pvOwDVW6TbynFCxzQjVPI3jLDIBI/IUnH1SqsfWF4pqObsqx1bnSnvj1q2qc+LdZXgDQFNlsRr1GLg5e5KrqNmKhG1hL3DqyczaGJSxcBr0CE5TAPPcW8ImDr2OKUI+JZfCRLD6Zi86mROHf91k7+ITsUn+xlulM5MX3SjRN45zvk1h4TExhC+/AIxjCAFW5nIf8w0SiSuFIaQX7Th/iW0Abg+UdJ1AaqmTyu3sZCAwuGJ/3GrFoS5k2cSrahZUH80PG4bvAPw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SN7PR12MB7809.namprd12.prod.outlook.com (2603:10b6:806:34e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.16; Fri, 14 Feb 2025 19:41:54 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:41:54 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/8] nodemask: numa: reorganize inclusion path Date: Fri, 14 Feb 2025 20:40:01 +0100 Message-ID: <20250214194134.658939-3-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0207.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:e5::16) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SN7PR12MB7809:EE_ X-MS-Office365-Filtering-Correlation-Id: 2882b00a-b830-4a76-5fe6-08dd4d2fa2d8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: cmxXnvDF4Fe005C7LmuJoPgOyNFJaFZJ7Gu+LzBpLrDulQpaNGFsWo3MoZ4AAVUyRza6kecaRk2Jp1Us42o/NvSA/91TJ3KggOR6AYLJ1ACy17YVuKxAxKUHBIZYZr1W6lGodZDN6X+rhFeHCap4oE8x2lBFhELYlutl6Iw9FSSPmP3snOA0yFOJSkN055tn97mDKKI1vugnDFfCX5LUNb+qLP5xMWvMYNvpw2r5/GJTXOcxeRlUM3RcT1oE7kz4YqyEL04zhV/fHfUQGDjihBYImqiD15x8yHhCjqP+ImhVosXkXHgnz9ba4Aoz0xnWCk0dxZN5bRCEpMbVY5t04+X6Ea4IyAJSbEmJ/uTJJCPM0rZhQ6euuiFlIZAIH8OGFfC3MtTvm6JtV1ZrpPFTdSO3NkPyDlfEvZ7FFKaKWzH3q9BdT+60PujEC7JDEYzR/HQLmNOSNkTFQpK4iNhlyUOta51ugxyFuQ7hWeqgLcnxICRnqfZrfIEYICOfUvG6aurC/wXVMQeF5dkh8MqYsHmF3W+lX9XRE7Pi6dxvcWtY9YvTD7V0baDKzv9SydVMdOCirw6HH0gCcwvVar3RQIIT99SxvOmboZriBBzhlrMbKGr+dHkNOutyrs7EDS48J1mdIdZUK4YoS7iYwcFX7SheoOvsXJahI5jaW6ua/dcAXVmOkkFrqYBUJmf9KyzpFmEOnKviAVlHl7bLAFbN79pqPzmsoIk5FVba9sn+FWBjzF9Bv0JPUJyqoWkDwWkyXVk1qe9Pcl9/cppDGYikuT6bs2SjEmojS4vFE8oRBYLBHPshotVvd/yG1bNufXUary9oVcyGyimx9jajm2y99uDoCPS9gIxq+ZO6Fp5PS/E9mj68/7yHxRiwSN4Ki549p4Yx6CJ2f50tCj+L10J0pdP+CFx3u1a/BGYtzwRufL19w+szQieOPM7HY3VONIYNMhI7rsmZE0TlW/qB4Txwui10Qme0nyZCKRCqfxJiDipzjvjf1v/SxcrwrzINQhJbbiZh9RI3CHhlwGiiO1Jh46RkrjuRzDOvEdNcuVmj5gv1vf5x5CvLgcgSNOT2E4P0CCAnKq6FRqUfiHjNhgbmY+q5o5KCsSB3OjskXW5gXtJ34BvPLhCjyglc+JNA/l1Pmk+h2Hr+P8Bo9sDA3RnC/FAEkdOTLkxT2URcG6RQvy7m/TWqbCyMoA+ffwZ+kqefynDM4bocImnBodbq4GVUHSd+QoIMuNrZaSaixTcgXz2cNFGOXKj9W3mhg4uTWMK9J64tn9vMm/Fh5LzfN9Yc0JeGgwGAKbgwqIm02F0tRJe7KS0KX2AvsxvPru6bRvx19428JInV2FJfW3XCxafBC+vwTHCideAGNVFtIxHOp28DFUk3dYMBS9G2qQuukHky X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: uthDUXcYcKy9a4/pjadTNu0SJDlw5hwjCEqU2aeYDXnJ18EGj24D7kG7vF/1sOnUreEktQRy+iZciDNp5Jfxn6uOLpPmcMk1lPa1p8NfCr9KfdqwGS1KGAYf83QrRinj5l8At4rNaOXFQwat0bi3q+XKYPgupt77jPIbCBS/e+cMyZnqZullnbISk7JsDyLat2+OK5U2KgxS/J7/+V1fv848HOa6qzuzKerRI57e36rT4SDJAQhbfYRB6la4xAQZ0i+5/a83oFe3eBbtbn/oVC2+xChyji/TFngTm940N05Jh/HjGHWrbJeisbygzmnwbk4Pt8ynDdasmIjAXVzBV+Rxm2qbqirR5xO/Z3zlcvYoyvtb7zTP1TPoV0XaPiqbmJwAgcqQcMbrZXQTJGcdP7Vo2YzD5kdj6A1jk5MLrgzNT+7NAa8ZEulnEd+0xxzqEuLM1SN7UXUBkAhMP/b4Yaujvy16PvlW0CyI5zXk106HfeQxFjFrr7ePlI4e7otv/dqTgITnJa/QoIn9yG2FKmVI+kauoq0CriZ3w60jJ5ckh/fSgsbFFWFqA4B5bnb5sd80B3OO+YzG+5uqc9q0tZ1HPUtxa4oD6SeyH+Yp+B2TKt2ONDErS85q1V382GaHiboMLXkelp70/GB0XDpWn4z8ZuEOkHEjf2YSMyjoPJUgKRMpMYchHnLeJWNm1p24nu9/SSq33hX345+C1YTQA+N24xsHcYjrceGX4K2FZLF7uCDM1S/RaP/s3hA3odVjcEH/GNKTQFUxe6x7cTDEkYXfbN30u0QlEzp+eu2ZvoebHcfsltzKtPC44Ew1K0CaYrqybZ4Ffuz0Aas4yaiSPG9mLHwNpVowYJXR9NEIxiOIQaFVFK5CqeAy1xpRU80AgCeypbwFIDr31Y7MMHm/fmTZ3j4XjeoJGXvg4eNx029m3HbF1U3sCxGQQETgEh9FyOMDPBn3oTaO7r8Dt2Hw2rTPe14WNdQYiD0i0hNomwrxY3iNllRBxy82XvGKf4wrYkkYfyoP9Pp1MbGrUTpivWK78tJAuZwdIGP9mORo5DGWr6xikVgDxXDUzmp9hw41dHLxS3ntlsl6+asfWbwXVayDWttkGvG9og+kT0/RR812PyCECSIxrvg/3WnUJhu4S1lDItna22hGKNm9YdeiK5hIKGhEr1Goi0wCzgVRj2zQNSgufqy4fxxttbqiD1YyjZhMhRSqrHVKuU7e/Fgrc2KiJIGfIS/KYQTHtFyvc9nLRLeZ6IPh11as5fJv3QmJOmJIkG/JUQFwWKAZK0lzWJ0zHgezKw/O0wybpjy76RJ+vKteTGK/7pyyjN6eLQEOGKWGFwfoDVTT9N7u4LJgty/V0VJrZvjpHSqFihooPmCMSysnXtUwyzwJ8TcJxkOh2C+xD2X3lh3sVULOje/QJ2MqFF4AcXATVEcWHexbeTa/M9GYh59U4cMSJqEh6WWfHjBDHeb3lMOdzgFXMU/Zjo6BMwLrUo8O4MAzJmG8IyPKZx9FSQbtCZBRJtSLH6NXzmrA3Bhf3y1gs/ZdyB/XHTrvkK54gkt6cr3zBKkXF6vaez+ShilgKNH1Q34UkvRa X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2882b00a-b830-4a76-5fe6-08dd4d2fa2d8 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:41:54.8288 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qeJaAy5wtX2VliysgkpgOSNg5yU1C55kEiC6o8OXawol4VpzfXRqA8+ZYa1wIFHUPxyriBCl8VQlharV3X4p6Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7809 From: Yury Norov Nodemasks now pull linux/numa.h for MAX_NUMNODES and NUMA_NO_NODE macros. This series makes numa.h depending on nodemasks, so we hit a circular dependency. Nodemasks library is highly employed by NUMA code, and it would be logical to resolve the circular dependency by making NUMA headers dependent nodemask.h. Signed-off-by: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi --- include/linux/nodemask.h | 1 - include/linux/nodemask_types.h | 11 ++++++++++- include/linux/numa.h | 10 +--------- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h index 41cf43c4e70f2..f0ac0633366b9 100644 --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -94,7 +94,6 @@ #include #include #include -#include #include extern nodemask_t _unused_nodemask_arg_; diff --git a/include/linux/nodemask_types.h b/include/linux/nodemask_types.h index 6b28d97ea6ed0..f850a48742f1f 100644 --- a/include/linux/nodemask_types.h +++ b/include/linux/nodemask_types.h @@ -3,7 +3,16 @@ #define __LINUX_NODEMASK_TYPES_H #include -#include + +#ifdef CONFIG_NODES_SHIFT +#define NODES_SHIFT CONFIG_NODES_SHIFT +#else +#define NODES_SHIFT 0 +#endif + +#define MAX_NUMNODES (1 << NODES_SHIFT) + +#define NUMA_NO_NODE (-1) typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; diff --git a/include/linux/numa.h b/include/linux/numa.h index 3567e40329ebc..31d8bf8a951a7 100644 --- a/include/linux/numa.h +++ b/include/linux/numa.h @@ -3,16 +3,8 @@ #define _LINUX_NUMA_H #include #include +#include -#ifdef CONFIG_NODES_SHIFT -#define NODES_SHIFT CONFIG_NODES_SHIFT -#else -#define NODES_SHIFT 0 -#endif - -#define MAX_NUMNODES (1 << NODES_SHIFT) - -#define NUMA_NO_NODE (-1) #define NUMA_NO_MEMBLK (-1) static inline bool numa_valid_node(int nid) From patchwork Fri Feb 14 19:40:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975564 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2075.outbound.protection.outlook.com [40.107.237.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE83D1C8637; Fri, 14 Feb 2025 19:42:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.237.75 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562123; cv=fail; b=Gyib4WG/t3EvFKlNkBLuToVy9etfRoFpFs3kuw+BAwwzgn/XZ3HlfnneyhhzLEvxlfY4VIKOeZwyb5PWi4zS5xIicoIqgUWzW+ohWsABUibQFMFSuGLRTaNdEqQ1hu2tewGYOcYdG9UxIraTj4sdUZvYx3yMEC152B/NG+MaDY4= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562123; c=relaxed/simple; bh=51OlaCzwGlOAaE1HyiIlI9UKSp9Sqo6BoXksah3f3Rg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=dsuScdiZdoUj6DjuUDXAuIGjHzhaehPqo3kM0bV8cRRMpwk9AKAC+z4BsqRb8XdCyqPaP83Rss/PwHVvh9UkgKWCKcbkTA9fqIZSheuGV4MxKQIqq246fCSmQ5A4vjB5yaIuPKH+ebyZIWjT5xyHKEGJeFsgJLC0CxH5lcB6LR0= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=QBzUUNF9; arc=fail smtp.client-ip=40.107.237.75 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="QBzUUNF9" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=AD1Q2vzfuirizcz/ci7CGFbn9ynljxrs2gtw1zckjB03oPRhu4iOS7raOIjS6rRpfy128WvaKNywV0yx1c7RYlX70YL+MOglPJN9HdPKK+rrn3ObrQ07vuaxwua/sAHu+p6uGGeMH3dUNklsx9YHAuentcuT7pVT7ggFr/zjKX/R1y2hzqlZb/C6QDiBHyeK3Nsjj8eKxgKFniWMH0f6Gp5uu/NZUu7zTtXCncDclPBosStoqwRzkmnUF4PibHRawFtXfxszrqFlysDxkW/be1A0zRhETI11izIJLPT0oKZ8dCGxxupp6NQMLt3D2fUER146xL58ayb5ZxMc3DbyWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bd6VQNRR5xJaWH9XJbyL6g1uRW8Ufqd4Dy2fAhdtZU4=; b=GzBGYgW0Ioip/e3cXEX+FQSvyM7t+2gTvda6p9WAz7XXohg2a0CR+8F89TQvdjSbM1jRyakqK+CbMj0xjKO0snIdErVTxCOOOsdNMjYt1v9u3OPkfwXt7I/ROuzcAxuUVkGNLJBYbF6mRE82EBwpJCRbWNpPlEYsd9UG19omnaYbma1RYNGgpFtVN9PKF26wIXmvYMdvOCkTQRR9jpqvsfiF9GYlW39csMD7w48OsNV3PCHU9emRKymntyhZX3xU1zJos24zJUWTB0CIF0xSw+Ah/avdcv79lvPWQVclgauZ/2NIgx+oap09uaDgAwyCDT3Qn+kibqusOPv6OCpLVA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bd6VQNRR5xJaWH9XJbyL6g1uRW8Ufqd4Dy2fAhdtZU4=; b=QBzUUNF9WpLoqR0V7dCSDgPgKt4hYCM37eEeEKNicV8QsGsHn1j6MwjzqVkEtsUkcXn+eVR4/5aC+Cjr4Hk1SHaVYwm93xmgNeWV8MMRotRDPQHf01Uz4pZQfKB+LjNStFra2e4uwLN/Wq+UYrFtsSHzcTM6j1jGuz34PQsm9GFHjWt3bZlxj/0r+Z1jMmRErCJQUxn2ONa5PR7PzuGKE6ZxPKbz4f9GilpVrARtsTBl8a+cpyeNN5jbHSKt72Tat4w1ZZizcyb2Ua/XhFvWKjIlhs7rIP4fASTbDoS92tGxy5DdcCBGLD6034GvWrQQYKY2zfOLE8KT4a4OvCVjBg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by SN7PR12MB7809.namprd12.prod.outlook.com (2603:10b6:806:34e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.16; Fri, 14 Feb 2025 19:41:59 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:41:58 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/8] mm/numa: Introduce nearest_node_nodemask() Date: Fri, 14 Feb 2025 20:40:02 +0100 Message-ID: <20250214194134.658939-4-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0287.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:e7::12) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|SN7PR12MB7809:EE_ X-MS-Office365-Filtering-Correlation-Id: 90761c16-f9da-489a-6590-08dd4d2fa547 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: 8A1hSmfXh8VhH2wcvAE7Qag8iDkmY2co3gziqpjt58HNWP4s6v8Vjz179+t5DKAgCr6ZvlbcUDD5l4JowzKOcOuKP2CAdooILi6VxukIbFm6DxZ3tMlyON150WOs0iBWTwo6mC7NezPOhMwBDrW9CZPnFz4+sblqZzSbp9RhBZTeXBEmpIfpZ6ZmaarPwmhRt8/7iSAis5eUsgsuajHtw3vXqGVoOv7TG8768OfP2B1AWFWmD8xQfE3/BDD+4yrB28B+tbyOnxuii6VbFJO5CSRuPg93ij1eFlQlG/oSoHONbaoIlWE2bVcHOzIP6ZNBIoJX2zXtUjKjlHt2DALiyQjQxYnalGP3Dp7pNVRQpoi9LC6oLbeWwiauVvmCe7qkjMOxnujyknuxw9I1lGVgJLi13xabPKENXn2iAy/LiPLIAS4R98uVzZPT1aSpEGbkIkAXLwRbM6TgktS4DUO8wfEEKOVTlTAm7hvFlY/bhazg7ffkPcO2zvVd25czTaS+8ZuYDellS2U2Tu9GxYvW19JMmmwxsMIH5rK7YFnTQnUYCDrZp75FwGT1nPJ5eDy6uWa6EHkkbnPn8q5utAACtknr/2cceTzkN62+dpz0cXHLxJSNzLy3GtGAQRQmMfp4ravxsMAwT/jdPXHmfGii0YDACJppsLYkqrtI9Ip/REGgrCoU9VfWq6AgmKnihZT6NzXED/XKhY3ATRrZGbQJX2pRdAmgw4m+U85cy6S/41E7SerPEzNQaPHhlb/Ta2CM4ZaCXtNgzsHpoZEbWdIVMtBc4S/lGDd0RCH7qYsznNKYCWM06jl5nU4ITtU7/8aH14JQBjMmSW0DcmR9dtnYXtbY6qBhgI1+h8FA2XpEhFSRkY+6zo0gknfA4ksr8kMEC1f4eH1UI5MM7BLFA+tBeIqBchl9zg7zf59qivq3j3hWix5JwhsIVI+IN+TuIJDfXVaxkzB2+tLg/YWPDOBPiE2530h+KzMTkNOVGbuv+T0ce/To+RMTg2zufroR3BfmjQlZ6Gh2Z7ocl+cnclBEbWoDvf0i6wRIPgVa+w+S9tUbqWs3fIvdlPDUfZrcPNNF/WKPBeoQLcMg2Tm3BapGeyxarRox5OVSJx7Um8GkXPueQOFJQfzPNGuAdE/8K/uwNH7LDEkV0qp6O/FHRQgyfpK61IXetr5z8p78V+FH0beubMTXeP5cdFdnqwupQ0YJDIQ918fqwbldMMzmq6QF0UDGJvPxriirW800SQRzd9dNB94SEJZznlDCAaYMafS5iZ/d/2/LQ00my5v7atgep3w0ocHVuC8us1vQIUkJH/zQtfZV/MabzVJpATPqlx9pp3S6oblUbEdbpPG7Hro35gtWJIv+6I7q7TgYMHipgtNXahg5QYQD+23Cnq041bWA X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JzegQ26JQ0qtg0ECdbRpovbVH4Hbl+rWJxGcKE8EOlxCElbOGb07bkfXl4v9kSKP3z0+t6ecA4Z9Ek4LgVcWALqFM78JrLRjOKuFC4aWG2Rj2iKLSPUgvEkZ69fNnyQc1vTyvhCEMf1sLYO/8vICn0sLi17x9x1fYwiwWjmA/xAc9Bxvmkn+ARSh5ml7iJdvWWiKZlmcktIVhXxCtNH3W5jIkRXJeyivIZliGcGWokLRGQ6duXrhtuwpPeMiYghfnngzKYROh2SKJx+nEnrAD/6PDyYtWU02eRLcmfU8P6wXQ5cajKq4uMGdIufVmxEv3Z9PkVtKMLkFb7n8qacPzkFz1sx4bnO7i4AE6z+jT1QMVIpJC5KsW/8W+MuE1VdsHrL1ygzZ9/oX4e9jNqD6OcagoKijrcHW98HK74+8VJn8fKkDZL9xrre0fKF9olDggcSzxRH9XaTNxUyC1ZYGLJ2icxWup4h3h76SBN1zlKxGLoo0GZdbwmvxOyeKWCFWBT+mgs+EVQ9eaRrkbtX/TN+SyQuRaGk7rUPFjLa17NcrKJz8G51zDQRTS+PQNsYkTXGzlawB/vJ3E9MXH9P79OjtaVYH7Hj8qfrx+DEAHUTS1NvLiR0a6f6IFJPrcCGEWRNTjrIP0ZUBXzBml90gGy90RfZsIX8IwDtng5CKutGqdZpsu8kod1n0s6djU0JtfidT+LZoVZTKu214yGZfjihGtFVvT5nx9PK7xwnnVK7ByxYkC3oz02tU/gc6fWEJ6yX9ZXW+9p0vuKHtp0xM48DUebwZGMfI1b8oisnYvn9fQ6wm1l0FIhTejNl8aUXO7zdMijZqBapeoMpRaGQ+CH4wWAFpI//DCr927uhBE26/OVeQwaaY8Qfw1mSYUopP4Vcdmpp2QQESmXqOfjCzoEFhOCQKXraU4yC+jmnoxgRBZQwWT9toDo/Jzexd22R4DROJJP5IFueSbUeRCDbu8NXjbZ2XlV9wxiGcSbf+Dju7QWPf+5QayiaK3HlQnV6deQy0+F9ZeCD2XBmegZG6ul6t2bZMdN3G/0rWrONHGVO5WBaB3tRFQNtq4c+zN6qovbuaee+emNpGlS6TdfP2nTCwtg60sMHIPAn6nLVlU4+h+xWbRu+bnfH2CfV/eRi2ei9Z/HRQ+tYHcPK/KOclKxqXO62t//COByOGbvBD3EScoeMzFVUxqeBoRniecGmOL2RTG0CnXdwYvN0TVg4YmxDyuQLRi4Z4avO8z2JluHjKkeJRjq8qhVo4fpiNQlerTHjoAuTCkRFgZWIHj6jQUDGaFbgpk8Z93wpW21LiJ9r+yiJbXwv6iV2HIh5L6hGcghv34rDmEdVu0OimdKyETiWiG/E49Tv4/nBNGWjPqt6oB6cnre7E0uuHVzX6SBoS/B5naPj1wnhAHEqCKmzk3TBchnsT+wjV7ZHE6H5zo1gz8wQBrnyx0tq2BgDeU82GtGGuOz+4/9/KMlo2n3A9C3X8ol4iLYXXfrSpM6fDyPpHsMEtq8qMKLskxTsSqibP3zvINJ9ZxMKkyoYGpFdWOjKq0je3m0tum9c5LnucmRGIt+5Cd2U+5NKg85loi0y/ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 90761c16-f9da-489a-6590-08dd4d2fa547 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:41:58.9054 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5AKAhyMejZA/s3k221JgIJrVcWSATHIjTqspLQgNNLweG4Snxwfbw+G7LLR45yfgI70wMzlp06ih+j8/IgOtfQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7809 Introduce the new helper nearest_node_nodemask() to find the closest node in a specified nodemask from a given starting node. Returns MAX_NUMNODES if no node is found. Suggested-by: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi Acked-by: Yury Norov [NVIDIA] --- include/linux/numa.h | 7 +++++++ mm/mempolicy.c | 31 +++++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/include/linux/numa.h b/include/linux/numa.h index 31d8bf8a951a7..e6baaf6051bcf 100644 --- a/include/linux/numa.h +++ b/include/linux/numa.h @@ -31,6 +31,8 @@ void __init alloc_offline_node_data(int nid); /* Generic implementation available */ int numa_nearest_node(int node, unsigned int state); +int nearest_node_nodemask(int node, nodemask_t *mask); + #ifndef memory_add_physaddr_to_nid int memory_add_physaddr_to_nid(u64 start); #endif @@ -47,6 +49,11 @@ static inline int numa_nearest_node(int node, unsigned int state) return NUMA_NO_NODE; } +static inline int nearest_node_nodemask(int node, nodemask_t *mask) +{ + return NUMA_NO_NODE; +} + static inline int memory_add_physaddr_to_nid(u64 start) { return 0; diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 162407fbf2bc7..488cad280efb3 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -196,6 +196,37 @@ int numa_nearest_node(int node, unsigned int state) } EXPORT_SYMBOL_GPL(numa_nearest_node); +/** + * nearest_node_nodemask - Find the node in @mask at the nearest distance + * from @node. + * + * @node: a valid node ID to start the search from. + * @mask: a pointer to a nodemask representing the allowed nodes. + * + * This function iterates over all nodes in @mask and calculates the + * distance from the starting @node, then it returns the node ID that is + * the closest to @node, or MAX_NUMNODES if no node is found. + * + * Note that @node must be a valid node ID usable with node_distance(), + * providing an invalid node ID (e.g., NUMA_NO_NODE) may result in crashes + * or unexpected behavior. + */ +int nearest_node_nodemask(int node, nodemask_t *mask) +{ + int dist, n, min_dist = INT_MAX, min_node = MAX_NUMNODES; + + for_each_node_mask(n, *mask) { + dist = node_distance(node, n); + if (dist < min_dist) { + min_dist = dist; + min_node = n; + } + } + + return min_node; +} +EXPORT_SYMBOL_GPL(nearest_node_nodemask); + struct mempolicy *get_task_policy(struct task_struct *p) { struct mempolicy *pol = p->mempolicy; From patchwork Fri Feb 14 19:40:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975565 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2067.outbound.protection.outlook.com [40.107.93.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E19E51DA31F; Fri, 14 Feb 2025 19:42:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.67 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562128; cv=fail; b=kFcHaTHr4M+o8ljRLr3Il7IBt5VqdRmKI2L7zhEdvQsfFWNpBT5+bLCL9ra96kFFiQVf69C8A8NW2i62QZTjFaLBTXyWEDC9NjY01mxEy69hFleAfPcwSvthBrBbS3ZwZ6MV06VB7LFQHZYtYwUvcQ8zpuDuMQ9e+hNeFXJVM6A= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562128; c=relaxed/simple; bh=pVAiNBUT6jVisQKGQ7Va71KZvIlnoTy2TLlx0HM4OSc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=L7sl+UUgooxgTvpmOKbVTOs3WVZpqZOPCWbqntLPCugVnk6FgSvyFibl2FOtRkb/w37zX2SguRwqgbFEsMP2FQbJhGPNAlx9rMFALC7MfSVFGuFWnbcPQQclzh2hjFxFpP5tL76V2XvU3/G3ww1QLNe87RBiQrO2IhkJfjhK+d8= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=fkJc0Ru4; arc=fail smtp.client-ip=40.107.93.67 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="fkJc0Ru4" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kgn25fYO+gBevTF5pxKKuU7/L6vW1zql0PUiPemGvoM/2fl8VMSeAikFD2CKr9tpN8PSR/iiehrW7+RFsAINCjJly7jPo7Mf7tM9z65VEEgcharBooV5mCWWLvcTKklp3onxu6/V0v3UYuhuJKCTrGaTvKkDVjN1QoFFGqULu9VVvOHOuR/sPgWxc5qp3GlQqjEXtggCREzmX4qFAWuhKHm/hROpWwKwXpixZi3Pe7PIwvyrGxMOC7gBPSW9ocoPpdP1gWY286LAFTdFAc0JwmFUNlkk7WF6ZCacDTt4ej9OFlpRZ4vrmfT8xSDkbudoOmNIjX4q5mevOwOqhLQA2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bHIKu4nBCiUgwyvw8fABVAjzYFlRa1GSTcgtMxEW9K0=; b=yIY+QTqYk4eDFb1aTJce0kFKwdnEKh/B7oz2kETiwBWCcbRWmXV1Rbx9yBGCBVsDYdQkXY9vpTKKmqlIYsUm4XERf+GTWDmLHGTAjUEWf32bTQLP/8gkVaKq7qNRNvKgD3+tjbVRolzIwcnVLeTMW5HHU66QL6Oooss2BxwI1lLuTDIhxHpKUah2neofxRV+BHY5XfW//nVV11SZUUKSrN+Z3jwh1a8/AENxOlWX0EbPHVca0ncibJo3ZB76Kb7hUpv/wW2giQ5oJ+bgDa5o3MnvyuPLZvnS3IDpDRkH2P5uOZXgHjG/uHjjxj0VSULBARHjvTiFNIBq5NHhcv6EMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bHIKu4nBCiUgwyvw8fABVAjzYFlRa1GSTcgtMxEW9K0=; b=fkJc0Ru4yE+/mfS/ZGjh5bQ2YgOzF5kV2M5yfQkcZ2UH2bXr8tQa9XDhxA6Jzu3JL4doD57RzPVelbMBJvTzytqXK7fhgvIPU9qb++45yaSHGPDCCg7vaw5Tw3TQe+HQ6VagRGq3mgl2iLa9ijubnMZeREHKJeEeIhkjjlZ03FWUmWoQYE3lCwPz7EMlWylG37M3QLoGPvz3baWlb60neXF2u2c9Lg41qHt6dSeZUghN68YlkIrWgHsKSMaTiOxX+n6Afe/dtapGOyQjaq4AXTUTxeQUJObLrYo/1VhFtM9cRoTOVoGkMnN6Advu+KnqJLGCQf7Mi2TZyIMWpnwo3A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MN2PR12MB4360.namprd12.prod.outlook.com (2603:10b6:208:266::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Fri, 14 Feb 2025 19:42:03 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:42:03 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/8] sched/topology: Introduce for_each_node_numadist() iterator Date: Fri, 14 Feb 2025 20:40:03 +0100 Message-ID: <20250214194134.658939-5-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0203.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:e5::6) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MN2PR12MB4360:EE_ X-MS-Office365-Filtering-Correlation-Id: 6c72032e-eee2-449d-1598-08dd4d2fa7d8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: USYEpbzV9TypuRHFDA2cvoBT5GX6AzjQxMhdBBlZby0/Cw3unrtkOg5KapuzB1Qtg6Pj97VSULVgcvoneeRKBGpTKHKVFSDzywkgJ38x4O4UP0jTkeJMA2VGY4YJfK2UxRUORfzLytSKqpmbGnp/QQocA0+PY6cF9eVf1UIZfxbbH8i9qI7nxbCWK1OuqXHmn3uGU8FBn8Cp4OUqA4oHfvdyTHiomzGl5AoWWHGDDvRNY6E79LyT5QEQTCumw+S361w+RH+cIOJlz7/SsgDKK/OfVeMou7CWrcu5WQidxfooSRljwT+TY5kev3IvTgkeNJDXWn9wNHtaXtzH0uw67mkzUAyBwKfnwsxdsGlMLHczBr6G/TVbQOyKjQZG7FX41GSwR/PoId+F4j/i0ba7u76bwRKAUXX0kzcbqQw/gjNoW03JanYKWY7ImCvKZ0Iy3D2qTBw14S8fDHF9EpOt/zpr4p5/x44s9Me6gO5af/nXHmcKZRJLhB4sQzPoRF0nmykCawGHkDQcglI9uE+WmVNf9bzzHCsSSrv+7tIuzaXi2uT1061oryslUZMEzs1GXuIwJuSLX6/ZtoqKs0eo/DrxmVxCIbKS83mIpN6M/oVVQ3dfd+YOlKJ8xcKs5Vh/S1vu61Q6JdGtNc9Gezdu3/aZD+ldkZsYDhzPFgixxPwWrFunreTs842qDGRGuFK9CAY5hrMNSQPr/g9L6f7gvuR/imutWfef1Sqe5ziG8bMgmknNC40gQutUyKJyzKsB2JOl7xxsFBUjzUFfZoKWWIybRePhZjEgVeGtKmpL//KANl9qJrP1TnTQlLaPLCz/JpkGTjgNysuq0eiqFA4zOM+ULs0x/D3UjGQSYS31Ao302myqi/wfYd+U/6O6OTkmGmLHljT5kRa+U7p+XZWqXuYbOgI9lvIneVoYGEhYCB7UynnOMVJYuF+uwV4EiiRg8EC0onTN6O4DVoGKSZT8E10enU/wMqGxwTqhGup2g+z8tyUZ+yBQ99WSVyDhK+HyMyLVRXVdmMNWgXmm8sIsmeDqXvScoxKBbox4SEP77l9Nnm8BJSHpT020UZUGyiuGpnAnNedoSWJZBmfxZCKbvwa4T7r8Joc0ETo9MwguhFIbVPyKPo7TcxU4xsUWbmxL8hpUrjtVpWj+u/AuHzL4+XqpmhmWSLGUZ1tJDRUF9aqjNxZDedg5SvGkUb/PJ2AYw7jda3KBXVsni054YU2lN7qeh2l+jgprQMltF/Em8g6ZfVhfEUxQNNJf3pP6v3d5NW4cu8PjHfDVH8bnb99Febo6jmQSrm3DHOqLwn/JRORplTuaxXh9LkgAimcMzSQ7l6/JMdFDwAk91hkkaXgrGcz9eVc2It9ow9P3LuXcBWVekGEPf5cQ/KtB3gtW1MQJ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: RImc71S43xn+6AkZwNlFPvkTUB+Lai3fyf59otAVjE0LutrIkbJMdCNQm8aj6h1UaoWx/AtWEV/IuOMGOvASdvFlhex1NfmgHx3N/Ru6G0Rstwo3eY9vNvNE5t5AUG6bjO/JKsn+E1TezPW9BOztltVhzhkmAnJjvhxvfRHM+Whi2yIFgEZGOdx5q+LQse6FHFL5uG4YZz6ztcgAoQ1cPSEfrXwW8ZB4JP8CWH3K8sb/TVPQeXycdk8rTvF+eTH/oNSGe0fQiXMDF0DRJOhdeF2mnHIkRTtJdLlnWeRre/KVwkVLYVUFH4UUVmxlDnvs/MD4Pqz3zVOJmenST4LUO+O6F/yJSA4scRZHQermd+Nyrjz/uDlNJx6TMWBcIlcjAPs+gllb1bWkBU/AIPNXl8JyW1/LL2Q65T5shg5wfK1O09tThBQCswF8/Q1yy4Wutbyr/x8NnU+qjouOMA1ApbwdT1hYyZ+vXnCUqW3JxNy1vUN5VSzE98So1PCTojmzr7U0yawFvBKRKg8eklqJR2bWrrSyK79mjVNcoeLft4G2yxINytHzj/XAdLQQ5+2IqisHW3+nBXf2bQKq3qb8VJMtlP/aDdRC71SHTZP3NPStq2zsoAk6VlwJJ9TEtNO1ph/8NUffK7Iy8bc05qXl1NdhGFDarrrplDVbSejwQBoqLjlDVXHhORnU/BGZZVVDFuIbQE2dYBNDazbR8DcsJik8O9GkUBvuGdVGblcxFucREBsxIMLe1wShcFNxhRdpdw7b4qZpHwmFsDzrkgr6TEl4+c9skwt6uEarhsNDvXEwZvsA7JcHcaGOygHAd3Bobxfd40z1sPtOJYpOD7cX3qTcbIKlqU61IXMALWt/EBWv6/r2WRUAY7Y0X/pLsLTjcgGlcv1f0knI0f0F6nfWUjRpk61+/2jqF/dzyNmtRCc3MBL3behshAJRsEn4YzhBsE3tga2zPuD3V4mku5/ofqZHmQBqcHSUStacCSRtdXmFUgumPLP2ECJ/96gQl/sVaRLol2DQks/jmNhSBGewZ0AHHxllcLAWNIgiEzxg7GBcpuebPSaTHnpmSYvj4g4sv3E9u3qSju4qloyQ8MMSzJuSBsVBmUIaX5qAmtYpjgk9ABctsXHcqRTtNjUnlUHgnDnY2jQ9AE1MAmEjAtBV3LrVfQOyBMMFIAFb3b8MGtJEo3OWkrK9xLSY+f71fCBojc9QyL8p7sac6Xg6sTpsuFrfqR4f5z+LDLSjwkSXABGZOPLSKGwqacp6bW5v0CY6dfs/7zFTTXJID4zAAKCpQTLN4NG108J0XH6MTX03ZwEJtK4F3ciHyztEbOQpbaE8dOWtXew5kgtXOI4cRJcrcnOilkAD7kpNBgy50wsgUTBxqJkf9WbsxNasNtVDg5hSHOZ+FrEbs2+mmbgHpdwopIxnRrTTv3pN2KSyU/1b/Oier8fzm7cgcGKclWaHfHNYWEaAkcjfVbLYFAJVW9KJ74Tde3tWBtwRDT3kfA1LT3FTYLFWVbdKz/5gUzw5ViYmOn98LE5Fca4dCi4IrUQ31tMui6xDUizQYfzZovEOfL5BnTc6VQCeynyXl/M/+BbS X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6c72032e-eee2-449d-1598-08dd4d2fa7d8 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:42:03.2657 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Aj+r+qJJMGQ3yTR14tHJWK23rUUy5rXDDVo114GBNLGrKcujBNLz90P2JO1sbk4hB3OfgiSPhML8jTI4i7hcwA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4360 Introduce the new helper for_each_node_numadist() to iterate over node IDs in order of increasing NUMA distance from a given starting node. This iterator is somehow similar to for_each_numa_hop_mask(), but instead of providing a cpumask at each iteration, it provides a node ID. Example usage: nodemask_t unvisited = NODE_MASK_ALL; int node, start = cpu_to_node(smp_processor_id()); node = start; for_each_node_numadist(node, unvisited) pr_info("node (%d, %d) -> %d\n", start, node, node_distance(start, node)); On a system with equidistant nodes: $ numactl -H ... node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10 Output of the example above (on node 0): [ 7.367022] node (0, 0) -> 10 [ 7.367151] node (0, 1) -> 20 [ 7.367186] node (0, 2) -> 20 [ 7.367247] node (0, 3) -> 20 On a system with non-equidistant nodes (simulated using virtme-ng): $ numactl -H ... node distances: node 0 1 2 3 0: 10 51 31 41 1: 51 10 21 61 2: 31 21 10 11 3: 41 61 11 10 Output of the example above (on node 0): [ 8.953644] node (0, 0) -> 10 [ 8.953712] node (0, 2) -> 31 [ 8.953764] node (0, 3) -> 41 [ 8.953817] node (0, 1) -> 51 Suggested-by: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi Acked-by: Yury Norov [NVIDIA] --- include/linux/topology.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/include/linux/topology.h b/include/linux/topology.h index 52f5850730b3e..a1815f4395ab6 100644 --- a/include/linux/topology.h +++ b/include/linux/topology.h @@ -261,6 +261,36 @@ sched_numa_hop_mask(unsigned int node, unsigned int hops) } #endif /* CONFIG_NUMA */ +/** + * for_each_node_numadist() - iterate over nodes in increasing distance + * order, starting from a given node + * @node: the iteration variable and the starting node. + * @unvisited: a nodemask to keep track of the unvisited nodes. + * + * This macro iterates over NUMA node IDs in increasing distance from the + * starting @node and yields MAX_NUMNODES when all the nodes have been + * visited. + * + * Note that by the time the loop completes, the @unvisited nodemask will + * be fully cleared, unless the loop exits early. + * + * The difference between for_each_node() and for_each_node_numadist() is + * that the former allows to iterate over nodes in numerical order, whereas + * the latter iterates over nodes in increasing order of distance. + * + * This complexity of this iterator is O(N^2), where N represents the + * number of nodes, as each iteration involves scanning all nodes to + * find the one with the shortest distance. + * + * Requires rcu_lock to be held. + */ +#define for_each_node_numadist(node, unvisited) \ + for (int __start = (node), \ + (node) = nearest_node_nodemask((__start), &(unvisited)); \ + (node) < MAX_NUMNODES; \ + node_clear((node), (unvisited)), \ + (node) = nearest_node_nodemask((__start), &(unvisited))) + /** * for_each_numa_hop_mask - iterate over cpumasks of increasing NUMA distance * from a given node. From patchwork Fri Feb 14 19:40:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975566 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2069.outbound.protection.outlook.com [40.107.92.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B6801D8E07; Fri, 14 Feb 2025 19:42:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.92.69 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562133; cv=fail; b=Y58CDLh7N8N2F0hvoHSxCNm+XrYc4ddzkVscyhr1VKa9KkBKZXjXRJP18MCQz/DoBKbDyOtLLj8xmXrKNc/FyB3dfG48aFpd2Xom707lKQfoPCoK05OvpZmD+KOLKTDDwIOUpuBjyzNXO59sGzjQCYGr8L5S1nFSo78EDrSz7G8= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562133; c=relaxed/simple; bh=CItsehmqbdTccVtapmnGN5dnL03FZ+kLAg26xi1qK9o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Cf8PN87bCxitTIMQkcMORSCPgM8fGKpQMzDavdBOo1qBftQx+h70p2Ni8Pa0vF0rac+w4PsJh8TVr5fIwniV6yYyP6nJsuZBAx/RI0HAVAo3aNxTTO85HMl/A3graT443WsqDh40+2nuKrW/rG/VRYPaDOigMwkOO0gItkhILIo= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=QybbOVze; arc=fail smtp.client-ip=40.107.92.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="QybbOVze" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=le+/Ch97s24D3MZcCsj109OLScQqUa043gJ75qb4hpjsIJjGqU1jpmJGnlT0Z6lZFcd9aH8n2rpheuPOfVZFrXHgkJpPFZJnqqgxa8tiEwQEjIgMdHcYc+bWDZwWa4gtynU14MWpNLo33I1U6pU+AXWOe8QTH21sTzWZs+tLEVZxHjTGVmZ3E6fG3DQYPp5RhsbJ0Nbv6O5501qUdmdknZorJfFLvWIjEy5wOuI3Q/odQaVl1SbRW0INVa1DJzztykFGt/lFLlrlG2sJ32cbpkrYReMdNiIh9h/Brd7AZy/SX1JlBMvyRYp4ztMLPA9KH2aNI4ubZTjRplrv1IFIAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=EkYo7l1lX54LOJduEAk6T7PRoCqM4p2mBOaSQz+O2Fs=; b=xrdoXekAbX2So9WVGNNIXx39A1g7RkTCPQiELmJELDubkW975cj2B0vPPmXiOd9H+4ia9uhip4A2X/VTVVqA3QvLsht/IO+YGUF0h8BMjl153Y38edFIMJO8xjlRYNr3HgjPUmQW3KpSXZWBBAQVpmE8KzvfDu141Bmf6PwjJ/w5AMvcgW68RuSVQ0j78NV9c95UToOWgB8ZmvrxlW8ObtuvPYSQQ7/GG0APmEa0XbnB6RrpXLwN/vwICrpKcbkgPEOB9jHhU24Re3hWgrjehpoOxNU03eZiz0QEdgAm2T3yCkODk3qvGWpYycDYW/shEGFNaENXiMJA90Iicr9fWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EkYo7l1lX54LOJduEAk6T7PRoCqM4p2mBOaSQz+O2Fs=; b=QybbOVzeXkS+P8Ok0qDaYzfIlZzA6oQu+oXZy7yKjIeJsYrp91Q7LXvAX72jrmAdBEZ5mb4oW7O4EDsBRT1b8IOzLv+7H9g/VEQ7WRf581UTS414O1yicEce4c0SHHJp3HGNV/UkWZWof90S0+Hrs/V4jMuHyI5D5Eg9nDoEme+u8t6HM377xOyc1XQxZtw3F7M2TtShLPTUIA70w4WUQPRmnylgZB0bj1PWP4gQajhFgYr6sMH7+UyKDux6ZzN5qpExj01o+OfVZ+lVi13zIS3MWkJtlfvycAswXyhTZXrUrCU+rNWp/psKFl146HUFDulT1Pj3WFUyZbfQflEs/g== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MN2PR12MB4360.namprd12.prod.outlook.com (2603:10b6:208:266::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Fri, 14 Feb 2025 19:42:07 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:42:07 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 5/8] sched_ext: idle: Make idle static keys private Date: Fri, 14 Feb 2025 20:40:04 +0100 Message-ID: <20250214194134.658939-6-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0201.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:e5::17) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MN2PR12MB4360:EE_ X-MS-Office365-Filtering-Correlation-Id: 5dd5258a-0184-4ec7-1716-08dd4d2faa97 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: UmumLfuZYnpFuytHPVlr14sDllgvY66x9IoptGAriARla5r+hBiMTpT2vjsi6gfgB1ZXOH1Ba3cQFD2BFzvJZIliNNEclu//zxlH+Em/CkXrHGlabGNe4ylFcyAZQkatixje2fDR7e/jYVU8YlZG085EqD9WWdo783qNM5KnMfpEejCArhw+n7FDVVxA+6w5/EPry8u6+dfPJWqKYJQuJUyJ83qpbYw0EGsFavuu3bnjS4LNIROJfXlThlBflhYWwPFuiUWlyd+64FnUoH+s5aRtPmHtR4ubw6+rhKqTATyQgbDVMRhk9yqa9HZ6rYujI5fn7+s/wBhAGwl47yPFrzb566vTQLDnScPZeyTy+1WcSXg4ef4k0EIGHvUmtTciqVK3YFfl1szJz/wTket3/lEGBdK2TP1QmZq+okZvY9YjZUdUuAfvKThqWFI2MPtEvYIzjVUUWHSAdEyByT+J1kfyp9Bmp938mDwd+kpfFmoK4GhO66isi8N/Y0RT8ZcsSJi0OZAGITilxSK90cZELiSXsx73QtucCPRrC1k9o14FqIvfUkCPDSYoGJbSbo72fPTKMdt7Mixwl6wpojkbEFccO1uNhjY1uvKmvlB94W1rj7ghpOWrwqyM5NQXjhZy4z3+kdxZBxf1O2ozAC/xA4J/eT7ALnfY3qRTCA3z5pKjeLPXpxElEynosDdV91421TOjTOoZCjGS7p18xBE4nrcJJ/A9L2n6n856kcJnqjWA8cblgkaP7w1f9tYsc0hv07t2DiKiXJ7vTcO/a9vz2u0PDV6JtXeLtfYSVo3/ruCdigNbKszyYC2q2LUalcO23+OIP/yexqA9iXeolI0c6MX2bciKBhVWamEaqet5VNSoO7gLOgxheiLjZe0znXagoz+wpCpc1wCF3oqc6oTdGkd/7r/R6TOLcJYIJurZonESKUKaE1NAR26W1RgJ6hC80FhzgC0vo5SvL8oYwcl3fv9uIfQWCYCZMzGWH0S7rKCm67cVh43120iOp4vLhRH0ThUMoDYpHUr+xsZ5Gu+W7xEePgQhVnQg/c9q/p3AGZvWo3Fnsfk3ntaV6HU5NGQmhmTSCN0iegwxtUEhRWOHgSnKJZJl9jjXaQ3XiGUpS8UT1zbF/geJYnT8/X4jcy2n8IXw/d8ijayFSFz89L6kiOg0MG3ak2ZBIFWqAIm3XTA07kUrWESCXcnb+Gnownf4LxGCzIWVntZ+YraHmx6/yfpYhcw3g1Pjfvmly7aLnN1pX4YVfagOCuhWS5yscgaVcoOcCKce7Y4vUwXqRnjqlKgFkQpAfIRFzg+QwMXy/hPqYAOE640xNnBxcdXx2fv0wbbEr5tLZsBn3qmWsx+575MBUfiFAKmkdPfEVEXypDzOlWLG4Osdpkpc0zrF7pZy X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: yNuIFJusOWomKzmm8iG8p74VX23WZyaVUMJ7RAjm3WHfx8f8FVOz8E5lmkhETGKCP7ZEW53Sapr3NkI4RtAQtuk+Q7xHu5qflINN03OL33dinPSfG0bl8jmkBn6cRtljIkkhivPe17drYbO3Md7uR7nll6sVWkZa9/W4W1/qrXDZTza3oiwz4sUZhDBEqfm6726BI8n+gct3+H2TfHFCRsMwmeLNfs+dAVXAV9O3k555gzgEpXvgTmGFvCX7GYoR41DxCs+hL0RImH5OgNw/075AkX5P17F4JYF6yvziqjN5oylwoVHj4mZFKbVpJX4WZE6bc32RgIV23+0H1gE3034hr6G4QXI9aGi8pIQ+cvaAYMDYOQyd1LOjRTkjpi1OSvSsC8Pyb28hfg2eKCa+A6h+nlQAQZ6sGyYRX1RH+E2Svbufg+Xe8W8t4N6i1D/Tk1iCRUoHDLBx2re6R4SQ/ieMFS0wzHMokKWOppUfAdvKNgtxQrjuP7KKH5FGp/SVyQ3AQ9vVHQ/R5RxgVZqjjwWJ9x39pM0XdKkvfMQ8eEEWgokjPn7uz2amkSMirscqpZRkY7IJxPz+idrb2Ygq0MIw/av7jlsDPsnB1ZbqkpzY4Wd7mrH2bCieZgehLNRsgdDZRaRyC5yeqBSJq5qrhz/AqOnfgTTbSw4IFOoYk5fdB1d/L+5bOTTE6HqUIIfqe5HAZmQ7TWkK+JJzev/na+SF0s0ExqV9isH289YA78MH5JH5zF7Jo7WKti3y1K8iBzmbBk5FtdrITgROmhBVVN4oJ6EEM/NeXf5J4RyUnmk5yMNxliL082QS6l16dFsPy1OwYyHgBSStARTTQqJJnBueCXoWd+xNmEf+93CnsKBsQYdcEC1yyCF196pibjS45d8G63BnicFVFHADETnET0eA8tDNAZhjOpfI+iN2EeuGNsgpBgT8inmhXbPTwtV8me41N3X4n6M4235DjMNFy61tevhQHhARjOD9aPMAsDib6m5vLKUpuzQQ9fORfYCT/2Vbj3kaWfWwx10Zkd5DtIKi0aNt0BCfClAbfVr56hPl/rzzUy7jj35CcW+WX/05Qio6XXs6NpWwkTIvTeBteL6RMpD4OxqC0tcTDuMs1sMX0zNDjUgzDtIBiTmz1t2+P8ZBXTZFnyLi7JO7AjhmwjHCsuxJOD6awD9Y1WHRZB+gfyykrdBiWgxEHyMuNer8eEuCygRk9OJMqlcysUkMUbcfzvXHUBYXTc5wQMX170z2r05dpoOif88W/4c/yU/pvVP4enZp+7fAYSrjWtAOfGHOaGqjQMgiUqEzDsX857l6iLkr+2WaTXjpsIfjI0nq72xO0u9gIISZkH7i1jfdxH/j0hr8hSMWIQQEqITX0UYkmIk6760H8rpLMI7b3cKbHW10gZZKM45S4JM0tXs2PNe5vLuf5BcfHyd2f8om5YNdGbmwYhtOV/WK9S6mmScVaWZIKv/EN3f6dTKvNE0rNWq4LIjWKgRX7QlCpN1v0At4w3ASAnmtne0h5wATaT6wvhfGsFmnWASVLdzx+nUScfGHW00y94bwiGrrRF/nLIeF8L9TRh2ybhlYkFHiOvxI X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5dd5258a-0184-4ec7-1716-08dd4d2faa97 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:42:07.8156 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: T1AZuWVTVFnooCxw7AjZnbw23tlefjCcwjAhzd7UmJGZ9YsWH8jLgI6vW/5WCQbHByND7BHweqn0jyDAbMMuhg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4360 Make all the static keys used by the idle CPU selection policy private to ext_idle.c. This avoids unnecessary exposure in headers and improves code encapsulation. Cc: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi --- kernel/sched/ext.c | 9 ++------- kernel/sched/ext_idle.c | 39 ++++++++++++++++++++++++++------------- kernel/sched/ext_idle.h | 12 ++++-------- 3 files changed, 32 insertions(+), 28 deletions(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index aec098efd6fbc..7c17e05ed15b1 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -4765,7 +4765,7 @@ static void scx_ops_disable_workfn(struct kthread_work *work) static_branch_disable(&scx_ops_enq_exiting); static_branch_disable(&scx_ops_enq_migration_disabled); static_branch_disable(&scx_ops_cpu_preempt); - static_branch_disable(&scx_builtin_idle_enabled); + scx_idle_disable(); synchronize_rcu(); if (ei->kind >= SCX_EXIT_ERROR) { @@ -5403,12 +5403,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link) if (scx_ops.cpu_acquire || scx_ops.cpu_release) static_branch_enable(&scx_ops_cpu_preempt); - if (!ops->update_idle || (ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) { - scx_idle_reset_masks(); - static_branch_enable(&scx_builtin_idle_enabled); - } else { - static_branch_disable(&scx_builtin_idle_enabled); - } + scx_idle_enable(ops); /* * Lock out forks, cgroup on/offlining and moves before opening the diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index cb981956005b4..ed1804506585b 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -12,7 +12,7 @@ #include "ext_idle.h" /* Enable/disable built-in idle CPU selection policy */ -DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); +static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); #ifdef CONFIG_SMP #ifdef CONFIG_CPUMASK_OFFSTACK @@ -22,10 +22,10 @@ DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); #endif /* Enable/disable LLC aware optimizations */ -DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); +static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); /* Enable/disable NUMA aware optimizations */ -DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); +static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); static struct { cpumask_var_t cpu; @@ -441,16 +441,6 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool return cpu; } -void scx_idle_reset_masks(void) -{ - /* - * Consider all online cpus idle. Should converge to the actual state - * quickly. - */ - cpumask_copy(idle_masks.cpu, cpu_online_mask); - cpumask_copy(idle_masks.smt, cpu_online_mask); -} - void scx_idle_init_masks(void) { BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL)); @@ -532,6 +522,29 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify) } #endif /* CONFIG_SMP */ +void scx_idle_enable(struct sched_ext_ops *ops) +{ + if (ops->update_idle && !(ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) { + static_branch_disable(&scx_builtin_idle_enabled); + return; + } + static_branch_enable(&scx_builtin_idle_enabled); + +#ifdef CONFIG_SMP + /* + * Consider all online cpus idle. Should converge to the actual state + * quickly. + */ + cpumask_copy(idle_masks.cpu, cpu_online_mask); + cpumask_copy(idle_masks.smt, cpu_online_mask); +#endif +} + +void scx_idle_disable(void) +{ + static_branch_disable(&scx_builtin_idle_enabled); +} + /******************************************************************************** * Helpers that can be called from the BPF scheduler. */ diff --git a/kernel/sched/ext_idle.h b/kernel/sched/ext_idle.h index 7a13a74815ba7..bbac0fd9a5ddd 100644 --- a/kernel/sched/ext_idle.h +++ b/kernel/sched/ext_idle.h @@ -10,20 +10,15 @@ #ifndef _KERNEL_SCHED_EXT_IDLE_H #define _KERNEL_SCHED_EXT_IDLE_H -extern struct static_key_false scx_builtin_idle_enabled; +struct sched_ext_ops; #ifdef CONFIG_SMP -extern struct static_key_false scx_selcpu_topo_llc; -extern struct static_key_false scx_selcpu_topo_numa; - void scx_idle_update_selcpu_topology(void); -void scx_idle_reset_masks(void); void scx_idle_init_masks(void); bool scx_idle_test_and_clear_cpu(int cpu); s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags); #else /* !CONFIG_SMP */ static inline void scx_idle_update_selcpu_topology(void) {} -static inline void scx_idle_reset_masks(void) {} static inline void scx_idle_init_masks(void) {} static inline bool scx_idle_test_and_clear_cpu(int cpu) { return false; } static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) @@ -33,7 +28,8 @@ static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flag #endif /* CONFIG_SMP */ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool *found); - -extern int scx_idle_init(void); +void scx_idle_enable(struct sched_ext_ops *ops); +void scx_idle_disable(void); +int scx_idle_init(void); #endif /* _KERNEL_SCHED_EXT_IDLE_H */ From patchwork Fri Feb 14 19:40:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975567 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2054.outbound.protection.outlook.com [40.107.93.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 161961DC98B; Fri, 14 Feb 2025 19:42:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.93.54 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562136; cv=fail; b=ebHCP1/TWwUUvcD0qVZruqfWVwd9hnOamPhV3mQzNWqNX6tt8By5mwppZGDZyYQqkz1RpvgBbQ8W68aB4PeEG7YsQOZHITZFvNCv/H4bU4GWS1ZVoWzy+KuuBaAunKjDVrdJJHUruCDt1KtbZoPWbBRTqx+0Ngg9zTJUBN0VFKk= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562136; c=relaxed/simple; bh=eyCBC3PTs9vH9iWT4ZEG18MQN/1V5kGQsOT/6Wnbe+k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=pQvSLxIimtbizL0VGzByvn2KSfxBpthN9hN/NXY4uRVjEjb6ntRyWr0o9LoddW2GhDWwA/qKGGLLH1j84ki8+mZJ0T+hGaKjMSa3S8XNfOLfMJ6tuvsbhUhuj7eWLaGgXdwCiVHG5dOoGTBEmXRviVtHeJ2l5n4cznIy0UTJo4s= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=b1aPd2If; arc=fail smtp.client-ip=40.107.93.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="b1aPd2If" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=od5FwhcbcqJofOvu3llkzXU7ofUCRYewQEqXmTJTYC1bsen5m7qG5TFsEgVsX/fGQs5kQ51ceWalZjN315AR4Ev5QROFOpBsMNDgQboVTM7HYXBs5SoeTOi9xWCEyDZQPZJg3CKCrkjejbUWKcW0ygQURoLzHOOa8F7g8tfTcxRieLOdk7WKkQDO23GUPISfkP8Tm36wx2OXD18TjDbOajc8wj5N+pO+WzCc8P7ih1fajh9YQXnAnNWHDbasvyLiP0t/EoEIYxI2LOAAndV5e7+1dyKR7dxMh/19GLanXhnAYPteNBtoQRN62B4sbXzMBTa9ElhPqCA4h7OcMHhTNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+hOZsbB52FscI8RKHYq3eciI39scM1aGcO/3NoWBeFk=; b=ph1l/P0sGU/MF7rytZ4FbgIVXdTaq4GF+/dZqd0oqs1FL4Aj0Q3L7THMN0qc+d9Psx3X7yXMjLG7Xa7fYYDT8DFOLjYmk95x+3+a+QnktbgLH3K+gikuE3WWLBXzd8G6RmC6s+tUTMSPPjN40NIAz4HuaySD8tTr+ZKmVmyb5makEa3SPARyohortqVDQl7avpJfuDHAt+W+Et9phRTh0DBympdw3w+xWZW5+ul/W1uEaMYNLdJTR2rFUqyyZJ6a4xjGtiweBPM+9ag0gN8ZzgMHewqtFhEs8lBLuygFkTWvJUpRKpNk+g50AHgVjsv8TwbAqbIlhEsJxbQHL9VDWg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+hOZsbB52FscI8RKHYq3eciI39scM1aGcO/3NoWBeFk=; b=b1aPd2IfAxuPqFGbSz2EfOZpoyBznct9cQl5BN/3/PJ0ojz7u/GCCv6SXRYfcTmx7jbxFz4ECkTjVzTA74Iiom9+X0D1E3KrUW9gKTiURh3mlTC8+1RqEp6xqGxy6u+/UjkqqUvetbEQgn+pMmHbyP72ycCFrTWBS6u6XeAzXN+cZHID4zkbdop2mZZR7u85yiA1sdfxeEH/03tYffUdt/nV9yA5+8j5FHtDnM/VXuvv/J1jO8opmEnPZGA1UFZnZ7Ld/n862atPL+DDjibMMY0lLU3rw2JLF4hBloyd25XGmDkXsHZnB/2ZpFLDWFH8Bq/wHyq64Miw1CguEGDeYw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MN2PR12MB4360.namprd12.prod.outlook.com (2603:10b6:208:266::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Fri, 14 Feb 2025 19:42:12 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:42:12 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 6/8] sched_ext: idle: Introduce SCX_OPS_BUILTIN_IDLE_PER_NODE Date: Fri, 14 Feb 2025 20:40:05 +0100 Message-ID: <20250214194134.658939-7-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0137.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b8::17) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MN2PR12MB4360:EE_ X-MS-Office365-Filtering-Correlation-Id: f7ed88ec-66e2-4413-9623-08dd4d2fad44 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: UbxhyQNku2oKmsJzXFOq2OoDY0skc3k1FUx3KoBnoHEKbjpE/AuN6/V6K7+nrMyuiFlOuBE+So4SPZQL1PYXS/ENwUPMfPeIzz8eOoVhuDZUNYouWvmwV9zHt8sqv265fGLDyw0jZ4HjzPEPNyTKylRsStOwb9YkfP4IB9E7wNwf7C/Wi6SjBc7yQUCS9rTln1biNa64XggMM+vhbyYPCSQ5uyvao4cKaqQ35cyRWJVG2nFdgetoYVNuhAan2lvVaUH3NySyMBcAKWGsnH2mhMtPRwhwrhqzq/4FI8D+SDeIPPhniQrd/YI+zEfS8jhG1IHOlv1rsInmIolItk6cti5b243BBV845aEOcPyPaw/E3tIt4kLojO+c//YmSOqXifTHvFhbVM+LZZXLz5juE5QNS7N90OgoTtvz7/p2RkrT7EhKx5Bw7jmCG00PvPs2nl/jOum2jhLGS5R9mE6wlxeN5T4hOGTzE6+p7u3yczxhklc9rexC/9OhzAPRqm6bSaSpoLnDPyd/nnhcyvTTWFFzvxDQ0WuSzZ/2rGTyVVQjEG0MfL6FEHTU1W7YT/rTepyVgu+NPPt+RwKsAD55c8iw0f4JcVzJ3y0UuK3uIDV3zhUKn+2rGOh6Q0aUz9RWup/TNW33kh3OOV8IMmWfyRhSEigfVAiYVAqPzcrKGItAY/wHGifeAoXCLIdPMNQUBW5G1wae9IDb1vYO5dTMUfue+9OZOf7gI52wA4yGSPNnc75VC5l/7CrU1RTp/QOHF8Gl1tlM50id9FLINA9TbwbtBCkSlaPy7u9nnCm/EDZMaOAnwAZx9OtYu6/abJ2S4TfghBfzsLoK0DAvooyjdhapMXn073Wy1oPF0jxZ8CrJF3993eRcnGHm8XqDkyDtzB46S8jEAbVA+3/33FBBlxlogEoaCOWA1NIZ479r4NjjUNR2mSXbnxG9sb8saGgJdbXAzsyAxHiOEoyZrACQ4PWLZloh6Gj6OLCVgjrzbRRY6Pd4BfctRdY1b6FQ40Gwz6F9BY26h6XWzLKhSQVR7p7dgpA0iFOyrXexJiw81axLW5aGxW4RJNxXUluZS2tMq35Jpcj0udszxHynytW220SZH/+L/T4Hli4CYmqa6l0W3Hi/CIHglZATLfu3YtpOLNEjNwkiuyaiDseZVpVlaxwkW+YfyYJb2WYQKRlXcs6ERBT5wM2SRw+H9Vnjhr5beoQ3jyUeKOc5vN2hnXZQFMElBA8r26KgXa1p3GS24xAvorz2I0oVeZPzpMqZ5D9o84oRS61IWSGNuzQ/NukOPaqi7sYYFxRCdGN3V7h9j4ci8HYm+lE9CIKj2AkwN5R2QnMNOnEosYA9ktuyM+8jvydxvmwy1dpuNSSmV9cPvoYU5+dcYXhBuHHzU04AMNVp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /1+FC4tQnRcBQMH6xc0kPMPtPXg4VqbDlzojI1yw5IV2puur6HkJPXEz2uUKsD7C9PrMKSUFJIXg55pN5UNwCQ5R09t/x+BjxvaD745+BkMlgSK561lK4FcwyVoQMd5TZ5To9RGUsh1cwfI3rjq8TqCC4nPtLLLmEl/fWTFfPG/W3x9t75dDBr1+8tFCEFuczEhkiHh67ARfRElb7SDZCFOXwICuBjhCG7mF1R3AbFW9nPlGy8CLNUGyDWlhAnutu0HgEpskNLcdJm3BEURe6p/uW1vckFnzzX9jrS4+XUQR7oskDqssNpiEBurOeTx4bu9yhsXu3Fd5bdfHZjCQbR+63cvpq/8KZKHhfG/4lENZXJbzQNFo5HLA83FZ60UeV7AQ1MjTXQ0ObYFrWBVJtFoyltqe9TlfctWI2MM0CmEhuiWGqfcuRKoDL66Ji/XxUT6qODyiADysY6b4sJWgdPtaa/ulW7oyOwVjk3PtW5l4NLV6fZoPDzlJBh0fdbNfVtgSH+idVstNz4XWHnceSXoSPq4ZXB3qlAAo/NYaWK3G0YPPnlXeyayPV8zjmIWwxLRSg7vQ6l5kX2B0DFN+zuDpPOEmo3n1f1G4eWDT5rxOoQnuzwMEtm9nkR4AyU3gkir4Uj0ZHyjbLlAo7FMIYefit+Y2/G2OQ7iU25fczA6boGfkfN/tTZQ+KBvXC09jQYPIFa0g72R5y64RNlByJlb/XaK2E9Z8D+3fxDmMvs34ClgQ6ZgbK+vdv5cBz2A25QeFJskQeKP8ZiOqLvDy9C4zrFztkTSILbqD0/p0z2vhvxPYwXTIkeTPASXL+W2IBD3yl5iV0sMF+PezErdbkHeOPUl1jjkf8cMzE5Jv1dojhrkXlUeZ5FryJJNcRFsyYvyowbgQeTBha+AGw70xopy94iDVdQwktN7ZhnUWhC9DaypSyu8AsGrT29bUjfNU0EWeV74SutfZts+HKbseP0GKeqgYl2F06P2ZJw00iS2jOCcXvkzxPoHeTZCF+2OGCkybpjbPqhPCilALlbIZgYWWD7AqnK2NJmVcClpLnA9hc+Yilsty/ly95WN/74eYIBQHe8OxfZovW/S5Z1TviwoBZJEa72kEytIF3vaiO5nTJuR3nvGOuvfi/tXp1gFzJXVtfPGGOE9xyDbIJ8nRFbKNw9Nr95oPu8Wj7rIFW3hs/z+bSqrJVxfNSf9wVqTdJXsIjHsQJWdJRCixvV3zknBD4nmsO7VgFRSeq+ka4Z95euL60Yp/WDyltZ3BBNWDJHlwbJnN58c/8axRh3sEaJFjVWrImPX14wgqkhEouZaWnM/FOvcO4GV00YLfy2Va0CUbZ4BuzVLZSQuWObzsoTEj+fhnx6Rnp4j6+vwwHfyFWuTftLR88W+nTW8V7Qxnhxcqo1+YgnYcdlpmQ70mm9IMoBI++zRO9nnqnQZwPIKQ8QoN6JK+KWWU3O5qLLdHqp4Tk0we5jhXd1aEzfzWhmsBUqzA6bc1Y3JW6+tnoDd/0FaqLfGXExPV14cgbGXFJhd3cdSkhRMWoOo5ENifkViodGPIDtpoT1Jczs6MIA9r5/YaejufGmei2Hhlpzks X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f7ed88ec-66e2-4413-9623-08dd4d2fad44 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:42:12.3067 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bT5206bejQ5IacjOyKjajZz1Zdmugll37J59WjXhvBo3VxrILjtIDhRZhFcQ2QhH+26nvHbUs27YffcUV+2Dbw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4360 Add the new scheduler flag SCX_OPS_BUILTIN_IDLE_PER_NODE, which allows BPF schedulers to select between using a global flat idle cpumask or multiple per-node cpumasks. This only introduces the flag and the mechanism to enable/disable this feature without affecting any scheduling behavior. Cc: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi Reviewed-by: Yury Norov [NVIDIA] --- kernel/sched/ext.c | 21 ++++++++++++++++++-- kernel/sched/ext_idle.c | 29 +++++++++++++++++++++------- kernel/sched/ext_idle.h | 4 ++-- tools/sched_ext/include/scx/compat.h | 3 +++ 4 files changed, 46 insertions(+), 11 deletions(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 7c17e05ed15b1..330a359d79301 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -154,6 +154,12 @@ enum scx_ops_flags { */ SCX_OPS_ALLOW_QUEUED_WAKEUP = 1LLU << 5, + /* + * If set, enable per-node idle cpumasks. If clear, use a single global + * flat idle cpumask. + */ + SCX_OPS_BUILTIN_IDLE_PER_NODE = 1LLU << 6, + /* * CPU cgroup support flags */ @@ -165,6 +171,7 @@ enum scx_ops_flags { SCX_OPS_ENQ_MIGRATION_DISABLED | SCX_OPS_ALLOW_QUEUED_WAKEUP | SCX_OPS_SWITCH_PARTIAL | + SCX_OPS_BUILTIN_IDLE_PER_NODE | SCX_OPS_HAS_CGROUP_WEIGHT, }; @@ -3427,7 +3434,7 @@ static void handle_hotplug(struct rq *rq, bool online) atomic_long_inc(&scx_hotplug_seq); if (scx_enabled()) - scx_idle_update_selcpu_topology(); + scx_idle_update_selcpu_topology(&scx_ops); if (online && SCX_HAS_OP(cpu_online)) SCX_CALL_OP(SCX_KF_UNLOCKED, cpu_online, cpu); @@ -5228,6 +5235,16 @@ static int validate_ops(const struct sched_ext_ops *ops) return -EINVAL; } + /* + * SCX_OPS_BUILTIN_IDLE_PER_NODE requires built-in CPU idle + * selection policy to be enabled. + */ + if ((ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) && + (ops->update_idle && !(ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE))) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE requires CPU idle selection enabled"); + return -EINVAL; + } + return 0; } @@ -5352,7 +5369,7 @@ static int scx_ops_enable(struct sched_ext_ops *ops, struct bpf_link *link) static_branch_enable_cpuslocked(&scx_has_op[i]); check_hotplug_seq(ops); - scx_idle_update_selcpu_topology(); + scx_idle_update_selcpu_topology(ops); cpus_read_unlock(); diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index ed1804506585b..0912f94b95cdc 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -14,6 +14,9 @@ /* Enable/disable built-in idle CPU selection policy */ static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); +/* Enable/disable per-node idle cpumasks */ +static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_per_node); + #ifdef CONFIG_SMP #ifdef CONFIG_CPUMASK_OFFSTACK #define CL_ALIGNED_IF_ONSTACK @@ -204,7 +207,7 @@ static bool llc_numa_mismatch(void) * CPU belongs to a single LLC domain, and that each LLC domain is entirely * contained within a single NUMA node. */ -void scx_idle_update_selcpu_topology(void) +void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops) { bool enable_llc = false, enable_numa = false; unsigned int nr_cpus; @@ -237,13 +240,19 @@ void scx_idle_update_selcpu_topology(void) * If all CPUs belong to the same NUMA node and the same LLC domain, * enabling both NUMA and LLC optimizations is unnecessary, as checking * for an idle CPU in the same domain twice is redundant. + * + * If SCX_OPS_BUILTIN_IDLE_PER_NODE is enabled ignore the NUMA + * optimization, as we would naturally select idle CPUs within + * specific NUMA nodes querying the corresponding per-node cpumask. */ - nr_cpus = numa_weight(cpu); - if (nr_cpus > 0) { - if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) - enable_numa = true; - pr_debug("sched_ext: NUMA=%*pb weight=%u\n", - cpumask_pr_args(numa_span(cpu)), numa_weight(cpu)); + if (!(ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE)) { + nr_cpus = numa_weight(cpu); + if (nr_cpus > 0) { + if (nr_cpus < num_online_cpus() && llc_numa_mismatch()) + enable_numa = true; + pr_debug("sched_ext: NUMA=%*pb weight=%u\n", + cpumask_pr_args(numa_span(cpu)), nr_cpus); + } } rcu_read_unlock(); @@ -530,6 +539,11 @@ void scx_idle_enable(struct sched_ext_ops *ops) } static_branch_enable(&scx_builtin_idle_enabled); + if (ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) + static_branch_enable(&scx_builtin_idle_per_node); + else + static_branch_disable(&scx_builtin_idle_per_node); + #ifdef CONFIG_SMP /* * Consider all online cpus idle. Should converge to the actual state @@ -543,6 +557,7 @@ void scx_idle_enable(struct sched_ext_ops *ops) void scx_idle_disable(void) { static_branch_disable(&scx_builtin_idle_enabled); + static_branch_disable(&scx_builtin_idle_per_node); } /******************************************************************************** diff --git a/kernel/sched/ext_idle.h b/kernel/sched/ext_idle.h index bbac0fd9a5ddd..339b6ec9c4cb7 100644 --- a/kernel/sched/ext_idle.h +++ b/kernel/sched/ext_idle.h @@ -13,12 +13,12 @@ struct sched_ext_ops; #ifdef CONFIG_SMP -void scx_idle_update_selcpu_topology(void); +void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops); void scx_idle_init_masks(void); bool scx_idle_test_and_clear_cpu(int cpu); s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags); #else /* !CONFIG_SMP */ -static inline void scx_idle_update_selcpu_topology(void) {} +static inline void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops) {} static inline void scx_idle_init_masks(void) {} static inline bool scx_idle_test_and_clear_cpu(int cpu) { return false; } static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h index b50280e2ba2ba..d63cf40be8eee 100644 --- a/tools/sched_ext/include/scx/compat.h +++ b/tools/sched_ext/include/scx/compat.h @@ -109,6 +109,9 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field #define SCX_OPS_SWITCH_PARTIAL \ __COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_SWITCH_PARTIAL") +#define SCX_OPS_BUILTIN_IDLE_PER_NODE \ + __COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_BUILTIN_IDLE_PER_NODE") + static inline long scx_hotplug_seq(void) { int fd; From patchwork Fri Feb 14 19:40:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975568 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2072.outbound.protection.outlook.com [40.107.94.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18F841B532F; Fri, 14 Feb 2025 19:42:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562145; cv=fail; b=ex6J/z5qxylmMPkKyvCgGYJnyt5UgDt5wogQo9ZLOAtHF8TjSm+YNbVdbsskS9mosu2xHNeEKKEKk+CaCqF+m+kSbfIOHBDvRe9C37PUtzpE79j7KOxQ2rBfLdr549mqMHPfakqf6RZBhNmaH43SDUHiT8STN9A4P+5r7ti+Rzc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562145; c=relaxed/simple; bh=jD+9ppsYnmBPN+GWsYn+3+WWXwdqOTOZuKqRWYBiaoM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=sitWLmFQKUpT68xcVyZOw1GRoM+32oK4GnhCRzgC2VRAvDODfqscK5zBS6AQjQqMVel4BoWLPlJaX/Js3Owkj36/T/Iu3Ey9a0SDyA1bJfjh3L/oAkAbTp6mUk1rUVUZoTFe5GkoSZGhCyXVNTIAuPT5V9+3uWTb1GA5JVHJ49U= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=GhPpzMDs; arc=fail smtp.client-ip=40.107.94.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GhPpzMDs" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ZtrdiRBPYXkWpimef5ujFS0Fn6ae0jEAeXqEFWeigl0CArxO+l6IciI/uSudSHWaioWMqgY9LUSdssf8ADlA9UwhSqd/sJcsEz5Wx2VOWTAwRd0KIf45XQElkDyn53F/N99xT02FIPil3bQ2TsjX6cE/NcLqNxjXvvrWyF24nhboUBbzq3Iqm/InuV55V1xdoehpuwSv4ckz4kTfWkTIz2qVTHm+6OjYp0B16icZCinVVbIMZCLjTz4DuxnxAvMxQFgHVJqel3laJpC4RApx6hpUv0gPueZ1b9UXSzlk5T/lkMfYwD1paBGaxVJzdK0oVgLEkGA/4FeKpLPCkxclFQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=inNLLHgawdAoU3N+S83EPoOL12tgEPPAO6gkI1ZK0AY=; b=s1FMW/pYVeiRHJFn9V7hExgTYt8Yy5psraYNKpvVu0BrtG0B8Xf44eN/9FYKcF7JMHXMHah0pefCTp/UO7MHHdoVY/ripUtci6xSOWNGkUGYDGy4pQCbHGZgTyPfc1I/ZxrvGwTuZcNAmHqE89z5tq7aUyek6VyY+ZNSewrCZw0gSAublqD2eOX6J2LOJ6t9YTODpov0v6kvGZV+p6wG/J++Hv9QtYiJcUNh7iR74FK/8SSpylWGe9T3E7iZ4YbIk+qppls4QfcAJN47Dh8uiQwoyXlNM+AycokgOPPmBZxpslGkSXL/7oUNfG5zXuaFAsvuncGQlo/b0IBJNLTWew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=inNLLHgawdAoU3N+S83EPoOL12tgEPPAO6gkI1ZK0AY=; b=GhPpzMDs557ewBiqcczfi1B2zzoyRwYR82wlQuPa2oSeJOAWzNWoI0VpcxFQIu1iJ8WVHhmUeworEXYNNro/k4wdMr6JI0c62k/GXsYHHLHGz7EkszwN2N1U9zGsl+4b0J1HThL1Xco6zFFgqUAiAvg3nsD7FXJb8eXwc9ywjoL4sPu8dKjQsOwt1bh1pOcsXsyKygnIPpBAJSUal+D+XrCSb1ZTQnstb+j0hFd+MIODfe5jDuIR9MJuOnPjg6uIfNuR97canRGgehNe4LsJhLlXdtO/PbRxR5egCwioElm80wVVBYjJ15VHeJ8P26wlQld7ONKHK57Rfr5qR64Ofw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MN2PR12MB4360.namprd12.prod.outlook.com (2603:10b6:208:266::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Fri, 14 Feb 2025 19:42:16 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:42:16 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 7/8] sched_ext: idle: Per-node idle cpumasks Date: Fri, 14 Feb 2025 20:40:06 +0100 Message-ID: <20250214194134.658939-8-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0141.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:b8::9) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MN2PR12MB4360:EE_ X-MS-Office365-Filtering-Correlation-Id: 46f92a3b-d300-4821-3e94-08dd4d2fafdd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: SMv1QMX5MVYyNvBkbltxAuFeLJtPP5KK9/afgytKXUcm0kzVXz3Hj2yL0KUDx2sEtJREEEHjcbvXlZZwk+rQsmfaYhGjLcQt7fY3OfjZgQmCZw2zKSGJbhdftwIs28X7i/QyI7ReFboYjb+c4wix2GrEznRMaUE2/LelA9+YMPUBukVmR5PMYkrAdrwPlE+Hia8pUu8hFBiZVnPvQR9FdcveBind5u0xmylvz0Q/7HjfUu1sqVib5oGRS8hjbhn1oJfvT8eWVIrQuCXlNaanNHEy/4IMlP0hKKdGwzlRCpWWIpICFcDFSfQ1Bmw7hkpMzgADD3c2DHQWvLJVv+eDBft4eUaagXeI4AqI35xtVkR0am5PhbZzG/rH/3g5AO7wmz6K4lmKDKBcb7OKDGqhTIPSgKuxg6Xm8ky+qVBePCQCj1/4+skV7YtBxbFEnvhvvu/e9qugDlzx6y1m4ovy40xOL0hv84GZ15kkgb0fsM6SP9NlcT02LkRGwn8BPwU6l603qZd3FiNlhkAYIYjvbGkcyKjSyGGSptgQzp/GiCMjhyAr42fz2HxeIJZDzK0avZCJkZpVMlJptLZqda9sRAJblAfHzKdbjHNBaFV6l7AW4oIP/67s1c1PllLUjW0Em0CHpmKmslhxthCCk+YczWUGmZzQNqfg4PQuy/ftrFvSPibFrbI1H7nIrM6aqYlH/zSRaNujQI46SzFCEKOHQxeLhOaWUXrcA8xoPTW95f08Bmg32QPYZ6vqXRcwOMF0m8Oq3t6mxyWwQjQCtwLDGHTinUfJxenRngakcrsvAAhNXlw8pncx3RAf4rjsB6ui2RF51fBfRYhi4Wslt3zaicG2jTjFm5/gX6AqAHcaCBslZYgppxKUizzLd4nGdZyOnWf1IaDXlJoxaWO+a4qkjnIki2iOt7eOPPHd2+MLMQELE7bqxy4Yl5BsJfeg1ClsgQcgi8iTwGrXPLOsJQ0p9/wVnhamDyTag2RH07L5GVZnMpD8mnjhwtXQrOlS3u6LemW/+WhR6wHpU2Gz8Qt6j6yetrARJtxtTDViKyTixm5a/heMwn6c/cvl7+3dNy1bcUPs6oYhn6UdOf8hYvleNLRKc31wmv7YUZN1F5Hos/Npwvoefz0QgUqA1ma3BT6CR5aRGNiLFjSwSaaC9mTD0MC8snF7PaObHrwWO/1NXbtA5QKlbFcy5tRR88HFcdr5MvlzrVs5EW3n5BcNDQ3vpETQD6JVbRz3hfjQlb8VMVM7URlUUTVogbXzJ7gEacV6f2+DfloFnkfTS52PmutNz49M4ntQWz0UX4Ivx4cROlvjJk8hFfAsay3cBDMgv+LgicDiTbEWsxRv9fNeKaMxZnIUPb3OOtmHFZB8fqkmXjggI3n1Ptlmb1eRQGVWEaze X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: PeozNRI3NN5Ma3RiZCnnqzvic7FobdqyiKw4LMTN6nQt9PjA3pzy6D4ZammdszDs0rcMahg/57OPJsv4/BgCVmkO07x/glvoGm7y9eXX4ON+Xqauthh20KHIIlkrmxgMDkNop+w2+IfqKxpLFutTp0/1m/+DyYvquaDxNLy+TUgpZw0y2aZgre9hDKSRGPlr6RntFO+VY69abYxROa6BntUA5yy04/MkRJu1yQ7ikSnM3Rme2uL/5n1A6irlPW4DiMFdA9Yvipvs5bc1AuOFsa0pWPNkvfq4kcRytXs4JLH4TwSCasD+0uo+Rhw+ZiuCd86GapBBr02lA6PsUETkSzAC8UNPn4YSJ9GijceCDuiTMOQZsjh7ZLUGhDEyh24EgS4JK0enoH6IOCGOXY9gUshAPNAdsfQVGMIhzesxF4czPD1vAoxKEDrwnNoMJsMMvjGgdUaen3IEAn4TBNgRt59t8iSH4IhPzoZvGjWZC00mXsMiO+POnLqDDjK1uBfskS9itY+4M4yRSYWqv2r69muR07HGULAdios2ccy3YI/Yt7IowdkjClr1lKxUViBdYiQw/rwboLFxoSG9rThxkPhxEkwqGs5s938Pfb5xMDAX4QJnJpqgd4NGbcCYWtDp7XQd976qWl9TOP1qvf26A3EDKEx5lTGU6ntaRfBL0R6Qrs6UG3k/bSZWQZ+IkVvy1q9HNNI2XJRRDp6uFcwadszOvHoI5CK1rcRXGo5xU0uc8Q89MKJ5GHMdLaYqZz5YUcLbRITlyf5jMpYwd2vrFYJFnw7ajhU/oTZDxYb3GyaGHfa/xOzXF0IK/y8T6XuldZ4NLaYuL7GGjHuy5QN/VaZevum6USzATZAxeQyFncBKMPUZLXt4tNVxoGPh7FuDowA+uMiL3fIPdhtPFJf4axGlQ5f02gnfphC5c6aDh0iRz2s1MeAh0SwsD2IpCLDkzuX7h5Yi4Bs82Y32soMJbPFiVev70IDBmLb9HMUDFJT5z2/3NO2btwgvXm5e3CeRceuF3YwQS087l6LrfDEj5R/ifaDhUfs+5E9X5LCvPcab3pV0uTeN9HReY35LL9xkoj+10JYhJMfQde8P/An6RnlFpgj3Xin7XAewNExIET49IOL1UXPdei6CMF7fuBPRMR9RARcvXnTubKSS6wE6bOUslbx1ah18tPG70B/3C0ZtgGQ0qEh0+GbQhmU0zx3p/RA1fNf1K9JMwhFi9Hj2b5gTF+uXW3MLs+xvT2Rf67qx7MNormvk2hgiNw2ja4Mxqs60aIzXZtl/idEYeLKVrGI/lWABZXwJ+fC1E4kmXSDHGwgzAgNPKFJZva9V8Ey1lj6jYF4zm51tqoHEha2KYceUYv88F3/PvzemzgmILm98JlNvbDTiPFQcyVRioiTVqAce3Hj49ITltDgVBhfWIM/fv5oNpaUBK5gLd3mQQAnQIuUAXzlOoJMzW3wXs7r11HLQpqpEFER7KvzXPmSKpkFKXWxFG8M+k8sN6ESlP4wVstzgom9a+uhtzGvndKgtr2vvvL1ySPinTgo8CwjhlT/4ZEh7pxD9+/8fZFfKAjQG7FTsArpMLodlwez4odmz X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 46f92a3b-d300-4821-3e94-08dd4d2fafdd X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:42:16.8381 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: LwiJMeBM4b7khGdxOxoyz2/YjAZ8UD+wHEst1U2AiGZJQWl089vZoFZrSrrZcp+V81KoLyyqyu41s7KekOKgNw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4360 Using a single global idle mask can lead to inefficiencies and a lot of stress on the cache coherency protocol on large systems with multiple NUMA nodes, since all the CPUs can create a really intense read/write activity on the single global cpumask. Therefore, split the global cpumask into multiple per-NUMA node cpumasks to improve scalability and performance on large systems. The concept is that each cpumask will track only the idle CPUs within its corresponding NUMA node, treating CPUs in other NUMA nodes as busy. In this way concurrent access to the idle cpumask will be restricted within each NUMA node. The split of multiple per-node idle cpumasks can be controlled using the SCX_OPS_BUILTIN_IDLE_PER_NODE flag. By default SCX_OPS_BUILTIN_IDLE_PER_NODE is not enabled and a global host-wide idle cpumask is used, maintaining the previous behavior. NOTE: if a scheduler explicitly enables the per-node idle cpumasks (via SCX_OPS_BUILTIN_IDLE_PER_NODE), scx_bpf_get_idle_cpu/smtmask() will trigger an scx error, since there are no system-wide cpumasks. = Test = Hardware: - System: DGX B200 - CPUs: 224 SMT threads (112 physical cores) - Processor: INTEL(R) XEON(R) PLATINUM 8570 - 2 NUMA nodes Scheduler: - scx_simple [1] (so that we can focus at the built-in idle selection policy and not at the scheduling policy itself) Test: - Run a parallel kernel build `make -j $(nproc)` and measure the average elapsed time over 10 runs: avg time | stdev ---------+------ before: 52.431s | 2.895 after: 50.342s | 2.895 = Conclusion = Splitting the global cpumask into multiple per-NUMA cpumasks helped to achieve a speedup of approximately +4% with this particular architecture and test case. The same test on a DGX-1 (40 physical cores, Intel Xeon E5-2698 v4 @ 2.20GHz, 2 NUMA nodes) shows a speedup of around 1.5-3%. On smaller systems, I haven't noticed any measurable regressions or improvements with the same test (parallel kernel build) and scheduler (scx_simple). Moreover, with a modified scx_bpfland that uses the new NUMA-aware APIs I observed an additional +2-2.5% performance improvement with the same test. [1] https://github.com/sched-ext/scx/blob/main/scheds/c/scx_simple.bpf.c Cc: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi Reviewed-by: Yury Norov [NVIDIA] --- kernel/sched/ext.c | 1 + kernel/sched/ext_idle.c | 283 ++++++++++++++++++++++----- kernel/sched/ext_idle.h | 4 +- tools/sched_ext/include/scx/compat.h | 3 + 4 files changed, 236 insertions(+), 55 deletions(-) diff --git a/kernel/sched/ext.c b/kernel/sched/ext.c index 330a359d79301..95603db36f043 100644 --- a/kernel/sched/ext.c +++ b/kernel/sched/ext.c @@ -806,6 +806,7 @@ enum scx_deq_flags { enum scx_pick_idle_cpu_flags { SCX_PICK_IDLE_CORE = 1LLU << 0, /* pick a CPU whose SMT siblings are also idle */ + SCX_PICK_IDLE_IN_NODE = 1LLU << 1, /* pick a CPU in the same target NUMA node */ }; enum scx_kick_flags { diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 0912f94b95cdc..8dacccc82ed63 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -18,25 +18,61 @@ static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_enabled); static DEFINE_STATIC_KEY_FALSE(scx_builtin_idle_per_node); #ifdef CONFIG_SMP -#ifdef CONFIG_CPUMASK_OFFSTACK -#define CL_ALIGNED_IF_ONSTACK -#else -#define CL_ALIGNED_IF_ONSTACK __cacheline_aligned_in_smp -#endif - /* Enable/disable LLC aware optimizations */ static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_llc); /* Enable/disable NUMA aware optimizations */ static DEFINE_STATIC_KEY_FALSE(scx_selcpu_topo_numa); -static struct { +/* + * cpumasks to track idle CPUs within each NUMA node. + * + * If SCX_OPS_BUILTIN_IDLE_PER_NODE is not enabled, a single global cpumask + * from is used to track all the idle CPUs in the system. + */ +struct scx_idle_cpus { cpumask_var_t cpu; cpumask_var_t smt; -} idle_masks CL_ALIGNED_IF_ONSTACK; +}; + +/* + * Global host-wide idle cpumasks (used when SCX_OPS_BUILTIN_IDLE_PER_NODE + * is not enabled). + */ +static struct scx_idle_cpus scx_idle_global_masks; + +/* + * Per-node idle cpumasks. + */ +static struct scx_idle_cpus **scx_idle_node_masks; + +/* + * Return the idle masks associated to a target @node. + * + * NUMA_NO_NODE identifies the global idle cpumask. + */ +static struct scx_idle_cpus *idle_cpumask(int node) +{ + return node == NUMA_NO_NODE ? &scx_idle_global_masks : scx_idle_node_masks[node]; +} + +/* + * Returns the NUMA node ID associated with a @cpu, or NUMA_NO_NODE if + * per-node idle cpumasks are disabled. + */ +static int scx_cpu_node_if_enabled(int cpu) +{ + if (!static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) + return NUMA_NO_NODE; + + return cpu_to_node(cpu); +} bool scx_idle_test_and_clear_cpu(int cpu) { + int node = scx_cpu_node_if_enabled(cpu); + struct cpumask *idle_cpus = idle_cpumask(node)->cpu; + #ifdef CONFIG_SCHED_SMT /* * SMT mask should be cleared whether we can claim @cpu or not. The SMT @@ -45,33 +81,38 @@ bool scx_idle_test_and_clear_cpu(int cpu) */ if (sched_smt_active()) { const struct cpumask *smt = cpu_smt_mask(cpu); + struct cpumask *idle_smts = idle_cpumask(node)->smt; /* * If offline, @cpu is not its own sibling and * scx_pick_idle_cpu() can get caught in an infinite loop as - * @cpu is never cleared from idle_masks.smt. Ensure that @cpu - * is eventually cleared. + * @cpu is never cleared from the idle SMT mask. Ensure that + * @cpu is eventually cleared. * * NOTE: Use cpumask_intersects() and cpumask_test_cpu() to * reduce memory writes, which may help alleviate cache * coherence pressure. */ - if (cpumask_intersects(smt, idle_masks.smt)) - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); - else if (cpumask_test_cpu(cpu, idle_masks.smt)) - __cpumask_clear_cpu(cpu, idle_masks.smt); + if (cpumask_intersects(smt, idle_smts)) + cpumask_andnot(idle_smts, idle_smts, smt); + else if (cpumask_test_cpu(cpu, idle_smts)) + __cpumask_clear_cpu(cpu, idle_smts); } #endif - return cpumask_test_and_clear_cpu(cpu, idle_masks.cpu); + + return cpumask_test_and_clear_cpu(cpu, idle_cpus); } -s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) +/* + * Pick an idle CPU in a specific NUMA node. + */ +static s32 pick_idle_cpu_in_node(const struct cpumask *cpus_allowed, int node, u64 flags) { int cpu; retry: if (sched_smt_active()) { - cpu = cpumask_any_and_distribute(idle_masks.smt, cpus_allowed); + cpu = cpumask_any_and_distribute(idle_cpumask(node)->smt, cpus_allowed); if (cpu < nr_cpu_ids) goto found; @@ -79,7 +120,7 @@ s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) return -EBUSY; } - cpu = cpumask_any_and_distribute(idle_masks.cpu, cpus_allowed); + cpu = cpumask_any_and_distribute(idle_cpumask(node)->cpu, cpus_allowed); if (cpu >= nr_cpu_ids) return -EBUSY; @@ -90,6 +131,85 @@ s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) goto retry; } +/* + * Tracks nodes that have not yet been visited when searching for an idle + * CPU across all available nodes. + */ +static DEFINE_PER_CPU(nodemask_t, per_cpu_unvisited); + +/* + * Search for an idle CPU across all nodes, excluding @node. + */ +static s32 pick_idle_cpu_from_online_nodes(const struct cpumask *cpus_allowed, int node, u64 flags) +{ + nodemask_t *unvisited; + s32 cpu = -EBUSY; + + preempt_disable(); + unvisited = this_cpu_ptr(&per_cpu_unvisited); + + /* + * Restrict the search to the online nodes (excluding the current + * node that has been visited already). + */ + nodes_copy(*unvisited, node_states[N_ONLINE]); + node_clear(node, *unvisited); + + /* + * Traverse all nodes in order of increasing distance, starting + * from @node. + * + * This loop is O(N^2), with N being the amount of NUMA nodes, + * which might be quite expensive in large NUMA systems. However, + * this complexity comes into play only when a scheduler enables + * SCX_OPS_BUILTIN_IDLE_PER_NODE and it's requesting an idle CPU + * without specifying a target NUMA node, so it shouldn't be a + * bottleneck is most cases. + * + * As a future optimization we may want to cache the list of nodes + * in a per-node array, instead of actually traversing them every + * time. + */ + for_each_node_numadist(node, *unvisited) { + cpu = pick_idle_cpu_in_node(cpus_allowed, node, flags); + if (cpu >= 0) + break; + } + preempt_enable(); + + return cpu; +} + +/* + * Find an idle CPU in the system, starting from @node. + */ +s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags) +{ + s32 cpu; + + /* + * Always search in the starting node first (this is an + * optimization that can save some cycles even when the search is + * not limited to a single node). + */ + cpu = pick_idle_cpu_in_node(cpus_allowed, node, flags); + if (cpu >= 0) + return cpu; + + /* + * Stop the search if we are using only a single global cpumask + * (NUMA_NO_NODE) or if the search is restricted to the first node + * only. + */ + if (node == NUMA_NO_NODE || flags & SCX_PICK_IDLE_IN_NODE) + return -EBUSY; + + /* + * Extend the search to the other online nodes. + */ + return pick_idle_cpu_from_online_nodes(cpus_allowed, node, flags); +} + /* * Return the amount of CPUs in the same LLC domain of @cpu (or zero if the LLC * domain is not defined). @@ -302,6 +422,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool { const struct cpumask *llc_cpus = NULL; const struct cpumask *numa_cpus = NULL; + int node = scx_cpu_node_if_enabled(prev_cpu); s32 cpu; *found = false; @@ -359,9 +480,9 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * piled up on it even if there is an idle core elsewhere on * the system. */ - if (!cpumask_empty(idle_masks.cpu) && - !(current->flags & PF_EXITING) && - cpu_rq(cpu)->scx.local_dsq.nr == 0) { + if (!(current->flags & PF_EXITING) && + cpu_rq(cpu)->scx.local_dsq.nr == 0 && + !cpumask_empty(idle_cpumask(cpu_to_node(cpu))->cpu)) { if (cpumask_test_cpu(cpu, p->cpus_ptr)) goto cpu_found; } @@ -375,7 +496,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool /* * Keep using @prev_cpu if it's part of a fully idle core. */ - if (cpumask_test_cpu(prev_cpu, idle_masks.smt) && + if (cpumask_test_cpu(prev_cpu, idle_cpumask(node)->smt) && scx_idle_test_and_clear_cpu(prev_cpu)) { cpu = prev_cpu; goto cpu_found; @@ -385,7 +506,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any fully idle core in the same LLC domain. */ if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, SCX_PICK_IDLE_CORE); + cpu = pick_idle_cpu_in_node(llc_cpus, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } @@ -394,15 +515,19 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any fully idle core in the same NUMA node. */ if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, SCX_PICK_IDLE_CORE); + cpu = pick_idle_cpu_in_node(numa_cpus, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } /* * Search for any full idle core usable by the task. + * + * If NUMA aware idle selection is enabled, the search will + * begin in prev_cpu's node and proceed to other nodes in + * order of increasing distance. */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, SCX_PICK_IDLE_CORE); + cpu = scx_pick_idle_cpu(p->cpus_ptr, node, SCX_PICK_IDLE_CORE); if (cpu >= 0) goto cpu_found; } @@ -419,7 +544,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any idle CPU in the same LLC domain. */ if (llc_cpus) { - cpu = scx_pick_idle_cpu(llc_cpus, 0); + cpu = pick_idle_cpu_in_node(llc_cpus, node, 0); if (cpu >= 0) goto cpu_found; } @@ -428,7 +553,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool * Search for any idle CPU in the same NUMA node. */ if (numa_cpus) { - cpu = scx_pick_idle_cpu(numa_cpus, 0); + cpu = pick_idle_cpu_in_node(numa_cpus, node, 0); if (cpu >= 0) goto cpu_found; } @@ -436,7 +561,7 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool /* * Search for any idle CPU usable by the task. */ - cpu = scx_pick_idle_cpu(p->cpus_ptr, 0); + cpu = scx_pick_idle_cpu(p->cpus_ptr, node, 0); if (cpu >= 0) goto cpu_found; @@ -450,30 +575,54 @@ s32 scx_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, u64 wake_flags, bool return cpu; } +/* + * Initialize global and per-node idle cpumasks. + */ void scx_idle_init_masks(void) { - BUG_ON(!alloc_cpumask_var(&idle_masks.cpu, GFP_KERNEL)); - BUG_ON(!alloc_cpumask_var(&idle_masks.smt, GFP_KERNEL)); + int node; + + /* Allocate global idle cpumasks */ + BUG_ON(!alloc_cpumask_var(&scx_idle_global_masks.cpu, GFP_KERNEL)); + BUG_ON(!alloc_cpumask_var(&scx_idle_global_masks.smt, GFP_KERNEL)); + + /* Allocate per-node idle cpumasks */ + scx_idle_node_masks = kcalloc(num_possible_nodes(), + sizeof(*scx_idle_node_masks), GFP_KERNEL); + BUG_ON(!scx_idle_node_masks); + + for_each_node(node) { + scx_idle_node_masks[node] = kzalloc_node(sizeof(**scx_idle_node_masks), + GFP_KERNEL, node); + BUG_ON(!scx_idle_node_masks[node]); + + BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[node]->cpu, GFP_KERNEL, node)); + BUG_ON(!alloc_cpumask_var_node(&scx_idle_node_masks[node]->smt, GFP_KERNEL, node)); + } } static void update_builtin_idle(int cpu, bool idle) { - assign_cpu(cpu, idle_masks.cpu, idle); + int node = scx_cpu_node_if_enabled(cpu); + struct cpumask *idle_cpus = idle_cpumask(node)->cpu; + + assign_cpu(cpu, idle_cpus, idle); #ifdef CONFIG_SCHED_SMT if (sched_smt_active()) { const struct cpumask *smt = cpu_smt_mask(cpu); + struct cpumask *idle_smts = idle_cpumask(node)->smt; if (idle) { /* - * idle_masks.smt handling is racy but that's fine as - * it's only for optimization and self-correcting. + * idle_smt handling is racy but that's fine as it's + * only for optimization and self-correcting. */ - if (!cpumask_subset(smt, idle_masks.cpu)) + if (!cpumask_subset(smt, idle_cpus)) return; - cpumask_or(idle_masks.smt, idle_masks.smt, smt); + cpumask_or(idle_smts, idle_smts, smt); } else { - cpumask_andnot(idle_masks.smt, idle_masks.smt, smt); + cpumask_andnot(idle_smts, idle_smts, smt); } } #endif @@ -529,15 +678,36 @@ void __scx_update_idle(struct rq *rq, bool idle, bool do_notify) if (do_notify || is_idle_task(rq->curr)) update_builtin_idle(cpu, idle); } + +static void reset_idle_masks(struct sched_ext_ops *ops) +{ + int node; + + /* + * Consider all online cpus idle. Should converge to the actual state + * quickly. + */ + if (!(ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE)) { + cpumask_copy(idle_cpumask(NUMA_NO_NODE)->cpu, cpu_online_mask); + cpumask_copy(idle_cpumask(NUMA_NO_NODE)->smt, cpu_online_mask); + return; + } + + for_each_node(node) { + const struct cpumask *node_mask = cpumask_of_node(node); + + cpumask_and(idle_cpumask(node)->cpu, cpu_online_mask, node_mask); + cpumask_and(idle_cpumask(node)->smt, cpu_online_mask, node_mask); + } +} #endif /* CONFIG_SMP */ void scx_idle_enable(struct sched_ext_ops *ops) { - if (ops->update_idle && !(ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) { + if (!ops->update_idle || (ops->flags & SCX_OPS_KEEP_BUILTIN_IDLE)) + static_branch_enable(&scx_builtin_idle_enabled); + else static_branch_disable(&scx_builtin_idle_enabled); - return; - } - static_branch_enable(&scx_builtin_idle_enabled); if (ops->flags & SCX_OPS_BUILTIN_IDLE_PER_NODE) static_branch_enable(&scx_builtin_idle_per_node); @@ -545,12 +715,7 @@ void scx_idle_enable(struct sched_ext_ops *ops) static_branch_disable(&scx_builtin_idle_per_node); #ifdef CONFIG_SMP - /* - * Consider all online cpus idle. Should converge to the actual state - * quickly. - */ - cpumask_copy(idle_masks.cpu, cpu_online_mask); - cpumask_copy(idle_masks.smt, cpu_online_mask); + reset_idle_masks(ops); #endif } @@ -610,15 +775,21 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking * per-CPU cpumask. * - * Returns NULL if idle tracking is not enabled, or running on a UP kernel. + * Returns an empty mask if idle tracking is not enabled, or running on a + * UP kernel. */ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) { + if (static_branch_unlikely(&scx_builtin_idle_per_node)) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled"); + return cpu_none_mask; + } + if (!check_builtin_idle_enabled()) return cpu_none_mask; #ifdef CONFIG_SMP - return idle_masks.cpu; + return idle_cpumask(NUMA_NO_NODE)->cpu; #else return cpu_none_mask; #endif @@ -629,18 +800,24 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) * per-physical-core cpumask. Can be used to determine if an entire physical * core is free. * - * Returns NULL if idle tracking is not enabled, or running on a UP kernel. + * Returns an empty mask if idle tracking is not enabled, or running on a + * UP kernel. */ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask(void) { + if (static_branch_unlikely(&scx_builtin_idle_per_node)) { + scx_ops_error("SCX_OPS_BUILTIN_IDLE_PER_NODE enabled"); + return cpu_none_mask; + } + if (!check_builtin_idle_enabled()) return cpu_none_mask; #ifdef CONFIG_SMP if (sched_smt_active()) - return idle_masks.smt; + return idle_cpumask(NUMA_NO_NODE)->smt; else - return idle_masks.cpu; + return idle_cpumask(NUMA_NO_NODE)->cpu; #else return cpu_none_mask; #endif @@ -707,7 +884,7 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, if (!check_builtin_idle_enabled()) return -EBUSY; - return scx_pick_idle_cpu(cpus_allowed, flags); + return scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); } /** @@ -730,7 +907,7 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, s32 cpu; if (static_branch_likely(&scx_builtin_idle_enabled)) { - cpu = scx_pick_idle_cpu(cpus_allowed, flags); + cpu = scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); if (cpu >= 0) return cpu; } diff --git a/kernel/sched/ext_idle.h b/kernel/sched/ext_idle.h index 339b6ec9c4cb7..68c4307ce4f6f 100644 --- a/kernel/sched/ext_idle.h +++ b/kernel/sched/ext_idle.h @@ -16,12 +16,12 @@ struct sched_ext_ops; void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops); void scx_idle_init_masks(void); bool scx_idle_test_and_clear_cpu(int cpu); -s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags); +s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags); #else /* !CONFIG_SMP */ static inline void scx_idle_update_selcpu_topology(struct sched_ext_ops *ops) {} static inline void scx_idle_init_masks(void) {} static inline bool scx_idle_test_and_clear_cpu(int cpu) { return false; } -static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) +static inline s32 scx_pick_idle_cpu(const struct cpumask *cpus_allowed, int node, u64 flags) { return -EBUSY; } diff --git a/tools/sched_ext/include/scx/compat.h b/tools/sched_ext/include/scx/compat.h index d63cf40be8eee..03e06bd15c738 100644 --- a/tools/sched_ext/include/scx/compat.h +++ b/tools/sched_ext/include/scx/compat.h @@ -112,6 +112,9 @@ static inline bool __COMPAT_struct_has_field(const char *type, const char *field #define SCX_OPS_BUILTIN_IDLE_PER_NODE \ __COMPAT_ENUM_OR_ZERO("scx_ops_flags", "SCX_OPS_BUILTIN_IDLE_PER_NODE") +#define SCX_PICK_IDLE_IN_NODE \ + __COMPAT_ENUM_OR_ZERO("scx_pick_idle_cpu_flags", "SCX_PICK_IDLE_IN_NODE") + static inline long scx_hotplug_seq(void) { int fd; From patchwork Fri Feb 14 19:40:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Righi X-Patchwork-Id: 13975569 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2072.outbound.protection.outlook.com [40.107.94.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 915011C8622; Fri, 14 Feb 2025 19:42:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.94.72 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562147; cv=fail; b=kJA9hbORQFJny0+QK68hjOUzVZ9uoiw7B9fMhW4hiMcmIhrVI3OCA996qHXwJULB0gaGY4XRtEOHcaeO1y9Og6bDV83Z5YxQWn4MaF3QiDjgXw/PPhZA8kSRqUCjUpc6K/L7NQAQ/1sBpPtZWsTMiShOyoWDw0K5vRv84il+OCc= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1739562147; c=relaxed/simple; bh=sDGVPfb6up2rSLDcy8cbOnTnN+8t80l7w90UXys48WY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=C+O0WBTnzNs1Kn0POX0UM3pRCiwv0rr+V8ek0dgGJUa2UyKQSFxOjzF2CuYDauRVXkj6WHFFUpXt5y88sZma+6vg7TDlL1jzojKTzfqLduPpOIne+s3JyQUd5RkIYw5rIlmW/37G1vX/7uj9XjYmP65bNvRF6wUanTFGiz3cwwM= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=DMYqj1se; arc=fail smtp.client-ip=40.107.94.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="DMYqj1se" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=pyDsED4gUjz5ljCgV6oGwaOBSl2qHDEyhYRC3Hj40OIpMAZvzRWCjaDfz/xcXJVzVZgc3FN9nlh23IkB3jzsrL+4VIW1kfnk7QWMED6UgY7OTm/l3f/UMQzZCPitnU/pP82ZYNdtCfKZZtWSQLb+n72CUega0977NNQ25j9E8ZY0dWb+nEO3yY/VgaqhFbQCTawl4dgvcc2zL/eT/C4RLhbTWuYRM7Fp+TSQldUJAPxqiPLn7h/2e7M28X9QQZ6lUGF7oeQ8lCvmxdGJ/b8Zm7g8b4mu4Odtor9/5WSasDXxkxLgbSO5/+ROo+D9eHWKUs/0W+UXNUPzlUJ9THKJZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9NTjs8nLjgluBnTMHtJJ1+bXvBbVUWkfPeBqnBtq4gI=; b=c/ZJPj2LTuZqKyGj0FjDVWK//U+IhnERoMzIE5YsMSGD2IxUmJ6Nmbw+e2YxvyF93HuuPpNvT2AA7CQutq4/la38EgsbR1MT8m17k8ae27ancytaP5Awm6a+1U4+nOx3uDO0bCIgnn+qQHvNz9I4z9X70A3Rfl/zolXhM9PPssYsU8gQtWkgGBUYGk8Pzvl9IneqrDh3rDmI9DtVfuyKnM0tnUQQjigKrmW4BG/610aVO6fJs0L1wzlsAKI0bDuk4biO16Bxh0r+vY/LfUHVd4DyUNvoSVwk0+zF+EoZwA65RbDFJ3yLTP2vo0ES0LwlYg2tf0zTr7cC1VX2Hz3zcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9NTjs8nLjgluBnTMHtJJ1+bXvBbVUWkfPeBqnBtq4gI=; b=DMYqj1sehSDhY/aSFamE7ST57qpoVu2dvtcYzIGznph40DCrB8QZVWp99mmw53SCKNCjcJuZGr4foWvoD9CyiYkdG9JL1CQ+Z3ZlKe4C2dChQRPc9Vj2ey60PZUzAbPSrjbKoPM8t8e1rtNSZZI7Ds10czVJRN/efaxaOIpbuSH5sguPuTvRQ72Mdujdf7rQ61IoIqzFoRlOI3j/p4twEQIu9B33+Ct1JstZ8JM8UvYnMAli1rBiKJOZIYMg+TQjmml7uvOVPdPn9u6TUGIMe+WfpZT3owPTlnxoXw4C+wu8juRUNTFnPtS45xjPSoaD4/DZOfI9L+wNi3qXEMA6Jg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) by MN2PR12MB4360.namprd12.prod.outlook.com (2603:10b6:208:266::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Fri, 14 Feb 2025 19:42:21 +0000 Received: from CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5]) by CY5PR12MB6405.namprd12.prod.outlook.com ([fe80::2119:c96c:b455:53b5%7]) with mapi id 15.20.8445.013; Fri, 14 Feb 2025 19:42:21 +0000 From: Andrea Righi To: Tejun Heo , David Vernet , Changwoo Min Cc: Yury Norov , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Joel Fernandes , Ian May , bpf@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 8/8] sched_ext: idle: Introduce node-aware idle cpu kfunc helpers Date: Fri, 14 Feb 2025 20:40:07 +0100 Message-ID: <20250214194134.658939-9-arighi@nvidia.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250214194134.658939-1-arighi@nvidia.com> References: <20250214194134.658939-1-arighi@nvidia.com> X-ClientProxiedBy: FR4P281CA0008.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:c8::19) To CY5PR12MB6405.namprd12.prod.outlook.com (2603:10b6:930:3e::17) Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY5PR12MB6405:EE_|MN2PR12MB4360:EE_ X-MS-Office365-Filtering-Correlation-Id: 0902b99d-d460-4a0b-7948-08dd4d2fb2b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|376014|7416014|1800799024|366016; X-Microsoft-Antispam-Message-Info: JXcWRRN6wAKcnx+/p2/8hqUI83ngV39jNZ96qebIMKufUHtOId04U/Pmpnc8zG5ff3Aa5NUkESOMbpib8P7nPczKl/rlTkIpoH67aEVycd0fVg+VK3SbQ/ljsMKZrduHOkApNzcNb6KTCTz9g8XLaugqc3JAT/qz0QSrJmCRf+f8RE7rAQ0iWjvJ2+lb+MMbSYKKQoNtlxoKVmTMznauhC/2j/8np5IGvJzkrYi3h4Ai26t2PqYQA9UMRc+g3CxeU2QAWNv4UIT5T8wK8N4kCXXLLE+QWK/GNwEdPh3J6pkriOWruHAL82rLOsk2PfGXDXaS/xMNKf1fefnhzGtrmV3BcrMZ4v/+9hgkAdpY7jhVivwXotj1/fE+j5HbeMDNlftmt21Td+RBG/o0zJZTblF305/QVX4IZOCsdqwT88xUhxYNJS61JxQSBYZasjPdq0dFTtwXCbIvwkv3oBWVtscYVNYUJXPw388yLYwkBLuPxlE71Wmg5l+bjfjKJ+l6Mbk2wbueRXKx1i2EEsu23YpjgC7a/uCoYmWWi/yz5EaeTCa0ZU8mqI8CfEMETpUcxCuAzAgQfi+kYqJtO7T4sxL0kDqmwTK2+ZMRaHeI/aIhurrhC6UzHk77sobr6aoZlGQyzYhH3J5bjNsgM+wtpzTMwkKRWC+13TmaByopedAyPwiR1bnrP6szSJtvwFe0xD0KFJBdvAIht3GSKAmk7GkWUfHwVbfFQCIY+pni9RwtyRKNJAlSSChaN1q8XZ8brgONWeuydDkw2eQphWLU+1b6cy4rHgIRybZ8XB/2mACYVOW3guOAsZZNwttkvSuNL3oBmLJzep7zcACBVRxfKhL14wLKzYAc6SRyfQ8crM1PxbrVlzxiJcMg26Y2xAqP6D2Uqrm4V94CZmtZw2CD5gcP8J44zCsjwrH5EonEiQW3S40lfZrydvWUBtfXlGC5Xg9a2V/Nfjowj0f2ZQKUEp9N7FrxcjkKG/PQkFIeo6MXQdmdc0lfoP4RbnbAsNrSC1yayNrvcJUfKmG3tcAqWS1uoNen9Hy3vrLdeW4ykkIwNWzoVb75zwn/XcOvwh27dDmwjAbsmRW9YNEtzz1Za83h9rS0iWwe3WrPeP1kSW5g9S/OjGVngpXtj7dUqPjf/G1RhevOrBtp22v/aIgheP7e0kzVujBW9T8jWN5DCxRGImjU5QHijZqo/VE7D4X/yi6rijp8V+/Tr8tvuHVI4wtCD7p6e6drrXAQ6Pk8X6qqkDtaHCks5QbXziZwmtoEUm30rOpLxlfP5uGcdyyd1rLjSdgP9PGUVwiDntxA7c26UOz/ACFp4oZJRBvidua1g9JvtDy5p0jMzboYfXdbM5kJ4oM/yZtPLi1df7FwKWHognfAWH84iRwTcR8IMKtp X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY5PR12MB6405.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(7416014)(1800799024)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tESzkwkbOjECdpuWZItJghw2Cvk353kqpA4N2RiVlMDcssAveoKuswJvUY8pPAciknCps6+xjClU0WIuLlqcGcI3Slde6HmxuFvly9Lirmg6J2sk5c/NUthSsNjDxyIjmPRrs/rlJ7PoIRYcOy+qBBti2zVaJwKfHVjqhRqUTKO/rQDy3nF2L/p/5WGydvgYIk9sbp2E/CXnW0V/DNYDhPtvYjh5/AMNsNXjfhdkqw5bCWhhs4EYQ1c0BB3Np1/zSuxAHX79zhGaW/StCJiB9yGCFIuqm3hEazYLN+hmjVF6MMNtjCBlrQeqvudH2u2+KqiEFxRuTnIPOf8nWQvhXj5IbNaE+FvrcKaAjUBmkPBYFbjPgWEZsyRsWej1Ifxf+nIotjLgwwr5msaTHQTnZXaFoJn35C2HWVodvHctKe+DgXr1NHs3DgfVQwvMfGJaXEpN8QdkKNHDyYnWaolOHQkDFu5JpEuh070rPiHBblx3WmxTtsz//PXzAy5QACd0/dAkVUoLUa3X5n03UZIMc735PWWO1LiIJkGyJRudJnkTFZVXhq4YRLiKeip3fDgtjaI9maYyVH/brwGKxMsT+tH3KwHjkqOSn23gNOEcB4aVWFwAc0CGe2T6ScyLW2etsGcccnISAt2ToxeNP5FKeteXL+2UffN40IeBZ1jFkDpi0WDbxJWp2f0cNsipvMjbgf3Z6xcpqUH1mn2mmJ6KXD84gR2XOjXYSWPzimkVKoXfnokaotUS1LOvljGh5kYoVuFaHSTXlVceti0+/qw6y3M6CAaWlIu09mcJVPgC9IWGh2eSYWhpzlex8u5t5AvMxfmnP2SBIBJvbV2F1LR/ZPC677Z2fstq9UV7trACIWYV4Bx8WprAoP6P94h5/dlhZdUoGgrxWA7u6ufChOpbNg8quXGL1KKrjrp2y2iSdplqggweIrUc1expQKTZMRwOakidztyjDZf+OdXlYb6gfQYfFhW4BNYqYyA/vO3e0nN7zSN0JhvwEju5aVSiVYm+pPwnDtYiAjp6Ge3NblF1AscLdXrOuHZzU07cgnsRcHDRSC8Vijn/8MptJrKCRKIs8bPdkTy7ruTZRjtaYL70P9o2V/m1uElZ+5TsCihRJK9h91wRg1cUbsNnpdVOaut+bKCCnSiVIeOwpRJi5KVLHIahslKRJJ67juicFgz2+G+1Rhu/LKzSKXs5M1Sv+P0vrJFqECsVj2DNTSdciBNBssPjE4BAkyzsDXbq5Z6K9cGKkY2dhnO++6v/mv9CNTe3881np5xYv/MUv7tmN3FrUoG7MQc1oWHBMCnynUOGS7Pbk0/agS1nAXopbyGOUGueWc5hO9tEfYBkZddDtc1z/F4cZ8Rospyv2mLQ/taIfIwv8WfwBbpf1Vg/GGkgV3n6ovV+y5l3inxfcTYObCLSfqmK5rbNtaNVJ6KAAsrNieRxVmnLKHHmxiVEEuc5PdSIsK76oM7O4lVh6QGcmHxsgo0jGHljAMecqNrXYEP+vJ+iZnZViRZdjH/SG4qlzkxExiYAgi9eDEp2rOWlCrLFTiTPjn1+U7RZ2ispRzT3Mgxi5YHU8PEax67k9SqIYoUe X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0902b99d-d460-4a0b-7948-08dd4d2fb2b5 X-MS-Exchange-CrossTenant-AuthSource: CY5PR12MB6405.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Feb 2025 19:42:21.5651 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lAUpyjRK0ZIypzXOTwiIQMnKhTED9uBSIlDL9ylhPcWztT2I0jL7a8f5hy++NGZVVkwUlJzEn1Dr7SCuBCR5jw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4360 Introduce a new kfunc to retrieve the node associated to a CPU: int scx_bpf_cpu_node(s32 cpu) Add the following kfuncs to provide BPF schedulers direct access to per-node idle cpumasks information: const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) s32 scx_bpf_pick_idle_cpu_in_node(const cpumask_t *cpus_allowed, int node, u64 flags) s32 scx_bpf_pick_any_cpu_node(const cpumask_t *cpus_allowed, int node, u64 flags) Moreover, trigger an scx error when any of the non-node aware idle CPU kfuncs are used when SCX_OPS_BUILTIN_IDLE_PER_NODE is enabled. Cc: Yury Norov [NVIDIA] Signed-off-by: Andrea Righi Reviewed-by: Yury Norov [NVIDIA] --- kernel/sched/ext_idle.c | 180 +++++++++++++++++++++++ tools/sched_ext/include/scx/common.bpf.h | 5 + tools/sched_ext/include/scx/compat.bpf.h | 31 ++++ 3 files changed, 216 insertions(+) diff --git a/kernel/sched/ext_idle.c b/kernel/sched/ext_idle.c index 8dacccc82ed63..5c062affd622c 100644 --- a/kernel/sched/ext_idle.c +++ b/kernel/sched/ext_idle.c @@ -728,6 +728,33 @@ void scx_idle_disable(void) /******************************************************************************** * Helpers that can be called from the BPF scheduler. */ + +static int validate_node(int node) +{ + if (!static_branch_likely(&scx_builtin_idle_per_node)) { + scx_ops_error("per-node idle tracking is disabled"); + return -EOPNOTSUPP; + } + + /* Return no entry for NUMA_NO_NODE (not a critical scx error) */ + if (node == NUMA_NO_NODE) + return -ENOENT; + + /* Make sure node is in a valid range */ + if (node < 0 || node >= nr_node_ids) { + scx_ops_error("invalid node %d", node); + return -EINVAL; + } + + /* Make sure the node is part of the set of possible nodes */ + if (!node_possible(node)) { + scx_ops_error("unavailable node %d", node); + return -EINVAL; + } + + return node; +} + __bpf_kfunc_start_defs(); static bool check_builtin_idle_enabled(void) @@ -739,6 +766,23 @@ static bool check_builtin_idle_enabled(void) return false; } +/** + * scx_bpf_cpu_node - Return the NUMA node the given @cpu belongs to, or + * trigger an error if @cpu is invalid + * @cpu: target CPU + */ +__bpf_kfunc int scx_bpf_cpu_node(s32 cpu) +{ +#ifdef CONFIG_NUMA + if (!ops_cpu_valid(cpu, NULL)) + return NUMA_NO_NODE; + + return cpu_to_node(cpu); +#else + return 0; +#endif +} + /** * scx_bpf_select_cpu_dfl - The default implementation of ops.select_cpu() * @p: task_struct to select a CPU for @@ -771,6 +815,27 @@ __bpf_kfunc s32 scx_bpf_select_cpu_dfl(struct task_struct *p, s32 prev_cpu, return prev_cpu; } +/** + * scx_bpf_get_idle_cpumask_node - Get a referenced kptr to the + * idle-tracking per-CPU cpumask of a target NUMA node. + * + * Returns an empty cpumask if idle tracking is not enabled, if @node is + * not valid, or running on a UP kernel. In this case the actual error will + * be reported to the BPF scheduler via scx_ops_error(). + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) +{ + node = validate_node(node); + if (node < 0) + return cpu_none_mask; + +#ifdef CONFIG_SMP + return idle_cpumask(node)->cpu; +#else + return cpu_none_mask; +#endif +} + /** * scx_bpf_get_idle_cpumask - Get a referenced kptr to the idle-tracking * per-CPU cpumask. @@ -795,6 +860,31 @@ __bpf_kfunc const struct cpumask *scx_bpf_get_idle_cpumask(void) #endif } +/** + * scx_bpf_get_idle_smtmask_node - Get a referenced kptr to the + * idle-tracking, per-physical-core cpumask of a target NUMA node. Can be + * used to determine if an entire physical core is free. + * + * Returns an empty cpumask if idle tracking is not enabled, if @node is + * not valid, or running on a UP kernel. In this case the actual error will + * be reported to the BPF scheduler via scx_ops_error(). + */ +__bpf_kfunc const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) +{ + node = validate_node(node); + if (node < 0) + return cpu_none_mask; + +#ifdef CONFIG_SMP + if (sched_smt_active()) + return idle_cpumask(node)->smt; + else + return idle_cpumask(node)->cpu; +#else + return cpu_none_mask; +#endif +} + /** * scx_bpf_get_idle_smtmask - Get a referenced kptr to the idle-tracking, * per-physical-core cpumask. Can be used to determine if an entire physical @@ -859,6 +949,35 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) return false; } +/** + * scx_bpf_pick_idle_cpu_in_node - Pick and claim an idle cpu from @node + * @cpus_allowed: Allowed cpumask + * @node: target NUMA node + * @flags: %SCX_PICK_IDLE_* flags + * + * Pick and claim an idle cpu in @cpus_allowed from the NUMA node @node. + * + * Returns the picked idle cpu number on success, or -%EBUSY if no matching + * cpu was found. + * + * The search starts from @node and proceeds to other online NUMA nodes in + * order of increasing distance (unless SCX_PICK_IDLE_IN_NODE is specified, + * in which case the search is limited to the target @node). + * + * Always returns an error if ops.update_idle() is implemented and + * %SCX_OPS_KEEP_BUILTIN_IDLE is not set, or if + * %SCX_OPS_BUILTIN_IDLE_PER_NODE is not set. + */ +__bpf_kfunc s32 scx_bpf_pick_idle_cpu_in_node(const struct cpumask *cpus_allowed, + int node, u64 flags) +{ + node = validate_node(node); + if (node < 0) + return node; + + return scx_pick_idle_cpu(cpus_allowed, node, flags); +} + /** * scx_bpf_pick_idle_cpu - Pick and claim an idle cpu * @cpus_allowed: Allowed cpumask @@ -877,16 +996,64 @@ __bpf_kfunc bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) * * Unavailable if ops.update_idle() is implemented and * %SCX_OPS_KEEP_BUILTIN_IDLE is not set. + * + * Always returns an error if %SCX_OPS_BUILTIN_IDLE_PER_NODE is set, use + * scx_bpf_pick_idle_cpu_in_node() instead. */ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, u64 flags) { + if (static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) { + scx_ops_error("per-node idle tracking is enabled"); + return -EBUSY; + } + if (!check_builtin_idle_enabled()) return -EBUSY; return scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); } +/** + * scx_bpf_pick_any_cpu_in_node - Pick and claim an idle cpu if available + * or pick any CPU from @node + * @cpus_allowed: Allowed cpumask + * @node: target NUMA node + * @flags: %SCX_PICK_IDLE_CPU_* flags + * + * Pick and claim an idle cpu in @cpus_allowed. If none is available, pick any + * CPU in @cpus_allowed. Guaranteed to succeed and returns the picked idle cpu + * number if @cpus_allowed is not empty. -%EBUSY is returned if @cpus_allowed is + * empty. + * + * The search starts from @node and proceeds to other online NUMA nodes in + * order of increasing distance (unless SCX_PICK_IDLE_IN_NODE is specified, + * in which case the search is limited to the target @node). + * + * If ops.update_idle() is implemented and %SCX_OPS_KEEP_BUILTIN_IDLE is not + * set, this function can't tell which CPUs are idle and will always pick any + * CPU. + */ +__bpf_kfunc s32 scx_bpf_pick_any_cpu_in_node(const struct cpumask *cpus_allowed, + int node, u64 flags) +{ + s32 cpu; + + node = validate_node(node); + if (node < 0) + return node; + + cpu = scx_pick_idle_cpu(cpus_allowed, node, flags); + if (cpu >= 0) + return cpu; + + cpu = cpumask_any_distribute(cpus_allowed); + if (cpu < nr_cpu_ids) + return cpu; + else + return -EBUSY; +} + /** * scx_bpf_pick_any_cpu - Pick and claim an idle cpu if available or pick any CPU * @cpus_allowed: Allowed cpumask @@ -900,12 +1067,20 @@ __bpf_kfunc s32 scx_bpf_pick_idle_cpu(const struct cpumask *cpus_allowed, * If ops.update_idle() is implemented and %SCX_OPS_KEEP_BUILTIN_IDLE is not * set, this function can't tell which CPUs are idle and will always pick any * CPU. + * + * Always returns an error if %SCX_OPS_BUILTIN_IDLE_PER_NODE is set, use + * scx_bpf_pick_any_cpu_in_node() instead. */ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, u64 flags) { s32 cpu; + if (static_branch_maybe(CONFIG_NUMA, &scx_builtin_idle_per_node)) { + scx_ops_error("per-node idle tracking is enabled"); + return -EBUSY; + } + if (static_branch_likely(&scx_builtin_idle_enabled)) { cpu = scx_pick_idle_cpu(cpus_allowed, NUMA_NO_NODE, flags); if (cpu >= 0) @@ -922,11 +1097,16 @@ __bpf_kfunc s32 scx_bpf_pick_any_cpu(const struct cpumask *cpus_allowed, __bpf_kfunc_end_defs(); BTF_KFUNCS_START(scx_kfunc_ids_idle) +BTF_ID_FLAGS(func, scx_bpf_cpu_node) +BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask_node, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_idle_cpumask, KF_ACQUIRE) +BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask_node, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_get_idle_smtmask, KF_ACQUIRE) BTF_ID_FLAGS(func, scx_bpf_put_idle_cpumask, KF_RELEASE) BTF_ID_FLAGS(func, scx_bpf_test_and_clear_cpu_idle) +BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu_in_node, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_pick_idle_cpu, KF_RCU) +BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu_in_node, KF_RCU) BTF_ID_FLAGS(func, scx_bpf_pick_any_cpu, KF_RCU) BTF_KFUNCS_END(scx_kfunc_ids_idle) diff --git a/tools/sched_ext/include/scx/common.bpf.h b/tools/sched_ext/include/scx/common.bpf.h index 77bbe0199a32c..cd1659c5d3f46 100644 --- a/tools/sched_ext/include/scx/common.bpf.h +++ b/tools/sched_ext/include/scx/common.bpf.h @@ -71,14 +71,19 @@ u32 scx_bpf_cpuperf_cap(s32 cpu) __ksym __weak; u32 scx_bpf_cpuperf_cur(s32 cpu) __ksym __weak; void scx_bpf_cpuperf_set(s32 cpu, u32 perf) __ksym __weak; u32 scx_bpf_nr_cpu_ids(void) __ksym __weak; +int scx_bpf_cpu_node(s32 cpu) __ksym __weak; const struct cpumask *scx_bpf_get_possible_cpumask(void) __ksym __weak; const struct cpumask *scx_bpf_get_online_cpumask(void) __ksym __weak; void scx_bpf_put_cpumask(const struct cpumask *cpumask) __ksym __weak; +const struct cpumask *scx_bpf_get_idle_cpumask_node(int node) __ksym __weak; const struct cpumask *scx_bpf_get_idle_cpumask(void) __ksym; +const struct cpumask *scx_bpf_get_idle_smtmask_node(int node) __ksym __weak; const struct cpumask *scx_bpf_get_idle_smtmask(void) __ksym; void scx_bpf_put_idle_cpumask(const struct cpumask *cpumask) __ksym; bool scx_bpf_test_and_clear_cpu_idle(s32 cpu) __ksym; +s32 scx_bpf_pick_idle_cpu_in_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; s32 scx_bpf_pick_idle_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; +s32 scx_bpf_pick_any_cpu_in_node(const cpumask_t *cpus_allowed, int node, u64 flags) __ksym __weak; s32 scx_bpf_pick_any_cpu(const cpumask_t *cpus_allowed, u64 flags) __ksym; bool scx_bpf_task_running(const struct task_struct *p) __ksym; s32 scx_bpf_task_cpu(const struct task_struct *p) __ksym; diff --git a/tools/sched_ext/include/scx/compat.bpf.h b/tools/sched_ext/include/scx/compat.bpf.h index e5fa72f9bf22b..617ed0ec85dc4 100644 --- a/tools/sched_ext/include/scx/compat.bpf.h +++ b/tools/sched_ext/include/scx/compat.bpf.h @@ -182,6 +182,37 @@ static inline bool __COMPAT_is_enq_cpu_selected(u64 enq_flags) scx_bpf_now() : \ bpf_ktime_get_ns()) +/* + * + * v6.15: Introduce NUMA-aware kfuncs to operate with per-node idle + * cpumasks. + * + * Preserve the following __COMPAT_scx_*_node macros until v6.17. + */ +#define __COMPAT_scx_bpf_cpu_node(cpu) \ + (bpf_ksym_exists(scx_bpf_cpu_node) ? \ + scx_bpf_cpu_node(cpu) : 0) + +#define __COMPAT_scx_bpf_get_idle_cpumask_node(node) \ + (bpf_ksym_exists(scx_bpf_get_idle_cpumask_node) ? \ + scx_bpf_get_idle_cpumask_node(node) : \ + scx_bpf_get_idle_cpumask()) \ + +#define __COMPAT_scx_bpf_get_idle_smtmask_node(node) \ + (bpf_ksym_exists(scx_bpf_get_idle_smtmask_node) ? \ + scx_bpf_get_idle_smtmask_node(node) : \ + scx_bpf_get_idle_smtmask()) + +#define __COMPAT_scx_bpf_pick_idle_cpu_in_node(cpus_allowed, node, flags) \ + (bpf_ksym_exists(scx_bpf_pick_idle_cpu_in_node) ? \ + scx_bpf_pick_idle_cpu_in_node(cpus_allowed, node, flags) : \ + scx_bpf_pick_idle_cpu(cpus_allowed, flags)) + +#define __COMPAT_scx_bpf_pick_any_cpu_in_node(cpus_allowed, node, flags) \ + (bpf_ksym_exists(scx_bpf_pick_any_cpu_in_node) ? \ + scx_bpf_pick_any_cpu_in_node(cpus_allowed, node, flags) : \ + scx_bpf_pick_any_cpu(cpus_allowed, flags)) + /* * Define sched_ext_ops. This may be expanded to define multiple variants for * backward compatibility. See compat.h::SCX_OPS_LOAD/ATTACH().