From patchwork Wed Aug 2 11:24:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13338050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5FA0C04E69 for ; Wed, 2 Aug 2023 11:25:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7888F28015A; Wed, 2 Aug 2023 07:25:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 628E4280143; Wed, 2 Aug 2023 07:25:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49E8628015B; Wed, 2 Aug 2023 07:25:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 05988280143 for ; Wed, 2 Aug 2023 07:25:43 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CE0F5160DD4 for ; Wed, 2 Aug 2023 11:25:42 +0000 (UTC) X-FDA: 81078934524.26.85B4FC7 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf10.hostedemail.com (Postfix) with ESMTP id 1C750C000C for ; Wed, 2 Aug 2023 11:25:40 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=cIQ2HGS3; dmarc=none; spf=none (imf10.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690975541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=Ms9C/9A0FSfh2mm0WkpMIkoUR7i908GJu9ttwY7f2Do=; b=J/FcNc2vvt+8ojcleXbhSbXPw0LeOg0lCXICyjLlDKsBWOtXdRkMypHQbdIdmIEkksEYtT LAepZa6lgG1E8CQjt83Jg0ngfpC6Z4FqZqflXSaxRZ5sAnpJ0MhdYCZbzhTl878nD/AMUB lP8p/xQyFnAiY/2tGdSm2l3BuHmmZIU= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=cIQ2HGS3; dmarc=none; spf=none (imf10.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690975541; a=rsa-sha256; cv=none; b=6iCKqoFDHPlQhSGLrO3/GP0bjPqKok4clTJmD8o+Lpmk+s/NuM7xGZ+Aq4t8rlc3v3PeAH VmR6i2OiJ+LyMj6FRRMkjYluV+dXh/FHGswEHcP7hxiNmVCfNTvXTqN9LwqpBa07woEXnZ 7ObobXpbiU2DgEw7jAVXlzpRCIEoHDw= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=Ms9C/9A0FSfh2mm0WkpMIkoUR7i908GJu9ttwY7f2Do=; b=cIQ2HGS3cP+XZMAtZq1Byn+y8c U6DlXjpXUogUNUSsU3Z/1ZMmPDYlKpmMaVpsqTeUfuzKfKAtuEa8fIhvGTCGikB9NwA4+OyMZP0O2 F1SjnDk9AAd/CZiynZUd/LrY33NGVtR02jhaOeD4VHdmMkUTF5iTCMJSIZTaqabUyV2Idp2V2x2eu WOMLrKY0EFewFV2acVGKDuowTdZtXhDvF5Xq+hIMucWL4URCGJ0fxBrUJ9FUby0KkdYUxQtTw9mYe dmr/LSi244V/6B2b6X+UyISaTrNIpJX1KZ6WfW0mVLGa9NsttKQ9+45uvDf0XNcOePanj4+Cxcwjx 13N4T+3w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1qR9yx-00FtnU-16; Wed, 02 Aug 2023 11:25:35 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 9D59F3001DD; Wed, 2 Aug 2023 13:25:34 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 812F52107C43B; Wed, 2 Aug 2023 13:25:34 +0200 (CEST) Message-ID: <20230802112525.564928240@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 13:24:59 +0200 From: Peter Zijlstra To: yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk Cc: peterz@infradead.org, Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, rppt@kernel.org Subject: [PATCH 1/2] mm: Mark nr_node_ids __ro_after_init References: <20230802112458.230221601@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1C750C000C X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: d3nndeos78rkcgfwha5ousonu6ahcuuu X-HE-Tag: 1690975540-262610 X-HE-Meta: U2FsdGVkX18YlZehGMV+uTUThm4dfhP/av97mK5SIDajorq6FSE6X/UpzX44H19Yuu+XpKOOkv+gMj33QsJrdh2kVud5LWY1EeGG9kRqNBtilaciwErfLPnidd2oDxL/zZEjQ3Y8L2L18WreXPUZSlkXqTm/ijkAsR1UFdUSE8xHOAO2tU7GbRQUcz5bxkGIfOnXc3c8+dMRX3XEjVG0qJhtkDZ0mJvT8i6lRcwuoCtPJmBjrX3ykXikVtKiRV/JvoKr6/v15UW9ZUMQepgg2qYnnpo6aT14xU6kzN77/n5G9gZ/ipLgE5wrJVMi+cfs9lfpnsmv3sYjGBZ6jYPo39Bs/ymuuAVcp7OjvwmERoy/YbCivIlJZm+psKx+EUCJOhj5j6+4wlVMSLNMqXhPXA2HMt2tt8h9LZBIGwBuaJYuKmpb8JFJnVliMWHrh50vDA7gCR4bT/+JFGeM8n23SZGM6Fw5ilz0Ts4+pWxEZLVdgB5Psxv2/c2eFvzZYsSFFFULdL5ZBm9QSTLf6V941KgWWd35hBO7Y4JU4uFYFUMFP8W5Ezc1LSIvvukWUXDYE2Wmo/rLfKgmTauA7gXbEcLiQ/qL7A2sIzthx60/UTKzl9LVEAJ3/igApECyNIA0H/+zv9MCrOOx8IYph8D7VdL7K6pMlxf5kGArsXY4PI/Nz8WYFi/g+WRDYUeJ9mo76qRqUwNc9vVhzL2dykhtMyR23URxz9ayssa5UxMfGs/S2OtqrB9XUwR8VhafMGvoUx0qX1+B/y96QaFoYjAksCCwM4gUguTgOn/7pLYC4MsPYZwSxz8EPf2qhURUKvw+Iz05B2KeTs3me7CgHvPMOH+I21+tf45lWrjG61rF0dUySqDAjuikOwpqzNkRI6tI9qqLXttPdi+HUHIfREmX6T0ILfcjpRpmjQR3C0+Z4jKUbXM6totWoagkgbgfPplWsAr0/tpADM9EzZCHnHN wYlD743/ imwGhHMWFC9kzwb1tu7Lih7ZHPkQ2KtyYMkrm7TOn1KQ9oTncLutN0K42Ylf7JYLZOT9DBafAyiEDkYWJx2aQrwePmOsv9SfFhtpXtmbflsEO5GQewI7LshPGyHTpJJn0ui+1tQsj1SY5ruA9MBKSuVcKMuB4a3FQye++m5PwI+hk/OTYxfLouIx86J32hKXH/+n7mBzwUVM34NmoBvgwtlTni4QcF7CCi94H8A60o1GHLnaHryozCkwJegsjacpY66ijTb61sMdtW4YGS9YFdGMhEL3bTBqSKJOYGLQhXPyMdQBClFDKisOSe3b852E1G8wxQyL+k497FTTsN9o42CJztyaycLX7A0P1DlZ61g+boSunV7mJ0Nfa6IsMLNMUJ0Mff6ZLAmSiE78UGYqddhC05wqEKDpbs5on X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like nr_cpu_ids, nr_node_ids covers the side of possible_map and is thus constant after init. Mark the variable __ro_after_init to both improve performance and enhance security, since the variable is exported. A module writing to it would be 'unfortunate'. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Mike Rapoport (IBM) --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -305,7 +305,7 @@ int movable_zone; EXPORT_SYMBOL(movable_zone); #if MAX_NUMNODES > 1 -unsigned int nr_node_ids __read_mostly = MAX_NUMNODES; +unsigned int nr_node_ids __ro_after_init = MAX_NUMNODES; unsigned int nr_online_nodes __read_mostly = 1; EXPORT_SYMBOL(nr_node_ids); EXPORT_SYMBOL(nr_online_nodes); From patchwork Wed Aug 2 11:25:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13338049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA7FC001DF for ; Wed, 2 Aug 2023 11:25:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CA13280158; Wed, 2 Aug 2023 07:25:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4519C280143; Wed, 2 Aug 2023 07:25:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A4EA280158; Wed, 2 Aug 2023 07:25:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 13A44280143 for ; Wed, 2 Aug 2023 07:25:42 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CE7D61C9E2A for ; Wed, 2 Aug 2023 11:25:41 +0000 (UTC) X-FDA: 81078934482.04.6261826 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf27.hostedemail.com (Postfix) with ESMTP id 92D5B4000A for ; Wed, 2 Aug 2023 11:25:39 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XIis5ISh; dmarc=none; spf=none (imf27.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690975540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=S5a4dcigoffDzWno2NBShJb6tIhiVcSw+KbJyctBM6A=; b=piUUkmjf8LfDEAiYMH1wFfw5XiKTJzOMOS4hv/1EqRV8UZnTJUKyMbb38I0woxW0FV+t9B N+tH1J7rhqqj3msmSnr4v/dtggu4wAbPTWhP4g1MfggQGsjms9itUt5vu6Bvgzqc12xIKM /1oVF7i+pRVne7li96QTkMFCgZ/AkWY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XIis5ISh; dmarc=none; spf=none (imf27.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690975540; a=rsa-sha256; cv=none; b=WJx40buBl0VFfTnXQeGWDcEVOa4Q2pWyeabas7Sx9su/XIebh+oHnCyAlmCIDZ4elu5fb+ 8RcmkG2lhLMjJCGubUrkPNb3CpShbJQpsGJSt87FspbR4mYysKF/dwioOeSgdSkBnSyoIc nRlyBCGkfWKNGowTYhGPYwS0KmdFb2Y= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=S5a4dcigoffDzWno2NBShJb6tIhiVcSw+KbJyctBM6A=; b=XIis5ISh0bWAVmuT3K6CV9fdRK t12x9krGXJHpfheEDox47UFv4B1HX5GycBJzzIJV+Qe+nJnGPBTUpNYNWUS2gsUW9H/5bXSl4o5EM gGjFeSk6d4JSUuwc9tRg/KxlplqAOREyX0gg9d8BtwLqL9YEtZEsYckL2fXDsKgljUvr8ODXKtHcf K85q7AYq1U0ZUJGXsgcxBWAb2d6H05FICNMz1oJSd6PWcRAvY8qt4Gl/oZvb05zAUh28VDg+N5xA8 de5BymDU6PeFX8fhu1A6e9P0WdIrd/9kFAjU9iV1owUyjWtb0Run2RHCNv/loT7gag6bkinOO1H94 xuGDXx6g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qR9yx-00Efr0-BH; Wed, 02 Aug 2023 11:25:35 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 9F97D3006E2; Wed, 2 Aug 2023 13:25:34 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 83121205FF770; Wed, 2 Aug 2023 13:25:34 +0200 (CEST) Message-ID: <20230802112525.633758009@infradead.org> User-Agent: quilt/0.66 Date: Wed, 02 Aug 2023 13:25:00 +0200 From: Peter Zijlstra To: yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk Cc: peterz@infradead.org, Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, rppt@kernel.org Subject: [PATCH 2/2] mm,nodemask: Use nr_node_ids References: <20230802112458.230221601@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 92D5B4000A X-Stat-Signature: y45e5i4xqfrwxg6i8ff8551pp8quabzn X-HE-Tag: 1690975539-915356 X-HE-Meta: U2FsdGVkX197feU/vLbxLg6FsiqJk9WuvHi5IJ/N8TWjjPOasf08p0vCquor2irRDBOkgr5is6B4MULLyfgtrpJVpWXSIONi5vVbrwMasoNukKtioLx2uvctQvQBtRoP79SfkMz4jVxSjnmNnUM0DUXZrT71zq02lTuH8vcc7rT2oYvAkrJgspODoSP4rRsH2KYEqo1fqLOncPXLJbWRKAYWga78yQYM4Eyrh91kXMG57vI4YLa01F6AAXZNV220wV52XDAEL3YIeBL09WgHMS9+1kmfQ7D03mSeGZPmrPCuFvuyZMLWw9a7mzWWAd3vfOaeeMyDZzL2tRlWs84s0/73tU4G0N36QdQCRbFW8P75fN2TR52ZavHXxh09CaNLUM94HYjDy8Wsie4yKPCNgENeh/yHdcH+GQ1QBtFxt59iXCnGtvHJPjnZElVZmT0WEt0eGhxm7XTTiidLcnW9oKv14YfEGDbTD49MOBRs0YEQYEHzsrWas/IdKrnCeTDa5Dkv5h512GIZskb8Ht4kjTgd232g595DTZyKdm0b3Bs6xnkcuLa2AhG93SwMgYtIhny+EefNcZCVMO59wgOQU7m9VEwuK1NDGEuVQ9e9adNstj0KZxu81jdhFftH7L+8yWPfGT/j4I6mVyQS5UdQjsyntJCpPCFOGSJLiMAe4UY38RhfybLVEYmVN5EQKntloCIa1Z68TEO5urGNPv4RPINNFlTuv2hh7dx+2ajwzIoeNttG5cr5EpMJcQIE7Td6IPcbN99VgjLD5R+alKd953g8ful7t8Yg28zQg1thHKt4WoL7nmXbeYERvBnSOB0aHLDnVIvQYzkNF3LqK66nXckiGMgeVdeiWh2qF/zg0AFvcwkyp691obIk2qgxeY/etVLuw7T0Uu+ULdUub6t2l8I9XxuKRAr2FIKMGFRawzQnilgSGJjdAyg5tIxrETScqrOJnlE7VdyX5+8gKXR 0yxF816g ZCOXayKKiPiRRYpAMZV93spmBc/rpXN0o//LPHOSZdjQC6KDBtswl2mZH5GKTgr4ZySxIGJ+Pm8+bvNgcgjnef72xUwHkwrKNj3kguIdJ19YtQ9YrKZCH16SZy2sAjqV/grfZ01MWvaAos+Q3jIPuvml2t6eUEUt35TWlO4ZZlKDEQEPQem8OUmdBY6hQrq14701FeMwX2JSTVRrFuan6SMwzA3yb3bsBLVrD1KDnxrs5Xrn7xUoMxxdk6SdAkhUbI/E92r745j8uiQGdjnn7TKjrf0YYiWJ0Rn+05J+XVztgh9bSRGsozOW1n4oCly5UqWUJmC7ItTnYnQfPX5cASDPbMDa8X9EualvL1htwKtqvHXmvMh8v8symXm3Hi4diWC427WUM1ntWtZ4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like how cpumask uses nr_cpu_ids to limit the bitmap scanning, make nodemask use nr_node_ids. Since current users expect MAX_NUMNODES as the end-of-bitmap marker, retain this behaviour for now. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Mike Rapoport (IBM) --- include/linux/nodemask.h | 121 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 89 insertions(+), 32 deletions(-) --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -99,6 +99,48 @@ typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t; extern nodemask_t _unused_nodemask_arg_; +#if MAX_NUMNODES > 1 +extern unsigned int nr_node_ids; +#else +#define nr_node_ids 1U +#endif + +/* + * We have several different "preferred sizes" for the nodemask + * operations, depending on operation. + * + * For example, the bitmap scanning and operating operations have + * optimized routines that work for the single-word case, but only when + * the size is constant. So if NR_CPUS fits in one single word, we are + * better off using that small constant, in order to trigger the + * optimized bit finding. That is 'small_nodemask_size'. + * + * The clearing and copying operations will similarly perform better + * with a constant size, but we limit that size arbitrarily to four + * words. We call this 'large_nodemask_size'. + * + * Finally, some operations just want the exact limit, either because + * they set bits or just don't have any faster fixed-sized versions. We + * call this just 'nr_nodemask_bits'. + * + * Note that these optional constants are always guaranteed to be at + * least as big as 'nr_node_ids' itself is, and all our nodemask + * allocations are at least that size (see nodemask_size()). The + * optimization comes from being able to potentially use a compile-time + * constant instead of a run-time generated exact number of CPUs. + */ +#if MAX_NUMNODES <= BITS_PER_LONG + #define small_nodemask_bits ((unsigned int)MAX_NUMNODES) + #define large_nodemask_bits ((unsigned int)MAX_NUMNODES) +#elif MAX_NUMNODES <= 4*BITS_PER_LONG + #define small_nodemask_bits nr_node_ids + #define large_nodemask_bits ((unsigned int)MAX_NUMNODES) +#else + #define small_nodemask_bits nr_node_ids + #define large_nodemask_bits nr_node_ids +#endif +#define nr_nodemask_bits nr_node_ids + /** * nodemask_pr_args - printf args to output a nodemask * @maskp: nodemask to be printed @@ -109,7 +151,7 @@ extern nodemask_t _unused_nodemask_arg_; __nodemask_pr_bits(maskp) static inline unsigned int __nodemask_pr_numnodes(const nodemask_t *m) { - return m ? MAX_NUMNODES : 0; + return m ? nr_nodemask_bits : 0; } static inline const unsigned long *__nodemask_pr_bits(const nodemask_t *m) { @@ -137,13 +179,13 @@ static inline void __node_clear(int node clear_bit(node, dstp->bits); } -#define nodes_setall(dst) __nodes_setall(&(dst), MAX_NUMNODES) +#define nodes_setall(dst) __nodes_setall(&(dst), large_nodemask_bits) static inline void __nodes_setall(nodemask_t *dstp, unsigned int nbits) { bitmap_fill(dstp->bits, nbits); } -#define nodes_clear(dst) __nodes_clear(&(dst), MAX_NUMNODES) +#define nodes_clear(dst) __nodes_clear(&(dst), large_nodemask_bits) static inline void __nodes_clear(nodemask_t *dstp, unsigned int nbits) { bitmap_zero(dstp->bits, nbits); @@ -160,7 +202,7 @@ static inline bool __node_test_and_set(i } #define nodes_and(dst, src1, src2) \ - __nodes_and(&(dst), &(src1), &(src2), MAX_NUMNODES) + __nodes_and(&(dst), &(src1), &(src2), small_nodemask_bits) static inline void __nodes_and(nodemask_t *dstp, const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { @@ -168,7 +210,7 @@ static inline void __nodes_and(nodemask_ } #define nodes_or(dst, src1, src2) \ - __nodes_or(&(dst), &(src1), &(src2), MAX_NUMNODES) + __nodes_or(&(dst), &(src1), &(src2), small_nodemask_bits) static inline void __nodes_or(nodemask_t *dstp, const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { @@ -176,7 +218,7 @@ static inline void __nodes_or(nodemask_t } #define nodes_xor(dst, src1, src2) \ - __nodes_xor(&(dst), &(src1), &(src2), MAX_NUMNODES) + __nodes_xor(&(dst), &(src1), &(src2), small_nodemask_bits) static inline void __nodes_xor(nodemask_t *dstp, const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { @@ -184,7 +226,7 @@ static inline void __nodes_xor(nodemask_ } #define nodes_andnot(dst, src1, src2) \ - __nodes_andnot(&(dst), &(src1), &(src2), MAX_NUMNODES) + __nodes_andnot(&(dst), &(src1), &(src2), small_nodemask_bits) static inline void __nodes_andnot(nodemask_t *dstp, const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { @@ -192,7 +234,7 @@ static inline void __nodes_andnot(nodema } #define nodes_complement(dst, src) \ - __nodes_complement(&(dst), &(src), MAX_NUMNODES) + __nodes_complement(&(dst), &(src), small_nodemask_bits) static inline void __nodes_complement(nodemask_t *dstp, const nodemask_t *srcp, unsigned int nbits) { @@ -200,7 +242,7 @@ static inline void __nodes_complement(no } #define nodes_equal(src1, src2) \ - __nodes_equal(&(src1), &(src2), MAX_NUMNODES) + __nodes_equal(&(src1), &(src2), small_nodemask_bits) static inline bool __nodes_equal(const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { @@ -208,7 +250,7 @@ static inline bool __nodes_equal(const n } #define nodes_intersects(src1, src2) \ - __nodes_intersects(&(src1), &(src2), MAX_NUMNODES) + __nodes_intersects(&(src1), &(src2), small_nodemask_bits) static inline bool __nodes_intersects(const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { @@ -216,33 +258,33 @@ static inline bool __nodes_intersects(co } #define nodes_subset(src1, src2) \ - __nodes_subset(&(src1), &(src2), MAX_NUMNODES) + __nodes_subset(&(src1), &(src2), small_nodemask_bits) static inline bool __nodes_subset(const nodemask_t *src1p, const nodemask_t *src2p, unsigned int nbits) { return bitmap_subset(src1p->bits, src2p->bits, nbits); } -#define nodes_empty(src) __nodes_empty(&(src), MAX_NUMNODES) +#define nodes_empty(src) __nodes_empty(&(src), small_nodemask_bits) static inline bool __nodes_empty(const nodemask_t *srcp, unsigned int nbits) { return bitmap_empty(srcp->bits, nbits); } -#define nodes_full(nodemask) __nodes_full(&(nodemask), MAX_NUMNODES) +#define nodes_full(nodemask) __nodes_full(&(nodemask), small_nodemask_bits) static inline bool __nodes_full(const nodemask_t *srcp, unsigned int nbits) { return bitmap_full(srcp->bits, nbits); } -#define nodes_weight(nodemask) __nodes_weight(&(nodemask), MAX_NUMNODES) +#define nodes_weight(nodemask) __nodes_weight(&(nodemask), small_nodemask_bits) static inline int __nodes_weight(const nodemask_t *srcp, unsigned int nbits) { return bitmap_weight(srcp->bits, nbits); } #define nodes_shift_right(dst, src, n) \ - __nodes_shift_right(&(dst), &(src), (n), MAX_NUMNODES) + __nodes_shift_right(&(dst), &(src), (n), small_nodemask_bits) static inline void __nodes_shift_right(nodemask_t *dstp, const nodemask_t *srcp, int n, int nbits) { @@ -250,26 +292,38 @@ static inline void __nodes_shift_right(n } #define nodes_shift_left(dst, src, n) \ - __nodes_shift_left(&(dst), &(src), (n), MAX_NUMNODES) + __nodes_shift_left(&(dst), &(src), (n), small_nodemask_bits) static inline void __nodes_shift_left(nodemask_t *dstp, const nodemask_t *srcp, int n, int nbits) { bitmap_shift_left(dstp->bits, srcp->bits, n, nbits); } -/* FIXME: better would be to fix all architectures to never return - > MAX_NUMNODES, then the silly min_ts could be dropped. */ +/* + * FIXME: audit tree to move end-of-nodemask to >= nr_nodemask_bits; + * for now, map it to >= MAX_NUMNODES. + */ #define first_node(src) __first_node(&(src)) static inline unsigned int __first_node(const nodemask_t *srcp) { - return min_t(unsigned int, MAX_NUMNODES, find_first_bit(srcp->bits, MAX_NUMNODES)); + unsigned int bit = find_first_bit(srcp->bits, small_nodemask_bits); + + if (bit >= nr_nodemask_bits) + bit = MAX_NUMNODES; + + return bit; } #define next_node(n, src) __next_node((n), &(src)) static inline unsigned int __next_node(int n, const nodemask_t *srcp) { - return min_t(unsigned int, MAX_NUMNODES, find_next_bit(srcp->bits, MAX_NUMNODES, n+1)); + unsigned int bit = find_next_bit(srcp->bits, small_nodemask_bits, n+1); + + if (bit >= nr_nodemask_bits) + bit = MAX_NUMNODES; + + return bit; } /* @@ -283,6 +337,7 @@ static inline unsigned int __next_node_i if (ret == MAX_NUMNODES) ret = __first_node(srcp); + return ret; } @@ -306,8 +361,12 @@ static inline void init_nodemask_of_node #define first_unset_node(mask) __first_unset_node(&(mask)) static inline unsigned int __first_unset_node(const nodemask_t *maskp) { - return min_t(unsigned int, MAX_NUMNODES, - find_first_zero_bit(maskp->bits, MAX_NUMNODES)); + unsigned int bit = find_first_zero_bit(maskp->bits, small_nodemask_bits); + + if (bit >= nr_nodemask_bits) + bit = MAX_NUMNODES; + + return bit; } #define NODE_MASK_LAST_WORD BITMAP_LAST_WORD_MASK(MAX_NUMNODES) @@ -337,21 +396,21 @@ static inline unsigned int __first_unset #define nodes_addr(src) ((src).bits) #define nodemask_parse_user(ubuf, ulen, dst) \ - __nodemask_parse_user((ubuf), (ulen), &(dst), MAX_NUMNODES) + __nodemask_parse_user((ubuf), (ulen), &(dst), nr_nodemask_bits) static inline int __nodemask_parse_user(const char __user *buf, int len, nodemask_t *dstp, int nbits) { return bitmap_parse_user(buf, len, dstp->bits, nbits); } -#define nodelist_parse(buf, dst) __nodelist_parse((buf), &(dst), MAX_NUMNODES) +#define nodelist_parse(buf, dst) __nodelist_parse((buf), &(dst), nr_nodemask_bits) static inline int __nodelist_parse(const char *buf, nodemask_t *dstp, int nbits) { return bitmap_parselist(buf, dstp->bits, nbits); } #define node_remap(oldbit, old, new) \ - __node_remap((oldbit), &(old), &(new), MAX_NUMNODES) + __node_remap((oldbit), &(old), &(new), nr_nodemask_bits) static inline int __node_remap(int oldbit, const nodemask_t *oldp, const nodemask_t *newp, int nbits) { @@ -359,7 +418,7 @@ static inline int __node_remap(int oldbi } #define nodes_remap(dst, src, old, new) \ - __nodes_remap(&(dst), &(src), &(old), &(new), MAX_NUMNODES) + __nodes_remap(&(dst), &(src), &(old), &(new), nr_nodemask_bits) static inline void __nodes_remap(nodemask_t *dstp, const nodemask_t *srcp, const nodemask_t *oldp, const nodemask_t *newp, int nbits) { @@ -367,7 +426,7 @@ static inline void __nodes_remap(nodemas } #define nodes_onto(dst, orig, relmap) \ - __nodes_onto(&(dst), &(orig), &(relmap), MAX_NUMNODES) + __nodes_onto(&(dst), &(orig), &(relmap), nr_nodemask_bits) static inline void __nodes_onto(nodemask_t *dstp, const nodemask_t *origp, const nodemask_t *relmapp, int nbits) { @@ -375,7 +434,7 @@ static inline void __nodes_onto(nodemask } #define nodes_fold(dst, orig, sz) \ - __nodes_fold(&(dst), &(orig), sz, MAX_NUMNODES) + __nodes_fold(&(dst), &(orig), sz, nr_nodemask_bits) static inline void __nodes_fold(nodemask_t *dstp, const nodemask_t *origp, int sz, int nbits) { @@ -385,7 +444,7 @@ static inline void __nodes_fold(nodemask #if MAX_NUMNODES > 1 #define for_each_node_mask(node, mask) \ for ((node) = first_node(mask); \ - (node) < MAX_NUMNODES; \ + (node) < nr_nodemask_bits; \ (node) = next_node((node), (mask))) #else /* MAX_NUMNODES == 1 */ #define for_each_node_mask(node, mask) \ @@ -452,7 +511,6 @@ static inline unsigned int next_memory_n return next_node(nid, node_states[N_MEMORY]); } -extern unsigned int nr_node_ids; extern unsigned int nr_online_nodes; static inline void node_set_online(int nid) @@ -494,7 +552,6 @@ static inline int num_node_state(enum no #define first_memory_node 0 #define next_online_node(nid) (MAX_NUMNODES) #define next_memory_node(nid) (MAX_NUMNODES) -#define nr_node_ids 1U #define nr_online_nodes 1U #define node_set_online(node) node_set_state((node), N_ONLINE) @@ -516,7 +573,7 @@ static inline int node_random(const node bit = first_node(*maskp); break; default: - bit = find_nth_bit(maskp->bits, MAX_NUMNODES, get_random_u32_below(w)); + bit = find_nth_bit(maskp->bits, nr_node_ids, get_random_u32_below(w)); break; } return bit;