From patchwork Thu Mar 7 01:45:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 13584961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87773C5475B for ; Thu, 7 Mar 2024 01:45:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B11006B00CD; Wed, 6 Mar 2024 20:45:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A99C56B00CE; Wed, 6 Mar 2024 20:45:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 939C36B00CF; Wed, 6 Mar 2024 20:45:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7E5CA6B00CD for ; Wed, 6 Mar 2024 20:45:07 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2A01C410A3 for ; Thu, 7 Mar 2024 01:45:07 +0000 (UTC) X-FDA: 81868549854.08.4F96247 Received: from gentwo.org (gentwo.org [62.72.0.81]) by imf29.hostedemail.com (Postfix) with ESMTP id 9EF90120005 for ; Thu, 7 Mar 2024 01:45:05 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none ("invalid DKIM record") header.d=gentwo.org header.s=default header.b=WRye+pJL; spf=pass (imf29.hostedemail.com: domain of cl@gentwo.org designates 62.72.0.81 as permitted sender) smtp.mailfrom=cl@gentwo.org; dmarc=pass (policy=none) header.from=gentwo.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709775905; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=guH4P4vfe1Ds3Rh97BFkr3T7MIyFL/YNkYpLZapfYT8=; b=Y7DtjsBxAUcl1J5+B5/g0boKR0Qcw2kWf4pB8MIgffiVwu40y7no/rkRNnebsH7G6gC4tu /wvalr5FTnINamqMq2g5oGuQncfSvv79V4sUszWVy2K/5vtBw4wb/ct8Wut0bMrBLcq6Fy dQlljZ3C7Pt1KlYbMTp+gjYGLJVzlQ4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709775905; a=rsa-sha256; cv=none; b=CLe0GnRBB5k0j5ErCkIJs0aE6z8IGSp1guylT4t8vXMPPzk7DOE96FmtdKyF24D6tzb+N4 aphRjTaeVNUTUuBVKEnCtT2C/YjCscf3nqLpNH4bkrgIpMDtbGZMiYhzidR3hxCFpk1Ff3 6rX7O6rBeOSk0DohcHKqUCXI6W41UCg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none ("invalid DKIM record") header.d=gentwo.org header.s=default header.b=WRye+pJL; spf=pass (imf29.hostedemail.com: domain of cl@gentwo.org designates 62.72.0.81 as permitted sender) smtp.mailfrom=cl@gentwo.org; dmarc=pass (policy=none) header.from=gentwo.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gentwo.org; s=default; t=1709775904; bh=aWSwUSkgwGQnBYXx/O6wZsg2pJVaxfIK5ZiFu+4pJiM=; h=Date:From:To:cc:Subject:From; b=WRye+pJLR+trpFSDcZ4fJaP+w34FsxMv4jhCYGzeYBguKZj9sMfw4npzfW5R8+vbt ff+pUuLlIOGn02EhmiXQ/u/sL3lpx97bT45tNjeYLGkTxKv82V2Fi3kqErb/hndxF7 y0tOM0LoYWspgM4XpLviFf82ReGKWy1xgMrmBwo0Zy7BTuYlBCEE1s9lYLQHBu4CnW 6E/qupm8E4VN91Y2UBGfOPzeaNqBECudKuo8XTFINnJoaWInraLbRvhJpxZViYel7V hc73NxJOs8nsNGoBxUHjubsZl13uOqQk0ncgN3THtki13nDyjrWfsGMGEv6Qmezpy1 sd3LWJ9sjRWYQ== Received: by gentwo.org (Postfix, from userid 1003) id 3501440A94; Wed, 6 Mar 2024 17:45:04 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by gentwo.org (Postfix) with ESMTP id 30B1240789; Wed, 6 Mar 2024 17:45:04 -0800 (PST) Date: Wed, 6 Mar 2024 17:45:04 -0800 (PST) From: "Christoph Lameter (Ampere)" To: Mark Rutland cc: catalin.marinas@arm.com, Will Deacon , Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com, Valentin.Schneider@arm.com, akpm@linux-foundation.org, anshuman.khandual@arm.com, Eric Mackay , dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux@armlinux.org.uk, robin.murphy@arm.com, vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com Subject: [PATCH v3] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512 Message-ID: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9EF90120005 X-Rspam-User: X-Stat-Signature: 5ayfa37qm3pjeszymnega6cka8z5i1hn X-Rspamd-Server: rspam03 X-HE-Tag: 1709775905-588035 X-HE-Meta: U2FsdGVkX18le6tA4ZciwzPz3CPnZpRlIUdHAE7y3rq2kG9ZRNw1sm2gmh6Qsp65+ICVCfu4bk+ZQcZlRvZ2ieUQIIYn8Lx9bxji1g48g7A2VVJM5lGQhcRpUQwX+8t6fzoeoFgyCJqPowmUfdmo/rpRXUyvh/vBSsYzLB1o5wDCh19SaOnH0l/6yl18CK3SbZbxcESVDWmTCUsLp2Ffxxmevfg95c6L3cyIilLz71en1aBDXVYjHW+b3BNUhwXKjFAI0RVN2ChHrcGcSigOnXL8Z3XbY6mgcFq0sIZerSczCJUgkKUIGcYZai78e1v0FM72sHBGDd2+33A4vPHWYP8bS4EzCmYSNSoYRgN6UITKQYWbtJKt6DYE7IB6z8MYHfE+7UJe9FprykAVCE0sY4/3UTJmQ2J53S0b+5nsAHVgpVyxKhdnx/BeDdNVBkEzUUaCbPcFIQsOLmOhkI2Mvv2D9kSB2ZDIBRscVavvliPYwaJzheMNxRcOWjFHoaWhAozTL7xOZJFrIDQibv0xO3amDfTDp6tr/gRd2tnlHQbiWVU/9nES3DLuRNxCio5xdr0O1G4SYo3Ldn4X/869UhBt/YIVWyiIWkHKjoCkvAkiElAeve3YbXYMjS/lO5K7z5iIRTUD3X+YiAi5qtDvvXmzRCZZQLVupPMwTgtw1JA1gFWDoKua8SINPpSq0RVLD+P5E+awmRlMdYH7dF2sMwS74AttYjwdR2TodI/IO+uUHFekN7MbOZBslmlsX8ZiBj9NHBs0B9WDGtWfADn6cou8xuDtpymaTi5TnkpsCdLhHNVFUOoJ98wyEIu0v28lIlvLV/mvOaEbdaLahg1xgpwoDGpdVkOE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently defconfig selects NR_CPUS=256, but some vendors (e.g. Ampere Computing) are planning to ship systems with 512 CPUs. So that all CPUs on these systems can be used with defconfig, we'd like to bump NR_CPUS to 512. Therefore this patch increases the default NR_CPUS from 256 to 512. As increasing NR_CPUS will increase the size of cpumasks, there's a fear that this might have a significant impact on stack usage due to code which places cpumasks on the stack. To mitigate that concern, we can select CPUMASK_OFFSTACK. As that doesn't seem to be a problem today with NR_CPUS=256, we only select this when NR_CPUS > 256. CPUMASK_OFFSTACK configures the cpumasks in the kernel to be dynamically allocated. This was used in the X86 architecture in the past to enable support for larger CPU configurations up to 8k cpus. With that is becomes possible to dynamically size the allocation of the cpu bitmaps depending on the quantity of processors detected on bootup. Memory used for cpumasks will increase if the kernel is run on a machine with more cores. Further increases may be needed if ARM processor vendors start supporting more processors. Given the current inflationary trends in core counts from multiple processor manufacturers this may occur. There are minor regressions for hackbench. The kernel data size for 512 cpus is smaller with offstack than with onstack. Benchmark results using hackbench average over 10 runs of hackbench -s 512 -l 2000 -g 15 -f 25 -P on Altra 80 Core Support for 256 CPUs on stack. Baseline 7.8564 sec Support for 512 CUs on stack. 7.8713 sec + 0.18% 512 CPUS offstack 7.8916 sec + 0.44% Kernel size comparison: text data filename Difference to onstack256 baseline 25755648 9589248 vmlinuz-6.8.0-rc4-onstack256 25755648 9607680 vmlinuz-6.8.0-rc4-onstack512 +0.19% 25755648 9603584 vmlinuz-6.8.0-rc4-offstack512 +0.14% Tested-by: Eric Mackay Reviewed-by: Russell King (Oracle) Signed-off-by: Christoph Lameter (Ampere) Acked-by: Mark Rutland --- Original post: https://www.spinics.net/lists/linux-mm/msg369701.html V2: https://lkml.org/lkml/2024/2/7/505 V1->V2 - Keep quotation marks - Remove whiltespace damage - Add tested by V2->V3: - Add test results - Rework descriptions arch/arm64/Kconfig | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index aa7c1d435139..4e767dede47d 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1427,7 +1427,21 @@ config SCHED_SMT config NR_CPUS int "Maximum number of CPUs (2-4096)" range 2 4096 - default "256" + default "512" + +# +# Determines the placement of cpumasks. +# +# With CPUMASK_OFFSTACK the cpumasks are dynamically allocated. +# Useful for machines with lots of core because it avoids increasing +# the size of many of the data structures in the kernel. +# +# If this is off then the cpumasks have a static sizes and are +# embedded within data structures. +# + config CPUMASK_OFFSTACK + def_bool y + depends on NR_CPUS > 256 config HOTPLUG_CPU bool "Support for hot-pluggable CPUs"