From patchwork Tue Mar 4 22:19:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D45C3C021B8 for ; Tue, 4 Mar 2025 22:24:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version: Content-Transfer-Encoding:Content-Type:References:In-Reply-To:Message-ID:Date :Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vdST2mWAb0UFHFUtZG5BrAT83pl4Qk9noVNXHU9QDvI=; b=nDqWJtM8Ju84wFgwhx05jS4lNB 5rccvbJ6k2JzVx2LmGsvtDzq8k/js6mOo6T+FbOUmkks3o6Hv1v/SCwS7BI4/ejDnqpQNgQeSV9ND +8/hnZqXiz46kjXQazM7WDNccqht0mmA8aUZHjhj01LB/Wpj3HzxqY8zbgc2bTJrNe9EVHdCzpQ5+ 2dqT5D3lVPl6F7qzJpLYXmIo6PtQhWGDXwenxID8JJ59OQRr6YLv5O69DBgGM9+eLd+7cFOf2jAwN wHU6a4epfIP6SGRGsJiLmTy5OZIyNt/U4sUjCj4r6KxuEvE1eNfECSLYyo8TxCHmt9YzyEvhV/jCS 73YUqGAw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpagF-00000006Oli-3hIz; Tue, 04 Mar 2025 22:24:03 +0000 Received: from mail-eastus2azlp170100001.outbound.protection.outlook.com ([2a01:111:f403:c110::1] helo=BN1PR04CU002.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadA-00000006OM9-0l66 for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:20:53 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=vI0qgrH5KiaPoIUA7VQG1YjmZVudbc743MY0hAA7dD0Mae5BhP1y8Pyzyx8ihg/FJbpmFUvNx3AK2npI/YtOpN4yHeWmhzr/6yyvuCIOl6ZN8S0hdJf+qmlo1ArOCF176qXc0qXXezZ8ENqj5Wo1Mj057/W5EtBYSeuCf4O5YSTi2wRaDXDYg86KHVakn39qY7/+QBUFwlDM0uCk3OzQ8joyB8bwM9TDOsTPC6tVI8kkQHiXZinGNUi5wMsXaSwWob8SjEzwDFwthYsfPDsnpCmK1G0mjc+zaK80FLwrZXMZBKMGejcsCB2sd+fnFBVW1IjhkfATcXIzjLUnVcqZbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vdST2mWAb0UFHFUtZG5BrAT83pl4Qk9noVNXHU9QDvI=; b=Uv8l4vyWcbZOqQ0wLsqlg9Fcy5Vp2lIV09WOhCX75L5Q1hAGCOq4Gqif9uI7WIHTAS5wpITAlFy7LaDc2S33U4k5ftmM6dHseZzOgJUuxLbF21GdQYi32I1m3XPqbPsc8TfUEnoCCY2bcDxGiiqv7Br/5cV2w/GOG3rspDrQ7W99iHqFFPqlCLZtLgH6COwkcZD6DgaWLg3ckg/gfRGl71vxkRWgEFuoQpxZPaPu6bZ+4U/E8mBOlws2BGMw7IYw66sjdQdrnRgb6yoTNQKKFxxjtHGB/9gsdSeDqn33HCR8FWuUj8omOE6TCdz1zYeVBEqMENjT16VNkqUjAYmX7w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vdST2mWAb0UFHFUtZG5BrAT83pl4Qk9noVNXHU9QDvI=; b=qZExP4dK/lPGygJlgMBCcpLzDwP/6eU1WuVxF3GpCc9hhanD13zOkv57JOXv3ZcZJL/xEZF4f2WTJq9lP6x/KW0Vk9QIXpGGswe5tVdkzxGSps6M/g5PTJxbrbDsrtt/5xMPEqauIPMG8As23SQI2bWwgnRbJXuftlU9PNfZ3S4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by PH7PR01MB7931.prod.exchangelabs.com (2603:10b6:510:275::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.26; Tue, 4 Mar 2025 22:20:46 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:46 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 1/6] arm64: Add BBM Level 2 cpu feature Date: Tue, 4 Mar 2025 14:19:26 -0800 Message-ID: <20250304222018.615808-2-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|PH7PR01MB7931:EE_ X-MS-Office365-Filtering-Correlation-Id: 0b6305a3-9871-4444-8218-08dd5b6acf50 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|52116014|38350700014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?/uQhG2GqoJnbMtnU3XVhjbxLT6bgCv0?= =?utf-8?q?pJuV+TZiGBYjNIcb+oUWohBaE6NiCs4vmXnKf+5fDYhHqpq8wFrwNdIlBwB1M5nOE?= =?utf-8?q?LR4k6n/QmxzehAh7hIy+age7RbBh5iXzVhDbC1N6o5Eai8kYGDwB3aP6vbx0ohWPy?= =?utf-8?q?RR+Py2E3e+yeIzO+yE++YMVZGCVJwebDgNusZE7rzCe8Ex7kNwlkMVXz4G7OR67vc?= =?utf-8?q?Vxqu/pUvh0dO5k+KzKdIQFOvmLJAvQNMtyQYMKkvKtDzgdIqUGLPR+8o1Pue/7uU1?= =?utf-8?q?xtO00ZV4xg5hvlS66v82EAwMWg1QgnBCtimLNCRCPoKsIX51euBghPtW4d+ErmrTy?= =?utf-8?q?caciAwe85jge3ZSBmFn1mmheduraAJRizLl5YUiX+2vvLED0CZPjPT3ePt9T4bfnU?= =?utf-8?q?0xnreKU0g7sdaUyY3geVxmFrzCruh+pd2S78P6YlPV8yI8CxMo1Buz8Dd1Qs+jy70?= =?utf-8?q?M3u3SlpyCyCFJsPX1WB02a/t1L88POSYyx3sDrGX+g1G6zSs3gt1u7W03CTnXdhoB?= =?utf-8?q?+mmmgObJN3uxAVVM2hs1sInIT3UmSS9vQh+pJVYrKEhD43RHQIfHT2TIukkmaajxB?= =?utf-8?q?wo6fOkx9CPQIw0emg1L33p+UcoMiz1EhBNFpvP2vFR1xw9N8MSUynsLBODcOeFpR2?= =?utf-8?q?9YHn5vOl6/HsL5tlo4qLGwQI/deDMpszzhVCPDDGTQmUOcC3lkiUzD5+YttHgw4aI?= =?utf-8?q?OqXGkqKaGZ/b5Rvile68kr4e9TLBFpFZc8lFKPvlMj7NrV8c3yzSxwkVM6GaWJARe?= =?utf-8?q?5DO1OFvwbsZT+og36W3N/+5hPOWjDmnrWl7tx3ul0BJ3A3FfW4o6f22a00JrlEVxn?= =?utf-8?q?/XV9TW5fILpQp+O24d3DN8k3JPompUSKDYTBdWgRBbwkKGOgN5W26WklXXKWZB5RK?= =?utf-8?q?5RXdbxCNJ5bSDefG8Yck8evoTSRP1zlwmAEEWcSorZLpkd1M9qbrJdF2ZsUmix56+?= =?utf-8?q?l/5iQ2hJnlrTVF4Fdm+JegyV3CrlqcP1dw+lqXOTIyfdeikN67BeCFRIDi2t5WbWr?= =?utf-8?q?wDqMq2//Wy+HTnekJwC3iig9Bsqrqq83YyZ6lf+NFSB7JDXQO/4q6JTA1lvIjfDKI?= =?utf-8?q?v8ntPBjMj6MGpCs/Bv9A7RMikYcWV7y45Buyg4kUxTSrkaOpoTIJSBZ7BJTfoZ/u+?= =?utf-8?q?KBwFKw1x/nVjY5TaAaHrg7Eh3LaU4EHQSVB4TiQ7UXEZ64stg7cc3ZQlS5CdOnuM9?= =?utf-8?q?phczPDE6A8LILu33IoOG8bpRX57kDtqFwRS0ZdSn3mwEVSEJ7L1cy+G5HYWYQFfac?= =?utf-8?q?J55h0Vv/o2T9SUoKo65f9reDOalspbRFuKmRXYbYjQAg5TwUufoYc+/w67/P8dH0h?= =?utf-8?q?Qk/cJeEmxODB54QSUgyMo2EB2XHAQwTcbzlmEL0F9A8ZjA3FdjbOq9anIlSpXPG28?= =?utf-8?q?NaCTQdGDh5j?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?uXw0NI2IpEjLZ0nzCqEWt2p/27k8?= =?utf-8?q?i+a9YVc7rpcVqT3v3t6WbzL20jPjj+IXyThZjRTIQDEjs/5Afvqr8NJgm1r7R/jRF?= =?utf-8?q?g+R+n1n3Vx90tvK1UrkpEWpsHCSwA+idTuWx9VPeHTQ2NR28imKmCOGeRvbn74PU4?= =?utf-8?q?SEU6dpziFeSNR0+gD2o0DXBqtLl9e/6wng7Pr2k1mhG2k1WO34r1GNPeu30/HWeNs?= =?utf-8?q?sVb8utFkAJkHlaeLxHT+xDi+ai4sRBjXpGbHlId1XvtXGzwTzpdtxuVSMYuGZ0UNM?= =?utf-8?q?ue2rwSL7hxi0mUG33qfF6XauwrDogtyjkQzT8bgkrbPgx1NCsfobO+qaSnG8jl1/Z?= =?utf-8?q?1aHqlcpUn4jlQo9ge7reo+0UrNptykggwp/kahtVIMDxMWXfUurCHsZLXHz+praDH?= =?utf-8?q?4Gt++P2JcFxZdZvYAY7mWttU2rG9KVV0VMVmyVR14qipRKh0xrgKFRftBYxgmew5O?= =?utf-8?q?dNvyIX5K8/mWijATHZoJGJ4eNcSVwdendbjpQv9g9jexSdl8em0bA+C+c9JJn+YIj?= =?utf-8?q?W7Ss2+iAxBayObbqCcR5Y5KjKKpyD/sK05UUU722+JQINNICejDRkj+kk4DaOkB3r?= =?utf-8?q?ZjmKyhQGjt+Mz2UWOMviTAwiqyUYE45xqRrspyZXlHJAiFBpotPK1iuTEhp6e0cgS?= =?utf-8?q?U87MHeXzVCto5hVVXhurl6VlQ74unx2MDhORADJo5RxCMZJgMy2jazqUrMKB/l6Zj?= =?utf-8?q?NNTdZxCbZ0un3Pg37iEIVMGzOXWUA40VaJnI57h3wQIdIND1pNPdv2wefN6JoYj3n?= =?utf-8?q?4qXVlYOTnA2KsyCPl5vUFAqq5es+jz7+izP75Gic5OfIO/gL8Anj3VYRphXVFNM/K?= =?utf-8?q?RIsFYGGWBlytFqGC6VB5HV4YQ65NaNGNFVP4669goqFRZE+ZXPAkME29+HB5oCNQE?= =?utf-8?q?W/t6QbsQK1q7y94U/v9GkUW+O5l4gGpgRNft1wc8afk8gjNk1kdXkAs4vykAAwJbN?= =?utf-8?q?SdW1UtNYTJYnsEBFIEWsk9LLKY8s2ywCauLr7s3fuS+aROHoTGn6XJD1f4n2sAikM?= =?utf-8?q?KC5fcVi4cIGxQa4QDPzmf2TY09T9V0W2B1hXmQ6HK5D7QCYUvngK8gWFpuyuTnWUk?= =?utf-8?q?r5pbi01T9sgKCdTXPwWSe2O4jzHbahgtbHoEYC1Pj0akH1KhO3oJlALFmuBESdj12?= =?utf-8?q?kQobgS0gX2FuVQ7wE/YqzoquFApD49UymdGaYAd8ONLksjn4h/eJXdywJXyfP2M/q?= =?utf-8?q?EoZYxFi7e6BAQaSpDYyId6VPn8sRZzRFHpDgPmvflG2NrQ5YBElGt0iGLX0SxKAYY?= =?utf-8?q?7wkYFX4ZPGzCe4kWKovGCIkBZttT3wNQaDMve5q/lYUglqQulTNLiN0M9tw2p1UEd?= =?utf-8?q?ZXSNK2Ovjr2Me+po3IrXh6Hn9COB9GuBs0ReoKDYiQyPDlsjRI1tmveetSDpSN/d+?= =?utf-8?q?roU6Iw7bwZHwvWsx1jN+w2Kp4pr2iLmEzZ+b1lx8ft0dj1rbjwV98yXJkhvAzixzk?= =?utf-8?q?IL31rNSxNyBfvZxoe8eeB24dnix97guLk0Mrg+/0BIdA7a5+cluY8Y8+Ij385zvX1?= =?utf-8?q?kBGIvSQSrgoXhwapQCLgICXHMKB+j6so+n6PfWk32jXW535VR5sZDa4=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0b6305a3-9871-4444-8218-08dd5b6acf50 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:46.0830 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4oGf5/5jnhAfPugm64dteAeDq0QS0AOpthggg+u9zIIHMq+2AP/fbVzBFWVHODo/M9DcTLlM2oMMVRqYu6+6/pqF9BU3R1BzEXL1Tq7fFsA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7931 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142052_297405_2002ADF0 X-CRM114-Status: GOOD ( 21.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Mikołaj Lenczewski The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for. This is a system feature as we might have a big.LITTLE architecture where some cores support BBML2 and some don't, but we want all cores to be available and BBM to default to level 0 (as opposed to having cores without BBML2 not coming online). To support BBML2 in as wide a range of contexts as we can, we want not only the architectural guarantees that BBML2 makes, but additionally want BBML2 to not create TLB conflict aborts. Not causing aborts avoids us having to prove that no recursive faults can be induced in any path that uses BBML2, allowing its use for arbitrary kernel mappings. Support detection of such CPUs. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/Kconfig | 11 +++++ arch/arm64/include/asm/cpucaps.h | 2 + arch/arm64/include/asm/cpufeature.h | 5 +++ arch/arm64/kernel/cpufeature.c | 69 +++++++++++++++++++++++++++++ arch/arm64/tools/cpucaps | 1 + 5 files changed, 88 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 940343beb3d4..49deda2b22ae 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2057,6 +2057,17 @@ config ARM64_TLB_RANGE The feature introduces new assembly instructions, and they were support when binutils >= 2.30. +config ARM64_BBML2_NOABORT + bool "Enable support for Break-Before-Make Level 2 detection and usage" + default y + help + FEAT_BBM provides detection of support levels for break-before-make + sequences. If BBM level 2 is supported, some TLB maintenance requirements + can be relaxed to improve performance. We additonally require the + property that the implementation cannot ever raise TLB Conflict Aborts. + Selecting N causes the kernel to fallback to BBM level 0 behaviour + even if the system supports BBM level 2. + endmenu # "ARMv8.4 architectural features" menu "ARMv8.5 architectural features" diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 0b5ca6e0eb09..2d6db33d4e45 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -23,6 +23,8 @@ cpucap_is_possible(const unsigned int cap) return IS_ENABLED(CONFIG_ARM64_PAN); case ARM64_HAS_EPAN: return IS_ENABLED(CONFIG_ARM64_EPAN); + case ARM64_HAS_BBML2_NOABORT: + return IS_ENABLED(CONFIG_ARM64_BBML2_NOABORT); case ARM64_SVE: return IS_ENABLED(CONFIG_ARM64_SVE); case ARM64_SME: diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index e0e4478f5fb5..108ef3fbbc00 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -866,6 +866,11 @@ static __always_inline bool system_supports_mpam_hcr(void) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } +static inline bool system_supports_bbml2_noabort(void) +{ + return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d561cf3b8ac7..7934c6dd493e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2176,6 +2176,68 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } +static bool cpu_has_bbml2_noabort(unsigned int cpu_midr) +{ + /* We want to allow usage of bbml2 in as wide a range of kernel contexts + * as possible. This list is therefore an allow-list of known-good + * implementations that both support bbml2 and additionally, fulfill the + * extra constraint of never generating TLB conflict aborts when using + * the relaxed bbml2 semantics (such aborts make use of bbml2 in certain + * kernel contexts difficult to prove safe against recursive aborts). + * + * Note that implementations can only be considered "known-good" if their + * implementors attest to the fact that the implementation never raises + * TLBI conflict aborts for bbml2 mapping granularity changes. + */ + static const struct midr_range supports_bbml2_noabort_list[] = { + MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), + MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), + {} + }; + + return is_midr_in_range_list(cpu_midr, supports_bbml2_noabort_list); +} + +static inline unsigned int __cpu_read_midr(int cpu) +{ + WARN_ON_ONCE(!cpu_online(cpu)); + + return per_cpu(cpu_data, cpu).reg_midr; +} + +static bool has_bbml2_noabort(const struct arm64_cpu_capabilities *caps, int scope) +{ + if (!IS_ENABLED(CONFIG_ARM64_BBML2_NOABORT)) + return false; + + if (scope & SCOPE_SYSTEM) { + int cpu; + + /* We are a boot CPU, and must verify that all enumerated boot + * CPUs have MIDR values within our allowlist. Otherwise, we do + * not allow the BBML2 feature to avoid potential faults when + * the insufficient CPUs access memory regions using BBML2 + * semantics. + */ + for_each_online_cpu(cpu) { + if (!cpu_has_bbml2_noabort(__cpu_read_midr(cpu))) + return false; + } + + return true; + } else if (scope & SCOPE_LOCAL_CPU) { + /* We are a hot-plugged CPU, so only need to check our MIDR. + * If we have the correct MIDR, but the kernel booted on an + * insufficient CPU, we will not use BBML2 (this is safe). If + * we have an incorrect MIDR, but the kernel booted on a + * sufficient CPU, we will not bring up this CPU. + */ + return cpu_has_bbml2_noabort(read_cpuid_id()); + } + + return false; +} + #ifdef CONFIG_ARM64_PAN static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused) { @@ -2926,6 +2988,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, + { + .desc = "BBM Level 2 without conflict abort", + .capability = ARM64_HAS_BBML2_NOABORT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_bbml2_noabort, + ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) + }, { .desc = "52-bit Virtual Addressing for KVM (LPA2)", .capability = ARM64_HAS_LPA2, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 1e65f2fb45bd..b03a375e5507 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -14,6 +14,7 @@ HAS_ADDRESS_AUTH_ARCH_QARMA5 HAS_ADDRESS_AUTH_IMP_DEF HAS_AMU_EXTN HAS_ARMv8_4_TTL +HAS_BBML2_NOABORT HAS_CACHE_DIC HAS_CACHE_IDC HAS_CNP From patchwork Tue Mar 4 22:19:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B14A3C282D0 for ; Tue, 4 Mar 2025 22:25:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=+Ch3mUlEhrZc5Yn/m5R1uPyYkZxYLTxGAckLc2kMhPI=; b=PAa8tKX3dqRGjhuNHwXONQKEtg XZ8B4wms9kBLYXGTG+ioc9647ZlOSuoC3xU+nWqYwQBna9Ki8VvgOuSx7Jvxo/tVj3n38fPT9F+6m BzzNjVzUoNEXM6ZWkwieVy29prRV5jvTqHTv+LPxg6zfGcQbJ0M0AZ0SLsUW3himE/unLTmCDA/1H rVeOFTH9y9ZP+lbBZ22+BjsKg4/FacSMO0G1InmgAq5UjlUZjKNB0Wforqygu0yUOon3BxEF+2ciC 0Pl/Lva1xM5YZRu3F6PDPpKiz1in34uRIBY+/wr2HAslIjvLQN8cU/EzAYxtbqtEm/pQzGiNNnhMX wdrHPwXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpahm-00000006P66-26aq; Tue, 04 Mar 2025 22:25:38 +0000 Received: from mail-eastus2azlp170100001.outbound.protection.outlook.com ([2a01:111:f403:c110::1] helo=BN1PR04CU002.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadB-00000006OM9-2pd9 for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:20:54 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=wgexTAPtA7vfearZRRJlhds5VL111VtFTlNHDTvkyIzV/02TjdrdYWudKzR4qf/jd2mzvBZDYbg9YUlL6j8hiLd7MbwN16CIqV4h9+CC89F7Iju9482gaizaJC+rcejnZMgaYvZA2dvEyMl4Eb+dvrPcyn3kAVdYtOYlJSUGVMhSyK/IFFzmN4ooJzQkpi8jwGFajyO2wqVzQgLxBrlbbpoH0U7NumBA7gomsde1tr9wq2K/89PD968Lwp5zUkOLN0anVssPOdJMqb7Jwx5LAolNpwIQnji2SwQh2vPNUpjnrHmcHf68nJL7OgGAhPiZTWSUa5eYjp5a5oLxUSWVrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+Ch3mUlEhrZc5Yn/m5R1uPyYkZxYLTxGAckLc2kMhPI=; b=dQ9skztfXau9c13zvxX/mkrC0oXQsT4/DfphVq1sRU99kWe31JTB+8yoNbwLI08Cyjk7B8+Ky5bkAhWF09gfv6bl2eIDMmYJ9SHpvVCkaJ7WS7yZiT1xm+HqVmn5W2MuPNBT0ki5vXQF1h3bxcK/s/bxL+pDyZhRuPhHD1ZCEzj298+wjFqTchnv7aX68dB5Zpom5gUQmCwDozDouZQKFYm/C6wueL68uDvjpnc0vVdtP8hvS45m4CM3v/dDDCkBgwTB7RtYS0XWMHlFp0RrcTYJQkccW+iJSK6tQPCB8+dfnS41sDpsJ4y5Wj9iC8tVEajQX4EiQqvS106iHWCazw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+Ch3mUlEhrZc5Yn/m5R1uPyYkZxYLTxGAckLc2kMhPI=; b=rqjhM64OvqqYf836H05YwT+3c8huiskeXUVapSRTiQQ+JGxLHMviZoYhZgqmmjcRmOs9XrqO3IfAE6t72cwYsVb+Zx8nL8Aa6Wt/iJ3IjWAI2wVNTQE2vGpnE/5Tz/GtV65Evpbr9qsByh6zbDF5ytDcttdZQ4TOxTOS0g0rA5E= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by PH7PR01MB7931.prod.exchangelabs.com (2603:10b6:510:275::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.26; Tue, 4 Mar 2025 22:20:47 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:47 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 2/6] arm64: cpufeature: add AmpereOne to BBML2 allow list Date: Tue, 4 Mar 2025 14:19:27 -0800 Message-ID: <20250304222018.615808-3-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|PH7PR01MB7931:EE_ X-MS-Office365-Filtering-Correlation-Id: ebec83d3-1ef0-47e2-74ab-08dd5b6ad009 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|52116014|38350700014; X-Microsoft-Antispam-Message-Info: YhHLX5zZ8FWz2+z+hDWDQioPgTXmeATVOM4JuHF+UB788ORau5VVnTvOAxi93Zb8Y/BLWOu537voLGzWcLMENUOYlLGj15vVfCCn/0gn4tE/O7cu6x/vWS4wIqNXoSThXDmrFKsSOmmEE+HA9xJLavmK8M/gtiPEdLdpwEvVXlxsIiiSFd7UrlfBytojl5pTZv3PPNkNddtFEle9NHNryXGS1q/rgTgEuFDKXzu+wAJzLs6WU2XGIdKPidWKuAPUMUDEvAB58dYlZzSNaq2YHSLx9lq6X1GvUVXvgjEJl8qaFyr4SyB7Gt7XBAR4UVu4gwolbUK7okblvuOY+9JMqLbGAYWIlyGIfd8EtNt1mcLJLMFAr0cqR5Ft715luUIZ4++as9dpKuL4wgj5YzQsDhwXJJID41CgY+ZcY8SFaEE/wgLimdtF78LrugyEZCvCIiB7aIDtA2LRjm2SQynWiGKKNqL4H0DB7nMACsucq1uRAtDxh+XWQFpF6PEfM4y5EOLQF6Sq21olBUNrFsdNar6qNwzkffzAE6n0IHBsekLF/vAmGdBGICwi6Hw4GdF0nJQEtX/2D4LXvu2WHGzYlR5nqC9qSSVgO8FNyyodCv4IkkRQ8Zgh9by0rF2/vi5OEsNumNhIHn0beZSzCMEG6wv8hOhW9SExlwKD5Lc3jiKmgknzm99DS2f8i9dxRL3Jc58LABhjBMyFQtprqdhqOr5ToIffPwvRBGKq9kWhHOIwl4ZFmLs3qK4YUPROEmQvaieDmUGKR7l6L5QorVFAIu+0/ByIgQg5NZaR0lpMJ6YUkGn0XF/ur5Ot/kyqw0MfjTZqL7Wg/hnTaZO8Fq2BD8vBAsbYds4/P+oOX8lE6UJ0lWeIuwIQAHKkXp7f0bGBlcduP2YfGwAwHgEfkUG7t21oyZv8naq97ZEJoW1/HvBwVLxvXrSTa52BLLzfFJMEOa+hQfng6PsU+epKBuUf69g78V8BeTDnxXkj1RlCYaHYpKaia9nGjWb7+HnRNp61+pjot3Mk3rh38Y9Lm3SLR67l6teSF6ue+g3mYU6fRCtdEBwJtqwkd/bf2DN8yrymp9AfeVZdxNTNVp02QcUSneFWBG5YxQvIvUTtNcba/bLF5neVC7WuH5dA73+OMNrLZFVeTCquGLMA6Amb20TVYm6uKA8n6bfuc/IoLB7GPlbNwbie6NWw7+vQ3nn7hxND4Zmu5p9l4mtZXtUe3TSb43O+T7feys2IPbryYb+MeWO5A/GRp9vic2VtVCtaghDAoDrB0Jdk8lPHaiHgXi+uVXgHbrIBIXzVSqml3xbSQmZ98ldVMbr88INSln/wpULIf9RCYOUtTOOdDnO2mnmJYT1fCaKNXqWO0+CNI/ij5VhOaQJQD1iScfbhJ8YKd4acigNLH5VSwcefGqEWA+DQOqVKHjm7hdDvwu+9HlF0ETwKBKJK1J8Q+UUgyGYKQ112 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UuAIeapEuzDQJ4tqlkEcQS0n/qds0eiWk6wTeP0sbGbsBgVoW8HRxizr0xDnfqZtKQCMJFgdtY0lE1tIk9OQ+L8E6M37B9DHng7ddXCtKtEO+qK5huVoRxhQDTMSt/tqtkr6/ecb4HV9ywbkvZIbmZ3WQt7KEt8o0ATo5OLaO8KBOyG/IW2c6wi8KHnvbAhjVnf6tCKLiqCXP7+gjXEcyTM9QnslqCSGheTzBx/qX1dj+GGuaCyhn0ZFBH1DjFz9tWlSC62AuM8HxsxMMXjp1N+11eN+5H+ONW6YGKwr/FG39Ikn5CtjzDcmowCpJPYDmlBharSqkiWwFXX26DOLUlR28W2AvfiPpsDvS9+8EuqDiO3hAx212yt2ooRL/EDyjsNd+PHcTIsM6nigeLA3Cd+4YepDId+FJeZl85PBIkgW0E0NLNOCF2IDxyISzuZiJo2eLAVoGuhyRS7XXSzwsTfcrvDM0Wc/dPRkPTMEDm5ifSmhgVgfP6g7e5m5k9erG1Gkgoes95ULs+/ralavUYgTZhfAM2R8XerYYwQCGT61zxgb19xigWjUS1mfhZMN91IDilm+TpClyAWFHwiQKKqsadeIUgGEfIT/TvhP10sCZglC0PIAFVJjAB+W2zKYiF59HDzv4dZAYyg7X6REkx27O2I4qmWhMELYrEg57Bm/QgEtyFMZRK5T/fOE0d0quvn8FG08VPN7ALe6+5L/eK0+QWg0pwRHcdv/byXyfwbngPlloyfMeqzR0jV7ksiXMfHa54EWQJhECZ7eHddeBhEnmiR3GPEZg2/FAfwJHKPXuWVD02lLmJGPacFlc/eNgnN8KxLJo885wzn6dbgx9DTRjTStqzmuSw+hTq4LucGcGTvyT1vvXH9Zz24+szxfrXnpwrtvAaRqe/sHm0dPwkQ6PhI9Gc2ZSVkFmVokTu6tY6eS0lYZ5FqwlggkSLUzxWi5rXZLdAw57kG9EsumdYt9FPr7na9k06gWmnE6Qp163i/4/ge8aVRqpyxlQ2eNsRMz/PhrsfISGvJgY7IX3PisscnpX5fdrL6bHeDKUnLqCx4E5qkrFSAdE2IoAaTMWVxoWD1pt6XeuBqSChnfEPLC2uc53IwJTZbEHzBd/EsWYLxMmuXgJDyWouCnpc2plkFsOh8lLmCLxTiwOgIkyAEgdNs3ZFL1JD6TOBkGxhcbT6piw7gP23jJWxHF8bowds8GF1tthSOQM5NRBLrboC6vNKyBZnHw0X0nHQAkCz18/fhA3r/sRFgAwlvi6vOOc2tdtPmZ8cUePyPO0tPnYn30aJ57LakuQQUZBtd4fdYAX9HRY1gLvCeqxah8h0NpT0zl++S5IkiIyoZsbccmcZ0P0hpHZT95a2pbYXeahgEXNe4aMr959+wdQt4yikHT/TirQvU9r2qsGYsNDyZzSiIXt6nrn0wJxf0cWTkLW68uMLwpT9Obg9oR3MPMmDCBcNQM7QzImiyqavaztnIXWqp5Htlg9mEc0PIYghXMG1t3joCdNeaQFjAF6kklNH23OL9IMK+9ja/yiN4xuWin6JuVG/75QH8WQEWWKyUCWaeViwYxDYWsBTdv+8MAIgP9hup6xUYn19KH2U4fHfYpLw9+PyAIVPz8e0SM6MQyHTc= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: ebec83d3-1ef0-47e2-74ab-08dd5b6ad009 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:47.3487 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ir3LDNr8RtBM7auA6sVxkvrTcXE4aM2Ye66L7PgvLyj7g4L0Nf6NN2YYnyFsS53TiZZw1Zp+8qLzgIgzH48xzT75umpcn5M7cTCSZaeEVUo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7931 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142053_712520_6CE80D39 X-CRM114-Status: UNSURE ( 9.20 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org AmpereOne supports BBML2 without conflict abort, add to the allow list. Signed-off-by: Yang Shi --- arch/arm64/kernel/cpufeature.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7934c6dd493e..bf3df8407ca3 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2192,6 +2192,8 @@ static bool cpu_has_bbml2_noabort(unsigned int cpu_midr) static const struct midr_range supports_bbml2_noabort_list[] = { MIDR_REV_RANGE(MIDR_CORTEX_X4, 0, 3, 0xf), MIDR_REV_RANGE(MIDR_NEOVERSE_V3, 0, 2, 0xf), + MIDR_ALL_VERSIONS(MIDR_AMPERE1), + MIDR_ALL_VERSIONS(MIDR_AMPERE1A), {} }; From patchwork Tue Mar 4 22:19:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A406C021B8 for ; Tue, 4 Mar 2025 23:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bc8S2OTbZME42pyY8V87xNGR1wOTsn5d4wbfYh2jm/s=; b=E+ESfVojlUxoheH2TpH98mFiXd BFHJk7Z9psWcFVIfsMxuQ6XnZl4g4qRXTX4lYJxW1PstihE9gbVNy3z0/Dr3Be/Wk57posx5TM+Ky So9+vDux2yJUf8+fJiDsKG4zo7rRod0eaqV8jUAzpEehTQvLx1GggXrfp8IV6mVwhCaHxdnFjoBU/ 2MG5Bz8+I214Db1SaG/XeK3+uZhg3eEggltL7EX/qYq1Aaq3GfwpUeUUOBWVXkPblvDvqgesUGxdO PYE2iIlLpE4AhP5yXcSvOyc123G2ZBNgHhPdUgFXV0nKf8oknb0o9SO+dGBQcn2/BSmMDWz+zUBDp VKh2mmxg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpbhs-00000006Wqc-3fFk; Tue, 04 Mar 2025 23:29:48 +0000 Received: from mail-eastus2azlp170100001.outbound.protection.outlook.com ([2a01:111:f403:c110::1] helo=BN1PR04CU002.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadD-00000006OM9-0M0o for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:20:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ctmcsDz28ZWlzGRlPG96/CbChrHM1mmH9vKyvMLxESysPWp5Zof2rbngbx81HEDEH0CwoTSCD+T3SLUsZkpw/eLtoo9fz5aJ/sUGv03URB89Patnaladj/iCtHc64Mxo7/Qd42Qb/3Od0mWeIAeqjALi9dAiR/elDsrizDI5B1R4oe/+Ya1qBpEkFKaZQwNba0aP7HsguTJDugZGxGCbSDnbwinkjMoIMo+oW5tC+Wa6r+DavohzLGmOeRb7ps1CVGpnHG00+y0D+NfRAC8BOMYoaAOYEJj6EATxtGZePmEpAZIVi4ydqpI0zm5zvs7W49AQqtc+KhH1SagCOkm82Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bc8S2OTbZME42pyY8V87xNGR1wOTsn5d4wbfYh2jm/s=; b=YLxBFJsJVcBCWonl06ZSNwIr7pCuNBey+JcFZMAeYTEtI+afJ1Zgp767YTcANKiuUr014W7tliMtyZ8m99UNYw7r3Egh9vyR1RzXLesYIj2Bk5xASRw/KQnjRcS+iTYTzLgQiWheActcxpZ+6AHybn/kmdLKQKX33KhbrbebBU0tCltQ88y//A6svmLBeqqtABFNeKvKEtIw1p/UNqVYFAmHfJ9HiUQ2zpkfqJLHuyO+wImCb2AeE5BbY0fhA6y/ZJeZWub0JNT8i8H3kuA5RYsEWwq9hx8Tvz0F0bj5OLCwR2lW6/USdAUv9X3EkyjW2rVhAUuXVYgNGOA5RlyxtQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bc8S2OTbZME42pyY8V87xNGR1wOTsn5d4wbfYh2jm/s=; b=l95yYa9UaEDL7PgaSRZE6SzqrXnVcNVdFBfek0Y2GcSPLHw7mTP6NBB0bcNeywTyqiafUUeN+M4osf8TOJY3o0JUqPATfCypKjVpyUemmbr1BykbjwgD1Z9vG68lwJRcdN0ZsK4IM19973OnJCudEsR9iTpbJKthSqhMwBZnpm4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by PH7PR01MB7931.prod.exchangelabs.com (2603:10b6:510:275::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.26; Tue, 4 Mar 2025 22:20:48 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:48 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 3/6] arm64: mm: make __create_pgd_mapping() and helpers non-void Date: Tue, 4 Mar 2025 14:19:28 -0800 Message-ID: <20250304222018.615808-4-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|PH7PR01MB7931:EE_ X-MS-Office365-Filtering-Correlation-Id: 72ca7119-25e7-400d-f34e-08dd5b6ad0bd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|52116014|7053199007|38350700014; X-Microsoft-Antispam-Message-Info: WmpGZg1mjm2zsnrtHGa1zMKDmaPY8t489ZRZT0AUaVcylGUMneQRI+uYuEq5Ch/rLFwiblQxK0IQnudleXP5h9drxL9s9OKUlaoiIHOl+J2AGqhBEm5yBAY5rwgheAe4wY5zp7W7uxhOQE+hP748C1RFTY83Q6q9tA/NkrICvi0J/CqSkpAhKLtQ/OmvTMDSncHJG7J3J2n3Va8wl+VELdYEpqB4q7FECFtP8bTmQO7rrLizX6NCupaAaFiA2s+dNJIKdkBrzKPR2zVcX012p9I+XtLfA4cFX9NJmNu8crPfEN0GrZwmYoJtp2GKFL8nSByJZGBTH2jT14W/F/bXEkdPvwugm9dh1RwHDQadAkq5k7eHkuGZnv+6oxMSu55clpr0w2kiNs8pbpr3fudnG0a484UQYDjOhIhRY8a+HN4vd4Sy0QUXE/TVrK5NHK7eEaqRrtCXBwrbqN6E6L6taTL/Oh5uzyDhlf/1cCWlw8Dz9D8k6A2ZIhf9E3XiRsgrMDVwuRoTjXGIPrqitfmuunwR79iROeGcfc2VSXIhZKWBBOWWlrXGHpDjngE/+b3+Q1IpmbXD52SagXKAmnKF1zkNPy4jXHAJspc0oYyNbRxq4eO2JzvvTlgw8sLJ/gDD9pxapQXYp6SsIbQeYT4ekrBKIuySkTFc/04HLS6GG9ptwT3+3AwNvPLTvvKQps1KXDOYDhovZ2QDe1na4Url6cWdI/6c+7hRfZwiM3ggwatI0HDqlVVRYt9XYQZB6Z8IFCYGjCNeyKBgRM5T6XeHbW9qKfp6Qpu6d+PAEBCc7GyMzXF2yuUq8/TgiYW0fdLyGn17QuutIyGL3DfG5ZiD6c+fwH0iKNkSbcs92F+K9IoJlbbBz3vT/RKcuxd28XgCzXYO7H6fKEjVPw2aETVvGvtF5POgJEZ+atCyz2k8LzRc3vwNyLTiS6jMnXtCSb5jtIJZntcl2u2v7qBJ0m4tILP1e2a0xLWCEK5zSH1Z5hel0qit2dAtvNwp/AbXfHLcYHqfbIcFWMAlX1INeFxc/32sWwqi/N2cgIFhn54dYwy0nIhNBOfG7h2FXaNOlNzzgKip8COmYTahyATCDshqMWpl3kfLX6BQjh4PMnHInSqjkuXQrPnx9SeOzhs3as0wXeLF519vk2deH80fLwJO8lJwhRrP6ZWXHdE6gKtWy7WHeNBmZcYuIIGNf+TZ4u9GT64Gv0vAffPCEB8yz/MYtNZXD8U1ofUkBJzSh8yUyc1zvYKbiqSjh301rqYKVw2wVmaiaQ9vsrZdX0fFnvrz3ol1LudiqsMd0d6auSuhzggMh370m0wez00gOHjhHTQmf6pJ4DNQbpRZmxPyMaBJMiYLiEXTBX81LsqM0F2DBZRh13JfAbuBdG16vTnnbY84sBygN3OOkP5o/GW2lvtnJ3YPbEk2iUM6QM66r/RQabo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(52116014)(7053199007)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qgFq6QhRBOQ6J8oeEpGXvr17hwGbtmGHd9Mo2VtoWXQCTwCose8yfu5dPhPP7w+H9a3FRFVOebkwe/XCYNySLhOATkEyl63HjWK/gFIsrMQ1fpfDg03Vd5cTg8Vro333AxSQOERswn+QUmORVWEWBTllgSlES8MrAjWr67idMcGAy1kxqRuE0p+12CYScxrgUvHhVCztOJszFsWjWv6uNh6uK9swBYcx+vfOFrRRo4LS3C9yKILtS0UNCOL/8URjtc6riPkq5UUSUv0lMwXnzfrSQeBmF3/3WI89DxB9tMfEuaZ8O7APffqR45utWDuNFzBR+bV78ypo4PpNmhQDundAQMmon6AwOH5XVxCy33y5zEOo2L3QL8kpw2ua3hJy6JlVLU9PMOS5qta5nOFt423ezdYBtHrd6gxhGtUva/dSFVhD+vtfZPWMOQw2T0gjI14sgcAZHaeAMJ1BEs6pWHVYHjaPEj9dZVf8xM/UG69xKZZZ9izBocQlEwBrbb+4RLqxmPKIl7E0xy4fx13tskOcvW7FMsNtVoeHroDC0XeouDQF8koGMak09XNLyQZ9lsHTGNWjKwLZbd1P8RjMnraYKQaAW7dbSifDMrGQKe7JX3+FW8dEh4ZEn6/iHeH+MPyiofhjrvebhoBh48I5oIAJ5hFwg8tuNVOjws8NyYflMj06vQSLUJ4nuXapWxOa/smJdjroPIHd0Yw0WZSBrpE7XgQHthqWUpSdaqMCXZsqSUqFksPQ3kP3715Jq+o42kzdnkXHH98jgmmkyZPMJ7+0QOLWIGa9tFxqq5CA0pxnu+4mtsZteH0nQFoPWqDqW/2u/+IdlzPk5m0dc4m7njWL9wLj1VWJdvbRLzEOJ7wVpZR9A1hPObkz8GPuQx6MqJua6yaTyPlAAMgoF7xwBz5fiSLIhnDg0EEenmRjNaq1HppsCnNhrXUqgHvnwGFzKLt/V/Jz5KLr99bEj4aijuVyy+k9eiPbxl58wqeBseyFkI1HxIJzwstSAUfdxVO9Zigvtpm+NPRCL30Pl7lOSSp978UkIK4RPAgau76TqFlcMt758ChalX+Y4C1EumGKrypzq65wh+uF9iOpH/jE/wOtOvzQe0t8vNfCA4tNEZjSxwU+4N3NWOjqUy/ZhKEjtB4yFBpvVXVOmCMCd4EErAGUd7ibn7kN6IyQ3cVGuouNInYcNGYdBEwkHTdU1VPW6dq1FfgJ2Y0UKrOrRkweJd9z2hnf6VjgW1/L1Rb+mhovg9l6TseByEh8fYwhUAIEVxVz2WCb2gh6edsVvH93W3Nd0yy+D68PGe+x9gcZRKe2hxOc8pc2WVtnCV94/GFOaMWkCijC30HnWeJtFh/2qfIWvAjwKowZazKNJuYW1cO4IYE1VMvIG9fZXgnalYqTsNV7rcp5HGXr0HhgIqQpQ93MqLnXQQfhzAwv/wd2uBiz5+2DUrTooMIvpz/juagGfevKnqaNb690rliq5r+3TN1ZHpEGm31/LpzoiOisaM/Z8cefcbj4dwPHZZ3fR6dJHd0zCKAGd5gTiAfUECCa0WGV0/Oi9ZQkzcm8lBAfDb+X1b0peuPuzw3v3sPi9gSyOOh2NIM2nHryScueVPtCAu6zt/MQ5LqSc2ni0yaIEkE= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 72ca7119-25e7-400d-f34e-08dd5b6ad0bd X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:48.4636 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PC7AaB4426Opvqc210fk/En2zA+SU+jw5I90ZbzMdiFSN71dOIOt6b3J9FAEGuj34O/67gt+2zprkHjszE2W5kzlSguvJzURS3E8CRyi0w4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7931 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142055_129073_10BA2338 X-CRM114-Status: GOOD ( 13.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The later patch will enhance __create_pgd_mapping() and related helpers to split kernel linear mapping, it requires have return value. So make __create_pgd_mapping() and helpers non-void functions. And move the BUG_ON() out of page table alloc helper since failing splitting kernel linear mapping is not fatal and can be handled by the callers in the later patch. Have BUG_ON() after __create_pgd_mapping_locked() returns to keep the current callers behavior intact. Suggested-by: Ryan Roberts Signed-off-by: Yang Shi --- arch/arm64/mm/mmu.c | 127 ++++++++++++++++++++++++++++++-------------- 1 file changed, 86 insertions(+), 41 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b4df5bc5b1b8..dccf0877285b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -189,11 +189,11 @@ static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, } while (ptep++, addr += PAGE_SIZE, addr != end); } -static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; pmd_t pmd = READ_ONCE(*pmdp); @@ -208,6 +208,8 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, pmdval |= PMD_TABLE_PXN; BUG_ON(!pgtable_alloc); pte_phys = pgtable_alloc(PAGE_SHIFT); + if (!pte_phys) + return -ENOMEM; ptep = pte_set_fixmap(pte_phys); init_clear_pgtable(ptep); ptep += pte_index(addr); @@ -239,13 +241,16 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, * walker. */ pte_clear_fixmap(); + + return 0; } -static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) +static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; + int ret = 0; do { pmd_t old_pmd = READ_ONCE(*pmdp); @@ -264,22 +269,27 @@ static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), READ_ONCE(pmd_val(*pmdp)))); } else { - alloc_init_cont_pte(pmdp, addr, next, phys, prot, + ret = alloc_init_cont_pte(pmdp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; BUG_ON(pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); } phys += next - addr; } while (pmdp++, addr = next, addr != end); + + return ret; } -static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) +static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; + int ret = 0; pud_t pud = READ_ONCE(*pudp); pmd_t *pmdp; @@ -295,6 +305,8 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, pudval |= PUD_TABLE_PXN; BUG_ON(!pgtable_alloc); pmd_phys = pgtable_alloc(PMD_SHIFT); + if (!pmd_phys) + return -ENOMEM; pmdp = pmd_set_fixmap(pmd_phys); init_clear_pgtable(pmdp); pmdp += pmd_index(addr); @@ -314,21 +326,26 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + ret = init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + if (ret) + break; pmdp += pmd_index(next) - pmd_index(addr); phys += next - addr; } while (addr = next, addr != end); pmd_clear_fixmap(); + + return ret; } -static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; + int ret = 0; p4d_t p4d = READ_ONCE(*p4dp); pud_t *pudp; @@ -340,6 +357,8 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, p4dval |= P4D_TABLE_PXN; BUG_ON(!pgtable_alloc); pud_phys = pgtable_alloc(PUD_SHIFT); + if (!pud_phys) + return -ENOMEM; pudp = pud_set_fixmap(pud_phys); init_clear_pgtable(pudp); pudp += pud_index(addr); @@ -369,8 +388,10 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), READ_ONCE(pud_val(*pudp)))); } else { - alloc_init_cont_pmd(pudp, addr, next, phys, prot, + ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; BUG_ON(pud_val(old_pud) != 0 && pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); @@ -379,14 +400,17 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, } while (pudp++, addr = next, addr != end); pud_clear_fixmap(); + + return ret; } -static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; + int ret = 0; pgd_t pgd = READ_ONCE(*pgdp); p4d_t *p4dp; @@ -398,6 +422,8 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, pgdval |= PGD_TABLE_PXN; BUG_ON(!pgtable_alloc); p4d_phys = pgtable_alloc(P4D_SHIFT); + if (!p4d_phys) + return -ENOMEM; p4dp = p4d_set_fixmap(p4d_phys); init_clear_pgtable(p4dp); p4dp += p4d_index(addr); @@ -412,8 +438,10 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, next = p4d_addr_end(addr, end); - alloc_init_pud(p4dp, addr, next, phys, prot, + ret = alloc_init_pud(p4dp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; BUG_ON(p4d_val(old_p4d) != 0 && p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); @@ -422,23 +450,26 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, } while (p4dp++, addr = next, addr != end); p4d_clear_fixmap(); + + return ret; } -static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long addr, end, next; pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + int ret = 0; /* * If the virtual and physical address don't have the same offset * within a page, we cannot map the region as the caller expects. */ if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) - return; + return -EINVAL; phys &= PAGE_MASK; addr = virt & PAGE_MASK; @@ -446,29 +477,38 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); - alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, + ret = alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; phys += next - addr; } while (pgdp++, addr = next, addr != end); + + return ret; } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { + int ret; + mutex_lock(&fixmap_lock); - __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, + ret = __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, pgtable_alloc, flags); + BUG_ON(ret); mutex_unlock(&fixmap_lock); + + return ret; } #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 extern __alias(__create_pgd_mapping_locked) -void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, - phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags); +int create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags); #endif static phys_addr_t __pgd_pgtable_alloc(int shift) @@ -476,13 +516,17 @@ static phys_addr_t __pgd_pgtable_alloc(int shift) /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO); - BUG_ON(!ptr); + if (!ptr) + return 0; + return __pa(ptr); } static phys_addr_t pgd_pgtable_alloc(int shift) { phys_addr_t pa = __pgd_pgtable_alloc(shift); + if (!pa) + goto out; struct ptdesc *ptdesc = page_ptdesc(phys_to_page(pa)); /* @@ -498,6 +542,7 @@ static phys_addr_t pgd_pgtable_alloc(int shift) else if (shift == PMD_SHIFT) BUG_ON(!pagetable_pmd_ctor(ptdesc)); +out: return pa; } From patchwork Tue Mar 4 22:19:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 966CAC021B8 for ; Tue, 4 Mar 2025 22:28:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=94OWQgZSNmZciVJ66VwsvsrSEv9n1ZXhYxuZKkj/4x4=; b=WuDc9U0m4PlFaqVxmACGrHFUzM QvqvhDHyWNuxihGoKSWSgaj8oXWWJYkZuabf3nqGi4GEo5vVtbss5Lg/hE9XsUjKohdaJdNJWfZ0h ZEveNIop+qmxhlk8iHsjQsyQgCF8ztWsOzyphUEAy++Y9Zkxk4zgyYQ0ai/q/me7NQyCKGH9dfiNr 4j4rbwUDBx+1dzpGpWhrFwlWyxxsE5xfHO4st6HO2CMxdjGp3aKSlSC4dYOjbt4DgA0Z/pHKJcrpL i9ajy+5RK/VP3agY/KQv665gY8UviXWQ4coSP5VwyUlaavxyPknwaC8DivD7ZfAHV8XbbJ8JG4j7W 8oTWtRpw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpaks-00000006Pl9-1SIX; Tue, 04 Mar 2025 22:28:50 +0000 Received: from mail-eastus2azlp170100001.outbound.protection.outlook.com ([2a01:111:f403:c110::1] helo=BN1PR04CU002.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadE-00000006OM9-3su5 for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:20:58 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=yUghz798qnfaVRHjKlOdBxkBlwbkh6yxheU8L4E0zQJcMKoGRsfHmGGN9yHqDKk+PQqBptdfKbi/y8+DXecVtf0KQCCQy10tatYQWO1ETC4UHvkPsS9a/DBIuUmOL810zg78MxihPulz/u2gsfa8zFjZfH/A3xyApeYEKyuq9SltdjPPfmNAxDmDplpx1M43GWl1wiDzCEygfCmPTuwnhpGoeFNcCwUrKDGmAhIjeRIDL5tYFHppddxWUiTPugBElRqNXSjpJGtpgXt6fqM0rXRpcvwmMElnqhtIza4lMlcyZOHk/M2hAFg0jpXyHgmsvKP04BmNBJ/T8u8s9xpZQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=94OWQgZSNmZciVJ66VwsvsrSEv9n1ZXhYxuZKkj/4x4=; b=Jx3UcM729qq12JOugZYNJI2Svexk77ctIw1nKo5RVYzRGyoA3ObKsSy1yjnRn7uBN0Ib6WcHHUrFLjAuuak4TNnxQu9b0Ebh0lTDgx4ijiyKMf98v0PVUOdZwmdKBT5p10YTrskkoROAja/imNaqktzdsFani6b3TijWfC30Y5chTQCrMyKm6KkCg2ft0Y/0u0oEhrwZ5kcYEPzIlt9YVQ5lRB896+FKA5yBvWznmuyP8QJY/SSfa6cxnrhHjkA/uMhWhs9y1AWuOfOcJNAm8jt2zi8bqAolE64A/8cuNbYo8f/z1JpX1AxKNwKRJdenqfhqNHh6qLEx/0WM3M7ODA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=94OWQgZSNmZciVJ66VwsvsrSEv9n1ZXhYxuZKkj/4x4=; b=Gm0RMFd2wXfvwKq/Akg5/igHnS27pcnJoAGBjRKDQqX9tMZlnr45cwd9UY7HNHXIgIJRLbw29+mD4xlbir+FThRobBvS135+YErTFsXSnNj2wMOsShkcChyYrK4Oytf0oE2kdFnoFX6mC5eVgTc1TOTUb8zoXz69ewzYHPt6Cws= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by PH7PR01MB7931.prod.exchangelabs.com (2603:10b6:510:275::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.26; Tue, 4 Mar 2025 22:20:49 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:49 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 4/6] arm64: mm: support large block mapping when rodata=full Date: Tue, 4 Mar 2025 14:19:29 -0800 Message-ID: <20250304222018.615808-5-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|PH7PR01MB7931:EE_ X-MS-Office365-Filtering-Correlation-Id: 1cb76ef4-9d37-44fc-d3a2-08dd5b6ad163 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|52116014|38350700014; X-Microsoft-Antispam-Message-Info: KQOAKu2FhjmgK2ANpIzHXVLDh/wATgYhSwwNO5e8v5mxksmid4hsycPByJDfO3n9Uw+bru5hXKvhgKmQ18qmCpoaN9VrPBS1ydhQ9lusNL/0k5pdlG80Z7KPXEwLhOL93UkyT22do+50qzlr0Ot/K2w3tnLP7tW8k6KDngSuO7lleFktdiNtifnzoc1O3eTiBEII5l+hXLSNXwD/50cNzNDkuKWqbA78Tffu2WhtCmrXyP8a5AyXwmHfYNGYc2Nc2w6Jj9enyklZ0dDgoSQd6qCkiINVPht/Xz/diopzFKaruXbGamAAZR6x+FyZUIXNvaS9R5jRfKKr9dkfHYE9Hw7XOiFG55ry7Dwa8uVlSa4Bb8NzKGTxdGgOU2JOJFEVKhx7ljDfbRNTnfAURx86/rpw9VUU69rmHyeGkVp4REpbBeRxst4//Up04Myuz7cjb4F65kwsvn9Gi2/8qfURm/7JQuYIa2o22+AsQr8RMKeFpnbQ5PphGX6eDpNTnxCmSAcy/cYQCg47oC80Y7sCqQB1yjZJ1ayuqFmJNSHOTvASejBU5r74odY7pdq0MtOdDeNGGOrWLfYrZHUS/HpwHtukN3injM2Jc38JPXk08Ytd0rvnVYt9zBQiiUH2dFgkFtyF0yol0cqzJFGf3sipyAhVVVfd+Rfg0gzUIYHaxXGLmCJlx+2gWDO8pBwt5726JtXjZ24iI2CrL4CGiTXh2laXj7irRxhiXWRcQEWeNNRt89YAvmyBTyigUmEmCxhJVIDOD9YdxsuskabY5/ViL7Ie3x1X6F9GPM9wFPkkloV9A1j0PvRcu2QlOAyZY1emeTItI9Jc2W/FafOQB4K4iYxF0Vv9UA8N+RA6gDA4hSZ5Yi9m2GP7yE3mK8EtEU2xCVs55MFJKOdQTHtSpsHQVs4PylbvwgilRzAr/8gnEe/joz3P2rD2utkSxnHgXaOaKgCKDV8/S7mhMuGBJKAMkXuZdMNSzQJXr37fvxOTesL2vDCjTbco5NgabTI7lxPrpcG8mmJ/KCj9ywFDIqxnmldaeiU1c27EYcRLIHU01GOcAbUFbdKjNFUNtswqSHYbQpB0MgQwIJPdZK/hxq3yVBcCdLtZkvHsVo0Kd1kwg5uzrijz8Tvzo+mDr7Y2JQseKXHUoLsMYiWYu3mb8aE3yYN/9vJCaCRgcBCgea61E/siqAFrnTbh+qaXiqlbzHP2G1HGpLl33OVu7iFrgbSoOLcDa/Pa1RR0XvZmEGs7Hs0nOSxHplwaFAQ1wTfcxrsJleOnLrCIPvL0DprTemDWCXCoWUg/eAMsIIui3pKVfn2L9To+XEo9f74VEsXCbGPCvFebJWVDth5r3+Lrz3T6AcHFWDGoehRSGT29thfIo30FQQuK9htsfKrse4Bu95au5W463yCmpny2RmNdwHjCMSKeIjWMBJ2K3bGYUgL2hRHb996hT/IH4iAd8omeNuDm X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Hr0WsYr/0WTVP4PD/0C7UlAXVuSk8bZp5D1apCcULKn8t1o5RwneP2jgZa/5ql1flQ7aknJ6iZG782DBm4M9a+CLxpd07GfRLbdWzkvCx4Eg9SSl8OCNxJOVbiGEc5oAU/ta2Hwqb+vajVd1HGfxfCdQ1RrAwkejjV0FAJ8RZXpRtc+2yyyRPLsf6nXEGlCeXb32ZKetVx270ST0kdejnCgxHEJVehkjPBuxZXGzISXibhaau05r7wl+xtBA+VkaLIkHAaBVR+tTNynNeyqv6R5UrggljTK59IEMG3xXTJzMOBDmcJxu10OVuau7hEQEalul7uOO4fBWSHdfGnx4kg4NZly3LFn7JdyYhlGDgi82FHDqm9u4UKie/IkHmztg+uZRASdtK7AjI3zl51H29Q+Ghyo1Bj4fSQ9WzinfmGEVLPeuR8VVsq4kUJWERBL9QXkKb0xtCAFgcV0UZx6maPHwkY6aU05pQiAZfk87pvCkYSY54TL9B14VlJRKnYmKfm4LyUgJO9kGXQsB/J/sgoVBYKcQI6iG5QDeLIEQvLV1TeUK47LF6TblApPxI2yUbebqGRt/NUtLQ96G4taJBAqjQbsfg1Iz21nTzRUqEyq7i5alvYj8gGTC2dJWaLoTa4Yl545ASLZioXUfb46ycI/Nuh9y6ZnaBHc9CMsHnR/NLU7LN8cU9eCOiMw3NJH5gLK+O20S8jcr17XS9lWf1H7VT0xmTa8X+bOozMs/cEEQ6RhW/L1igt4VMuN/z2lwcrGovTaVU54kQepanBmmQ//1eMxI5sn4rXuwRECcGQTFOh9JJl4AHss2iOXWKGYzJ7meFo/ELC9M7Atcc7G6BHo5V/FJej+eF8zipNq4mQtXDR3DMRgdiLugZ6yzc2wYH9REcwYyBv/WSnYZnmC+aD03O3g0vHAh6iqH6O8wo0Rxwf6M9QvKmHbIfnkBXHXzIDJLcR8ZsncvvQlBgdAeGlr9tK70oo08aof1+s+NTFuCppLH87ROEExmZcMXiWGyFqHcGejCCivaFaNA0eYDr63ZQyFYOuMY5lEmQ7KgS0fCWhFlfbKIapcNLnxdtSozkeHfH9303Jz/WqYVj77QtRqt1es/7eyqkB7DK47+/qNfHort2figsmee/59jB4CDkKkf0TX9jlLlQ0gzdFmxgPnsCyJQBLHyNOzMfMHjiivpXFLmg6fqv8Re1T1KAnz0hx3+KKjJn1thOSpSNfTVe3lKW/Prz6C/MtpWqKsLIWcOl6qabcaY4h2v5bvDc97S8a+oR1R8r5FT2tiUeBisy7W7YNcfJa5ohLdkgzVc2PoP0lodO2EXgNdJcdv3YLNNHjQrlNkBC2QlEq2aO/TfGnVz5n5mFd0g6YB29sWL3bpYoOzT3nMxv+UuSYI/bObhBipkYpoLnMR+zbnhBchbj+oEG8FEnv/3gOB8R9zM/PBL0ZwSU17UiOZd7nyJYI9Udbx8aEuyfKWMQQAb4irUnKTCT8huCASyT7bPvEIkybSlW5zf3s68NCMawGg27DpEQnSl40vsKA6IEWrpjdDi6sPuDkBtkn8IrUOkIQEB5CoeZ45kBMzvMkCMHcGXzzZfVF8hN223yP+Q7ZCRkDqVyQUEaaPE9SrsXGNAve+Tcts= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1cb76ef4-9d37-44fc-d3a2-08dd5b6ad163 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:49.5461 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: G/mhj33U4nyGXBKa/qG4/dnEx3jLHA4eZ6T74xvi8Fv6/WKmILTqmKF5F+4NFcJUSCLOV+y2y1LYaMK07cIS79NnU5D0PmZzq513KSrCJ58= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7931 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142056_971730_04C0CA8F X-CRM114-Status: GOOD ( 21.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When rodata=full is specified, kernel linear mapping has to be mapped at PTE level since large page table can't be split due to break-before-make rule on ARM64. This resulted in a couple of problems: - performance degradation - more TLB pressure - memory waste for kernel page table With FEAT_BBM level 2 support, splitting large block page table to smaller ones doesn't need to make the page table entry invalid anymore. This allows kernel split large block mapping on the fly. Add kernel page table split support and use large block mapping by default when FEAT_BBM level 2 is supported for rodata=full. When changing permissions for kernel linear mapping, the page table will be split to PTE level. The machine without FEAT_BBM level 2 will fallback to have kernel linear mapping PTE-mapped when rodata=full. With this we saw significant performance boost with some benchmarks and much less memory consumption on my AmpereOne machine (192 cores, 1P) with 256GB memory. * Memory use after boot Before: MemTotal: 258988984 kB MemFree: 254821700 kB After: MemTotal: 259505132 kB MemFree: 255410264 kB Around 500MB more memory are free to use. The larger the machine, the more memory saved. * Memcached We saw performance degradation when running Memcached benchmark with rodata=full vs rodata=on. Our profiling pointed to kernel TLB pressure. With this patchset we saw ops/sec is increased by around 3.5%, P99 latency is reduced by around 9.6%. The gain mainly came from reduced kernel TLB misses. The kernel TLB MPKI is reduced by 28.5%. The benchmark data is now on par with rodata=on too. * Disk encryption (dm-crypt) benchmark Ran fio benchmark with the below command on a 128G ramdisk (ext4) with disk encryption (by dm-crypt). fio --directory=/data --random_generator=lfsr --norandommap --randrepeat 1 \ --status-interval=999 --rw=write --bs=4k --loops=1 --ioengine=sync \ --iodepth=1 --numjobs=1 --fsync_on_close=1 --group_reporting --thread \ --name=iops-test-job --eta-newline=1 --size 100G The IOPS is increased by 90% - 150% (the variance is high, but the worst number of good case is around 90% more than the best number of bad case). The bandwidth is increased and the avg clat is reduced proportionally. * Sequential file read Read 100G file sequentially on XFS (xfs_io read with page cache populated). The bandwidth is increased by 150%. Signed-off-by: Yang Shi --- arch/arm64/include/asm/cpufeature.h | 10 ++ arch/arm64/include/asm/mmu.h | 1 + arch/arm64/include/asm/pgtable.h | 7 +- arch/arm64/kernel/cpufeature.c | 2 +- arch/arm64/mm/mmu.c | 169 +++++++++++++++++++++++++++- arch/arm64/mm/pageattr.c | 35 +++++- 6 files changed, 211 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 108ef3fbbc00..e24edc32b0bd 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -871,6 +871,16 @@ static inline bool system_supports_bbml2_noabort(void) return alternative_has_cap_unlikely(ARM64_HAS_BBML2_NOABORT); } +bool cpu_has_bbml2_noabort(unsigned int cpu_midr); +/* + * Called at early boot stage on boot CPU before cpu info and cpu feature + * are ready. + */ +static inline bool bbml2_noabort_available(void) +{ + return cpu_has_bbml2_noabort(read_cpuid_id()); +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index 662471cfc536..d658a33df266 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -71,6 +71,7 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgprot_t prot, bool page_mappings_only); extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); +extern int split_linear_mapping(unsigned long start, unsigned long end); /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 0b2a2ad1b9e8..ed2fc1dcf7ae 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -749,7 +749,7 @@ static inline bool in_swapper_pgdir(void *addr) ((unsigned long)swapper_pg_dir & PAGE_MASK); } -static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) +static inline void __set_pmd_nosync(pmd_t *pmdp, pmd_t pmd) { #ifdef __PAGETABLE_PMD_FOLDED if (in_swapper_pgdir(pmdp)) { @@ -759,6 +759,11 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) #endif /* __PAGETABLE_PMD_FOLDED */ WRITE_ONCE(*pmdp, pmd); +} + +static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) +{ + __set_pmd_nosync(pmdp, pmd); if (pmd_valid(pmd)) { dsb(ishst); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index bf3df8407ca3..d39637d5aeab 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2176,7 +2176,7 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } -static bool cpu_has_bbml2_noabort(unsigned int cpu_midr) +bool cpu_has_bbml2_noabort(unsigned int cpu_midr) { /* We want to allow usage of bbml2 in as wide a range of kernel contexts * as possible. This list is therefore an allow-list of known-good diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index dccf0877285b..ad0f1cc55e3a 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,6 +45,7 @@ #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ +#define SPLIT_MAPPINGS BIT(3) u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); @@ -166,6 +167,73 @@ static void init_clear_pgtable(void *table) dsb(ishst); } +static int split_pmd(pmd_t *pmdp, pmd_t pmdval, + phys_addr_t (*pgtable_alloc)(int)) +{ + unsigned long pfn; + pgprot_t prot; + phys_addr_t pte_phys; + pte_t *ptep; + + if (!pmd_leaf(pmdval)) + return 0; + + pfn = pmd_pfn(pmdval); + prot = pmd_pgprot(pmdval); + + pte_phys = pgtable_alloc(PAGE_SHIFT); + if (!pte_phys) + return -ENOMEM; + + ptep = (pte_t *)phys_to_virt(pte_phys); + init_clear_pgtable(ptep); + prot = __pgprot(pgprot_val(prot) | PTE_TYPE_PAGE); + for (int i = 0; i < PTRS_PER_PTE; i++, ptep++) + __set_pte_nosync(ptep, pfn_pte(pfn + i, prot)); + + dsb(ishst); + + set_pmd(pmdp, pfn_pmd(__phys_to_pfn(pte_phys), + __pgprot(PMD_TYPE_TABLE))); + + return 0; +} + +static int split_pud(pud_t *pudp, pud_t pudval, + phys_addr_t (*pgtable_alloc)(int)) +{ + unsigned long pfn; + pgprot_t prot; + pmd_t *pmdp; + phys_addr_t pmd_phys; + unsigned int step; + + if (!pud_leaf(pudval)) + return 0; + + pfn = pud_pfn(pudval); + prot = pud_pgprot(pudval); + step = PMD_SIZE >> PAGE_SHIFT; + + pmd_phys = pgtable_alloc(PMD_SHIFT); + if (!pmd_phys) + return -ENOMEM; + + pmdp = (pmd_t *)phys_to_virt(pmd_phys); + init_clear_pgtable(pmdp); + for (int i = 0; i < PTRS_PER_PMD; i++, pmdp++) { + __set_pmd_nosync(pmdp, pfn_pmd(pfn, prot)); + pfn += step; + } + + dsb(ishst); + + set_pud(pudp, pfn_pud(__phys_to_pfn(pmd_phys), + __pgprot(PUD_TYPE_TABLE))); + + return 0; +} + static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot) { @@ -251,12 +319,21 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, { unsigned long next; int ret = 0; + bool split = flags & SPLIT_MAPPINGS; do { pmd_t old_pmd = READ_ONCE(*pmdp); next = pmd_addr_end(addr, end); + if (split) { + ret = split_pmd(pmdp, old_pmd, pgtable_alloc); + if (ret) + break; + + continue; + } + /* try section mapping first */ if (((addr | next | phys) & ~PMD_MASK) == 0 && (flags & NO_BLOCK_MAPPINGS) == 0) { @@ -292,11 +369,19 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, int ret = 0; pud_t pud = READ_ONCE(*pudp); pmd_t *pmdp; + bool split = flags & SPLIT_MAPPINGS; /* * Check for initial section mappings in the pgd/pud. */ BUG_ON(pud_sect(pud)); + + if (split) { + BUG_ON(pud_none(pud)); + pmdp = pmd_offset(pudp, addr); + goto split_pgtable; + } + if (pud_none(pud)) { pudval_t pudval = PUD_TYPE_TABLE | PUD_TABLE_UXN | PUD_TABLE_AF; phys_addr_t pmd_phys; @@ -316,6 +401,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, pmdp = pmd_set_fixmap_offset(pudp, addr); } +split_pgtable: do { pgprot_t __prot = prot; @@ -334,7 +420,8 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, phys += next - addr; } while (addr = next, addr != end); - pmd_clear_fixmap(); + if (!split) + pmd_clear_fixmap(); return ret; } @@ -348,6 +435,13 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, int ret = 0; p4d_t p4d = READ_ONCE(*p4dp); pud_t *pudp; + bool split = flags & SPLIT_MAPPINGS; + + if (split) { + BUG_ON(p4d_none(p4d)); + pudp = pud_offset(p4dp, addr); + goto split_pgtable; + } if (p4d_none(p4d)) { p4dval_t p4dval = P4D_TYPE_TABLE | P4D_TABLE_UXN | P4D_TABLE_AF; @@ -368,11 +462,25 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, pudp = pud_set_fixmap_offset(p4dp, addr); } +split_pgtable: do { pud_t old_pud = READ_ONCE(*pudp); next = pud_addr_end(addr, end); + if (split) { + ret = split_pud(pudp, old_pud, pgtable_alloc); + if (ret) + break; + + ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot, + pgtable_alloc, flags); + if (ret) + break; + + continue; + } + /* * For 4K granule only, attempt to put down a 1GB block */ @@ -399,7 +507,8 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, phys += next - addr; } while (pudp++, addr = next, addr != end); - pud_clear_fixmap(); + if (!split) + pud_clear_fixmap(); return ret; } @@ -413,6 +522,13 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, int ret = 0; pgd_t pgd = READ_ONCE(*pgdp); p4d_t *p4dp; + bool split = flags & SPLIT_MAPPINGS; + + if (split) { + BUG_ON(pgd_none(pgd)); + p4dp = p4d_offset(pgdp, addr); + goto split_pgtable; + } if (pgd_none(pgd)) { pgdval_t pgdval = PGD_TYPE_TABLE | PGD_TABLE_UXN | PGD_TABLE_AF; @@ -433,6 +549,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, p4dp = p4d_set_fixmap_offset(pgdp, addr); } +split_pgtable: do { p4d_t old_p4d = READ_ONCE(*p4dp); @@ -449,7 +566,8 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, phys += next - addr; } while (p4dp++, addr = next, addr != end); - p4d_clear_fixmap(); + if (!split) + p4d_clear_fixmap(); return ret; } @@ -546,6 +664,23 @@ static phys_addr_t pgd_pgtable_alloc(int shift) return pa; } +int split_linear_mapping(unsigned long start, unsigned long end) +{ + int ret = 0; + + if (!system_supports_bbml2_noabort()) + return 0; + + mmap_write_lock(&init_mm); + ret = __create_pgd_mapping_locked(init_mm.pgd, virt_to_phys((void *)start), + start, (end - start), __pgprot(0), + __pgd_pgtable_alloc, SPLIT_MAPPINGS); + mmap_write_unlock(&init_mm); + flush_tlb_kernel_range(start, end); + + return ret; +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -665,6 +800,24 @@ static inline void arm64_kfence_map_pool(phys_addr_t kfence_pool, pgd_t *pgdp) { #endif /* CONFIG_KFENCE */ +static inline bool force_pte_mapping(void) +{ + /* + * Can't use cpufeature API to determine whether BBML2 supported + * or not since cpufeature have not been finalized yet. + * + * Checking the boot CPU only for now. If the boot CPU has + * BBML2, paint linear mapping with block mapping. If it turns + * out the secondary CPUs don't support BBML2 once cpufeature is + * fininalized, the linear mapping will be repainted with PTE + * mapping. + */ + return (rodata_full && !bbml2_noabort_available()) || + debug_pagealloc_enabled() || + arm64_kfence_can_set_direct_map() || + is_realm_world(); +} + static void __init map_mem(pgd_t *pgdp) { static const u64 direct_map_end = _PAGE_END(VA_BITS_MIN); @@ -690,9 +843,12 @@ static void __init map_mem(pgd_t *pgdp) early_kfence_pool = arm64_kfence_alloc_pool(); - if (can_set_direct_map()) + if (force_pte_mapping()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + if (rodata_full) + flags |= NO_CONT_MAPPINGS; + /* * Take care not to create a writable alias for the * read-only text and rodata sections of the kernel image. @@ -1388,9 +1544,12 @@ int arch_add_memory(int nid, u64 start, u64 size, VM_BUG_ON(!mhp_range_allowed(start, size, true)); - if (can_set_direct_map()) + if (force_pte_mapping()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + if (rodata_full) + flags |= NO_CONT_MAPPINGS; + __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 39fd1f7ff02a..5d42d87ea7e1 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -80,8 +81,9 @@ static int change_memory_common(unsigned long addr, int numpages, unsigned long start = addr; unsigned long size = PAGE_SIZE * numpages; unsigned long end = start + size; + unsigned long l_start; struct vm_struct *area; - int i; + int i, ret; if (!PAGE_ALIGNED(addr)) { start &= PAGE_MASK; @@ -118,7 +120,12 @@ static int change_memory_common(unsigned long addr, int numpages, if (rodata_full && (pgprot_val(set_mask) == PTE_RDONLY || pgprot_val(clear_mask) == PTE_RDONLY)) { for (i = 0; i < area->nr_pages; i++) { - __change_memory_common((u64)page_address(area->pages[i]), + l_start = (u64)page_address(area->pages[i]); + ret = split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + + __change_memory_common(l_start, PAGE_SIZE, set_mask, clear_mask); } } @@ -174,6 +181,9 @@ int set_memory_valid(unsigned long addr, int numpages, int enable) int set_direct_map_invalid_noflush(struct page *page) { + unsigned long l_start; + int ret; + struct page_change_data data = { .set_mask = __pgprot(0), .clear_mask = __pgprot(PTE_VALID), @@ -182,13 +192,21 @@ int set_direct_map_invalid_noflush(struct page *page) if (!can_set_direct_map()) return 0; + l_start = (unsigned long)page_address(page); + ret = split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + l_start, PAGE_SIZE, change_page_range, + &data); } int set_direct_map_default_noflush(struct page *page) { + unsigned long l_start; + int ret; + struct page_change_data data = { .set_mask = __pgprot(PTE_VALID | PTE_WRITE), .clear_mask = __pgprot(PTE_RDONLY), @@ -197,9 +215,14 @@ int set_direct_map_default_noflush(struct page *page) if (!can_set_direct_map()) return 0; + l_start = (unsigned long)page_address(page); + ret = split_linear_mapping(l_start, l_start + PAGE_SIZE); + if (WARN_ON_ONCE(ret)) + return ret; + return apply_to_page_range(&init_mm, - (unsigned long)page_address(page), - PAGE_SIZE, change_page_range, &data); + l_start, PAGE_SIZE, change_page_range, + &data); } static int __set_memory_enc_dec(unsigned long addr, From patchwork Tue Mar 4 22:19:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12723C021B8 for ; Tue, 4 Mar 2025 22:30:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tkmV7x3mcQBAyxbvWzDtrcqC+7mmWkYjzNradsUNkVU=; b=v6Ri3P31hN3VFnkf5azKQruwpm 0t51pgL3k2WwtAXIlO+XjRYqPBW3lYcM7BwDl1x4/EduCZRyEOe0VpZKjUp7tdRzIJbFEenrJHCk/ eBk65PbfEd7W92JCPeE9gcRXf5HJQX7iy+1RtlA6RObn6i4k6w4yc7bzjOIX9DZK0OZzy/i4rTObn 4VWUIPuImtLOgZ0/MWDUnuGD9yrXMgKJhy8ItVL3D7vdkonue1Gga/FIbu9dHn5dUgtgoqMKJ/mZP Es/VI2WrkyPhzATIO0PcTBTa32Mnarr1gEGD+qJ0Ga0yhh04tdQYbb5PuTedN6bB/CZLJKC35xHbi FnsPH7/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpamQ-00000006QI4-0fFw; Tue, 04 Mar 2025 22:30:26 +0000 Received: from mail-eastus2azlp170100001.outbound.protection.outlook.com ([2a01:111:f403:c110::1] helo=BN1PR04CU002.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadG-00000006OM9-1nRz for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:20:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=q+PVzPjWEgusvsnqd+pXygerLo6RUJ8oN08vI4lGBvjB7BR12qIWwvj16cAIaiqrXIdevesJDtaCeIBY0MaozsFSFgt5B5NI3DCLbUGM3+Rq78h8TR/MWZyYhlQWxA9ulgF14vwj8imO1giAaLlK/3jyjvbXnULULSl/rVe0EosMmrkVtYCEpWVBlEiGYz52no7mDZhlnT0x39cutE8kq3/DUSINJamkRaZIutmkr+FlbeAkAxJRBpHEc/yMuqHgODYX2dziLh5I3awxtzun97eH8aFGK3CDPr6mc7s3FuIx8ia0sH8Yp3E2VnFhLpYU4M84tpkBXoffUH+C6RPf8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tkmV7x3mcQBAyxbvWzDtrcqC+7mmWkYjzNradsUNkVU=; b=F8ZQbKOyfNQGKpAK8qJoT1pyxJwd9FhKU7zILrWsACOp4Af4HsUTkBcPeaJAJItF5HQf+Wz4uC5xTj6qQYvRT+Iy3bdmhLdB0gkygma23kAKeqIK1eJg8BAD2AvSB/4ZXeyzH9Bvs6enzX2lfXheJJvXM+KCbNZCg8XCMzLasrPrU2eYXj2O1aL15+zRkD0OSb5xMFrO9ot3iESJJKhwAx2/fMY6Qtm3iuzxeOvqnC1GKEXvv5qhLHqLE7lx2Mal9aXlt1fcEne/Mr9BhY+ymT/Z08dWgnUEIyK2YNPG3EhCaySMNmutaPNxuZDlZE+rN2yVjHFGUsQobyYACT1A/A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tkmV7x3mcQBAyxbvWzDtrcqC+7mmWkYjzNradsUNkVU=; b=aPFTIeHUjVx6Oya+l8C1SNhzJsO2iC4+oVaz9ZQwhGZomv7X64szM3xHLPubK5p0FxAtC+7dyXy/5vZMFi4gv2v/MUrth4/5oZW8iVr/RZVYnpxS/fIfndFLcSnBbK0jjQvrqWkXSAoP77Sg4mtQ9IbSThHPVFl2z4KqAPQa5ts= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by PH7PR01MB7931.prod.exchangelabs.com (2603:10b6:510:275::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.26; Tue, 4 Mar 2025 22:20:50 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:50 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 5/6] arm64: mm: support split CONT mappings Date: Tue, 4 Mar 2025 14:19:30 -0800 Message-ID: <20250304222018.615808-6-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|PH7PR01MB7931:EE_ X-MS-Office365-Filtering-Correlation-Id: ebf135ca-f1ff-4deb-a5fa-08dd5b6ad226 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|52116014|38350700014; X-Microsoft-Antispam-Message-Info: AYrdNVuVE4Zpjm50pgIWQ1cQlC6zH8tELKx5InYgCHuXvh7cJ5XC2LZ0QnKF56tl3hpsaprsAteEhCcmMBSfkSyYvx7M9pLwxIQwiZTkjK5AQNy0KFZxT+C4HDQUv07zPQPCxCbNO/ONwgwN79NkZYI+0tCzxn5ZKlgUGHrdwxPuKmx2WcP+nM1ZtqS1YRdg/ZJDcAveXOfwxUMT6CNUcINhjRwaYKNypbMXNiMRoYPrwr6wpzm3Z+gAIi+AhNJY5/GpIKcLtoBDo/W02MqsBt+re5pc/1npyECFmTV9w2kuuZhnxPFstpiZMLuF3HOLbodIMlieU/AbSj+8aHpBfsGhOqvN6o1GRvGSbwFhbPLzodvl204qejIEEd7UqRoZdvfMnziyPsELAh2Sxm0wMTFmLM5eqOha5q7jUPNJxgYxFaUiunNYziA/zJw64dNROyGrBQPaAma39FLO54FmXfCPXhe/9pWdMayT9oOzdCYSrq9eYtu7681PeU+IpH14KSwTVJi7pqYXyaHbC46dfhxj3pVmbspK+Hm58Vi+P0G2uGmVxcgwTJvLrJREkxwcXpPvel4VQM8kxdZyToH2Tzbyq1RzY31GdZ750LLoDjP/ndoZ0KrsVYpW7AZnj+XmzDa4airhGM5mMPtQO3LKNOfpobTcCX4ARKM0sBqSiw8KUNdHXcZx7lkDaaAkh35zVmkAWarUuiNsnYq3MDLFWwQxBYzjpSv0SMGBWC/QBQYM7tgE1JEBn2YtoAmpYpDsDnR8N81/eAweHsliBoeP22t8/5r70sEbrfl0yUA/IGhO7jsBJGtiJyNLo8JmZc+YYJ4yvQ8nB3QBZLfVr4a1x3fp3RIUJPOrUHavzWRxvP79mK9WlqCnWEQSkR/hfwOgJO2S4yBQIx1OSDUL54lLVcD08nQZrrrtrrsMUniXjCjp9I+3ilZu8/UaF6VqlUcTlPcKHbZfVNAdzHpW6QgM+qAhWh83G1dGV7vTaGoRuu16TZ66SA4BnQBOf9anIgmRSRxM4o1OygTohRY+exVgZMgwijBecwIn54K+8zGA/57RVscL9TBdk+USnmoFV8SglEYEBRn9jor6Sfx3yC2JO2/gThr5Ps31gkEvdGosYZzhon8Svhxv1aXyrGH1ifKcH7yc0bovjOLPibm6lclANHPtVXbqS4QGGMqcduV+FqY1jzsjsuh4turJk5RdJOGtP8UuOQ7M5xIfo3h3Qc/0wkMEborh8RCFOp7M0bLHKjtoUDqD/BNSl2BSOUsaBY9M96NTVXsWZq1ElvDtyejY8aL8zdt0a6EGBAhBcdEFUNy2VnnZr7TQnLK1FJ1RbVr8Hx3RRqovKhmcBNFCe4ZCo/J0bANbPEEhnjrqL8+mcUQG+71XRJvPP50AkGDtdY33puDue57MCmXEp0ResfT0DNajb7HbluBcbZ5CZgnXdKPqAxWUdhSzrlqJcs/C2Kfb X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(52116014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gDymRz06GBCH3t7cNxtqi5AU7JRs+IAxQPr2pqfnE8DY4RNRy/r0crKBUbG/7WvHaf2eWT9Qv+wBDR6oCCScoDZJ5p3iuQJ7BkXBGCJ7m05P/ILOc/OaECPPxcVLkkzNOVDyDCTDG8Y1yL3JnIwdnppY83PdK8EFsEZrm56qMkRRIxAkhlkX0isr4KjZetu6wm8KSisWcRmz10VdTN5o6Eof2BV6ojFQLGVfSq3QZRPEHP3E/qeCPMpMrlaFk+aZ9Wh+adTPB6QKrM50ikjonMPYDnDYUvD91cJ/8XHzFofo+lGVUHI/U9UfQJXDmRUsoSth8UfVD2TdGlZ8KruqYujxD8+5UIaThangFkHNQ7M86KiUK5m8msH4hVsI9Jb37mNo+YtTRfmfiF45c6WksFC+ZrUobGeVaueGPIopq7h9y7Oh9Zn0Dee9n01bhocm3xMDEQ2kFkWmmtH1fy7afo4Kt1Nybjj5D13dVLbKXXP4CbAY4J+QHOqEa6ku4+KV53igl53uEJvFYTaXax3zQbytxnPqFyeUDvC2gMAEWfLWPR301aI+HNVA/EBoV6RV7Au6nwEXqG2sHLn80NF7tL/YNDNShGZ1vFQ8G03wZFY8czVARCkIDFze1I0Pse0ct+0vTO+tlwz7QCyNovoVkfFpPtkpgDj2Td+j/ULQYIaraTrNH6i605d8ZOroNR7egNwktYgZQn2JL7c+jYMrWx2m7kcbdjZ+KDb5FO5FPbV0zmnGG+jwzaFU1eBrCdZotrnCFpkET5ShakIrK7BJTryLiyI9hZHbrlzQs28KQ2a7bLCFQGw4IC5CJczepBZAEaaPNXwyQfOlAGpE+MEiaqvtCny61YZwGeyiQmhCqV1JOhhjY4jAsXvgh84BHHPOpi5OhbBzw33M4TjIHT1XOeClCDUKoq2qC3NXvFqQ0QKVLijVPME5KKbOdsCcs26D28k23iVwKH6v1P4m8HVH7GwWmV+jGppnuuwGUFqW0mSKpPGOujBxNfF0tC+5IG+tZqkyaIcIz/YwSy0N0qEqANTp/IuxbreESQ/YryFQCqCcrZO+bIu/VVCy19j++l8TLy2k2IZkMOgzgWQ65dJCrQ/8/yqQI4bMAjrCwd69PHXan0v9EltoYIjqwocmlVL6276PjsQU36voEqrGFG7fJqk/Sgd+cO1r74LPcf9pnvwDzEpnhlPViW5fUxSzKLUZrfsnamGTiWRWPKiIHkaXxC6dIkkSsNfbnUxhFogeZlWMJp6x8Ii1KxEkuymG8U3gSsl93oajdgmXsf3fYcjYQJQokY0G2Pxg1f8ZwjIcCp+9RQycoP5UgySgThnrCDenxq+AbSUors7hGlmzhk+dqFJdQOK//7yTqAqkCtsCN5aNkUdqJiWoo+W33/a7G1wm4PnT1dNCDyPp+T4PMeFvq/seuooEBTz6CpY0WJE/3KPlS3cQ18iCnMwDUfPKCt89FKojYqlbS4qpUsmk4g76VkHvA0TCrMV/yYwaucMH/oTVkg9y67/7qRabgP9yXMpPgZmul7BvWOy5i58lO1+jtySOvW9uItKgXRbBoFfhczibNeu+3dREsq1fQe4DufZTPa85V5mc6tY3YGJhe1dzLHm5wtioYy8eZ7j7V7tV4iE= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: ebf135ca-f1ff-4deb-a5fa-08dd5b6ad226 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:50.8476 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CMDqNUQQ89PmMfJSaW0o0mWFHLBZNZRcmT71Q+8UsEU97T4ZE0Op4C/dtrBcZCWyxLFBtw7PhHSq+Ah9TGuk4xZySwgLJ+wC91BXG0iMmW0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7931 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142058_468397_8AD167B6 X-CRM114-Status: GOOD ( 20.25 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add split CONT mappings support in order to support CONT mappings for direct map. This should help reduce TLB pressure further. When splitting PUD, all PMDs will have CONT bit set since the leaf PUD must be naturally aligned. When splitting PMD, all PTEs will have CONT bit set since the leaf PMD must be naturally aligned too, but the PMDs in the cont range of split PMD will have CONT bit cleared. Splitting CONT PTEs by clearing CONT bit for all PTEs in the range. Signed-off-by: Yang Shi --- arch/arm64/include/asm/pgtable.h | 5 ++ arch/arm64/mm/mmu.c | 82 ++++++++++++++++++++++++++------ arch/arm64/mm/pageattr.c | 2 + 3 files changed, 75 insertions(+), 14 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ed2fc1dcf7ae..3c6ef47f5813 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -290,6 +290,11 @@ static inline pmd_t pmd_mkcont(pmd_t pmd) return __pmd(pmd_val(pmd) | PMD_SECT_CONT); } +static inline pmd_t pmd_mknoncont(pmd_t pmd) +{ + return __pmd(pmd_val(pmd) & ~PMD_SECT_CONT); +} + static inline pte_t pte_mkdevmap(pte_t pte) { return set_pte_bit(pte, __pgprot(PTE_DEVMAP | PTE_SPECIAL)); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ad0f1cc55e3a..d4dfeabc80e9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -167,19 +167,36 @@ static void init_clear_pgtable(void *table) dsb(ishst); } +static void split_cont_pte(pte_t *ptep) +{ + pte_t *_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); + pte_t _pte; + for (int i = 0; i < CONT_PTES; i++, _ptep++) { + _pte = READ_ONCE(*_ptep); + _pte = pte_mknoncont(_pte); + __set_pte_nosync(_ptep, _pte); + } + + dsb(ishst); + isb(); +} + static int split_pmd(pmd_t *pmdp, pmd_t pmdval, - phys_addr_t (*pgtable_alloc)(int)) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long pfn; pgprot_t prot; phys_addr_t pte_phys; pte_t *ptep; + bool cont; + int i; if (!pmd_leaf(pmdval)) return 0; pfn = pmd_pfn(pmdval); prot = pmd_pgprot(pmdval); + cont = pgprot_val(prot) & PTE_CONT; pte_phys = pgtable_alloc(PAGE_SHIFT); if (!pte_phys) @@ -188,11 +205,27 @@ static int split_pmd(pmd_t *pmdp, pmd_t pmdval, ptep = (pte_t *)phys_to_virt(pte_phys); init_clear_pgtable(ptep); prot = __pgprot(pgprot_val(prot) | PTE_TYPE_PAGE); - for (int i = 0; i < PTRS_PER_PTE; i++, ptep++) + + /* It must be naturally aligned if PMD is leaf */ + if ((flags & NO_CONT_MAPPINGS) == 0) + prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + for (i = 0; i < PTRS_PER_PTE; i++, ptep++) __set_pte_nosync(ptep, pfn_pte(pfn + i, prot)); dsb(ishst); + /* Clear CONT bit for the PMDs in the range */ + if (cont) { + pmd_t *_pmdp, _pmd; + _pmdp = PTR_ALIGN_DOWN(pmdp, sizeof(*pmdp) * CONT_PMDS); + for (i = 0; i < CONT_PMDS; i++, _pmdp++) { + _pmd = READ_ONCE(*_pmdp); + _pmd = pmd_mknoncont(_pmd); + set_pmd(_pmdp, _pmd); + } + } + set_pmd(pmdp, pfn_pmd(__phys_to_pfn(pte_phys), __pgprot(PMD_TYPE_TABLE))); @@ -200,7 +233,7 @@ static int split_pmd(pmd_t *pmdp, pmd_t pmdval, } static int split_pud(pud_t *pudp, pud_t pudval, - phys_addr_t (*pgtable_alloc)(int)) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long pfn; pgprot_t prot; @@ -221,6 +254,11 @@ static int split_pud(pud_t *pudp, pud_t pudval, pmdp = (pmd_t *)phys_to_virt(pmd_phys); init_clear_pgtable(pmdp); + + /* It must be naturally aligned if PUD is leaf */ + if ((flags & NO_CONT_MAPPINGS) == 0) + prot = __pgprot(pgprot_val(prot) | PTE_CONT); + for (int i = 0; i < PTRS_PER_PMD; i++, pmdp++) { __set_pmd_nosync(pmdp, pfn_pmd(pfn, prot)); pfn += step; @@ -235,11 +273,18 @@ static int split_pud(pud_t *pudp, pud_t pudval, } static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, int flags) { do { pte_t old_pte = __ptep_get(ptep); + if (flags & SPLIT_MAPPINGS) { + if (pte_cont(old_pte)) + split_cont_pte(ptep); + + continue; + } + /* * Required barriers to make this visible to the table walker * are deferred to the end of alloc_init_cont_pte(). @@ -266,8 +311,16 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long next; pmd_t pmd = READ_ONCE(*pmdp); pte_t *ptep; + bool split = flags & SPLIT_MAPPINGS; BUG_ON(pmd_sect(pmd)); + + if (split) { + BUG_ON(pmd_none(pmd)); + ptep = pte_offset_kernel(pmdp, addr); + goto split_pgtable; + } + if (pmd_none(pmd)) { pmdval_t pmdval = PMD_TYPE_TABLE | PMD_TABLE_UXN | PMD_TABLE_AF; phys_addr_t pte_phys; @@ -287,6 +340,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, ptep = pte_set_fixmap_offset(pmdp, addr); } +split_pgtable: do { pgprot_t __prot = prot; @@ -297,7 +351,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pte(ptep, addr, next, phys, __prot); + init_pte(ptep, addr, next, phys, __prot, flags); ptep += pte_index(next) - pte_index(addr); phys += next - addr; @@ -308,7 +362,8 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, * ensure that all previous pgtable writes are visible to the table * walker. */ - pte_clear_fixmap(); + if (!split) + pte_clear_fixmap(); return 0; } @@ -327,7 +382,12 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, next = pmd_addr_end(addr, end); if (split) { - ret = split_pmd(pmdp, old_pmd, pgtable_alloc); + ret = split_pmd(pmdp, old_pmd, pgtable_alloc, flags); + if (ret) + break; + + ret = alloc_init_cont_pte(pmdp, addr, next, phys, prot, + pgtable_alloc, flags); if (ret) break; @@ -469,7 +529,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, next = pud_addr_end(addr, end); if (split) { - ret = split_pud(pudp, old_pud, pgtable_alloc); + ret = split_pud(pudp, old_pud, pgtable_alloc, flags); if (ret) break; @@ -846,9 +906,6 @@ static void __init map_mem(pgd_t *pgdp) if (force_pte_mapping()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - if (rodata_full) - flags |= NO_CONT_MAPPINGS; - /* * Take care not to create a writable alias for the * read-only text and rodata sections of the kernel image. @@ -1547,9 +1604,6 @@ int arch_add_memory(int nid, u64 start, u64 size, if (force_pte_mapping()) flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - if (rodata_full) - flags |= NO_CONT_MAPPINGS; - __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 5d42d87ea7e1..25c068712cb5 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -43,6 +43,8 @@ static int change_page_range(pte_t *ptep, unsigned long addr, void *data) struct page_change_data *cdata = data; pte_t pte = __ptep_get(ptep); + BUG_ON(pte_cont(pte)); + pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); From patchwork Tue Mar 4 22:19:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 17163C021B8 for ; Tue, 4 Mar 2025 22:32:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/KDHg9m5RQuafqimCOTWucTmGtJk6oGY9o9n630bLHs=; b=May7lLANucCuB1cKpBfyxBYgXt cezjyKyUdakJGIApSVK2dCw5/EHu7rTcVqbeSlOc9IuUBz1mELX3d2Jjqk2wjloxzBnREfrhfuRGk mMsseqkCFDc07tTFXZ9gjYJBLgngvJ0QQEkrUQBiywuSoDKotCJbYB5GjQtjVaudzb7gO5RfDfDEx xha4Zwe4xPbsL7cCmxTRD9aAh+V+1NSsMqz0+AH1hInYwq0uf/9RRUx8c3yPrupinY3wrOeItwnG9 AGY5mSwwQOv1kIrOhWXJ5H3i19fMnrHDUY6nL8VQpEFNaMJ1HQ2aw0uYQsf/W/eWPqgTYTYHsC8gh SOx5R6EQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpanx-00000006QSI-3W20; Tue, 04 Mar 2025 22:32:01 +0000 Received: from mail-dm6nam12on2070b.outbound.protection.outlook.com ([2a01:111:f403:2417::70b] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadR-00000006ORJ-3Kbk for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:21:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Xvfe0MqKb51Je9wLEh1TiPUjnzHUTGLw1uGIOZ/1CXK8WQ0Fd9x9l6q+vKVSp9LV++KWfJU1cI/NeZCP+PmWCQ/dc/+MQ7VTKO4gmqtxFS3bu//emHO5K351RCcUHNVe7/ugOCfbcHZ+dbzecbdNnM4DCL2a4UNntXqde57+e/TLMRPRETdH3UTe3aFI3tAhL+zGUisYzBTGnCqyjwlMqn3R083KMBH9QLhAoFTFsYr+SCsHaag/vZoexprAfPHISl7/wGAU4tnwm7oIJqzjp+v6H61iG4dWflNAnceqQGp8wUR7Rfn3qA2xcMHWmfb4U91M1nFJO3tGudWKBqIEtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/KDHg9m5RQuafqimCOTWucTmGtJk6oGY9o9n630bLHs=; b=qno18n2StlAoHY9zoZbZoxZf2Ww9hQrIo2naxABDfzFSj3eOL+Me5h7ohyktT9lbe76x4qwZGaR696lyraRk89XrxOoSZtZcQ3KJfvSSRIsrqdQUTYYSYG+qItE139CYMFv+oKmsK9+FZe5L0xEA3heG0QbjOXLJmfsLeXaH2rCSPV3HibJ87P/dEFyzSXBRIR4Y802wqoJ1z1miesm65WbteKEzDzmm20Dv5NqxrPoI16SEfI4NrWNet880uGpm61oXhRbTG1Z3yYHKvTdwOh/dg/CsesT/f37PT3TQNwEd6CipiH4gLwUklvGXs6W4NC2/Jlsx/Y7uQwLbRGLecw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/KDHg9m5RQuafqimCOTWucTmGtJk6oGY9o9n630bLHs=; b=ZDBsuHd+Lx9aHQ25KfjeHLQ34nBSj05D4Z0L4oMyMYUXczme1Rxy5j2le1EQsLX6qZXLOCY2CzVlDkfTSZAI9Tw/+D49JOjJvH6EXhIn8rIk6F3EaE4a0JUuFoqxAcVuNsIDntrG/CCQVGWMM3G15mZKjbjI8q/05PdVMyxtlPM= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by MN0PR01MB7803.prod.exchangelabs.com (2603:10b6:208:37e::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8511.17; Tue, 4 Mar 2025 22:20:52 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:52 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 6/6] arm64: mm: split linear mapping if BBML2 is not supported on secondary CPUs Date: Tue, 4 Mar 2025 14:19:31 -0800 Message-ID: <20250304222018.615808-7-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|MN0PR01MB7803:EE_ X-MS-Office365-Filtering-Correlation-Id: 31ef06aa-5cf6-4201-e18a-08dd5b6ad2f8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|1800799024|52116014|376014|38350700014; X-Microsoft-Antispam-Message-Info: ZyhBeGBfy9khsjwEAStR8dKkd13Ad2jf+MJHqxQvTqH7Xj6LaA3PFpgRmLeyuxQL+xkaFR0boyY2dXQoa+8zCIk9rv4swsH6NIDfhW+tRx0R7f+5vbDBcj7jwzEB9buABmL1Go65ZTFY96uM/mygr/+YCZA5d/kbKmtL3I8a3F9BQA3rwhJOq3L5dTROkUlodYsXX373/RKUKwq9thIfWuJsdohruhrSqkzg7pJtQFS7ikuoyGqsBZfdoZXW/u8FKJiQ+xhQYW5Wbkuxs67PzclkJK7SD/KTsrrFLKQ1JXwrWioIXUMb6yT2dr9ltA5FwqKDKDyMBIAecdWqu5oAcGBVzZ9dhpxqrYOyJY5M+zqmtNXGZukYXaknj1IpVIWstl1UWlfN1HPVjERrNvUJiKh4YzTYsjgFKmY2yP+B3DjUdEDuTzwwzMuoa5RaD8uYCF/bOszX8vXF77YQr5t9srXSfEi7b1gV9Ufz1eAihCfCWiONwmzV72xcEm2YU3buPrrhBduJdHqIVF7ZHdrG4cbTtKKmUqUexaXQU/vKxvEwM2QWSG2XEneexvmKAJYJ6+PTUV9dHlgYNY5FwpqzyUCjfEFFuL2JpawuXz/5REHMr2a3RqIQe+eNwaKPYxrSRuoiPhN6OEu0Ggcxm2bFHYTK57vIThKJVf5GsOrU09WI7TviOSfwDU6tKbNGFgfCPoQiJFNTrkn0tiLIBKa+Ndua6SeiW90+Wx739Z/KwhDyhgOxxFmD+0ZAMTtWBIz8vE7sNUUKPW7i74CQuD0HHFsTpRRMXCDHIjc7uA72vz0QUw65nW5rKJK46cJO93LQun7PhTcGjqBV6KsY9F8rbRW3pcdfaP5MObLAfw+gOHWhnPqFHL4zafGswFAL3Z5+VtqJCleH23/KOMZH5SfLd3LlKYpPHzvnwxyiW1uYFNSgnqY7DHyRKAC8+l6TklEvfD7AEwLbsWEpE+BhwS1yi4CzJLxAE2T+i1ovEgL+QqoqU/g0POaaKDxI7+Zvo3hf5HkUTqc6+1BgzhlnQxiAVb97QIBu3+WHYDRAHdlHXKVID2m6z1PuToHczv5ezVUHbqsMooHgC5nL5B7MRpZTsCFVeQj8ClodUK5GU+HH8CyjfVLBKV5NoWsgThvd99r5ixtVmoTlOGpt0ZlVGw9BxJ+kZfaAeK1DHEvoq40bfuyNUKe0UUu4LZeHy5lgoBBx9jwxtCSP8+ihsLnA+XBHuXAbQkwpH2u23WMFSIKzwX0CBIy4VBEVqBAhfO/5MlRq0LFoUmx6BdhYFmj0uyXBLPHyTtA52BTCaivoWjPt6F2bkt5ofmO0PRsSth6gAUZXmV3NVoSyDPzemNOWMZzrXo+PbVr20AIDJJFXTbhQ94V+J4FKP6m/9KBwqOzQ5AHL5mMTjrfcAn7OCL07Im9AY26V+KFJbKRHHzjVlWZUh1C9w8PQCZzpxBwda9XnZ1mB X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(1800799024)(52116014)(376014)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TpBIXw8L9Fv3jCzNUeeVOphRXwzzEZ45VAt7M21AgjkJ6EcdSJma/N8yPDKlYgXTpz9bYkgQj6jsL+rDifeLYs/xO89RLNXEmRlSf8fekJv/cW/cSIi07ADB4LLOJadqZ+yMEEfwTYSxcDdrtox4HuEDImggoZS3+AGjptBKeLWs8nlH8/UDZTqEER9PkIinTMtgcqAWRGgT13zDUAt51oLe99x3GMZ7t+oYwrhS+btMM0z24vz14Ted+JjkbvksEBKbwWbYPDEn7+yx7K8Vcs+JnJJN9lQgrB+VHubG3LZp0c0wLgCT/rkydPXhrwrEJoPEPjueJ5v90agfW9gZi+f8WyZuDnsGr0Tjd7tcQdDR8h1k5/IDMe12C8KFVa1cR2mUY/YXLX44YzqmjhNYHEEjO9KGJF6uxHHlftDy5h4zqdphNYwSp6m4X1cjhPstGfJkhrePyeJMAyPjxFbpYf3Coj4KXwOIJLAqUrTwyRPFIl4Kcr6Tpx6vM2sCjQ3tKAyZr3tvfSrsU5Zwdea/gCaoT54PfG7aAaq0Fqto8auyUQ8x8oMrMN/JOrFSCP8rNI8M1NSpIhWSMoJPo++1VUgA3JJxhbnGPB1mnxv0x9aCZbcxZoEARLfjUzzVwkUbQrP6oqnY3DTpew4jbpHeFMIprLhNPnMjJHH9sdnThHPwVYphiyfkCPAYfLK8f7N+XdQAmSnJB32xONerS4qvClBHiteXMe7jMxPHkJ0dzi+63zKj4Iq8DnANCJO2cNTkl2Lq6tvjuJcTEAOY2feJfoyQ501cRNHPL/g4W3WxvSFjcExOF82+qF0yaLPR22VBUaHSZFbqCTo/FKJ6dx1KLrUgYWQkNYixU0OfHSUkowfnCuRiA2kdtOE1X4O74jautW7ftkQd6d9ve/qcdRNet0g7nwJbd6zNAREP1AKlPmJTyz8put+FMLtePL1XtdpkUHGGfcQNTB2MbUdLVGFQEsyGwJC52kjsSeQPuI2rP3TpQn/rw5o8NuiMt+pwAw8Y/NxuJrQCIGyyJz0i0RTGvCZtil5Y0UcfL5T+G+fbwJUlTZFcGYr7j9GpVKqXmo4zSJGa7RejAiQGkRcV3bxPsiX7NL1KXKzx7LvfN9kjJWH3CZ3SlsLLdg04zkjp0rGaF8WbE79DibXr/CxWcxl78Xl3YD1XuJS0hDiZIZ/mU6Rq1lK2a6SkNP0szaj61iCbni27vhKFCdf2VLxtmoHznryVPNonySUiu7831j41xJEv+U/JxyicYZalPeClAfOGD17CLL5PfmWzL2zuzVz/y8xsL65v33QIEnfbcHaB0JZBGuQjUZ1GznTag5aRFRZMygiWFctlJlnGAYnktUskXkJDcjLBzYeddz0WVqCYAXNQHnkgj6pSFjtQAB74O65oQL22RpRFyEJsYnN8hxYoQC8Anl+795OiHig5vvUQtS+whGWX9TE5yv1y511PgoS4wxA2FzSNedFvIpG5eRDg0vPyxAZJXch9CGc3tCpMVSqO9bKPDTAzDRPFoTbgZCyQ91jE4r+wo+DxLhBZ+5IfWj4J8b5SXwIBlO+0maZzybMzNi9OW+64XYK0RZxyIk81n//xd+EgXa+d31uF0GdltPvwHbXDVt1WEApxai7BHmI= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 31ef06aa-5cf6-4201-e18a-08dd5b6ad2f8 X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:52.2224 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9Ye4M2usk+FDas2CIrPhWIR2rlctS3n02VF3DoPUAPmyY3VbMGNc78aBOTnJ9ULh0zSr/tXfNXDiTF+bKo6mPkem1hV1fLkpXmz5EWYzt4I= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR01MB7803 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142109_836782_A96E675C X-CRM114-Status: GOOD ( 25.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The kernel linear mapping is painted in very early stage of system boot. The cpufeature has not been finalized yet at this point. So the linear mapping is determined by the capability of boot CPU. If the boot CPU supports BBML2, large block mapping will be used for linear mapping. But the secondary CPUs may not support BBML2, so repaint the linear mapping if large block mapping is used and the secondary CPUs don't support BBML2 once cpufeature is finalized on all CPUs. If the boot CPU doesn't support BBML2 or the secondary CPUs have the same BBML2 capability with the boot CPU, repainting the linear mapping is not needed. Signed-off-by: Yang Shi --- arch/arm64/include/asm/mmu.h | 3 +++ arch/arm64/kernel/cpufeature.c | 24 +++++++++++++++++++ arch/arm64/mm/mmu.c | 43 +++++++++++++++++++++++++++++++++- 3 files changed, 69 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index d658a33df266..181649424317 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -56,6 +56,8 @@ typedef struct { */ #define ASID(mm) (atomic64_read(&(mm)->context.id) & 0xffff) +extern bool block_mapping; + static inline bool arm64_kernel_unmapped_at_el0(void) { return alternative_has_cap_unlikely(ARM64_UNMAP_KERNEL_AT_EL0); @@ -72,6 +74,7 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); extern int split_linear_mapping(unsigned long start, unsigned long end); +extern int __repaint_linear_mappings(void *__unused); /* * This check is triggered during the early boot before the cpufeature diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index d39637d5aeab..ffb797bc2dba 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -1972,6 +1973,28 @@ static int __init __kpti_install_ng_mappings(void *__unused) return 0; } +static void __init repaint_linear_mappings(void) +{ + struct cpumask bbml2_cpus; + + if (!block_mapping) + return; + + if (!rodata_full) + return; + + if (system_supports_bbml2_noabort()) + return; + + /* + * Need to guarantee repainting linear mapping is called on the + * boot CPU since boot CPU supports BBML2. + */ + cpumask_clear(&bbml2_cpus); + cpumask_set_cpu(smp_processor_id(), &bbml2_cpus); + stop_machine(__repaint_linear_mappings, NULL, &bbml2_cpus); +} + static void __init kpti_install_ng_mappings(void) { /* Check whether KPTI is going to be used */ @@ -3814,6 +3837,7 @@ void __init setup_system_features(void) { setup_system_capabilities(); + repaint_linear_mappings(); kpti_install_ng_mappings(); sve_setup(); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d4dfeabc80e9..015b30567ad1 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -209,6 +209,8 @@ static int split_pmd(pmd_t *pmdp, pmd_t pmdval, /* It must be naturally aligned if PMD is leaf */ if ((flags & NO_CONT_MAPPINGS) == 0) prot = __pgprot(pgprot_val(prot) | PTE_CONT); + else + prot = __pgprot(pgprot_val(prot) & ~PTE_CONT); for (i = 0; i < PTRS_PER_PTE; i++, ptep++) __set_pte_nosync(ptep, pfn_pte(pfn + i, prot)); @@ -258,6 +260,8 @@ static int split_pud(pud_t *pudp, pud_t pudval, /* It must be naturally aligned if PUD is leaf */ if ((flags & NO_CONT_MAPPINGS) == 0) prot = __pgprot(pgprot_val(prot) | PTE_CONT); + else + prot = __pgprot(pgprot_val(prot) & ~PTE_CONT); for (int i = 0; i < PTRS_PER_PMD; i++, pmdp++) { __set_pmd_nosync(pmdp, pfn_pmd(pfn, prot)); @@ -806,6 +810,37 @@ void __init mark_linear_text_alias_ro(void) PAGE_KERNEL_RO); } +int __init __repaint_linear_mappings(void *__unused) +{ + phys_addr_t kernel_start = __pa_symbol(_stext); + phys_addr_t kernel_end = __pa_symbol(__init_begin); + phys_addr_t start, end; + unsigned long vstart, vend; + u64 i; + int ret; + + memblock_mark_nomap(kernel_start, kernel_end - kernel_start); + /* Split the whole linear mapping */ + for_each_mem_range(i, &start, &end) { + if (start >= end) + return -EINVAL; + + vstart = __phys_to_virt(start); + vend = __phys_to_virt(end); + ret = __create_pgd_mapping_locked(init_mm.pgd, start, + vstart, (end - start), __pgprot(0), + __pgd_pgtable_alloc, + NO_CONT_MAPPINGS | SPLIT_MAPPINGS); + if (ret) + panic("Failed to split linear mappings\n"); + + flush_tlb_kernel_range(vstart, vend); + } + memblock_clear_nomap(kernel_start, kernel_end - kernel_start); + + return 0; +} + #ifdef CONFIG_KFENCE bool __ro_after_init kfence_early_init = !!CONFIG_KFENCE_SAMPLE_INTERVAL; @@ -860,6 +895,8 @@ static inline void arm64_kfence_map_pool(phys_addr_t kfence_pool, pgd_t *pgdp) { #endif /* CONFIG_KFENCE */ +bool block_mapping; + static inline bool force_pte_mapping(void) { /* @@ -888,6 +925,8 @@ static void __init map_mem(pgd_t *pgdp) int flags = NO_EXEC_MAPPINGS; u64 i; + block_mapping = true; + /* * Setting hierarchical PXNTable attributes on table entries covering * the linear region is only possible if it is guaranteed that no table @@ -903,8 +942,10 @@ static void __init map_mem(pgd_t *pgdp) early_kfence_pool = arm64_kfence_alloc_pool(); - if (force_pte_mapping()) + if (force_pte_mapping()) { + block_mapping = false; flags |= NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; + } /* * Take care not to create a writable alias for the