From patchwork Tue Mar 4 22:19:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 14001629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7A406C021B8 for ; Tue, 4 Mar 2025 23:29:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:Content-Type: Content-Transfer-Encoding:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bc8S2OTbZME42pyY8V87xNGR1wOTsn5d4wbfYh2jm/s=; b=E+ESfVojlUxoheH2TpH98mFiXd BFHJk7Z9psWcFVIfsMxuQ6XnZl4g4qRXTX4lYJxW1PstihE9gbVNy3z0/Dr3Be/Wk57posx5TM+Ky So9+vDux2yJUf8+fJiDsKG4zo7rRod0eaqV8jUAzpEehTQvLx1GggXrfp8IV6mVwhCaHxdnFjoBU/ 2MG5Bz8+I214Db1SaG/XeK3+uZhg3eEggltL7EX/qYq1Aaq3GfwpUeUUOBWVXkPblvDvqgesUGxdO PYE2iIlLpE4AhP5yXcSvOyc123G2ZBNgHhPdUgFXV0nKf8oknb0o9SO+dGBQcn2/BSmMDWz+zUBDp VKh2mmxg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tpbhs-00000006Wqc-3fFk; Tue, 04 Mar 2025 23:29:48 +0000 Received: from mail-eastus2azlp170100001.outbound.protection.outlook.com ([2a01:111:f403:c110::1] helo=BN1PR04CU002.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tpadD-00000006OM9-0M0o for linux-arm-kernel@lists.infradead.org; Tue, 04 Mar 2025 22:20:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ctmcsDz28ZWlzGRlPG96/CbChrHM1mmH9vKyvMLxESysPWp5Zof2rbngbx81HEDEH0CwoTSCD+T3SLUsZkpw/eLtoo9fz5aJ/sUGv03URB89Patnaladj/iCtHc64Mxo7/Qd42Qb/3Od0mWeIAeqjALi9dAiR/elDsrizDI5B1R4oe/+Ya1qBpEkFKaZQwNba0aP7HsguTJDugZGxGCbSDnbwinkjMoIMo+oW5tC+Wa6r+DavohzLGmOeRb7ps1CVGpnHG00+y0D+NfRAC8BOMYoaAOYEJj6EATxtGZePmEpAZIVi4ydqpI0zm5zvs7W49AQqtc+KhH1SagCOkm82Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bc8S2OTbZME42pyY8V87xNGR1wOTsn5d4wbfYh2jm/s=; b=YLxBFJsJVcBCWonl06ZSNwIr7pCuNBey+JcFZMAeYTEtI+afJ1Zgp767YTcANKiuUr014W7tliMtyZ8m99UNYw7r3Egh9vyR1RzXLesYIj2Bk5xASRw/KQnjRcS+iTYTzLgQiWheActcxpZ+6AHybn/kmdLKQKX33KhbrbebBU0tCltQ88y//A6svmLBeqqtABFNeKvKEtIw1p/UNqVYFAmHfJ9HiUQ2zpkfqJLHuyO+wImCb2AeE5BbY0fhA6y/ZJeZWub0JNT8i8H3kuA5RYsEWwq9hx8Tvz0F0bj5OLCwR2lW6/USdAUv9X3EkyjW2rVhAUuXVYgNGOA5RlyxtQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bc8S2OTbZME42pyY8V87xNGR1wOTsn5d4wbfYh2jm/s=; b=l95yYa9UaEDL7PgaSRZE6SzqrXnVcNVdFBfek0Y2GcSPLHw7mTP6NBB0bcNeywTyqiafUUeN+M4osf8TOJY3o0JUqPATfCypKjVpyUemmbr1BykbjwgD1Z9vG68lwJRcdN0ZsK4IM19973OnJCudEsR9iTpbJKthSqhMwBZnpm4= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) by PH7PR01MB7931.prod.exchangelabs.com (2603:10b6:510:275::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8489.26; Tue, 4 Mar 2025 22:20:48 +0000 Received: from CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460]) by CH0PR01MB6873.prod.exchangelabs.com ([fe80::3850:9112:f3bf:6460%2]) with mapi id 15.20.8489.025; Tue, 4 Mar 2025 22:20:48 +0000 From: Yang Shi To: ryan.roberts@arm.com, will@kernel.org, catalin.marinas@arm.com, Miko.Lenczewski@arm.com, scott@os.amperecomputing.com, cl@gentwo.org Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [v3 PATCH 3/6] arm64: mm: make __create_pgd_mapping() and helpers non-void Date: Tue, 4 Mar 2025 14:19:28 -0800 Message-ID: <20250304222018.615808-4-yang@os.amperecomputing.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20250304222018.615808-1-yang@os.amperecomputing.com> References: <20250304222018.615808-1-yang@os.amperecomputing.com> X-ClientProxiedBy: SN7P222CA0014.NAMP222.PROD.OUTLOOK.COM (2603:10b6:806:124::8) To CH0PR01MB6873.prod.exchangelabs.com (2603:10b6:610:112::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH0PR01MB6873:EE_|PH7PR01MB7931:EE_ X-MS-Office365-Filtering-Correlation-Id: 72ca7119-25e7-400d-f34e-08dd5b6ad0bd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|1800799024|52116014|7053199007|38350700014; X-Microsoft-Antispam-Message-Info: WmpGZg1mjm2zsnrtHGa1zMKDmaPY8t489ZRZT0AUaVcylGUMneQRI+uYuEq5Ch/rLFwiblQxK0IQnudleXP5h9drxL9s9OKUlaoiIHOl+J2AGqhBEm5yBAY5rwgheAe4wY5zp7W7uxhOQE+hP748C1RFTY83Q6q9tA/NkrICvi0J/CqSkpAhKLtQ/OmvTMDSncHJG7J3J2n3Va8wl+VELdYEpqB4q7FECFtP8bTmQO7rrLizX6NCupaAaFiA2s+dNJIKdkBrzKPR2zVcX012p9I+XtLfA4cFX9NJmNu8crPfEN0GrZwmYoJtp2GKFL8nSByJZGBTH2jT14W/F/bXEkdPvwugm9dh1RwHDQadAkq5k7eHkuGZnv+6oxMSu55clpr0w2kiNs8pbpr3fudnG0a484UQYDjOhIhRY8a+HN4vd4Sy0QUXE/TVrK5NHK7eEaqRrtCXBwrbqN6E6L6taTL/Oh5uzyDhlf/1cCWlw8Dz9D8k6A2ZIhf9E3XiRsgrMDVwuRoTjXGIPrqitfmuunwR79iROeGcfc2VSXIhZKWBBOWWlrXGHpDjngE/+b3+Q1IpmbXD52SagXKAmnKF1zkNPy4jXHAJspc0oYyNbRxq4eO2JzvvTlgw8sLJ/gDD9pxapQXYp6SsIbQeYT4ekrBKIuySkTFc/04HLS6GG9ptwT3+3AwNvPLTvvKQps1KXDOYDhovZ2QDe1na4Url6cWdI/6c+7hRfZwiM3ggwatI0HDqlVVRYt9XYQZB6Z8IFCYGjCNeyKBgRM5T6XeHbW9qKfp6Qpu6d+PAEBCc7GyMzXF2yuUq8/TgiYW0fdLyGn17QuutIyGL3DfG5ZiD6c+fwH0iKNkSbcs92F+K9IoJlbbBz3vT/RKcuxd28XgCzXYO7H6fKEjVPw2aETVvGvtF5POgJEZ+atCyz2k8LzRc3vwNyLTiS6jMnXtCSb5jtIJZntcl2u2v7qBJ0m4tILP1e2a0xLWCEK5zSH1Z5hel0qit2dAtvNwp/AbXfHLcYHqfbIcFWMAlX1INeFxc/32sWwqi/N2cgIFhn54dYwy0nIhNBOfG7h2FXaNOlNzzgKip8COmYTahyATCDshqMWpl3kfLX6BQjh4PMnHInSqjkuXQrPnx9SeOzhs3as0wXeLF519vk2deH80fLwJO8lJwhRrP6ZWXHdE6gKtWy7WHeNBmZcYuIIGNf+TZ4u9GT64Gv0vAffPCEB8yz/MYtNZXD8U1ofUkBJzSh8yUyc1zvYKbiqSjh301rqYKVw2wVmaiaQ9vsrZdX0fFnvrz3ol1LudiqsMd0d6auSuhzggMh370m0wez00gOHjhHTQmf6pJ4DNQbpRZmxPyMaBJMiYLiEXTBX81LsqM0F2DBZRh13JfAbuBdG16vTnnbY84sBygN3OOkP5o/GW2lvtnJ3YPbEk2iUM6QM66r/RQabo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH0PR01MB6873.prod.exchangelabs.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(1800799024)(52116014)(7053199007)(38350700014);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qgFq6QhRBOQ6J8oeEpGXvr17hwGbtmGHd9Mo2VtoWXQCTwCose8yfu5dPhPP7w+H9a3FRFVOebkwe/XCYNySLhOATkEyl63HjWK/gFIsrMQ1fpfDg03Vd5cTg8Vro333AxSQOERswn+QUmORVWEWBTllgSlES8MrAjWr67idMcGAy1kxqRuE0p+12CYScxrgUvHhVCztOJszFsWjWv6uNh6uK9swBYcx+vfOFrRRo4LS3C9yKILtS0UNCOL/8URjtc6riPkq5UUSUv0lMwXnzfrSQeBmF3/3WI89DxB9tMfEuaZ8O7APffqR45utWDuNFzBR+bV78ypo4PpNmhQDundAQMmon6AwOH5XVxCy33y5zEOo2L3QL8kpw2ua3hJy6JlVLU9PMOS5qta5nOFt423ezdYBtHrd6gxhGtUva/dSFVhD+vtfZPWMOQw2T0gjI14sgcAZHaeAMJ1BEs6pWHVYHjaPEj9dZVf8xM/UG69xKZZZ9izBocQlEwBrbb+4RLqxmPKIl7E0xy4fx13tskOcvW7FMsNtVoeHroDC0XeouDQF8koGMak09XNLyQZ9lsHTGNWjKwLZbd1P8RjMnraYKQaAW7dbSifDMrGQKe7JX3+FW8dEh4ZEn6/iHeH+MPyiofhjrvebhoBh48I5oIAJ5hFwg8tuNVOjws8NyYflMj06vQSLUJ4nuXapWxOa/smJdjroPIHd0Yw0WZSBrpE7XgQHthqWUpSdaqMCXZsqSUqFksPQ3kP3715Jq+o42kzdnkXHH98jgmmkyZPMJ7+0QOLWIGa9tFxqq5CA0pxnu+4mtsZteH0nQFoPWqDqW/2u/+IdlzPk5m0dc4m7njWL9wLj1VWJdvbRLzEOJ7wVpZR9A1hPObkz8GPuQx6MqJua6yaTyPlAAMgoF7xwBz5fiSLIhnDg0EEenmRjNaq1HppsCnNhrXUqgHvnwGFzKLt/V/Jz5KLr99bEj4aijuVyy+k9eiPbxl58wqeBseyFkI1HxIJzwstSAUfdxVO9Zigvtpm+NPRCL30Pl7lOSSp978UkIK4RPAgau76TqFlcMt758ChalX+Y4C1EumGKrypzq65wh+uF9iOpH/jE/wOtOvzQe0t8vNfCA4tNEZjSxwU+4N3NWOjqUy/ZhKEjtB4yFBpvVXVOmCMCd4EErAGUd7ibn7kN6IyQ3cVGuouNInYcNGYdBEwkHTdU1VPW6dq1FfgJ2Y0UKrOrRkweJd9z2hnf6VjgW1/L1Rb+mhovg9l6TseByEh8fYwhUAIEVxVz2WCb2gh6edsVvH93W3Nd0yy+D68PGe+x9gcZRKe2hxOc8pc2WVtnCV94/GFOaMWkCijC30HnWeJtFh/2qfIWvAjwKowZazKNJuYW1cO4IYE1VMvIG9fZXgnalYqTsNV7rcp5HGXr0HhgIqQpQ93MqLnXQQfhzAwv/wd2uBiz5+2DUrTooMIvpz/juagGfevKnqaNb690rliq5r+3TN1ZHpEGm31/LpzoiOisaM/Z8cefcbj4dwPHZZ3fR6dJHd0zCKAGd5gTiAfUECCa0WGV0/Oi9ZQkzcm8lBAfDb+X1b0peuPuzw3v3sPi9gSyOOh2NIM2nHryScueVPtCAu6zt/MQ5LqSc2ni0yaIEkE= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: 72ca7119-25e7-400d-f34e-08dd5b6ad0bd X-MS-Exchange-CrossTenant-AuthSource: CH0PR01MB6873.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2025 22:20:48.4636 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PC7AaB4426Opvqc210fk/En2zA+SU+jw5I90ZbzMdiFSN71dOIOt6b3J9FAEGuj34O/67gt+2zprkHjszE2W5kzlSguvJzURS3E8CRyi0w4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR01MB7931 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250304_142055_129073_10BA2338 X-CRM114-Status: GOOD ( 13.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The later patch will enhance __create_pgd_mapping() and related helpers to split kernel linear mapping, it requires have return value. So make __create_pgd_mapping() and helpers non-void functions. And move the BUG_ON() out of page table alloc helper since failing splitting kernel linear mapping is not fatal and can be handled by the callers in the later patch. Have BUG_ON() after __create_pgd_mapping_locked() returns to keep the current callers behavior intact. Suggested-by: Ryan Roberts Signed-off-by: Yang Shi --- arch/arm64/mm/mmu.c | 127 ++++++++++++++++++++++++++++++-------------- 1 file changed, 86 insertions(+), 41 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b4df5bc5b1b8..dccf0877285b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -189,11 +189,11 @@ static void init_pte(pte_t *ptep, unsigned long addr, unsigned long end, } while (ptep++, addr += PAGE_SIZE, addr != end); } -static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; pmd_t pmd = READ_ONCE(*pmdp); @@ -208,6 +208,8 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, pmdval |= PMD_TABLE_PXN; BUG_ON(!pgtable_alloc); pte_phys = pgtable_alloc(PAGE_SHIFT); + if (!pte_phys) + return -ENOMEM; ptep = pte_set_fixmap(pte_phys); init_clear_pgtable(ptep); ptep += pte_index(addr); @@ -239,13 +241,16 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, * walker. */ pte_clear_fixmap(); + + return 0; } -static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) +static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; + int ret = 0; do { pmd_t old_pmd = READ_ONCE(*pmdp); @@ -264,22 +269,27 @@ static void init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end, BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), READ_ONCE(pmd_val(*pmdp)))); } else { - alloc_init_cont_pte(pmdp, addr, next, phys, prot, + ret = alloc_init_cont_pte(pmdp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; BUG_ON(pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); } phys += next - addr; } while (pmdp++, addr = next, addr != end); + + return ret; } -static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) +static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; + int ret = 0; pud_t pud = READ_ONCE(*pudp); pmd_t *pmdp; @@ -295,6 +305,8 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, pudval |= PUD_TABLE_PXN; BUG_ON(!pgtable_alloc); pmd_phys = pgtable_alloc(PMD_SHIFT); + if (!pmd_phys) + return -ENOMEM; pmdp = pmd_set_fixmap(pmd_phys); init_clear_pgtable(pmdp); pmdp += pmd_index(addr); @@ -314,21 +326,26 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + ret = init_pmd(pmdp, addr, next, phys, __prot, pgtable_alloc, flags); + if (ret) + break; pmdp += pmd_index(next) - pmd_index(addr); phys += next - addr; } while (addr = next, addr != end); pmd_clear_fixmap(); + + return ret; } -static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; + int ret = 0; p4d_t p4d = READ_ONCE(*p4dp); pud_t *pudp; @@ -340,6 +357,8 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, p4dval |= P4D_TABLE_PXN; BUG_ON(!pgtable_alloc); pud_phys = pgtable_alloc(PUD_SHIFT); + if (!pud_phys) + return -ENOMEM; pudp = pud_set_fixmap(pud_phys); init_clear_pgtable(pudp); pudp += pud_index(addr); @@ -369,8 +388,10 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), READ_ONCE(pud_val(*pudp)))); } else { - alloc_init_cont_pmd(pudp, addr, next, phys, prot, + ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; BUG_ON(pud_val(old_pud) != 0 && pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); @@ -379,14 +400,17 @@ static void alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end, } while (pudp++, addr = next, addr != end); pud_clear_fixmap(); + + return ret; } -static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long next; + int ret = 0; pgd_t pgd = READ_ONCE(*pgdp); p4d_t *p4dp; @@ -398,6 +422,8 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, pgdval |= PGD_TABLE_PXN; BUG_ON(!pgtable_alloc); p4d_phys = pgtable_alloc(P4D_SHIFT); + if (!p4d_phys) + return -ENOMEM; p4dp = p4d_set_fixmap(p4d_phys); init_clear_pgtable(p4dp); p4dp += p4d_index(addr); @@ -412,8 +438,10 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, next = p4d_addr_end(addr, end); - alloc_init_pud(p4dp, addr, next, phys, prot, + ret = alloc_init_pud(p4dp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; BUG_ON(p4d_val(old_p4d) != 0 && p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp))); @@ -422,23 +450,26 @@ static void alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end, } while (p4dp++, addr = next, addr != end); p4d_clear_fixmap(); + + return ret; } -static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { unsigned long addr, end, next; pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + int ret = 0; /* * If the virtual and physical address don't have the same offset * within a page, we cannot map the region as the caller expects. */ if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) - return; + return -EINVAL; phys &= PAGE_MASK; addr = virt & PAGE_MASK; @@ -446,29 +477,38 @@ static void __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys, do { next = pgd_addr_end(addr, end); - alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, + ret = alloc_init_p4d(pgdp, addr, next, phys, prot, pgtable_alloc, flags); + if (ret) + break; phys += next - addr; } while (pgdp++, addr = next, addr != end); + + return ret; } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) +static int __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) { + int ret; + mutex_lock(&fixmap_lock); - __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, + ret = __create_pgd_mapping_locked(pgdir, phys, virt, size, prot, pgtable_alloc, flags); + BUG_ON(ret); mutex_unlock(&fixmap_lock); + + return ret; } #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 extern __alias(__create_pgd_mapping_locked) -void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, - phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags); +int create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags); #endif static phys_addr_t __pgd_pgtable_alloc(int shift) @@ -476,13 +516,17 @@ static phys_addr_t __pgd_pgtable_alloc(int shift) /* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL & ~__GFP_ZERO); - BUG_ON(!ptr); + if (!ptr) + return 0; + return __pa(ptr); } static phys_addr_t pgd_pgtable_alloc(int shift) { phys_addr_t pa = __pgd_pgtable_alloc(shift); + if (!pa) + goto out; struct ptdesc *ptdesc = page_ptdesc(phys_to_page(pa)); /* @@ -498,6 +542,7 @@ static phys_addr_t pgd_pgtable_alloc(int shift) else if (shift == PMD_SHIFT) BUG_ON(!pagetable_pmd_ctor(ptdesc)); +out: return pa; }