From patchwork Mon May 31 08:45:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12288905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18252C47082 for ; Mon, 31 May 2021 08:48:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D2706611AC for ; Mon, 31 May 2021 08:48:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D2706611AC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pCZ4EC8KNpg4PsqBW7em3i8MTWB71YdLNMc/JSMVZIo=; b=nJCpnKTGnpJUAc 2NRoVamz/tpO0NSyZH1RxFzXIfAw6MWM92my35qKLzKjqdzaOfDLn+qjySQjWnuT3YUW3FmvkqExA 3hnwNa26cjKknmXChl7H0GAOZ04sKCBgNhGz+SKTqUj5JkbY9Cec0HQCvVl19b500VpTFxnPqvdGM 3f3igco79MBdnvAmeTn+k6JbyCLqwr8gkVQ8qzqfNnFLfQUj+LT/GyQamycHYUYwh08QqSMDHMwX1 7Zr9HwRWIQ/oVm3P607L/4jqlbpkndArww61clqdEqnljIvnqcn8X0iGpRHZG3s1dzFv7qkr1lBJT CXMALLhbG5uHy36pDpmQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZ2-00BWAB-OW; Mon, 31 May 2021 08:46:24 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndYp-00BW8f-Vx for linux-arm-kernel@lists.infradead.org; Mon, 31 May 2021 08:46:14 +0000 Received: by mail-pl1-x635.google.com with SMTP id v13so4860595ple.9 for ; Mon, 31 May 2021 01:46:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=U0waVifzBAtCWMpyTCX/XUC7ce55I6cPGggbE8PRRcE=; b=FUPxp1dBJwvmNZfKJDJhHoOZTJ/ufpAULm+4Hpm2nYz9rsJaSZqzYKz9xUtyCLccRs jifH4NEsoM5sNuGYv+T2ltVuAQDnY3H5lqkmnMbLWtut6dcFB1ayutGdljmxU1IlGWTg pTZAR3/I+/IlPjXN0DF4g4+xUBKlfRSR+MJHZiPvsB2V1e8hwvMOVbJMELuYHak9GAe7 jYyqrs0CKPK2nmFmwnQzK0olieEMMLrq0WrYxSNSEkp8uIORiQmHoqsm1g6812qTwNDb 6tEDRfQf5DO/SxAEYQ6QouK6/Sy3cegzmL+8Z7sH6t9TZgdhPMnrfBkP9QG9GQM5c9VI Ga0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=U0waVifzBAtCWMpyTCX/XUC7ce55I6cPGggbE8PRRcE=; b=td4iVlK9Uj2nhncIU80nNaTTP05Vco1lqjgLRL/CT5E8QTxdxwDyxq9z3rBVeOQikn lRmtf9iiEU9pt6rJ/HAl2PN4qRB68bzr5stzIOatjgBCi4OvYwiMrpjZlq29uLjH7uzq 4o3kCjvCepYQR2Vj9CpgTN0zxj8HKz84WJH9Kb9e0gu+2dw5iSu7kbg1Mi7zw0LZC+Vm qLCY9v/5dmfnA1UrLNfCllfSHEcO72ym5CU1P49xmvVcAF/ItfdGz9aUYc6uLYgDsht0 ssFdODV7VCvYOP+SJXPsw2UbavCTA4Q3ifptnpUoj2Yd+fFC+HknmBns1jsUWiXw4sJG hhuA== X-Gm-Message-State: AOAM533179aVfZN/sWHoVl94EfDH7uUISxQiwEXmbJ3y0v9NRJvQpJQV Kb5qGPZAlWRwQdVaMn2/ksv63tdz5FdBzuw= X-Google-Smtp-Source: ABdhPJwH6wLHQbE3IKiiFGY7Zj1jKFMhJWBZSdXwGrStlOZunKqufpGmJxqkAOQwgJozDElLU2oIRQ== X-Received: by 2002:a17:902:c1d2:b029:101:656b:8c06 with SMTP id c18-20020a170902c1d2b0290101656b8c06mr13079691plc.77.1622450770790; Mon, 31 May 2021 01:46:10 -0700 (PDT) Received: from qualcomm-amberwing-rep-18.khw4.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id fs24sm5125677pjb.6.2021.05.31.01.46.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 01:46:10 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv3 1/5] arm64/mm: introduce pgtable allocator for idmap_pg_dir and init_pg_dir Date: Mon, 31 May 2021 04:45:36 -0400 Message-Id: <20210531084540.78546-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210531084540.78546-1-kernelfans@gmail.com> References: <20210531084540.78546-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210531_014612_181321_47ED33F4 X-CRM114-Status: GOOD ( 12.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When building a pgtable, __create_pgd_mapping() resorts to pgtable allocator to alloc mem. By introducing an allocator, __create_pgd_mapping() can consider both idmap_pg_dir and init_pg_dir as a memory pool, and get memory from them during the building of pgtable. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/mm/mmu.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 6dd9369e3ea0..f4fc905718ca 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -90,6 +90,28 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); +struct headpool { + phys_addr_t start; + unsigned long size; + unsigned long next_idx; +}; + +static struct headpool cur_pool __initdata; + +void __init set_cur_headpool(phys_addr_t start, unsigned long size) +{ + cur_pool.start = start; + cur_pool.size = size; + cur_pool.next_idx = 0; +} + +phys_addr_t __init head_pgtable_alloc(unsigned long unused_a) +{ + unsigned long idx = cur_pool.next_idx++; + + return cur_pool.start + (idx << PAGE_SHIFT); +} + static phys_addr_t __init early_pgtable_alloc(int shift) { phys_addr_t phys; From patchwork Mon May 31 08:45:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12288911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1279AC47096 for ; Mon, 31 May 2021 08:48:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CC9D960FEE for ; Mon, 31 May 2021 08:48:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC9D960FEE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NO1jfQblCVM3lIbCu/jTdwoHgNRwko06dtpW4ZkJ8OU=; b=rt3zo8ymLe5XQf yk7g4JK9SWEv9CzRn0PFpAPW0fONhkFl022VmtVMYJJt+DIwwhYXEe4uIvuMwGh2R2DdKxa69bBew IghavhXfDej1du72Oyu4AmPenpJINHpjBF/AnWTVnF6SvgeGd7kempD46mJ1P88dCHPH41rFXSSGP 5Sc8DAfw9FvGG4hqWqIMtqODCptwd0HxojFcLL0Vl/eYYO2oIsz9+g56kha37TjgMlE3C92CNroVT e17l8fz0IUA8luXmu4cvD4jdxEmZ09pdhMDGFgwmXXWhOXeXZu6lz4eXpmzcBpZFJarG1HjnY2p4E 6c9dnfYo5Ey97dRWPqjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZF-00BWB2-FW; Mon, 31 May 2021 08:46:37 +0000 Received: from mail-pj1-x1032.google.com ([2607:f8b0:4864:20::1032]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndYy-00BW9m-La for linux-arm-kernel@lists.infradead.org; Mon, 31 May 2021 08:46:22 +0000 Received: by mail-pj1-x1032.google.com with SMTP id 22-20020a17090a0c16b0290164a5354ad0so945815pjs.2 for ; Mon, 31 May 2021 01:46:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2F86xve07XA7uY1FsbnVFLtV22c4Q57gB5yrbCwHOc4=; b=UnrcLB8bLa1eaU6ZB1j4Br+m2s5fCISs5QA0ZzKOWgJ5jGo5Kza902/xRGM0MYm02K FLP0b9sKGbKCDlKE5Nzl/l7dTtTWClXmttrvB0CwPj/iII6eINU3QLUl4cX6GN53AbIf bCs4Kb5ear5ZPwf3MAJaUwzkUTnVm2VWOE4KU4aaiun7tMQPICHQ5zVN3716it7xJWK9 8eWbwMYcndaBBYJRSejPYSVtPXe5zxttcJxWEYUvFDDLNNNgvC7JnHFOHlz3K8qhlONh jQ3g5A4q83+RY3czffV39RLA3UIzKDLQTpb68iEAFP+XclnugrUCj8MKmTNrxVU3bH8C QbQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2F86xve07XA7uY1FsbnVFLtV22c4Q57gB5yrbCwHOc4=; b=WlTICbIDKYNNkDcoTAJ9AtlwusZMPcdgJpyHy2fcHpNyAh/xEu6gdVJwulwdf74B/p QGQEWicJ9CPDPKJJ5tA1KHK0R51nCeeYSY33H8qQ8/uSWkpZVJLJCD/X8zdUpylia714 nhmfKba5rmLdEgSln1uNXRt66fCxF8GJxpG6IDEuGe9Y4vS6/EN+MbTKwhTTKfOD+CGb 2DNZNPiz90XIUrPldU0MDY8Mkqzs27QfpQpVuiurTeHe1bVg4eRqlaWg+GmQ6xOGNPNe gtIa0AaL/OmdOrmd6DsP5BohUT9XJL34ENmfqiqjrqTNM5HOubpT/qAhjeiYNuyUiY05 3xWA== X-Gm-Message-State: AOAM533YTNt42WNheQJ2nnugmiHA4JvJyasgN+Ljn2YybmSmQ73u8bDq IYor13RLqIntQRxrgWhw24tLmGiJNexuPOE= X-Google-Smtp-Source: ABdhPJyzNhYZQp/0j9tklWJpgprZFB6KYspaPyLBT7aVtdP7N0gg9S/C7SEgLeDALPUEpJmf5+F4pA== X-Received: by 2002:a17:90a:520f:: with SMTP id v15mr17794149pjh.23.1622450779476; Mon, 31 May 2021 01:46:19 -0700 (PDT) Received: from qualcomm-amberwing-rep-18.khw4.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id fs24sm5125677pjb.6.2021.05.31.01.46.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 01:46:19 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv3 2/5] arm64/mm: disable WARN_ON() and BUG_ON() in __create_pgd_mapping() if too early Date: Mon, 31 May 2021 04:45:37 -0400 Message-Id: <20210531084540.78546-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210531084540.78546-1-kernelfans@gmail.com> References: <20210531084540.78546-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210531_014620_737790_B59B2530 X-CRM114-Status: GOOD ( 17.79 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch is still one of the preparation for calling __create_pgd_mapping() from head.S When calling from head.S, printk() is not ready to work. Hence define SAFE_BUG_ON()/SAFE_WARN_ON(), wrapping around BUG_ON()/WARN_ON() to protect against early calling. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/mm/mmu.c | 42 ++++++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f4fc905718ca..fe16f235d1fa 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -40,6 +40,7 @@ #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ +#define BOOT_HEAD BIT(3) u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; @@ -139,6 +140,15 @@ static phys_addr_t __init early_pgtable_alloc(int shift) return phys; } +/* + * printk is not ready in the very early stage. And this pair macro should be used + * instead + */ +#define SAFE_BUG_ON(flags, cond) \ + do { if (likely(!(flags & BOOT_HEAD))) BUG_ON(cond); } while (0) +#define SAFE_WARN_ON(flags, cond) \ + ({ (flags & BOOT_HEAD) ? false : WARN_ON(cond); }) + static bool pgattr_change_is_safe(u64 old, u64 new) { /* @@ -174,7 +184,7 @@ static bool pgattr_change_is_safe(u64 old, u64 new) } static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, int flags) { pte_t *ptep; @@ -188,7 +198,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, * After the PTE entry has been populated once, we * only allow updates to the permission attributes. */ - BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), + SAFE_BUG_ON(flags, !pgattr_change_is_safe(pte_val(old_pte), READ_ONCE(pte_val(*ptep)))); phys += PAGE_SIZE; @@ -206,19 +216,19 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long next; pmd_t pmd = READ_ONCE(*pmdp); - BUG_ON(pmd_sect(pmd)); + SAFE_BUG_ON(flags, pmd_sect(pmd)); if (pmd_none(pmd)) { pmdval_t pmdval = PMD_TYPE_TABLE | PMD_TABLE_UXN; phys_addr_t pte_phys; if (flags & NO_EXEC_MAPPINGS) pmdval |= PMD_TABLE_PXN; - BUG_ON(!pgtable_alloc); + SAFE_BUG_ON(flags, !pgtable_alloc); pte_phys = pgtable_alloc(PAGE_SHIFT); __pmd_populate(pmdp, pte_phys, pmdval); pmd = READ_ONCE(*pmdp); } - BUG_ON(pmd_bad(pmd)); + SAFE_BUG_ON(flags, pmd_bad(pmd)); do { pgprot_t __prot = prot; @@ -230,7 +240,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pte(pmdp, addr, next, phys, __prot); + init_pte(pmdp, addr, next, phys, __prot, flags); phys += next - addr; } while (addr = next, addr != end); @@ -258,13 +268,13 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, * After the PMD entry has been populated once, we * only allow updates to the permission attributes. */ - BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), + SAFE_BUG_ON(flags, !pgattr_change_is_safe(pmd_val(old_pmd), READ_ONCE(pmd_val(*pmdp)))); } else { alloc_init_cont_pte(pmdp, addr, next, phys, prot, pgtable_alloc, flags); - BUG_ON(pmd_val(old_pmd) != 0 && + SAFE_BUG_ON(flags, pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); } phys += next - addr; @@ -284,19 +294,19 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, /* * Check for initial section mappings in the pgd/pud. */ - BUG_ON(pud_sect(pud)); + SAFE_BUG_ON(flags, pud_sect(pud)); if (pud_none(pud)) { pudval_t pudval = PUD_TYPE_TABLE | PUD_TABLE_UXN; phys_addr_t pmd_phys; if (flags & NO_EXEC_MAPPINGS) pudval |= PUD_TABLE_PXN; - BUG_ON(!pgtable_alloc); + SAFE_BUG_ON(flags, !pgtable_alloc); pmd_phys = pgtable_alloc(PMD_SHIFT); __pud_populate(pudp, pmd_phys, pudval); pud = READ_ONCE(*pudp); } - BUG_ON(pud_bad(pud)); + SAFE_BUG_ON(flags, pud_bad(pud)); do { pgprot_t __prot = prot; @@ -342,12 +352,12 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (flags & NO_EXEC_MAPPINGS) p4dval |= P4D_TABLE_PXN; - BUG_ON(!pgtable_alloc); + SAFE_BUG_ON(flags, !pgtable_alloc); pud_phys = pgtable_alloc(PUD_SHIFT); __p4d_populate(p4dp, pud_phys, p4dval); p4d = READ_ONCE(*p4dp); } - BUG_ON(p4d_bad(p4d)); + SAFE_BUG_ON(flags, p4d_bad(p4d)); pudp = pud_set_fixmap_offset(p4dp, addr); do { @@ -366,13 +376,13 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, * After the PUD entry has been populated once, we * only allow updates to the permission attributes. */ - BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), + SAFE_BUG_ON(flags, !pgattr_change_is_safe(pud_val(old_pud), READ_ONCE(pud_val(*pudp)))); } else { alloc_init_cont_pmd(pudp, addr, next, phys, prot, pgtable_alloc, flags); - BUG_ON(pud_val(old_pud) != 0 && + SAFE_BUG_ON(flags, pud_val(old_pud) != 0 && pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); } phys += next - addr; @@ -394,7 +404,7 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, * If the virtual and physical address don't have the same offset * within a page, we cannot map the region as the caller expects. */ - if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) + if (SAFE_WARN_ON(flags, (phys ^ virt) & ~PAGE_MASK)) return; phys &= PAGE_MASK; From patchwork Mon May 31 08:45:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12288909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7F34C47094 for ; Mon, 31 May 2021 08:48:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8E49E61186 for ; Mon, 31 May 2021 08:48:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E49E61186 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TtTHG1nUpZIOo8CsrTcQl/IQxV6xrQMoz0MLaRqMUGE=; b=gThCi++HZSP+vn rUDipvjg3BbZ4oD3TUKUybWTpVzmOcPvgM/+pq2egps6Fo+AuEV+gaO7TvtW6hPZNxK3Dlp5x2cqa ZuaDgelxAukGhZ4pxYINNd+Dj+LaEIWZY6oh/BcJk8zbWeaC6yKpk872ZxEXWKif9X/TYyrlLboTv kVwraoSQjvcrcQuv3oNvm7f9Oyi2XOiT/7yDhi2YS7uU1vbANyoA0rc91TbEBKFgkoa7RlsiwTFxn ONW6KZA6RcuS0BP/0MPp8PaU/cD7nObVoI4VgdBfvDhkkOv1mR8elLNTRLjvmNSYOhV8BCOYLDnbh pbSkoHJGrPGhjYRiWvVA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZT-00BWDc-8e; Mon, 31 May 2021 08:46:51 +0000 Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZ7-00BWAL-AH for linux-arm-kernel@lists.infradead.org; Mon, 31 May 2021 08:46:30 +0000 Received: by mail-pl1-x632.google.com with SMTP id e1so610756pld.13 for ; Mon, 31 May 2021 01:46:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/zyJIT7w/eGeWmQ4G0rkEMHAeL6fPSPkJSYhJ9pV1g4=; b=UgcoYLr7v9nKLPo4Qzo71PC+JN5LVIBY54596ObzRpp8EfzTdN4Bc3/Y1cyQS6dRYB Mt2v3haAsJ3OBHgLGXGw4iTyakqgpwk02XR6qaGRsWfyxtQxkDVfHchntNh2vrbAYP2o QyFnrDjJctIoRKlhI7sj4KMh6w0Me6A6gzu13OiZE4HzPRP8CQkm/wpsT6OllazsXCSl 7NBgJe/o3Njdc3DN1ZMwoE6P8hxrHFP40vjEkTM0WCOruNM8B4kdKcjQYKbYwAWSF1CW wuoUbyxSj1dZLa5MpAO+y5RMHCsZD9pFb0aDeWgLrZqGcYxs4UgLK3iJB1iKOA5Wojut IUGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/zyJIT7w/eGeWmQ4G0rkEMHAeL6fPSPkJSYhJ9pV1g4=; b=bdnNBnA45GJQDmo5Bdhj/gi1BaXBsRWlFhACg7hXpKmYPz0By7Src5N8+G2ZpMYTZz hjkqn9coxP2K9j9Iw2xwK2ZZSLS+WlAAU7J+YAaRWwkMucoePuAOk5I6GKRG91fGl+bp 5KpfjPMSoAeRrbzx6Jy1nMKMJkSO9KAW4V7SoG8FGI0dat+0S6YMX8zNTy4urSXpdEQx vRz2x0Wy0TCZLk0VDDM0xzu1O3Eu5SyU0x9RKC3kB5UEJRfiJnDoiHJXdPUeJPWi1Upq sk0e1fU2H8PuXeSbVXGYGdfjLaEkC+eXtVSmPE+IPnb6t/HdgTzN2XbjeDDdRjnpxzhp NXlA== X-Gm-Message-State: AOAM531arMK4yv3sKAALM4XFSBdCGwYCQj19QRWctsO70Su2EO46cHE+ AnnYHAylPh9Pky3mAR0uxUchvAMDHUBoEF8= X-Google-Smtp-Source: ABdhPJwsQHoGIQdCvURQG7Lg+E7EqukEge1/ZlcutjNZjsg9TvkUXViO0buZQ1l52PMlOcfwS6BcpQ== X-Received: by 2002:a17:902:a988:b029:101:86c3:df24 with SMTP id bh8-20020a170902a988b029010186c3df24mr12783937plb.64.1622450788124; Mon, 31 May 2021 01:46:28 -0700 (PDT) Received: from qualcomm-amberwing-rep-18.khw4.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id fs24sm5125677pjb.6.2021.05.31.01.46.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 01:46:27 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv3 3/5] arm64/mm: unconditionally set IDMAP_PGTABLE_LEVELS to max pgtable level Date: Mon, 31 May 2021 04:45:38 -0400 Message-Id: <20210531084540.78546-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210531084540.78546-1-kernelfans@gmail.com> References: <20210531084540.78546-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210531_014629_407391_CE5F5374 X-CRM114-Status: GOOD ( 17.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After commit 5dfe9d7d23c2 ("arm64: reduce ID map to a single page"), idmap only costs a page. While with 4K page, the current code enforces section mapping (2MB) on it due to the prot SWAPPER_MM_MMUFLAGS. But if switching to __create_pgd_mapping() to create pgtable, the code will decide the proper mapping level by itself, as the code snippet in init_pmd(): if (((addr | next | phys) & ~SECTION_MASK) == 0 && (flags & NO_BLOCK_MAPPINGS) == 0) pmd_set_huge(pmdp, phys, prot); As the case of .idmap.text, it requires pgtable up to pte level. Hence IDMAP_PGTABLE_LEVELS should be large enough to assure the capacity of IDMAP_DIR_SIZE. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/kernel-pgtable.h | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index d44df9d62fc9..249ab132fced 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -24,23 +24,26 @@ #endif /* - * The idmap and swapper page tables need some space reserved in the kernel - * image. Both require pgd, pud (4 levels only) and pmd tables to (section) - * map the kernel. With the 64K page configuration, swapper and idmap need to + * The swapper page table needs some space reserved in the kernel + * image. It require pgd, pud (4 levels only) and pmd tables to (section) + * map the kernel. With the 64K page configuration, swapper needs to * map to pte level. The swapper also maps the FDT (see __create_page_tables - * for more information). Note that the number of ID map translation levels - * could be increased on the fly if system RAM is out of reach for the default - * VA range, so pages required to map highest possible PA are reserved in all - * cases. + * for more information). */ #if ARM64_SWAPPER_USES_SECTION_MAPS #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) -#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT) - 1) #else #define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) -#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT)) #endif +/* + * idmap also needs some space reserved in the kernel image. Since ".idmap.text" + * only occupies less than a page, idmap needs to map to pte level. + * Note that the number of ID map translation levels could be increased on the fly + * if system RAM is out of reach for the default VA range, so pages required to + * map highest possible PA are reserved in all cases. + */ +#define IDMAP_PGTABLE_LEVELS (ARM64_HW_PGTABLE_LEVELS(PHYS_MASK_SHIFT)) /* * If KASLR is enabled, then an offset K is added to the kernel address From patchwork Mon May 31 08:45:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12288913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE37BC47082 for ; Mon, 31 May 2021 08:49:00 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB0B361186 for ; Mon, 31 May 2021 08:49:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB0B361186 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3++co9pY2xWjjNWlKwB2SOC9wf5RCYfgxnwjh8fMikk=; b=NnunOvas1EUX4g VEeL2S8PatFarH+Jbcjk79QUqXhzD3sQUWGcNS4JvHeJO+jJqOH7BMQLq2l+ywAyAnUPL3GIaDf8D auztr1+mAX24YwpJEKpng4uT3ou88n3XEPN/zblU8vS2r2nju4AQ7JswLUpQNklAsX8bHtJHIz3n2 J8VavC5wqkvXWb7ugt8SLI1bOONHY90iChPMS/cjGo2Pv22ncTF8X3FYno894JoUHQHtDhhnNYkZO nEnsVQE/JPfRRjN8hrgaVUVs2a9Y/IQU8PhmkQwPRamzMLj6MSO0XatEaSzVggJ3uqTFDbj2qqyo3 AhSYjBfqno0J/MuP95jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZg-00BWHP-3Y; Mon, 31 May 2021 08:47:04 +0000 Received: from mail-pj1-x102e.google.com ([2607:f8b0:4864:20::102e]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZF-00BWB7-AD for linux-arm-kernel@lists.infradead.org; Mon, 31 May 2021 08:46:38 +0000 Received: by mail-pj1-x102e.google.com with SMTP id g24so6204128pji.4 for ; Mon, 31 May 2021 01:46:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xxogjRX7faegKuuduNieefIAKpNTkkEuy15CEkfsdKM=; b=qNhAIjwqusqrhi7tyyZ4nB3kvTWecd+lMMq0uuszQ29RYh5qcLhWhPzQANHlkZAWMh /+nJ1jJVMd8mvFQfJdh8Q3Awg2USNcEXX+9DV/NHcGH6Yr/KcYTe47N5Ic01cbzb7jhs 9i6Ty38B/T35i6PHE2KS8bdKpXWXyh8n8+rVnbNaIxXbxYxOcUB7xJw1Y8HG3Kwle7Yi L6gL8If4MUmI23xHwjT07vH+IOUNkLsSo7JJ8Z0cceQjQu5bGJbsVURaBDiuj4sZ2Qn2 EfhS9teaCHG/aniDx6/SdyW8l4wJA8PRNXl+JGg9Zo8JF7OoZCTojjlyXbii4v6wDiKK oXRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xxogjRX7faegKuuduNieefIAKpNTkkEuy15CEkfsdKM=; b=AXUwNqk/kicNq/ClvPcEpBj22RFS9R1X32j/oMMlrdgbwbyfUbQbidgGrP/WCNgZZh KJNushQ/o8eeC8dUDvZj5qzbvdX2bLCLNNMRjbfKAZXmeQ4p3A26LHvjz1RmAUWDNfnV vo5aZjS3QMh/ujdP2PXCoKIKjT6ByEa/Vuik/nHJxd1x6m5F8S2Tc5RtOwJfspLoX3pu G5a+mr5ei/RFHEBa6KT4rDMr4x+WmtKxRmBkFFk7rwVW3WFMPbfDF5NIMn2HlotK82wy Gy98IjtN7nJ6sOMDcizNq0AhZLXosdi5Hgy8dEjWk8P6m1kDP3vLz4SUzPHf6P4mM0+Z lXqA== X-Gm-Message-State: AOAM532PK6dmFUdqUasaO9FWhQ6xLepjU1IQB41gfDes1t8vtjXM9nfJ wwnzTXZVeDTmJwmJMCCKxaLvqdZ/sja8mdI= X-Google-Smtp-Source: ABdhPJwxd9jVht6YICfWbYYQH4vDsde2c/msYUsIhHJMAMivejKu/fs1jCPT3zX3b8zOkIL1p4b6uA== X-Received: by 2002:a17:902:9685:b029:ef:70fd:a5a2 with SMTP id n5-20020a1709029685b02900ef70fda5a2mr19658561plp.20.1622450796714; Mon, 31 May 2021 01:46:36 -0700 (PDT) Received: from qualcomm-amberwing-rep-18.khw4.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id fs24sm5125677pjb.6.2021.05.31.01.46.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 01:46:36 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv3 4/5] arm64/mm: make __create_pgd_mapping() capable to handle pgtable's paddr Date: Mon, 31 May 2021 04:45:39 -0400 Message-Id: <20210531084540.78546-5-kernelfans@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210531084540.78546-1-kernelfans@gmail.com> References: <20210531084540.78546-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210531_014637_392515_A01B2BB1 X-CRM114-Status: GOOD ( 14.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This patch is the last preparation for calling __create_pgd_mapping() from head.S. Under mmu-offset situation, __create_pgd_mapping() handles paddr of each pgtable level. All of pud_t */pmd_t */pte_t * points to paddr, and they should be handled carefully to avoid the involvement of __va(). Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/mm/mmu.c | 39 ++++++++++++++++++++++++++++++++++----- 1 file changed, 34 insertions(+), 5 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index fe16f235d1fa..5f717552b524 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -189,6 +189,12 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, pte_t *ptep; ptep = pte_set_fixmap_offset(pmdp, addr); + if (likely(!(flags & BOOT_HEAD))) + ptep = pte_set_fixmap_offset(pmdp, addr); + else + /* for head.S, there is no __va() */ + ptep = (pte_t *)__pmd_to_phys(*pmdp) + pte_index(addr); + do { pte_t old_pte = READ_ONCE(*ptep); @@ -204,7 +210,8 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); - pte_clear_fixmap(); + if (likely(!(flags & BOOT_HEAD))) + pte_clear_fixmap(); } static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, @@ -253,7 +260,17 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, unsigned long next; pmd_t *pmdp; - pmdp = pmd_set_fixmap_offset(pudp, addr); + if (likely(!(flags & BOOT_HEAD))) { + pmdp = pmd_set_fixmap_offset(pudp, addr); + } else { +#if CONFIG_PGTABLE_LEVELS > 2 + /* for head.S, there is no __va() */ + pmdp = (pmd_t *)__pud_to_phys(*pudp) + pmd_index(addr); +#else + pmdp = (pmd_t *)pudp; +#endif + } + do { pmd_t old_pmd = READ_ONCE(*pmdp); @@ -280,7 +297,8 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys += next - addr; } while (pmdp++, addr = next, addr != end); - pmd_clear_fixmap(); + if (likely(!(flags & BOOT_HEAD))) + pmd_clear_fixmap(); } static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, @@ -359,7 +377,17 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, } SAFE_BUG_ON(flags, p4d_bad(p4d)); - pudp = pud_set_fixmap_offset(p4dp, addr); + if (likely(!(flags & BOOT_HEAD))) { + pudp = pud_set_fixmap_offset(p4dp, addr); + } else { +#if CONFIG_PGTABLE_LEVELS > 3 + /* for head.S, there is no __va() */ + pudp = (pud_t *)__p4d_to_phys(*p4dp) + pud_index(addr); +#else + pudp = (pud_t *)p4dp; +#endif + } + do { pud_t old_pud = READ_ONCE(*pudp); @@ -388,7 +416,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys += next - addr; } while (pudp++, addr = next, addr != end); - pud_clear_fixmap(); + if (likely(!(flags & BOOT_HEAD))) + pud_clear_fixmap(); } static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, From patchwork Mon May 31 08:45:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12288915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52206C4708F for ; Mon, 31 May 2021 08:49:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1743F61186 for ; Mon, 31 May 2021 08:49:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1743F61186 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Bqh75mcMTbE/Dgu/YYrRARDOb1DyKn7x2XYgw2hu7g0=; b=kBXMrgXhBN+BER 6Fc+Ie5Ewrkn60tkcTljKnOIcZ2eSfjPYnf4IdVMtkgxgYTviTf81y0ZrFNhGPZefvZQb06vYcHMM rP4zLndckXH6LlFAt/LhQ9/8TlWEFLlHMG8d7XVoFdQOw98Dzs6HHcI7PrQVHMhWi+UEBx4MGOKoK EZcGz/EYg9FVMHUYQWfoip6KgO+MYuBrguGnYGOZFezcImIvggnQ07gtdtlo29MSPM3lHxomecJ9i iU1+5V596uK42I4+m768F8kmRU6aTIU5xDe/69xW53ggmtkQdxtIyxjeJmx1FvbM1ylBw373qkv6+ QTTyRPe2KCNPDVvIPS8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZu-00BWM9-Aw; Mon, 31 May 2021 08:47:18 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lndZR-00BWDa-9S for linux-arm-kernel@lists.infradead.org; Mon, 31 May 2021 08:46:51 +0000 Received: by mail-pf1-x429.google.com with SMTP id 22so8493378pfv.11 for ; Mon, 31 May 2021 01:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Zel3PPNBalQvU0nQThY/lOELl5rVDzxGrrqCtqzuEa0=; b=SH/S6z8LoO4JchsNLVFhMwJBwa13G0Up4h8p0GzJW57z73NDLI5DSZECNiHbxGrQwo 3FQryHjrqKDC4mcg6hTxsNkUYUitXKEL1v3kJGZYufaEzF6S0xXxPWtoLPtTytKXSrCb c2W/JliThtesTj48RCyk6qzrH1rQVvqGBPUHqROQa3IK4Cg0wtgY0/l0/gKcSY62b0wN VlokPPuJmvDqgyZxVVVmLiPp9xZTtwK/CcWxybcA2uwlXkFpgJ9JCtn39OaglA9+eJ2Z zE6VqZKET/hpcRMNAldJ8DEAzCLmuIdNah6nOPcCrkulp1Jyav8CWbhIIB2bIICIeTWz 25uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Zel3PPNBalQvU0nQThY/lOELl5rVDzxGrrqCtqzuEa0=; b=KANtj5AMG0JSghte/cE1XnRzHXgIOiPsjpnuDtymPzx5pW/xHPSeVoc1tu7dE4WWhE 9I39cjdx0eQtrR0jxVkQSO8Fh6TnEZqtIPgGQedpmeSDNPfpixs9Edsti6AH+s/2aHZE 97UWQFsb86syw/5gTixySpWljmrTECUMgPEEqCMR7WmpoI7TmPG2d8qNtTAmSWXYiDcA h4MrnV4A9nG7z5dOas+uvSophjf/H81HjBwzNqWwFhX953VmpgHPvOgje9KAueWnbFtL rQjOf1Cid5mAUSVclDNvpHUV3Qyp6YIIDOsoBxNaC87v0MmrpeH8VXHHmDTydBSDt8mZ OJaA== X-Gm-Message-State: AOAM530wL/SwrQSRpEbGPbAfTsTi5umFRH6iK/OTzSgkwUvFCrAC60gu I8KtEUmd2DHutFJ1rx6LAYCLjhK38FusFTw= X-Google-Smtp-Source: ABdhPJypAyObrFNjX9KOn1kNNAnNzGXzY6XXfosGxSA7t2Xa5AagB6okQ+TvMG1jkGu4MWKSZDhs3g== X-Received: by 2002:a62:4ed1:0:b029:2e4:df13:fbd8 with SMTP id c200-20020a624ed10000b02902e4df13fbd8mr15832483pfb.68.1622450807871; Mon, 31 May 2021 01:46:47 -0700 (PDT) Received: from qualcomm-amberwing-rep-18.khw4.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id fs24sm5125677pjb.6.2021.05.31.01.46.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 May 2021 01:46:47 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv3 5/5] arm64/mm: use __create_pgd_mapping() to create pgtable for idmap_pg_dir and init_pg_dir Date: Mon, 31 May 2021 04:45:40 -0400 Message-Id: <20210531084540.78546-6-kernelfans@gmail.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210531084540.78546-1-kernelfans@gmail.com> References: <20210531084540.78546-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210531_014649_430661_B7960CD2 X-CRM114-Status: GOOD ( 24.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now, everything is ready for calling __create_pgd_mapping() from head.S. Switching to these C routine and remove the asm counterpart. This patch has been successfully tested with the following config value: PAGE_SIZE VA PA PGTABLE_LEVEL 4K 48 48 4 4K 39 48 3 16K 48 48 4 16K 47 48 3 64K 52 52 3 64K 48 52 3 64K 42 52 2 Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/kernel-pgtable.h | 12 +- arch/arm64/include/asm/pgalloc.h | 9 ++ arch/arm64/kernel/head.S | 164 +++++++----------------- arch/arm64/mm/mmu.c | 7 +- 4 files changed, 60 insertions(+), 132 deletions(-) diff --git a/arch/arm64/include/asm/kernel-pgtable.h b/arch/arm64/include/asm/kernel-pgtable.h index 249ab132fced..121856008a0e 100644 --- a/arch/arm64/include/asm/kernel-pgtable.h +++ b/arch/arm64/include/asm/kernel-pgtable.h @@ -108,15 +108,11 @@ /* * Initial memory map attributes. + * When using ARM64_SWAPPER_USES_SECTION_MAPS, init_pmd()->pmd_set_huge() + * sets up section mapping. */ -#define SWAPPER_PTE_FLAGS (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define SWAPPER_PMD_FLAGS (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) - -#if ARM64_SWAPPER_USES_SECTION_MAPS -#define SWAPPER_MM_MMUFLAGS (PMD_ATTRINDX(MT_NORMAL) | SWAPPER_PMD_FLAGS) -#else -#define SWAPPER_MM_MMUFLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS) -#endif +#define SWAPPER_PAGE_FLAGS ((PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) | \ + PTE_ATTRINDX(MT_NORMAL)) /* * To make optimal use of block mappings when laying out the linear diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 31fbab3d6f99..9a6fb96ff291 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -8,6 +8,8 @@ #ifndef __ASM_PGALLOC_H #define __ASM_PGALLOC_H +#ifndef __ASSEMBLY__ + #include #include #include @@ -89,3 +91,10 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) #define pmd_pgtable(pmd) pmd_page(pmd) #endif + +#define NO_BLOCK_MAPPINGS BIT(0) +#define NO_CONT_MAPPINGS BIT(1) +#define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ +#define BOOT_HEAD BIT(3) + +#endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 96873dfa67fd..7158987f52b1 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -93,6 +94,8 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag + adrp x4, init_thread_union + add sp, x4, #THREAD_SIZE bl __create_page_tables /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -144,112 +147,6 @@ SYM_CODE_END(preserve_boot_args) add \tbl, \tbl, #PAGE_SIZE // next level table page .endm -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - * shift: shift used to transform virtual address into index - * ptrs: number of entries in page table - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend, shift, ptrs - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, ptrs, istart, iend, count - lsr \iend, \vend, \shift - mov \istart, \ptrs - sub \istart, \istart, #1 - and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1) - mov \istart, \ptrs - mul \istart, \istart, \count - add \iend, \iend, \istart // iend += (count - 1) * ptrs - // our entries span multiple tables - - lsr \istart, \vstart, \shift - mov \count, \ptrs - sub \count, \count, #1 - and \istart, \istart, \count - - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: start address to map - * vend: end address to map - we map [vstart, vend] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * pgds: the number of pgd entries - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, vend, flags - * Corrupts: tbl, rtbl, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, pgds, istart, iend, tmp, count, sv - add \rtbl, \tbl, #PAGE_SIZE - mov \sv, \rtbl - mov \count, #0 - compute_indices \vstart, \vend, #PGDIR_SHIFT, \pgds, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #PTRS_PER_PUD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #PTRS_PER_PMD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #PTRS_PER_PTE, \istart, \iend, \count - bic \count, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \count, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - /* * Setup the initial page tables. We only setup the barest amount which is * required to get the kernel running. The following sections are required: @@ -284,8 +181,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) subs x1, x1, #64 b.ne 1b - mov x7, SWAPPER_MM_MMUFLAGS - /* * Create the identity mapping. */ @@ -357,21 +252,54 @@ SYM_FUNC_START_LOCAL(__create_page_tables) mov x5, x3 // __pa(__idmap_text_start) adr_l x6, __idmap_text_end // __pa(__idmap_text_end) - map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 + /* + * x0 points to either idmap_pg_dir or idmap_pg_dir + PAGE_SIZE + */ + stp x0, x1, [sp, #-64]! + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #32] + stp x6, x7, [sp, #16] + + adrp x1, idmap_pg_end + sub x1, x1, x0 + bl set_cur_headpool + mov x0, #0 + bl head_pgtable_alloc // return x0, containing the appropriate pgtable level + + adrp x1, __idmap_text_start + adrp x2, __idmap_text_start // va = pa for idmap + adr_l x3, __idmap_text_end + sub x3, x3, x1 + ldr x4, =SWAPPER_PAGE_FLAGS + adr_l x5, head_pgtable_alloc + mov x6, #BOOT_HEAD + bl __create_pgd_mapping /* * Map the kernel image (starting with PHYS_OFFSET). */ adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) - add x5, x5, x23 // add KASLR displacement - mov x4, PTRS_PER_PGD - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) - - map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 + adrp x1, init_pg_end + sub x1, x1, x0 + bl set_cur_headpool + mov x0, #0 + bl head_pgtable_alloc // return x0, containing init_pg_dir + + adrp x1, _text // runtime __pa(_text) + mov_q x2, KIMAGE_VADDR // compile time __va(_text) + add x2, x2, x23 // add KASLR displacement + adrp x3, _end // runtime __pa(_end) + sub x3, x3, x1 // _end - _text + + ldr x4, =SWAPPER_PAGE_FLAGS + adr_l x5, head_pgtable_alloc + mov x6, #BOOT_HEAD + bl __create_pgd_mapping + + ldp x6, x7, [sp, #16] + ldp x4, x5, [sp, #32] + ldp x2, x3, [sp, #48] + ldp x0, x1, [sp], #64 /* * Since the page tables have been populated with non-cacheable diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5f717552b524..b3295523607e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -37,11 +37,6 @@ #include #include -#define NO_BLOCK_MAPPINGS BIT(0) -#define NO_CONT_MAPPINGS BIT(1) -#define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -#define BOOT_HEAD BIT(3) - u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; @@ -420,7 +415,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, +void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int),