From patchwork Sat Apr 10 09:56:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59AB7C433ED for ; Sat, 10 Apr 2021 09:59:32 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D12A9610A2 for ; Sat, 10 Apr 2021 09:59:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D12A9610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/D/lQYi1KpwSp6g4iVfabfyasGeCOex2QV+4VZksOb0=; b=CoYMv7AUOQHgBx/drzb+PvEDt j4jetv0r56QaiKVMcrBoU7kohfznlvUPFwF7rjr5E0eN6yz78YwvCLIGX4AuT9ul8c8r/vWlHXx2D vocjhBROVv3nM5nIjXkIQsLbgKTXt0ECm4osJXwoxPDiTYQrfV7ynJKNTL+yfZ+tv0XdbeNODvmtR Exf/5yfHS7GvpY/FV7x2KNAJ/Yhm9RFC8Qjk17UtXtmw6wkHTsGIdrT5//7HXuotgCIPBuOzTOTPV +3JjuOgeoRHhnlcVmnoXBfIY4aVBQUvJBAvEmiaZ6mQncRhGxBvJ+aICfcz1j3ZcLbhJ99QCYUc00 lHb0tJKNg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMt-002bQB-3v; Sat, 10 Apr 2021 09:57:32 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMi-002bOf-1T for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=nYpEtuTnhMZJDjsOy576PWjaFXEy4VEKhrE6/XIqMTQ=; b=pgi/ENwEGmN4XvFY5mlYQhFqwH y223aSSeNM4C5yLOqbNyRrO0+fTe4aKFb08ESfRbmOLhbAJsU8OUf+sG8QpvATyiogrGga181/ONS t+1TJ3Qm1EMyyoZOP3Rv4ehHszCbR7PoeyeG8l7exqcSl21RCYBiNKenLANNbKINxm6GYUE2ItsW+ YFCwJ0nnL8Mwom0Pqb3WXV601re6LiAAWR9VOP598dlAZQ20w4jpU12glTA0TzrMN39JnslGGix3y Cj8hEBNUxJWH8AXVkUJa7igC93CQcSPIKukSPFZ3nlaLbaVc39yrVTYew864iKOu42VDAjmyciJYM gB9kr99A==; Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMe-0051YV-BE for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:18 +0000 Received: by mail-pl1-x630.google.com with SMTP id ay2so3932668plb.3 for ; Sat, 10 Apr 2021 02:57:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nYpEtuTnhMZJDjsOy576PWjaFXEy4VEKhrE6/XIqMTQ=; b=ivXk0iS2zP5ZTHwy7M1Lzc2U1/WrAQJbLqGqBYUGf7gMxwQmmUust+pwYu1TXTeGrc rcdthqA9aM3Cv4h44tem/mLrKxYImRcks34ry7Oee3kfgC/nH10X1iiQ7OeGplCKy2Xf V791TiqlArJsKI1d/YHH0b9J3luT718Dh2wxAxM7qJNSGj4M3y61+WnVv+kFIyNYWj3E s5g2RzgQ9wwGjF4qTGEqNxmi+byv7MJE/1NPyReBuy4k0vdFSqtlPgH7iu/5le8eigso k5eFHIzF0GxzsTFJ2Rz8lsBeAPeO8CLDvsgpR+EhfN9nrpSZUYgOh+g5NOPkgFiHXzfz X2QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nYpEtuTnhMZJDjsOy576PWjaFXEy4VEKhrE6/XIqMTQ=; b=mvNke+ftuHlyGVzp1nZL215EfuizNcu8SBKpFyxKmq7r0i8Ck3gZIOH4odDOGMpnoS ZrhKsKMrkcmbcW9TRfitRYnAZnpIAkZoY7+PW54aty8EQHHOSXJmwVOSnj0XqBxTiPOQ fskSXpn0W4uMA7B9+FcZDdQ7MXNFQf1rR8sIkGN/G7FpRJNiCj7Rx6fUDt2uwR1G8HMN U52BF8H1Yi3hUXjYQePf/P0WEQBhNaIzcvr/B8bLtiLOsSu0tPDc9LQEyjLEZyxyN6sT 4KHFX69Z1StHQOPV4hrlRJqCveo0HgTSflc3UhvMr+u/3T+0VnTWOpbpv/iN6nW/e0+B 5mlg== X-Gm-Message-State: AOAM531zrlVQ26Uo9DhqpvEcymYkNfmiWibKjquzaT/lhOPyasWv5yxh RUko2v3p5kHjba0k6iz53myTLY5sdGpI X-Google-Smtp-Source: ABdhPJy6/xpDWky59IUqG52DsVPidnmbyAUZCq2OYaU593bN6gnbQwhuARWHa/+3o+nqDtU9Ot/2yQ== X-Received: by 2002:a17:90a:f298:: with SMTP id fs24mr195459pjb.176.1618048634956; Sat, 10 Apr 2021 02:57:14 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:14 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 1/8] arm64/mm: split out __create_pgd_mapping() routines Date: Sat, 10 Apr 2021 17:56:47 +0800 Message-Id: <20210410095654.24102-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025716_420368_7EDEB5C8 X-CRM114-Status: GOOD ( 29.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Split out the routines for __create_pgd_mapping(), in order to use it to generate two sets of operations for CONFIG_PGTABLE_LEVELS and CONFIG_PGTABLE_LEVELS + 1 Later the one generated with 'CONFIG_PGTABLE_LEVELS + 1' can be used for idmap if VA_BITS is too small to cover system RAM, which is located sufficiently high in the physical address space. Later, idmap can be created by __create_pgd_mapping() directly. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/Kconfig | 4 + arch/arm64/mm/Makefile | 2 + arch/arm64/mm/idmap_mmu.c | 45 ++++++ arch/arm64/mm/mmu.c | 263 +----------------------------------- arch/arm64/mm/mmu_include.c | 262 +++++++++++++++++++++++++++++++++++ 5 files changed, 315 insertions(+), 261 deletions(-) create mode 100644 arch/arm64/mm/idmap_mmu.c create mode 100644 arch/arm64/mm/mmu_include.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index e4e1b6550115..989fc501a1b4 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -327,6 +327,10 @@ config PGTABLE_LEVELS default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 +config IDMAP_PGTABLE_EXPAND + def_bool y + depends on (ARM64_4K_PAGES && ARM64_VA_BITS_39) || (ARM64_64K_PAGES && ARM64_VA_BITS_42) + config ARCH_SUPPORTS_UPROBES def_bool y diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index f188c9092696..f9283cb9a201 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,8 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o + +obj-$(CONFIG_IDMAP_PGTABLE_EXPAND) += idmap_mmu.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c new file mode 100644 index 000000000000..7e9a4f4017d3 --- /dev/null +++ b/arch/arm64/mm/idmap_mmu.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if CONFIG_IDMAP_PGTABLE_EXPAND + +#if CONFIG_PGTABLE_LEVELS == 2 +#define EXTEND_LEVEL 3 +#elif CONFIG_PGTABLE_LEVELS == 3 +#define EXTEND_LEVEL 4 +#endif + +#undef CONFIG_PGTABLE_LEVELS +#define CONFIG_PGTABLE_LEVELS EXTEND_LEVEL + + +#include "./mmu_include.c" + +void __create_pgd_mapping_extend(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, prot, pgtable_alloc, flags); +} +#endif + + diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5d9550fdb9cf..56e4f25e8d6d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -37,9 +37,6 @@ #include #include -#define NO_BLOCK_MAPPINGS BIT(0) -#define NO_CONT_MAPPINGS BIT(1) - u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; @@ -116,264 +113,6 @@ static phys_addr_t __init early_pgtable_alloc(int shift) return phys; } -static bool pgattr_change_is_safe(u64 old, u64 new) -{ - /* - * The following mapping attributes may be updated in live - * kernel mappings without the need for break-before-make. - */ - pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; - - /* creating or taking down mappings is always safe */ - if (old == 0 || new == 0) - return true; - - /* live contiguous mappings may not be manipulated at all */ - if ((old | new) & PTE_CONT) - return false; - - /* Transitioning from Non-Global to Global is unsafe */ - if (old & ~new & PTE_NG) - return false; - - /* - * Changing the memory type between Normal and Normal-Tagged is safe - * since Tagged is considered a permission attribute from the - * mismatched attribute aliases perspective. - */ - if (((old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || - (old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED)) && - ((new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || - (new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED))) - mask |= PTE_ATTRINDX_MASK; - - return ((old ^ new) & ~mask) == 0; -} - -static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) -{ - pte_t *ptep; - - ptep = pte_set_fixmap_offset(pmdp, addr); - do { - pte_t old_pte = READ_ONCE(*ptep); - - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); - - /* - * After the PTE entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); - - phys += PAGE_SIZE; - } while (ptep++, addr += PAGE_SIZE, addr != end); - - pte_clear_fixmap(); -} - -static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long next; - pmd_t pmd = READ_ONCE(*pmdp); - - BUG_ON(pmd_sect(pmd)); - if (pmd_none(pmd)) { - phys_addr_t pte_phys; - BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(PAGE_SHIFT); - __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); - pmd = READ_ONCE(*pmdp); - } - BUG_ON(pmd_bad(pmd)); - - do { - pgprot_t __prot = prot; - - next = pte_cont_addr_end(addr, end); - - /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && - (flags & NO_CONT_MAPPINGS) == 0) - __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - - init_pte(pmdp, addr, next, phys, __prot); - - phys += next - addr; - } while (addr = next, addr != end); -} - -static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) -{ - unsigned long next; - pmd_t *pmdp; - - pmdp = pmd_set_fixmap_offset(pudp, addr); - do { - pmd_t old_pmd = READ_ONCE(*pmdp); - - next = pmd_addr_end(addr, end); - - /* try section mapping first */ - if (((addr | next | phys) & ~SECTION_MASK) == 0 && - (flags & NO_BLOCK_MAPPINGS) == 0) { - pmd_set_huge(pmdp, phys, prot); - - /* - * After the PMD entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), - READ_ONCE(pmd_val(*pmdp)))); - } else { - alloc_init_cont_pte(pmdp, addr, next, phys, prot, - pgtable_alloc, flags); - - BUG_ON(pmd_val(old_pmd) != 0 && - pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); - } - phys += next - addr; - } while (pmdp++, addr = next, addr != end); - - pmd_clear_fixmap(); -} - -static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) -{ - unsigned long next; - pud_t pud = READ_ONCE(*pudp); - - /* - * Check for initial section mappings in the pgd/pud. - */ - BUG_ON(pud_sect(pud)); - if (pud_none(pud)) { - phys_addr_t pmd_phys; - BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(PMD_SHIFT); - __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); - pud = READ_ONCE(*pudp); - } - BUG_ON(pud_bad(pud)); - - do { - pgprot_t __prot = prot; - - next = pmd_cont_addr_end(addr, end); - - /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PMD_MASK) == 0) && - (flags & NO_CONT_MAPPINGS) == 0) - __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - - init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); - - phys += next - addr; - } while (addr = next, addr != end); -} - -static inline bool use_1G_block(unsigned long addr, unsigned long next, - unsigned long phys) -{ - if (PAGE_SHIFT != 12) - return false; - - if (((addr | next | phys) & ~PUD_MASK) != 0) - return false; - - return true; -} - -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); - p4d_t p4d = READ_ONCE(*p4dp); - - if (p4d_none(p4d)) { - phys_addr_t pud_phys; - BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(PUD_SHIFT); - __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); - p4d = READ_ONCE(*p4dp); - } - BUG_ON(p4d_bad(p4d)); - - pudp = pud_set_fixmap_offset(p4dp, addr); - do { - pud_t old_pud = READ_ONCE(*pudp); - - next = pud_addr_end(addr, end); - - /* - * For 4K granule only, attempt to put down a 1GB block - */ - if (use_1G_block(addr, next, phys) && - (flags & NO_BLOCK_MAPPINGS) == 0) { - pud_set_huge(pudp, phys, prot); - - /* - * After the PUD entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); - } else { - alloc_init_cont_pmd(pudp, addr, next, phys, prot, - pgtable_alloc, flags); - - BUG_ON(pud_val(old_pud) != 0 && - pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); - } - phys += next - addr; - } while (pudp++, addr = next, addr != end); - - pud_clear_fixmap(); -} - -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); - - /* - * If the virtual and physical address don't have the same offset - * within a page, we cannot map the region as the caller expects. - */ - if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) - return; - - phys &= PAGE_MASK; - addr = virt & PAGE_MASK; - end = PAGE_ALIGN(virt + size); - - do { - next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, - flags); - phys += next - addr; - } while (pgdp++, addr = next, addr != end); -} - static phys_addr_t __pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); @@ -404,6 +143,8 @@ static phys_addr_t pgd_pgtable_alloc(int shift) return pa; } +#include "./mmu_include.c" + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c new file mode 100644 index 000000000000..e9ebdffe860b --- /dev/null +++ b/arch/arm64/mm/mmu_include.c @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#define NO_BLOCK_MAPPINGS BIT(0) +#define NO_CONT_MAPPINGS BIT(1) + +static bool pgattr_change_is_safe(u64 old, u64 new) +{ + /* + * The following mapping attributes may be updated in live + * kernel mappings without the need for break-before-make. + */ + pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; + + /* creating or taking down mappings is always safe */ + if (old == 0 || new == 0) + return true; + + /* live contiguous mappings may not be manipulated at all */ + if ((old | new) & PTE_CONT) + return false; + + /* Transitioning from Non-Global to Global is unsafe */ + if (old & ~new & PTE_NG) + return false; + + /* + * Changing the memory type between Normal and Normal-Tagged is safe + * since Tagged is considered a permission attribute from the + * mismatched attribute aliases perspective. + */ + if (((old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || + (old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED)) && + ((new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || + (new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED))) + mask |= PTE_ATTRINDX_MASK; + + return ((old ^ new) & ~mask) == 0; +} + +static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot) +{ + pte_t *ptep; + + ptep = pte_set_fixmap_offset(pmdp, addr); + do { + pte_t old_pte = READ_ONCE(*ptep); + + set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + + /* + * After the PTE entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), + READ_ONCE(pte_val(*ptep)))); + + phys += PAGE_SIZE; + } while (ptep++, addr += PAGE_SIZE, addr != end); + + pte_clear_fixmap(); +} + +static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pmd_t pmd = READ_ONCE(*pmdp); + + BUG_ON(pmd_sect(pmd)); + if (pmd_none(pmd)) { + phys_addr_t pte_phys; + BUG_ON(!pgtable_alloc); + pte_phys = pgtable_alloc(PAGE_SHIFT); + __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); + pmd = READ_ONCE(*pmdp); + } + BUG_ON(pmd_bad(pmd)); + + do { + pgprot_t __prot = prot; + + next = pte_cont_addr_end(addr, end); + + /* use a contiguous mapping if the range is suitably aligned */ + if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && + (flags & NO_CONT_MAPPINGS) == 0) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + init_pte(pmdp, addr, next, phys, __prot); + + phys += next - addr; + } while (addr = next, addr != end); +} + +static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) +{ + unsigned long next; + pmd_t *pmdp; + + pmdp = pmd_set_fixmap_offset(pudp, addr); + do { + pmd_t old_pmd = READ_ONCE(*pmdp); + + next = pmd_addr_end(addr, end); + + /* try section mapping first */ + if (((addr | next | phys) & ~SECTION_MASK) == 0 && + (flags & NO_BLOCK_MAPPINGS) == 0) { + pmd_set_huge(pmdp, phys, prot); + + /* + * After the PMD entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), + READ_ONCE(pmd_val(*pmdp)))); + } else { + alloc_init_cont_pte(pmdp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(pmd_val(old_pmd) != 0 && + pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); + } + phys += next - addr; + } while (pmdp++, addr = next, addr != end); + + pmd_clear_fixmap(); +} + +static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) +{ + unsigned long next; + pud_t pud = READ_ONCE(*pudp); + + /* + * Check for initial section mappings in the pgd/pud. + */ + BUG_ON(pud_sect(pud)); + if (pud_none(pud)) { + phys_addr_t pmd_phys; + BUG_ON(!pgtable_alloc); + pmd_phys = pgtable_alloc(PMD_SHIFT); + __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); + pud = READ_ONCE(*pudp); + } + BUG_ON(pud_bad(pud)); + + do { + pgprot_t __prot = prot; + + next = pmd_cont_addr_end(addr, end); + + /* use a contiguous mapping if the range is suitably aligned */ + if ((((addr | next | phys) & ~CONT_PMD_MASK) == 0) && + (flags & NO_CONT_MAPPINGS) == 0) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); + + phys += next - addr; + } while (addr = next, addr != end); +} + +static inline bool use_1G_block(unsigned long addr, unsigned long next, + unsigned long phys) +{ + if (PAGE_SHIFT != 12) + return false; + + if (((addr | next | phys) & ~PUD_MASK) != 0) + return false; + + return true; +} + +static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pud_t *pudp; + p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t p4d = READ_ONCE(*p4dp); + + if (p4d_none(p4d)) { + phys_addr_t pud_phys; + BUG_ON(!pgtable_alloc); + pud_phys = pgtable_alloc(PUD_SHIFT); + __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); + p4d = READ_ONCE(*p4dp); + } + BUG_ON(p4d_bad(p4d)); + + pudp = pud_set_fixmap_offset(p4dp, addr); + do { + pud_t old_pud = READ_ONCE(*pudp); + + next = pud_addr_end(addr, end); + + /* + * For 4K granule only, attempt to put down a 1GB block + */ + if (use_1G_block(addr, next, phys) && + (flags & NO_BLOCK_MAPPINGS) == 0) { + pud_set_huge(pudp, phys, prot); + + /* + * After the PUD entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), + READ_ONCE(pud_val(*pudp)))); + } else { + alloc_init_cont_pmd(pudp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(pud_val(old_pud) != 0 && + pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); + } + phys += next - addr; + } while (pudp++, addr = next, addr != end); + + pud_clear_fixmap(); +} + +static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long addr, end, next; + pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + + /* + * If the virtual and physical address don't have the same offset + * within a page, we cannot map the region as the caller expects. + */ + if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) + return; + + phys &= PAGE_MASK; + addr = virt & PAGE_MASK; + end = PAGE_ALIGN(virt + size); + + do { + next = pgd_addr_end(addr, end); + alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + flags); + phys += next - addr; + } while (pgdp++, addr = next, addr != end); +} From patchwork Sat Apr 10 09:56:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 871D3C433B4 for ; Sat, 10 Apr 2021 09:59:48 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA74961001 for ; Sat, 10 Apr 2021 09:59:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA74961001 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VyWyj6I6p/sI0N7u2oBTQTFkwTAlvOYZBeoqFZMKDYc=; b=cYgnSi+4dPSi5bDZRpU72PWkz nmmETBJt4y8DmDXD3YjWQL0cZ1J2aNqcdeF9DE4GOeDnpcMRvMCzbPOgqtNOWriWGRpyCfjxZKLp7 pMPDlJbZuQuo82dJrRlsI9RwKFmuK/ErOx6vxzPEK17alTVyihMGPXcS6Wj3QXM6q/bPq1Yx6EnRX 8Wx614dWvViwTYKap/A/yDRZp3HKjRBM+vPIsT7X3tuUvXrTCjVFG3naHFDJJh0L7LnKoFSNiIyLu vbAn5OdVomuCdviUQ+Silv9N24X4zXcYEn1wngIrYU3rsub5VZBkV3i/XIsjJQFM7Tl9+bNQZjZcq POLUsBfXQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVANC-002bTF-Ho; Sat, 10 Apr 2021 09:57:50 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMm-002bP2-RF for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=kfZ3yY7rkniJE2QSo9AxiDLn15/ogNXSmhdqvQvKk5Q=; b=gcap4erg8NghVURxXNsOdkO1x4 xsIEd14oNapMqDfHwHEV28PP/POpdnlRp7G3W6JlC3AdeqGnO43PPN56RW4UCsWD22WPiswlvzPrT t1Tjb0tmkIOiSl3jByDDoJMoXDwiwqwFauNQEzxn5Gyz/3K99JC/r8inPrGGTs/oEBNZK9rUa1dzO J44GVpJVj/BOokkS1KxAnDuvEYCYnuglADZJ+8Lt1BivoAmjEJa3Qohz2VlOb7qA7Pt38EM+8v0o3 cXH3URTEzVX4KxoddI8XVuncfnZ8FPA4OFTtC6+ijxtMbqhffLPIVPtrBlbFCpewuNtk+87N11xAG NKQFvX+A==; Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMj-0051Yv-34 for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:22 +0000 Received: by mail-pf1-x42e.google.com with SMTP id d124so5836620pfa.13 for ; Sat, 10 Apr 2021 02:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kfZ3yY7rkniJE2QSo9AxiDLn15/ogNXSmhdqvQvKk5Q=; b=ck8ji4ZyXKDKgfydWa1JEzX7EmHxu8i/cG/+tiZJsJqUoJMhWaoqj9J0DCH0ankYCi UQjcQJ8v2Kxpuo5mcnLQohGCp1a7QVdakBMDv3rR/HJg0tQaRHHAnE6DCPt3rsbwVe6X D/Ej/eqEs8N1NQ72gfQ+9kyq/JIhFNrsQMkavaBnf8PVej5MrTOSzzNbkMa9E9Fxb1ju CMt7QT2FyKsKrVon7fapNQZoB5IE9E0Lp0TEuUt0uQ3NIJygR/NsE3u1jz6nF6Wq+U57 EVLUtvxabJKjEYvAhsHrYl1tOE9ARCKTPCDVqGHhL/sWI81mK40Lug1rw3Vf1aRqV8WI +MLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kfZ3yY7rkniJE2QSo9AxiDLn15/ogNXSmhdqvQvKk5Q=; b=E9I2xLJa/ucbZDOn0oBoOPGno4xvMrZivdA+sX3medJ7okBUVVA52deKGAYbaEzCwz dCpwefYyvcmiw3iutsG5HcxbcSjFh0kJdpV0qGOcmQ/I4lHsXFcWovJv5H32igR8nvjx IZDygfTcQoTgRYMD5/AnYiApPWfNdgA9dNupqO6tTZCEddW9/Fg6/u8YHEaTECpKOpmc ZO2Lc9FLd2Mw4bIwV2fufKpF8zfj94bHNPhli7oe7ek1DjsrgxfN6NEH1/8WQEQLcqo7 bt/xw4++HiZWC4kL2+xZzJkiEjrT09YQrVZ1bucA5TJhZTduMoIhTXtzEQbHIVyY6Bpm m5hQ== X-Gm-Message-State: AOAM532Fczw/TkgpBFW/qBdkjUBowBQZzwcD+cxI71BVA7WeZq2qJghA cR2/N3vna76M5mDkqmtIqDk/MYAHqDgp X-Google-Smtp-Source: ABdhPJzYw+cKPHhspjN7j05J4DyH6i0aZIomTCc2SGK69+HrOqGiDmVSk+ZU+OAwaLNNjzoI812aqg== X-Received: by 2002:a62:2cce:0:b029:21d:97da:833e with SMTP id s197-20020a622cce0000b029021d97da833emr15532224pfs.40.1618048640155; Sat, 10 Apr 2021 02:57:20 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:19 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 2/8] arm64/mm: change __create_pgd_mapping() prototype to accept nr_entries and introduce create_idmap() Date: Sat, 10 Apr 2021 17:56:48 +0800 Message-Id: <20210410095654.24102-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025721_154817_1B3400E4 X-CRM114-Status: GOOD ( 19.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As idmap may have pgd entries more than PTRS_PER_PGD, change the prototype of __create_pgd_mapping() to fix that. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 7 ++++++ arch/arm64/kernel/head.S | 3 +++ arch/arm64/mm/mmu.c | 41 ++++++++++++++++++++++++++------ arch/arm64/mm/mmu_include.c | 10 ++++++-- 4 files changed, 52 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 3c6a7f5988b1..555792921af0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,4 +83,11 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) +extern void __create_pgd_mapping_extend(pgd_t *pgdir, + unsigned int entries_cnt, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags); + #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 840bda1869e9..e19649dbbafb 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -341,6 +341,9 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #if VA_BITS != EXTRA_SHIFT #error "Mismatch between VA_BITS and page size/number of translation levels" #endif + adr_l x4, idmap_extend_pgtable + mov x5, #1 + str x5, [x4] //require expanded pagetable mov x4, EXTRA_PTRS create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 56e4f25e8d6d..30afd6ed275f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -145,6 +145,33 @@ static phys_addr_t pgd_pgtable_alloc(int shift) #include "./mmu_include.c" +int idmap_extend_pgtable; + +/* + * todo: tear down part of idmap + * todo: lock protection for concurrent population + */ +void create_idmap(pgd_t *pgdir, phys_addr_t phys, + phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + u64 ptrs_per_pgd = idmap_ptrs_per_pgd; + +#if CONFIG_IDMAP_PGTABLE_EXPAND + if (idmap_extend_pgtable) + __create_pgd_mapping_extend(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); + else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#endif +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -158,7 +185,7 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, &phys, virt); return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); } @@ -173,7 +200,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, if (page_mappings_only) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(mm->pgd, phys, virt, size, prot, + __create_pgd_mapping(mm->pgd, PTRS_PER_PGD, phys, virt, size, prot, pgd_pgtable_alloc, flags); } @@ -186,7 +213,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); /* flush the TLBs after updating live kernel mappings */ @@ -196,7 +223,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, phys_addr_t end, pgprot_t prot, int flags) { - __create_pgd_mapping(pgdp, start, __phys_to_virt(start), end - start, + __create_pgd_mapping(pgdp, start, PTRS_PER_PGD, __phys_to_virt(start), end - start, prot, early_pgtable_alloc, flags); } @@ -297,7 +324,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, + __create_pgd_mapping(pgdp, PTRS_PER_PGD, pa_start, (unsigned long)va_start, size, prot, early_pgtable_alloc, flags); if (!(vm_flags & VM_NO_GUARD)) @@ -341,7 +368,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); - __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, + __create_pgd_mapping(tramp_pg_dir, PTRS_PER_PGD, pa_start, TRAMP_VALIAS, PAGE_SIZE, prot, __pgd_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ @@ -1233,7 +1260,7 @@ int arch_add_memory(int nid, u64 start, u64 size, IS_ENABLED(CONFIG_KFENCE)) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), + __create_pgd_mapping(swapper_pg_dir, PTRS_PER_PGD, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index e9ebdffe860b..1cf5af7e2aeb 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -233,14 +233,20 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, +static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + pgd_t *pgdp; + + if (likely(entries_cnt == PTRS_PER_PGD)) + pgdp = pgd_offset_pgd(pgdir, virt); + else { + pgdp = pgdir + ((virt >> PGDIR_SHIFT) & (entries_cnt - 1)); + } /* * If the virtual and physical address don't have the same offset From patchwork Sat Apr 10 09:56:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03316C433ED for ; Sat, 10 Apr 2021 09:59:52 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 76181610A2 for ; Sat, 10 Apr 2021 09:59:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76181610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=C8hFvXZq5b+JKuKKC8JmzA4XyZC9FApKhIAHtyNphGQ=; b=KrrCbhdeVVbsp8hjY8DyxLakt hUbHpAxLt5r0QUegaXMbwQ1eCDeqfRwbznIO6qWeymPIQJwZXORchmHw8U4UDRcJjjBFLuktpVQgv RYTDkAhOQCx+jCwjFBE/0+c9wwMcjr7RHqwMWK9QFWG/mlLflxNHcspxss/qfSPehOfwvXoFPcdTR 2pqCXrLxIPpbcx/cH7kaCyUDlaZcAmcapRoiRqwbtwvE8uJzCL81cT7mDnTKzUeD9gM3C6Ty0o2fi IBRdmtRPZj1QvvAyRED5kiiyzB7ptLMzldMeqW9EtQ9GxIsPF9z2UYZLDoKKXcHskGdAkuslou/Na icC6YKiiA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVANP-002bVJ-Er; Sat, 10 Apr 2021 09:58:03 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMr-002bPb-8f for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:29 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=KI1OOnbo35577kxAOfgRYQQwPrpJtshmmXLoljeLfjU=; b=CnHjsDKVhxVdshNTrBKDACyzSV MmvtkEaDGWii7itdWFYYnBGfYFntDqtvQWr6C5iUtixaIHvpcL6KvNorpF/O+zVxdg6a5xQIQc5SP 0vc3loTCThK8Husd3R5rF7F2LRo3TlaiBM6T6Ui0NsSfSGIweOYshWvKNkgR3/vfZqgbDk9qC6Tdq Xrz0+gixd8kZX0uCKMbsYLRIAT87rLAwdbn9LX7zDsSf+ArJwzK5fxHlpoxRzeh4mQ0bapfiYkO30 MDIuOKe+V7d+ZbUa0G+RLvaKJ/Cd+dQ3XLiV4I6201iX6Kw4q6B0ys37Lr6tFXZx07wqOgNrUf6m9 eTQzGwxA==; Received: from mail-pg1-x531.google.com ([2607:f8b0:4864:20::531]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMo-0051ZE-5N for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:28 +0000 Received: by mail-pg1-x531.google.com with SMTP id z16so5680709pga.1 for ; Sat, 10 Apr 2021 02:57:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KI1OOnbo35577kxAOfgRYQQwPrpJtshmmXLoljeLfjU=; b=gUF8s1Auju0vLdkI1fxNI9ZwtajfwOv0TknHMqhjkTfsCdcGUXjRES1gxAc6q6/H/7 Q5YgnH7Qfvmp3oIeTz00lk6BetDnyMUYC+042aJz2DMDO9DzpIBLdqSxunNBjL2o6MBb mQv4hzE3MxiC3yypxTgr5X1bpg+jz10cDbpjLyULfLSVOIFLO64QNN22PCV8xgoie8qV maN6rJTAx1zTKTo4YsTBxpS2SnJsw2A6hFXguCJvo472INT2h5AtdoRfbnljojdW4vQB aaRgBbi+rBCSYN5VHxROxddrJ8w0bNRIZAoT5MHJSoWyV6A3s64NovG/3fKpml/JFCHy nSmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KI1OOnbo35577kxAOfgRYQQwPrpJtshmmXLoljeLfjU=; b=dTs5bpNAJUTyiZP8J4WBtKfC5TOkUJLx57qeYE8FxuXyD2UhOKLhDiq9yYDlXgO1ig uDuZuWFiYObBXdx4HrGbtYXMK+cexJqpsGvXx4b87Ym71CQn7Ycy1fPQn+1CMi5e07jN lP1qrO7uhw7jP25CRvG9QiR4Z7qluYnhQyVs+CqDGmQyFYBrWtf5LS/2ZGGhFxEvopu4 LQLeqwnFAfK9H5GiJhCCZmjTOoTAeruQb7ZD87E6qEv0OCHZRHA+IWaN+GSk2AI1ihhA yjo5Y8yHdQGTS66Mmdt4Qdci2RP/YJ6TDpsi1dLQwZ5QnBk5lF3gGCb5PQMpjdvUqu3i Z1SA== X-Gm-Message-State: AOAM532W/XxrdFjGNs33bAlhxeHfGPFuWl4y6DKncKgQzUzMARFVRedV UdQ2B3zlZeDiLDzvNnR349gOxKMSx1xC X-Google-Smtp-Source: ABdhPJx13Gq1meu7XxZKN1DctJehlJT1d7ZId5rUJf75GpwhKSe0GcZXh4I+/olTkVX4542ZiX0ITQ== X-Received: by 2002:a05:6a00:24ca:b029:23e:e09:8ed0 with SMTP id d10-20020a056a0024cab029023e0e098ed0mr15932657pfv.20.1618048645194; Sat, 10 Apr 2021 02:57:25 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:24 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 3/8] arm64/mm: change __create_pgd_mapping() prototype to accept extra info for allocator Date: Sat, 10 Apr 2021 17:56:49 +0800 Message-Id: <20210410095654.24102-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025726_232810_B3DEB401 X-CRM114-Status: GOOD ( 17.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Incoming allocator needs extra info to get the memory pool info. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 5 +++- arch/arm64/mm/idmap_mmu.c | 5 ++-- arch/arm64/mm/mmu.c | 31 +++++++++++++------------ arch/arm64/mm/mmu_include.c | 39 +++++++++++++++++++------------- 4 files changed, 46 insertions(+), 34 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 555792921af0..42f602528b90 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,11 +83,14 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) +typedef phys_addr_t (*pgtable_alloc)(unsigned long shift, void *data); + extern void __create_pgd_mapping_extend(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags); #endif diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c index 7e9a4f4017d3..9d9fb77ce0e9 100644 --- a/arch/arm64/mm/idmap_mmu.c +++ b/arch/arm64/mm/idmap_mmu.c @@ -35,10 +35,11 @@ void __create_pgd_mapping_extend(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { - __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, prot, pgtable_alloc, flags); + __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, prot, allocator, info, flags); } #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 30afd6ed275f..0f183aaf98c9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -86,7 +86,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc(int shift) +static phys_addr_t __init early_pgtable_alloc(unsigned long unused_a, void *unused_b) { phys_addr_t phys; void *ptr; @@ -113,7 +113,7 @@ static phys_addr_t __init early_pgtable_alloc(int shift) return phys; } -static phys_addr_t __pgd_pgtable_alloc(int shift) +static phys_addr_t __pgd_pgtable_alloc(unsigned long unused_a, void *unused_b) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); BUG_ON(!ptr); @@ -123,9 +123,9 @@ static phys_addr_t __pgd_pgtable_alloc(int shift) return __pa(ptr); } -static phys_addr_t pgd_pgtable_alloc(int shift) +static phys_addr_t pgd_pgtable_alloc(unsigned long shift, void *unused) { - phys_addr_t pa = __pgd_pgtable_alloc(shift); + phys_addr_t pa = __pgd_pgtable_alloc(shift, unused); /* * Call proper page table ctor in case later we need to @@ -154,7 +154,8 @@ int idmap_extend_pgtable; void create_idmap(pgd_t *pgdir, phys_addr_t phys, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { u64 ptrs_per_pgd = idmap_ptrs_per_pgd; @@ -162,13 +163,13 @@ void create_idmap(pgd_t *pgdir, phys_addr_t phys, #if CONFIG_IDMAP_PGTABLE_EXPAND if (idmap_extend_pgtable) __create_pgd_mapping_extend(pgdir, ptrs_per_pgd, - phys, phys, size, prot, pgtable_alloc, flags); + phys, phys, size, prot, allocator, info, flags); else __create_pgd_mapping(pgdir, ptrs_per_pgd, - phys, phys, size, prot, pgtable_alloc, flags); + phys, phys, size, prot, allocator, info, flags); #else __create_pgd_mapping(pgdir, ptrs_per_pgd, - phys, phys, size, prot, pgtable_alloc, flags); + phys, phys, size, prot, allocator, info, flags); #endif } @@ -186,7 +187,7 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, return; } __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, - NO_CONT_MAPPINGS); + NULL, NO_CONT_MAPPINGS); } void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, @@ -201,7 +202,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(mm->pgd, PTRS_PER_PGD, phys, virt, size, prot, - pgd_pgtable_alloc, flags); + pgd_pgtable_alloc, NULL, flags); } static void update_mapping_prot(phys_addr_t phys, unsigned long virt, @@ -214,7 +215,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, } __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, - NO_CONT_MAPPINGS); + NULL, NO_CONT_MAPPINGS); /* flush the TLBs after updating live kernel mappings */ flush_tlb_kernel_range(virt, virt + size); @@ -224,7 +225,7 @@ static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, phys_addr_t end, pgprot_t prot, int flags) { __create_pgd_mapping(pgdp, start, PTRS_PER_PGD, __phys_to_virt(start), end - start, - prot, early_pgtable_alloc, flags); + prot, early_pgtable_alloc, NULL, flags); } void __init mark_linear_text_alias_ro(void) @@ -325,7 +326,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(size)); __create_pgd_mapping(pgdp, PTRS_PER_PGD, pa_start, (unsigned long)va_start, size, prot, - early_pgtable_alloc, flags); + early_pgtable_alloc, NULL, flags); if (!(vm_flags & VM_NO_GUARD)) size += PAGE_SIZE; @@ -369,7 +370,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, PTRS_PER_PGD, pa_start, TRAMP_VALIAS, PAGE_SIZE, - prot, __pgd_pgtable_alloc, 0); + prot, __pgd_pgtable_alloc, NULL, 0); /* Map both the text and data into the kernel page table */ __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); @@ -1261,7 +1262,7 @@ int arch_add_memory(int nid, u64 start, u64 size, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(swapper_pg_dir, PTRS_PER_PGD, start, __phys_to_virt(start), - size, params->pgprot, __pgd_pgtable_alloc, + size, params->pgprot, __pgd_pgtable_alloc, NULL, flags); memblock_clear_nomap(start, size); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 1cf5af7e2aeb..371afc7d4502 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -64,7 +64,8 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { unsigned long next; @@ -73,8 +74,8 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, BUG_ON(pmd_sect(pmd)); if (pmd_none(pmd)) { phys_addr_t pte_phys; - BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(PAGE_SHIFT); + BUG_ON(!allocator); + pte_phys = allocator(PAGE_SHIFT, info); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); pmd = READ_ONCE(*pmdp); } @@ -98,7 +99,9 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) + pgtable_alloc allocator, + void *info, + int flags) { unsigned long next; pmd_t *pmdp; @@ -122,7 +125,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, READ_ONCE(pmd_val(*pmdp)))); } else { alloc_init_cont_pte(pmdp, addr, next, phys, prot, - pgtable_alloc, flags); + allocator, info, flags); BUG_ON(pmd_val(old_pmd) != 0 && pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); @@ -136,7 +139,9 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) + pgtable_alloc allocator, + void *info, + int flags) { unsigned long next; pud_t pud = READ_ONCE(*pudp); @@ -147,8 +152,8 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, BUG_ON(pud_sect(pud)); if (pud_none(pud)) { phys_addr_t pmd_phys; - BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(PMD_SHIFT); + BUG_ON(!allocator); + pmd_phys = allocator(PMD_SHIFT, info); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); pud = READ_ONCE(*pudp); } @@ -164,7 +169,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); + init_pmd(pudp, addr, next, phys, __prot, allocator, info, flags); phys += next - addr; } while (addr = next, addr != end); @@ -184,7 +189,8 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { unsigned long next; @@ -194,8 +200,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (p4d_none(p4d)) { phys_addr_t pud_phys; - BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(PUD_SHIFT); + BUG_ON(!allocator); + pud_phys = allocator(PUD_SHIFT, info); __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); p4d = READ_ONCE(*p4dp); } @@ -222,7 +228,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, READ_ONCE(pud_val(*pudp)))); } else { alloc_init_cont_pmd(pudp, addr, next, phys, prot, - pgtable_alloc, flags); + allocator, info, flags); BUG_ON(pud_val(old_pud) != 0 && pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); @@ -236,7 +242,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), + pgtable_alloc allocator, + void *info, int flags) { unsigned long addr, end, next; @@ -261,8 +268,8 @@ static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_ad do { next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, - flags); + alloc_init_pud(pgdp, addr, next, phys, prot, allocator, + info, flags); phys += next - addr; } while (pgdp++, addr = next, addr != end); } From patchwork Sat Apr 10 09:56:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E88CC433B4 for ; Sat, 10 Apr 2021 10:00:02 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5AFCA610A2 for ; Sat, 10 Apr 2021 10:00:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5AFCA610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=L0zyYUeQRi4j223GCr7vmlf2gD0btLKjdkGVgPVK3TI=; b=AhizcQR9+V8sK3/sblX58EF0U J4M2SlC2Qb8/5lgkWXbaCwiHq5WWOwcCDSO+9E7qv5IQKL9j4n/YnxhHwVPaeRBTmxCJ8TdDIhxbc EeuzHGmOngJw4p1jbCDg4f6h5f5cYJpnFKkbvXLXphnCtduvvDr8qU/Q/h7NLiPuZ+Ek4Cw3tHKQz ilDZz6fkPzoZvAK4meWZdDLfh3TooOCeCTIIcFSHAqMluFS+M5ZyjlLYCNIkUDB0PT0rIk35rWB3k 7wQ0615YCfnF9rtMVsFluNQT6xNOY3nPIbBO7s9o2rNlEsGxhXRAMUHK28Up9QHO32/bAu5TCl7MP BARulIh/g==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVANi-002bZu-Vx; Sat, 10 Apr 2021 09:58:23 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMw-002bQu-6x for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=JDkzlZvoMVLGV8peQuSm7JE3RB4/MO3qP78CDl7IL/c=; b=p1D001DJrqQxDomX0mfhS5x8/7 stMX+L21yPDak7EJHh0SMCSmXsb2WC712ByJlZPZYMeaMlYViIBPRr+ZUf2oF/jVe1qBlNfzYLLWy XGOQCtVodVHB/lCwUPyc36mkXuuUvkDqiQjDt6hr0lLoOD4Lcp/iPcsa6hV7IFoZYMoUwpv5HDoMR OB9t8H+ONRsa9n6kwGxwmnVe2Ax8Mr4LTKQFiUW1WCP09T64HffQGbRpMd5Xh7p6IP4HHIhdPjNWk LzpylXf4fiK9oU/CNavg5geV00irwVi8vwDiCRA7rMAfU2h5upL1bpqRABOsedWxDWvqbPx6DfYfG QJiCeqbA==; Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMt-0051ZX-FM for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:33 +0000 Received: by mail-pl1-x629.google.com with SMTP id g10so3915540plt.8 for ; Sat, 10 Apr 2021 02:57:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JDkzlZvoMVLGV8peQuSm7JE3RB4/MO3qP78CDl7IL/c=; b=XrA0DyEWHBIP2L6R0yhONYUZYVjAbbX10OqRyaxKxj/GV4VhaU0tVE4m2s3ZccBgti Ab73jHB0laNe//6k/GxWnJ9m6tXPuiQFsChV9Mqym9GD3r64oDSDp6mfEAvmEwuuZ2ak K8kyP+/5gFf2tjOn16Joe9iLEqIUHqwSVkRh2yYPRX2xpxPbUguWcNsIN7J9/nJF5GMV oepXeH48jWXcobqnpCxlw8BZB9xvY9okev5VNBA9AApjtHqgFQVDp5am0BgIFbkeehkW IEr8YINFjdHvDGuYe9LmIDZLESLa8rF+ZyXtWji7QsXbQgsPGgJWzILEzYUsVPUTt2hX EnHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JDkzlZvoMVLGV8peQuSm7JE3RB4/MO3qP78CDl7IL/c=; b=QUwtuK6BZcdN4YhbeLRe7/eY7tAURrL6a+//VxkbDWNhln8gfjL/1pPWNgDZQq0OFu Qd0xapfIE8Tp15nG6J4bzEvkazt28EzjpTFjdxqglB2rheZTt5wVYpPpOi56gDpMPwW8 U1CjtlQSf4MlcrnhLSMfS7ODXu8VfNCj1sIGrr6OubUu23hYbEvRIZW5SzpaCez83joD Fdp0/KJO+5s0UdIM0CHPgIJHlhUY7rBLIifMfczW0318AoJBQOHfUc6b6HvUviM+YUTT RiZbI0aGIDJIvbZBapwrzplHS1TNFcEen7IMNdsEqu0j6hRc+WA/lUuDlFR9OKZ+MqAV nk8Q== X-Gm-Message-State: AOAM533TBfDYVClxBmB8RtxPkGjzliB9EpPRLIf2ytKcfgeYKPemziS5 +xSZ9Z47AlDHgLh2k979cBzNP44/CROI X-Google-Smtp-Source: ABdhPJwdCf6loTjnYxNSOtTu5FsK+3m0YsvnEW/TBL+wY/ti4tMsGH45VXUSuiMhwxOTmX7rUCBVkA== X-Received: by 2002:a17:90a:7e4:: with SMTP id m91mr17281497pjm.46.1618048650577; Sat, 10 Apr 2021 02:57:30 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:30 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 4/8] arm64/mm: enable __create_pgd_mapping() to run across different pgtable Date: Sat, 10 Apr 2021 17:56:50 +0800 Message-Id: <20210410095654.24102-5-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025731_533991_408BBD87 X-CRM114-Status: GOOD ( 14.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org PUP/PMD/PTE fixmap can not be shared across different pgtable concurrently. Also change the return type from phys_addr_t to unsigned long, since allocator may return virtual address directly. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 6 +++++- arch/arm64/mm/mmu.c | 6 +++--- arch/arm64/mm/mmu_include.c | 31 ++++++++++++++++++++----------- 3 files changed, 28 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 42f602528b90..6e9f1e218300 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,7 +83,7 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) -typedef phys_addr_t (*pgtable_alloc)(unsigned long shift, void *data); +typedef unsigned long (*pgtable_alloc)(unsigned long shift, void *data); extern void __create_pgd_mapping_extend(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, @@ -93,4 +93,8 @@ extern void __create_pgd_mapping_extend(pgd_t *pgdir, void *info, int flags); +#define NO_BLOCK_MAPPINGS BIT(0) +#define NO_CONT_MAPPINGS BIT(1) +#define NO_FIXMAP BIT(2) + #endif diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 0f183aaf98c9..628752c3cfd0 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -86,7 +86,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc(unsigned long unused_a, void *unused_b) +static unsigned long __init early_pgtable_alloc(unsigned long unused_a, void *unused_b) { phys_addr_t phys; void *ptr; @@ -113,7 +113,7 @@ static phys_addr_t __init early_pgtable_alloc(unsigned long unused_a, void *unus return phys; } -static phys_addr_t __pgd_pgtable_alloc(unsigned long unused_a, void *unused_b) +static unsigned long __pgd_pgtable_alloc(unsigned long unused_a, void *unused_b) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); BUG_ON(!ptr); @@ -123,7 +123,7 @@ static phys_addr_t __pgd_pgtable_alloc(unsigned long unused_a, void *unused_b) return __pa(ptr); } -static phys_addr_t pgd_pgtable_alloc(unsigned long shift, void *unused) +static unsigned long pgd_pgtable_alloc(unsigned long shift, void *unused) { phys_addr_t pa = __pgd_pgtable_alloc(shift, unused); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index 371afc7d4502..adad0f93cd53 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -1,8 +1,5 @@ // SPDX-License-Identifier: GPL-2.0-only -#define NO_BLOCK_MAPPINGS BIT(0) -#define NO_CONT_MAPPINGS BIT(1) - static bool pgattr_change_is_safe(u64 old, u64 new) { /* @@ -38,11 +35,14 @@ static bool pgattr_change_is_safe(u64 old, u64 new) } static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) + phys_addr_t phys, pgprot_t prot, int flags) { pte_t *ptep; - ptep = pte_set_fixmap_offset(pmdp, addr); + if (likely(!(flags & NO_FIXMAP))) + ptep = pte_set_fixmap_offset(pmdp, addr); + else + ptep = pte_offset_kernel(pmdp, addr); do { pte_t old_pte = READ_ONCE(*ptep); @@ -58,7 +58,8 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); - pte_clear_fixmap(); + if (likely(!(flags & NO_FIXMAP))) + pte_clear_fixmap(); } static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, @@ -91,7 +92,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, (flags & NO_CONT_MAPPINGS) == 0) __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - init_pte(pmdp, addr, next, phys, __prot); + init_pte(pmdp, addr, next, phys, __prot, flags); phys += next - addr; } while (addr = next, addr != end); @@ -106,7 +107,10 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, unsigned long next; pmd_t *pmdp; - pmdp = pmd_set_fixmap_offset(pudp, addr); + if (likely(!(flags & NO_FIXMAP))) + pmdp = pmd_set_fixmap_offset(pudp, addr); + else + pmdp = pmd_offset(pudp, addr); do { pmd_t old_pmd = READ_ONCE(*pmdp); @@ -133,7 +137,8 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys += next - addr; } while (pmdp++, addr = next, addr != end); - pmd_clear_fixmap(); + if (likely(!(flags & NO_FIXMAP))) + pmd_clear_fixmap(); } static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, @@ -207,7 +212,10 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, } BUG_ON(p4d_bad(p4d)); - pudp = pud_set_fixmap_offset(p4dp, addr); + if (likely(!(flags & NO_FIXMAP))) + pudp = pud_set_fixmap_offset(p4dp, addr); + else + pudp = pud_offset(p4dp, addr); do { pud_t old_pud = READ_ONCE(*pudp); @@ -236,7 +244,8 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys += next - addr; } while (pudp++, addr = next, addr != end); - pud_clear_fixmap(); + if (likely(!(flags & NO_FIXMAP))) + pud_clear_fixmap(); } static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, From patchwork Sat Apr 10 09:56:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B68E2C433ED for ; Sat, 10 Apr 2021 10:00:38 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 55C9861001 for ; Sat, 10 Apr 2021 10:00:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 55C9861001 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=66k47o0H/P9/dE4VyqRVZHLcT5BgzDdnr5R2qWw8Z2Q=; b=EAx8pUYhgRrP2nc7+5k8PQ/gR 9gu9aZsJOOC5IN4KKJ/ZsP8HfkihzGw8VzYNMNDp6XN09POd0b7PYM8+qBFWKGBpA1DGnYMKL7J+N cMC7koRvpcTFRAXg+qnRhOTQevnKQE443ey/QD7qbrNhfWW7P0wy1R2QGudRQy/u2DNYTgbCWYbWI RTKVMqLRej7yqG1mQBxvfjjt2TPPd2JQcmsa6ggWgNMxSlbNLhDQoTvbCyfbKtbEndfKxnX4ZywCF 8J6OGG3IJvKAO4EiAD1W1Vz2FeQdBonU9giKZOMX24Y1diqMmCVMRhOuycWrAo00ox4dQywPvUcjC O5u2JaAwQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVANv-002bdU-Ui; Sat, 10 Apr 2021 09:58:36 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAN1-002bRK-Gf for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:39 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=ebzxZEiBIG4H4lXdCyPtungiZLvBTkaali2R4+XSI1E=; b=HmV5xQrRf8JXSf7qsnZjG+41+S eQB5zPKK/Df3nAD3bNutZSbR+6WRzNrDwSQmSt4BN2G0hGIeUheNVVJeAZVbpm7VcKc5fH9u/3C6W zbahSze8CGOYvZAV5wyLkpdvOBk8P/etQVsSRwYNiSpmoFd0gMfIH3DzpFT1NRr1r4U/sPqYDBhx6 7qSv+2WpZGywZuDG8VGVDug0BcMpJYaB5iAODZu4gmvQIW3PX5XxSVjx1QISJQ2MZ2lckZWdkV8XP gRYYTUjYZHyqgEmiStPgcSffn+Mw7yCc4eCQnzFJKFG1YtkjyJAIxa/nOVVeeoi3cH3qO2S56iw7A JdzWm7Kg==; Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMy-0051Zn-VE for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:38 +0000 Received: by mail-pj1-x102b.google.com with SMTP id ot17-20020a17090b3b51b0290109c9ac3c34so6187178pjb.4 for ; Sat, 10 Apr 2021 02:57:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ebzxZEiBIG4H4lXdCyPtungiZLvBTkaali2R4+XSI1E=; b=Fh+13w75g3g8mU6JGXclOI+3G/txBOJkK3A3GlhwlmVzSqj6wQwJDtPZ3gCsWnmoMV 2xi9vxd5A1XIEdSB5n5Ekpva6ge3w9lo6SYuxAXlPMdYWowrx6aKVxDEhJGCrM8fsFSK QK7tg+MbPv3dfmpwmNsp4xjb+OqdqBtOQzdbmmIy1vhrnEhGFcSwdfnHTFDo8ImNZZSO SNAzbBYBX595POok74qhtjscAIPZrlBv3LZzVGhGS1KatTwbBgBHz5ftYCXGhEXajQY0 duMs8NgGxx6Mhijl3QbLsGGErAkkTwEgH8nu3Hd/YFg9jzwnhLtDYdJx2emrZaV1MCN2 yfwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ebzxZEiBIG4H4lXdCyPtungiZLvBTkaali2R4+XSI1E=; b=GwaU3VTsDOElbdcaieo33PQhg3tEtN4pqVXVhfNt4FK594P6FUmR9wGHch3wPcBbM8 acaebe7vl149dxuUbw3RIoIfdobpctOGUxvOI56vX5gYc1MSwvBuLqwcxIX7oHdvyJHm LbVaCkZraX1xDxEVMhTw55qAVcxYTtLOXjwLMpkh3wX8Ir4CwQeBBZycDAgqT/AEN91i /VhwboWQNMoiZu/jr0lfuyMIATH6zhkiAyoOsiKXXltkTEbq/Jm0xlQkRfx+edECEFTi g7B7EITnWGEJwHU3xnNU5VIR+wnJnS0isxuJVju6F4LN7NpKwTcDwMHbTdXDIvcBtCUs 0tMw== X-Gm-Message-State: AOAM5306epbYz3qdtqrMVqSCbrCPQNoYf/OyrmZ3lo26DSL/cy16f38W VqvJMSlaecEISF7kIR40+CD/9QRm69Vc X-Google-Smtp-Source: ABdhPJxnr7uMB5MKBnBuAWtsS2Acp+r7BDqCchInppccITVVYuezEPXRijon5dR7ErDjn6WhtOepug== X-Received: by 2002:a17:903:22c2:b029:ea:b40b:7f88 with SMTP id y2-20020a17090322c2b02900eab40b7f88mr4322891plg.61.1618048656036; Sat, 10 Apr 2021 02:57:36 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:35 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 5/8] arm64/mm: make trans_pgd_idmap_page() use create_idmap() Date: Sat, 10 Apr 2021 17:56:51 +0800 Message-Id: <20210410095654.24102-6-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025737_025489_E7B4BA18 X-CRM114-Status: GOOD ( 14.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org At present, trans_pgd_idmap_page() has its own logic to set up idmap. To make code simple, it can reuse create_idmap(). Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 9 +++++ arch/arm64/mm/trans_pgd.c | 59 +++++++++++++++----------------- 2 files changed, 36 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 6e9f1e218300..f848a0300228 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -93,6 +93,15 @@ extern void __create_pgd_mapping_extend(pgd_t *pgdir, void *info, int flags); +extern int idmap_extend_pgtable; + +extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, + phys_addr_t size, + pgprot_t prot, + pgtable_alloc allocator, + void *info, + int flags); + #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_FIXMAP BIT(2) diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 527f0a39c3da..004ccbadd647 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -274,6 +274,14 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, return 0; } +static unsigned long allocator_trans_alloc(unsigned long unused, void *info) +{ + unsigned long *p; + + p = trans_alloc(info); + return (unsigned long)p; +} + /* * The page we want to idmap may be outside the range covered by VA_BITS that * can be built using the kernel's p?d_populate() helpers. As a one off, for a @@ -287,38 +295,25 @@ int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans_ttbr0, unsigned long *t0sz, void *page) { - phys_addr_t dst_addr = virt_to_phys(page); - unsigned long pfn = __phys_to_pfn(dst_addr); - int max_msb = (dst_addr & GENMASK(52, 48)) ? 51 : 47; - int bits_mapped = PAGE_SHIFT - 4; - unsigned long level_mask, prev_level_entry, *levels[4]; - int this_level, index, level_lsb, level_msb; - - dst_addr &= PAGE_MASK; - prev_level_entry = pte_val(pfn_pte(pfn, PAGE_KERNEL_EXEC)); - - for (this_level = 3; this_level >= 0; this_level--) { - levels[this_level] = trans_alloc(info); - if (!levels[this_level]) - return -ENOMEM; - - level_lsb = ARM64_HW_PGTABLE_LEVEL_SHIFT(this_level); - level_msb = min(level_lsb + bits_mapped, max_msb); - level_mask = GENMASK_ULL(level_msb, level_lsb); - - index = (dst_addr & level_mask) >> level_lsb; - *(levels[this_level] + index) = prev_level_entry; - - pfn = virt_to_pfn(levels[this_level]); - prev_level_entry = pte_val(pfn_pte(pfn, - __pgprot(PMD_TYPE_TABLE))); - - if (level_msb == max_msb) - break; - } - - *trans_ttbr0 = phys_to_ttbr(__pfn_to_phys(pfn)); - *t0sz = TCR_T0SZ(max_msb + 1); + pgd_t * pgdir = trans_alloc(info); + int flags = NO_FIXMAP; + unsigned long base, step, level, va_bits; + +#ifdef CONFIG_ARM64_64K_PAGES + base = 16; + step = 13; +#elif CONFIG_ARM64_4K_PAGES + base = 12; + step = 9; +#endif + create_idmap(pgdir, virt_to_phys(page), PAGE_SIZE, PAGE_KERNEL_EXEC, + allocator_trans_alloc, info, flags); + + *trans_ttbr0 = phys_to_ttbr(__virt_to_phys(pgdir)); + level = CONFIG_PGTABLE_LEVELS + idmap_extend_pgtable ? 1 : 0; + va_bits = base + step * level; + va_bits = min(va_bits, vabits_actual); + *t0sz = 64 - va_bits; return 0; } From patchwork Sat Apr 10 09:56:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D32EC43462 for ; Sat, 10 Apr 2021 10:00:53 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1921F610A2 for ; Sat, 10 Apr 2021 10:00:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1921F610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=KOybZY7To4V21iK6gWRe/mF2R/jDk8n4PArqRhHCzXE=; b=bCNjbM10yNmDGRX6CMthncXfo 9DRcn7Bvfl3VFbOfQQb34wgebCKldYhSqVfvi1p4rH7hNceJKXOnnZ9etSme97P9dCdKKei0kuQ1f 8sNQfntrwkXAeunUCAMQ904NlsyDx6UTDPT3u5QWC5BPMoc8Tte9HNYpmJkzPK29lx5LqvZft7G0I GO9Sg5wDbADVqm82G7ujoXpb08QqVTOwwTasyI5gq/sbSC87EJorR1vif6PxK0KCpjncnhECFj+GX dctAVJ7Xkve77wID1jNd3osNUDnVDxIK+rUR54wfxjazmghkhYZqCL97zD4Qpwlh9FYi5qK/BGhjs a6LL6Lgmg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVAO7-002bgO-9D; Sat, 10 Apr 2021 09:58:47 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAN6-002bS2-LA for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=4k9OCzuAlGUr1qN4c20ToMLFDYt1W8fv6NNCJVMwj6E=; b=BYeF2Vw2pFaAp3BxDOU/bsBp+0 Zwo431pfboPPdtkixChh1dU0yzXcEzG4gJ+VxmmFKUIN1DoWoIr3Lk1cgZOxRP5vsecSxJNDujvDK Gkx14jvx5x0YdtDbUdZLkVZllQNHMnPswWZV9pZLWQDVDHbDr38WAW/3Yy9x0gm12YUUPzwA0otOf XK/4zDVsmktnR2N+MCFug9FwJ7iXpyWkR4IwFF2Wvdq4CiDToguXABo2dzvY2WKp65L1XnMfPLVWh /et/FeJt58eGWhE1+o1UHcyIGQQTEBC23Hp3GqEypkw2qCgLkEOVltpq/drwat2tmc4AurMtScaaE 1gXy6CEw==; Received: from mail-pg1-x535.google.com ([2607:f8b0:4864:20::535]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAN4-0051aA-7H for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:43 +0000 Received: by mail-pg1-x535.google.com with SMTP id d10so5660256pgf.12 for ; Sat, 10 Apr 2021 02:57:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4k9OCzuAlGUr1qN4c20ToMLFDYt1W8fv6NNCJVMwj6E=; b=tTs6ql0MPJNA7zh7rONg2jJCmvqJ9dJhGUGCk/UywB2JrjstCQDYDikt80Ek4RD6u7 /alVX+hx2Vl7SkTAFiceeFwKeRih7HDbcixlgT24pUpidGRzancSJCIG6KHPeetEWp6Q xSLfYwys6fROXgXnAZ8M9vlyYMg7JqrN+Lsdx1HftFgN2KNfmFtTeSWR82vcoMmfYhe5 5KHpXctP/I+/UF9NgyMusmTtXojCG5re1lgiIINBNWOJd7hEL6liV8O2PcjABb21pjQS NZq3pws6gV/nvF2hECX7ZuTHpL6YT3a6IAlYerC1xTlhndNqc2e59vSrnOHITLDP0VRf ylGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4k9OCzuAlGUr1qN4c20ToMLFDYt1W8fv6NNCJVMwj6E=; b=b8tUyKyp7h+QH8xUxV3paeA8IxG7c/Fhm2lehXjm+c4eu3fXBvdj6Z59w/YZaWBf+1 /+gaO+dV/WYXIGh/KMvLGRNguOPz0/TvQmvq8xz4KkWKS9ZkGRoN7Y8QcXyLe0JZgFys ecfOGDpAdRf9OqIeqKrQFBxq8lRg7f+OPYQfFWfyvLH295MFpWoEQMzYorBfqJgoK8bF 65zkkjdKDTATJ/Q+rUHJjp6E4sX2l3oa9dNRf0a/bLGDS3ttvStiKAwMhbYKalZ8cFMw Qx9KXtiojy+3cYIckIVGvNG5cI9JBjWx3NhH5At1ViaiQw0AuYOsiJx31UVu88hOls/G yvUg== X-Gm-Message-State: AOAM531K1iDwmgIecJF+Y4Tji4JgzyhEDtE88+XZKRh/kBWNRvh4rETa ZfohHOBBlmusKBXyJ4qdo27A0xqREg== X-Google-Smtp-Source: ABdhPJxmkEniuTDO2wFQeKRt+swrMZkECeyIwnPsK1LAT+FR1lDvsfTLyUGdzlao6oz9GLHP8a3/Ww== X-Received: by 2002:a65:5c4d:: with SMTP id v13mr559053pgr.161.1618048661298; Sat, 10 Apr 2021 02:57:41 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:41 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 6/8] arm64/mm: introduce pgtable allocator for head Date: Sat, 10 Apr 2021 17:56:52 +0800 Message-Id: <20210410095654.24102-7-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025742_283286_3E6A93C8 X-CRM114-Status: GOOD ( 10.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From the view of __create_pgd_mapping(), both idmap_pg_dir and init_pg_dir can be treated as a memory pool. Introduce an allocator working on them, so __create_pgd_mapping() can create mapping. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/mm/mmu.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 628752c3cfd0..b546e47543e2 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -86,6 +86,28 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); +struct mempool { + unsigned long start; + unsigned long size; + unsigned long next_idx; +}; + +struct mempool cur_pool; + +void set_cur_mempool(unsigned long start, unsigned long size) +{ + cur_pool.start = start; + cur_pool.size = size; + cur_pool.next_idx = 0; +} + +unsigned long __init head_pgtable_alloc(unsigned long unused_a, void *unused_b) +{ + unsigned long idx = cur_pool.next_idx++; + + return cur_pool.start + (idx << PAGE_SHIFT); +} + static unsigned long __init early_pgtable_alloc(unsigned long unused_a, void *unused_b) { phys_addr_t phys; From patchwork Sat Apr 10 09:56:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23DE1C43461 for ; Sat, 10 Apr 2021 10:00:55 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B3ED861001 for ; Sat, 10 Apr 2021 10:00:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B3ED861001 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cZsDL54/Lcewphqf7PXftSM7Nza8qvdCoPxgohMIOJ8=; b=Yxt8OUJEZi8TNr+wULUbRXfbq /k+ac1ZEaTqmvOgubx9SICWUVrZ1Ntej5HxndZO3vXh2yJsRq6VQdrkPrrJWZ7no/4EKJgBxSL/DI fXzwtyvLx8fs5XgZOoFnKK5733fD28hC1B9jtkUi8Vbezu0ssav9cSAJ5omRBehguLhSz3KADrWAY lWl6EkI0S0cDvCyOsNuYeN1sD6cpR8gmJAG9Km/d3PboinoTstW4y5ZDlym04Pb1r6JJtTkyXN3u/ m8gnDRNnFejl4rSgNvsCBJ0FKSGtqwyeV3HIYrXoIeZa8ucQbQPSE/S4hIU39eAmUjZeTJ/yF90AY ABMesqdsA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVAOK-002bjs-Rl; Sat, 10 Apr 2021 09:59:01 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVANC-002bTC-6L for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=zrd7bBLovHvnijnmMdqZkS7Wy6EC6D6qV3B22r4kYsY=; b=mF/C9tLSbMWvTc0P54tGkYJtvK u4G5wfx3XPMqk6x4YyS8eAX/teqAivbyt6f4KHgK/R8Ix+jGj5xQNK/xW8bDe5wrLRIywuZNzQTrq 7XPxquAxLQBZG/zIJzG6vD6p6euxVKCBLYk7rThVRirpx3n43naMB/em72Ul0RCjxO3ByWJ+PsumA DLc3eBa9UY5HT2L7iwn5ucqo0JVKx3BhQvhBSQNLrNUjNOw0TMBRKv+6sg4imZtWMhXtI9dAtXnKI OQHIMZ3WPo2iAsJ42tdvpODpYWh+wAu5TfgCaN8I4czhT17HOSvz34+3uquk854zoQWtDB4cNkk3v sBCIbVQg==; Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAN9-0051aR-NO for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:49 +0000 Received: by mail-pf1-x436.google.com with SMTP id w8so2389474pfn.9 for ; Sat, 10 Apr 2021 02:57:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zrd7bBLovHvnijnmMdqZkS7Wy6EC6D6qV3B22r4kYsY=; b=kmDTbLootSstV4Gv4MndxXmNuae44mMGjYz3am4sjq0zcKh2hELFopGiJXbINQ1dQP 5r5zUKyk603h5s2OBJWJhRTcP1ynH1XvpJ3qJQW9QhjjnNOH5QeEz8vIk+J58Ka4zpSY Js+j580RZY6j1NVQr4fVQzzgrh2RMHKS/W+hVKIwdMciILw9UiqVx4DtZgDJNIDOUkBz WDBP0kQT9rBw4T9Xtu7IDcQgGwjAIzxxba40+C+uI3WHdcqnAXvNyFJbNXFB+GWk6tJe CS10w5bfsVztH1pnf9x9/3S9i1QIYnC6jgtKrI3f7m3qB7rvXn3rDh+Ez7T6pCd17L6h F9wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zrd7bBLovHvnijnmMdqZkS7Wy6EC6D6qV3B22r4kYsY=; b=Ei3bK8ewIBHUuTbKUBdBLPuqCk0jEYImFYssRUn09MgEnB2nYrjODKOGqhRMaqGzzL /4eBapQligv9RJwR//iWoyy0YZytbFHXP/pZPcr9yCbbPzD/SqM9e5omjQwk6b4Su2Pm cfgOxCgyNhNoiYDlTegCwsfSK8SJMHiVMoJGzOisCoI2uN1HNAv19RXqQSlGXBJcr6Z5 4SR9wwRznXK6Pjcixoc2lmtJ7iQO5kSHQLjEnon+N4HIQQAtyF5t/mBo4SIs1ndA1V7r kYGE6Xk3novzczX4MiJhC2NxsD66q6lExs3RtN7zlkROkI4GJourVS542Fh6rY1VdtBN XLOA== X-Gm-Message-State: AOAM531qbsKep2yBhmAHlSDEuphV5eExBg1d7E8JyhtoZD2SN1mROxFC 85gtUgWxi4DxHgQatPBCv1C3XnTFnfNk X-Google-Smtp-Source: ABdhPJyvVmbXrQ1e7NSFfhexzrkc97/ZpolsbiKNztm6qpeZH7HwhRM4NMF8MivNkU3vQxp3YhiTsQ== X-Received: by 2002:a63:fd0a:: with SMTP id d10mr9886701pgh.94.1618048666772; Sat, 10 Apr 2021 02:57:46 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:46 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 7/8] arm64/pgtable-prot.h: reorganize to cope with asm Date: Sat, 10 Apr 2021 17:56:53 +0800 Message-Id: <20210410095654.24102-8-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025747_781665_236C5C17 X-CRM114-Status: GOOD ( 12.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org In order to refer PAGE_KERNEL_EXEC in head.S, reorganize this file. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgtable-prot.h | 34 +++++++++++++++++---------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index 9a65fb528110..424fc5e6fd69 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -33,9 +33,6 @@ extern bool arm64_use_ng_mappings; -#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) -#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) - #define PTE_MAYBE_NG (arm64_use_ng_mappings ? PTE_NG : 0) #define PMD_MAYBE_NG (arm64_use_ng_mappings ? PMD_SECT_NG : 0) @@ -49,6 +46,26 @@ extern bool arm64_use_ng_mappings; #define PTE_MAYBE_GP 0 #endif +#define PAGE_S2_MEMATTR(attr) \ + ({ \ + u64 __val; \ + if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) \ + __val = PTE_S2_MEMATTR(MT_S2_FWB_ ## attr); \ + else \ + __val = PTE_S2_MEMATTR(MT_S2_ ## attr); \ + __val; \ + }) + +#endif /* __ASSEMBLY__ */ + +#ifdef __ASSEMBLY__ +#define PTE_MAYBE_NG 0 +#define __pgprot(x) (x) +#endif + +#define _PROT_DEFAULT (PTE_TYPE_PAGE | PTE_AF | PTE_SHARED) +#define _PROT_SECT_DEFAULT (PMD_TYPE_SECT | PMD_SECT_AF | PMD_SECT_S) + #define PROT_DEFAULT (_PROT_DEFAULT | PTE_MAYBE_NG) #define PROT_SECT_DEFAULT (_PROT_SECT_DEFAULT | PMD_MAYBE_NG) @@ -71,15 +88,7 @@ extern bool arm64_use_ng_mappings; #define PAGE_KERNEL_EXEC __pgprot(PROT_NORMAL & ~PTE_PXN) #define PAGE_KERNEL_EXEC_CONT __pgprot((PROT_NORMAL & ~PTE_PXN) | PTE_CONT) -#define PAGE_S2_MEMATTR(attr) \ - ({ \ - u64 __val; \ - if (cpus_have_const_cap(ARM64_HAS_STAGE2_FWB)) \ - __val = PTE_S2_MEMATTR(MT_S2_FWB_ ## attr); \ - else \ - __val = PTE_S2_MEMATTR(MT_S2_ ## attr); \ - __val; \ - }) + #define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) /* shared+writable pages are clean by default, hence PTE_RDONLY|PTE_WRITE */ @@ -106,6 +115,5 @@ extern bool arm64_use_ng_mappings; #define __S110 PAGE_SHARED_EXEC #define __S111 PAGE_SHARED_EXEC -#endif /* __ASSEMBLY__ */ #endif /* __ASM_PGTABLE_PROT_H */ From patchwork Sat Apr 10 09:56:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A815C433ED for ; Sat, 10 Apr 2021 10:01:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F773610A2 for ; Sat, 10 Apr 2021 10:01:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F773610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ev9ATT0yvdEW2r7xofLE1D+CeuuCHe2opGdox1xOU9o=; b=Lf4jVbjOWmr7m2G3S+OmA/DVu cEgY7jDmleaBbUJZy5+jP587f1SBK5S7Ept33ReRu4tZsvC5e5slFhSaM0K3orJDFvWQv6oCYqBm5 XU/Yxb6GfnaAOTasBTtIDaHzxmnESBvNbJ9EyV+bYtybFqMNA0cQW0ML66GMeTp5Gh1tOtU58pH7Q czN/heGQHd/ynhAc+/7fw0kZ/j9XzEI1vesKGTt3JXZqb0vMBwr3p09k7R0yZxXWd/gRaVB6TcQFC Lbby/KllHbvjbm26JLw/lV7y0dTw8eKFu5Q7nHKx+xOfpkOtQ29Z3Ye2JH7Oumq10pmfVADJ/nSkS i8E6622uQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVAOj-002bre-Py; Sat, 10 Apr 2021 09:59:28 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVANH-002bTl-SN for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=80v3vFe215kri+0o/k30XuUjT5mfA41TLcwr3Ulv96o=; b=LH3JJytVz1595KNRQvegefBrLl wyCRrNSRJ2XmLlzA9ohcMNzuL/jW34Vs+rceMRPZ2EQ/SkJKM3wRMKaJgrA18mZ+NW8nIbRGD38Gg q5CbKnbv2EVQb/X0wOBf0gjCz8AxO7h1YwR6Zw5XJmQlp6Q6XkuO2jXn8Phm5YtDSC5GVKmggIJd9 0dqsRBZbHRnpgpo3F0pQW8lwIy+lXYIqXUWtZz6Bqi92385xJeP+zEstSgAcME3/Bu8OD5+6/hgFf qtdGWDKzPu3G6lyezFI0TYr9yebyRVZBgou1OZQW2olm0lBfQkBT5qXr3HemvjSQdyYfBH7gORo3g vnM9vsvA==; Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVANE-0051ar-PH for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:54 +0000 Received: by mail-pl1-x62a.google.com with SMTP id p10so3946787pld.0 for ; Sat, 10 Apr 2021 02:57:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=80v3vFe215kri+0o/k30XuUjT5mfA41TLcwr3Ulv96o=; b=LhtLlALC+5z1pYJSGS4S6JU+yfgH8LOx4yXrQsibhjvQVoZebw5GASueQdTp4V2sxr M4hcU8JUccWRxbQm9/vt5Nl3O3YlVG6xKt77aHHbUgolitKUam18LQ6tt9T/UYYm/0yF E8XIt69b2ngCOVRgacf88rG4G58IERpb0ak/Z/YTdgK5TcfeRPEfxKOPY/54zZHK3X8I +j3+yPCrtUgUjLqt3zOVu1haMMYWS/KwkqmpRJLaMmoAKAbnG9otTciVhaJncDNd06eB 3d6SZ1QciMBn/aTO718fdCFpyCsoApE3KnYXgz29dHmU7E5gB/GwJxRWF5VQGaXc4rLV z7lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=80v3vFe215kri+0o/k30XuUjT5mfA41TLcwr3Ulv96o=; b=bqaAXjoZyjfrby+uAl39R758NOckacFbArtoS2w3BgdTOd/4zFGk7PbY7a3jMGD8gy ytm0ezDox7BJV7SJUUVvFhWs1FN2SaQOSN+JZ6UAIJzYca2U5GhZ/auQFRK6oh65oSZQ Gisv7Fy8FiD9LJibUVrgaeBFXGNiYLK8RAIuMtagO3MhubpugoThCdIYmoXo9kMWr/H5 hN18MFAg6SIimJk4O6oC6FSaTAMfJqXTyTZUID7NgX1Pmvka9KH+i3BJYPhTzXoO6GGy e8uUmLPOMf6fplSTiXFtWsz0IsOjwhbA7JpD/kFVBKdtZdc3NN++mX7S6iM9ETJTGRCB hrtA== X-Gm-Message-State: AOAM531poRnvSsH5AFyBMODhmnNO8CvKL1sKWuHEDE405X2bfSGyLRgh Uv6EXCyVMxh2bEXy0z7U6fvdH5jhi77d X-Google-Smtp-Source: ABdhPJyny5WDhNZvmOTPl8nyiEGR+/+fQmyqqoY+nfYFowNlKq6fKk9edyRE+kpEO+Y/RJRIUbAg3A== X-Received: by 2002:a17:902:d70f:b029:ea:83ae:2336 with SMTP id w15-20020a170902d70fb02900ea83ae2336mr5813147ply.4.1618048671857; Sat, 10 Apr 2021 02:57:51 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:51 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 8/8] arm64/head: convert idmap_pg_dir and init_pg_dir to __create_pgd_mapping() Date: Sat, 10 Apr 2021 17:56:54 +0800 Message-Id: <20210410095654.24102-9-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025752_850381_9C933C9C X-CRM114-Status: GOOD ( 20.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 5 + arch/arm64/kernel/head.S | 187 +++++++------------------------ arch/arm64/mm/mmu.c | 9 ++ 3 files changed, 55 insertions(+), 146 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index f848a0300228..128d784d78d4 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -8,6 +8,9 @@ #ifndef __ASM_PGALLOC_H #define __ASM_PGALLOC_H +#include + +#ifndef __ASSEMBLY__ #include #include #include @@ -102,6 +105,8 @@ extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, void *info, int flags); +#endif + #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_FIXMAP BIT(2) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e19649dbbafb..3b0c9359ab70 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -28,6 +28,8 @@ #include #include #include +#include +#include #include #include #include @@ -93,6 +95,8 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag + adrp x4, init_thread_union + add sp, x4, #THREAD_SIZE bl __create_page_tables /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -121,135 +125,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b __inval_dcache_area // tail call SYM_CODE_END(preserve_boot_args) -/* - * Macro to create a table entry to the next page. - * - * tbl: page table address - * virt: virtual address - * shift: #imm page table shift - * ptrs: #imm pointers per table page - * - * Preserves: virt - * Corrupts: ptrs, tmp1, tmp2 - * Returns: tbl -> next level table page address - */ - .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 - add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp2, \tmp1 - orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type - lsr \tmp1, \virt, #\shift - sub \ptrs, \ptrs, #1 - and \tmp1, \tmp1, \ptrs // table index - str \tmp2, [\tbl, \tmp1, lsl #3] - add \tbl, \tbl, #PAGE_SIZE // next level table page - .endm - -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - * shift: shift used to transform virtual address into index - * ptrs: number of entries in page table - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend, shift, ptrs - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, ptrs, istart, iend, count - lsr \iend, \vend, \shift - mov \istart, \ptrs - sub \istart, \istart, #1 - and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1) - mov \istart, \ptrs - mul \istart, \istart, \count - add \iend, \iend, \istart // iend += (count - 1) * ptrs - // our entries span multiple tables - - lsr \istart, \vstart, \shift - mov \count, \ptrs - sub \count, \count, #1 - and \istart, \istart, \count - - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: start address to map - * vend: end address to map - we map [vstart, vend] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * pgds: the number of pgd entries - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, vend, flags - * Corrupts: tbl, rtbl, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, pgds, istart, iend, tmp, count, sv - add \rtbl, \tbl, #PAGE_SIZE - mov \sv, \rtbl - mov \count, #0 - compute_indices \vstart, \vend, #PGDIR_SHIFT, \pgds, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #PTRS_PER_PUD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #PTRS_PER_PMD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #PTRS_PER_PTE, \istart, \iend, \count - bic \count, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \count, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - /* * Setup the initial page tables. We only setup the barest amount which is * required to get the kernel running. The following sections are required: @@ -344,9 +219,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adr_l x4, idmap_extend_pgtable mov x5, #1 str x5, [x4] //require expanded pagetable - - mov x4, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 #else /* * If VA_BITS == 48, we don't have to configure an additional @@ -356,25 +228,50 @@ SYM_FUNC_START_LOCAL(__create_page_tables) str_l x4, idmap_ptrs_per_pgd, x5 #endif 1: - ldr_l x4, idmap_ptrs_per_pgd - mov x5, x3 // __pa(__idmap_text_start) - adr_l x6, __idmap_text_end // __pa(__idmap_text_end) + stp x0, x1, [sp, #-64]! + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #32] + stp x6, x7, [sp, #16] - map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 + adrp x0, idmap_pg_dir + adrp x1, idmap_pg_end + sub x1, x1, x0 + bl set_cur_mempool + + adrp x1, __idmap_text_start + adr_l x2, __idmap_text_end + sub x2, x2, x1 + ldr x3, =PAGE_KERNEL_EXEC + adr_l x4, head_pgtable_alloc + mov x5, #0 + mov x6, #NO_FIXMAP + bl create_idmap /* * Map the kernel image (starting with PHYS_OFFSET). */ adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) - add x5, x5, x23 // add KASLR displacement - mov x4, PTRS_PER_PGD - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) + adrp x1, init_pg_end + sub x1, x1, x0 + bl set_cur_mempool - map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 + mov x1, PTRS_PER_PGD + adrp x3, _text // runtime __pa(_text) + mov_q x4, KIMAGE_VADDR // compile time __va(_text) + add x4, x4, x23 // add KASLR displacement + adrp x5, _end // runtime __pa(_end) + sub x5, x5, x3 // _end - _text + + ldr x3, =PAGE_KERNEL_EXEC + adr_l x4, head_pgtable_alloc + mov x5, #0 + mov x6, #NO_FIXMAP + + bl create_init_pgd_mapping + ldp x6, x7, [sp, #16] + ldp x4, x5, [sp, #32] + ldp x2, x3, [sp, #48] + ldp x0, x1, [sp], #-64 /* * Since the page tables have been populated with non-cacheable @@ -402,8 +299,6 @@ SYM_FUNC_END(__create_page_tables) * x0 = __PHYS_OFFSET */ SYM_FUNC_START_LOCAL(__primary_switched) - adrp x4, init_thread_union - add sp, x4, #THREAD_SIZE adr_l x5, init_task msr sp_el0, x5 // Save thread_info diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b546e47543e2..b886332a7c3f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -167,6 +167,15 @@ static unsigned long pgd_pgtable_alloc(unsigned long shift, void *unused) #include "./mmu_include.c" +void create_init_pgd_mapping(pgd_t * pgdir, unsigned int entries_cnt, + phys_addr_t phys, unsigned long virt, phys_addr_t size, + pgprot_t prot, pgtable_alloc allocator, + void * info, int flags) +{ + __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, + prot, allocator, info, flags); +} + int idmap_extend_pgtable; /*