From patchwork Sun Apr 25 14:12:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12223323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F089EC43460 for ; Sun, 25 Apr 2021 14:15:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4F55B61369 for ; Sun, 25 Apr 2021 14:15:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4F55B61369 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=GvB5m35kEXsgveAtV/LCKDoDTK0IQ7QByh7LQIdrjvY=; b=qMWeYGPhipWZyKrVFQ6UKB9+3 7oQ9AZt3WqAKtOyJqJ4gnU8EXr4MJNitN6XJd5ZYvHjmK5jDYq/bI7ZBgOycNuCuM3zzWSjQ8HSlz BI/PRZxJ6tbIJwQ1kpxlPYLGpH9FstxH8Hz2GCnVaprc6UZlOV0ORib51yxlT4N58rd7WOr5IYRNO xawMhAxvmhdbjjnzRQjB2JzCNB+xHihCw+OVQU9yoOG8zx09gsqtzsA5omokzWvZGHNuMYwrB8KxE 7hnIPDlPEpUdCydvfOF17rLO04F3U5uvDX6/uT0YiEp2QQy6KdVz1NDFg5tSA+lw7A2TnMpIptU0J NWYBG08vQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lafW2-005iFq-Cd; Sun, 25 Apr 2021 14:13:42 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafVz-005iFf-AD for linux-arm-kernel@desiato.infradead.org; Sun, 25 Apr 2021 14:13:40 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=KW55YbQnlz7SIqfPLToGbEidmfkJ1g+UVynIfCIMsRE=; b=mLV15XgjswALmkS4TMyxUq/WuZ AzamfVyTw2DLCQTHNlO4mOrsqJuSggEy6FogophL7uhnhjfkYQvytbU3sktbVdjhv5L9CXa0K1njc Hgw2sEiwjMUW+4VvEbECZgDI9TvSNCkNS2+Jgr1dU09SRWEs/caAu0aspiVDkXWBqREt6ZQJvKpMJ La09S+ObZTsSdO/UvUZqkuDo5Mo4ES4B46XYeXJ5h2qOpN2Y1R0PvvHfqXZptFsgVEBtsWkNKPrsl nGaDwF1L4AZF6a6UiC/+7l+QU+92BnPRUwoREVNgj1G5UBE5ICfJOEpYEE7YX+bRi/LRxDJsA2oYC RJb0QYLQ==; Received: from mail-pj1-x1035.google.com ([2607:f8b0:4864:20::1035]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lafVv-00FT3x-K1 for linux-arm-kernel@lists.infradead.org; Sun, 25 Apr 2021 14:13:38 +0000 Received: by mail-pj1-x1035.google.com with SMTP id y22-20020a17090a8b16b0290150ae1a6d2bso3808897pjn.0 for ; Sun, 25 Apr 2021 07:13:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KW55YbQnlz7SIqfPLToGbEidmfkJ1g+UVynIfCIMsRE=; b=J/I88VtUm4TabcGsDPLY+FoRRZxAcj6Fz4AFVQdPVM5KydnQdB5oEf13HPojCWkm5w jpBhVkYG2IUpYEaJmvl7TFU7IY6qG4mOqZ3XjTv/fcyckV7SQcWKQIYHvtpIiXJmOxv0 g2UdNGw29770Tod6y8WNeD5KrTq6MermXcBsHJE0+WhT5CSWyDTPA8kCkMP03a3zdOm7 7E1t+iSHkLtq0HNiUBnNhnX5Bjr7MBOsY35xbpHvW4Sd/xCg9H0PCqaVxwOD3P8RJb7x JMBQCQrjxl1pSM6BRPkIxYOEDkd4BuwK1pTb0yZoZjwQv/IvE8CT/mLD9VMyrLT+C9ss DpTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KW55YbQnlz7SIqfPLToGbEidmfkJ1g+UVynIfCIMsRE=; b=hI5t+kNrWYik07T6qK+KkziVuKxlHoZNITI4zCT2d4/wyXECrgljFf3re+CL57VlkN 7237JzD46gU+HBXGvtAXij5DVGbo6WyhD5h5rtyKCeDlxqaUlnQZaCkhK+ilpYm2mkxk rry+keRD5IloMB27qfK6TZsKvr4gBKPGVB9a99HTZG065DyTa4jFss9dGdRGWOfG77YF mosggh/zX8AoHhdFWWeZVn8dbDdA2AsDQBNQK8+lzVg+IRU0qwdBoy/K9l1iuTTHk51x KkKQ8OQLGOVHMxOwotEySmw1dCYYegBCmccI3rRV1d/WlIVFO6221zqHRmQvlN5UAxtw u8jA== X-Gm-Message-State: AOAM532XxppVo4AlIVnx1VVQxBckjCOHOZ55eTRwZY67gT6nmV1TkvNn gbKijHAW0yQxd9b08X3l6dgNU4+qww== X-Google-Smtp-Source: ABdhPJxJPERun+Wu2dMUnxngmCzDCmfk83f7A+WB0VlC1b4ZrBvr2pAD5NF0bjzhpOn+HaqkJiNYzQ== X-Received: by 2002:a17:90a:f298:: with SMTP id fs24mr15768349pjb.129.1619360013457; Sun, 25 Apr 2021 07:13:33 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id r32sm1007139pgm.49.2021.04.25.07.13.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Apr 2021 07:13:26 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [PATCHv2 01/10] arm64/mm: split out __create_pgd_mapping() routines Date: Sun, 25 Apr 2021 22:12:55 +0800 Message-Id: <20210425141304.32721-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210425141304.32721-1-kernelfans@gmail.com> References: <20210425141304.32721-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210425_071335_693705_33D892A1 X-CRM114-Status: GOOD ( 30.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If VA_BITS < 48, an additional table level should be used to allow an ID mapping to cover system RAM. At present, it is done by macro create_table_entry in head.S. But with second thoughts, this expansion can be easily achieved by redefinition of CONFIG_PGTABLE_LEVELS. Split out the routines for __create_pgd_mapping(), so that two sets of code will be generated to manipulate pgtable under two different CONFIG_PGTABLE_LEVELS. Later the one generated under 'CONFIG_PGTABLE_LEVELS + 1' can be used for idmap. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- RFC -> v2: consider the config ARM64_16K_PAGES && ARM64_VA_BITS_36 and move redefinition of CONFIG_PGTABLE_LEVELS to the head of idmap_mmu.c --- arch/arm64/Kconfig | 4 + arch/arm64/mm/Makefile | 2 + arch/arm64/mm/idmap_mmu.c | 34 +++++ arch/arm64/mm/mmu.c | 263 +---------------------------------- arch/arm64/mm/mmu_include.c | 270 ++++++++++++++++++++++++++++++++++++ 5 files changed, 312 insertions(+), 261 deletions(-) create mode 100644 arch/arm64/mm/idmap_mmu.c create mode 100644 arch/arm64/mm/mmu_include.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index e4e1b6550115..79755ade5d27 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -327,6 +327,10 @@ config PGTABLE_LEVELS default 3 if ARM64_16K_PAGES && ARM64_VA_BITS_47 default 4 if !ARM64_64K_PAGES && ARM64_VA_BITS_48 +config IDMAP_PGTABLE_EXPAND + def_bool y + depends on (ARM64_16K_PAGES && ARM64_VA_BITS_36) || (ARM64_64K_PAGES && ARM64_VA_BITS_42) || (ARM64_4K_PAGES && ARM64_VA_BITS_39) || (ARM64_16K_PAGES && ARM64_VA_BITS_47) + config ARCH_SUPPORTS_UPROBES def_bool y diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index f188c9092696..f9283cb9a201 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,8 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o + +obj-$(CONFIG_IDMAP_PGTABLE_EXPAND) += idmap_mmu.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/idmap_mmu.c b/arch/arm64/mm/idmap_mmu.c new file mode 100644 index 000000000000..42a27dd5cc9f --- /dev/null +++ b/arch/arm64/mm/idmap_mmu.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#ifdef CONFIG_IDMAP_PGTABLE_EXPAND + +#if CONFIG_PGTABLE_LEVELS == 2 +#define EXTEND_LEVEL 3 +#elif CONFIG_PGTABLE_LEVELS == 3 +#define EXTEND_LEVEL 4 +#endif + +#undef CONFIG_PGTABLE_LEVELS +#define CONFIG_PGTABLE_LEVELS EXTEND_LEVEL + +#include + +#include +#include +#include +#include +#include + +#include "./mmu_include.c" + +void __create_pgd_mapping_extend(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + __create_pgd_mapping(pgdir, phys, virt, size, prot, pgtable_alloc, flags); +} +#endif + + diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 5d9550fdb9cf..56e4f25e8d6d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -37,9 +37,6 @@ #include #include -#define NO_BLOCK_MAPPINGS BIT(0) -#define NO_CONT_MAPPINGS BIT(1) - u64 idmap_t0sz = TCR_T0SZ(VA_BITS_MIN); u64 idmap_ptrs_per_pgd = PTRS_PER_PGD; @@ -116,264 +113,6 @@ static phys_addr_t __init early_pgtable_alloc(int shift) return phys; } -static bool pgattr_change_is_safe(u64 old, u64 new) -{ - /* - * The following mapping attributes may be updated in live - * kernel mappings without the need for break-before-make. - */ - pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; - - /* creating or taking down mappings is always safe */ - if (old == 0 || new == 0) - return true; - - /* live contiguous mappings may not be manipulated at all */ - if ((old | new) & PTE_CONT) - return false; - - /* Transitioning from Non-Global to Global is unsafe */ - if (old & ~new & PTE_NG) - return false; - - /* - * Changing the memory type between Normal and Normal-Tagged is safe - * since Tagged is considered a permission attribute from the - * mismatched attribute aliases perspective. - */ - if (((old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || - (old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED)) && - ((new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || - (new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED))) - mask |= PTE_ATTRINDX_MASK; - - return ((old ^ new) & ~mask) == 0; -} - -static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot) -{ - pte_t *ptep; - - ptep = pte_set_fixmap_offset(pmdp, addr); - do { - pte_t old_pte = READ_ONCE(*ptep); - - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); - - /* - * After the PTE entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); - - phys += PAGE_SIZE; - } while (ptep++, addr += PAGE_SIZE, addr != end); - - pte_clear_fixmap(); -} - -static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long next; - pmd_t pmd = READ_ONCE(*pmdp); - - BUG_ON(pmd_sect(pmd)); - if (pmd_none(pmd)) { - phys_addr_t pte_phys; - BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(PAGE_SHIFT); - __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); - pmd = READ_ONCE(*pmdp); - } - BUG_ON(pmd_bad(pmd)); - - do { - pgprot_t __prot = prot; - - next = pte_cont_addr_end(addr, end); - - /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && - (flags & NO_CONT_MAPPINGS) == 0) - __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - - init_pte(pmdp, addr, next, phys, __prot); - - phys += next - addr; - } while (addr = next, addr != end); -} - -static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) -{ - unsigned long next; - pmd_t *pmdp; - - pmdp = pmd_set_fixmap_offset(pudp, addr); - do { - pmd_t old_pmd = READ_ONCE(*pmdp); - - next = pmd_addr_end(addr, end); - - /* try section mapping first */ - if (((addr | next | phys) & ~SECTION_MASK) == 0 && - (flags & NO_BLOCK_MAPPINGS) == 0) { - pmd_set_huge(pmdp, phys, prot); - - /* - * After the PMD entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), - READ_ONCE(pmd_val(*pmdp)))); - } else { - alloc_init_cont_pte(pmdp, addr, next, phys, prot, - pgtable_alloc, flags); - - BUG_ON(pmd_val(old_pmd) != 0 && - pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); - } - phys += next - addr; - } while (pmdp++, addr = next, addr != end); - - pmd_clear_fixmap(); -} - -static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, - unsigned long end, phys_addr_t phys, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), int flags) -{ - unsigned long next; - pud_t pud = READ_ONCE(*pudp); - - /* - * Check for initial section mappings in the pgd/pud. - */ - BUG_ON(pud_sect(pud)); - if (pud_none(pud)) { - phys_addr_t pmd_phys; - BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(PMD_SHIFT); - __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); - pud = READ_ONCE(*pudp); - } - BUG_ON(pud_bad(pud)); - - do { - pgprot_t __prot = prot; - - next = pmd_cont_addr_end(addr, end); - - /* use a contiguous mapping if the range is suitably aligned */ - if ((((addr | next | phys) & ~CONT_PMD_MASK) == 0) && - (flags & NO_CONT_MAPPINGS) == 0) - __prot = __pgprot(pgprot_val(prot) | PTE_CONT); - - init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); - - phys += next - addr; - } while (addr = next, addr != end); -} - -static inline bool use_1G_block(unsigned long addr, unsigned long next, - unsigned long phys) -{ - if (PAGE_SHIFT != 12) - return false; - - if (((addr | next | phys) & ~PUD_MASK) != 0) - return false; - - return true; -} - -static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, - phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long next; - pud_t *pudp; - p4d_t *p4dp = p4d_offset(pgdp, addr); - p4d_t p4d = READ_ONCE(*p4dp); - - if (p4d_none(p4d)) { - phys_addr_t pud_phys; - BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(PUD_SHIFT); - __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); - p4d = READ_ONCE(*p4dp); - } - BUG_ON(p4d_bad(p4d)); - - pudp = pud_set_fixmap_offset(p4dp, addr); - do { - pud_t old_pud = READ_ONCE(*pudp); - - next = pud_addr_end(addr, end); - - /* - * For 4K granule only, attempt to put down a 1GB block - */ - if (use_1G_block(addr, next, phys) && - (flags & NO_BLOCK_MAPPINGS) == 0) { - pud_set_huge(pudp, phys, prot); - - /* - * After the PUD entry has been populated once, we - * only allow updates to the permission attributes. - */ - BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), - READ_ONCE(pud_val(*pudp)))); - } else { - alloc_init_cont_pmd(pudp, addr, next, phys, prot, - pgtable_alloc, flags); - - BUG_ON(pud_val(old_pud) != 0 && - pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); - } - phys += next - addr; - } while (pudp++, addr = next, addr != end); - - pud_clear_fixmap(); -} - -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, - unsigned long virt, phys_addr_t size, - pgprot_t prot, - phys_addr_t (*pgtable_alloc)(int), - int flags) -{ - unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); - - /* - * If the virtual and physical address don't have the same offset - * within a page, we cannot map the region as the caller expects. - */ - if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) - return; - - phys &= PAGE_MASK; - addr = virt & PAGE_MASK; - end = PAGE_ALIGN(virt + size); - - do { - next = pgd_addr_end(addr, end); - alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, - flags); - phys += next - addr; - } while (pgdp++, addr = next, addr != end); -} - static phys_addr_t __pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(GFP_PGTABLE_KERNEL); @@ -404,6 +143,8 @@ static phys_addr_t pgd_pgtable_alloc(int shift) return pa; } +#include "./mmu_include.c" + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c new file mode 100644 index 000000000000..95ff35a3c6cb --- /dev/null +++ b/arch/arm64/mm/mmu_include.c @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * This file is shared + * And these functions should be PIC because they are called under both MMU-disable + * and MMU-enable + */ + +#define NO_BLOCK_MAPPINGS BIT(0) +#define NO_CONT_MAPPINGS BIT(1) + +static bool pgattr_change_is_safe(u64 old, u64 new) +{ + /* + * The following mapping attributes may be updated in live + * kernel mappings without the need for break-before-make. + */ + pteval_t mask = PTE_PXN | PTE_RDONLY | PTE_WRITE | PTE_NG; + + /* creating or taking down mappings is always safe */ + if (old == 0 || new == 0) + return true; + + /* live contiguous mappings may not be manipulated at all */ + if ((old | new) & PTE_CONT) + return false; + + /* Transitioning from Non-Global to Global is unsafe */ + if (old & ~new & PTE_NG) + return false; + + /* + * Changing the memory type between Normal and Normal-Tagged is safe + * since Tagged is considered a permission attribute from the + * mismatched attribute aliases perspective. + */ + if (((old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || + (old & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED)) && + ((new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL) || + (new & PTE_ATTRINDX_MASK) == PTE_ATTRINDX(MT_NORMAL_TAGGED))) + mask |= PTE_ATTRINDX_MASK; + + return ((old ^ new) & ~mask) == 0; +} + +static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot) +{ + pte_t *ptep; + + ptep = pte_set_fixmap_offset(pmdp, addr); + do { + pte_t old_pte = READ_ONCE(*ptep); + + set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + + /* + * After the PTE entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), + READ_ONCE(pte_val(*ptep)))); + + phys += PAGE_SIZE; + } while (ptep++, addr += PAGE_SIZE, addr != end); + + pte_clear_fixmap(); +} + +static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pmd_t pmd = READ_ONCE(*pmdp); + + BUG_ON(pmd_sect(pmd)); + if (pmd_none(pmd)) { + phys_addr_t pte_phys; + + BUG_ON(!pgtable_alloc); + pte_phys = pgtable_alloc(PAGE_SHIFT); + __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); + pmd = READ_ONCE(*pmdp); + } + BUG_ON(pmd_bad(pmd)); + + do { + pgprot_t __prot = prot; + + next = pte_cont_addr_end(addr, end); + + /* use a contiguous mapping if the range is suitably aligned */ + if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && + (flags & NO_CONT_MAPPINGS) == 0) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + init_pte(pmdp, addr, next, phys, __prot); + + phys += next - addr; + } while (addr = next, addr != end); +} + +static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) +{ + unsigned long next; + pmd_t *pmdp; + + pmdp = pmd_set_fixmap_offset(pudp, addr); + do { + pmd_t old_pmd = READ_ONCE(*pmdp); + + next = pmd_addr_end(addr, end); + + /* try section mapping first */ + if (((addr | next | phys) & ~SECTION_MASK) == 0 && + (flags & NO_BLOCK_MAPPINGS) == 0) { + pmd_set_huge(pmdp, phys, prot); + + /* + * After the PMD entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd), + READ_ONCE(pmd_val(*pmdp)))); + } else { + alloc_init_cont_pte(pmdp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(pmd_val(old_pmd) != 0 && + pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp))); + } + phys += next - addr; + } while (pmdp++, addr = next, addr != end); + + pmd_clear_fixmap(); +} + +static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, + unsigned long end, phys_addr_t phys, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), int flags) +{ + unsigned long next; + pud_t pud = READ_ONCE(*pudp); + + /* + * Check for initial section mappings in the pgd/pud. + */ + BUG_ON(pud_sect(pud)); + if (pud_none(pud)) { + phys_addr_t pmd_phys; + + BUG_ON(!pgtable_alloc); + pmd_phys = pgtable_alloc(PMD_SHIFT); + __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); + pud = READ_ONCE(*pudp); + } + BUG_ON(pud_bad(pud)); + + do { + pgprot_t __prot = prot; + + next = pmd_cont_addr_end(addr, end); + + /* use a contiguous mapping if the range is suitably aligned */ + if ((((addr | next | phys) & ~CONT_PMD_MASK) == 0) && + (flags & NO_CONT_MAPPINGS) == 0) + __prot = __pgprot(pgprot_val(prot) | PTE_CONT); + + init_pmd(pudp, addr, next, phys, __prot, pgtable_alloc, flags); + + phys += next - addr; + } while (addr = next, addr != end); +} + +static inline bool use_1G_block(unsigned long addr, unsigned long next, + unsigned long phys) +{ + if (PAGE_SHIFT != 12) + return false; + + if (((addr | next | phys) & ~PUD_MASK) != 0) + return false; + + return true; +} + +static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, + phys_addr_t phys, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long next; + pud_t *pudp; + p4d_t *p4dp = p4d_offset(pgdp, addr); + p4d_t p4d = READ_ONCE(*p4dp); + + if (p4d_none(p4d)) { + phys_addr_t pud_phys; + + BUG_ON(!pgtable_alloc); + pud_phys = pgtable_alloc(PUD_SHIFT); + __p4d_populate(p4dp, pud_phys, PUD_TYPE_TABLE); + p4d = READ_ONCE(*p4dp); + } + BUG_ON(p4d_bad(p4d)); + + pudp = pud_set_fixmap_offset(p4dp, addr); + do { + pud_t old_pud = READ_ONCE(*pudp); + + next = pud_addr_end(addr, end); + + /* + * For 4K granule only, attempt to put down a 1GB block + */ + if (use_1G_block(addr, next, phys) && + (flags & NO_BLOCK_MAPPINGS) == 0) { + pud_set_huge(pudp, phys, prot); + + /* + * After the PUD entry has been populated once, we + * only allow updates to the permission attributes. + */ + BUG_ON(!pgattr_change_is_safe(pud_val(old_pud), + READ_ONCE(pud_val(*pudp)))); + } else { + alloc_init_cont_pmd(pudp, addr, next, phys, prot, + pgtable_alloc, flags); + + BUG_ON(pud_val(old_pud) != 0 && + pud_val(old_pud) != READ_ONCE(pud_val(*pudp))); + } + phys += next - addr; + } while (pudp++, addr = next, addr != end); + + pud_clear_fixmap(); +} + +static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + unsigned long addr, end, next; + pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + + /* + * If the virtual and physical address don't have the same offset + * within a page, we cannot map the region as the caller expects. + */ + if (WARN_ON((phys ^ virt) & ~PAGE_MASK)) + return; + + phys &= PAGE_MASK; + addr = virt & PAGE_MASK; + end = PAGE_ALIGN(virt + size); + + do { + next = pgd_addr_end(addr, end); + alloc_init_pud(pgdp, addr, next, phys, prot, pgtable_alloc, + flags); + phys += next - addr; + } while (pgdp++, addr = next, addr != end); +}