From patchwork Sat Apr 10 09:56:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A815C433ED for ; Sat, 10 Apr 2021 10:01:36 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F773610A2 for ; Sat, 10 Apr 2021 10:01:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F773610A2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ev9ATT0yvdEW2r7xofLE1D+CeuuCHe2opGdox1xOU9o=; b=Lf4jVbjOWmr7m2G3S+OmA/DVu cEgY7jDmleaBbUJZy5+jP587f1SBK5S7Ept33ReRu4tZsvC5e5slFhSaM0K3orJDFvWQv6oCYqBm5 XU/Yxb6GfnaAOTasBTtIDaHzxmnESBvNbJ9EyV+bYtybFqMNA0cQW0ML66GMeTp5Gh1tOtU58pH7Q czN/heGQHd/ynhAc+/7fw0kZ/j9XzEI1vesKGTt3JXZqb0vMBwr3p09k7R0yZxXWd/gRaVB6TcQFC Lbby/KllHbvjbm26JLw/lV7y0dTw8eKFu5Q7nHKx+xOfpkOtQ29Z3Ye2JH7Oumq10pmfVADJ/nSkS i8E6622uQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVAOj-002bre-Py; Sat, 10 Apr 2021 09:59:28 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVANH-002bTl-SN for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=80v3vFe215kri+0o/k30XuUjT5mfA41TLcwr3Ulv96o=; b=LH3JJytVz1595KNRQvegefBrLl wyCRrNSRJ2XmLlzA9ohcMNzuL/jW34Vs+rceMRPZ2EQ/SkJKM3wRMKaJgrA18mZ+NW8nIbRGD38Gg q5CbKnbv2EVQb/X0wOBf0gjCz8AxO7h1YwR6Zw5XJmQlp6Q6XkuO2jXn8Phm5YtDSC5GVKmggIJd9 0dqsRBZbHRnpgpo3F0pQW8lwIy+lXYIqXUWtZz6Bqi92385xJeP+zEstSgAcME3/Bu8OD5+6/hgFf qtdGWDKzPu3G6lyezFI0TYr9yebyRVZBgou1OZQW2olm0lBfQkBT5qXr3HemvjSQdyYfBH7gORo3g vnM9vsvA==; Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVANE-0051ar-PH for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:54 +0000 Received: by mail-pl1-x62a.google.com with SMTP id p10so3946787pld.0 for ; Sat, 10 Apr 2021 02:57:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=80v3vFe215kri+0o/k30XuUjT5mfA41TLcwr3Ulv96o=; b=LhtLlALC+5z1pYJSGS4S6JU+yfgH8LOx4yXrQsibhjvQVoZebw5GASueQdTp4V2sxr M4hcU8JUccWRxbQm9/vt5Nl3O3YlVG6xKt77aHHbUgolitKUam18LQ6tt9T/UYYm/0yF E8XIt69b2ngCOVRgacf88rG4G58IERpb0ak/Z/YTdgK5TcfeRPEfxKOPY/54zZHK3X8I +j3+yPCrtUgUjLqt3zOVu1haMMYWS/KwkqmpRJLaMmoAKAbnG9otTciVhaJncDNd06eB 3d6SZ1QciMBn/aTO718fdCFpyCsoApE3KnYXgz29dHmU7E5gB/GwJxRWF5VQGaXc4rLV z7lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=80v3vFe215kri+0o/k30XuUjT5mfA41TLcwr3Ulv96o=; b=bqaAXjoZyjfrby+uAl39R758NOckacFbArtoS2w3BgdTOd/4zFGk7PbY7a3jMGD8gy ytm0ezDox7BJV7SJUUVvFhWs1FN2SaQOSN+JZ6UAIJzYca2U5GhZ/auQFRK6oh65oSZQ Gisv7Fy8FiD9LJibUVrgaeBFXGNiYLK8RAIuMtagO3MhubpugoThCdIYmoXo9kMWr/H5 hN18MFAg6SIimJk4O6oC6FSaTAMfJqXTyTZUID7NgX1Pmvka9KH+i3BJYPhTzXoO6GGy e8uUmLPOMf6fplSTiXFtWsz0IsOjwhbA7JpD/kFVBKdtZdc3NN++mX7S6iM9ETJTGRCB hrtA== X-Gm-Message-State: AOAM531poRnvSsH5AFyBMODhmnNO8CvKL1sKWuHEDE405X2bfSGyLRgh Uv6EXCyVMxh2bEXy0z7U6fvdH5jhi77d X-Google-Smtp-Source: ABdhPJyny5WDhNZvmOTPl8nyiEGR+/+fQmyqqoY+nfYFowNlKq6fKk9edyRE+kpEO+Y/RJRIUbAg3A== X-Received: by 2002:a17:902:d70f:b029:ea:83ae:2336 with SMTP id w15-20020a170902d70fb02900ea83ae2336mr5813147ply.4.1618048671857; Sat, 10 Apr 2021 02:57:51 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:51 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 8/8] arm64/head: convert idmap_pg_dir and init_pg_dir to __create_pgd_mapping() Date: Sat, 10 Apr 2021 17:56:54 +0800 Message-Id: <20210410095654.24102-9-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025752_850381_9C933C9C X-CRM114-Status: GOOD ( 20.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 5 + arch/arm64/kernel/head.S | 187 +++++++------------------------ arch/arm64/mm/mmu.c | 9 ++ 3 files changed, 55 insertions(+), 146 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index f848a0300228..128d784d78d4 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -8,6 +8,9 @@ #ifndef __ASM_PGALLOC_H #define __ASM_PGALLOC_H +#include + +#ifndef __ASSEMBLY__ #include #include #include @@ -102,6 +105,8 @@ extern void create_idmap(pgd_t *pgdir, phys_addr_t phys, void *info, int flags); +#endif + #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) #define NO_FIXMAP BIT(2) diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index e19649dbbafb..3b0c9359ab70 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -28,6 +28,8 @@ #include #include #include +#include +#include #include #include #include @@ -93,6 +95,8 @@ SYM_CODE_START(primary_entry) adrp x23, __PHYS_OFFSET and x23, x23, MIN_KIMG_ALIGN - 1 // KASLR offset, defaults to 0 bl set_cpu_boot_mode_flag + adrp x4, init_thread_union + add sp, x4, #THREAD_SIZE bl __create_page_tables /* * The following calls CPU setup code, see arch/arm64/mm/proc.S for @@ -121,135 +125,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) b __inval_dcache_area // tail call SYM_CODE_END(preserve_boot_args) -/* - * Macro to create a table entry to the next page. - * - * tbl: page table address - * virt: virtual address - * shift: #imm page table shift - * ptrs: #imm pointers per table page - * - * Preserves: virt - * Corrupts: ptrs, tmp1, tmp2 - * Returns: tbl -> next level table page address - */ - .macro create_table_entry, tbl, virt, shift, ptrs, tmp1, tmp2 - add \tmp1, \tbl, #PAGE_SIZE - phys_to_pte \tmp2, \tmp1 - orr \tmp2, \tmp2, #PMD_TYPE_TABLE // address of next table and entry type - lsr \tmp1, \virt, #\shift - sub \ptrs, \ptrs, #1 - and \tmp1, \tmp1, \ptrs // table index - str \tmp2, [\tbl, \tmp1, lsl #3] - add \tbl, \tbl, #PAGE_SIZE // next level table page - .endm - -/* - * Macro to populate page table entries, these entries can be pointers to the next level - * or last level entries pointing to physical memory. - * - * tbl: page table address - * rtbl: pointer to page table or physical memory - * index: start index to write - * eindex: end index to write - [index, eindex] written to - * flags: flags for pagetable entry to or in - * inc: increment to rtbl between each entry - * tmp1: temporary variable - * - * Preserves: tbl, eindex, flags, inc - * Corrupts: index, tmp1 - * Returns: rtbl - */ - .macro populate_entries, tbl, rtbl, index, eindex, flags, inc, tmp1 -.Lpe\@: phys_to_pte \tmp1, \rtbl - orr \tmp1, \tmp1, \flags // tmp1 = table entry - str \tmp1, [\tbl, \index, lsl #3] - add \rtbl, \rtbl, \inc // rtbl = pa next level - add \index, \index, #1 - cmp \index, \eindex - b.ls .Lpe\@ - .endm - -/* - * Compute indices of table entries from virtual address range. If multiple entries - * were needed in the previous page table level then the next page table level is assumed - * to be composed of multiple pages. (This effectively scales the end index). - * - * vstart: virtual address of start of range - * vend: virtual address of end of range - * shift: shift used to transform virtual address into index - * ptrs: number of entries in page table - * istart: index in table corresponding to vstart - * iend: index in table corresponding to vend - * count: On entry: how many extra entries were required in previous level, scales - * our end index. - * On exit: returns how many extra entries required for next page table level - * - * Preserves: vstart, vend, shift, ptrs - * Returns: istart, iend, count - */ - .macro compute_indices, vstart, vend, shift, ptrs, istart, iend, count - lsr \iend, \vend, \shift - mov \istart, \ptrs - sub \istart, \istart, #1 - and \iend, \iend, \istart // iend = (vend >> shift) & (ptrs - 1) - mov \istart, \ptrs - mul \istart, \istart, \count - add \iend, \iend, \istart // iend += (count - 1) * ptrs - // our entries span multiple tables - - lsr \istart, \vstart, \shift - mov \count, \ptrs - sub \count, \count, #1 - and \istart, \istart, \count - - sub \count, \iend, \istart - .endm - -/* - * Map memory for specified virtual address range. Each level of page table needed supports - * multiple entries. If a level requires n entries the next page table level is assumed to be - * formed from n pages. - * - * tbl: location of page table - * rtbl: address to be used for first level page table entry (typically tbl + PAGE_SIZE) - * vstart: start address to map - * vend: end address to map - we map [vstart, vend] - * flags: flags to use to map last level entries - * phys: physical address corresponding to vstart - physical memory is contiguous - * pgds: the number of pgd entries - * - * Temporaries: istart, iend, tmp, count, sv - these need to be different registers - * Preserves: vstart, vend, flags - * Corrupts: tbl, rtbl, istart, iend, tmp, count, sv - */ - .macro map_memory, tbl, rtbl, vstart, vend, flags, phys, pgds, istart, iend, tmp, count, sv - add \rtbl, \tbl, #PAGE_SIZE - mov \sv, \rtbl - mov \count, #0 - compute_indices \vstart, \vend, #PGDIR_SHIFT, \pgds, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl - -#if SWAPPER_PGTABLE_LEVELS > 3 - compute_indices \vstart, \vend, #PUD_SHIFT, #PTRS_PER_PUD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv - mov \sv, \rtbl -#endif - -#if SWAPPER_PGTABLE_LEVELS > 2 - compute_indices \vstart, \vend, #SWAPPER_TABLE_SHIFT, #PTRS_PER_PMD, \istart, \iend, \count - populate_entries \tbl, \rtbl, \istart, \iend, #PMD_TYPE_TABLE, #PAGE_SIZE, \tmp - mov \tbl, \sv -#endif - - compute_indices \vstart, \vend, #SWAPPER_BLOCK_SHIFT, #PTRS_PER_PTE, \istart, \iend, \count - bic \count, \phys, #SWAPPER_BLOCK_SIZE - 1 - populate_entries \tbl, \count, \istart, \iend, \flags, #SWAPPER_BLOCK_SIZE, \tmp - .endm - /* * Setup the initial page tables. We only setup the barest amount which is * required to get the kernel running. The following sections are required: @@ -344,9 +219,6 @@ SYM_FUNC_START_LOCAL(__create_page_tables) adr_l x4, idmap_extend_pgtable mov x5, #1 str x5, [x4] //require expanded pagetable - - mov x4, EXTRA_PTRS - create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 #else /* * If VA_BITS == 48, we don't have to configure an additional @@ -356,25 +228,50 @@ SYM_FUNC_START_LOCAL(__create_page_tables) str_l x4, idmap_ptrs_per_pgd, x5 #endif 1: - ldr_l x4, idmap_ptrs_per_pgd - mov x5, x3 // __pa(__idmap_text_start) - adr_l x6, __idmap_text_end // __pa(__idmap_text_end) + stp x0, x1, [sp, #-64]! + stp x2, x3, [sp, #48] + stp x4, x5, [sp, #32] + stp x6, x7, [sp, #16] - map_memory x0, x1, x3, x6, x7, x3, x4, x10, x11, x12, x13, x14 + adrp x0, idmap_pg_dir + adrp x1, idmap_pg_end + sub x1, x1, x0 + bl set_cur_mempool + + adrp x1, __idmap_text_start + adr_l x2, __idmap_text_end + sub x2, x2, x1 + ldr x3, =PAGE_KERNEL_EXEC + adr_l x4, head_pgtable_alloc + mov x5, #0 + mov x6, #NO_FIXMAP + bl create_idmap /* * Map the kernel image (starting with PHYS_OFFSET). */ adrp x0, init_pg_dir - mov_q x5, KIMAGE_VADDR // compile time __va(_text) - add x5, x5, x23 // add KASLR displacement - mov x4, PTRS_PER_PGD - adrp x6, _end // runtime __pa(_end) - adrp x3, _text // runtime __pa(_text) - sub x6, x6, x3 // _end - _text - add x6, x6, x5 // runtime __va(_end) + adrp x1, init_pg_end + sub x1, x1, x0 + bl set_cur_mempool - map_memory x0, x1, x5, x6, x7, x3, x4, x10, x11, x12, x13, x14 + mov x1, PTRS_PER_PGD + adrp x3, _text // runtime __pa(_text) + mov_q x4, KIMAGE_VADDR // compile time __va(_text) + add x4, x4, x23 // add KASLR displacement + adrp x5, _end // runtime __pa(_end) + sub x5, x5, x3 // _end - _text + + ldr x3, =PAGE_KERNEL_EXEC + adr_l x4, head_pgtable_alloc + mov x5, #0 + mov x6, #NO_FIXMAP + + bl create_init_pgd_mapping + ldp x6, x7, [sp, #16] + ldp x4, x5, [sp, #32] + ldp x2, x3, [sp, #48] + ldp x0, x1, [sp], #-64 /* * Since the page tables have been populated with non-cacheable @@ -402,8 +299,6 @@ SYM_FUNC_END(__create_page_tables) * x0 = __PHYS_OFFSET */ SYM_FUNC_START_LOCAL(__primary_switched) - adrp x4, init_thread_union - add sp, x4, #THREAD_SIZE adr_l x5, init_task msr sp_el0, x5 // Save thread_info diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b546e47543e2..b886332a7c3f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -167,6 +167,15 @@ static unsigned long pgd_pgtable_alloc(unsigned long shift, void *unused) #include "./mmu_include.c" +void create_init_pgd_mapping(pgd_t * pgdir, unsigned int entries_cnt, + phys_addr_t phys, unsigned long virt, phys_addr_t size, + pgprot_t prot, pgtable_alloc allocator, + void * info, int flags) +{ + __create_pgd_mapping(pgdir, entries_cnt, phys, virt, size, + prot, allocator, info, flags); +} + int idmap_extend_pgtable; /*