From patchwork Sat Apr 10 09:56:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12195549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 871D3C433B4 for ; Sat, 10 Apr 2021 09:59:48 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DA74961001 for ; Sat, 10 Apr 2021 09:59:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA74961001 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=VyWyj6I6p/sI0N7u2oBTQTFkwTAlvOYZBeoqFZMKDYc=; b=cYgnSi+4dPSi5bDZRpU72PWkz nmmETBJt4y8DmDXD3YjWQL0cZ1J2aNqcdeF9DE4GOeDnpcMRvMCzbPOgqtNOWriWGRpyCfjxZKLp7 pMPDlJbZuQuo82dJrRlsI9RwKFmuK/ErOx6vxzPEK17alTVyihMGPXcS6Wj3QXM6q/bPq1Yx6EnRX 8Wx614dWvViwTYKap/A/yDRZp3HKjRBM+vPIsT7X3tuUvXrTCjVFG3naHFDJJh0L7LnKoFSNiIyLu vbAn5OdVomuCdviUQ+Silv9N24X4zXcYEn1wngIrYU3rsub5VZBkV3i/XIsjJQFM7Tl9+bNQZjZcq POLUsBfXQ==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lVANC-002bTF-Ho; Sat, 10 Apr 2021 09:57:50 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMm-002bP2-RF for linux-arm-kernel@desiato.infradead.org; Sat, 10 Apr 2021 09:57:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=kfZ3yY7rkniJE2QSo9AxiDLn15/ogNXSmhdqvQvKk5Q=; b=gcap4erg8NghVURxXNsOdkO1x4 xsIEd14oNapMqDfHwHEV28PP/POpdnlRp7G3W6JlC3AdeqGnO43PPN56RW4UCsWD22WPiswlvzPrT t1Tjb0tmkIOiSl3jByDDoJMoXDwiwqwFauNQEzxn5Gyz/3K99JC/r8inPrGGTs/oEBNZK9rUa1dzO J44GVpJVj/BOokkS1KxAnDuvEYCYnuglADZJ+8Lt1BivoAmjEJa3Qohz2VlOb7qA7Pt38EM+8v0o3 cXH3URTEzVX4KxoddI8XVuncfnZ8FPA4OFTtC6+ijxtMbqhffLPIVPtrBlbFCpewuNtk+87N11xAG NKQFvX+A==; Received: from mail-pf1-x42e.google.com ([2607:f8b0:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lVAMj-0051Yv-34 for linux-arm-kernel@lists.infradead.org; Sat, 10 Apr 2021 09:57:22 +0000 Received: by mail-pf1-x42e.google.com with SMTP id d124so5836620pfa.13 for ; Sat, 10 Apr 2021 02:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kfZ3yY7rkniJE2QSo9AxiDLn15/ogNXSmhdqvQvKk5Q=; b=ck8ji4ZyXKDKgfydWa1JEzX7EmHxu8i/cG/+tiZJsJqUoJMhWaoqj9J0DCH0ankYCi UQjcQJ8v2Kxpuo5mcnLQohGCp1a7QVdakBMDv3rR/HJg0tQaRHHAnE6DCPt3rsbwVe6X D/Ej/eqEs8N1NQ72gfQ+9kyq/JIhFNrsQMkavaBnf8PVej5MrTOSzzNbkMa9E9Fxb1ju CMt7QT2FyKsKrVon7fapNQZoB5IE9E0Lp0TEuUt0uQ3NIJygR/NsE3u1jz6nF6Wq+U57 EVLUtvxabJKjEYvAhsHrYl1tOE9ARCKTPCDVqGHhL/sWI81mK40Lug1rw3Vf1aRqV8WI +MLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kfZ3yY7rkniJE2QSo9AxiDLn15/ogNXSmhdqvQvKk5Q=; b=E9I2xLJa/ucbZDOn0oBoOPGno4xvMrZivdA+sX3medJ7okBUVVA52deKGAYbaEzCwz dCpwefYyvcmiw3iutsG5HcxbcSjFh0kJdpV0qGOcmQ/I4lHsXFcWovJv5H32igR8nvjx IZDygfTcQoTgRYMD5/AnYiApPWfNdgA9dNupqO6tTZCEddW9/Fg6/u8YHEaTECpKOpmc ZO2Lc9FLd2Mw4bIwV2fufKpF8zfj94bHNPhli7oe7ek1DjsrgxfN6NEH1/8WQEQLcqo7 bt/xw4++HiZWC4kL2+xZzJkiEjrT09YQrVZ1bucA5TJhZTduMoIhTXtzEQbHIVyY6Bpm m5hQ== X-Gm-Message-State: AOAM532Fczw/TkgpBFW/qBdkjUBowBQZzwcD+cxI71BVA7WeZq2qJghA cR2/N3vna76M5mDkqmtIqDk/MYAHqDgp X-Google-Smtp-Source: ABdhPJzYw+cKPHhspjN7j05J4DyH6i0aZIomTCc2SGK69+HrOqGiDmVSk+ZU+OAwaLNNjzoI812aqg== X-Received: by 2002:a62:2cce:0:b029:21d:97da:833e with SMTP id s197-20020a622cce0000b029021d97da833emr15532224pfs.40.1618048640155; Sat, 10 Apr 2021 02:57:20 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id m14sm3836322pfh.89.2021.04.10.02.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 02:57:19 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Marc Zyngier , Kristina Martsenko , James Morse , Steven Price , Jonathan Cameron , Pavel Tatashin , Anshuman Khandual , Atish Patra , Mike Rapoport , Logan Gunthorpe , Mark Brown Subject: [RFC 2/8] arm64/mm: change __create_pgd_mapping() prototype to accept nr_entries and introduce create_idmap() Date: Sat, 10 Apr 2021 17:56:48 +0800 Message-Id: <20210410095654.24102-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210410095654.24102-1-kernelfans@gmail.com> References: <20210410095654.24102-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210410_025721_154817_1B3400E4 X-CRM114-Status: GOOD ( 19.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As idmap may have pgd entries more than PTRS_PER_PGD, change the prototype of __create_pgd_mapping() to fix that. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Marc Zyngier Cc: Kristina Martsenko Cc: James Morse Cc: Steven Price Cc: Jonathan Cameron Cc: Pavel Tatashin Cc: Anshuman Khandual Cc: Atish Patra Cc: Mike Rapoport Cc: Logan Gunthorpe Cc: Mark Brown To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/pgalloc.h | 7 ++++++ arch/arm64/kernel/head.S | 3 +++ arch/arm64/mm/mmu.c | 41 ++++++++++++++++++++++++++------ arch/arm64/mm/mmu_include.c | 10 ++++++-- 4 files changed, 52 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 3c6a7f5988b1..555792921af0 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -83,4 +83,11 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgtable_t ptep) } #define pmd_pgtable(pmd) pmd_page(pmd) +extern void __create_pgd_mapping_extend(pgd_t *pgdir, + unsigned int entries_cnt, phys_addr_t phys, + unsigned long virt, phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags); + #endif diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 840bda1869e9..e19649dbbafb 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -341,6 +341,9 @@ SYM_FUNC_START_LOCAL(__create_page_tables) #if VA_BITS != EXTRA_SHIFT #error "Mismatch between VA_BITS and page size/number of translation levels" #endif + adr_l x4, idmap_extend_pgtable + mov x5, #1 + str x5, [x4] //require expanded pagetable mov x4, EXTRA_PTRS create_table_entry x0, x3, EXTRA_SHIFT, x4, x5, x6 diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 56e4f25e8d6d..30afd6ed275f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -145,6 +145,33 @@ static phys_addr_t pgd_pgtable_alloc(int shift) #include "./mmu_include.c" +int idmap_extend_pgtable; + +/* + * todo: tear down part of idmap + * todo: lock protection for concurrent population + */ +void create_idmap(pgd_t *pgdir, phys_addr_t phys, + phys_addr_t size, + pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int), + int flags) +{ + u64 ptrs_per_pgd = idmap_ptrs_per_pgd; + +#if CONFIG_IDMAP_PGTABLE_EXPAND + if (idmap_extend_pgtable) + __create_pgd_mapping_extend(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); + else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#else + __create_pgd_mapping(pgdir, ptrs_per_pgd, + phys, phys, size, prot, pgtable_alloc, flags); +#endif +} + /* * This function can only be used to modify existing table entries, * without allocating new levels of table. Note that this permits the @@ -158,7 +185,7 @@ static void __init create_mapping_noalloc(phys_addr_t phys, unsigned long virt, &phys, virt); return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); } @@ -173,7 +200,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, if (page_mappings_only) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(mm->pgd, phys, virt, size, prot, + __create_pgd_mapping(mm->pgd, PTRS_PER_PGD, phys, virt, size, prot, pgd_pgtable_alloc, flags); } @@ -186,7 +213,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, return; } - __create_pgd_mapping(init_mm.pgd, phys, virt, size, prot, NULL, + __create_pgd_mapping(init_mm.pgd, PTRS_PER_PGD, phys, virt, size, prot, NULL, NO_CONT_MAPPINGS); /* flush the TLBs after updating live kernel mappings */ @@ -196,7 +223,7 @@ static void update_mapping_prot(phys_addr_t phys, unsigned long virt, static void __init __map_memblock(pgd_t *pgdp, phys_addr_t start, phys_addr_t end, pgprot_t prot, int flags) { - __create_pgd_mapping(pgdp, start, __phys_to_virt(start), end - start, + __create_pgd_mapping(pgdp, start, PTRS_PER_PGD, __phys_to_virt(start), end - start, prot, early_pgtable_alloc, flags); } @@ -297,7 +324,7 @@ static void __init map_kernel_segment(pgd_t *pgdp, void *va_start, void *va_end, BUG_ON(!PAGE_ALIGNED(pa_start)); BUG_ON(!PAGE_ALIGNED(size)); - __create_pgd_mapping(pgdp, pa_start, (unsigned long)va_start, size, prot, + __create_pgd_mapping(pgdp, PTRS_PER_PGD, pa_start, (unsigned long)va_start, size, prot, early_pgtable_alloc, flags); if (!(vm_flags & VM_NO_GUARD)) @@ -341,7 +368,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); - __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, + __create_pgd_mapping(tramp_pg_dir, PTRS_PER_PGD, pa_start, TRAMP_VALIAS, PAGE_SIZE, prot, __pgd_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ @@ -1233,7 +1260,7 @@ int arch_add_memory(int nid, u64 start, u64 size, IS_ENABLED(CONFIG_KFENCE)) flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; - __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), + __create_pgd_mapping(swapper_pg_dir, PTRS_PER_PGD, start, __phys_to_virt(start), size, params->pgprot, __pgd_pgtable_alloc, flags); diff --git a/arch/arm64/mm/mmu_include.c b/arch/arm64/mm/mmu_include.c index e9ebdffe860b..1cf5af7e2aeb 100644 --- a/arch/arm64/mm/mmu_include.c +++ b/arch/arm64/mm/mmu_include.c @@ -233,14 +233,20 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, pud_clear_fixmap(); } -static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, +static void __create_pgd_mapping(pgd_t *pgdir, unsigned int entries_cnt, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, end, next; - pgd_t *pgdp = pgd_offset_pgd(pgdir, virt); + pgd_t *pgdp; + + if (likely(entries_cnt == PTRS_PER_PGD)) + pgdp = pgd_offset_pgd(pgdir, virt); + else { + pgdp = pgdir + ((virt >> PGDIR_SHIFT) & (entries_cnt - 1)); + } /* * If the virtual and physical address don't have the same offset