From patchwork Wed Dec 5 16:41:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Capper X-Patchwork-Id: 10714603 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B7A1613BB for ; Wed, 5 Dec 2018 16:42:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A74192DDC6 for ; Wed, 5 Dec 2018 16:42:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 915982DE49; Wed, 5 Dec 2018 16:42:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD6252DDC6 for ; Wed, 5 Dec 2018 16:42:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 251C96B754E; Wed, 5 Dec 2018 11:42:09 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1DBBF6B754F; Wed, 5 Dec 2018 11:42:09 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0033C6B7550; Wed, 5 Dec 2018 11:42:08 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by kanga.kvack.org (Postfix) with ESMTP id CE1216B754E for ; Wed, 5 Dec 2018 11:42:08 -0500 (EST) Received: by mail-oi1-f200.google.com with SMTP id 73so12499504oii.12 for ; Wed, 05 Dec 2018 08:42:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=lFR32fSr1ly6v5yn7Uw54cPiu8J3gkqBA0EbNA51000=; b=UY8Iqpth+Fm9j3p03sb6Oz43SkruxwtQIASHjx2kytnQvjwDqBFg7FDhF2NfVYmMFn UHxpdO4GuKsRV2G/mG6NL0Ymz0YqGXIb50pDcgqog1n3GYaFQfEZQPgXUOPshjrX3Dp2 VLIziJToyesBJ9WFfABLZ701GxlH5ArQFSJN+7sbhp6N8DHUoPkc7XoVbfPAGp7nInJi IlbfmuPhlj9bk79Ngw49XUSAxAubJSYTtylnmpMInC9anSbwubT8OyPTA9RjWAif5/Fv tQywI/nnJHytCPoE7T+WneD5av5e/AOaGuQ063bdhDGIJiv1w+fEMv4ngvYqzNzEKcDh hpXA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of steve.capper@arm.com designates 217.140.101.70 as permitted sender) smtp.mailfrom=steve.capper@arm.com X-Gm-Message-State: AA+aEWa7gmhwcjyCoLpX3sWBJYxgqoRPns1etX6BWdZNmaJLDSzkf7SC 82Y5WhFwgHc6UDCEBjRX3UuxHfME2zv75cUrp1jUFLZ4YtOPHST33znyIeR21FJe2MFP7l3IMbn bNun3bmr6d86c/nlg7GytR+bnzriMzfaCur8BEW3oZv4/rpEGew6yOt/TGz3vqetXOg== X-Received: by 2002:aca:ce86:: with SMTP id e128mr16092550oig.247.1544028128551; Wed, 05 Dec 2018 08:42:08 -0800 (PST) X-Google-Smtp-Source: AFSGD/UeOxFgedcT3fEBy4mh+glAvHWWGUkSwx4CKoxw0k+P+HyWE8GYcL/axBxX6ZEwej6+pbi8 X-Received: by 2002:aca:ce86:: with SMTP id e128mr16092505oig.247.1544028127584; Wed, 05 Dec 2018 08:42:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544028127; cv=none; d=google.com; s=arc-20160816; b=U82iW8pdEav4Q/fZ8Q0euQGQBv/hpPv958ibY8wCPUXT/6jPZHxkwm+smvOwNv8Hu/ mVE6UYFvI3MxqIpT081P7AE+NmdH+u7Maz0IjuIj7dzcr8M8f2G8/xE1N8sguepQNF4V O25pAl18Udzxhqq+s3JPGDYZXUmvoWpxUduTLO14A0p00yHBbLVGTaJxfftqKppDrUm6 6qIIht/1U+aIRKggG9W/3WGP2MPKE1lBZaFMEd1CPqORoa1YDNpHNnovZptrKR2ol4f6 2Wssgh59anG2/iuIMlZD3ivvAIR/KQg/lP3p1GbuEqb0f1MVuzWj6QiCBQoukJM9JK9k GfdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from; bh=lFR32fSr1ly6v5yn7Uw54cPiu8J3gkqBA0EbNA51000=; b=fecuoGkPPDXLOWUGZKhuLhaLD5e7HEKic47almHtxlbbS4gcuzWFkVvECjj6l/EcGz CdgKELkfFjrMso1RSjBj2MDuEr6p0TG5p1tPtkSMKdLQrNGM/9BISOPjKDowshHgV/J1 kYeGjKFfruS/ZhOY36r7te9bLKqqWSWeT6H9XADLjeORNuSPeVPXwWk7jrFTeIVWWFlF ZR3uEANlCs7KKpHFY579rTxP69aA5LNySdD6YghTfhugwQg7rdqZl6kZmgGesX8Rl49n 7elHUxL8BkEZVwak5lHYAecJ4LtQ9/t9md5Y38VNgsD7Uma4hcfyYOoVSzQ4EbPAl3sl 90lA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of steve.capper@arm.com designates 217.140.101.70 as permitted sender) smtp.mailfrom=steve.capper@arm.com Received: from foss.arm.com (foss.arm.com. [217.140.101.70]) by mx.google.com with ESMTP id x188si9127266oix.138.2018.12.05.08.42.07 for ; Wed, 05 Dec 2018 08:42:07 -0800 (PST) Received-SPF: pass (google.com: domain of steve.capper@arm.com designates 217.140.101.70 as permitted sender) client-ip=217.140.101.70; Authentication-Results: mx.google.com; spf=pass (google.com: domain of steve.capper@arm.com designates 217.140.101.70 as permitted sender) smtp.mailfrom=steve.capper@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DECAC168F; Wed, 5 Dec 2018 08:42:06 -0800 (PST) Received: from capper-debian.emea.arm.com (C02R32KKFVH8.manchester.arm.com [10.32.102.139]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 95B283F5AF; Wed, 5 Dec 2018 08:42:05 -0800 (PST) From: Steve Capper To: linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Cc: catalin.marinas@arm.com, will.deacon@arm.com, ard.biesheuvel@linaro.org, jcm@redhat.com, Steve Capper Subject: [PATCH V4 4/6] arm64: mm: Offset TTBR1 to allow 52-bit PTRS_PER_PGD Date: Wed, 5 Dec 2018 16:41:43 +0000 Message-Id: <20181205164145.24568-5-steve.capper@arm.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20181205164145.24568-1-steve.capper@arm.com> References: <20181205164145.24568-1-steve.capper@arm.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Enabling 52-bit VAs on arm64 requires that the PGD table expands from 64 entries (for the 48-bit case) to 1024 entries. This quantity, PTRS_PER_PGD is used as follows to compute which PGD entry corresponds to a given virtual address, addr: pgd_index(addr) -> (addr >> PGDIR_SHIFT) & (PTRS_PER_PGD - 1) Userspace addresses are prefixed by 0's, so for a 48-bit userspace address, uva, the following is true: (uva >> PGDIR_SHIFT) & (1024 - 1) == (uva >> PGDIR_SHIFT) & (64 - 1) In other words, a 48-bit userspace address will have the same pgd_index when using PTRS_PER_PGD = 64 and 1024. Kernel addresses are prefixed by 1's so, given a 48-bit kernel address, kva, we have the following inequality: (kva >> PGDIR_SHIFT) & (1024 - 1) != (kva >> PGDIR_SHIFT) & (64 - 1) In other words a 48-bit kernel virtual address will have a different pgd_index when using PTRS_PER_PGD = 64 and 1024. If, however, we note that: kva = 0xFFFF << 48 + lower (where lower[63:48] == 0b) and, PGDIR_SHIFT = 42 (as we are dealing with 64KB PAGE_SIZE) We can consider: (kva >> PGDIR_SHIFT) & (1024 - 1) - (kva >> PGDIR_SHIFT) & (64 - 1) = (0xFFFF << 6) & 0x3FF - (0xFFFF << 6) & 0x3F // "lower" cancels out = 0x3C0 In other words, one can switch PTRS_PER_PGD to the 52-bit value globally provided that they increment ttbr1_el1 by 0x3C0 * 8 = 0x1E00 bytes when running with 48-bit kernel VAs (TCR_EL1.T1SZ = 16). For kernel configuration where 52-bit userspace VAs are possible, this patch offsets ttbr1_el1 and sets PTRS_PER_PGD corresponding to the 52-bit value. Suggested-by: Catalin Marinas Signed-off-by: Steve Capper --- This patch is new in V4 of the series --- arch/arm64/include/asm/asm-uaccess.h | 4 ++++ arch/arm64/include/asm/assembler.h | 23 +++++++++++++++++++++++ arch/arm64/include/asm/pgtable-hwdef.h | 9 +++++++++ arch/arm64/include/asm/uaccess.h | 4 ++++ arch/arm64/kernel/head.S | 1 + arch/arm64/kernel/hibernate-asm.S | 1 + arch/arm64/mm/proc.S | 4 ++++ 7 files changed, 46 insertions(+) diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h index 4128bec033f6..cd361dd16b12 100644 --- a/arch/arm64/include/asm/asm-uaccess.h +++ b/arch/arm64/include/asm/asm-uaccess.h @@ -14,11 +14,13 @@ #ifdef CONFIG_ARM64_SW_TTBR0_PAN .macro __uaccess_ttbr0_disable, tmp1 mrs \tmp1, ttbr1_el1 // swapper_pg_dir + restore_ttbr1 \tmp1 bic \tmp1, \tmp1, #TTBR_ASID_MASK sub \tmp1, \tmp1, #RESERVED_TTBR0_SIZE // reserved_ttbr0 just before swapper_pg_dir msr ttbr0_el1, \tmp1 // set reserved TTBR0_EL1 isb add \tmp1, \tmp1, #RESERVED_TTBR0_SIZE + offset_ttbr1 \tmp1 msr ttbr1_el1, \tmp1 // set reserved ASID isb .endm @@ -27,8 +29,10 @@ get_thread_info \tmp1 ldr \tmp1, [\tmp1, #TSK_TI_TTBR0] // load saved TTBR0_EL1 mrs \tmp2, ttbr1_el1 + restore_ttbr1 \tmp2 extr \tmp2, \tmp2, \tmp1, #48 ror \tmp2, \tmp2, #16 + offset_ttbr1 \tmp2 msr ttbr1_el1, \tmp2 // set the active ASID isb msr ttbr0_el1, \tmp1 // set the non-PAN TTBR0_EL1 diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 6142402c2eb4..e2fe378d2a63 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -515,6 +515,29 @@ USER(\label, ic ivau, \tmp2) // invalidate I line PoU mrs \rd, sp_el0 .endm +/* + * Offset ttbr1 to allow for 48-bit kernel VAs set with 52-bit PTRS_PER_PGD. + * orr is used as it can cover the immediate value (and is idempotent). + * In future this may be nop'ed out when dealing with 52-bit kernel VAs. + * ttbr: Value of ttbr to set, modified. + */ + .macro offset_ttbr1, ttbr +#ifdef CONFIG_ARM64_52BIT_VA + orr \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET +#endif + .endm + +/* + * Perform the reverse of offset_ttbr1. + * bic is used as it can cover the immediate value and, in future, won't need + * to be nop'ed out when dealing with 52-bit kernel VAs. + */ + .macro restore_ttbr1, ttbr +#ifdef CONFIG_ARM64_52BIT_VA + bic \ttbr, \ttbr, #TTBR1_BADDR_4852_OFFSET +#endif + .endm + /* * Arrange a physical address in a TTBR register, taking care of 52-bit * addresses. diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 1d7d8da2ef9b..4a29c7e03ae4 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -80,7 +80,11 @@ #define PGDIR_SHIFT ARM64_HW_PGTABLE_LEVEL_SHIFT(4 - CONFIG_PGTABLE_LEVELS) #define PGDIR_SIZE (_AC(1, UL) << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE-1)) +#ifdef CONFIG_ARM64_52BIT_VA +#define PTRS_PER_PGD (1 << (52 - PGDIR_SHIFT)) +#else #define PTRS_PER_PGD (1 << (VA_BITS - PGDIR_SHIFT)) +#endif /* * Section address mask and size definitions. @@ -306,4 +310,9 @@ #define TTBR_BADDR_MASK_52 (((UL(1) << 46) - 1) << 2) #endif +#ifdef CONFIG_ARM64_52BIT_VA +#define TTBR1_BADDR_4852_OFFSET (((UL(1) << (52 - PGDIR_SHIFT)) - \ + (UL(1) << (48 - PGDIR_SHIFT))) * 8) +#endif + #endif diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 07c34087bd5e..df60b3978568 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -124,7 +124,11 @@ static inline void __uaccess_ttbr0_disable(void) ttbr = read_sysreg(ttbr1_el1); ttbr &= ~TTBR_ASID_MASK; /* reserved_ttbr0 placed before swapper_pg_dir */ +#ifdef CONFIG_ARM64_52BIT_VA + write_sysreg((ttbr & ~TTBR1_BADDR_4852_OFFSET) - RESERVED_TTBR0_SIZE, ttbr0_el1); +#else write_sysreg(ttbr - RESERVED_TTBR0_SIZE, ttbr0_el1); +#endif isb(); /* Set reserved ASID */ write_sysreg(ttbr, ttbr1_el1); diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S index 4471f570a295..f60081be9a1b 100644 --- a/arch/arm64/kernel/head.S +++ b/arch/arm64/kernel/head.S @@ -769,6 +769,7 @@ ENTRY(__enable_mmu) phys_to_ttbr x1, x1 phys_to_ttbr x2, x2 msr ttbr0_el1, x2 // load TTBR0 + offset_ttbr1 x1 msr ttbr1_el1, x1 // load TTBR1 isb msr sctlr_el1, x0 diff --git a/arch/arm64/kernel/hibernate-asm.S b/arch/arm64/kernel/hibernate-asm.S index dd14ab8c9f72..fe36d85c60bd 100644 --- a/arch/arm64/kernel/hibernate-asm.S +++ b/arch/arm64/kernel/hibernate-asm.S @@ -40,6 +40,7 @@ tlbi vmalle1 dsb nsh phys_to_ttbr \tmp, \page_table + offset_ttbr1 \tmp msr ttbr1_el1, \tmp isb .endm diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 2c75b0b903ae..2db1c491d45d 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -182,6 +182,7 @@ ENDPROC(cpu_do_switch_mm) .macro __idmap_cpu_set_reserved_ttbr1, tmp1, tmp2 adrp \tmp1, empty_zero_page phys_to_ttbr \tmp2, \tmp1 + offset_ttbr1 \tmp2 msr ttbr1_el1, \tmp2 isb tlbi vmalle1 @@ -200,6 +201,7 @@ ENTRY(idmap_cpu_replace_ttbr1) __idmap_cpu_set_reserved_ttbr1 x1, x3 + offset_ttbr1 x0 msr ttbr1_el1, x0 isb @@ -254,6 +256,7 @@ ENTRY(idmap_kpti_install_ng_mappings) pte .req x16 mrs swapper_ttb, ttbr1_el1 + restore_ttbr1 swapper_ttb adr flag_ptr, __idmap_kpti_flag cbnz cpu, __idmap_kpti_secondary @@ -373,6 +376,7 @@ __idmap_kpti_secondary: cbnz w18, 1b /* All done, act like nothing happened */ + offset_ttbr1 swapper_ttb msr ttbr1_el1, swapper_ttb isb ret