From patchwork Wed Feb 14 12:29:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13556545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D39AC48BC4 for ; Wed, 14 Feb 2024 13:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=SZDldf1thjDqVtUEB1g2hhQivmpncLv6uobqZIIeh90=; b=lm/EL9kVj3D3evgCnMM5SxTrmc VAjrovuLEB7mun1XVUkVuTsnXo4ePEq0Kb0Aoa0Xixxko5QscAaEz6oPKnXbzVj+nRZPYTeeg0nSq 5Nj1eiVpJPDC22M44g6Q9j3CYfoeAQV7J4jqdLuzM9uIKrTxR1QJTfomLleYkyzzFcutvjuog/Z6D DNoS8c08dVQJvGBtv+T9+idg+3JbDSc8JfogpZFfBYcJqh6t5Q9nh19iH+hL8cGubdI16qpHwDftq DeBuuE0P6kM6b/Sm8cS2rSU4499DR3z2xH/uLrGgeu6fCPG7NPYLdM4wPlxsiZdrVZA3QE5rlfGBQ 5ED0filw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1raFa6-0000000D3fY-1VaL; Wed, 14 Feb 2024 13:45:46 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPg-0000000CozD-3Tp6 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Feb 2024 12:30:56 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Message-ID:References:Mime-Version:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=XS5iesF0HNBEmWWUWvb99bP+C1kfTRanM7cS9JJcyAQ=; b=hm75TOavJOGiifcjv2AR4ZQ4h+ IzA43zBP7pJRWmpyiM4ZfH83gjXXaGYmJRDlFTWU7L4bwYYJ8GjJ89YtJvpg8pbqfiNersiRPRExr kPrVRhy6TtizSniNPhqH4//9TwzRhOOAOqrO4IHb6ZYNlYD4kbp31jr4ImMSeSJmB16Aumiv/Th8U kR+r90ShsHza8iS/Y1Whb6ZafccmHF9OZw+XZUMS50bFPOH92hvw3VqFag+pqnn8ZFDVvEGi6qrgb ln/+4pD+X6zIfTc2cFah2cZImmKHxEfW8Rp3O46JD4AdJGD8wUZgVYvA9c5pz8ynG3uc1Y+pvyh8v T2b2Nrwg==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1raEPZ-0000000GE8f-14mW for linux-arm-kernel@lists.infradead.org; Wed, 14 Feb 2024 12:30:53 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5eba564eb3fso110708327b3.1 for ; Wed, 14 Feb 2024 04:30:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707913847; x=1708518647; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=XS5iesF0HNBEmWWUWvb99bP+C1kfTRanM7cS9JJcyAQ=; b=lMY7EzYqj9lJ5/KiaASm3s2rj1fifnIwxHA9PeZWYu9XfI7S+nSO7YV3/W5qDEQJgB btZ9LNWlUcxMBp5BIn+YlD3Q/mhKDrwV1NO0tT0ohlew5Am6Qg7ltrhYy2TJcFr+4XJk mG/v0edgj0uYTUcVKbEN22j9HNDTp5yceaNPWSqCShZoPfxKA02KGvEOt+K6uK1JwF9i 4FOB8As/RsMfhaYURUqo6dYfUp7/AOdx+JewaHbPlxlwubtwEB+003b99ovplyB2uTLl OknJY3s+sG6GHR7O4C8rpwBal9VG0nCPCK/wys34utT7NF0bz7vdQg4yS4Tn8jAi2rxK 22qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707913847; x=1708518647; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XS5iesF0HNBEmWWUWvb99bP+C1kfTRanM7cS9JJcyAQ=; b=Gc0DDisuiO4IotSiyuZsFLmo3P66i6rLy8bV+AuUs9m+YboPhg1zoMWwFQMtnSCpSn LrBrZtVEStXCvxYX+fQY4Ch10g2dIsETam0G8XXk/bHn7Fv4LWyxRQgfiFczMGGQQCE/ ow0JNlZ2I71EDUO+jvXs5tOvcjiGizFVthNvuxyHGmtJGxqBKsNg2hgl7GbFn4jvq0I7 auRJ0Dk3bkJDdLl4/bwPBX1WFBy4rGWEexGkXjRGml7vkbzNw+HLEtFoIYv5CDxb1bqH QM4XfGQqDsmjWApuNNpCuRa+OnIxKbynvNeouNez/DQ9ZD1/caBFNKkUvtYw/mblCVIs 868w== X-Gm-Message-State: AOJu0Yy2d0pJOeGVEWuk585fi0AIOkCxftUwZn3+yCGBhDUL91hmsppZ HFipZTuHPxmhN2LAbXAV2nt4q1+QVFWep6ZiyIa7Vf4InaJ6Y+Q6d4lY6rInW+MFayTykC/AWyv Z+KB1SPs9gxGwebPtIWNjUZckk9QKb+mi6LrzsB/2/LTBsNxIUShmr+XWgAcYAxqgasbVCKujIF PvJNxF5YjWsKHn2Vmb0gopBoVYuiTsliQaVK/j9I5D X-Google-Smtp-Source: AGHT+IHI8MGiGaICtaaucPLtig05RYHAf3O3BfbZfPiJgfDSXWvX1mEoHpKVYS+ZekatiEXc2POsKGQ5 X-Received: from palermo.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:118a]) (user=ardb job=sendgmr) by 2002:a05:6902:1826:b0:dcd:25be:aefb with SMTP id cf38-20020a056902182600b00dcd25beaefbmr494352ybb.13.1707913847174; Wed, 14 Feb 2024 04:30:47 -0800 (PST) Date: Wed, 14 Feb 2024 13:29:16 +0100 In-Reply-To: <20240214122845.2033971-45-ardb+git@google.com> Mime-Version: 1.0 References: <20240214122845.2033971-45-ardb+git@google.com> X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 X-Developer-Signature: v=1; a=openpgp-sha256; l=4645; i=ardb@kernel.org; h=from:subject; bh=bhq1UZsl34o3/V5QqWl176mqS7CVCTze3JDjfKMi/9U=; b=owGbwMvMwCFmkMcZplerG8N4Wi2JIfXMJjl9n1nLCmZoCa6X2SlmEX5/ln/P/wk/Jyx3e3wqK Pd87ePyjlIWBjEOBlkxRRaB2X/f7Tw9UarWeZYszBxWJpAhDFycAjARcX5GhiNcUVbSUxZlL+Lc k3fcVKOybKWSWTb3wVOd6qcObv3qzcnw3y0u74V215eVGRXN3Ht/+VlYTdx1v7uw93R5/afTEXZ lfAA= X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214122845.2033971-75-ardb+git@google.com> Subject: [PATCH v8 30/43] arm64: mm: Add LPA2 support to phys<->pte conversion routines From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org Cc: Ard Biesheuvel , Catalin Marinas , Will Deacon , Marc Zyngier , Mark Rutland , Ryan Roberts , Anshuman Khandual , Kees Cook X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240214_123049_449157_76A07356 X-CRM114-Status: GOOD ( 15.39 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Ard Biesheuvel In preparation for enabling LPA2 support, introduce the mask values for converting between physical addresses and their representations in a page table descriptor. While at it, move the pte_to_phys asm macro into its only user, so that we can freely modify it to use its input value register as a temp register. For LPA2, the PTE_ADDR_MASK contains two non-adjacent sequences of zero bits, which means it no longer fits into the immediate field of an ordinary ALU instruction. So let's redefine it to include the bits in between as well, and only use it when converting from physical address to PTE representation, where the distinction does not matter. Also update the name accordingly to emphasize this. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/assembler.h | 16 ++-------------- arch/arm64/include/asm/pgtable-hwdef.h | 10 +++++++--- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/proc.S | 8 ++++++++ 4 files changed, 20 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 68a99b116256..7eedcb36ebe0 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -612,25 +612,13 @@ alternative_endif .macro phys_to_pte, pte, phys #ifdef CONFIG_ARM64_PA_BITS_52 - /* - * We assume \phys is 64K aligned and this is guaranteed by only - * supporting this configuration with 64K pages. - */ - orr \pte, \phys, \phys, lsr #36 - and \pte, \pte, #PTE_ADDR_MASK + orr \pte, \phys, \phys, lsr #PTE_ADDR_HIGH_SHIFT + and \pte, \pte, #PHYS_TO_PTE_ADDR_MASK #else mov \pte, \phys #endif .endm - .macro pte_to_phys, phys, pte - and \phys, \pte, #PTE_ADDR_MASK -#ifdef CONFIG_ARM64_PA_BITS_52 - orr \phys, \phys, \phys, lsl #PTE_ADDR_HIGH_SHIFT - and \phys, \phys, GENMASK_ULL(PHYS_MASK_SHIFT - 1, PAGE_SHIFT) -#endif - .endm - /* * tcr_clear_errata_bits - Clear TCR bits that trigger an errata on this CPU. */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index b770f98fc0b5..4426f48f2ae0 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -155,13 +155,17 @@ #define PTE_PXN (_AT(pteval_t, 1) << 53) /* Privileged XN */ #define PTE_UXN (_AT(pteval_t, 1) << 54) /* User XN */ -#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) +#define PTE_ADDR_LOW (((_AT(pteval_t, 1) << (50 - PAGE_SHIFT)) - 1) << PAGE_SHIFT) #ifdef CONFIG_ARM64_PA_BITS_52 +#ifdef CONFIG_ARM64_64K_PAGES #define PTE_ADDR_HIGH (_AT(pteval_t, 0xf) << 12) -#define PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #define PTE_ADDR_HIGH_SHIFT 36 +#define PHYS_TO_PTE_ADDR_MASK (PTE_ADDR_LOW | PTE_ADDR_HIGH) #else -#define PTE_ADDR_MASK PTE_ADDR_LOW +#define PTE_ADDR_HIGH (_AT(pteval_t, 0x3) << 8) +#define PTE_ADDR_HIGH_SHIFT 42 +#define PHYS_TO_PTE_ADDR_MASK GENMASK_ULL(49, 8) +#endif #endif /* diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 522c21348ae8..61de7b1516bc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,15 +80,16 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #ifdef CONFIG_ARM64_PA_BITS_52 static inline phys_addr_t __pte_to_phys(pte_t pte) { + pte_val(pte) &= ~PTE_MAYBE_SHARED; return (pte_val(pte) & PTE_ADDR_LOW) | ((pte_val(pte) & PTE_ADDR_HIGH) << PTE_ADDR_HIGH_SHIFT); } static inline pteval_t __phys_to_pte_val(phys_addr_t phys) { - return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PTE_ADDR_MASK; + return (phys | (phys >> PTE_ADDR_HIGH_SHIFT)) & PHYS_TO_PTE_ADDR_MASK; } #else -#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_MASK) +#define __pte_to_phys(pte) (pte_val(pte) & PTE_ADDR_LOW) #define __phys_to_pte_val(phys) (phys) #endif diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 7c46f8cfd6ae..d03434b7bca5 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -205,6 +205,14 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .pushsection ".idmap.text", "a" + .macro pte_to_phys, phys, pte + and \phys, \pte, #PTE_ADDR_LOW +#ifdef CONFIG_ARM64_PA_BITS_52 + and \pte, \pte, #PTE_ADDR_HIGH + orr \phys, \phys, \pte, lsl #PTE_ADDR_HIGH_SHIFT +#endif + .endm + .macro kpti_mk_tbl_ng, type, num_entries add end_\type\()p, cur_\type\()p, #\num_entries * 8 .Ldo_\type: