From patchwork Sat Nov 2 00:08:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13859849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A602BE6F097 for ; Sat, 2 Nov 2024 00:09:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=S+llj3H3A7IJcjYa1ZoEK/UQYmbZdnS8jH91AZf5w9g=; b=tUboBC/9HYGCLT d0yKbtMUId7VomcCGgJUesoMJ53C/flKrmLhE5EDQXgTBLLgFAf6Vq3/NgKmuHG/gh6cXj7Bka41L B1VMH12zBPciABIAn/rnjH3V1OTwFJqM6TYYenvHf250D2/MOQQhM5Crr7yEWJljGIGtDXhc3XcR5 aqTljuCN98CA5+njSvCAjtvb+VDCs+1HTXrPzRexu2r2v9LIc5qjnbZO636xIuOq91HZ4OjWbxTSf tilMXuuaQxga5j2gkGOIhlAnLRRICOlF2l8WrqVYDHD7kxo/H8GnVhy5pQPVM+JfbVyvz5Pn86FAG QbQYIbRxkQZjo77YKmtw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t71hQ-00000008a8L-43Za; Sat, 02 Nov 2024 00:09:04 +0000 Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t71hM-00000008a57-2rdz for linux-riscv@lists.infradead.org; Sat, 02 Nov 2024 00:09:02 +0000 Received: by mail-pj1-x102b.google.com with SMTP id 98e67ed59e1d1-2e2eba31d3aso1867962a91.2 for ; Fri, 01 Nov 2024 17:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1730506140; x=1731110940; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bMdTiw85E/ndb+s2ImQQp45t4KJbvNlvAuA5mb4sn38=; b=TBmMpVayzillyTzQrVzB6RtyW6AwbTIZu5n7DusFahHS+WUtpe40OFJNgtqDBOtI/f ak/+pKsXQemyDvHb+0yIpHRTxpF7A/MDauOyRROKvaM6DsF6G+q5wmcR45tH+tuM9AxU /s9lmsNIFTCMuJEMbA+icQTSpDZ8BXomWJVddTVq/8g6TEMfeGvgscF2qT7Lyu+tuJm7 pFUPdo4tgzo4bq/QiEW1Bi6SEW+d7fCUYDdN88GIaZT3ZRT9HeVaz03Bk+Zt/OA4YtjT c5VfV6E/sLV7WySAnbAj/74w8+KZFBrM1/Bc27nABGeOKwbEeWogpaaquotRQ8Y7+J8G EcnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730506140; x=1731110940; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bMdTiw85E/ndb+s2ImQQp45t4KJbvNlvAuA5mb4sn38=; b=cara75FYr10D78kIzthf3GnUJ6BRcJVI1BKf/v6+oQCcfCvvCsbTxziZlht24SBTRO AXm6taJhb3NMBbILUpmIvzEvpOnV0vpRcKaASQhWuTR0rVuI8ARQOuwiObWYYu5abUEQ WVg2CEbUOAEBFPps5YnbcoF/ri4cMa9ohqhpkEzsxRSdhUCqOidXPYDfTXPmmClquit3 yM9TME96fJvvK6rDDAJKWZ4WYJbOfVyVhILbsD0i8nbdVfmaCRQ0DqwU7AY0hq01bW+j YXWK2n1d6CQe9X7DawkV2GIzDWC32OUo9ud7i8BNFNbGFnELM4Z4tfU8ibeosIa+59M8 ljnA== X-Forwarded-Encrypted: i=1; AJvYcCUzeamULCvkDTZ0JPLmK8u3Hn9gjBjC0DBfuQu5ZklVcS/ABUpuRe9y2OhrjsbabRsQcBTilroteNZEwg==@lists.infradead.org X-Gm-Message-State: AOJu0YyhU/HC0g7q939wTtftcpG83Q9LycbhBfwCCBJcoFcjtCDR4Kz2 UAHOzROqRa5DnFCYm5W5qD2L7STx7GGhhF7jnoXUCvph4nC7g7moH3ja/RW2tDI= X-Google-Smtp-Source: AGHT+IHB07oy//WNtEG4aQ2tSWxlcw5NVWtvl+A9a2GkhEm/z9LgVjJWHeMoDLv894FIO26fgWOlJg== X-Received: by 2002:a17:90b:390e:b0:2e2:b64e:f4f7 with SMTP id 98e67ed59e1d1-2e8f11b8c2amr26674437a91.29.1730506140045; Fri, 01 Nov 2024 17:09:00 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e92fc00856sm5505749a91.54.2024.11.01.17.08.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Nov 2024 17:08:59 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Conor Dooley Cc: devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, Alexandre Ghiti , Lad Prabhakar , Emil Renner Berthing , Rob Herring , Krzysztof Kozlowski , Samuel Holland Subject: [PATCH 10/11] riscv: mm: Use physical memory aliases to apply PMAs Date: Fri, 1 Nov 2024 17:08:04 -0700 Message-ID: <20241102000843.1301099-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20241102000843.1301099-1-samuel.holland@sifive.com> References: <20241102000843.1301099-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241101_170900_756004_EE6422DD X-CRM114-Status: GOOD ( 30.13 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On some RISC-V platforms, RAM is mapped to multiple physical address ranges, with each alias having a different set of statically-determined Physical Memory Attributes (PMAs). Software selects the PMAs for a page by choosing a PFN from the corresponding physical address range. Implement this by transforming the PFN when writing page tables. If the memory type field is nonzero, replace the PFN with the corresponding PFN from the noncached alias. Similarly, when reading from the page tables, if the PFN is found in a noncached alias, replace it with the corresponding PFN from the cached alias, and insert _PAGE_NOCACHE. The rest of the kernel sees only the cached PFNs and _PAGE_MTMASK values as if Svpbmt was implemented. Memory alias pairs are determined from the devicetree. A new cpufeature bit is required because that is the only way to trigger alternative patching. Signed-off-by: Samuel Holland --- arch/riscv/Kconfig | 3 + arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/include/asm/pgtable-64.h | 27 ++++++-- arch/riscv/include/asm/pgtable.h | 8 +++ arch/riscv/kernel/cpufeature.c | 6 ++ arch/riscv/kernel/setup.c | 1 + arch/riscv/mm/Makefile | 1 + arch/riscv/mm/memory-alias.S | 101 ++++++++++++++++++++++++++++ arch/riscv/mm/pgtable.c | 91 +++++++++++++++++++++++++ 9 files changed, 235 insertions(+), 4 deletions(-) create mode 100644 arch/riscv/mm/memory-alias.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 62545946ecf4..d28d1dab5f26 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -566,6 +566,9 @@ config RISCV_ISA_SVPBMT The Svpbmt extension is only available on 64-bit cpus. + This option also controls selection of memory type based on + physical memory aliases. + If you don't know what to do here, say Y. config TOOLCHAIN_HAS_V diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 46d9de54179e..8a37e22f4223 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -94,6 +94,7 @@ #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_SVVPTC 86 +#define RISCV_ISA_EXT_XLINUXMEMALIAS 126 #define RISCV_ISA_EXT_XLINUXENVCFG 127 #define RISCV_ISA_EXT_MAX 128 diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 174b6a5837c2..6b4af408a37a 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -124,27 +124,46 @@ enum napot_cont_order { */ #define ALT_FIXUP_MT(_val) \ - asm(ALTERNATIVE_2("addi t0, zero, 0x3\n\t" \ + asm(ALTERNATIVE_3("addi t0, zero, 0x3\n\t" \ "slli t0, t0, 61\n\t" \ "not t0, t0\n\t" \ "and %0, %0, t0\n\t" \ "nop\n\t" \ "nop\n\t" \ + "nop\n\t" \ "nop", \ - __nops(7), \ + __nops(8), \ 0, RISCV_ISA_EXT_SVPBMT, CONFIG_RISCV_ISA_SVPBMT, \ + "addi t0, zero, 0x3\n\t" \ + "slli t0, t0, 61\n\t" \ + "and t0, %0, t0\n\t" \ + "beqz t0, 2f\n\t" \ + "xor t1, %0, t0\n\t" \ + "1: auipc t0, %%pcrel_hi(riscv_fixup_memory_alias)\n\t" \ + "jalr t0, t0, %%pcrel_lo(1b)\n\t" \ + "mv %0, t1\n" \ + "2:", \ + 0, RISCV_ISA_EXT_XLINUXMEMALIAS, CONFIG_RISCV_ISA_SVPBMT, \ "srli t0, %0, 59\n\t" \ "seqz t1, t0\n\t" \ "slli t1, t1, 1\n\t" \ "or t0, t0, t1\n\t" \ "xori t0, t0, 0x5\n\t" \ "slli t0, t0, 60\n\t" \ - "xor %0, %0, t0", \ + "xor %0, %0, t0\n\t" \ + "nop", \ THEAD_VENDOR_ID, ERRATA_THEAD_MAE, CONFIG_ERRATA_THEAD_MAE) \ : "+r" (_val) :: "t0", "t1") #define ALT_UNFIX_MT(_val) \ - asm(ALTERNATIVE(__nops(6), \ + asm(ALTERNATIVE_2(__nops(6), \ + "mv t1, %0\n\t" \ + "1: auipc t0, %%pcrel_hi(riscv_unfix_memory_alias)\n\t" \ + "jalr t0, t0, %%pcrel_lo(1b)\n\t" \ + "mv %0, t1\n\t" \ + "nop\n\t" \ + "nop", \ + 0, RISCV_ISA_EXT_XLINUXMEMALIAS, CONFIG_RISCV_ISA_SVPBMT, \ "srli t0, %0, 60\n\t" \ "andi t0, t0, 0xd\n\t" \ "srli t1, t0, 1\n\t" \ diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 3ffcff76ac0d..0e52dfaaff63 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -949,6 +949,14 @@ extern u64 satp_mode; void paging_init(void); void misc_mem_init(void); +#ifdef CONFIG_RISCV_ISA_SVPBMT +bool __init riscv_have_memory_alias(void); +void __init riscv_init_memory_alias(void); +#else +static inline bool riscv_have_memory_alias(void) { return false; } +static inline void riscv_init_memory_alias(void) {} +#endif /* CONFIG_RISCV_ISA_SVPBMT */ + /* * ZERO_PAGE is a global shared page that is always zero, * used for zero-mapped memory areas, etc. diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 3a8eeaa9310c..ca36f8240a86 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -892,6 +892,12 @@ void __init riscv_fill_hwcap(void) elf_hwcap &= ~COMPAT_HWCAP_ISA_V; } + /* Vendor-independent alternatives require a bit in the ISA bitmap. */ + if (riscv_have_memory_alias()) { + set_bit(RISCV_ISA_EXT_XLINUXMEMALIAS, riscv_isa); + pr_info("Using physical memory alias for noncached mappings\n"); + } + memset(print_str, 0, sizeof(print_str)); for (i = 0, j = 0; i < NUM_ALPHA_EXTS; i++) if (riscv_isa[0] & BIT_MASK(i)) diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..ab718fc4538f 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -287,6 +287,7 @@ void __init setup_arch(char **cmdline_p) } riscv_init_cbo_blocksizes(); + riscv_init_memory_alias(); riscv_fill_hwcap(); init_rt_signal_env(); apply_boot_alternatives(); diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index cbe4d775ef56..50d843b298cd 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -33,3 +33,4 @@ endif obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o obj-$(CONFIG_RISCV_DMA_NONCOHERENT) += dma-noncoherent.o obj-$(CONFIG_RISCV_NONSTANDARD_CACHE_OPS) += cache-ops.o +obj-$(CONFIG_RISCV_ISA_SVPBMT) += memory-alias.o diff --git a/arch/riscv/mm/memory-alias.S b/arch/riscv/mm/memory-alias.S new file mode 100644 index 000000000000..df2e8cc3f69c --- /dev/null +++ b/arch/riscv/mm/memory-alias.S @@ -0,0 +1,101 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 SiFive + */ + +#include +#include +#include +#include + +#define CACHED_BASE_OFFSET (0 * RISCV_SZPTR) +#define NONCACHED_BASE_OFFSET (1 * RISCV_SZPTR) +#define SIZE_OFFSET (2 * RISCV_SZPTR) + +#define SIZEOF_PAIR (4 * RISCV_SZPTR) + +SYM_CODE_START(riscv_fixup_memory_alias) + addi sp, sp, -4 * SZREG + REG_S t2, (0 * SZREG)(sp) + REG_S t3, (1 * SZREG)(sp) + REG_S t4, (2 * SZREG)(sp) +#ifdef CONFIG_RISCV_ISA_SVNAPOT + REG_S t5, (3 * SZREG)(sp) + + /* Save and mask off _PAGE_NAPOT if present. */ + li t5, _PAGE_NAPOT + and t5, t1, t5 + xor t1, t1, t5 +#endif + + lla t2, memory_alias_pairs +.Lfixup_loop: + REG_L t3, SIZE_OFFSET(t2) + beqz t3, .Lfixup_end + REG_L t4, CACHED_BASE_OFFSET(t2) + sub t4, t1, t4 + bltu t4, t3, .Lfixup_found + addi t2, t2, SIZEOF_PAIR + j .Lfixup_loop + +.Lfixup_found: + REG_L t3, NONCACHED_BASE_OFFSET(t2) + add t1, t3, t4 + +.Lfixup_end: +#ifdef CONFIG_RISCV_ISA_SVNAPOT + xor t1, t1, t5 + + REG_L t5, (3 * SZREG)(sp) +#endif + REG_L t4, (2 * SZREG)(sp) + REG_L t3, (1 * SZREG)(sp) + REG_L t2, (0 * SZREG)(sp) + addi sp, sp, 4 * SZREG + jr t0 +SYM_CODE_END(riscv_fixup_memory_alias) + +SYM_CODE_START(riscv_unfix_memory_alias) + addi sp, sp, -4 * SZREG + REG_S t2, (0 * SZREG)(sp) + REG_S t3, (1 * SZREG)(sp) + REG_S t4, (2 * SZREG)(sp) +#ifdef CONFIG_RISCV_ISA_SVNAPOT + REG_S t5, (3 * SZREG)(sp) + + /* Save and mask off _PAGE_NAPOT if present. */ + li t5, _PAGE_NAPOT + and t5, t1, t5 + xor t1, t1, t5 +#endif + + lla t2, memory_alias_pairs +.Lunfix_loop: + REG_L t3, SIZE_OFFSET(t2) + beqz t3, .Lunfix_end + REG_L t4, NONCACHED_BASE_OFFSET(t2) + sub t4, t1, t4 + bltu t4, t3, .Lunfix_found + addi t2, t2, SIZEOF_PAIR + j .Lunfix_loop + +.Lunfix_found: + REG_L t3, CACHED_BASE_OFFSET(t2) + add t1, t3, t4 + + /* PFN was in the noncached alias, so mark it as such. */ + li t2, _PAGE_NOCACHE + or t1, t1, t2 + +.Lunfix_end: +#ifdef CONFIG_RISCV_ISA_SVNAPOT + xor t1, t1, t5 + + REG_L t5, (3 * SZREG)(sp) +#endif + REG_L t4, (2 * SZREG)(sp) + REG_L t3, (1 * SZREG)(sp) + REG_L t2, (0 * SZREG)(sp) + addi sp, sp, 4 * SZREG + jr t0 +SYM_CODE_END(riscv_unfix_memory_alias) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 4ae67324f992..8dd43001cd10 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -1,8 +1,12 @@ // SPDX-License-Identifier: GPL-2.0 #include +#include +#include #include #include +#include +#include #include int ptep_set_access_flags(struct vm_area_struct *vma, @@ -155,3 +159,90 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, return pmd; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +#ifdef CONFIG_RISCV_ISA_SVPBMT +struct memory_alias_pair { + unsigned long cached_base; + unsigned long noncached_base; + unsigned long size; + int index; +} memory_alias_pairs[5]; + +bool __init riscv_have_memory_alias(void) +{ + return memory_alias_pairs[0].size; +} + +void __init riscv_init_memory_alias(void) +{ + int na = of_n_addr_cells(of_root); + int ns = of_n_size_cells(of_root); + int nc = na + ns + 2; + const __be32 *prop; + int pairs = 0; + int len; + + prop = of_get_property(of_root, "riscv,physical-memory-regions", &len); + if (!prop) + return; + + len /= sizeof(__be32); + for (int i = 0; len >= nc; i++, prop += nc, len -= nc) { + unsigned long base = of_read_ulong(prop, na); + unsigned long size = of_read_ulong(prop + na, ns); + unsigned long flags = be32_to_cpup(prop + na + ns); + struct memory_alias_pair *pair; + int alias; + + /* We only care about non-coherent memory. */ + if ((flags & PMA_ORDER_MASK) != PMA_ORDER_MEMORY || (flags & PMA_COHERENT)) + continue; + + /* The cacheable alias must be usable memory. */ + if ((flags & PMA_CACHEABLE) && + !memblock_overlaps_region(&memblock.memory, base, size)) + continue; + + alias = FIELD_GET(PMR_ALIAS_MASK, flags); + if (alias) { + pair = NULL; + for (int j = 0; j < pairs; j++) { + if (alias == memory_alias_pairs[j].index) { + pair = &memory_alias_pairs[j]; + break; + } + } + if (!pair) + continue; + } else { + /* Leave room for the null sentinel. */ + if (pairs == ARRAY_SIZE(memory_alias_pairs) - 1) + continue; + pair = &memory_alias_pairs[pairs++]; + pair->index = i; + } + + /* Align the address and size with the page table PFN field. */ + base >>= PAGE_SHIFT - _PAGE_PFN_SHIFT; + size >>= PAGE_SHIFT - _PAGE_PFN_SHIFT; + + if (flags & PMA_CACHEABLE) + pair->cached_base = base; + else + pair->noncached_base = base; + pair->size = min_not_zero(pair->size, size); + } + + /* Remove any unmatched pairs. */ + for (int i = 0; i < pairs; i++) { + struct memory_alias_pair *pair = &memory_alias_pairs[i]; + + if (pair->cached_base && pair->noncached_base && pair->size) + continue; + + for (int j = i + 1; j < pairs; j++) + memory_alias_pairs[j - 1] = memory_alias_pairs[j]; + memory_alias_pairs[--pairs].size = 0; + } +} +#endif /* CONFIG_RISCV_ISA_SVPBMT */