From patchwork Wed Aug 14 08:55:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9943BC52D7F for ; Wed, 14 Aug 2024 08:57:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=p+ZWbM+3N2UqMrGeBrOGYCDvWi KwuS57AhifBbu2B9ywAJ45fNZyjhhvAX2BZecxBFWdFCY75pKge9Zcf1K+Xw0/dN2YV33qNfd+Pyt QJf75aLobsPPppJRAW09npBzRDvyDazZHOApEsuSItt8SyEryAHhgMg/0OZnN81bdganxcnjux3BY nD0ZzjIhOjGbTifnZmx8Xaxxu1vPrL+LnsPcItxYnm0iZqFDXCAMetmu05wYJLR09ou1Nj0CsoTOE 10WecMGaZ5+FWcdKXg9ZZbS2rPsEzR6Ekw3V6ukKE5sCP17dXgvHHMF8+7v9W+pCGSU/V4f96FWyb jodKm6YQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9p9-00000006Ieg-2Fvx; Wed, 14 Aug 2024 08:57:43 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9ny-00000006IBU-2ZC5 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=ji57IXUbZOXsGiFQoumo4XWXNa u3WMfFNoBuDTzXE8fv5jQ8mVY/vcbYEDr388RIKnmRmKKZBaBq7IsQMp4eBIA3CSBGD3u1L1Et3A5 0wHroIvkBD2GCirNN04MyxGBF7/UjvEFNBID89URVScl6TRSLDDa6mwX1noIIiBCzDud1TJqknLxt kqmk9kQx8r0jU4xlb+f2Sg4OZwXZ5ZzjzR0pQaLS3LlZHYGPLVZN533oYblrEDX4Eb4EdTllG5gLI xH4Yr+WWXgOgpRL8HWRbN9+7K4nPf/PORPYJi0M8hu0nIR6dVz6ylTJhg0tz4svw6lufiH4ZI8lLi qy6OyNNg==; Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9nt-00000007z9T-1sKW for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:28 +0000 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-76cb5b6b3e4so4215951a12.1 for ; Wed, 14 Aug 2024 01:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625782; x=1724230582; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=AUWey4GXIQve35UzoHaJIckjTrNWtsfsJn7FBXn+thjOedff3tRNySxU3ogku+plue HXJyLawFzNk6Aq7AzPbUZfDnqnps4ID9hIUpZhoZuXSHlATTE6OCS2+cPxuORIVkAOfG l6eKjBlWsiVA6DxpIQCr6oKd1u9z1mPau/9AeO7bvqy2v1bUrKnBTbsSzhxypJqBl2GJ 8y6hwIeBH9joXoIuOZetPlfCUxw5UQIxfinRcfU86Ww/j9+2njDbLiUYxTwhYt3dXajf ccVXScmbZSdzKLqz3+nPTYbWxkiK9YBaZXpLbQI70mDb8qQkQ8bzyZUx1SKA7V6eJyRF owyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625782; x=1724230582; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=i9G4It+59P9jTNpvNfe60Jvnkp7Jegto/MYm9GPMiIvqR3WtMXlQYtiq/NsBZJHrsS EArx3J77reI6hFusTShk4o6haWSIxxLi9eE6sL9wTje+QE1Ip5eXmynm1CpzfZgbeGWU erpXH6gDNWI2lpJeJC0MqHi8OoV+gSAg97gtT1RVbaHXEGRmswUK6CBIJ5jKQxo/Ur2o j6el0jH80bFM6GjGK8tiDJ9zHz2Rne05CKQVFChFBKQB1DPO+wwMRmA0wDOA5iNgBEB9 j+jedFRdnFLezg7KDI49AsmVIpUf4Q7FTRF7XLOrf72fUJ83tS9IGXgCQMbzRdC9qF15 meOA== X-Forwarded-Encrypted: i=1; AJvYcCXj/4Ta9yGpOgrzYWKKqOzBTNLLflzI7pnB05pNM+4y4PhQX9pQ+SG1E+bcmr+IdAsQnZJvAI/fq+EAAT1+YTrgA/LSutBkycNBqTdJMj5448KIRQc= X-Gm-Message-State: AOJu0YwSlOtPOCnT6fMJF9ZFDxHXjsMvRyqTTp3fe7+3IkoRh/REqB1U WIH6OH6Uc8llsGV0PCufGlihPfavsuGFrPauor392w2TWLITj7xUe5YeONuyBRM= X-Google-Smtp-Source: AGHT+IFEE/BC7r7DqzdXwRgEMrr+FQ81W6ResIpLXSNMMxD7udxBFFjplbf0XoQXYF4QlNhTBdsXYw== X-Received: by 2002:a05:6a21:9206:b0:1c3:b1e2:f826 with SMTP id adf61e73a8af0-1c8eaf5ab1bmr2775585637.35.1723625782207; Wed, 14 Aug 2024 01:56:22 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:21 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 1/7] kasan: sw_tags: Use arithmetic shift for shadow computation Date: Wed, 14 Aug 2024 01:55:29 -0700 Message-ID: <20240814085618.968833-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095625_798317_92F9B260 X-CRM114-Status: GOOD ( 25.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, kasan_mem_to_shadow() uses a logical right shift, which turns canonical kernel addresses into non-canonical addresses by clearing the high KASAN_SHADOW_SCALE_SHIFT bits. The value of KASAN_SHADOW_OFFSET is then chosen so that the addition results in a canonical address for the shadow memory. For KASAN_GENERIC, this shift/add combination is ABI with the compiler, because KASAN_SHADOW_OFFSET is used in compiler-generated inline tag checks[1], which must only attempt to dereference canonical addresses. However, for KASAN_SW_TAGS we have some freedom to change the algorithm without breaking the ABI. Because TBI is enabled for kernel addresses, the top bits of shadow memory addresses computed during tag checks are irrelevant, and so likewise are the top bits of KASAN_SHADOW_OFFSET. This is demonstrated by the fact that LLVM uses a logical right shift in the tag check fast path[2] but a sbfx (signed bitfield extract) instruction in the slow path[3] without causing any issues. Using an arithmetic shift in kasan_mem_to_shadow() provides a number of benefits: 1) The memory layout is easier to understand. KASAN_SHADOW_OFFSET becomes a canonical memory address, and the shifted pointer becomes a negative offset, so KASAN_SHADOW_OFFSET == KASAN_SHADOW_END regardless of the shift amount or the size of the virtual address space. 2) KASAN_SHADOW_OFFSET becomes a simpler constant, requiring only one instruction to load instead of two. Since it must be loaded in each function with a tag check, this decreases kernel text size by 0.5%. 3) This shift and the sign extension from kasan_reset_tag() can be combined into a single sbfx instruction. When this same algorithm change is applied to the compiler, it removes an instruction from each inline tag check, further reducing kernel text size by an additional 4.6%. These benefits extend to other architectures as well. On RISC-V, where the baseline ISA does not shifted addition or have an equivalent to the sbfx instruction, loading KASAN_SHADOW_OFFSET is reduced from 3 to 2 instructions, and kasan_mem_to_shadow(kasan_reset_tag(addr)) similarly combines two consecutive right shifts. Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp#L1316 [1] Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp#L895 [2] Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp#L669 [3] Signed-off-by: Samuel Holland --- arch/arm64/Kconfig | 10 +++++----- arch/arm64/include/asm/memory.h | 8 ++++++++ arch/arm64/mm/kasan_init.c | 7 +++++-- include/linux/kasan.h | 10 ++++++++-- scripts/gdb/linux/mm.py | 5 +++-- 5 files changed, 29 insertions(+), 11 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2f8ff354ca6..7df218cca168 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -402,11 +402,11 @@ config KASAN_SHADOW_OFFSET default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS - default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS - default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS - default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS - default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS - default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS + default 0xffff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS + default 0xffffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS + default 0xfffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS + default 0xffffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS + default 0xfffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff config UNWIND_TABLES diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 54fb014eba05..3af8d1e721af 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -82,6 +82,10 @@ * the mapping. Note that KASAN_SHADOW_OFFSET does not point to the start of * the shadow memory region. * + * For KASAN_GENERIC, addr is treated as unsigned. For KASAN_SW_TAGS, addr is + * treated as signed, so in that case KASAN_SHADOW_OFFSET points to the end of + * the shadow memory region. + * * Based on this mapping, we define two constants: * * KASAN_SHADOW_START: the start of the shadow memory region; @@ -100,7 +104,11 @@ */ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) +#ifdef CONFIG_KASAN_GENERIC #define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) + KASAN_SHADOW_OFFSET) +#else +#define KASAN_SHADOW_END KASAN_SHADOW_OFFSET +#endif #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (UL(1) << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) #define PAGE_END KASAN_SHADOW_START diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index b65a29440a0c..6836e571555c 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -198,8 +198,11 @@ static bool __init root_level_aligned(u64 addr) /* The early shadow maps everything to a single page of zeroes */ asmlinkage void __init kasan_early_init(void) { - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + else + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 70d6a8f6e25d..41f57e10ba03 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -58,8 +58,14 @@ int kasan_populate_early_shadow(const void *shadow_start, #ifndef kasan_mem_to_shadow static inline void *kasan_mem_to_shadow(const void *addr) { - return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT) - + KASAN_SHADOW_OFFSET; + void *scaled; + + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); + else + scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); + + return KASAN_SHADOW_OFFSET + scaled; } #endif diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index 7571aebbe650..2e63f3dedd53 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -110,12 +110,13 @@ class aarch64_page_ops(): self.KERNEL_END = gdb.parse_and_eval("_end") if constants.LX_CONFIG_KASAN_GENERIC or constants.LX_CONFIG_KASAN_SW_TAGS: + self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET if constants.LX_CONFIG_KASAN_GENERIC: self.KASAN_SHADOW_SCALE_SHIFT = 3 + self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET else: self.KASAN_SHADOW_SCALE_SHIFT = 4 - self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET - self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET + self.KASAN_SHADOW_END = self.KASAN_SHADOW_OFFSET self.PAGE_END = self.KASAN_SHADOW_END - (1 << (self.vabits_actual - self.KASAN_SHADOW_SCALE_SHIFT)) else: self.PAGE_END = self._PAGE_END(self.VA_BITS_MIN) From patchwork Wed Aug 14 08:55:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 48BCDC52D7F for ; Wed, 14 Aug 2024 08:58:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=FOU7P9jn4wor6Rdt5JEa4HOxmny7NogNXhx9YRYE+tY=; b=OoSDSJkhMGu57ehhxYCGaOCct3 Ck30XFAUlVrjv3kMV5eZNfhNic+yjRcwnKgY24y/dcVi9jHcEvJb8YOzHTn+Sq5mMqZGpMUX9mNfu Zamu8jwd38Bv9O7/ONFKl19fQNOzGUoxVV9nCwm6D0GXEIBKrbkelmq9iujuv7kVWyMK6Ts231Edb YVvR2gjTzLsbn0Q+lfjMPyyI+5zBiaW1O2i5MchH0XDQ55IHTTSCwFH6ieItnJuGQp7fi7tu+/GN5 wX9IcGnvFScig9olL/8d3WkbixeFOzof4Lob210ZHsKPY8G5I2mA0NlfIbgotZuG2zmLTz0VsBMwn +h+J8Oqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9pm-00000006IvZ-0HL9; Wed, 14 Aug 2024 08:58:22 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9nz-00000006IBz-3FIm for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=FOU7P9jn4wor6Rdt5JEa4HOxmny7NogNXhx9YRYE+tY=; b=UZgGkGl8P543Z/ah/97iq6ugIc HrpmZ0bkcMh2W8dCTYD046OMai7wAVe7aJD7pOgCm5frGWYU0irWvhhL/1++DWniTl+Y0vr4j/8O6 JJ/lUTDOPPC9bKuiB+wq52ZS7en9K/KF1eRXKxBXi4ki0IWjBbRKaqjl4Zl5lMOdYEayofqkHy77U 6fgH46h+8XGyx88nO+iVfyrrl3FEj3OHsVHPztXCB6qgS2xGdJFh+fhcv9pp120wk+LdD5u/tE3Ap zSYI6dn0WLWTGcYNJXhTMPi9FHWig2ALYjon2IZgSUb1UKb5y7jgM+KFQj3BXpVAn4KFSP2255sMB jozGrHig==; Received: from mail-pl1-x632.google.com ([2607:f8b0:4864:20::632]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9nv-00000007zAH-128Q for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:30 +0000 Received: by mail-pl1-x632.google.com with SMTP id d9443c01a7336-1fc56fd4de1so5203675ad.0 for ; Wed, 14 Aug 2024 01:56:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625784; x=1724230584; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FOU7P9jn4wor6Rdt5JEa4HOxmny7NogNXhx9YRYE+tY=; b=kt5xOkVkE27hap23SerKmwx8Libw7g19aEGHTXujuxLPKmPqzcKfN5eWEEgcnMP2Sx hRsYhGGfa790ay2ijEr7ye/vdEd0uEDnh5ycxEO3Uarj2PMFoDbOOHcDC8UfdSspfM3/ QRNdfH8ZhwLdg10Lv37QxPegrtmkBh/wZE1vl5C9AleDLCW1qlgTOqulZAF2F1YGU9TP lw/5WtnKfe6MRfR8ES4nGbWiRsjuSnrJ7sFKVL3Jow0EEwCMoTXvSZU/FezBdp1tr0Hq R4PRUDFQp8FJbA3U1hVGf/lEudtoFJ/zatso7rYNqC9Ev/OsBONZhGuh46eBpPZrnUQN 1/ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625784; x=1724230584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FOU7P9jn4wor6Rdt5JEa4HOxmny7NogNXhx9YRYE+tY=; b=tE73mMIONQRdOt3uy4e58ICCbUk0MwWCG5O+CcvNxicksYMOaWzQavoz3ROfQ2kQmU l8IzJilSIW5Cnex0MlCkOdOeWcFjoVnDrt6eGhJGgW4OfGDEo1FH138gBoyO4pButD++ Si5UFBNdh5aHcLSQN67vxSxQb3RBS+oocMSTxBr2IEbnTz9YIR2Uc6zw7wbLQmswxmfM eZAQlc04DPFPKHjRglby99TFlPFjZH5VDP5R5Sjv6Uh9pfu0IRqyGFc9WL0knnHwFMM8 wiCeFuWuiR9lyNgRzJQGP2SJSB07YcC9Zyt0ezQzRMnrUFNcdnaNsXhogqLdmXzThBzK WWxA== X-Forwarded-Encrypted: i=1; AJvYcCUrBumcMzBD4bYk75CMF8pzm/brs4+epddfJiDxzE78uIPsrWlGPEJLt19X3xZSUNCGNSpWRUoVzthlrajA9Hn7xGQTSQVnSbRvt0GLrhyYd4fhuYw= X-Gm-Message-State: AOJu0Yxha/QoJ7MlLfVhKh/Bivul8moJpQYvsGnZFkZ45uIYRGGk4zJm JyyBgrt4PODXtD8nuvc3DYmDdFhrogZu1+NxZsE3aq1cJ8fXLk+J++5maV/Lng8= X-Google-Smtp-Source: AGHT+IHtUXL6qE88NcA742NNnvRZ3OwS1PEsV1FbKiWQihqr5xxMbeis8+zR2wt/Qw0zy68klcF0mw== X-Received: by 2002:a17:902:e741:b0:201:dc7b:a88f with SMTP id d9443c01a7336-201dc7bae70mr15567155ad.25.1723625783955; Wed, 14 Aug 2024 01:56:23 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:23 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 2/7] kasan: sw_tags: Check kasan_flag_enabled at runtime Date: Wed, 14 Aug 2024 01:55:30 -0700 Message-ID: <20240814085618.968833-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095627_633499_9BE0979C X-CRM114-Status: GOOD ( 16.23 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On RISC-V, the ISA extension required to dereference tagged pointers is optional, and the interface to enable pointer masking requires firmware support. Therefore, we must detect at runtime if sw_tags is usable on a given machine. Reuse the logic from hw_tags to dynamically enable KASAN. This commit makes no functional change to the KASAN_HW_TAGS code path. Signed-off-by: Samuel Holland Reviewed-by: Andrey Konovalov --- include/linux/kasan-enabled.h | 15 +++++---------- mm/kasan/hw_tags.c | 10 ---------- mm/kasan/tags.c | 10 ++++++++++ 3 files changed, 15 insertions(+), 20 deletions(-) diff --git a/include/linux/kasan-enabled.h b/include/linux/kasan-enabled.h index 6f612d69ea0c..648bda9495b7 100644 --- a/include/linux/kasan-enabled.h +++ b/include/linux/kasan-enabled.h @@ -4,7 +4,7 @@ #include -#ifdef CONFIG_KASAN_HW_TAGS +#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) DECLARE_STATIC_KEY_FALSE(kasan_flag_enabled); @@ -13,23 +13,18 @@ static __always_inline bool kasan_enabled(void) return static_branch_likely(&kasan_flag_enabled); } -static inline bool kasan_hw_tags_enabled(void) -{ - return kasan_enabled(); -} - -#else /* CONFIG_KASAN_HW_TAGS */ +#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */ static inline bool kasan_enabled(void) { return IS_ENABLED(CONFIG_KASAN); } +#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */ + static inline bool kasan_hw_tags_enabled(void) { - return false; + return IS_ENABLED(CONFIG_KASAN_HW_TAGS) && kasan_enabled(); } -#endif /* CONFIG_KASAN_HW_TAGS */ - #endif /* LINUX_KASAN_ENABLED_H */ diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c index 9958ebc15d38..c3beeb94efa5 100644 --- a/mm/kasan/hw_tags.c +++ b/mm/kasan/hw_tags.c @@ -43,13 +43,6 @@ static enum kasan_arg kasan_arg __ro_after_init; static enum kasan_arg_mode kasan_arg_mode __ro_after_init; static enum kasan_arg_vmalloc kasan_arg_vmalloc __initdata; -/* - * Whether KASAN is enabled at all. - * The value remains false until KASAN is initialized by kasan_init_hw_tags(). - */ -DEFINE_STATIC_KEY_FALSE(kasan_flag_enabled); -EXPORT_SYMBOL(kasan_flag_enabled); - /* * Whether the selected mode is synchronous, asynchronous, or asymmetric. * Defaults to KASAN_MODE_SYNC. @@ -257,9 +250,6 @@ void __init kasan_init_hw_tags(void) kasan_init_tags(); - /* KASAN is now initialized, enable it. */ - static_branch_enable(&kasan_flag_enabled); - pr_info("KernelAddressSanitizer initialized (hw-tags, mode=%s, vmalloc=%s, stacktrace=%s)\n", kasan_mode_info(), kasan_vmalloc_enabled() ? "on" : "off", diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c index d65d48b85f90..c111d98961ed 100644 --- a/mm/kasan/tags.c +++ b/mm/kasan/tags.c @@ -32,6 +32,13 @@ enum kasan_arg_stacktrace { static enum kasan_arg_stacktrace kasan_arg_stacktrace __initdata; +/* + * Whether KASAN is enabled at all. + * The value remains false until KASAN is initialized by kasan_init_tags(). + */ +DEFINE_STATIC_KEY_FALSE(kasan_flag_enabled); +EXPORT_SYMBOL(kasan_flag_enabled); + /* Whether to collect alloc/free stack traces. */ DEFINE_STATIC_KEY_TRUE(kasan_flag_stacktrace); @@ -92,6 +99,9 @@ void __init kasan_init_tags(void) if (WARN_ON(!stack_ring.entries)) static_branch_disable(&kasan_flag_stacktrace); } + + /* KASAN is now initialized, enable it. */ + static_branch_enable(&kasan_flag_enabled); } static void save_stack_info(struct kmem_cache *cache, void *object, From patchwork Wed Aug 14 08:55:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22AFFC52D7F for ; Wed, 14 Aug 2024 09:01:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Kr1OlloTvp35mKilBnuEZzypgigPXmCly4KSd56Q7OI=; b=RLNl11Z7Vm9kXGYaU88Ft6Dkm0 wRn35SrLFzqNMPDAPnuoLexRr/6Kg3VVBHZHG0+F/B9gG/u7kywRD2R0kc4/I28bDzSv6aazC0Ntz irHRhxx5xjoZPteNVIH2CIoZoAksfupuwCUbdNg94Ef2TV+A/ATMNYSrMxRHNQxQrUHkq+gHovzvO do+xLi3FhK0OKSvBIOB/8tmz5lBTrZpe3MAtlzYwPEwHnQ2YTx0O1LpNNdsK6SMrK7zm7luhUqT/m 9nrrVWAIOc75OOHQXK47fWmN9+1Ljs07dcUILpdMoZ3KLw+fJTpIY9yMXmuV/v64hqYA0dIoJifHi 6L/LnLyA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9sj-00000006Jhn-2Cxr; Wed, 14 Aug 2024 09:01:25 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o1-00000006ICr-3Hgf for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:33 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Kr1OlloTvp35mKilBnuEZzypgigPXmCly4KSd56Q7OI=; b=B1/KZEJx5Vr9fQtj4hIEeBiQz/ MUtnuVJeVMn0gvOrd2JhcLC5dZjZheuDF8eNs7bWXtb3w5xpos45heBYGrtMuubxJ4cG/SC7hZSSF rE5mGI5xEQuKJj0Z/sFZalhytdN88fxO0la7GDPHLFyYm6tIxs4vrRiV/yeG+IvKap6GVkVgty3kT zIzK1DET4MrcQJsNHAucku42wKCFPUPCVZUEqf8znceVh3WGcuNO+iMc8+UrDi1uViGOfr1E1FezW vxeN4CIACo3pEPMKGbPAaiYHOWVY2PjeiyshJ/uXsqPBAYTviLZt523+lgoOPWHtwBhNGt8bR29SI ZRpXCxVA==; Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9nw-00000007zBB-3zCN for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:32 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1fec34f94abso55493755ad.2 for ; Wed, 14 Aug 2024 01:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625786; x=1724230586; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Kr1OlloTvp35mKilBnuEZzypgigPXmCly4KSd56Q7OI=; b=iEddrKr4i1PnAor6PaBO4WFxY8fFLVHe1ZAXi4dMJCW55fBHSFIoCntBILB+IGoKix 5otW38RKyh9z7afkeJpvChRdCo+ZWd38V4an1oO3e6k6wJteeS8BluSdhtEEFEt+dw/i koODw7tZSe3/OdcMsR+TeXCwGmLsqmcmTnSNBU1HY9bQgMgdTM0cm3gMvWxY+dy5F8iW VAyO0aRYiNBY38GEie3eIzMRfYaFfuI7VvkjUQaAOQ+VqgHS0AsLZKWYSzvkodyvbqcm 12X5UPZLL86klSW+RLpCPhqOywd/79fd5ZJgvVWoyZRKgvAE11OBe+yPa4f/LtASEoVB Lffw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625786; x=1724230586; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Kr1OlloTvp35mKilBnuEZzypgigPXmCly4KSd56Q7OI=; b=neGLFcyBhThYQyUlhVS1V8f1OppkOgZL5MLEEEpkD7i8O/VkJX2gNGt6iM2i7tcy6d mHxWhU5s4OsC75KRrSVdCJXsuXu8Z25ZLe1p4P1ow4ztV3HYn2yEInQelu6JTcB6i4s4 lWDbdFSKVEFfMQPm6KcTU9Ar5gBeF860DjBv78vH1oDvD+2mrzYV59COBZG+ODwDkJyM KyKDbAkJQE91KnnKZJmfSC1UwOxZ8skNaj25hX8QUpCuidgZ9egxhKt+HOq9WnBMZfnt EW+YVexHRFd4YZAo/7hicOl1C/LxwK7F8kLi4BWyhAsO2jlwO75NhfKCOMS/3Vcemvis 3+5Q== X-Forwarded-Encrypted: i=1; AJvYcCXJ4T1fvQ+MVBUUdE5QoAmETktsQ59H4A7iImYKj2+WFPd8I2n0KeT7sPmyRrm+SQH9iFSSceVH1NAIqzihgswqTw3P/kIZTeXYtB04McndnVVvCrA= X-Gm-Message-State: AOJu0YyY1a+seN/IycazobsoHLz5+VjonoTkFUCbryp+VK5igspzg/st IydpO+HKRRx/dXgnU+3jsvCYalo8PS9x1wbWVd28/IIfMo48n0BrR8ajEI11yIo= X-Google-Smtp-Source: AGHT+IE20LewatO25ueMR3XW0aRNE3Ho00UvK46l5CUL7vUBX+0A3K1zUsQysbjVk0qPCHhMgq/ydQ== X-Received: by 2002:a17:903:360d:b0:1ff:49a0:46b1 with SMTP id d9443c01a7336-201d6397db6mr25518915ad.6.1723625785721; Wed, 14 Aug 2024 01:56:25 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:25 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 3/7] kasan: sw_tags: Support tag widths less than 8 bits Date: Wed, 14 Aug 2024 01:55:31 -0700 Message-ID: <20240814085618.968833-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095629_065287_9C49F28F X-CRM114-Status: GOOD ( 12.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow architectures to override KASAN_TAG_KERNEL in asm/kasan.h. This is needed on RISC-V, which supports 57-bit virtual addresses and 7-bit pointer tags. For consistency, move the arm64 MTE definition of KASAN_TAG_MIN to asm/kasan.h, since it is also architecture-dependent; RISC-V's equivalent extension is expected to support 7-bit hardware memory tags. Signed-off-by: Samuel Holland Reviewed-by: Andrey Konovalov --- arch/arm64/include/asm/kasan.h | 6 ++++-- arch/arm64/include/asm/uaccess.h | 1 + include/linux/kasan-tags.h | 13 ++++++++----- 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kasan.h b/arch/arm64/include/asm/kasan.h index e1b57c13f8a4..4ab419df8b93 100644 --- a/arch/arm64/include/asm/kasan.h +++ b/arch/arm64/include/asm/kasan.h @@ -6,8 +6,10 @@ #include #include -#include -#include + +#ifdef CONFIG_KASAN_HW_TAGS +#define KASAN_TAG_MIN 0xF0 /* minimum value for random tags */ +#endif #define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) #define arch_kasan_reset_tag(addr) __tag_reset(addr) diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index 28f665e0975a..56a09f412272 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -22,6 +22,7 @@ #include #include #include +#include #include #include #include diff --git a/include/linux/kasan-tags.h b/include/linux/kasan-tags.h index 4f85f562512c..e07c896f95d3 100644 --- a/include/linux/kasan-tags.h +++ b/include/linux/kasan-tags.h @@ -2,13 +2,16 @@ #ifndef _LINUX_KASAN_TAGS_H #define _LINUX_KASAN_TAGS_H +#include + +#ifndef KASAN_TAG_KERNEL #define KASAN_TAG_KERNEL 0xFF /* native kernel pointers tag */ -#define KASAN_TAG_INVALID 0xFE /* inaccessible memory tag */ -#define KASAN_TAG_MAX 0xFD /* maximum value for random tags */ +#endif + +#define KASAN_TAG_INVALID (KASAN_TAG_KERNEL - 1) /* inaccessible memory tag */ +#define KASAN_TAG_MAX (KASAN_TAG_KERNEL - 2) /* maximum value for random tags */ -#ifdef CONFIG_KASAN_HW_TAGS -#define KASAN_TAG_MIN 0xF0 /* minimum value for random tags */ -#else +#ifndef KASAN_TAG_MIN #define KASAN_TAG_MIN 0x00 /* minimum value for random tags */ #endif From patchwork Wed Aug 14 08:55:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763218 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3BD6CC5320D for ; Wed, 14 Aug 2024 10:00:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=q+LcB4MC6Qgfr9kzwjEWl/9KLwOZpaXC6CA7cxVGJTI=; b=gbCw30DysIEV+S+7XCMMcZ19XR XoWA0p6phHL/l/ddq+GY05XOcd6PSOd3CaNNwn+SfO/VHaTXFMeIoMlYb1a059WVAgOftNvWH1PaN 5+Jk1AYlhbtcPighOkocHPQGn7Ekkz9+DkxnmhZMlgJ0BGJyA8YVCBNc9s3/jfaDkA+AQnDeozqSp C5wlDzPfc26yMS0OxT6a9YfZId8YmnBvF+5S/ASJRSngdw25Nt1g2vfVawi5sY3oPBHSZkZz3POq2 bZTCaCj8xKCyteFrywdKOdMBO32wJ4PXPJZSUPrTTjhJQbPgHpMC4bBpmsr4DDyzUQBNZy5Dq344Z +6W6XsnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1seAoA-00000006U5P-23Q8; Wed, 14 Aug 2024 10:00:46 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o3-00000006IDX-22p0 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:35 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=q+LcB4MC6Qgfr9kzwjEWl/9KLwOZpaXC6CA7cxVGJTI=; b=KN5LzflrBaJ9GYiwhUZxej494k 2302nnNLkdTOPr5+lPaF278IFyyaH/y3/7EOdBpgts5Lxw03DQUZYzX9LfD1gGr/zzPwGsjm+pxte 4JgYz1XehqMhsRDD7qs/2UtRLyoifkpp3Plm1F5r23sPoq8xk9R26SaUULGUg/k3yGio8WY00tqUI QKO0x5BPJyWsKVF8h2Y5NywckB+hTILjXbZsJAy3HJ4IJmC+ET3hPw7LfgyWNFaJ190vBGdFnb+IC lvrBoYVsw6Jp/d2LefKwzR4u5/NU4aBeTSJo1yrcoxKykVLjWyqUCtA3oTOLtQ7RNgySsFAq/MgJb PszFHwuQ==; Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9nz-00000007zDR-3Nv3 for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:33 +0000 Received: by mail-pg1-x529.google.com with SMTP id 41be03b00d2f7-76cb5b6b3e4so4216008a12.1 for ; Wed, 14 Aug 2024 01:56:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625787; x=1724230587; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=q+LcB4MC6Qgfr9kzwjEWl/9KLwOZpaXC6CA7cxVGJTI=; b=YhSDj7gRNo1/FoN5CSeSwviOQxYfDKhu40XOpvup/HlXAz2IO8+8jrYrDbIN7tKX6H rZy0pG9o2SgpxDugw/bUeoEK/N+eLHdX4LAmlbrm3HFMcMbztaeZnPk4tWbTBNBUzhCg Ewa06QrC3e1XpEtmX2O45DPpPuEvizmqkvAAfk3pM5UtdNNquObl8MpKFYfsFv2EGbKJ s3rdSri5A7jGW+1FO/9kqIO5xEIr8MfiRNttwZscPAtJdJiIAhqnj+c/JjbRcj6Jw/jZ vvl3tDEsXj/vJUebG8U/IB//rzxqQ6SAxNy2ZWaHJV7IDiP+2/rYVlV4K5MCYMXERwvo qh5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625787; x=1724230587; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q+LcB4MC6Qgfr9kzwjEWl/9KLwOZpaXC6CA7cxVGJTI=; b=NbRdkI/4sXLpx14pP1jSnpWsLIWcHwHfUVHH6cPwdWL4ccGyd9hEAU7VfMk7jHhbvB tY8qojLL7VqFfKNvJtjTyxgyrRdqpvoRQg4ydIcNbHGsTdHgnOuMOJJOdK8MBWP5UHtf d9zpLlfkRNASGvS/edhN2FQEPNFF+83z3PbD+EaXlIkd1l3HZAuwpBNwnGgMPmrsF+qi cfF7KpNA7noL9JU6AvwZH0EVDnblsm97OAaZJ/Q4aoXtGwPo5mJKHjSsteEwtLH9UxnL XjFM8UAGqAWja153S/vNdX2eAislM1DmR2y3tU0Cp0T/qZp0GfzP4BmxVJwKcXHxwq1Z vraQ== X-Forwarded-Encrypted: i=1; AJvYcCVuzLd/8KQxUfI0sxwrZshlBrHmKU8sqCTMjmXAQ9Ww4GensBuQP5OjMWvHkPKGDwzsHUH8hncHjwiE4qYsY7d/55h5Cw8yX88yOpKIsIFp13BwhPw= X-Gm-Message-State: AOJu0Yx/jPGzov2F/hN9D5BZEXO/oKF/ysbNamu7icSjTDsgEHTFMoyo WYNlt3wcU84Qs/CgCJUxG6xB1/eMJrXaG4+2w2SkD1xZeOemiWOWPLOOKn5Z0c0= X-Google-Smtp-Source: AGHT+IHeop/4tDtqPSIvDLepGfrszpRVvQ8NysDbzpZjr3VmMUvS3QBFyPVJf7l/syUiJf9jHvayiw== X-Received: by 2002:a05:6a20:c88b:b0:1c4:c879:b770 with SMTP id adf61e73a8af0-1c8eae8dd79mr2680167637.23.1723625787615; Wed, 14 Aug 2024 01:56:27 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:27 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 4/7] riscv: Do not rely on KASAN to define the memory layout Date: Wed, 14 Aug 2024 01:55:32 -0700 Message-ID: <20240814085618.968833-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095632_038292_44486DCA X-CRM114-Status: GOOD ( 17.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Commit 66673099f734 ("riscv: mm: Pre-allocate vmemmap/direct map/kasan PGD entries") used the start of the KASAN shadow memory region to represent the end of the linear map, since the two memory regions were immediately adjacent. This is no longer the case for Sv39; commit 5c8405d763dc ("riscv: Extend sv39 linear mapping max size to 128G") introduced a 4 GiB hole between the regions. Introducing KASAN_SW_TAGS will cut the size of the shadow memory region in half, creating an even larger hole. Avoid wasting PGD entries on this hole by using the size of the linear map (KERN_VIRT_SIZE) to compute PAGE_END. Since KASAN_SHADOW_START/KASAN_SHADOW_END are used inside an IS_ENABLED block, it's not possible to completely hide the constants when KASAN is disabled, so provide dummy definitions for that case. Signed-off-by: Samuel Holland --- arch/riscv/include/asm/kasan.h | 11 +++++++++-- arch/riscv/mm/init.c | 2 +- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h index e6a0071bdb56..a4e92ce9fa31 100644 --- a/arch/riscv/include/asm/kasan.h +++ b/arch/riscv/include/asm/kasan.h @@ -6,6 +6,8 @@ #ifndef __ASSEMBLY__ +#ifdef CONFIG_KASAN + /* * The following comment was copied from arm64: * KASAN_SHADOW_START: beginning of the kernel virtual addresses. @@ -33,13 +35,18 @@ #define KASAN_SHADOW_START ((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK) #define KASAN_SHADOW_END MODULES_LOWEST_VADDR -#ifdef CONFIG_KASAN #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) void kasan_init(void); asmlinkage void kasan_early_init(void); void kasan_swapper_init(void); -#endif +#else /* CONFIG_KASAN */ + +#define KASAN_SHADOW_START MODULES_LOWEST_VADDR +#define KASAN_SHADOW_END MODULES_LOWEST_VADDR + +#endif /* CONFIG_KASAN */ + #endif #endif /* __ASM_KASAN_H */ diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 8b698d9609e7..1667f1b05f5a 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -1492,7 +1492,7 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon panic("Failed to pre-allocate %s pages for %s area\n", lvl, area); } -#define PAGE_END KASAN_SHADOW_START +#define PAGE_END (PAGE_OFFSET + KERN_VIRT_SIZE) void __init pgtable_cache_init(void) { From patchwork Wed Aug 14 08:55:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1678EC3DA4A for ; Wed, 14 Aug 2024 09:00:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=BtxoKtjWvRqhjS0z2DQnl5gBuyvoRF+9CVzVIMvmZ6E=; b=niYI9+BCRmhFwxMNH1bEMmC04k HGHTLs1nCB/UCd3BNMU3qI2d/QHTkeOnbHKg2tEYP6cbg6qmy9uvhE1H/SlHqzT9c70/wYVsAyQWZ SB5Vpetk4n0DXIBYQkLtSZRuI/8ZX7jaoCBOBRcZDK89kAn7yVaZfmn1IM6xceNvu8XPYwB+Dqovi p6pLtaq6/TqwpfplMu3SpqVu3aSzhvqK+AjmtJ0ZvPHJsfwld8nFOTmBvGbsofM5Sg3CASl4XUKnb 6+vFz1HFXmBKWuyWRaBrbnR9xUTg3ZPWOEoKSMtAMezrtTlXUneMRnZnn12i5LlfMqYogcv646GoI mJ0z5PeA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9rK-00000006JRI-2Gbx; Wed, 14 Aug 2024 08:59:58 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o4-00000006IDm-0zRX for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=BtxoKtjWvRqhjS0z2DQnl5gBuyvoRF+9CVzVIMvmZ6E=; b=c7DQgCem7J72kvVSvWKD1LBVpk O7Pf1Y2CraW93TuZRTl5KXPsbtKguuTVaxOgBS7FrZTKb0TxePGQ/i9hjCpVn95CzDwzdaaFHQ94p iZbmY/ymCM6d7cYbziZYKuay/xVRWKSAfgRmAtxflsQClZ+Uqh4bdVLSQjhFLVtyZ5smP9a1VzDIx NVzlZsV/zYRqQdhR17f2qyhNdm0m6TMFyk4SvCjroNSaP9hjoEDMWN6BOxomfUEd6Zba1YDLer1CV TzbtLGfn2cMWSFttNudOHm1j6anaaXDuoAD4aPosjR1g/IZpSdGPctMyyJcatf9Zb2YdHDGL1+79/ YdPqU10w==; Received: from mail-pg1-x52e.google.com ([2607:f8b0:4864:20::52e]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o0-00000007zDr-3FZt for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:34 +0000 Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-7ab09739287so3936597a12.3 for ; Wed, 14 Aug 2024 01:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625789; x=1724230589; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BtxoKtjWvRqhjS0z2DQnl5gBuyvoRF+9CVzVIMvmZ6E=; b=i1zzzHt2msT2iFsRD8CrqNjzDSSwTxFgCBfaHcpzeZ632kl8SWGu0RjXLbNi/gyijw fXJ60UUr9Vh5ZVrFPooy01r5CQLype1DKjCrjqGnT2cMKqiA+cO6DkWTKWu9h+oomIuo ioSic6AX4lD4FPzSYJ38QniI8FDewSAqZ4WKYj3tW873tcrp5wNuKl9CjcAn18/pfQg7 1uFLNn5ug+LjJTmi+n2CmjQ8whMisDUelKcu9C1XP8oAtSWQNynE7HhOiqzFZJ482rAG mhQ+uJjvemXVszaX05THega/UZq/d4hTZpB22ZEkJ7zT5lcSvjcyEeD32sZukRPPrgsS RGbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625789; x=1724230589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BtxoKtjWvRqhjS0z2DQnl5gBuyvoRF+9CVzVIMvmZ6E=; b=bP9t9c9hGHdjIWxBkFAvDdMNRkq4DES7qQLranAzpYPE8pMVYpJ1z3seIjuy7bqxgR S2ghrd02QdnBUMqMXuEoV5UK6JTQqqerxgn8dsos5bqTgHPiMOmKYlOpdecnp7N4Hek2 A8piTIqpyyBCAIUKm1kNLZON0TvEH7jZcd6WNvKgeKTOUriGnebUEVlmUj2HFUxfTxMG tNiElxq+VZWlD6UCIXTAn2W8UpvHc+P56FueDZhKIYqMqoipqeSTZ+aodlsS0hk6AcVA FQ03tq7kMJAyElSG7gqPnVgxaMhI+SyRSB+HOJA9zqPFWh62LqMZAvPJn8l8CoQOyk+L 8TFQ== X-Forwarded-Encrypted: i=1; AJvYcCXtlehFj1u033UiUHFqFD1X1oHNR76unbJKni57sc7DaM/rE6EGWrhpgeU0Fj9tuFI7XhdPtPRzJ8o4EkB40eJGqBR8V+vDIx9RvqJpnvWS7yUgkIs= X-Gm-Message-State: AOJu0YydE+6nSh8ZDdnWE3Znxz+X/wa/FtVfAERMl86MFnTPfWpSMagm odEV7sz8UHlw2MbU0xhOOfgJkH3TPJzaDRrZGxHAHA7Bsc6MuCwJq3wFZz4rGQA= X-Google-Smtp-Source: AGHT+IGKwvtrSq7d8JlTCA95HrODYZub/jNh1CadsEDvmYgd02vgRRSnG6ohafRUbcvHgfRX7yetxA== X-Received: by 2002:a05:6a20:4f25:b0:1c8:edba:b9ca with SMTP id adf61e73a8af0-1c8edbabc7fmr1261808637.1.1723625789313; Wed, 14 Aug 2024 01:56:29 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:28 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 5/7] riscv: Align the sv39 linear map to 16 GiB Date: Wed, 14 Aug 2024 01:55:33 -0700 Message-ID: <20240814085618.968833-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095633_096191_195F3A3B X-CRM114-Status: GOOD ( 14.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The KASAN implementation on RISC-V requires the shadow memory for the vmemmap and linear map regions to be aligned to a PMD boundary (1 GiB). For KASAN_GENERIC (KASAN_SHADOW_SCALE_SHIFT == 3), this enforces 8 GiB alignment for the memory regions themselves. KASAN_SW_TAGS uses 16-byte granules (KASAN_SHADOW_SCALE_SHIFT == 4), so now the memory regions must be aligned to a 16 GiB boundary. Signed-off-by: Samuel Holland --- Documentation/arch/riscv/vm-layout.rst | 10 +++++----- arch/riscv/include/asm/page.h | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/Documentation/arch/riscv/vm-layout.rst b/Documentation/arch/riscv/vm-layout.rst index 077b968dcc81..ed71b3b1b784 100644 --- a/Documentation/arch/riscv/vm-layout.rst +++ b/Documentation/arch/riscv/vm-layout.rst @@ -47,11 +47,11 @@ RISC-V Linux Kernel SV39 | Kernel-space virtual memory, shared between all processes: ____________________________________________________________|___________________________________________________________ | | | | - ffffffc4fea00000 | -236 GB | ffffffc4feffffff | 6 MB | fixmap - ffffffc4ff000000 | -236 GB | ffffffc4ffffffff | 16 MB | PCI io - ffffffc500000000 | -236 GB | ffffffc5ffffffff | 4 GB | vmemmap - ffffffc600000000 | -232 GB | ffffffd5ffffffff | 64 GB | vmalloc/ioremap space - ffffffd600000000 | -168 GB | fffffff5ffffffff | 128 GB | direct mapping of all physical memory + ffffffc2fea00000 | -244 GB | ffffffc2feffffff | 6 MB | fixmap + ffffffc2ff000000 | -244 GB | ffffffc2ffffffff | 16 MB | PCI io + ffffffc300000000 | -244 GB | ffffffc3ffffffff | 4 GB | vmemmap + ffffffc400000000 | -240 GB | ffffffd3ffffffff | 64 GB | vmalloc/ioremap space + ffffffd400000000 | -176 GB | fffffff3ffffffff | 128 GB | direct mapping of all physical memory | | | | fffffff700000000 | -36 GB | fffffffeffffffff | 32 GB | kasan __________________|____________|__________________|_________|____________________________________________________________ diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 7ede2111c591..09d15567b0b8 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -37,7 +37,7 @@ * define the PAGE_OFFSET value for SV48 and SV39. */ #define PAGE_OFFSET_L4 _AC(0xffffaf8000000000, UL) -#define PAGE_OFFSET_L3 _AC(0xffffffd600000000, UL) +#define PAGE_OFFSET_L3 _AC(0xffffffd400000000, UL) #else #define PAGE_OFFSET _AC(CONFIG_PAGE_OFFSET, UL) #endif /* CONFIG_64BIT */ From patchwork Wed Aug 14 08:55:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E118C52D7F for ; Wed, 14 Aug 2024 10:00:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=xswnBLwHxJrHYsJkrECwj9jBmYSXx50wzmQACaoOJYE=; b=AVfRA8huqunjY5Si0sLTgcBZi9 qb/VxcrC5bliMGui30JhPkbXjSUqIqTI4rjdBA3XMTWq82I1f26yynOOtUNXW0dpVH4/SCuYX0RQ5 IgG1Xi4WII41l6mntIJ/uNpIs+udO/FRSJ+yr7DsmJoFtl50RLHDkStXfom7wwW51XlzMnw0MDTS6 KxhKIKW6a38abgBZoDZw7I17iIYBpFDbV35QX37BXpuaNi0YGduI9tqQeZ+ERL5bxzwMvEDF2enav saSU2/6X/g5vr0qt1IULxhE5/s2vS7OJvrluTF2JUKkYG79RtKGhWXNxi06JPc2AVQbKQTtv6zhYF I16SDMYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1seAoB-00000006U6h-1gKQ; Wed, 14 Aug 2024 10:00:47 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o4-00000006IDs-1alp for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xswnBLwHxJrHYsJkrECwj9jBmYSXx50wzmQACaoOJYE=; b=YMYn+XSnvMXvce7FNykqwoIq5i 7I3S7T7AXBpiFG9wgSaUJCfiyjWBQp3D7DpdDmbP71hWlr9zwnJ1QjmePD2nhAFrF0OFCfsd/9dvd r8i8daAPczp4CN99HjpT4enb9qzB1IgiV4ClnZzoXfU2n8fwBeKRyWIcfGCe68DRvyVS2O2+wWQe5 allBzEMcpj+hm9DyMmDNAWONap/Z004lcV6w807kq7duAl6RFZ4BEfF8h0qeG09tLw6QLGW8GPyMu WcpWah1NgipBjA+v8NDKWgwNpCT8R4FVjnzawffXBLI9Hm1TvUfNu2EKRqSt86u+3ryadqwuM7Q08 48ET4gnw==; Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o0-00000007zEe-2uGK for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:35 +0000 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-7163489149eso4458499a12.1 for ; Wed, 14 Aug 2024 01:56:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625791; x=1724230591; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xswnBLwHxJrHYsJkrECwj9jBmYSXx50wzmQACaoOJYE=; b=Jt3O7M7cRIdUhne0U+zyWU4/Qzi3y8wQBG++2mcesu4+nI8S/MioXMFuxVltbDYKl2 2OKvsrWtBzxwzmS7r7Ue0vhDThI2jRU7hK6igCEvdL1hW7UbeH8E35Q6NxpwBJJWu+wO 75beyUUqxefEyzctrUqF8X7Ygrl1ytw77R3LObKq0pGJPemV56WpsyjHvkokzA/cbeUd vr5pIRyCpCgghBHLpMZGgw2fVIiqUEJ50uXik6iKOoz2VXip+aj/aV5nJiw1xWoH/J31 4/MMBLvx+foDCGnLYtdGK9B+BSvKvNkU8S1ihEAHl45rpm0PmRlsIIi7kVmhJXdXinNC DBJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625791; x=1724230591; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xswnBLwHxJrHYsJkrECwj9jBmYSXx50wzmQACaoOJYE=; b=P7Sxo77ZfU4d7tPq8NvwYZ1aHd5YCkKee2y3GEwAZFOzh6GSOi2pwFLUz4DkWrZIcx qkrN2TsPbe/CqR8kTgH2dmdXk/nUzOOXepJRali8HFsVCD3JoR2F7i2fPSMSkioPOEJa I8W7UoPAIARfMm26xJCOXFZqq/Xq0HQ4Y6flwUGeUeFfbjK4Xc1IheBHPamw18R6TvT8 63WZXgrHf4ufvTpvkEUOKYAKMGYbpdbZZ5yBdSqEHV6iJj9NNVpoip+b/fLDxVuqdPvy cvpgC3XDP6UcKFhklCgTi6wITpY7N6tlvgb+d/WfmQ0AQ0xq8Sk7ZASGljUV3HZlTQU3 5NtA== X-Forwarded-Encrypted: i=1; AJvYcCUtgNeL4AOWftFkN+We6a/gOLk+rqCUTCULlD4rgEuBqpnbT7dIHft/9+x3Ao1+7ydYQD5/1C/6YUBbhaoSKuNP@lists.infradead.org X-Gm-Message-State: AOJu0YxpL184LjXOHXmVphUhAGJhI5qA6RrLxsBtqXJzDWwbuMZDltZE /PnJlfpMv1fHpuxxGUSvFwi6NlpNMq78h545OkuQ/987CEFoJWur09Q5SFtq1ww= X-Google-Smtp-Source: AGHT+IHCL/FoPksetPP9uSrL57foa2RYf5VItY5DJy2ZZShxFbar/elQ054Ynd2YhHuu4Nhm6f2xQQ== X-Received: by 2002:a05:6a20:9e4b:b0:1c4:bde5:174b with SMTP id adf61e73a8af0-1c8eaf7db28mr2708359637.41.1723625791103; Wed, 14 Aug 2024 01:56:31 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:30 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 6/7] riscv: Implement KASAN_SW_TAGS Date: Wed, 14 Aug 2024 01:55:34 -0700 Message-ID: <20240814085618.968833-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095633_213586_5DF57338 X-CRM114-Status: GOOD ( 34.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement support for software tag-based KASAN using the RISC-V pointer masking extension, which supports 7 and/or 16-bit tags. This implemen- tation uses 7-bit tags, so it is compatible with either hardware mode. Pointer masking is an optional ISA extension, and it must be enabled using an SBI call to firmware on each CPU. This SBI call must be made very early in smp_callin(), as dereferencing any tagged pointers before that point will crash the kernel. If the SBI call fails on the boot CPU, then KASAN is globally disabled, and the kernel boots normally (unless stack tagging is enabled). If the SBI call fails on any other CPU, that CPU is excluded from the system. When pointer masking is enabled for the kernel's privilege mode, it must be more careful about accepting tagged pointers from userspace. Normally, __access_ok() accepts tagged aliases of kernel memory as long as the MSB is zero, since those addresses cannot be dereferenced -- they will cause a page fault in the uaccess routines. But when the kernel is using pointer masking, those addresses are dereferenceable, so __access_ok() must specifically check the most-significant non-tag bit. Pointer masking does not apply to the operands of fence instructions, so software is responsible for untagging those addresses. Signed-off-by: Samuel Holland --- Documentation/dev-tools/kasan.rst | 14 ++++--- arch/riscv/Kconfig | 4 +- arch/riscv/include/asm/cache.h | 4 ++ arch/riscv/include/asm/kasan.h | 20 ++++++++++ arch/riscv/include/asm/page.h | 19 ++++++++-- arch/riscv/include/asm/pgtable.h | 6 +++ arch/riscv/include/asm/tlbflush.h | 4 +- arch/riscv/kernel/setup.c | 6 +++ arch/riscv/kernel/smpboot.c | 8 +++- arch/riscv/lib/Makefile | 2 + arch/riscv/lib/kasan_sw_tags.S | 61 +++++++++++++++++++++++++++++++ arch/riscv/mm/kasan_init.c | 30 ++++++++++++++- arch/riscv/mm/physaddr.c | 4 ++ 13 files changed, 167 insertions(+), 15 deletions(-) create mode 100644 arch/riscv/lib/kasan_sw_tags.S diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst index d7de44f5339d..6548aebac57f 100644 --- a/Documentation/dev-tools/kasan.rst +++ b/Documentation/dev-tools/kasan.rst @@ -22,8 +22,8 @@ architectures, but it has significant performance and memory overheads. Software Tag-Based KASAN or SW_TAGS KASAN, enabled with CONFIG_KASAN_SW_TAGS, can be used for both debugging and dogfood testing, similar to userspace HWASan. -This mode is only supported for arm64, but its moderate memory overhead allows -using it for testing on memory-restricted devices with real workloads. +This mode is only supported on arm64 and riscv, but its moderate memory overhead +allows using it for testing on memory-restricted devices with real workloads. Hardware Tag-Based KASAN or HW_TAGS KASAN, enabled with CONFIG_KASAN_HW_TAGS, is the mode intended to be used as an in-field memory bug detector or as a @@ -340,12 +340,14 @@ Software Tag-Based KASAN ~~~~~~~~~~~~~~~~~~~~~~~~ Software Tag-Based KASAN uses a software memory tagging approach to checking -access validity. It is currently only implemented for the arm64 architecture. +access validity. It is currently only implemented for the arm64 and riscv +architectures. Software Tag-Based KASAN uses the Top Byte Ignore (TBI) feature of arm64 CPUs -to store a pointer tag in the top byte of kernel pointers. It uses shadow memory -to store memory tags associated with each 16-byte memory cell (therefore, it -dedicates 1/16th of the kernel memory for shadow memory). +or the pointer masking (Sspm) feature of RISC-V CPUs to store a pointer tag in +the top byte of kernel pointers. It uses shadow memory to store memory tags +associated with each 16-byte memory cell (therefore, it dedicates 1/16th of the +kernel memory for shadow memory). On each memory allocation, Software Tag-Based KASAN generates a random tag, tags the allocated memory with this tag, and embeds the same tag into the returned diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0f3cd7c3a436..b963f7cea3b8 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -117,6 +117,7 @@ config RISCV select HAVE_ARCH_JUMP_LABEL if !XIP_KERNEL select HAVE_ARCH_JUMP_LABEL_RELATIVE if !XIP_KERNEL select HAVE_ARCH_KASAN if MMU && 64BIT + select HAVE_ARCH_KASAN_SW_TAGS if MMU && 64BIT select HAVE_ARCH_KASAN_VMALLOC if MMU && 64BIT select HAVE_ARCH_KFENCE if MMU && 64BIT select HAVE_ARCH_KGDB if !XIP_KERNEL @@ -277,7 +278,8 @@ config PAGE_OFFSET config KASAN_SHADOW_OFFSET hex - depends on KASAN_GENERIC + depends on KASAN + default 0xffffffff00000000 if KASAN_SW_TAGS default 0xdfffffff00000000 if 64BIT default 0xffffffff if 32BIT diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h index 570e9d8acad1..232288a060c6 100644 --- a/arch/riscv/include/asm/cache.h +++ b/arch/riscv/include/asm/cache.h @@ -16,6 +16,10 @@ #define ARCH_KMALLOC_MINALIGN (8) #endif +#ifdef CONFIG_KASAN_SW_TAGS +#define ARCH_SLAB_MINALIGN (1ULL << KASAN_SHADOW_SCALE_SHIFT) +#endif + /* * RISC-V requires the stack pointer to be 16-byte aligned, so ensure that * the flat loader aligns it accordingly. diff --git a/arch/riscv/include/asm/kasan.h b/arch/riscv/include/asm/kasan.h index a4e92ce9fa31..f6b378ba936d 100644 --- a/arch/riscv/include/asm/kasan.h +++ b/arch/riscv/include/asm/kasan.h @@ -25,7 +25,11 @@ * KASAN_SHADOW_OFFSET = KASAN_SHADOW_END - * (1ULL << (64 - KASAN_SHADOW_SCALE_SHIFT)) */ +#if defined(CONFIG_KASAN_GENERIC) #define KASAN_SHADOW_SCALE_SHIFT 3 +#elif defined(CONFIG_KASAN_SW_TAGS) +#define KASAN_SHADOW_SCALE_SHIFT 4 +#endif #define KASAN_SHADOW_SIZE (UL(1) << ((VA_BITS - 1) - KASAN_SHADOW_SCALE_SHIFT)) /* @@ -37,6 +41,14 @@ #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) +#ifdef CONFIG_KASAN_SW_TAGS +#define KASAN_TAG_KERNEL 0x7f /* native kernel pointers tag */ +#endif + +#define arch_kasan_set_tag(addr, tag) __tag_set(addr, tag) +#define arch_kasan_reset_tag(addr) __tag_reset(addr) +#define arch_kasan_get_tag(addr) __tag_get(addr) + void kasan_init(void); asmlinkage void kasan_early_init(void); void kasan_swapper_init(void); @@ -48,5 +60,13 @@ void kasan_swapper_init(void); #endif /* CONFIG_KASAN */ +#ifdef CONFIG_KASAN_SW_TAGS +bool kasan_boot_cpu_enabled(void); +int kasan_cpu_enable(void); +#else +static inline bool kasan_boot_cpu_enabled(void) { return false; } +static inline int kasan_cpu_enable(void) { return 0; } +#endif + #endif #endif /* __ASM_KASAN_H */ diff --git a/arch/riscv/include/asm/page.h b/arch/riscv/include/asm/page.h index 09d15567b0b8..d4f038466f1d 100644 --- a/arch/riscv/include/asm/page.h +++ b/arch/riscv/include/asm/page.h @@ -89,6 +89,16 @@ typedef struct page *pgtable_t; #define PTE_FMT "%08lx" #endif +#ifdef CONFIG_KASAN_SW_TAGS +#define __tag_set(addr, tag) ((void *)((((u64)(addr) << 7) >> 7) | ((u64)(tag) << 57))) +#define __tag_reset(addr) ((void *)((s64)((u64)(addr) << 7) >> 7)) +#define __tag_get(addr) ((u8)((u64)(addr) >> 57)) +#else +#define __tag_set(addr, tag) (addr) +#define __tag_reset(addr) (addr) +#define __tag_get(addr) 0 +#endif + #if defined(CONFIG_64BIT) && defined(CONFIG_MMU) /* * We override this value as its generic definition uses __pa too early in @@ -155,7 +165,7 @@ phys_addr_t linear_mapping_va_to_pa(unsigned long x); }) #define __va_to_pa_nodebug(x) ({ \ - unsigned long _x = x; \ + unsigned long _x = (unsigned long)__tag_reset(x); \ is_linear_mapping(_x) ? \ linear_mapping_va_to_pa(_x) : kernel_mapping_va_to_pa(_x); \ }) @@ -179,7 +189,10 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x); #define pfn_to_virt(pfn) (__va(pfn_to_phys(pfn))) #define virt_to_page(vaddr) (pfn_to_page(virt_to_pfn(vaddr))) -#define page_to_virt(page) (pfn_to_virt(page_to_pfn(page))) +#define page_to_virt(page) ({ \ + __typeof__(page) __page = page; \ + __tag_set(pfn_to_virt(page_to_pfn(__page)), page_kasan_tag(__page)); \ +}) #define page_to_phys(page) (pfn_to_phys(page_to_pfn(page))) #define phys_to_page(paddr) (pfn_to_page(phys_to_pfn(paddr))) @@ -196,7 +209,7 @@ static __always_inline void *pfn_to_kaddr(unsigned long pfn) #endif /* __ASSEMBLY__ */ #define virt_addr_valid(vaddr) ({ \ - unsigned long _addr = (unsigned long)vaddr; \ + unsigned long _addr = (unsigned long)__tag_reset(vaddr); \ (unsigned long)(_addr) >= PAGE_OFFSET && pfn_valid(virt_to_pfn(_addr)); \ }) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 089f3c9f56a3..1b3bd1ff643a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -910,7 +910,13 @@ static inline pte_t pte_swp_clear_exclusive(pte_t pte) */ #ifdef CONFIG_64BIT #define TASK_SIZE_64 (PGDIR_SIZE * PTRS_PER_PGD / 2) +/* + * When pointer masking is enabled for the kernel's privilege mode, + * __access_ok() must reject tagged aliases of kernel memory. + */ +#ifndef CONFIG_KASAN_SW_TAGS #define TASK_SIZE_MAX LONG_MAX +#endif #ifdef CONFIG_COMPAT #define TASK_SIZE_32 (_AC(0x80000000, UL) - PAGE_SIZE) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 72e559934952..68b3a85c6960 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -31,14 +31,14 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_SFENCE_VMA_ADDR(addr); + ALT_SFENCE_VMA_ADDR(__tag_reset(addr)); } static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { if (asid != FLUSH_TLB_NO_ASID) - ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + ALT_SFENCE_VMA_ADDR_ASID(__tag_reset(addr), asid); else local_flush_tlb_page(addr); } diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..fdc72edc4857 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -299,6 +299,12 @@ void __init setup_arch(char **cmdline_p) riscv_user_isa_enable(); } +void __init smp_prepare_boot_cpu(void) +{ + if (kasan_boot_cpu_enabled()) + kasan_init_sw_tags(); +} + bool arch_cpu_is_hotpluggable(int cpu) { return cpu_has_hotplug(cpu); diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index 0f8f1c95ac38..a1cc555691b0 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -210,7 +211,11 @@ void __init smp_cpus_done(unsigned int max_cpus) asmlinkage __visible void smp_callin(void) { struct mm_struct *mm = &init_mm; - unsigned int curr_cpuid = smp_processor_id(); + unsigned int curr_cpuid; + + /* Must be called first, before referencing any dynamic allocations */ + if (kasan_boot_cpu_enabled() && kasan_cpu_enable()) + return; if (has_vector()) { /* @@ -225,6 +230,7 @@ asmlinkage __visible void smp_callin(void) mmgrab(mm); current->active_mm = mm; + curr_cpuid = smp_processor_id(); store_cpu_topology(curr_cpuid); notify_cpu_starting(curr_cpuid); diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 8eec6b69a875..ae36616fe1f5 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -20,3 +20,5 @@ lib-$(CONFIG_RISCV_ISA_ZBC) += crc32.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o + +obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o diff --git a/arch/riscv/lib/kasan_sw_tags.S b/arch/riscv/lib/kasan_sw_tags.S new file mode 100644 index 000000000000..f7d3e0acba6a --- /dev/null +++ b/arch/riscv/lib/kasan_sw_tags.S @@ -0,0 +1,61 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2020 Google LLC + * Copyright (C) 2024 SiFive + */ + +#include + +/* + * Report a tag mismatch detected by tag-based KASAN. + * + * A compiler-generated thunk calls this with a custom calling convention. + * Upon entry to this function, the following registers have been modified: + * + * x1/ra: clobbered by call to this function + * x2/sp: decremented by 256 + * x6/t1: tag from shadow memory + * x7/t2: tag from pointer + * x10/a0: fault address + * x11/a1: fault description + * x28/t3: clobbered by thunk + * x29/t4: clobbered by thunk + * x30/t5: clobbered by thunk + * x31/t6: clobbered by thunk + * + * The caller has decremented the SP by 256 bytes, and stored the following + * registers in slots on the stack according to their number (sp + 8 * xN): + * + * x1/ra: return address to user code + * x8/s0/fp: saved value from user code + * x10/a0: saved value from user code + * x11/a1: saved value from user code + */ +SYM_CODE_START(__hwasan_tag_mismatch) + /* Store the remaining unclobbered caller-saved regs */ + sd t0, (8 * 5)(sp) + sd a2, (8 * 12)(sp) + sd a3, (8 * 13)(sp) + sd a4, (8 * 14)(sp) + sd a5, (8 * 15)(sp) + sd a6, (8 * 16)(sp) + sd a7, (8 * 17)(sp) + + /* a0 and a1 are already set by the thunk */ + ld a2, (8 * 1)(sp) + call kasan_tag_mismatch + + ld ra, (8 * 1)(sp) + ld t0, (8 * 5)(sp) + ld a0, (8 * 10)(sp) + ld a1, (8 * 11)(sp) + ld a2, (8 * 12)(sp) + ld a3, (8 * 13)(sp) + ld a4, (8 * 14)(sp) + ld a5, (8 * 15)(sp) + ld a6, (8 * 16)(sp) + ld a7, (8 * 17)(sp) + addi sp, sp, 256 + ret +SYM_CODE_END(__hwasan_tag_mismatch) +EXPORT_SYMBOL(__hwasan_tag_mismatch) diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index c301c8d291d2..b247c56206c5 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -11,6 +11,10 @@ #include #include +#ifdef CONFIG_KASAN_SW_TAGS +static bool __kasan_boot_cpu_enabled __ro_after_init; +#endif + /* * Kasan shadow region must lie at a fixed address across sv39, sv48 and sv57 * which is right before the kernel. @@ -323,8 +327,11 @@ asmlinkage void __init kasan_early_init(void) { uintptr_t i; - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + else + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); for (i = 0; i < PTRS_PER_PTE; ++i) set_pte(kasan_early_shadow_pte + i, @@ -356,6 +363,8 @@ asmlinkage void __init kasan_early_init(void) KASAN_SHADOW_START, KASAN_SHADOW_END); local_flush_tlb_all(); + + __kasan_boot_cpu_enabled = !kasan_cpu_enable(); } void __init kasan_swapper_init(void) @@ -534,3 +543,20 @@ void __init kasan_init(void) csr_write(CSR_SATP, PFN_DOWN(__pa(swapper_pg_dir)) | satp_mode); local_flush_tlb_all(); } + +#ifdef CONFIG_KASAN_SW_TAGS +bool kasan_boot_cpu_enabled(void) +{ + return __kasan_boot_cpu_enabled; +} + +int kasan_cpu_enable(void) +{ + struct sbiret ret; + + /* sbi_fwft_set(POINTER_MASKING_PMLEN, 7, 0); */ + ret = sbi_ecall(0x46574654, 0, 5, 7, 0, 0, 0, 0); + + return sbi_err_map_linux_errno(ret.error); +} +#endif diff --git a/arch/riscv/mm/physaddr.c b/arch/riscv/mm/physaddr.c index 18706f457da7..6d1cf6ffd54e 100644 --- a/arch/riscv/mm/physaddr.c +++ b/arch/riscv/mm/physaddr.c @@ -8,6 +8,8 @@ phys_addr_t __virt_to_phys(unsigned long x) { + x = __tag_reset(x); + /* * Boundary checking aginst the kernel linear mapping space. */ @@ -24,6 +26,8 @@ phys_addr_t __phys_addr_symbol(unsigned long x) unsigned long kernel_start = kernel_map.virt_addr; unsigned long kernel_end = kernel_start + kernel_map.size; + x = __tag_reset(x); + /* * Boundary checking aginst the kernel image mapping. * __pa_symbol should only be used on kernel symbol addresses. From patchwork Wed Aug 14 08:55:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD561C531DC for ; Wed, 14 Aug 2024 09:02:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9hz49lwxt/ore9sEZ/uYZptf0bZHlG9fVMF8g6BZbdk=; b=ZPM2Rz9GtocX3yyTX2MeAj8BHy NXIdG8S/DSUgUj6JzzsiU0SYufG0yMavtfGwFDfgPUbZV9bX9K0L2Bo6/HlNMlPWCz+jt9ebAnWhl sqoIA2ASxIutMevtZOB1v9DRFhmZ5ob1D9z/fH8G8QuGtMsYxVQNGuP4o/7g+KJfU5Tv5XAzgB+GR sRjYfmFi0L0R6GDA6XnOKLMslbZTAL1fP9+l9YkxKw6lReV8DzOsgMXQIazKRcL3Ha4kV/Smx0C8B sk5hMT8BNBmIqKiMnO3qYen/avwHgflfE4RzxEFnSR5p/wPLECq1ZoMEc17Q5oXzMPl9KKA+U4K9T i/4lyDfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9tJ-00000006Jno-2Lzn; Wed, 14 Aug 2024 09:02:01 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o5-00000006IEV-4BZB for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9hz49lwxt/ore9sEZ/uYZptf0bZHlG9fVMF8g6BZbdk=; b=IKneuVzCPsHhPtUAbd36G4dy9d d1bQTsxx5gWoIzJAbS4WiLVxwq1JTfw+6NNaCzbwfgs6qT/8QCtVmaG3lZ59HATPsNEK7GMMo/bCb ONTFE8ZDi1Iv9cUGkFe9aXQGTbrmea/bQBMjyleWH6PAbvRwS1WNXovtNcUkhmIIz3sKCbScrni+D hrLkhj/F0JXxBqnzSamdE3t3ZF2MngNmlwg1TDtfFKqvO+lCGDl2F//qTIQElTZRxWASguaoB8NF4 s09mtzaP+B5r8PSUwQpmTLSzYh/u/L7sJdEdj9QHljXmJghcQPzr0wItybG+kke0FwzAMrdLl4iJ/ CqlneHLg==; Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9o2-00000007zGG-1URy for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:36 +0000 Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1fd78c165eeso51138275ad.2 for ; Wed, 14 Aug 2024 01:56:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625793; x=1724230593; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9hz49lwxt/ore9sEZ/uYZptf0bZHlG9fVMF8g6BZbdk=; b=BmfK4k+OUsqPfzJxo03M8csv/ozcTptoALRIoq5jJFvG+ksMuL/7aD6KPhIb2CCCOU s8il86M7+jM79fZem5FQoz0oDbGuUIhgUuB5Ew5/gPF6opBDakFneB+Ptrdbnc3PWtXY fiXsf7TDv1Sdmb+7R4IBDKkom2gziRYiPHTwUF11O5gOEzDmN7NFYuKfK/otvXKQkIdr aOgKa6PS1a3/HTYAjIQtKy8jweCwTFmCXTzaKmAvW1gVw+QN3t2SBodCuQ569NN7uwRq Tfz06weKdshja9rdF4Qrgx0UsHvyh1zQNfT5I1VJwRxxyDRo/OoljnaEEgVpUM9AdXZR Cn8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625793; x=1724230593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9hz49lwxt/ore9sEZ/uYZptf0bZHlG9fVMF8g6BZbdk=; b=id21WgQA/oMqzjMZsf2zWkhc96egqQBP/qC+Wgfbrb/VGbn3FxVV+GS1PbrJNhfGdm /l1VgHAWtWSA+fXzVCn4uVWD/ZG3YuEbTqzy9bIW1KiIb6ZHFU7wt1MWbn2Sd00z6AcJ VWRzgA8o7mbVz8bQHW8MB41m+WoDx+xUHQSmwShX4Q/TQSmQlsbQKZ2nILc/KPKC8rus psTpnsZo2V2lLi8rEY43Vte8zsmBAqX3GZt/XFQhtmXd8FMmt5TDnumgEGxki9t3bu7J wsN7SKdYkFQJVn9UqKAOrPR6m0bmnaka+p588VTd94ZnJlO+FxkupaXSmQEqGXstFs81 WD0A== X-Forwarded-Encrypted: i=1; AJvYcCWT0HR9007ebMRqOf/PoE8knecEjFtMZz9l7ymuLlWqSmgW/QBbUiZn3je7+fu5iy5PAfwkxlw2K+VVb7hWmWqZukUzWAwrEIXfPlVZwoXhtz5BBRY= X-Gm-Message-State: AOJu0Yyf3ccjcvoo8+lNXs/h/wiOToAXrYmUHI6cmihqNeuHnfR9La1d fSCh0vrvpePRFNur/qXdVWrmVq8veDhDCBqsx+LJtb1b7EFC2zVNPsxbbw1ZcPg= X-Google-Smtp-Source: AGHT+IEjBmXCU7mNTlJDHLEXVaA9qjlxh2UbxM2PLdod3xVt/jyJ0BnNxQ5PpAdVoFfN1q2eB+NLCQ== X-Received: by 2002:a17:903:1c9:b0:1fc:2ee3:d46f with SMTP id d9443c01a7336-201d638d797mr29420295ad.11.1723625792838; Wed, 14 Aug 2024 01:56:32 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:32 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 7/7] kasan: sw_tags: Support runtime stack tagging control for RISC-V Date: Wed, 14 Aug 2024 01:55:35 -0700 Message-ID: <20240814085618.968833-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095634_550589_5E09D4B2 X-CRM114-Status: GOOD ( 11.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This allows the kernel to boot on systems without pointer masking support when stack tagging is enabled. Signed-off-by: Samuel Holland Reviewed-by: Andrey Konovalov --- mm/kasan/kasan.h | 2 ++ mm/kasan/sw_tags.c | 9 +++++++++ scripts/Makefile.kasan | 5 +++++ 3 files changed, 16 insertions(+) diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index fb2b9ac0659a..01e945cb111d 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -630,6 +630,8 @@ void *__asan_memset(void *addr, int c, ssize_t len); void *__asan_memmove(void *dest, const void *src, ssize_t len); void *__asan_memcpy(void *dest, const void *src, ssize_t len); +u8 __hwasan_generate_tag(void); + void __hwasan_load1_noabort(void *); void __hwasan_store1_noabort(void *); void __hwasan_load2_noabort(void *); diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c index 220b5d4c6876..32435d33583a 100644 --- a/mm/kasan/sw_tags.c +++ b/mm/kasan/sw_tags.c @@ -70,6 +70,15 @@ u8 kasan_random_tag(void) return (u8)(state % (KASAN_TAG_MAX + 1)); } +u8 __hwasan_generate_tag(void) +{ + if (!kasan_enabled()) + return KASAN_TAG_KERNEL; + + return kasan_random_tag(); +} +EXPORT_SYMBOL(__hwasan_generate_tag); + bool kasan_check_range(const void *addr, size_t size, bool write, unsigned long ret_ip) { diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan index 390658a2d5b7..f64c1aca3e97 100644 --- a/scripts/Makefile.kasan +++ b/scripts/Makefile.kasan @@ -73,6 +73,11 @@ ifeq ($(call clang-min-version, 150000)$(call gcc-min-version, 130000),y) CFLAGS_KASAN += $(call cc-param,hwasan-kernel-mem-intrinsic-prefix=1) endif +# RISC-V requires dynamically determining if stack tagging can be enabled. +ifdef CONFIG_RISCV +CFLAGS_KASAN += $(call cc-param,hwasan-generate-tags-with-calls=1) +endif + endif # CONFIG_KASAN_SW_TAGS export CFLAGS_KASAN CFLAGS_KASAN_NOSANITIZE