From patchwork Wed Aug 14 08:55:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 13763114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9943BC52D7F for ; Wed, 14 Aug 2024 08:57:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=p+ZWbM+3N2UqMrGeBrOGYCDvWi KwuS57AhifBbu2B9ywAJ45fNZyjhhvAX2BZecxBFWdFCY75pKge9Zcf1K+Xw0/dN2YV33qNfd+Pyt QJf75aLobsPPppJRAW09npBzRDvyDazZHOApEsuSItt8SyEryAHhgMg/0OZnN81bdganxcnjux3BY nD0ZzjIhOjGbTifnZmx8Xaxxu1vPrL+LnsPcItxYnm0iZqFDXCAMetmu05wYJLR09ou1Nj0CsoTOE 10WecMGaZ5+FWcdKXg9ZZbS2rPsEzR6Ekw3V6ukKE5sCP17dXgvHHMF8+7v9W+pCGSU/V4f96FWyb jodKm6YQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9p9-00000006Ieg-2Fvx; Wed, 14 Aug 2024 08:57:43 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9ny-00000006IBU-2ZC5 for linux-arm-kernel@bombadil.infradead.org; Wed, 14 Aug 2024 08:56:30 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=ji57IXUbZOXsGiFQoumo4XWXNa u3WMfFNoBuDTzXE8fv5jQ8mVY/vcbYEDr388RIKnmRmKKZBaBq7IsQMp4eBIA3CSBGD3u1L1Et3A5 0wHroIvkBD2GCirNN04MyxGBF7/UjvEFNBID89URVScl6TRSLDDa6mwX1noIIiBCzDud1TJqknLxt kqmk9kQx8r0jU4xlb+f2Sg4OZwXZ5ZzjzR0pQaLS3LlZHYGPLVZN533oYblrEDX4Eb4EdTllG5gLI xH4Yr+WWXgOgpRL8HWRbN9+7K4nPf/PORPYJi0M8hu0nIR6dVz6ylTJhg0tz4svw6lufiH4ZI8lLi qy6OyNNg==; Received: from mail-pg1-x534.google.com ([2607:f8b0:4864:20::534]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1se9nt-00000007z9T-1sKW for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 08:56:28 +0000 Received: by mail-pg1-x534.google.com with SMTP id 41be03b00d2f7-76cb5b6b3e4so4215951a12.1 for ; Wed, 14 Aug 2024 01:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1723625782; x=1724230582; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=AUWey4GXIQve35UzoHaJIckjTrNWtsfsJn7FBXn+thjOedff3tRNySxU3ogku+plue HXJyLawFzNk6Aq7AzPbUZfDnqnps4ID9hIUpZhoZuXSHlATTE6OCS2+cPxuORIVkAOfG l6eKjBlWsiVA6DxpIQCr6oKd1u9z1mPau/9AeO7bvqy2v1bUrKnBTbsSzhxypJqBl2GJ 8y6hwIeBH9joXoIuOZetPlfCUxw5UQIxfinRcfU86Ww/j9+2njDbLiUYxTwhYt3dXajf ccVXScmbZSdzKLqz3+nPTYbWxkiK9YBaZXpLbQI70mDb8qQkQ8bzyZUx1SKA7V6eJyRF owyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723625782; x=1724230582; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O7k+MLTxwWAputjEjU3HfIE7+9GwjR6EFArU6PwcDFo=; b=i9G4It+59P9jTNpvNfe60Jvnkp7Jegto/MYm9GPMiIvqR3WtMXlQYtiq/NsBZJHrsS EArx3J77reI6hFusTShk4o6haWSIxxLi9eE6sL9wTje+QE1Ip5eXmynm1CpzfZgbeGWU erpXH6gDNWI2lpJeJC0MqHi8OoV+gSAg97gtT1RVbaHXEGRmswUK6CBIJ5jKQxo/Ur2o j6el0jH80bFM6GjGK8tiDJ9zHz2Rne05CKQVFChFBKQB1DPO+wwMRmA0wDOA5iNgBEB9 j+jedFRdnFLezg7KDI49AsmVIpUf4Q7FTRF7XLOrf72fUJ83tS9IGXgCQMbzRdC9qF15 meOA== X-Forwarded-Encrypted: i=1; AJvYcCXj/4Ta9yGpOgrzYWKKqOzBTNLLflzI7pnB05pNM+4y4PhQX9pQ+SG1E+bcmr+IdAsQnZJvAI/fq+EAAT1+YTrgA/LSutBkycNBqTdJMj5448KIRQc= X-Gm-Message-State: AOJu0YwSlOtPOCnT6fMJF9ZFDxHXjsMvRyqTTp3fe7+3IkoRh/REqB1U WIH6OH6Uc8llsGV0PCufGlihPfavsuGFrPauor392w2TWLITj7xUe5YeONuyBRM= X-Google-Smtp-Source: AGHT+IFEE/BC7r7DqzdXwRgEMrr+FQ81W6ResIpLXSNMMxD7udxBFFjplbf0XoQXYF4QlNhTBdsXYw== X-Received: by 2002:a05:6a21:9206:b0:1c3:b1e2:f826 with SMTP id adf61e73a8af0-1c8eaf5ab1bmr2775585637.35.1723625782207; Wed, 14 Aug 2024 01:56:22 -0700 (PDT) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd14a7b8sm25439615ad.100.2024.08.14.01.56.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 01:56:21 -0700 (PDT) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org, Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com Cc: llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Alexandre Ghiti , Evgenii Stepanov , Andrew Morton , linux-arm-kernel@lists.infradead.org, Samuel Holland Subject: [RFC PATCH 1/7] kasan: sw_tags: Use arithmetic shift for shadow computation Date: Wed, 14 Aug 2024 01:55:29 -0700 Message-ID: <20240814085618.968833-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.45.1 In-Reply-To: <20240814085618.968833-1-samuel.holland@sifive.com> References: <20240814085618.968833-1-samuel.holland@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_095625_798317_92F9B260 X-CRM114-Status: GOOD ( 25.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, kasan_mem_to_shadow() uses a logical right shift, which turns canonical kernel addresses into non-canonical addresses by clearing the high KASAN_SHADOW_SCALE_SHIFT bits. The value of KASAN_SHADOW_OFFSET is then chosen so that the addition results in a canonical address for the shadow memory. For KASAN_GENERIC, this shift/add combination is ABI with the compiler, because KASAN_SHADOW_OFFSET is used in compiler-generated inline tag checks[1], which must only attempt to dereference canonical addresses. However, for KASAN_SW_TAGS we have some freedom to change the algorithm without breaking the ABI. Because TBI is enabled for kernel addresses, the top bits of shadow memory addresses computed during tag checks are irrelevant, and so likewise are the top bits of KASAN_SHADOW_OFFSET. This is demonstrated by the fact that LLVM uses a logical right shift in the tag check fast path[2] but a sbfx (signed bitfield extract) instruction in the slow path[3] without causing any issues. Using an arithmetic shift in kasan_mem_to_shadow() provides a number of benefits: 1) The memory layout is easier to understand. KASAN_SHADOW_OFFSET becomes a canonical memory address, and the shifted pointer becomes a negative offset, so KASAN_SHADOW_OFFSET == KASAN_SHADOW_END regardless of the shift amount or the size of the virtual address space. 2) KASAN_SHADOW_OFFSET becomes a simpler constant, requiring only one instruction to load instead of two. Since it must be loaded in each function with a tag check, this decreases kernel text size by 0.5%. 3) This shift and the sign extension from kasan_reset_tag() can be combined into a single sbfx instruction. When this same algorithm change is applied to the compiler, it removes an instruction from each inline tag check, further reducing kernel text size by an additional 4.6%. These benefits extend to other architectures as well. On RISC-V, where the baseline ISA does not shifted addition or have an equivalent to the sbfx instruction, loading KASAN_SHADOW_OFFSET is reduced from 3 to 2 instructions, and kasan_mem_to_shadow(kasan_reset_tag(addr)) similarly combines two consecutive right shifts. Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp#L1316 [1] Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp#L895 [2] Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp#L669 [3] Signed-off-by: Samuel Holland --- arch/arm64/Kconfig | 10 +++++----- arch/arm64/include/asm/memory.h | 8 ++++++++ arch/arm64/mm/kasan_init.c | 7 +++++-- include/linux/kasan.h | 10 ++++++++-- scripts/gdb/linux/mm.py | 5 +++-- 5 files changed, 29 insertions(+), 11 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2f8ff354ca6..7df218cca168 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -402,11 +402,11 @@ config KASAN_SHADOW_OFFSET default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS - default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS - default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS - default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS - default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS - default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS + default 0xffff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS + default 0xffffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS + default 0xfffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS + default 0xffffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS + default 0xfffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff config UNWIND_TABLES diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 54fb014eba05..3af8d1e721af 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -82,6 +82,10 @@ * the mapping. Note that KASAN_SHADOW_OFFSET does not point to the start of * the shadow memory region. * + * For KASAN_GENERIC, addr is treated as unsigned. For KASAN_SW_TAGS, addr is + * treated as signed, so in that case KASAN_SHADOW_OFFSET points to the end of + * the shadow memory region. + * * Based on this mapping, we define two constants: * * KASAN_SHADOW_START: the start of the shadow memory region; @@ -100,7 +104,11 @@ */ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) +#ifdef CONFIG_KASAN_GENERIC #define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) + KASAN_SHADOW_OFFSET) +#else +#define KASAN_SHADOW_END KASAN_SHADOW_OFFSET +#endif #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (UL(1) << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) #define PAGE_END KASAN_SHADOW_START diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index b65a29440a0c..6836e571555c 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -198,8 +198,11 @@ static bool __init root_level_aligned(u64 addr) /* The early shadow maps everything to a single page of zeroes */ asmlinkage void __init kasan_early_init(void) { - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + else + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 70d6a8f6e25d..41f57e10ba03 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -58,8 +58,14 @@ int kasan_populate_early_shadow(const void *shadow_start, #ifndef kasan_mem_to_shadow static inline void *kasan_mem_to_shadow(const void *addr) { - return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT) - + KASAN_SHADOW_OFFSET; + void *scaled; + + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); + else + scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); + + return KASAN_SHADOW_OFFSET + scaled; } #endif diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index 7571aebbe650..2e63f3dedd53 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -110,12 +110,13 @@ class aarch64_page_ops(): self.KERNEL_END = gdb.parse_and_eval("_end") if constants.LX_CONFIG_KASAN_GENERIC or constants.LX_CONFIG_KASAN_SW_TAGS: + self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET if constants.LX_CONFIG_KASAN_GENERIC: self.KASAN_SHADOW_SCALE_SHIFT = 3 + self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET else: self.KASAN_SHADOW_SCALE_SHIFT = 4 - self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET - self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET + self.KASAN_SHADOW_END = self.KASAN_SHADOW_OFFSET self.PAGE_END = self.KASAN_SHADOW_END - (1 << (self.vabits_actual - self.KASAN_SHADOW_SCALE_SHIFT)) else: self.PAGE_END = self._PAGE_END(self.VA_BITS_MIN)