| Message ID | 168f775c4587f3a1338271390204a9fe16b150dd.1739866028.git.maciej.wieczor-retman@intel.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | [v2,01/14] kasan: sw_tags: Use arithmetic shift for shadow computation | expand |
On Tue, Feb 18, 2025 at 9:16 AM Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com> wrote: > > From: Samuel Holland <samuel.holland@sifive.com> > > Currently, kasan_mem_to_shadow() uses a logical right shift, which turns > canonical kernel addresses into non-canonical addresses by clearing the > high KASAN_SHADOW_SCALE_SHIFT bits. The value of KASAN_SHADOW_OFFSET is > then chosen so that the addition results in a canonical address for the > shadow memory. > > For KASAN_GENERIC, this shift/add combination is ABI with the compiler, > because KASAN_SHADOW_OFFSET is used in compiler-generated inline tag > checks[1], which must only attempt to dereference canonical addresses. > > However, for KASAN_SW_TAGS we have some freedom to change the algorithm > without breaking the ABI. Because TBI is enabled for kernel addresses, > the top bits of shadow memory addresses computed during tag checks are > irrelevant, and so likewise are the top bits of KASAN_SHADOW_OFFSET. > This is demonstrated by the fact that LLVM uses a logical right shift > in the tag check fast path[2] but a sbfx (signed bitfield extract) > instruction in the slow path[3] without causing any issues. > > Using an arithmetic shift in kasan_mem_to_shadow() provides a number of > benefits: > > 1) The memory layout is easier to understand. KASAN_SHADOW_OFFSET > becomes a canonical memory address, and the shifted pointer becomes a > negative offset, so KASAN_SHADOW_OFFSET == KASAN_SHADOW_END regardless > of the shift amount or the size of the virtual address space. > > 2) KASAN_SHADOW_OFFSET becomes a simpler constant, requiring only one > instruction to load instead of two. Since it must be loaded in each > function with a tag check, this decreases kernel text size by 0.5%. > > 3) This shift and the sign extension from kasan_reset_tag() can be > combined into a single sbfx instruction. When this same algorithm change > is applied to the compiler, it removes an instruction from each inline > tag check, further reducing kernel text size by an additional 4.6%. > > These benefits extend to other architectures as well. On RISC-V, where > the baseline ISA does not shifted addition or have an equivalent to the > sbfx instruction, loading KASAN_SHADOW_OFFSET is reduced from 3 to 2 > instructions, and kasan_mem_to_shadow(kasan_reset_tag(addr)) similarly > combines two consecutive right shifts. > > Due to signed memory-to-shadow mapping kasan_non_canonical_hook() needs > changes - specifically the first part that tries to deduce if a faulty > address came from kasan_mem_to_shadow(). Previous value of > KASAN_SHADOW_OFFSET prevented any overflows when trying to map the > entire linear address space to shadow memory so the check in > kasan_non_canonical_hook() could consist of only checking whether the > address isn't below KASAN_SHADOW_OFFSET. > > The signed memory-to-shadow conversion means negative addresses will be > mapped below KASAN_SHADOW_OFFSET and positive addresses will map above > KASAN_SHADOW_OFFSET. When looking at the mapping of the entire address > space there will be an overflow when a big enough positive address will > be passed to kasan_mem_to_shadow(). Then the question of finding > addresses that couldn't come from kasan_mem_to_shadow() can be reduced > to figuring out if the address isn't above the highest overflowed value > (most positive address possible) AND below the most negative address > possible. Is there any reason we need this change for x86 SW_TAGS besides the optimization benefits? Is it required for the "x86: runtime_const used for KASAN_SHADOW_END" patch? If so, please check my comment there first. > > Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp#L1316 [1] > Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp#L895 [2] > Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp#L669 [3] > Signed-off-by: Samuel Holland <samuel.holland@sifive.com> > Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com> > --- > Changelog v2: (Maciej) > - Correct address range that's checked in kasan_non_canonical_hook(). > Adjust the comment inside. > - Remove part of comment from arch/arm64/include/asm/memory.h. > - Append patch message paragraph about the overflow in > kasan_non_canonical_hook(). > > arch/arm64/Kconfig | 10 +++++----- > arch/arm64/include/asm/memory.h | 14 +++++++++++++- > arch/arm64/mm/kasan_init.c | 7 +++++-- > include/linux/kasan.h | 10 ++++++++-- > mm/kasan/report.c | 26 ++++++++++++++++++++++---- > scripts/gdb/linux/mm.py | 5 +++-- > 6 files changed, 56 insertions(+), 16 deletions(-) > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index fcdd0ed3eca8..fe7d79b447c3 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -426,11 +426,11 @@ config KASAN_SHADOW_OFFSET > default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS > default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS > default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS > - default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS > - default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS > - default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS > - default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS > - default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS > + default 0xffff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS > + default 0xffffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS > + default 0xfffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS > + default 0xffffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS > + default 0xfffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS Ah, we also need to update Documentation/arch/arm64/kasan-offsets.sh, these offsets are generated by that script. Let's also point out in the commit message, that this change does not move the location of the shadow memory but only changes the way that location is calculated. > default 0xffffffffffffffff > > config UNWIND_TABLES > diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h > index 717829df294e..e71cdf036287 100644 > --- a/arch/arm64/include/asm/memory.h > +++ b/arch/arm64/include/asm/memory.h > @@ -89,7 +89,15 @@ > * > * KASAN_SHADOW_END is defined first as the shadow address that corresponds to > * the upper bound of possible virtual kernel memory addresses UL(1) << 64 > - * according to the mapping formula. > + * according to the mapping formula. For Generic KASAN, the address in the > + * mapping formula is treated as unsigned (part of the compiler's ABI), so the > + * end of the shadow memory region is at a large positive offset from > + * KASAN_SHADOW_OFFSET. For Software Tag-Based KASAN, the address in the > + * formula is treated as signed. Since all kernel addresses are negative, they > + * map to shadow memory below KASAN_SHADOW_OFFSET, making KASAN_SHADOW_OFFSET > + * itself the end of the shadow memory region. (User pointers are positive and > + * would map to shadow memory above KASAN_SHADOW_OFFSET, but shadow memory is > + * not allocated for them.) > * > * KASAN_SHADOW_START is defined second based on KASAN_SHADOW_END. The shadow > * memory start must map to the lowest possible kernel virtual memory address > @@ -100,7 +108,11 @@ > */ > #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) > #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) > +#ifdef CONFIG_KASAN_GENERIC > #define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) + KASAN_SHADOW_OFFSET) > +#else > +#define KASAN_SHADOW_END KASAN_SHADOW_OFFSET > +#endif > #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (UL(1) << ((va) - KASAN_SHADOW_SCALE_SHIFT))) > #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) > #define PAGE_END KASAN_SHADOW_START > diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c > index b65a29440a0c..6836e571555c 100644 > --- a/arch/arm64/mm/kasan_init.c > +++ b/arch/arm64/mm/kasan_init.c > @@ -198,8 +198,11 @@ static bool __init root_level_aligned(u64 addr) > /* The early shadow maps everything to a single page of zeroes */ > asmlinkage void __init kasan_early_init(void) > { > - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != > - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); > + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) > + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != > + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); > + else > + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); > BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); > BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); > BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); > diff --git a/include/linux/kasan.h b/include/linux/kasan.h > index 890011071f2b..b396feca714f 100644 > --- a/include/linux/kasan.h > +++ b/include/linux/kasan.h > @@ -61,8 +61,14 @@ int kasan_populate_early_shadow(const void *shadow_start, > #ifndef kasan_mem_to_shadow > static inline void *kasan_mem_to_shadow(const void *addr) > { > - return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT) > - + KASAN_SHADOW_OFFSET; > + void *scaled; > + > + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) > + scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); > + else > + scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); > + > + return KASAN_SHADOW_OFFSET + scaled; > } > #endif > > diff --git a/mm/kasan/report.c b/mm/kasan/report.c > index 3fe77a360f1c..5766714872d3 100644 > --- a/mm/kasan/report.c > +++ b/mm/kasan/report.c > @@ -645,15 +645,33 @@ void kasan_report_async(void) > */ > void kasan_non_canonical_hook(unsigned long addr) > { > + unsigned long max_shadow_size = BIT(BITS_PER_LONG - KASAN_SHADOW_SCALE_SHIFT); > unsigned long orig_addr; > const char *bug_type; > > /* > - * All addresses that came as a result of the memory-to-shadow mapping > - * (even for bogus pointers) must be >= KASAN_SHADOW_OFFSET. > + * With the default kasan_mem_to_shadow() algorithm, all addresses > + * returned by the memory-to-shadow mapping (even for bogus pointers) > + * must be within a certain displacement from KASAN_SHADOW_OFFSET. > + * > + * For Generic KASAN the displacement is unsigned so the mapping from zero > + * to the last kernel address needs checking. > + * > + * For Software Tag-Based KASAN, the displacement is signed, so > + * KASAN_SHADOW_OFFSET is the center of the range. Higher positive > + * addresses overflow, so the range that can't be part of > + * memory-to-shadow mapping is above the biggest positive address > + * mapping and below the lowest possible one. > */ > - if (addr < KASAN_SHADOW_OFFSET) > - return; > + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) { > + if (addr < KASAN_SHADOW_OFFSET || > + addr >= KASAN_SHADOW_OFFSET + max_shadow_size) > + return; > + } else { > + if (addr < KASAN_SHADOW_OFFSET - max_shadow_size / 2 && > + addr >= KASAN_SHADOW_OFFSET + max_shadow_size / 2) > + return; Ok, I think this would work for what I had in mind. However, I just realized that this check is not entirely precise. When doing the memory-to-shadow mapping, the memory address always has its top byte set to 0xff: both the inlined compiler code and the outline KASAN code do this. Thus, the possible values a shadow address can take are the result of the memory-to-shadow mapping applied to [0xff00000000000000, 0xffffffffffffffff], not to the whole address space. So we can make this check more precise. > + } > > orig_addr = (unsigned long)kasan_shadow_to_mem((void *)addr); > > diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py > index 7571aebbe650..2e63f3dedd53 100644 > --- a/scripts/gdb/linux/mm.py > +++ b/scripts/gdb/linux/mm.py > @@ -110,12 +110,13 @@ class aarch64_page_ops(): > self.KERNEL_END = gdb.parse_and_eval("_end") > > if constants.LX_CONFIG_KASAN_GENERIC or constants.LX_CONFIG_KASAN_SW_TAGS: > + self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET > if constants.LX_CONFIG_KASAN_GENERIC: > self.KASAN_SHADOW_SCALE_SHIFT = 3 > + self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET > else: > self.KASAN_SHADOW_SCALE_SHIFT = 4 > - self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET > - self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET > + self.KASAN_SHADOW_END = self.KASAN_SHADOW_OFFSET > self.PAGE_END = self.KASAN_SHADOW_END - (1 << (self.vabits_actual - self.KASAN_SHADOW_SCALE_SHIFT)) > else: > self.PAGE_END = self._PAGE_END(self.VA_BITS_MIN) We likely also need to update scripts/gdb/linux/kasan.py. Also, later in the series, you change KASAN_SHADOW_OFFSET from a config option into a runtime_const, which AFAIU would make these scripts stop working.
On 2025-02-20 at 00:29:14 +0100, Andrey Konovalov wrote: >On Tue, Feb 18, 2025 at 9:16 AM Maciej Wieczor-Retman ><maciej.wieczor-retman@intel.com> wrote: >> >> From: Samuel Holland <samuel.holland@sifive.com> >> >> Currently, kasan_mem_to_shadow() uses a logical right shift, which turns >> canonical kernel addresses into non-canonical addresses by clearing the >> high KASAN_SHADOW_SCALE_SHIFT bits. The value of KASAN_SHADOW_OFFSET is >> then chosen so that the addition results in a canonical address for the >> shadow memory. >> >> For KASAN_GENERIC, this shift/add combination is ABI with the compiler, >> because KASAN_SHADOW_OFFSET is used in compiler-generated inline tag >> checks[1], which must only attempt to dereference canonical addresses. >> >> However, for KASAN_SW_TAGS we have some freedom to change the algorithm >> without breaking the ABI. Because TBI is enabled for kernel addresses, >> the top bits of shadow memory addresses computed during tag checks are >> irrelevant, and so likewise are the top bits of KASAN_SHADOW_OFFSET. >> This is demonstrated by the fact that LLVM uses a logical right shift >> in the tag check fast path[2] but a sbfx (signed bitfield extract) >> instruction in the slow path[3] without causing any issues. >> >> Using an arithmetic shift in kasan_mem_to_shadow() provides a number of >> benefits: >> >> 1) The memory layout is easier to understand. KASAN_SHADOW_OFFSET >> becomes a canonical memory address, and the shifted pointer becomes a >> negative offset, so KASAN_SHADOW_OFFSET == KASAN_SHADOW_END regardless >> of the shift amount or the size of the virtual address space. >> >> 2) KASAN_SHADOW_OFFSET becomes a simpler constant, requiring only one >> instruction to load instead of two. Since it must be loaded in each >> function with a tag check, this decreases kernel text size by 0.5%. >> >> 3) This shift and the sign extension from kasan_reset_tag() can be >> combined into a single sbfx instruction. When this same algorithm change >> is applied to the compiler, it removes an instruction from each inline >> tag check, further reducing kernel text size by an additional 4.6%. >> >> These benefits extend to other architectures as well. On RISC-V, where >> the baseline ISA does not shifted addition or have an equivalent to the >> sbfx instruction, loading KASAN_SHADOW_OFFSET is reduced from 3 to 2 >> instructions, and kasan_mem_to_shadow(kasan_reset_tag(addr)) similarly >> combines two consecutive right shifts. >> >> Due to signed memory-to-shadow mapping kasan_non_canonical_hook() needs >> changes - specifically the first part that tries to deduce if a faulty >> address came from kasan_mem_to_shadow(). Previous value of >> KASAN_SHADOW_OFFSET prevented any overflows when trying to map the >> entire linear address space to shadow memory so the check in >> kasan_non_canonical_hook() could consist of only checking whether the >> address isn't below KASAN_SHADOW_OFFSET. >> >> The signed memory-to-shadow conversion means negative addresses will be >> mapped below KASAN_SHADOW_OFFSET and positive addresses will map above >> KASAN_SHADOW_OFFSET. When looking at the mapping of the entire address >> space there will be an overflow when a big enough positive address will >> be passed to kasan_mem_to_shadow(). Then the question of finding >> addresses that couldn't come from kasan_mem_to_shadow() can be reduced >> to figuring out if the address isn't above the highest overflowed value >> (most positive address possible) AND below the most negative address >> possible. > >Is there any reason we need this change for x86 SW_TAGS besides the >optimization benefits? I wanted to have the shadow memory boundries aligned properly, to not waste page table entries, so the memory map is more straight forward. This patch helps with that, I don't think it would have worked without it. > >Is it required for the "x86: runtime_const used for KASAN_SHADOW_END" >patch? If so, please check my comment there first. Probably not, even if the shadow memory boundry addresses are static defines it'd be nice to have this patch to already think about the mappings as signed and about offset as the end of the shadow memory space. I'll think if I can keep all the things mentioned above without runtime_const. I had some idea with non-canonical offsets but I need to write this out a bit more yet. > >> >> Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/AddressSanitizer.cpp#L1316 [1] >> Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Transforms/Instrumentation/HWAddressSanitizer.cpp#L895 [2] >> Link: https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/lib/Target/AArch64/AArch64AsmPrinter.cpp#L669 [3] >> Signed-off-by: Samuel Holland <samuel.holland@sifive.com> >> Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com> >> --- >> Changelog v2: (Maciej) >> - Correct address range that's checked in kasan_non_canonical_hook(). >> Adjust the comment inside. >> - Remove part of comment from arch/arm64/include/asm/memory.h. >> - Append patch message paragraph about the overflow in >> kasan_non_canonical_hook(). >> >> arch/arm64/Kconfig | 10 +++++----- >> arch/arm64/include/asm/memory.h | 14 +++++++++++++- >> arch/arm64/mm/kasan_init.c | 7 +++++-- >> include/linux/kasan.h | 10 ++++++++-- >> mm/kasan/report.c | 26 ++++++++++++++++++++++---- >> scripts/gdb/linux/mm.py | 5 +++-- >> 6 files changed, 56 insertions(+), 16 deletions(-) >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index fcdd0ed3eca8..fe7d79b447c3 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -426,11 +426,11 @@ config KASAN_SHADOW_OFFSET >> default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS >> default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS >> default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS >> - default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS >> - default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS >> - default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS >> - default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS >> - default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS >> + default 0xffff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS >> + default 0xffffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS >> + default 0xfffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS >> + default 0xffffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS >> + default 0xfffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS > >Ah, we also need to update Documentation/arch/arm64/kasan-offsets.sh, >these offsets are generated by that script. > >Let's also point out in the commit message, that this change does not >move the location of the shadow memory but only changes the way that >location is calculated. Will do, thanks :) > >> default 0xffffffffffffffff >> >> config UNWIND_TABLES >> diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h >> index 717829df294e..e71cdf036287 100644 >> --- a/arch/arm64/include/asm/memory.h >> +++ b/arch/arm64/include/asm/memory.h >> @@ -89,7 +89,15 @@ >> * >> * KASAN_SHADOW_END is defined first as the shadow address that corresponds to >> * the upper bound of possible virtual kernel memory addresses UL(1) << 64 >> - * according to the mapping formula. >> + * according to the mapping formula. For Generic KASAN, the address in the >> + * mapping formula is treated as unsigned (part of the compiler's ABI), so the >> + * end of the shadow memory region is at a large positive offset from >> + * KASAN_SHADOW_OFFSET. For Software Tag-Based KASAN, the address in the >> + * formula is treated as signed. Since all kernel addresses are negative, they >> + * map to shadow memory below KASAN_SHADOW_OFFSET, making KASAN_SHADOW_OFFSET >> + * itself the end of the shadow memory region. (User pointers are positive and >> + * would map to shadow memory above KASAN_SHADOW_OFFSET, but shadow memory is >> + * not allocated for them.) >> * >> * KASAN_SHADOW_START is defined second based on KASAN_SHADOW_END. The shadow >> * memory start must map to the lowest possible kernel virtual memory address >> @@ -100,7 +108,11 @@ >> */ >> #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) >> #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) >> +#ifdef CONFIG_KASAN_GENERIC >> #define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) + KASAN_SHADOW_OFFSET) >> +#else >> +#define KASAN_SHADOW_END KASAN_SHADOW_OFFSET >> +#endif >> #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (UL(1) << ((va) - KASAN_SHADOW_SCALE_SHIFT))) >> #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) >> #define PAGE_END KASAN_SHADOW_START >> diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c >> index b65a29440a0c..6836e571555c 100644 >> --- a/arch/arm64/mm/kasan_init.c >> +++ b/arch/arm64/mm/kasan_init.c >> @@ -198,8 +198,11 @@ static bool __init root_level_aligned(u64 addr) >> /* The early shadow maps everything to a single page of zeroes */ >> asmlinkage void __init kasan_early_init(void) >> { >> - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != >> - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); >> + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) >> + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != >> + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); >> + else >> + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); >> BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); >> BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); >> BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); >> diff --git a/include/linux/kasan.h b/include/linux/kasan.h >> index 890011071f2b..b396feca714f 100644 >> --- a/include/linux/kasan.h >> +++ b/include/linux/kasan.h >> @@ -61,8 +61,14 @@ int kasan_populate_early_shadow(const void *shadow_start, >> #ifndef kasan_mem_to_shadow >> static inline void *kasan_mem_to_shadow(const void *addr) >> { >> - return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT) >> - + KASAN_SHADOW_OFFSET; >> + void *scaled; >> + >> + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) >> + scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); >> + else >> + scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); >> + >> + return KASAN_SHADOW_OFFSET + scaled; >> } >> #endif >> >> diff --git a/mm/kasan/report.c b/mm/kasan/report.c >> index 3fe77a360f1c..5766714872d3 100644 >> --- a/mm/kasan/report.c >> +++ b/mm/kasan/report.c >> @@ -645,15 +645,33 @@ void kasan_report_async(void) >> */ >> void kasan_non_canonical_hook(unsigned long addr) >> { >> + unsigned long max_shadow_size = BIT(BITS_PER_LONG - KASAN_SHADOW_SCALE_SHIFT); >> unsigned long orig_addr; >> const char *bug_type; >> >> /* >> - * All addresses that came as a result of the memory-to-shadow mapping >> - * (even for bogus pointers) must be >= KASAN_SHADOW_OFFSET. >> + * With the default kasan_mem_to_shadow() algorithm, all addresses >> + * returned by the memory-to-shadow mapping (even for bogus pointers) >> + * must be within a certain displacement from KASAN_SHADOW_OFFSET. >> + * >> + * For Generic KASAN the displacement is unsigned so the mapping from zero >> + * to the last kernel address needs checking. >> + * >> + * For Software Tag-Based KASAN, the displacement is signed, so >> + * KASAN_SHADOW_OFFSET is the center of the range. Higher positive >> + * addresses overflow, so the range that can't be part of >> + * memory-to-shadow mapping is above the biggest positive address >> + * mapping and below the lowest possible one. >> */ >> - if (addr < KASAN_SHADOW_OFFSET) >> - return; >> + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) { >> + if (addr < KASAN_SHADOW_OFFSET || >> + addr >= KASAN_SHADOW_OFFSET + max_shadow_size) >> + return; >> + } else { >> + if (addr < KASAN_SHADOW_OFFSET - max_shadow_size / 2 && >> + addr >= KASAN_SHADOW_OFFSET + max_shadow_size / 2) >> + return; > >Ok, I think this would work for what I had in mind. > >However, I just realized that this check is not entirely precise. When >doing the memory-to-shadow mapping, the memory address always has its >top byte set to 0xff: both the inlined compiler code and the outline >KASAN code do this Do you mean that non-canonical addresses passed to kasan_mem_to_shadow() will map to the same space that the canonical version would map to? What does that? Does the compiler do something more than is in kasan_mem_to_shadow() when instrumenting functions? > Thus, the possible values a shadow address can >take are the result of the memory-to-shadow mapping applied to >[0xff00000000000000, 0xffffffffffffffff], not to the whole address >space. So we can make this check more precise. In case my question above didn't lead to this: what happens to the rest of the values if they get plugged into kasan_mem_to_shadow()? > >> + } >> >> orig_addr = (unsigned long)kasan_shadow_to_mem((void *)addr); >> >> diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py >> index 7571aebbe650..2e63f3dedd53 100644 >> --- a/scripts/gdb/linux/mm.py >> +++ b/scripts/gdb/linux/mm.py >> @@ -110,12 +110,13 @@ class aarch64_page_ops(): >> self.KERNEL_END = gdb.parse_and_eval("_end") >> >> if constants.LX_CONFIG_KASAN_GENERIC or constants.LX_CONFIG_KASAN_SW_TAGS: >> + self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET >> if constants.LX_CONFIG_KASAN_GENERIC: >> self.KASAN_SHADOW_SCALE_SHIFT = 3 >> + self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET >> else: >> self.KASAN_SHADOW_SCALE_SHIFT = 4 >> - self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET >> - self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET >> + self.KASAN_SHADOW_END = self.KASAN_SHADOW_OFFSET >> self.PAGE_END = self.KASAN_SHADOW_END - (1 << (self.vabits_actual - self.KASAN_SHADOW_SCALE_SHIFT)) >> else: >> self.PAGE_END = self._PAGE_END(self.VA_BITS_MIN) > >We likely also need to update scripts/gdb/linux/kasan.py. > >Also, later in the series, you change KASAN_SHADOW_OFFSET from a >config option into a runtime_const, which AFAIU would make these >scripts stop working. Thanks, I'll have to think about these.
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fcdd0ed3eca8..fe7d79b447c3 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -426,11 +426,11 @@ config KASAN_SHADOW_OFFSET default 0xdffffe0000000000 if ARM64_VA_BITS_42 && !KASAN_SW_TAGS default 0xdfffffc000000000 if ARM64_VA_BITS_39 && !KASAN_SW_TAGS default 0xdffffff800000000 if ARM64_VA_BITS_36 && !KASAN_SW_TAGS - default 0xefff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS - default 0xefffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS - default 0xeffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS - default 0xefffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS - default 0xeffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS + default 0xffff800000000000 if (ARM64_VA_BITS_48 || (ARM64_VA_BITS_52 && !ARM64_16K_PAGES)) && KASAN_SW_TAGS + default 0xffffc00000000000 if (ARM64_VA_BITS_47 || ARM64_VA_BITS_52) && ARM64_16K_PAGES && KASAN_SW_TAGS + default 0xfffffe0000000000 if ARM64_VA_BITS_42 && KASAN_SW_TAGS + default 0xffffffc000000000 if ARM64_VA_BITS_39 && KASAN_SW_TAGS + default 0xfffffff800000000 if ARM64_VA_BITS_36 && KASAN_SW_TAGS default 0xffffffffffffffff config UNWIND_TABLES diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index 717829df294e..e71cdf036287 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -89,7 +89,15 @@ * * KASAN_SHADOW_END is defined first as the shadow address that corresponds to * the upper bound of possible virtual kernel memory addresses UL(1) << 64 - * according to the mapping formula. + * according to the mapping formula. For Generic KASAN, the address in the + * mapping formula is treated as unsigned (part of the compiler's ABI), so the + * end of the shadow memory region is at a large positive offset from + * KASAN_SHADOW_OFFSET. For Software Tag-Based KASAN, the address in the + * formula is treated as signed. Since all kernel addresses are negative, they + * map to shadow memory below KASAN_SHADOW_OFFSET, making KASAN_SHADOW_OFFSET + * itself the end of the shadow memory region. (User pointers are positive and + * would map to shadow memory above KASAN_SHADOW_OFFSET, but shadow memory is + * not allocated for them.) * * KASAN_SHADOW_START is defined second based on KASAN_SHADOW_END. The shadow * memory start must map to the lowest possible kernel virtual memory address @@ -100,7 +108,11 @@ */ #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) #define KASAN_SHADOW_OFFSET _AC(CONFIG_KASAN_SHADOW_OFFSET, UL) +#ifdef CONFIG_KASAN_GENERIC #define KASAN_SHADOW_END ((UL(1) << (64 - KASAN_SHADOW_SCALE_SHIFT)) + KASAN_SHADOW_OFFSET) +#else +#define KASAN_SHADOW_END KASAN_SHADOW_OFFSET +#endif #define _KASAN_SHADOW_START(va) (KASAN_SHADOW_END - (UL(1) << ((va) - KASAN_SHADOW_SCALE_SHIFT))) #define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) #define PAGE_END KASAN_SHADOW_START diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index b65a29440a0c..6836e571555c 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -198,8 +198,11 @@ static bool __init root_level_aligned(u64 addr) /* The early shadow maps everything to a single page of zeroes */ asmlinkage void __init kasan_early_init(void) { - BUILD_BUG_ON(KASAN_SHADOW_OFFSET != - KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != + KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); + else + BUILD_BUG_ON(KASAN_SHADOW_OFFSET != KASAN_SHADOW_END); BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS), SHADOW_ALIGN)); BUILD_BUG_ON(!IS_ALIGNED(_KASAN_SHADOW_START(VA_BITS_MIN), SHADOW_ALIGN)); BUILD_BUG_ON(!IS_ALIGNED(KASAN_SHADOW_END, SHADOW_ALIGN)); diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 890011071f2b..b396feca714f 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -61,8 +61,14 @@ int kasan_populate_early_shadow(const void *shadow_start, #ifndef kasan_mem_to_shadow static inline void *kasan_mem_to_shadow(const void *addr) { - return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT) - + KASAN_SHADOW_OFFSET; + void *scaled; + + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) + scaled = (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT); + else + scaled = (void *)((long)addr >> KASAN_SHADOW_SCALE_SHIFT); + + return KASAN_SHADOW_OFFSET + scaled; } #endif diff --git a/mm/kasan/report.c b/mm/kasan/report.c index 3fe77a360f1c..5766714872d3 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -645,15 +645,33 @@ void kasan_report_async(void) */ void kasan_non_canonical_hook(unsigned long addr) { + unsigned long max_shadow_size = BIT(BITS_PER_LONG - KASAN_SHADOW_SCALE_SHIFT); unsigned long orig_addr; const char *bug_type; /* - * All addresses that came as a result of the memory-to-shadow mapping - * (even for bogus pointers) must be >= KASAN_SHADOW_OFFSET. + * With the default kasan_mem_to_shadow() algorithm, all addresses + * returned by the memory-to-shadow mapping (even for bogus pointers) + * must be within a certain displacement from KASAN_SHADOW_OFFSET. + * + * For Generic KASAN the displacement is unsigned so the mapping from zero + * to the last kernel address needs checking. + * + * For Software Tag-Based KASAN, the displacement is signed, so + * KASAN_SHADOW_OFFSET is the center of the range. Higher positive + * addresses overflow, so the range that can't be part of + * memory-to-shadow mapping is above the biggest positive address + * mapping and below the lowest possible one. */ - if (addr < KASAN_SHADOW_OFFSET) - return; + if (IS_ENABLED(CONFIG_KASAN_GENERIC)) { + if (addr < KASAN_SHADOW_OFFSET || + addr >= KASAN_SHADOW_OFFSET + max_shadow_size) + return; + } else { + if (addr < KASAN_SHADOW_OFFSET - max_shadow_size / 2 && + addr >= KASAN_SHADOW_OFFSET + max_shadow_size / 2) + return; + } orig_addr = (unsigned long)kasan_shadow_to_mem((void *)addr); diff --git a/scripts/gdb/linux/mm.py b/scripts/gdb/linux/mm.py index 7571aebbe650..2e63f3dedd53 100644 --- a/scripts/gdb/linux/mm.py +++ b/scripts/gdb/linux/mm.py @@ -110,12 +110,13 @@ class aarch64_page_ops(): self.KERNEL_END = gdb.parse_and_eval("_end") if constants.LX_CONFIG_KASAN_GENERIC or constants.LX_CONFIG_KASAN_SW_TAGS: + self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET if constants.LX_CONFIG_KASAN_GENERIC: self.KASAN_SHADOW_SCALE_SHIFT = 3 + self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET else: self.KASAN_SHADOW_SCALE_SHIFT = 4 - self.KASAN_SHADOW_OFFSET = constants.LX_CONFIG_KASAN_SHADOW_OFFSET - self.KASAN_SHADOW_END = (1 << (64 - self.KASAN_SHADOW_SCALE_SHIFT)) + self.KASAN_SHADOW_OFFSET + self.KASAN_SHADOW_END = self.KASAN_SHADOW_OFFSET self.PAGE_END = self.KASAN_SHADOW_END - (1 << (self.vabits_actual - self.KASAN_SHADOW_SCALE_SHIFT)) else: self.PAGE_END = self._PAGE_END(self.VA_BITS_MIN)