From patchwork Mon Aug 28 19:58:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13368285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11E32C83F15 for ; Mon, 28 Aug 2023 19:59:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=3YlFKRgOb4W9Wn8BBT51A+dt1yKSUpi6AiscbbM+xYg=; b=pH3LKaiMB7F+CjkAfZEg/rB9f4 gSVzldI9S1+gKjwoEI/hYpJQMeapULR/O1CIQarDC0FAq+Ypf29elE0MUcszuWMxiuf/Iv6RYOHTp a3cdv9glabLjh63LkbMCvam4MhWrNpJZJ0Hifs9yjTYJQGWH84wAwHRj9Xgc4oYkI63q1J0ufCntk 4cH1Xw86L2tCZcC6uzeMnP0jXuz58BJjgJrF7OfxJh5ofwd/7GLNCl+70QdLh4dDeuphMs6jz2XdK 6hX87s1bmeW8lNlhC078HCPSJSvYvoqrBGCuqJNb3UXpTckxnsimJp358B6iiDj2tPzrenDAqSEUz DnzDwa8w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNp-00ABux-2e; Mon, 28 Aug 2023 19:58:45 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNm-00ABth-2Z for linux-riscv@lists.infradead.org; Mon, 28 Aug 2023 19:58:44 +0000 Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-56fe210b7edso2222154a12.0 for ; Mon, 28 Aug 2023 12:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252720; x=1693857520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t72XOnwbzECF8GhVlL1pH0HFKftu91Ma7cUPYGPtetA=; b=WZrYBY3ho+HCCDP1FNKRdj5ezMO/wMySjNeFY9G+uOwejtvXlVrwbR8kmj6vddDtUc aqiPJNVaI2KWobFvj85tMLZkTndypS0s4b0dIqIcvCPoBeKWUJjd3sVYgpoScGFlaEw5 BCe/6mRhWNwoLP03aMvdOMvl55wDb+Wwkgpa5Jj/lPc+q2LXL+lIQ7pnEKtdiBr5tN1T dNWa1ELTP6r1biv9I1bMPFdmjmxGW+btzWAfC2cIncymnVG50+n09MBdmqLdyBP2vaD1 H3gM/2eLsiEq3Q+J9YZynhM/wfvf2NO4c70IB0Hy694bou6t1Czg63ZUxC/SLqYahrbi sSKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252720; x=1693857520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t72XOnwbzECF8GhVlL1pH0HFKftu91Ma7cUPYGPtetA=; b=gyLwZhmObGoGHzYAgD7TRVxUnygRRgQ56QFUQ33aSOT2tbtauC9dEe1z/kXOzxmWlJ H6Yd+JfGVClEiFIjBZ4f3GY03kxur6iwU6qGaLuAsG1BSj0hbs+d2Wz/kYrlof5qq84C umoINfAcDivHq0mxuLtv7ng5MsVnmOsUFylkPjeM0uC1kWA4jPCusoW5Go9oKmtrB4xm c4ybNqMsknLB1EasCiAZedbJ8h+biJS3LUlMtkFeP9YJAaDHWxSqZzlH13VvuYsh4HvX 4ISOKpmqqGE4FGYc18ydfmAeFV9HhaAJOYNJ66SDYty+ALVbgmJfBuzYQSUi59SB59ku JbXQ== X-Gm-Message-State: AOJu0YzTHx4P+tTyTg2fP9I10XHX0Qy7SIP3DxkB7kBm/Ig1cowXUSKc A/EjdXgFIotnMYOs302CtZHs2fRjDfhBX8ITIBM= X-Google-Smtp-Source: AGHT+IHbDMp1KPlbpSM031jXdTCRFJuh5xUwCvFAfRqKNHsphJHlbIQMX0YF0Z1FsX6MDgNL+BriZU4ZvulJC66z3Pg= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a63:3d04:0:b0:56b:cd71:6094 with SMTP id k4-20020a633d04000000b0056bcd716094mr4746709pga.1.1693252720142; Mon, 28 Aug 2023 12:58:40 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:35 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=10965; i=samitolvanen@google.com; h=from:subject; bh=UIBcAVgt1wQaHtPmKryqprOkmbFFoOTbmDsyS0eCtqk=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqvloXMV34+z11KS13LdGv1YQAUQNx6Wwp/ OrO+gdAcW6JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7q7bDACIdTbaLnmxOTBA4Mwg2Plne99OzA7EIUAwlRyp+BnE3ns/L8CLKVxiApCq8ThbhFVIrL0 14cAPD8SNMGK8h3G22DU3e/h9+eqjpsaBGIw3/qpl3ybi+qER/66VFv8ZbtyCgadix+wAFfDcBA KTn/Uh2gf3LTrXlDaFZG/cXJ+4QAsTUYMPOgpdNCaMef47uQUYsBSNB9G7kWGOacXWT9OKiH242 QR+q/F/h7zUSjjVs2VM90Mttf2051CUksVkSnbSSRKQJCDbs2w/mx1ylDyFNuTfsbP3jz75mtCw 8bybeR3fvLo741sE+s8DPIzmIh3Lox+7kGHuHZjvFET5H2DYN0BLia7quLSBd1Ji6jDEnxte7At E3D2WyCqfo56vU8UGzvdts2d613IrbSnSSanx3eFeayMlslmMfGECfXz8XkFCbGs5YAGw+5RtiJ Xp8LmpvhY1sa1l0W6YWZvq2KxnhAmtmKzb2fP6cnGCpqEUoZzYQJK1YPXMZP/8UOBemrs= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-9-samitolvanen@google.com> Subject: [PATCH v3 1/6] riscv: VMAP_STACK overflow detection thread-safe From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Jisheng Zhang , Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_125842_834676_A1112DF9 X-CRM114-Status: GOOD ( 19.36 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Deepak Gupta commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to `shadow_stack` temporarily before switching finally to per-cpu `overflow_stack`. If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because `shadow_stack` is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu `overflow_stack` and gets rid of `shadow_stack`. Following are the changes in this patch - Defines an asm macro to obtain per-cpu symbols in destination register. - In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel). Please see Links for additional relevant disccussion and alternative solution. Tested by `echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT` Kernel crash log below Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT Task stack: [0xff20000010a98000..0xff20000010a9c000] Overflow stack: [0xff600001f7d98370..0xff600001f7d99370] CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [] dump_backtrace+0x30/0x38 [] show_stack+0x40/0x4c [] dump_stack_lvl+0x44/0x5c [] dump_stack+0x18/0x20 [] panic+0x126/0x2fe [] walk_stackframe+0x0/0xf0 [] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]--- Cc: Guo Ren Cc: Jisheng Zhang Link: https://lore.kernel.org/linux-riscv/Y347B0x4VUNOd6V7@xhacker/T/#t Link: https://lore.kernel.org/lkml/20221124094845.1907443-1-debug@rivosinc.com/ Signed-off-by: Deepak Gupta Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Acked-by: Guo Ren Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm-prototypes.h | 1 - arch/riscv/include/asm/asm.h | 22 ++++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++--------------------- arch/riscv/kernel/traps.c | 36 +------------ 6 files changed, 34 insertions(+), 99 deletions(-) diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 61ba8ed43d8f..36b955c762ba 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -25,7 +25,6 @@ DECLARE_DO_ERROR_INFO(do_trap_ecall_s); DECLARE_DO_ERROR_INFO(do_trap_ecall_m); DECLARE_DO_ERROR_INFO(do_trap_break); -asmlinkage unsigned long get_overflow_stack(void); asmlinkage void handle_bad_stack(struct pt_regs *regs); asmlinkage void do_page_fault(struct pt_regs *regs); asmlinkage void do_irq(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 114bbadaef41..bfb4c26f113c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -82,6 +82,28 @@ .endr .endm +#ifdef CONFIG_SMP +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm +#else /* CONFIG_SMP */ +.macro asm_per_cpu dst sym tmp + la \dst, \sym +.endm +#endif /* CONFIG_SMP */ + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..d18ce0113ca1 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -34,9 +34,6 @@ #ifndef __ASSEMBLY__ -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)]; -extern unsigned long spin_shadow_stack; - #include #include diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..9f535d5de33f 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,7 @@ void asm_offsets(void) OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3d11aa3af105 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -10,9 +10,11 @@ #include #include #include +#include #include #include #include +#include SYM_CODE_START(handle_exception) /* @@ -170,67 +172,15 @@ SYM_CODE_END(ret_from_exception) #ifdef CONFIG_VMAP_STACK SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) - /* - * Takes the psuedo-spinlock for the shadow stack, in case multiple - * harts are concurrently overflowing their kernel stacks. We could - * store any value here, but since we're overflowing the kernel stack - * already we only have SP to use as a scratch register. So we just - * swap in the address of the spinlock, as that's definately non-zero. - * - * Pairs with a store_release in handle_bad_stack(). - */ -1: la sp, spin_shadow_stack - REG_AMOSWAP_AQ sp, sp, (sp) - bnez sp, 1b - - la sp, shadow_stack - addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE - - //save caller register to shadow stack - addi sp, sp, -(PT_SIZE_ON_STACK) - REG_S x1, PT_RA(sp) - REG_S x5, PT_T0(sp) - REG_S x6, PT_T1(sp) - REG_S x7, PT_T2(sp) - REG_S x10, PT_A0(sp) - REG_S x11, PT_A1(sp) - REG_S x12, PT_A2(sp) - REG_S x13, PT_A3(sp) - REG_S x14, PT_A4(sp) - REG_S x15, PT_A5(sp) - REG_S x16, PT_A6(sp) - REG_S x17, PT_A7(sp) - REG_S x28, PT_T3(sp) - REG_S x29, PT_T4(sp) - REG_S x30, PT_T5(sp) - REG_S x31, PT_T6(sp) - - la ra, restore_caller_reg - tail get_overflow_stack - -restore_caller_reg: - //save per-cpu overflow stack - REG_S a0, -8(sp) - //restore caller register from shadow_stack - REG_L x1, PT_RA(sp) - REG_L x5, PT_T0(sp) - REG_L x6, PT_T1(sp) - REG_L x7, PT_T2(sp) - REG_L x10, PT_A0(sp) - REG_L x11, PT_A1(sp) - REG_L x12, PT_A2(sp) - REG_L x13, PT_A3(sp) - REG_L x14, PT_A4(sp) - REG_L x15, PT_A5(sp) - REG_L x16, PT_A6(sp) - REG_L x17, PT_A7(sp) - REG_L x28, PT_T3(sp) - REG_L x29, PT_T4(sp) - REG_L x30, PT_T5(sp) - REG_L x31, PT_T6(sp) + /* we reach here from kernel context, sscratch must be 0 */ + csrrw x31, CSR_SCRATCH, x31 + asm_per_cpu sp, overflow_stack, x31 + li x31, OVERFLOW_STACK_SIZE + add sp, sp, x31 + /* zero out x31 again and restore x31 */ + xor x31, x31, x31 + csrrw x31, CSR_SCRATCH, x31 - //load per-cpu overflow stack - REG_L sp, -8(sp) addi sp, sp, -(PT_SIZE_ON_STACK) //save context to overflow stack diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index f798c853bede..a05905d88802 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -400,48 +400,14 @@ int is_valid_bugaddr(unsigned long pc) #endif /* CONFIG_GENERIC_BUG */ #ifdef CONFIG_VMAP_STACK -/* - * Extra stack space that allows us to provide panic messages when the kernel - * has overflowed its stack. - */ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* - * A temporary stack for use by handle_kernel_stack_overflow. This is used so - * we can call into C code to get the per-hart overflow stack. Usage of this - * stack must be protected by spin_shadow_stack. - */ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16); - -/* - * A pseudo spinlock to protect the shadow stack from being used by multiple - * harts concurrently. This isn't a real spinlock because the lock side must - * be taken without a valid stack and only a single register, it's only taken - * while in the process of panicing anyway so the performance and error - * checking a proper spinlock gives us doesn't matter. - */ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack(void) -{ - return (unsigned long)this_cpu_ptr(overflow_stack) + - OVERFLOW_STACK_SIZE; -} asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk = (unsigned long)current->stack; unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); - /* - * We're done with the shadow stack by this point, as we're on the - * overflow stack. Tell any other concurrent overflowing harts that - * they can proceed with panicing by releasing the pseudo-spinlock. - * - * This pairs with an amoswap.aq in handle_kernel_stack_overflow. - */ - smp_store_release(&spin_shadow_stack, 0); - console_verbose(); pr_emerg("Insufficient stack space to handle exception!\n"); From patchwork Mon Aug 28 19:58:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13368281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C967AC83F11 for ; Mon, 28 Aug 2023 19:58:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=/JSy5Tp1r36sUuf/MMWWq+wNz63wBPS4zmlo9d/1aOE=; b=kk0CvCGtGerhA/Fm3SOmnBak14 RPGF+HBZCAOscUX1MfQTmWgfdtheGRkz4A9F7g1vdEwDUq/dI93OF6/AugSdaI51dw05Md7COlqkb qw3RXBJsDZ8pYBM7tSaAKJjj5Aqk7bEWGFxN9aylmRiD5IopeMtvamux3N9PmPNZnElk9XOp63aZQ VlPGfuSa157YqdXGPl0WIr446wDnBzEtMRc3pBUQ2GZV28RbYUP3F9HkTYnhXQtRZDKvX4yQrqzGB dQ5VsXenCDWpWq3urHTBNFZnI45RG2e2Htq1g3Eej1Yt/lszxqYadKQqULkFx8/um2PWcN3D5cxqE BvBF8j5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNu-00ABwC-1q; Mon, 28 Aug 2023 19:58:50 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNp-00ABu1-0o for linux-riscv@lists.infradead.org; Mon, 28 Aug 2023 19:58:46 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d795e8483f8so5283663276.1 for ; Mon, 28 Aug 2023 12:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252722; x=1693857522; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SyPtEI03Xg1fBX3TZi3WfjXMOPTYu0dO+LXPL6FV3JY=; b=aeycJH5+6uxe8GlNC+YyRdXkjill87qRi7L6FkS96p4g2q1s4N94P2OZ5QDES1S1TC MfwdeCDtem0I3jQ+onVBjqfV8svuERSpLpxq6jjZ+Ufji5mcFbBp+g6ChKwsHhVNFNna LQonssHhBVrDczuTEO5l5yf74aTZFJ60kJfW3XcL7yn1/dPDDGFUudDGC9NGcXlPudY4 BuAw3Yfd3OAcCpnpMMIFLgvMXpik/3/GT/wQbEzdRIYCNBGI57sXpX998s0bzbLp+XDr +aCFpNwXtn4mlGFVj/IbkR53Q4xdpfP+oJruefxqPB++Dgcy9SSWoWGeTGKWxGTCKkQQ XlFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252722; x=1693857522; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SyPtEI03Xg1fBX3TZi3WfjXMOPTYu0dO+LXPL6FV3JY=; b=EZ89bbSkpdKYBA+wv2ql0vTtFlJBNrHqSypzdC3SHMoIfVJ3RXBDQR6YPJcwmOWdRJ CSISD/ldQdm1YHMw6Ga9VwZHYMck8d5qFMRMKwQ7djjFSzgpRlAfev6YkBz1oY5VI6Ea DXVVA33MXda8UC1x9JdeoQvg5S6D2YxKY4BdkhopvzDLKlZ2y4bNXmXvkQOXsBEjnuxk pV2NnSWQRmnvGPvZAiky3lZK3fb4Lv7p4/mnKWtqoRr4sQdyrHFWF9dkBMBz2I4NCo2O /u2oog4mLwk9EcvMejspW52aj1wB86Yr++80GR+SqS8Uw99CoLiCWFBDIP7/ipi7EWWQ lTeQ== X-Gm-Message-State: AOJu0YziZpJMjGiyhlGY7bp4ckqPUdx7ir5D3FT+Br5+37oJZQerh2HJ J9SoB6WIxW+HhaeSfJMfeNzJh+7GNsTkcomU0Hs= X-Google-Smtp-Source: AGHT+IFJVA5tn0i/teRMdLPB4BUylObsezBXemw+f2wLcIS31CUsFOoZ7JWpR/ILow1bMm25R+eFhC5JiSZJ6WzUrIw= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a05:6902:100d:b0:d47:3d35:1604 with SMTP id w13-20020a056902100d00b00d473d351604mr24925ybt.2.1693252722088; Mon, 28 Aug 2023 12:58:42 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:36 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=6654; i=samitolvanen@google.com; h=from:subject; bh=wnBUbn/4zTqpJwVvv6oK/jRIJiyG33tTx6IcCOYR/uQ=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqgcThGLqOU6/Nh1FKa2ioMm4jOSxmZxX+a sZvOfWhvQCJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7qN/DACO5167AKEHcx/87vqquJtHBN9KGNAuOkAYwEvKikfyASynXP4pukx0I5Ac1cOyZbdSKL8 Ce0qro1KPSj+B7uJLugyfen0UVTnlQTtNm24YRU2AnuqLCWm7zflBibZwWzhTqJsGZzImyJVLkB Yr3GszptMkh3O1gX6Tv2tmDjb+qkBlUMMRdltW45f/g9ibmbfcG5SO/vWQqcc3SMWqsc6m7JYwk keXj7naHVlBrPYSo+984uuAqdsHZ9q0aEVwqYIJOJIWhP5XgrEZZe3AJIp8OF8ZaefGmW5LiQ0I ixCukGxRFhOHpj9R9wdn6BFhLwcyzYzLwTqRW+Y7ko261wZrwLuApeBe44ru+05jqQ5SVj+y501 vjpXz+IlnWoGzvDP7e5QFY1rNNe2X4QdPODLjs6C/p2TfoPH/k92JNa6S7ITaa1o90sUmL9p4bO NCj0TY91rJ1SesDvIqMSsrklIMv4du5khyVUbh1YZyymDHBkAXomC5VVAKeZHqxyNk6oQ= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-10-samitolvanen@google.com> Subject: [PATCH v3 2/6] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_125845_290121_E1573B96 X-CRM114-Status: GOOD ( 14.35 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor Reviewed-by: Guo Ren --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/asm-offsets.c | 5 +++++ arch/riscv/kernel/entry.S | 30 +++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 35 +++++++----------------------- arch/riscv/kernel/traps.c | 32 ++++----------------------- 6 files changed, 55 insertions(+), 55 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index bfb4c26f113c..8e446be2d57c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -104,6 +104,11 @@ .endm #endif /* CONFIG_SMP */ +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/irq_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ DECLARE_PER_CPU(ulong *, irq_stack_ptr); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 9f535d5de33f..0af8860f9d68 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -14,6 +14,7 @@ #include #include #include +#include #include void asm_offsets(void); @@ -480,4 +481,8 @@ void asm_offsets(void) OFFSET(KERNEL_MAP_VIRT_ADDR, kernel_mapping, virt_addr); OFFSET(SBI_HART_BOOT_TASK_PTR_OFFSET, sbi_hart_boot_data, task_ptr); OFFSET(SBI_HART_BOOT_STACK_PTR_OFFSET, sbi_hart_boot_data, stack_ptr); + + DEFINE(STACKFRAME_SIZE_ON_STACK, ALIGN(sizeof(struct stackframe), STACK_ALIGN)); + OFFSET(STACKFRAME_FP, stackframe, fp); + OFFSET(STACKFRAME_RA, stackframe, ra); } diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..a306562636e4 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,36 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -STACKFRAME_SIZE_ON_STACK + REG_S ra, STACKFRAME_RA(sp) + REG_S s0, STACKFRAME_FP(sp) + addi s0, sp, STACKFRAME_SIZE_ON_STACK + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -STACKFRAME_SIZE_ON_STACK + REG_L ra, STACKFRAME_RA(sp) + REG_L s0, STACKFRAME_FP(sp) + addi sp, sp, STACKFRAME_SIZE_ON_STACK + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index a8efa053c4a5..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,35 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", -#ifndef CONFIG_FRAME_POINTER - "s0", -#endif - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index a05905d88802..1fe6b475cdfb 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -350,34 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state = irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", -#ifndef CONFIG_FRAME_POINTER - "s0", -#endif - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); irqentry_exit(regs, state); From patchwork Mon Aug 28 19:58:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13368284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11DE7C83F12 for ; Mon, 28 Aug 2023 19:59:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=+bQg87hIrWQCKvAcslQzH+SuPEjYRfVqHebA/IM2Dzs=; b=fcRvRmKgRywGstvNj9g3fOv+JD RC9pJ7wz8alQ8DPng/4yvH2Nwri8DV6Okxl2yYktf3URrN+3c0ytFyY5yvlQsHVJbXuAQpkCdJUuW NU8fVxgz7hbFXCupPKn53MS7bZyXrFH2HSXVgOT/2Sswdk/uDiA3wGLeOe5kd11OP0nABUHAyI12p TzA0uOzNFHFPi5ruEIlqXCBaOJSnpg8kmBbAmxZXdx/Lth17qdWJZ9JRmmSKE2pG17d0dY+hiPE2A IsUr6Dvx4ioWqRbNX/BoetaERp5qhM1xIc0/rGvpuKjzHa7fjFtznMj2ElPsnplpNsOTTEU4uXNZn D2Wjwm2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNv-00ABwe-0a; Mon, 28 Aug 2023 19:58:51 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNp-00ABuN-1j for linux-riscv@lists.infradead.org; Mon, 28 Aug 2023 19:58:47 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-d7493fcd829so4440078276.3 for ; Mon, 28 Aug 2023 12:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252724; x=1693857524; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GN6X1nlK+F0kjkEC50FG9os31FAuUCJYmCM49+1obnI=; b=q81rXqLshiF0sln/xS3h/1Wt74xT2Rdg2j7FkroNKWNJx+s5YHbY+oxX80kCoOEdXg Z7tbUBEmnha1x55o337Yu6Lp+PK9LnYIpjNDnXpf7Vvob1fBYdUFHdVNRYzgfh16aSLv eFNKGP5rRTkOAMSM3mrjAPNqs5knGLIBbNRZAC6qGU+CNi/lzLXAiQhO62xa0d/tNvJ+ HZQCnftScs5HC5H/QHtUIo0uBGDaDIJXYW2D2SPFvrSpGGn81MfWM1OEFhJRudgn2+le WexQrz+RIoMq4d/FZyHuGt5KMF8ZH4GobjeJX5y+t0Ma6T4e2S48xP0NJhR5CYYZChPH OKqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252724; x=1693857524; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GN6X1nlK+F0kjkEC50FG9os31FAuUCJYmCM49+1obnI=; b=MFXYZNQ2Q0I36eKM4iN94PpHYallDxRWzNQZmdVMdxIVs4Ho3ONUcP4J5bVOcJcief fKbNEaLjfSSQ+lI09slu5Zdbru0Ape6tthIuurasFYLxnK/Dh/lQhl/kHxqBWZpYGOC+ 1DvOjqtJfKoa4DKpvBCs4S7UekX6pFVca0wP7wEOpZu+ldQBnzDTL7d1QvCye6Hki2q0 CKj0KaxIR2+j8+q+J8GobraowTMaFCO73Y3p2ks8n4NYeQzppfOupWT7RLhWbcWcrJP8 8yAbxzPWFuujU5ShYZVZP5bIrVeua6AuwXtNsolWEtSFzkCo8JWyM4lu8a5eluFJYBG0 Go8g== X-Gm-Message-State: AOJu0Yw9muDPgjH19Ru4mxmwRfGS85soROopr3tHdpu83oeJO2fUOrzl qEAArUu9aPDxomoFPzL2Vd2gHViX2s8AOALcgJM= X-Google-Smtp-Source: AGHT+IH/uHwJj6+qOeylfRyEPqpB+nnMtItRxR7gf+ULc0gWYu3yZPJVS4kkyblR9fUvFWo2HvJU3EuGasR5rTd2B1w= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:9011:0:b0:d5d:b03c:49b2 with SMTP id s17-20020a259011000000b00d5db03c49b2mr820198ybl.11.1693252723904; Mon, 28 Aug 2023 12:58:43 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:37 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3271; i=samitolvanen@google.com; h=from:subject; bh=Chgku5okNNw1TPF+7h/tJxx5VnEIks6hZ2ypKsZTKYg=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqRbtkifAND8x/XfLzfQWcnICH3txr02qPt 4eiKf2aYjSJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7sqPC/4yfD8F4LJhON/0j9LDQmdASW7omk5GP9QTz9u9V2kMfasvvS5/J6shUYxruhqnrbLQBUp oZG+GjK1FZXYmm2NhWg4FLwoGp73evhYqS3wI+Ft3xabSmr5Na3lU51SZXDGqV8uyQ7euwUibuM 1IB+PWku6qCOxTHa+PmjyOlXV/KauzFtIzwmz+0x5d45qcRehgOD29B8x93CCG4ocGo16RTP8LI F/iGt9KvESUFJHI8k0VjTovZNvf6YxjaSUpyeHxe3VVQSccGWV/T3h55eOORae0j0Myri46bWi6 G6OIOjCjM5wvPYvwQTVyYFnbv5wTS0DJqhUWZt3ZEVviB/dFtXNBfxwtPWVuNItQMaZQ/8iaxqH ZZt1G2FWe/DvsQ5lHk95L6xpiDHTU8W8m68q4rh/0qju4f8K6jxJcN6sSbo2s98UyF1YVfErhSD UX5opXBgLT/rq9utajC5hslrYEgFPNP1pUHjVWRg8zD8f17xYD7/57e0gz+NISpgLceIs= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-11-samitolvanen@google.com> Subject: [PATCH v3 3/6] riscv: Move global pointer loading to a macro From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_125845_571502_FCE6A3AB X-CRM114-Status: UNSURE ( 9.98 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In Clang 17, -fsanitize=shadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead. Link: https://reviews.llvm.org/rGaa1d2693c256 Link: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 8e446be2d57c..f34dd1a526a1 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,14 @@ REG_L \dst, 0(\dst) .endm +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index a306562636e4..6215dcf2e83b 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 11c3b94c4534..79b5a863c782 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Switch to kernel page tables. A full fence is necessary in order to @@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of @@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of diff --git a/arch/riscv/kernel/suspend_entry.S b/arch/riscv/kernel/suspend_entry.S index 12b52afe09a4..556a4b166d8c 100644 --- a/arch/riscv/kernel/suspend_entry.S +++ b/arch/riscv/kernel/suspend_entry.S @@ -60,10 +60,7 @@ END(__cpu_suspend_enter) ENTRY(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer #ifdef CONFIG_MMU /* Save A0 and A1 */ From patchwork Mon Aug 28 19:58:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13368282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3D665C83F12 for ; Mon, 28 Aug 2023 19:58:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=wwaVlWg7wacduSBTt607Sm9wGG3wsE8UUwg29s9o1AE=; b=ngoNB4h/EBG6RLqQEBHsUm5iYR MKR7/RWl1uC8E/EJCtPmPonkufQvPoH9zjN4oe3mF9CoHrFsM4qetr8UWmkF6Sr8CXC0a5QpzC1Nr ymHsO3ume0yGDovyLO9PZLeJdAnh5gyosl4PHonCgfQUgPP/RpeyUMg5J2YmiR+InY4utBjtwQCB9 qgMDNwMPeO7LAGk2WteitXUwhHYCeeKJJarm0rKxcb5sz2hMxqOp4JBnw+43fhWI0L7ZmFpbjMxpd 2IVxrai0rjOkEI7iWmFUF+CPuhfljDt/KrqJSg+RRcjdGXG8unHIEoeAmtDIWRZkg+FWyODXEec2P PLhBTDaA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNv-00ABwy-2V; Mon, 28 Aug 2023 19:58:51 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNr-00ABv0-1s for linux-riscv@lists.infradead.org; Mon, 28 Aug 2023 19:58:49 +0000 Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d7493fcd829so4440100276.3 for ; Mon, 28 Aug 2023 12:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252725; x=1693857525; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aHEpjDWpZKheS5AgI4FhMURxdLRiA7f+7eI6PSwoShs=; b=oU7u4yIwCqEfIEncvirifpc8jGp8C2PonLKbfLyys1pYIlGszZDoAooaF/rA0xipsa tGEdMGobqAe00UCFtvQUmb7m/LoLfLDhGkakB7CJCf2UzXywKzO4OGDJx36TWR6p1UpO gH+CFvy2T/d4aEliQy8VqP+pcWw0DQUKphP1cBN2b9Cr6KoXrSCWy2xSHa4O70TgLTtV kcouhP3xTN66BUOTos30E+RKqDSHDRJfp+8zYKZh20qXio3GEQ7TFh7/q3ws6NvVxZiO 9DkaSSJqea9knPB9Fi+KfetI4RjF02r8Q43oiZstz6SsA+TRNeaIoRtwg6FkO64uo7DC 3gOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252725; x=1693857525; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aHEpjDWpZKheS5AgI4FhMURxdLRiA7f+7eI6PSwoShs=; b=HQJ1wXbShXCQibE4LfMjOrrxPCR3YOUxDMo1zEfVDHGhwwamHRIkRlOqri0ZAkWluA rse4pbWeSUibfHj9KzhGy8FwM/pQ7gZv1zIcngXT9CkC3XM7uPKa1YRxLk0cwtTTYuQV ILsZeZZg4bp0PXZk7lMwMCWioA344c5eOxj7N8tafc3m0B4K75Cgsh5axNTphfhj6wq6 jhiWkdq65MHZcONcy5hjDen7MHLYrLxb6YULEiu/YQX/00tXBscJbN+/sWg5Cgp34PUb VVuK8gHKu7IGJPGOkoUDGmxp1VXa32u/f1vEPnhO8e7aZi3C7PjGYH53xZwws2Qd44Wb JMyA== X-Gm-Message-State: AOJu0Yx5bdfXPbvIhRz64R+5UMaH+F4QOcfsVCWSP3xJTQ0lpwMC/7hX XS0fRvqt1px3HN013wWr9U5zOHW7W+UkooKA8CQ= X-Google-Smtp-Source: AGHT+IFPe1xW/cZcqMnEmhJXa9gkio6zfEjjBVIssP4qY8EDsEHxCzdXB9iRA0bQANGRB+YXwQYzP941dcW/ssXB124= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a25:d750:0:b0:d7a:bfcf:2d7 with SMTP id o77-20020a25d750000000b00d7abfcf02d7mr386698ybg.6.1693252725580; Mon, 28 Aug 2023 12:58:45 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:38 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=9699; i=samitolvanen@google.com; h=from:subject; bh=yzq9X6lovJwCj7M3MxLTb+ZbxWOKAC+68l1vG/LS5Go=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqnP8GY39pSf11gSk0fB0bvEpXRF2ZEN/E+ G3d+76nn0OJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7gsfC/9z0ZHF2b9t4rZr3ySGuRBhgmJMEDj1qu+NhecypGTqBV+8Cf+bL2247doUWGkApt+72LI Pie3DKwp/m/VGmPDKi5q/CHYiJLP8680OGP1+bTrgYosZW8hyJ9IhiKBiO1KBRy47o/T50jlhIc VLTG0/pv3Y+jeXP0qgdkSrHZlTv7QZPhn289pObdT/2Ec0VHRVw53yDsfnirmd1Z2H+N5PkP/6f 1xreDmv6SydopVQvfabiJYbA2ISjihl6rFRDqU2ArU8zwBiuubsGlLXzPpeoKDiqvBWN86lYmXY X2DoWBrpXBLJwxNGk+gdnntxZXM5+hHaCh1D1a5jW2C9OajMf4lqHP9lG/XUDynL1Gmxxrid5JG xgY811eAOdQva10xkT5N8Zv/OzuURC3GqI2P2Dhr0c3G4WTomuJ13WFUOPg5QDhJsU3a5B9DZv/ f1ZMyp4Vr2IT2tV5madananCzjiSG8k+hxqtfu1tvhCKQ4SOJW10fDi80IvVzGfHZqXy8= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-12-samitolvanen@google.com> Subject: [PATCH v3 4/6] riscv: Implement Shadow Call Stack From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_125847_623327_FD242F44 X-CRM114-Status: GOOD ( 20.85 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Implement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example. The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bea7b73e895d..93122634f9c2 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -46,6 +46,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU @@ -169,6 +170,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=8) +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=shadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8 diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6ec6d52a4180..e518a74640fb 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -55,6 +55,10 @@ endif endif endif +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS += --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f34dd1a526a1..b0487b39e674 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,11 @@ REG_L \dst, 0(\dst) .endm +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push @@ -116,6 +121,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h new file mode 100644 index 000000000000..94726ea773e3 --- /dev/null +++ b/arch/riscv/include/asm/scs.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* Load init_shadow_call_stack to gp. */ +.macro scs_load_init_stack + la gp, init_shadow_call_stack + XIP_FIXUP_OFFSET gp +.endm + +/* Load task_scs_sp(current) to gp. */ +.macro scs_load_current + REG_L gp, TASK_TI_SCS_SP(tp) +.endm + +/* Load task_scs_sp(current) to gp, but only if tp has changed. */ +.macro scs_load_current_if_task_changed prev + beq \prev, tp, _skip_scs + scs_load_current +_skip_scs: +.endm + +/* Save gp to task_scs_sp(current). */ +.macro scs_save_current + REG_S gp, TASK_TI_SCS_SP(tp) +.endm + +#else /* CONFIG_SHADOW_CALL_STACK */ + +.macro scs_load_init_stack +.endm +.macro scs_load_current +.endm +.macro scs_load_current_if_task_changed prev +.endm +.macro scs_save_current +.endm + +#endif /* CONFIG_SHADOW_CALL_STACK */ +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index d18ce0113ca1..574779900bfb 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -57,8 +57,20 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + void *scs_sp; +#endif }; +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base = init_shadow_call_stack, \ + .scs_sp = init_shadow_call_stack, +#else +#define INIT_SCS +#endif + /* * macros/functions for gaining access to the thread information structure * @@ -68,6 +80,7 @@ struct thread_info { { \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ + INIT_SCS \ } void arch_release_task_struct(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 0af8860f9d68..a03129f40c46 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,9 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK + OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); +#endif OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 6215dcf2e83b..52793193a763 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again. @@ -275,6 +282,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) @@ -292,6 +301,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to) diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 79b5a863c782..c3d0ee77483b 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "efi-header.S" @@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current .Lsecondary_start_common: @@ -293,6 +295,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0 @@ -311,6 +314,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_KASAN call kasan_early_init diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makefile index 6b1dba11bf6d..48c362c0cb3d 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds += -DHAS_VGETTIMEOFDAY endif # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) # Disable profiling and instrumentation for VDSO code GCOV_PROFILE := n diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index dc20e166983e..d5d60c040560 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -77,6 +77,10 @@ ifdef CONFIG_STACKPROTECTOR_STRONG PURGATORY_CFLAGS_REMOVE += -fstack-protector-strong endif +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE += $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o += $(PURGATORY_CFLAGS) From patchwork Mon Aug 28 19:58:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13368286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35B95C83F11 for ; Mon, 28 Aug 2023 19:59:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=HqqXWfw1fd/q3qTKAO+sWfU+Ac3E6i5K4oSaG233bxA=; b=iZaaqmpcCObFUZElEQMGFQ4opl TWyVR4tOtHRDyhwV1PumLoHD3E38xKf/IzcTOUwtrIMtNAwkR1DOyrq1W1MQb2Z+MT5DDWSh74RjW vWZkCXF83UfYlQnMVYf5KMisUNgAEcZPcA77ZxTD2QGDBRZkyKc1VcDS11DraOuFmUFF/4gmN0ZMw qGmaAGNdFDCaL0/9TZOjfh8LeGdFFotHMdf017yjpflXlpcv9v4S84TxyeXn8THQSOWcXQE2UHbWM IJcg4qwPKSorcAMQO4DV60MSqHhL2bOS79vx7anY4PgwpUO1+Ycov4JIyoDmZ9HSmC6T85NxCi9Wp 8mDE+9bg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qaiO2-00AByV-0I; Mon, 28 Aug 2023 19:58:58 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNu-00ABvk-2P for linux-riscv@lists.infradead.org; Mon, 28 Aug 2023 19:58:52 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-58d37b541a2so52899187b3.2 for ; Mon, 28 Aug 2023 12:58:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252727; x=1693857527; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=03jC+OBVw6NtqXGsEGvclVCdQhu3EQx6jlJc9yvphdo=; b=E+bfdH1lDaLKrKcjWlt3v7mLrI0mXX3/qYWVcii5PVVLZRAYiy1nsib+qeG6H8MOuT 9OhEHYpNgvWxBky8trYBSEV2JubnwBjeu7di8zHvRoZf0j/YO3KJeryt/leOQaOk8U0+ E0YLLPMtCwqk+7XZu6/A3/70OtYGnMDc9dICr0zRd4xccIbb8Fl0DxJGU04K1ySPvtRA rOrJmojR9TLloOAjWkHIpTDXARYBxhHwOkJ5h1zSqKkVSIGksfJHQ6WU6QjJ2FOyH2nJ Z25Phbti7WwdNvRX5aYN5oYcnIcIxNODEPr4eLAqGQxyHCx64dEv2WvC0HWZsxOdspdf wAyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252727; x=1693857527; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=03jC+OBVw6NtqXGsEGvclVCdQhu3EQx6jlJc9yvphdo=; b=OdYq/MR8bmEgRxgux9JssChCiToTp0b1e0E/VVrwO899zw3IHUOfZjMt3jg7mELn9H OHiurX+DTBZJxwUm2OWYZlulM0p1EOe2tB9EdO8ljwxDgFDoYdUMNJUVmMjSzI3OOdWE J6fGIY3x/Sgi75Ps/bzXii5zUFo8PzqKuCUEJmTDmGBASHGmi23ExDFaXctYWrYJEHLg Y3OVUkhinUfm25wvHLFVbCreid0Qpdx8dFa+oZMeNz1+N8ukYjomAMidvnG2NSbg9ZB1 oivZtmv5Zvd82Jlt6OwJIaTAWX9XiHJbMoGVmKqeCApLbUmwhWp1AMAhJiysyYVqQ8DW hzAg== X-Gm-Message-State: AOJu0YwTGwLr8HicZFc6yAMFx+f50CK61iCizhMer1a8bcynG9zJ40Ht eFA8oqUj4sHgYNSulKzzQljbJfeIAZRPSQC0eBg= X-Google-Smtp-Source: AGHT+IGX3a8SAyFuYllCKEJp1p8BAuBf4bX7iN13I4toDObzdsg0UQOHsOqXYzlSHWKRqNRQ4bgZw4/tHZOh9SXDs4o= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:a904:0:b0:576:8cb6:62a9 with SMTP id g4-20020a81a904000000b005768cb662a9mr947313ywh.6.1693252727342; Mon, 28 Aug 2023 12:58:47 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:39 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3226; i=samitolvanen@google.com; h=from:subject; bh=/mGBqVAJStUatVnFnltMn+I7N64iI2QEtl2iB/CZBSg=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqQdVKNNuAiMW7udPNqMcNpKCbHaCzUHSyh KrgH39THuGJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7guJC/9ByIZj6HC9xt8M48iEuc3eleOXi7LLtLxGuUV5BC0aCnuet8/CP1zkrGWtv2JZ39aif62 X2Lq1gXUXz4nIi7rwmDEz0XjtrBeZtsI5xghAkZCCubYNPWatl0y+0UTL0qa+VuYO+MUp7CwJ4q 19Any1fNwFxVAXnOTh+0UAcjst/pmBxL9W1CG4VnyA9NHMX3EXKgrQw1SpC0g1+mUut9n6bcrJr W0ppsOU/Jqpc/8Xg4TgExx30gJgt1XpXcZSyNaDZk4jXED/GJ0cOupaFz/3bhVzR1J6/y9z3y70 b9ZHkq5uFdJ6LPJDcdccev+aWmRxrkaw1vZ/4ksObnxApxpaGmFk+kc09Q4+Ab83M50knpeUZkB 2lPPYgLx0SeP3+XQdiJywRmtKnipviQqot/8K0XYRWNukxdtd30QBKhYtlFI7f5uDYw2bpedKeO 5hPzmPQaQUqbOlLYrH/VXQIVhJ5BZDRq9B+djftJejiGDFObOFuxsPk6QpMbtOq6LFIVA= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-13-samitolvanen@google.com> Subject: [PATCH v3 5/6] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_125850_790085_8B41EF4B X-CRM114-Status: GOOD ( 12.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen Tested-by: Nathan Chancellor --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 52793193a763..3a0db310325a 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -237,12 +237,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, STACKFRAME_FP(sp) addi s0, sp, STACKFRAME_SIZE_ON_STACK + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -STACKFRAME_SIZE_ON_STACK REG_L ra, STACKFRAME_RA(sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) = + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) From patchwork Mon Aug 28 19:58:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13368283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F30B1C83F11 for ; Mon, 28 Aug 2023 19:59:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=9uC6C4yvlAohCkqO49wNfIOXNSt0umaBUebIZYjKlLM=; b=zlT9NfwnonPQELdXx7y+LkTgS/ dt7HVcU0SPX5Cm9j8eVE4Cl22PiyaR1+QMPWXwUoGFnC/kTOWlt/gxQ6JxpfGcaP9i2XOUYuX/65u 26Mp08roxlpm09Hm6E1M8i/HLmnM2HRuDOlX9WbJaz6fei5h8hvx5wyFJjAQuQ/X3Reoo3e8VJbZC feMnjnmqvxjH6ycFIk7pTPhVpbFx9c46Wo6KVMyIfY9s5qxw+w//7AU7jFeBTSRtkTjkhnQgRkkJk w2JJ4JbybcWZjTpQ8OHpXP8SKfnDfXjLOyb4kpXJLBzUsRofk9iAKeTxaLW5ZG2qcI5vh4hZAuuLJ p3pqAdGw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qaiO1-00ABy4-1i; Mon, 28 Aug 2023 19:58:57 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qaiNu-00ABvs-0R for linux-riscv@lists.infradead.org; Mon, 28 Aug 2023 19:58:51 +0000 Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-56c556b5aceso3537976a12.1 for ; Mon, 28 Aug 2023 12:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1693252729; x=1693857529; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BqxXsJhK/PDnu5QKzOuhVQVuGBQ7YV59awRNcw8M9KA=; b=5vbcngL8ZVHRFE2eljBR/WQoqnwHuLVs7upb4C37eJWQbZVjr56f/ur647yJ4AUyqU odlSa8/RjIkDV07G0UBaBb4auy8DU4amPuju8M5IQqLJYH54J4qila2NhRd+E1O4RyAM uVYOKvozn6NT8UEW8VWKOZJe/DKWNoXAIn9cnQYVR4AleyEg1/b63QvqUxI0dcSLbnMx XLuUylSfyiMonef/eNWAAMq+tnS4EWsKMBu8wIhTW3R0YLJ6tJf56CDFzDgaITSCuq+s BLuYHuv7RfEgVzf9k8LBhuxRvd2HXWmtf2Vk/OTijRK93Crc/qqwJGPUvgiTw7PzvnTZ ll/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693252729; x=1693857529; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BqxXsJhK/PDnu5QKzOuhVQVuGBQ7YV59awRNcw8M9KA=; b=FmleVxaPHMMkUEB5761zKVSHf0+wxOckMuIpoKZpvvm3rYtZ0S0ffMC2TjKt9OSZRz yMaIaHmASnLD30BPTNXfRPU+6jvqyjJCCiVU8TS4znvqQymG5pyaLmhXf8y0z/7MZ/iF UfhamrRdRSb+6VNhuONvMQgAUw+c5agQC5vzIU8hRQsqF+XUQqLsHoFXiDuxy+gVMFw3 H587n11gDOpMJePKFvU6zuYa6b1nTCoQHAgIds9kGS2mWfqfYPW3rslWQ3YRe3z+VB9y cwaGlXLSTzR+f/DPxL79Dr74PmuEYQFjJ5J2XEOurpnCTbiLp/GzZWb5ZDVe+Y9DQFue DXjA== X-Gm-Message-State: AOJu0YypIo2YVb17INTno778qKMW2HVZaVcFdDchAwt0LWZZ9/uoLcwK FsZ8s2uoWtSA6+2M1eJQvyfKfsfykCKVmaykTOA= X-Google-Smtp-Source: AGHT+IFGsaaO0XSM/Y67KiglbiGwn9oCum6xrnQEFf9Pggoc6RCuQ0rJ2Xb5U12wKvaEKKMsCRfVVSyEDOKWowHVSW0= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a63:9351:0:b0:569:52d4:fb68 with SMTP id w17-20020a639351000000b0056952d4fb68mr2368998pgm.5.1693252729217; Mon, 28 Aug 2023 12:58:49 -0700 (PDT) Date: Mon, 28 Aug 2023 19:58:40 +0000 In-Reply-To: <20230828195833.756747-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230828195833.756747-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=2086; i=samitolvanen@google.com; h=from:subject; bh=P2l56HkzVV4iJDbhsS59ZlfJgm96l/buqsb6+trKTfA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk7PxqKl9k+yTdNALVx1ZMSiIiYkPp3ifD0Ueug LDeD+59dp2JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZOz8agAKCRBMtfaEi7xW 7ombC/44I2N+oNE0mhFWTI0lDlgL+4/JMQhfMKSufRM3U1T5OqabvtPwFHQhOXRz1TiwEcDr7i8 5yQiZuqOXgg3iZ4h+9F4cRzw28ArehJIEGz+O7FO70FgOBNXOSHaWGcyv7VuHdprWhST+C1v9Zk ZgslARn9fBw5HJwmwr6bN8M9im4r9jKtVsPr55L3xnUWbk+knu12/aZpE/GGE0bbf3QK+N4vt3A 0nNh0iiFlV4YayOAPFabrTDShrkvvWlE4rzF+fG11hd/h8APu37zoH57B2AywUWNj14p83mXT6k u9y2x57j40yu0iK/b1JflZvDK7CJ0s8ajtQhMOXhjDmApgyeWZVLMLAZTpKv4gs/oO2xodXiw4R ksl1GmDHjdIlhyeMTWWtCCcg/DIiAd6f1B740jP4kkXUDiP9Sy3P/G0MHt1K+RGieGHpaB5B13/ UvPppmx2PeR8h40O/fHIATSyyElr9SQ8Dg84t2Fd6nUgiqKsWp+OrKSdL0qF6hmaWxQ0o= X-Mailer: git-send-email 2.42.0.rc2.253.gd59a3bf2b4-goog Message-ID: <20230828195833.756747-14-samitolvanen@google.com> Subject: [PATCH v3 6/6] lkdtm: Fix CFI_BACKWARD on RISC-V From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Clement Leger , Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230828_125850_177430_2D61456D X-CRM114-Status: GOOD ( 12.45 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On RISC-V, the return address is before the current frame pointer, unlike on most other architectures. Use the correct offset on RISC-V to fix the CFI_BACKWARD test. Signed-off-by: Sami Tolvanen Acked-by: Kees Cook Tested-by: Nathan Chancellor --- drivers/misc/lkdtm/cfi.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/misc/lkdtm/cfi.c b/drivers/misc/lkdtm/cfi.c index fc28714ae3a6..6a33889d0902 100644 --- a/drivers/misc/lkdtm/cfi.c +++ b/drivers/misc/lkdtm/cfi.c @@ -68,12 +68,20 @@ static void lkdtm_CFI_FORWARD_PROTO(void) #define no_pac_addr(addr) \ ((__force __typeof__(addr))((uintptr_t)(addr) | PAGE_OFFSET)) +#ifdef CONFIG_RISCV +/* https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-cc.adoc#frame-pointer-convention */ +#define FRAME_RA_OFFSET (-1) +#else +#define FRAME_RA_OFFSET 1 +#endif + /* The ultimate ROP gadget. */ static noinline __no_ret_protection void set_return_addr_unchecked(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store. */ - unsigned long * volatile *ret_addr = (unsigned long **)__builtin_frame_address(0) + 1; + unsigned long * volatile *ret_addr = + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; /* Make sure we've found the right place on the stack before writing it. */ if (no_pac_addr(*ret_addr) == expected) @@ -88,7 +96,8 @@ static noinline void set_return_addr(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store. */ - unsigned long * volatile *ret_addr = (unsigned long **)__builtin_frame_address(0) + 1; + unsigned long * volatile *ret_addr = + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; /* Make sure we've found the right place on the stack before writing it. */ if (no_pac_addr(*ret_addr) == expected)