From patchwork Tue Aug 15 20:34:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13354277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4B9B4C001B0 for ; Tue, 15 Aug 2023 20:35:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Y74IIQtlYEiX+hBZTRdNAwV+G63NJo2VT+LAefPpt5c=; b=eIXSOSsMW7pHRBgsQzOwoA9WEZ W4fYHMkgyaHM1WlU7qTeQMQmDnLn9qaADD9QhoO6mZxwdzlt8HH8OELy4SZFybP6CY/fai24ZDUF2 4wb2dU2zR/GQ7BKXtAaPs/6gdX944jyj8rHo7uNyplX72xvxW02FWTY1RrsZ6RXSvgkAebITxEtED xM42f9bzlqxZgvvzNgpMA+F+zagMjVoj40MhU+tb0tClRCxupQ5h4gfz0O8YXKtTaGNjcV230v0HI adWdvHD+Ek1ZTIooP3JM5q16KLWxjgMkMqDHVD42ytCezzX31YxoX31fIV+LRln8ONUaw+io2H/E3 g2tXBQ1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kh-002QPS-1f; Tue, 15 Aug 2023 20:34:55 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kc-002QNv-0v for linux-riscv@lists.infradead.org; Tue, 15 Aug 2023 20:34:52 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5840ea40c59so75506517b3.2 for ; Tue, 15 Aug 2023 13:34:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131689; x=1692736489; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2g9kzHqhSNJwgBBDx7OBACW2GSQNvlkcjq1gGBNV2Bo=; b=5fhSxlGCoP9eFSdFtRHiL4qRSdSwqPd7gzbVSPTiAU38g2wOHlOIpVYbOfieO8z/+X IEpl6nfntGQWgA924+5Z5nTBocRQ7FPksFYyL4EZCQja00MCxhy36o2za0FYibqRE21z fhQVudLJ+ihYNMXZnwK0rM1AI6CbD6GzVHb8KRBMJvZZVXLvVusxB3MJII05LCR7ksRN PYnsC/MjI2Tcyl2WcvR8r3fx2Vi+tJ1PCApbwyEgQcH+auQ7/GUCjqEaAjfE2WA22rZc CubL4Er4FcAbsDiTz/MAVkn6/HhJ3YMRAc+duqqYw/4Ml47tNzlvl2g4SmXazUrX9UGJ H+Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131689; x=1692736489; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2g9kzHqhSNJwgBBDx7OBACW2GSQNvlkcjq1gGBNV2Bo=; b=Cuah3ywlOm6j8loUseazbSseV6/vcEf2KKqAWVyPsMnE56KnWMl3/QfiG/r5DKmfQ0 WRxP1BcktG7lrupjRGOG5gsoUIpDuoZZO7NCxCiXe9Fkv226ZRUz2vsOTcDz2BAQfViN P5SVk5x6s/aaf5pGDlDiyjcVkh0ps9aLsxshZI8XK65zgZUFRiEwVUmyhR19K7PCG3Gq hx2l9gKqBbndzAK7eE6cOg7HHTIDzx7paxDU3RpbPZxnFZxPdfyd7PoIV0r2H+u7Wgv9 ucyupJ96JhB/JxphXnjuM7eXKR6clJMnpY1xBGvqS1WR/0FkPg4oz/bMviX6/gxDVTPT k3SQ== X-Gm-Message-State: AOJu0YyAY34F9zTe937r123Uq4LBUG0NslemHKs6our5KCBdNsMhkVkR XZMPcm8TXchwIOy2mJc72H8huwiV99cOp90wNXA= X-Google-Smtp-Source: AGHT+IGivQ4nxBqRW7/5l/sAoV4/vI2cYX6H54kXeR8lMldcoVTYpg5DO9jfP7O9CrkY98boxli26xltNMvid9T8YFI= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:af1f:0:b0:56d:647:664b with SMTP id n31-20020a81af1f000000b0056d0647664bmr196328ywh.5.1692131689073; Tue, 15 Aug 2023 13:34:49 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:44 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=10263; i=samitolvanen@google.com; h=from:subject; bh=ZSehk0HlS5lFssHilJirKHoqNUMKSxbPAfpprDSxzWc=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiY/LmC6Mrof2Ek0FECg4Lyrb602OprdqPv Z36fbNfcwKJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7mndC/0bMg+3a2jsWzVZKgcRI3wlt+nRTUv0WtQprozA0ki2lLlThz02RnGre6VdcJGWDs2NOTt ES5Mvq6ZCaP40k/y1vg293dS4YTr9axfndkManB9r3jzd+46U+nbQPpqvBUCeDJm2kuPNft/q9I SuubRVgtai/4nbcZXB7QrsYWW3Lek10TYFv+6sCU7NLij5hPrKN/NvDqJie0mVeZzvQO8hd/he0 C/vRSL6xNpExXTskKfQamDInoer/hiOiLI5OhAZSi+5Ov+W2uO+6zP5dUGTJ0OkOXj4Yd06IKcB hh16MhU5PVBJUPfQUL3OC8hCPaqJN5p6mPquadLTO0YZD3P6CTCjenY1Of0PBz4Zgo01fOIb+c6 U1XeBHSUBI4dbMW7VzXngTAkYd09OV+JNLJtsg3Rz0zTtsYRCr7oVmq8WQbPB8sBujzWJ36CKPF AOY17PAK2ebKmBDAss2umHOFq2t+7DF3bxncYTA1g/IKXLt66QJ2pZQ/1Ws0Mywoykmxs= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-9-samitolvanen@google.com> Subject: [PATCH v2 1/6] riscv: VMAP_STACK overflow detection thread-safe From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Jisheng Zhang , Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230815_133450_328707_0F08B486 X-CRM114-Status: GOOD ( 19.25 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Deepak Gupta commit 31da94c25aea ("riscv: add VMAP_STACK overflow detection") added support for CONFIG_VMAP_STACK. If overflow is detected, CPU switches to `shadow_stack` temporarily before switching finally to per-cpu `overflow_stack`. If two CPUs/harts are racing and end up in over flowing kernel stack, one or both will end up corrupting each other state because `shadow_stack` is not per-cpu. This patch optimizes per-cpu overflow stack switch by directly picking per-cpu `overflow_stack` and gets rid of `shadow_stack`. Following are the changes in this patch - Defines an asm macro to obtain per-cpu symbols in destination register. - In entry.S, when overflow is detected, per-cpu overflow stack is located using per-cpu asm macro. Computing per-cpu symbol requires a temporary register. x31 is saved away into CSR_SCRATCH (CSR_SCRATCH is anyways zero since we're in kernel). Please see Links for additional relevant disccussion and alternative solution. Tested by `echo EXHAUST_STACK > /sys/kernel/debug/provoke-crash/DIRECT` Kernel crash log below Insufficient stack space to handle exception!/debug/provoke-crash/DIRECT Task stack: [0xff20000010a98000..0xff20000010a9c000] Overflow stack: [0xff600001f7d98370..0xff600001f7d99370] CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) epc : __memset+0x60/0xfc ra : recursive_loop+0x48/0xc6 [lkdtm] epc : ffffffff808de0e4 ra : ffffffff0163a752 sp : ff20000010a97e80 gp : ffffffff815c0330 tp : ff600000820ea280 t0 : ff20000010a97e88 t1 : 000000000000002e t2 : 3233206874706564 s0 : ff20000010a982b0 s1 : 0000000000000012 a0 : ff20000010a97e88 a1 : 0000000000000000 a2 : 0000000000000400 a3 : ff20000010a98288 a4 : 0000000000000000 a5 : 0000000000000000 a6 : fffffffffffe43f0 a7 : 00007fffffffffff s2 : ff20000010a97e88 s3 : ffffffff01644680 s4 : ff20000010a9be90 s5 : ff600000842ba6c0 s6 : 00aaaaaac29e42b0 s7 : 00fffffff0aa3684 s8 : 00aaaaaac2978040 s9 : 0000000000000065 s10: 00ffffff8a7cad10 s11: 00ffffff8a76a4e0 t3 : ffffffff815dbaf4 t4 : ffffffff815dbaf4 t5 : ffffffff815dbab8 t6 : ff20000010a9bb48 status: 0000000200000120 badaddr: ff20000010a97e88 cause: 000000000000000f Kernel panic - not syncing: Kernel stack overflow CPU: 1 PID: 205 Comm: bash Not tainted 6.1.0-rc2-00001-g328a1f96f7b9 #34 Hardware name: riscv-virtio,qemu (DT) Call Trace: [] dump_backtrace+0x30/0x38 [] show_stack+0x40/0x4c [] dump_stack_lvl+0x44/0x5c [] dump_stack+0x18/0x20 [] panic+0x126/0x2fe [] walk_stackframe+0x0/0xf0 [] recursive_loop+0x48/0xc6 [lkdtm] SMP: stopping secondary CPUs ---[ end Kernel panic - not syncing: Kernel stack overflow ]--- Cc: Guo Ren Cc: Jisheng Zhang Link: https://lore.kernel.org/linux-riscv/Y347B0x4VUNOd6V7@xhacker/T/#t Link: https://lore.kernel.org/lkml/20221124094845.1907443-1-debug@rivosinc.com/ Signed-off-by: Deepak Gupta Co-developed-by: Sami Tolvanen Signed-off-by: Sami Tolvanen Acked-by: Guo Ren --- arch/riscv/include/asm/asm.h | 22 +++++++++ arch/riscv/include/asm/thread_info.h | 3 -- arch/riscv/kernel/asm-offsets.c | 1 + arch/riscv/kernel/entry.S | 70 ++++------------------------ arch/riscv/kernel/traps.c | 36 +------------- 5 files changed, 34 insertions(+), 98 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 114bbadaef41..bfb4c26f113c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -82,6 +82,28 @@ .endr .endm +#ifdef CONFIG_SMP +#ifdef CONFIG_32BIT +#define PER_CPU_OFFSET_SHIFT 2 +#else +#define PER_CPU_OFFSET_SHIFT 3 +#endif + +.macro asm_per_cpu dst sym tmp + REG_L \tmp, TASK_TI_CPU_NUM(tp) + slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT + la \dst, __per_cpu_offset + add \dst, \dst, \tmp + REG_L \tmp, 0(\dst) + la \dst, \sym + add \dst, \dst, \tmp +.endm +#else /* CONFIG_SMP */ +.macro asm_per_cpu dst sym tmp + la \dst, \sym +.endm +#endif /* CONFIG_SMP */ + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..d18ce0113ca1 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -34,9 +34,6 @@ #ifndef __ASSEMBLY__ -extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)]; -extern unsigned long spin_shadow_stack; - #include #include diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..9f535d5de33f 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -39,6 +39,7 @@ void asm_offsets(void) OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3d11aa3af105 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -10,9 +10,11 @@ #include #include #include +#include #include #include #include +#include SYM_CODE_START(handle_exception) /* @@ -170,67 +172,15 @@ SYM_CODE_END(ret_from_exception) #ifdef CONFIG_VMAP_STACK SYM_CODE_START_LOCAL(handle_kernel_stack_overflow) - /* - * Takes the psuedo-spinlock for the shadow stack, in case multiple - * harts are concurrently overflowing their kernel stacks. We could - * store any value here, but since we're overflowing the kernel stack - * already we only have SP to use as a scratch register. So we just - * swap in the address of the spinlock, as that's definately non-zero. - * - * Pairs with a store_release in handle_bad_stack(). - */ -1: la sp, spin_shadow_stack - REG_AMOSWAP_AQ sp, sp, (sp) - bnez sp, 1b - - la sp, shadow_stack - addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE - - //save caller register to shadow stack - addi sp, sp, -(PT_SIZE_ON_STACK) - REG_S x1, PT_RA(sp) - REG_S x5, PT_T0(sp) - REG_S x6, PT_T1(sp) - REG_S x7, PT_T2(sp) - REG_S x10, PT_A0(sp) - REG_S x11, PT_A1(sp) - REG_S x12, PT_A2(sp) - REG_S x13, PT_A3(sp) - REG_S x14, PT_A4(sp) - REG_S x15, PT_A5(sp) - REG_S x16, PT_A6(sp) - REG_S x17, PT_A7(sp) - REG_S x28, PT_T3(sp) - REG_S x29, PT_T4(sp) - REG_S x30, PT_T5(sp) - REG_S x31, PT_T6(sp) - - la ra, restore_caller_reg - tail get_overflow_stack - -restore_caller_reg: - //save per-cpu overflow stack - REG_S a0, -8(sp) - //restore caller register from shadow_stack - REG_L x1, PT_RA(sp) - REG_L x5, PT_T0(sp) - REG_L x6, PT_T1(sp) - REG_L x7, PT_T2(sp) - REG_L x10, PT_A0(sp) - REG_L x11, PT_A1(sp) - REG_L x12, PT_A2(sp) - REG_L x13, PT_A3(sp) - REG_L x14, PT_A4(sp) - REG_L x15, PT_A5(sp) - REG_L x16, PT_A6(sp) - REG_L x17, PT_A7(sp) - REG_L x28, PT_T3(sp) - REG_L x29, PT_T4(sp) - REG_L x30, PT_T5(sp) - REG_L x31, PT_T6(sp) + /* we reach here from kernel context, sscratch must be 0 */ + csrrw x31, CSR_SCRATCH, x31 + asm_per_cpu sp, overflow_stack, x31 + li x31, OVERFLOW_STACK_SIZE + add sp, sp, x31 + /* zero out x31 again and restore x31 */ + xor x31, x31, x31 + csrrw x31, CSR_SCRATCH, x31 - //load per-cpu overflow stack - REG_L sp, -8(sp) addi sp, sp, -(PT_SIZE_ON_STACK) //save context to overflow stack diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index f910dfccbf5d..deb2144d9143 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -397,48 +397,14 @@ int is_valid_bugaddr(unsigned long pc) #endif /* CONFIG_GENERIC_BUG */ #ifdef CONFIG_VMAP_STACK -/* - * Extra stack space that allows us to provide panic messages when the kernel - * has overflowed its stack. - */ -static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], +DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)], overflow_stack)__aligned(16); -/* - * A temporary stack for use by handle_kernel_stack_overflow. This is used so - * we can call into C code to get the per-hart overflow stack. Usage of this - * stack must be protected by spin_shadow_stack. - */ -long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16); - -/* - * A pseudo spinlock to protect the shadow stack from being used by multiple - * harts concurrently. This isn't a real spinlock because the lock side must - * be taken without a valid stack and only a single register, it's only taken - * while in the process of panicing anyway so the performance and error - * checking a proper spinlock gives us doesn't matter. - */ -unsigned long spin_shadow_stack; - -asmlinkage unsigned long get_overflow_stack(void) -{ - return (unsigned long)this_cpu_ptr(overflow_stack) + - OVERFLOW_STACK_SIZE; -} asmlinkage void handle_bad_stack(struct pt_regs *regs) { unsigned long tsk_stk = (unsigned long)current->stack; unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack); - /* - * We're done with the shadow stack by this point, as we're on the - * overflow stack. Tell any other concurrent overflowing harts that - * they can proceed with panicing by releasing the pseudo-spinlock. - * - * This pairs with an amoswap.aq in handle_kernel_stack_overflow. - */ - smp_store_release(&spin_shadow_stack, 0); - console_verbose(); pr_emerg("Insufficient stack space to handle exception!\n"); From patchwork Tue Aug 15 20:34:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13354275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E33EBC04A94 for ; Tue, 15 Aug 2023 20:35:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Wolw91OAYw6wRmyPoMGY10TxRGq8en5Q4hbYZcUjKa4=; b=Nv26lp0zpwV1AA3tHcpN9rbGP7 bWTx6I5XIEshl7/C4NtwiSb+846Tgb2WNxQOjbFi7bWXW/eugDK0PizWnxVW2slCHHwFt8/6P0Wa4 1qn51JwnpsbvYom2poUlW1hQSG23pDaeqTBg5pj0BlPzKrWQ06shro++8XrZ6fUwLuriB2QxTJcVJ YG29toQyclUOhs9ickwzuCeUPDvLa9vSGxCSStCELaXvEK7qnTJJroffqfPrlCoNwt91JxB/m6sx8 aeOAOpLPbT/2De82ZB/X82A5PmYGM/cnTPvLozXbeB2oNx+oEa2/6Btjuika7r+PNXvBTxJC0i8Kd 2TChjGdQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kk-002QQT-0Q; Tue, 15 Aug 2023 20:34:58 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kf-002QOV-0x for linux-riscv@lists.infradead.org; Tue, 15 Aug 2023 20:34:55 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-26b2bc75c89so4164065a91.0 for ; Tue, 15 Aug 2023 13:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131691; x=1692736491; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WrfRqvXS/YQAQD3rvUURRhi4O4V3buFLgz+cT4E6CXY=; b=0b0AfFag2OqtlzKfyBXLWRGSF8gW1eMeleZQdhIEnHmK7L+mnBqsuJuqNSwV6rtOEU u0s8fw9OKaQxUqiJRpnU0Hj7p2gKX/LzNZrGl2doSZ/b3K7fqQBZZ0gtHwr3NNrZmsmB bElC4ZVwGPAE2nH9h5njd1+8dir9Iv7EJaT4Lvaplu8can7LpTu0q410aYDm4oygCKG6 CfmWyM/5WXnsrFI9sYk8nJqgbKJWDH+e0Ftx6cUw7Ad5/bw0dCu4KZjUmFeQGgW27LQW b1+64f0E00cHsDKGcITMxivXieyFjcRQHcbigwhvU5bczQjxjY4/w6pRvIEHpu+0KRhz IQEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131691; x=1692736491; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WrfRqvXS/YQAQD3rvUURRhi4O4V3buFLgz+cT4E6CXY=; b=LZiRLHf/Ost06trtS/pCAS6hOvK7KUA8CLWPugV/RFivLwXwe6iFiIFYYQr9aElLIy W7u9tTG5hNytaZBRHrRd6nHZmUWHqOwG6VJSInL3uo2v3auPNNlRB5BNdjfn6PW/iySd elWy6MtxSaJfaOChh2565gRR55GKAmKV4g5U5GfPGtaprj8wrBczTjaVSOwj1ocPaUgH 0gVckpMUbGIqHRvofX53wm+cFhspDsI14qgJyVeEfVsRaI7JutQr/lkW5etwvrPVCYen 797dqxrWBdCfHRhC5fkJezbytawuB0Innf23nRpZ/4XRvGCv4VC9ozpd5XwDdHKf4iNp ElWA== X-Gm-Message-State: AOJu0YxU6ptcwHrr1Te/HppRl+xi3yhpx6zlN+a+KsEUIDd6ZBBfWA7Z GX2hdcV5gToFkp88AutmLGpbHiUoJzOzX6RIa7Y= X-Google-Smtp-Source: AGHT+IEchNZaBEINpLlSn/GPNkBO+VrwWfHXdxqZqQVAkBDyQGibTsdMvRWQoxISxiHRWmaP5XqKzzpcu2EXo/lIxzI= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:90a:d502:b0:26b:5fe2:5892 with SMTP id t2-20020a17090ad50200b0026b5fe25892mr1185559pju.6.1692131690966; Tue, 15 Aug 2023 13:34:50 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:45 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=5593; i=samitolvanen@google.com; h=from:subject; bh=fOkgWkASfAqgpBfAvGFiQ8Ap5BTE+sMjcVhFVqoevU4=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiD9deMgIJ3fhkaSc5TjCFw9aOJgO3lojDl IJKQL6WcO2JAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7uETC/oDWTmoCmprkVOzJT880FOolG/86ApcG+2Vp5pFm9jTSXw0uhsv0QJYCKBTupTIjofrNuk yID9Hm9vbntt8SHaS+jwKoheRWZDOAdh0tJuWru56EB0QyTPLLuI6f2NTAFPPDF7/N0MEfFY9Q2 73sbckQRqQxt7tkF7oxFDkS6SMsogOzT8IZ2NuTszPpFCpFOybsEIX+sds97QqG22w0LjKn120J ABr9wCeGmcRNk7+5igryjV3nOw6/uMjWyZwGKIyzWrC3FFDqbHm5AuDcIgkhUd0DK5YK13HFgls qtnla0SK6MtOkM/Zg37Tt4SMgUlX9a75WKGDlVjX1j6KQSvU3TGRjp1Lyas587+RFyuvHXVjdX1 qYlEOUKf8qQjIcqiNnCqllRAsCjzds/YrdVFgoHxrkzeXt+aq89oaE2tLG8gh3OC9y2pQ04Bpz8 vLLi1qj7Wm/3TWsrEydTvIulNO8xWEYLDag4+LI+L8CYYpBAwjrgoVvB0cCcF1VGQ0U5Y= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-10-samitolvanen@google.com> Subject: [PATCH v2 2/6] riscv: Deduplicate IRQ stack switching From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230815_133453_336694_86652947 X-CRM114-Status: GOOD ( 13.92 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org With CONFIG_IRQ_STACKS, we switch to a separate per-CPU IRQ stack before calling handle_riscv_irq or __do_softirq. We currently have duplicate inline assembly snippets for stack switching in both code paths. Now that we can access per-CPU variables in assembly, implement call_on_irq_stack in assembly, and use that instead of redudant inline assembly. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/asm.h | 5 +++++ arch/riscv/include/asm/irq_stack.h | 3 +++ arch/riscv/kernel/entry.S | 32 ++++++++++++++++++++++++++++++ arch/riscv/kernel/irq.c | 32 ++++++++---------------------- arch/riscv/kernel/traps.c | 29 ++++----------------------- 5 files changed, 52 insertions(+), 49 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index bfb4c26f113c..8e446be2d57c 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -104,6 +104,11 @@ .endm #endif /* CONFIG_SMP */ +.macro load_per_cpu dst ptr tmp + asm_per_cpu \dst \ptr \tmp + REG_L \dst, 0(\dst) +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/include/asm/irq_stack.h b/arch/riscv/include/asm/irq_stack.h index e4042d297580..6441ded3b0cf 100644 --- a/arch/riscv/include/asm/irq_stack.h +++ b/arch/riscv/include/asm/irq_stack.h @@ -12,6 +12,9 @@ DECLARE_PER_CPU(ulong *, irq_stack_ptr); +asmlinkage void call_on_irq_stack(struct pt_regs *regs, + void (*func)(struct pt_regs *)); + #ifdef CONFIG_VMAP_STACK /* * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 3d11aa3af105..39875f5e08a6 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -218,6 +218,38 @@ SYM_CODE_START(ret_from_fork) tail syscall_exit_to_user_mode SYM_CODE_END(ret_from_fork) +#ifdef CONFIG_IRQ_STACKS +/* + * void call_on_irq_stack(struct pt_regs *regs, + * void (*func)(struct pt_regs *)); + * + * Calls func(regs) using the per-CPU IRQ stack. + */ +SYM_FUNC_START(call_on_irq_stack) + /* Create a frame record to save ra and s0 (fp) */ + addi sp, sp, -RISCV_SZPTR + REG_S ra, (sp) + addi sp, sp, -RISCV_SZPTR + REG_S s0, (sp) + addi s0, sp, 2*RISCV_SZPTR + + /* Switch to the per-CPU IRQ stack and call the handler */ + load_per_cpu t0, irq_stack_ptr, t1 + li t1, IRQ_STACK_SIZE + add sp, t0, t1 + jalr a1 + + /* Switch back to the thread stack and restore ra and s0 */ + addi sp, s0, -2*RISCV_SZPTR + REG_L s0, (sp) + addi sp, sp, RISCV_SZPTR + REG_L ra, (sp) + addi sp, sp, RISCV_SZPTR + + ret +SYM_FUNC_END(call_on_irq_stack) +#endif /* CONFIG_IRQ_STACKS */ + /* * Integer register context switch * The callee-saved registers must be saved and restored. diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index d0577cc6a081..95dafdcbd135 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -61,32 +61,16 @@ static void init_irq_stacks(void) #endif /* CONFIG_VMAP_STACK */ #ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +static void ___do_softirq(struct pt_regs *regs) +{ + __do_softirq(); +} + void do_softirq_own_stack(void) { -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "call __do_softirq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + if (on_thread_stack()) + call_on_irq_stack(NULL, ___do_softirq); + else __do_softirq(); } #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c index deb2144d9143..83319b6816da 100644 --- a/arch/riscv/kernel/traps.c +++ b/arch/riscv/kernel/traps.c @@ -350,31 +350,10 @@ static void noinstr handle_riscv_irq(struct pt_regs *regs) asmlinkage void noinstr do_irq(struct pt_regs *regs) { irqentry_state_t state = irqentry_enter(regs); -#ifdef CONFIG_IRQ_STACKS - if (on_thread_stack()) { - ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) - + IRQ_STACK_SIZE/sizeof(ulong); - __asm__ __volatile( - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" ra, (sp) \n" - "addi sp, sp, -"RISCV_SZPTR "\n" - REG_S" s0, (sp) \n" - "addi s0, sp, 2*"RISCV_SZPTR "\n" - "move sp, %[sp] \n" - "move a0, %[regs] \n" - "call handle_riscv_irq \n" - "addi sp, s0, -2*"RISCV_SZPTR"\n" - REG_L" s0, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - REG_L" ra, (sp) \n" - "addi sp, sp, "RISCV_SZPTR "\n" - : - : [sp] "r" (sp), [regs] "r" (regs) - : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", - "t0", "t1", "t2", "t3", "t4", "t5", "t6", - "memory"); - } else -#endif + + if (IS_ENABLED(CONFIG_IRQ_STACKS) && on_thread_stack()) + call_on_irq_stack(regs, handle_riscv_irq); + else handle_riscv_irq(regs); irqentry_exit(regs, state); From patchwork Tue Aug 15 20:34:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13354279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59900C04A94 for ; Tue, 15 Aug 2023 20:35:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=hbZpGxfvOK5MVV68XMc8cEQhHLKjd5ANoUcABkcaigk=; b=Dy3XIgQhGLmdIu1x24PFyV8wS6 14w7j30yJwTwi8gSR3PcJ3H2PZ6YZHVTqv7HgMxXhJFZ2cZIj4t6qnMiBB/B4M9I0HPcMckJOmFh8 82ucriV/o7SFJ+Wy9j0sH3ieYEub6G0fhHSI9vvHHWo0E6Z+ulocsjoE8fMZ3a141C6hijAEaVut9 anpPWoahRBFVt8nNxhWv5TVIk6NfgqBNLvrwQ77mR34yd0DLnuR5XMzYyE9oZGL2Izj+q+JPA7XO6 JCHUkgfB6Sj2xwZbz4Zt29o8zY8Ok6ge/fZFdpf4vxJAm32cY17cxLmRQjahcduViQykw3XswbUwT 8BBQmELQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qW0ko-002QRy-19; Tue, 15 Aug 2023 20:35:02 +0000 Received: from mail-pg1-x549.google.com ([2607:f8b0:4864:20::549]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qW0ki-002QOl-0y for linux-riscv@lists.infradead.org; Tue, 15 Aug 2023 20:34:57 +0000 Received: by mail-pg1-x549.google.com with SMTP id 41be03b00d2f7-565e5961ddbso1297976a12.2 for ; Tue, 15 Aug 2023 13:34:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131692; x=1692736492; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BfDLHnMbXQceNrVQLGjVUevqJ2xwYxenmhNK/tAF/sk=; b=DveYx2ypgHekd9E3zy9CT1ilJONYIrAnPqHRj9ZtXg944Rf0TWVVNG5jaFobMnoKx0 Qp6jSJbrWh3E2/h9IXPRd2IM3jaJArvh4oKLp7/khzJJ/4YoiwEv/DKqr19Ru89mBaYv 1gt7hiExW5s9lLZMvCZRWPyrP/GDG1LqMTngfLcIolNUd8lCABgY4ON75DRwtNUOCk8B JY5g55OHAze9zUBVPQ+rocjrZ/v9jCkE6obEf+/vP5i9YGYK/d6QF7IAgDOSXS1kHXVD FeNp6SoviL3M8abvX30FCaMuBBgTkMnTrZyfZofD7Pl87xvQwDtW7fNUw3gGmTWAH0AY dFmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131692; x=1692736492; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BfDLHnMbXQceNrVQLGjVUevqJ2xwYxenmhNK/tAF/sk=; b=YfgHRRcsda9T1y73vnk1T/wFoTxD05PRI0Q0Ppr6FFT1R4vwSzH4C8obVUX14tNnsb s5QhZ2ucwrW6dfsdVMvols/DHq5+MgFHhlhB+fmPj2gkIq+Rt0hLEa3R8Sf+c3x6hB9s lWm3py5WZS8DJJVFQQOYOJmoBbZylWLD7xYxKhxjGl253O4hThAxVFzpFL/LmTb92Gw5 nCBpwz4y602qSPVZGPfVUQbkcrQk3k+kqIkvijfqGD+EaCsa5oSXljuBbbdDj32zIt2I W0Xbd8AFyqVEm4EEUI3/5Xnb7SENnp0v22P1O5K3U2tyP4IX/g0ZwBkaIDzssyXYxxjZ J92Q== X-Gm-Message-State: AOJu0Yzw2QVMB1LTUMlInV59V6qdc8E8KYj3FYieAkMOXdRQfA3wJsSy 4+ddtt0Wso8ENfZ9MSLmIlBQGcdYuQyeIqGUVq8= X-Google-Smtp-Source: AGHT+IG4n+M3PSO3DhS/2Scpx+tUwpXnKVdLJNTSN6YSIUZ+vrp2chjur8+avcRYM+CJmcVopciJDOywc23KOj4HoFo= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a17:90b:4393:b0:263:3437:a0b0 with SMTP id in19-20020a17090b439300b002633437a0b0mr3063684pjb.3.1692131692589; Tue, 15 Aug 2023 13:34:52 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:46 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3217; i=samitolvanen@google.com; h=from:subject; bh=5ixoDk+W6BxwBv4tIu2eVSrOr56ebYlcn47cfqFXXgA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiHUFljIaux+9IC+0XLbF6reJW5Y5V3fbd/ /MD1Qt4OqyJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7kdQDACYUsshok1LuWlv5stJjrwH99NJQrdDSX0WI0YLs+JulAYvsnIPJAatZgneQWhTPWBynMO rJ3Ix+mbOgesN6Los5e0Z5iR+60FzUznS4csogCnjAI1+8EEVryA7ygmOnVm4/wnNpslAOjORp7 byQ+dXEu0N175Wnuc3vqbRnwlcaSIgJR+3bGVEuk8ezzcuTN42WVem7P7gBBgGWi+MzsEtJk9IA +oCPRG0zWkpcBkqHS77+B58Wg9Z/YAaYEeyeqUKq1AMU2MIUP1ul8YRN6CX93320mx9vMOO6tff vPHHw66ck3QzRW5YPmT1EYusoHAeXpCxw6RBC7i2r5Wyqw/QVNOqtKvrkJuzNmMRFJnl9zlGZbs C0IN9uG9VSOuYRky9JDPhBidXrnk7bvsMGzrNuj703Iz9WVkn+LfAUI+Nn/LiYYHYOqhR9w+PRl ERHb8HUO9XJqfcJPtffC4okUlO3NCdKwN6FcN6izZWDFGeuHyBokuvxqemECQWRpFEqkM= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-11-samitolvanen@google.com> Subject: [PATCH v2 3/6] riscv: Move global pointer loading to a macro From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230815_133456_339790_95A6BFC6 X-CRM114-Status: GOOD ( 10.31 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In Clang 17, -fsanitize=shadow-call-stack uses the newly declared platform register gp for storing shadow call stack pointers. As this is obviously incompatible with gp relaxation, in preparation for CONFIG_SHADOW_CALL_STACK support, move global pointer loading to a single macro, which we can cleanly disable when SCS is used instead. Link: https://reviews.llvm.org/rGaa1d2693c256 Link: https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/asm.h | 8 ++++++++ arch/riscv/kernel/entry.S | 6 ++---- arch/riscv/kernel/head.S | 15 +++------------ arch/riscv/kernel/suspend_entry.S | 5 +---- 4 files changed, 14 insertions(+), 20 deletions(-) diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index 8e446be2d57c..f34dd1a526a1 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,14 @@ REG_L \dst, 0(\dst) .endm +/* load __global_pointer to gp */ +.macro load_global_pointer +.option push +.option norelax + la gp, __global_pointer$ +.option pop +.endm + /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 REG_S x6, PT_T1(sp) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 39875f5e08a6..2b4248c6b0a9 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -75,10 +75,8 @@ _save_context: csrw CSR_SCRATCH, x0 /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer + move a0, sp /* pt_regs */ la ra, ret_from_exception diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 11c3b94c4534..79b5a863c782 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -110,10 +110,7 @@ relocate_enable_mmu: csrw CSR_TVEC, a0 /* Reload the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Switch to kernel page tables. A full fence is necessary in order to @@ -134,10 +131,7 @@ secondary_start_sbi: csrw CSR_IP, zero /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of @@ -228,10 +222,7 @@ pmp_done: #endif /* CONFIG_RISCV_M_MODE */ /* Load the global pointer */ -.option push -.option norelax - la gp, __global_pointer$ -.option pop + load_global_pointer /* * Disable FPU & VECTOR to detect illegal usage of diff --git a/arch/riscv/kernel/suspend_entry.S b/arch/riscv/kernel/suspend_entry.S index 12b52afe09a4..556a4b166d8c 100644 --- a/arch/riscv/kernel/suspend_entry.S +++ b/arch/riscv/kernel/suspend_entry.S @@ -60,10 +60,7 @@ END(__cpu_suspend_enter) ENTRY(__cpu_resume_enter) /* Load the global pointer */ - .option push - .option norelax - la gp, __global_pointer$ - .option pop + load_global_pointer #ifdef CONFIG_MMU /* Save A0 and A1 */ From patchwork Tue Aug 15 20:34:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13354276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4EE7DC41513 for ; Tue, 15 Aug 2023 20:35:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=6S0X7IemJvXAmuOB7OY9VV4drNP3MnjWk3rbDVZVqa4=; b=iUEiglAgNCsesuy999ATy2hEdI VMtvVV/6P3Lq3SQaMVFAwUI7L3YDPyXmFNV1u/ea4fLijwe25O/C+ij5cwy4p0zzEp8b97zSLa9lM HL/NXfIWQnsZOIZH0XNQ6SJ9cWeGerEponNDIGzF8CB/MtJr6NjYZIHQLdioAOrzmfovkd1LVFmbO 0WGlsvtJoZcRALuV1R03/rRI7uVI2PLDEyJTiifIMM85lhLa/LAXPN49u9zvRQF4sZdbTZNYrgVYj WKLHsj+qltW+5rDu8soFywzx+grTl0LnC4Tf3VEiHsYm2ADMWZ0owXTgGb/uep2Rt9RZ/jfKmy1hi sTtcFGKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kk-002QQl-2A; Tue, 15 Aug 2023 20:34:58 +0000 Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kh-002QP6-0H for linux-riscv@lists.infradead.org; Tue, 15 Aug 2023 20:34:57 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-58c583f885cso13283027b3.1 for ; Tue, 15 Aug 2023 13:34:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131694; x=1692736494; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N3ibQG9kPTeaqq1uk119WhHtvGpC05ARPoe95r4WB3Q=; b=ii3NiBwjW2S4FrV7yjM64htaZ/K+xMTAe8C3FjqVBGt9E6qnK9LrJmBelNkdwUPq0S /u8xXLY/b2D0v9G+ODd2yoH2w0MBi7+xjG488iodmtTpyqSxuJWj17X4fZWhEmWIX+fR wjELHT7fOZ01fKMWI8/otfmtTHXgHB8Fq+wgPvPcWxYFNX4dnVbEX57BoPyZuUGCDZWR 1O51jehCVbCgc9APBR35NAd6nxRB0SailOkfeWnWCnEHaQWkyZhKxC/zwlFuL/DZOHhC N3u254/fONTgoNDkrUS6Iwkbg/W8KeV4athdOLQ3LmKxCTJeJrUnNXykkj6KSDMpHum5 ICCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131694; x=1692736494; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N3ibQG9kPTeaqq1uk119WhHtvGpC05ARPoe95r4WB3Q=; b=iHDx+FKOzTk3dPjUA659fOsWKN59Eh2GGZ4Z8KLvLF3LeakO5awbFuNUHIXuQxxL6l bJ1GeOa7cBzULj9BfeF2KXFM0/MABcJKLhM0kXjSa7oiYJNxQisy6PZtVWIc6HrkOhiX uy9KEyTPeLPb5Rw+IA79tJdxuK9/LG+iLG/v9HhX8lWC+oODnQnwNCJemwK4SHseS71L gaPw5XTl4SxvqI1OCdHlE+oJHsosY3kJWS0ivb8H+5iCEsxeWv/ZEOmLeUtfzTWyruzx 4HkojvWWaDU2gzN1rXj5naFwRWtlOLz/18dZPu/LOg7Jt4259fNsVCIF091DdplHx2lR HuBg== X-Gm-Message-State: AOJu0YwosXGTLI4dVfwOeiS664RDZt90Am9CbRwVPm1M3lWqEBOLGOqq JKUIysUteiPeA7MOkvkl+77ppI247vBvYCJtcGM= X-Google-Smtp-Source: AGHT+IFOBtAfOTGqgJAn+DcvcJUfpiueeWWzkqYoVx3UNQD79ntr/Mc1j91DYQQN2fSTiPkbFkjokwMeqbqG2PDHuNw= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:b60d:0:b0:58c:6ddd:d27c with SMTP id u13-20020a81b60d000000b0058c6dddd27cmr5856ywh.6.1692131694123; Tue, 15 Aug 2023 13:34:54 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:47 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=9645; i=samitolvanen@google.com; h=from:subject; bh=e0JAEbcNV8QaCejgu3nDKo0gvZQXOTr0vBP/0yqZP/Q=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiTMC3KO4Q7y58NU00sySutTm4Un6tjcRZM NE+DfEzpYyJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7omFDACbtMcHmBN0cplT0k67u1z6qmmA2zIU82etEYjowZNjt2dLorEXMhCUCsSGBB+WOfE8wdK T1Ic5tQpvOODjDM1gAr0DzGeI4zHBtdL35iFmGDHe/K4JAihBOPM9ukDaPBywSyWMp7PwOT3R3E dpTpGqVyZCAz+ggnRAItZlj9hr1xlBPwi0Lgga40mOUDi0A83mhYPFar290fiu3bHBFHSkWWkEZ bE/cB9jlBzaDLK9fE7B2um3TvkudWYaCpimIbKNJhDGBrBr3Zk98tW4+jHe4VIu7SUc3i5JJcc7 jqK/+9Tl8CLtFl1lHwAqYHCz9esWAPVyWQtAQtyZd+bd8ZK+JUa5QpEso9V+8vl+/ntd8bl+5Te faYUB+jUrc5PH2ROCiBNrQYem3x8ezSXhtS6j5xHs0YoGWCQWNZFA2NqtOeZNDejk/2TFYbLSPl y7/WihDsjDhucR7Td+gZSUJ0WSB288e8icuCoB2C7SfPlj7zn6m7ktq/naHzMqTDS4c2c= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-12-samitolvanen@google.com> Subject: [PATCH v2 4/6] riscv: Implement Shadow Call Stack From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230815_133455_127070_8E0532C5 X-CRM114-Status: GOOD ( 21.10 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Implement CONFIG_SHADOW_CALL_STACK for RISC-V. When enabled, the compiler injects instructions to all non-leaf C functions to store the return address to the shadow stack and unconditionally load it again before returning, which makes it harder to corrupt the return address through a stack overflow, for example. The active shadow call stack pointer is stored in the gp register, which makes SCS incompatible with gp relaxation. Use --no-relax-gp to ensure gp relaxation is disabled and disable global pointer loading. Add SCS pointers to struct thread_info, implement SCS initialization, and task switching Signed-off-by: Sami Tolvanen --- arch/riscv/Kconfig | 6 ++++ arch/riscv/Makefile | 4 +++ arch/riscv/include/asm/asm.h | 6 ++++ arch/riscv/include/asm/scs.h | 47 ++++++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 13 ++++++++ arch/riscv/kernel/asm-offsets.c | 3 ++ arch/riscv/kernel/entry.S | 11 +++++++ arch/riscv/kernel/head.S | 4 +++ arch/riscv/kernel/vdso/Makefile | 2 +- arch/riscv/purgatory/Makefile | 4 +++ 10 files changed, 99 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/include/asm/scs.h diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4c07b9189c86..8fe31ec59da4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -46,6 +46,7 @@ config RISCV select ARCH_SUPPORTS_HUGETLBFS if MMU select ARCH_SUPPORTS_PAGE_TABLE_CHECK if MMU select ARCH_SUPPORTS_PER_VMA_LOCK if MMU + select ARCH_SUPPORTS_SHADOW_CALL_STACK if HAVE_SHADOW_CALL_STACK select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU @@ -169,6 +170,11 @@ config GCC_SUPPORTS_DYNAMIC_FTRACE def_bool CC_IS_GCC depends on $(cc-option,-fpatchable-function-entry=8) +config HAVE_SHADOW_CALL_STACK + def_bool $(cc-option,-fsanitize=shadow-call-stack) + # https://github.com/riscv-non-isa/riscv-elf-psabi-doc/commit/a484e843e6eeb51f0cb7b8819e50da6d2444d769 + depends on $(ld-option,--no-relax-gp) + config ARCH_MMAP_RND_BITS_MIN default 18 if 64BIT default 8 diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6ec6d52a4180..e518a74640fb 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -55,6 +55,10 @@ endif endif endif +ifeq ($(CONFIG_SHADOW_CALL_STACK),y) + KBUILD_LDFLAGS += --no-relax-gp +endif + # ISA string setting riscv-march-$(CONFIG_ARCH_RV32I) := rv32ima riscv-march-$(CONFIG_ARCH_RV64I) := rv64ima diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h index f34dd1a526a1..b0487b39e674 100644 --- a/arch/riscv/include/asm/asm.h +++ b/arch/riscv/include/asm/asm.h @@ -109,6 +109,11 @@ REG_L \dst, 0(\dst) .endm +#ifdef CONFIG_SHADOW_CALL_STACK +/* gp is used as the shadow call stack pointer instead */ +.macro load_global_pointer +.endm +#else /* load __global_pointer to gp */ .macro load_global_pointer .option push @@ -116,6 +121,7 @@ la gp, __global_pointer$ .option pop .endm +#endif /* CONFIG_SHADOW_CALL_STACK */ /* save all GPs except x1 ~ x5 */ .macro save_from_x6_to_x31 diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h new file mode 100644 index 000000000000..94726ea773e3 --- /dev/null +++ b/arch/riscv/include/asm/scs.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_SCS_H +#define _ASM_SCS_H + +#ifdef __ASSEMBLY__ +#include + +#ifdef CONFIG_SHADOW_CALL_STACK + +/* Load init_shadow_call_stack to gp. */ +.macro scs_load_init_stack + la gp, init_shadow_call_stack + XIP_FIXUP_OFFSET gp +.endm + +/* Load task_scs_sp(current) to gp. */ +.macro scs_load_current + REG_L gp, TASK_TI_SCS_SP(tp) +.endm + +/* Load task_scs_sp(current) to gp, but only if tp has changed. */ +.macro scs_load_current_if_task_changed prev + beq \prev, tp, _skip_scs + scs_load_current +_skip_scs: +.endm + +/* Save gp to task_scs_sp(current). */ +.macro scs_save_current + REG_S gp, TASK_TI_SCS_SP(tp) +.endm + +#else /* CONFIG_SHADOW_CALL_STACK */ + +.macro scs_load_init_stack +.endm +.macro scs_load_current +.endm +.macro scs_load_current_if_task_changed prev +.endm +.macro scs_save_current +.endm + +#endif /* CONFIG_SHADOW_CALL_STACK */ +#endif /* __ASSEMBLY__ */ + +#endif /* _ASM_SCS_H */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index d18ce0113ca1..574779900bfb 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -57,8 +57,20 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ +#ifdef CONFIG_SHADOW_CALL_STACK + void *scs_base; + void *scs_sp; +#endif }; +#ifdef CONFIG_SHADOW_CALL_STACK +#define INIT_SCS \ + .scs_base = init_shadow_call_stack, \ + .scs_sp = init_shadow_call_stack, +#else +#define INIT_SCS +#endif + /* * macros/functions for gaining access to the thread information structure * @@ -68,6 +80,7 @@ struct thread_info { { \ .flags = 0, \ .preempt_count = INIT_PREEMPT_COUNT, \ + INIT_SCS \ } void arch_release_task_struct(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index 9f535d5de33f..177cef43a2ee 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -38,6 +38,9 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); +#ifdef CONFIG_SHADOW_CALL_STACK + OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); +#endif OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 2b4248c6b0a9..ad34507d3c96 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -77,6 +78,9 @@ _save_context: /* Load the global pointer */ load_global_pointer + /* Load the kernel shadow call stack pointer if coming from userspace */ + scs_load_current_if_task_changed s5 + move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -123,6 +127,9 @@ SYM_CODE_START_NOALIGN(ret_from_exception) addi s0, sp, PT_SIZE_ON_STACK REG_S s0, TASK_TI_KERNEL_SP(tp) + /* Save the kernel shadow call stack pointer */ + scs_save_current + /* * Save TP into the scratch register , so we can find the kernel data * structures again. @@ -277,6 +284,8 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + /* Save the kernel shadow call stack pointer */ + scs_save_current /* Restore context from next->thread */ REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) @@ -294,6 +303,8 @@ SYM_FUNC_START(__switch_to) REG_L s11, TASK_THREAD_S11_RA(a4) /* The offset of thread_info in task_struct is zero. */ move tp, a1 + /* Switch to the next shadow call stack */ + scs_load_current ret SYM_FUNC_END(__switch_to) diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S index 79b5a863c782..c3d0ee77483b 100644 --- a/arch/riscv/kernel/head.S +++ b/arch/riscv/kernel/head.S @@ -14,6 +14,7 @@ #include #include #include +#include #include #include "efi-header.S" @@ -153,6 +154,7 @@ secondary_start_sbi: XIP_FIXUP_OFFSET a3 add a3, a3, a1 REG_L sp, (a3) + scs_load_current .Lsecondary_start_common: @@ -293,6 +295,7 @@ clear_bss_done: la sp, init_thread_union + THREAD_SIZE XIP_FIXUP_OFFSET sp addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_BUILTIN_DTB la a0, __dtb_start XIP_FIXUP_OFFSET a0 @@ -311,6 +314,7 @@ clear_bss_done: la tp, init_task la sp, init_thread_union + THREAD_SIZE addi sp, sp, -PT_SIZE_ON_STACK + scs_load_init_stack #ifdef CONFIG_KASAN call kasan_early_init diff --git a/arch/riscv/kernel/vdso/Makefile b/arch/riscv/kernel/vdso/Makefile index 6b1dba11bf6d..48c362c0cb3d 100644 --- a/arch/riscv/kernel/vdso/Makefile +++ b/arch/riscv/kernel/vdso/Makefile @@ -36,7 +36,7 @@ CPPFLAGS_vdso.lds += -DHAS_VGETTIMEOFDAY endif # Disable -pg to prevent insert call site -CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) # Disable profiling and instrumentation for VDSO code GCOV_PROFILE := n diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile index dc20e166983e..d5d60c040560 100644 --- a/arch/riscv/purgatory/Makefile +++ b/arch/riscv/purgatory/Makefile @@ -77,6 +77,10 @@ ifdef CONFIG_STACKPROTECTOR_STRONG PURGATORY_CFLAGS_REMOVE += -fstack-protector-strong endif +ifdef CONFIG_SHADOW_CALL_STACK +PURGATORY_CFLAGS_REMOVE += $(CC_FLAGS_SCS) +endif + CFLAGS_REMOVE_purgatory.o += $(PURGATORY_CFLAGS_REMOVE) CFLAGS_purgatory.o += $(PURGATORY_CFLAGS) From patchwork Tue Aug 15 20:34:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13354280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41B5FC001E0 for ; Tue, 15 Aug 2023 20:35:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=f9vnZPrvpl5tbPgxaBl0ZCiIgLnQpKjddkhGF/Ghcds=; b=GneCCttn4YajSBJBfkhGMFRA5e eb1fTx5mh0HJ5CMjKgd59zghwQb3YWlVUkp8LPB9NsqVQKIj/ZXX3SUsu4LQ+1+S2q187i7rQlx0R ijE+uypo4dlicmLf/1mjxxvyqqQMJBlygVPzjrsQqDyxc3FzPHM2rhAsiZ2/0HR14jldsf+K2aPBq NT4jm/hF47pUJ84ErN3clR+0ojzklcAINkxLbPuI6AuOigPBKhe3grTL968Kj6k4PWLzgfclZtC3U G7peKyBD9PF2jOu9ww6hdmZlYmV/IeSAI0nOxg1Mu2QUCvYVz1yEyfaFn7+uwAPWADI/WekZB5FPR PQ49fa2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qW0ko-002QSI-31; Tue, 15 Aug 2023 20:35:02 +0000 Received: from mail-yw1-x114a.google.com ([2607:f8b0:4864:20::114a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kj-002QPq-0Y for linux-riscv@lists.infradead.org; Tue, 15 Aug 2023 20:34:58 +0000 Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-584139b6b03so74737007b3.3 for ; Tue, 15 Aug 2023 13:34:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131696; x=1692736496; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uSOlMNCKHmfn5xfjrLh25HLPAdSumgQ6fSlbmoDJ2Yc=; b=qox904O6890VuPVeOFie+1gdmVBYEK7DiCslIx75yuPhU25HodT/JX1fwIBsNjAqoW iQp2mkqgx00drzHTCY7LI8aABLkp1VnAeuVSPs0VEuZYZ47KInVdm7L62A4CvkC5bp0l C5/HLIBI+E08srWstaA2cDxRL3JtGccv9yCImwYaUrvr3deqtmN7h8aeIdx6lyEucs/m pw4vrlK5aHRM1FnlmLa+RzgujHxZ4WrJcngnnxCUiIxM+P8tDmNNr9Vt8Ab6fbkG20n1 yF5suVbZp+f/JhUCo5ExAiU6zsPSve/ui3KtPmU9Ua6b1Fdva7K+Na15IB35x1Wnlhgr y1UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131696; x=1692736496; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uSOlMNCKHmfn5xfjrLh25HLPAdSumgQ6fSlbmoDJ2Yc=; b=HZ1Us9uXrGxhzP2RdvogJoK3CITzWPEmFpXpyzIro1GtrnzWeETXkFqUHU+LtT4++7 L2NVCUxzajoLxA2mkxcMwMTkkUmIGx6w67i+v1k4HfKn3n/XV0WBfJGN8tfPeeP6LCpQ qBC+oi4cnCyamriy2WSqQT4LE9dtGKu+v0HEaScuqR6v4wIfD24H7qnOernBRtVokryT R+wAykEtblaqNaodAw5HdbBVcbBcrTMS4KvxUQ50SvdHk4iA7sfa0OhH3dgLcLS5ezTn NSF2jse5lAqZhZA7dLIWUVMVZVQEFRCBD2b3wk30NiaajfkU4TyWRLOBEEVIDUsEiFr0 n4Rg== X-Gm-Message-State: AOJu0YweSn/McPAxonBsmEcB3Adhudab81Um4vz8O7vIokXPUb50CnaA AZei5qvFpfkpf/15b9rU/FYN5AntlkcUs1AlzbE= X-Google-Smtp-Source: AGHT+IFIG6x56rr09NEIucFsFTb2xdeaHALY+wrc10+vyz3Ri9z+DqUmjXG78S/oVc7ErGyWTEp4JYnxEZ20xlRZUd0= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a81:ad12:0:b0:589:a935:b13 with SMTP id l18-20020a81ad12000000b00589a9350b13mr197844ywh.5.1692131695824; Tue, 15 Aug 2023 13:34:55 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:48 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=3124; i=samitolvanen@google.com; h=from:subject; bh=+CT5zR4YANM8SQ9/yXUv6987s9+QQOw/OXe2qcTQ3NU=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+FiWKdkEjcK+oM60PCK6k6ESVE8u+T4K+xL3 kx9YUX2gkmJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7kXbDACVmNAL8YYcd9TlWO6FP17kmKyhCvdTHqqrxGafGKiyJDXD7dTQipBujrxw7uevlZIkOiM 2krCuz5YPDlaC9LMHF4iLkWccLPwZY1NMQV8VwQyWPIHG97WO4fo4aZ/kbZf/FVoXcEBkZzmXD8 8OIDeYf8fCjDiqITUNFS8qFFrppbk0LerRQp9AYy9rIXE9LIuxkoEJ0I892xLtE2jEJLbAj13UM vTaq0hnLWd9E5rxKWuGMqUheN3DI1xIYFHjZX6lYODTUzRVyukKdeCXAyG0XNeyaVi2AdmvgGo5 eMPQz8InG+kdQSqH3PSbaR/H788IhSheQImtpgsJhycp41G5XbIAVuq5ndC1pik0iTdXM/sQoR9 /Tr6kIR9wN8YC3VWXEKaLCPfVOvzGEta9WYy7NAxOQ1tAjE9PjNaLg75QDxE9Ji92wRuwPgCcqh 1avsH5hj5GulKKx9q9ZdyeIoLP/3xzhAq5SgONJcm7ldjpCnTpGVtm6muw6YiAJ0EUnNY= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-13-samitolvanen@google.com> Subject: [PATCH v2 5/6] riscv: Use separate IRQ shadow call stacks From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230815_133457_208448_7606E8A1 X-CRM114-Status: GOOD ( 12.47 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org When both CONFIG_IRQ_STACKS and SCS are enabled, also use a separate per-CPU shadow call stack. Signed-off-by: Sami Tolvanen --- arch/riscv/include/asm/scs.h | 7 +++++++ arch/riscv/kernel/entry.S | 7 +++++++ arch/riscv/kernel/irq.c | 21 +++++++++++++++++++++ 3 files changed, 35 insertions(+) diff --git a/arch/riscv/include/asm/scs.h b/arch/riscv/include/asm/scs.h index 94726ea773e3..0e45db78b24b 100644 --- a/arch/riscv/include/asm/scs.h +++ b/arch/riscv/include/asm/scs.h @@ -13,6 +13,11 @@ XIP_FIXUP_OFFSET gp .endm +/* Load the per-CPU IRQ shadow call stack to gp. */ +.macro scs_load_irq_stack tmp + load_per_cpu gp, irq_shadow_call_stack_ptr, \tmp +.endm + /* Load task_scs_sp(current) to gp. */ .macro scs_load_current REG_L gp, TASK_TI_SCS_SP(tp) @@ -34,6 +39,8 @@ .macro scs_load_init_stack .endm +.macro scs_load_irq_stack tmp +.endm .macro scs_load_current .endm .macro scs_load_current_if_task_changed prev diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index ad34507d3c96..c86b76584d2d 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -238,12 +238,19 @@ SYM_FUNC_START(call_on_irq_stack) REG_S s0, (sp) addi s0, sp, 2*RISCV_SZPTR + /* Switch to the per-CPU shadow call stack */ + scs_save_current + scs_load_irq_stack t0 + /* Switch to the per-CPU IRQ stack and call the handler */ load_per_cpu t0, irq_stack_ptr, t1 li t1, IRQ_STACK_SIZE add sp, t0, t1 jalr a1 + /* Switch back to the thread shadow call stack */ + scs_load_current + /* Switch back to the thread stack and restore ra and s0 */ addi sp, s0, -2*RISCV_SZPTR REG_L s0, (sp) diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c index 95dafdcbd135..7bfea97ee7e7 100644 --- a/arch/riscv/kernel/irq.c +++ b/arch/riscv/kernel/irq.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -34,6 +35,24 @@ EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode); #ifdef CONFIG_IRQ_STACKS #include +DECLARE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); + +#ifdef CONFIG_SHADOW_CALL_STACK +DEFINE_PER_CPU(ulong *, irq_shadow_call_stack_ptr); +#endif + +static void init_irq_scs(void) +{ + int cpu; + + if (!scs_is_enabled()) + return; + + for_each_possible_cpu(cpu) + per_cpu(irq_shadow_call_stack_ptr, cpu) = + scs_alloc(cpu_to_node(cpu)); +} + DEFINE_PER_CPU(ulong *, irq_stack_ptr); #ifdef CONFIG_VMAP_STACK @@ -76,6 +95,7 @@ void do_softirq_own_stack(void) #endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ #else +static void init_irq_scs(void) {} static void init_irq_stacks(void) {} #endif /* CONFIG_IRQ_STACKS */ @@ -87,6 +107,7 @@ int arch_show_interrupts(struct seq_file *p, int prec) void __init init_IRQ(void) { + init_irq_scs(); init_irq_stacks(); irqchip_init(); if (!handle_arch_irq) From patchwork Tue Aug 15 20:34:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sami Tolvanen X-Patchwork-Id: 13354278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6CF9FC04E69 for ; Tue, 15 Aug 2023 20:35:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PLiT581K0+Zch4mu877+m9RcyMYr+DKkEZljE+CZUJ0=; b=2W4Ga4tF0KuxL/z77/uvxAmJ6d ehlaRLLrw6UOevN1ijZavOY6M/agfWv4bHmbU0F94dG6waHR5rJ2MA9Dx0520WpuQS5T4oc/KlDqZ tf2yX5KXtA4EpnxwDK5BuLZkPtusF3WstJl7hfc5b7D4Uz7ihxYKWnX84uJRCoMevErprXCDlvrPF Bx7dL+S64+0a5a1crljEyGwZxgOxK0CRksxxtwOt4nuCtLK+cmghHvv7/f1BwMXxk8yo90pqaCpX8 WQ3yWv9DH2WXWaj/BAkktbY6UWzbT+KFTfy4pftRA9qn+SmsXTnIOndTeBgDMr0JnfjzeVH+2AVbZ GrlkSHag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kp-002QSn-1p; Tue, 15 Aug 2023 20:35:03 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qW0kl-002QQM-2i for linux-riscv@lists.infradead.org; Tue, 15 Aug 2023 20:35:01 +0000 Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-68843280241so1975723b3a.1 for ; Tue, 15 Aug 2023 13:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692131697; x=1692736497; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0Ykk2Dcm0sp1r3wFF3O+L+QDwz/a2RXVAzUE310xTCE=; b=c51n2/tJShODyeeUWChM6SNmOtBH4TwAB27gHlz0U2MW1KhJAFsJAI7HodLDVzEs2L h5EacJeK3yQ7xI1bDM7pS/8+NTTnu/pBkkf0mHnOLxusk130BgOMc3UgF/ILE4pTCv3b FoK/mdVbDTdzIm+PesccCvkjIraV3wLcVysrd2PMSHMHQsQbcViexWNnSAbHE6yrDtgT Mp+VyMCTaEctp9R1ryTin05vgD7DzxczuWzmHabxbBnipkxhOH1Z8NXKF+B+bk3ZWN2C pzEoTOjVu6w2TWjI1tCV+5ZHtdXlLljOrV5rvKK0dr/ZGKr7YBnabTma9ayXOLsdZ9cs ecvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692131697; x=1692736497; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0Ykk2Dcm0sp1r3wFF3O+L+QDwz/a2RXVAzUE310xTCE=; b=Z2B2I+bHR3mfpvjrNW38rPwgsr+zhl00aUCTfonVGFiEtfDPqxfItxv83qqNV+ylCa wUg8asrNAq5W9opOrB6JnTRkdg7QNf3WmP3slKYPsRrJNfk4lISeETQ52uuvqv/N9OzB CfcivkU16Mg+/+Xbs/qs+dF8idjGQU356C4ioUW706CEvHlcWgHQBbB3BRhHRY6I/pda 6oQtd2p6qvdQy1ZTrHOJ02ekZlTPI13n97LFMe5JOmhMZQ4G2iCnZEbxbbsjnvKgspE4 Bk0uw3rHUgICTkWTCSEgF45NbfppdXIL2Tu7/G/oP5YnMtX5lhijdpKy7kJ0gUOmfd5P YBZw== X-Gm-Message-State: AOJu0Ywdx3cHBnuFsQFiYtH4AbPmZOsI9j7aahoFoOhV2yIhmLoqEDWq sOHq6ESa3YrfBbqk5YSqjLQpbFhOpAEFtB7oty0= X-Google-Smtp-Source: AGHT+IFSPEKlQSmP4El45E8vBc0Epq9faMbNSeZpAP4jMdZu51DXHIvs320ofYVLaVzWyPCFTx03x5OLJ9JW1V1TgBQ= X-Received: from samitolvanen.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:4f92]) (user=samitolvanen job=sendgmr) by 2002:a05:6a00:1a87:b0:674:1663:1270 with SMTP id e7-20020a056a001a8700b0067416631270mr6012423pfv.4.1692131697326; Tue, 15 Aug 2023 13:34:57 -0700 (PDT) Date: Tue, 15 Aug 2023 20:34:49 +0000 In-Reply-To: <20230815203442.1608773-8-samitolvanen@google.com> Mime-Version: 1.0 References: <20230815203442.1608773-8-samitolvanen@google.com> X-Developer-Key: i=samitolvanen@google.com; a=openpgp; fpr=35CCFB63B283D6D3AEB783944CB5F6848BBC56EE X-Developer-Signature: v=1; a=openpgp-sha256; l=1987; i=samitolvanen@google.com; h=from:subject; bh=ki2ulz3S+Qkr8AIV7YWzX5sRx6qIOMgf7DcUTP9c/kA=; b=owEB7QES/pANAwAKAUy19oSLvFbuAcsmYgBk2+Fi+17Bq1AKxHm6SJ6quZHGmn+eubZLGDQWv bi76aScYZaJAbMEAAEKAB0WIQQ1zPtjsoPW0663g5RMtfaEi7xW7gUCZNvhYgAKCRBMtfaEi7xW 7uDxC/4yeVJIZQGthB7VhCrB1mQ9aPlWa4zl/qPRC5GlncPWQV/MwLuo1ExhOVxQY1M+pu82F+m 8PqQG33dyzALoBf1c1t0RJgk8hXrSg+on+5gsx856SahKsjV3+GpMyptJgjPEwnNHk+6wKMMK8A N/pPZs3mhzf+AhJROOGKLcZi1Ux1G5qizF4mL1TsJXLT2ky85P+u9iE+XnlUUOVj0o6+zdRc6E4 T3hgbyVAvct21OxraGNq6gZ4hFfI1gD0Vzy9TsNhYvkKUv6h9A3mjyabLDgnWYIneyLL5Aw8hLM 778p+uRcOJ4tKxAH+wveS7E7YVuTsPIR0MtZTeW4TlXWx284O7O/LdzF0u0artPihMYgdyyeXkT fbWExG0nGe0BC399mEX2b1gOK/3yeNpSy5DzlUGN+aTa7vNStcHxmPR4yr6Qlfw7uDrg4g305Ra AkJHJ/qsdN+6PCM8BiYrPK7icqJ3WmLfwFC54pjnKI93e+BJiBx3TlG5aczelxjKucC60= X-Mailer: git-send-email 2.41.0.694.ge786442a9b-goog Message-ID: <20230815203442.1608773-14-samitolvanen@google.com> Subject: [PATCH v2 6/6] lkdtm: Fix CFI_BACKWARD on RISC-V From: Sami Tolvanen To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Kees Cook Cc: Guo Ren , Deepak Gupta , Nathan Chancellor , Nick Desaulniers , Fangrui Song , linux-riscv@lists.infradead.org, llvm@lists.linux.dev, linux-kernel@vger.kernel.org, Sami Tolvanen X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230815_133459_915748_1F071491 X-CRM114-Status: GOOD ( 12.69 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On RISC-V, the return address is before the current frame pointer, unlike on most other architectures. Use the correct offset on RISC-V to fix the CFI_BACKWARD test. Signed-off-by: Sami Tolvanen Acked-by: Kees Cook --- drivers/misc/lkdtm/cfi.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/drivers/misc/lkdtm/cfi.c b/drivers/misc/lkdtm/cfi.c index fc28714ae3a6..6a33889d0902 100644 --- a/drivers/misc/lkdtm/cfi.c +++ b/drivers/misc/lkdtm/cfi.c @@ -68,12 +68,20 @@ static void lkdtm_CFI_FORWARD_PROTO(void) #define no_pac_addr(addr) \ ((__force __typeof__(addr))((uintptr_t)(addr) | PAGE_OFFSET)) +#ifdef CONFIG_RISCV +/* https://github.com/riscv-non-isa/riscv-elf-psabi-doc/blob/master/riscv-cc.adoc#frame-pointer-convention */ +#define FRAME_RA_OFFSET (-1) +#else +#define FRAME_RA_OFFSET 1 +#endif + /* The ultimate ROP gadget. */ static noinline __no_ret_protection void set_return_addr_unchecked(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store. */ - unsigned long * volatile *ret_addr = (unsigned long **)__builtin_frame_address(0) + 1; + unsigned long * volatile *ret_addr = + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; /* Make sure we've found the right place on the stack before writing it. */ if (no_pac_addr(*ret_addr) == expected) @@ -88,7 +96,8 @@ static noinline void set_return_addr(unsigned long *expected, unsigned long *addr) { /* Use of volatile is to make sure final write isn't seen as a dead store. */ - unsigned long * volatile *ret_addr = (unsigned long **)__builtin_frame_address(0) + 1; + unsigned long * volatile *ret_addr = + (unsigned long **)__builtin_frame_address(0) + FRAME_RA_OFFSET; /* Make sure we've found the right place on the stack before writing it. */ if (no_pac_addr(*ret_addr) == expected)