From patchwork Wed Dec 7 10:39:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13066961 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 064DAC352A1 for ; Wed, 7 Dec 2022 10:39:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229564AbiLGKj4 (ORCPT ); Wed, 7 Dec 2022 05:39:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36890 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229486AbiLGKjx (ORCPT ); Wed, 7 Dec 2022 05:39:53 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2A361C7 for ; Wed, 7 Dec 2022 02:39:52 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5EAE3612B7 for ; Wed, 7 Dec 2022 10:39:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 789D7C433D6; Wed, 7 Dec 2022 10:39:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670409591; bh=2VxWbj18Yrpe5S7MdQii2REpX63Y+WQlZ1akSuKTbsc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JVF2ccOkl1oWPKUYKUT0n+Lbqym3Fcte8F6bZY3OEhjj4bdd+Nz1n6oX89rVJ42cc gRhqKmHQTwZCSCp02c0AmIvkvCcmWr1Ui5+LnBCV2bxCEZmngtZCHdvU13kzIFhamk Vj6yO7H2kUG51JrbSUQWWsW1biCMsi+QR0HOvnQdYpj+S1EF2fMQpQe/tXXAfcQaVC EWkoIgt+Uf99nVWjfj+qf044i+XQSQXbMZcObE8c4Vy26dGcENxPunubyeTFKA/u4A vYBLHL6vC+ZH5jjxrw8n07NChOrA8S3InGcMluDJhjCEls8LxnKuvucFnvELXizCPD IgA4u8iqGOfbw== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , Linus Walleij , Arnd Bergmann Subject: [PATCH v2 1/2] ARM: vfp: Manipulate VFP state with softirqs disabled Date: Wed, 7 Dec 2022 11:39:35 +0100 Message-Id: <20221207103936.2198407-2-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221207103936.2198407-1-ardb@kernel.org> References: <20221207103936.2198407-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6070; i=ardb@kernel.org; h=from:subject; bh=2VxWbj18Yrpe5S7MdQii2REpX63Y+WQlZ1akSuKTbsc=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjkG1loJrT1eQlQs9RmMXqtJx12VX/eg+OQ017OEbd WD3YbwCJAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5BtZQAKCRDDTyI5ktmPJF9lC/ 93/RQWHZSrUxfwec+zQAO2jz0rzX9AmfnV049ZoosEOyUQ6n0hrbuTvTIrkACzZTbUGnClyUDRM0EA /4BlywjCB5MgFD9p+mqhDaorYnPeNo/suji5V33iT/OfWNNJ0uRiSIf9baFFHt4wkBeNZ9LfcbFwGR 3hUIzKqZV46lX3vLajyRyfKmj4L87jzy5aJJh6UWaeEBv95hOu0s8aaKlzrBHkabQL/msgqr2Nvhfu HpwX3UHF34cYeh4Gtg3Huk63shYAARkZyyOy00bDYS195igGirVg645/NtVBPCu4VfMpCzG4SzWfVY ltW3RQ1OWaJrDRbbBdvlEbZ2oN2tLjDYY/aLdGKUbc5AZew5hAkwHLqpWDiD7S+pkS9+LBZcvt8Moj hS03VwDUavqUSNcMbpK5Nh0n10Fm/cTz6/FiY0YNAvwHktk/zyo3cZMHCoxB9Ymd4ysvvVlrnm4EFP 5fdaUpUF3A2DgmPBm5fhKUNj9n2jUyqQQ6m0VrRzbH84s= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org In a subsequent patch, we will relax the kernel mode NEON policy, and permit kernel mode NEON to be used not only from task context, as is permitted today, but also from softirq context. Given that softirqs may trigger over the back of any IRQ unless they are explicitly disabled, we need to address the resulting races in the VFP state handling, by disabling softirq processing in two distinct but related cases: - kernel mode NEON will leave the FPU disabled after it completes, so any kernel code sequence that enables the FPU and subsequently accesses its registers needs to disable softirqs until it completes; - kernel_neon_begin() will preserve the userland VFP state in memory, and if it interrupts the ordinary VFP state preserve sequence, the latter will resume execution with the VFP registers corrupted, and happily save them to memory. Given that disabling softirqs also disables preemption, we can replace the existing preempt_disable/enable occurrences in the VFP state handling asm code with new macros that dis/enable softirqs instead. In the VFP state handling C code, add local_bh_disable/enable() calls in those places where the VFP state is preserved. One thing to keep in mind is that, once we allow NEON use in softirq context, the result of any such interruption is that the FPEXC_EN bit in the FPEXC register will be cleared, and vfp_current_hw_state[cpu] will be NULL. This means that any sequence that [conditionally] clears FPEXC_EN and/or sets vfp_current_hw_state[cpu] to NULL does not need to run with softirqs disabled, as the result will be the same. Furthermore, the handling of THREAD_NOTIFY_SWITCH is guaranteed to run with IRQs disabled, and so it does not need protection from softirq interruptions either. Signed-off-by: Ard Biesheuvel Reviewed-by: Linus Walleij --- arch/arm/include/asm/assembler.h | 19 ++++++++++++------- arch/arm/kernel/asm-offsets.c | 1 + arch/arm/vfp/entry.S | 4 ++-- arch/arm/vfp/vfphw.S | 4 ++-- arch/arm/vfp/vfpmodule.c | 8 +++++++- 5 files changed, 24 insertions(+), 12 deletions(-) diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h index 90fbe4a3f9c8472f..df999b75c0e25b01 100644 --- a/arch/arm/include/asm/assembler.h +++ b/arch/arm/include/asm/assembler.h @@ -236,21 +236,26 @@ THUMB( fpreg .req r7 ) sub \tmp, \tmp, #1 @ decrement it str \tmp, [\ti, #TI_PREEMPT] .endm - - .macro dec_preempt_count_ti, ti, tmp - get_thread_info \ti - dec_preempt_count \ti, \tmp - .endm #else .macro inc_preempt_count, ti, tmp .endm .macro dec_preempt_count, ti, tmp .endm +#endif + + .macro local_bh_disable, ti, tmp + ldr \tmp, [\ti, #TI_PREEMPT] + add \tmp, \tmp, #SOFTIRQ_DISABLE_OFFSET + str \tmp, [\ti, #TI_PREEMPT] + .endm - .macro dec_preempt_count_ti, ti, tmp + .macro local_bh_enable_ti, ti, tmp + get_thread_info \ti + ldr \tmp, [\ti, #TI_PREEMPT] + sub \tmp, \tmp, #SOFTIRQ_DISABLE_OFFSET + str \tmp, [\ti, #TI_PREEMPT] .endm -#endif #define USERL(l, x...) \ 9999: x; \ diff --git a/arch/arm/kernel/asm-offsets.c b/arch/arm/kernel/asm-offsets.c index 2c8d76fd7c66298a..38121c59cbc26cdd 100644 --- a/arch/arm/kernel/asm-offsets.c +++ b/arch/arm/kernel/asm-offsets.c @@ -56,6 +56,7 @@ int main(void) DEFINE(VFP_CPU, offsetof(union vfp_state, hard.cpu)); #endif #endif + DEFINE(SOFTIRQ_DISABLE_OFFSET,SOFTIRQ_DISABLE_OFFSET); #ifdef CONFIG_ARM_THUMBEE DEFINE(TI_THUMBEE_STATE, offsetof(struct thread_info, thumbee_state)); #endif diff --git a/arch/arm/vfp/entry.S b/arch/arm/vfp/entry.S index 27b0a1f27fbdf392..9a89264cdcc0b46e 100644 --- a/arch/arm/vfp/entry.S +++ b/arch/arm/vfp/entry.S @@ -22,7 +22,7 @@ @ IRQs enabled. @ ENTRY(do_vfp) - inc_preempt_count r10, r4 + local_bh_disable r10, r4 ldr r4, .LCvfp ldr r11, [r10, #TI_CPU] @ CPU number add r10, r10, #TI_VFPSTATE @ r10 = workspace @@ -30,7 +30,7 @@ ENTRY(do_vfp) ENDPROC(do_vfp) ENTRY(vfp_null_entry) - dec_preempt_count_ti r10, r4 + local_bh_enable_ti r10, r4 ret lr ENDPROC(vfp_null_entry) diff --git a/arch/arm/vfp/vfphw.S b/arch/arm/vfp/vfphw.S index 6f7926c9c1790f66..26c4f61ecfa39638 100644 --- a/arch/arm/vfp/vfphw.S +++ b/arch/arm/vfp/vfphw.S @@ -175,7 +175,7 @@ vfp_hw_state_valid: @ else it's one 32-bit instruction, so @ always subtract 4 from the following @ instruction address. - dec_preempt_count_ti r10, r4 + local_bh_enable_ti r10, r4 ret r9 @ we think we have handled things @@ -200,7 +200,7 @@ skip: @ not recognised by VFP DBGSTR "not VFP" - dec_preempt_count_ti r10, r4 + local_bh_enable_ti r10, r4 ret lr process_exception: diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index 2cb355c1b5b71694..8f5bc672b4aac04a 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -416,7 +416,7 @@ void VFP_bounce(u32 trigger, u32 fpexc, struct pt_regs *regs) if (exceptions) vfp_raise_exceptions(exceptions, trigger, orig_fpscr, regs); exit: - preempt_enable(); + local_bh_enable(); } static void vfp_enable(void *unused) @@ -517,6 +517,8 @@ void vfp_sync_hwstate(struct thread_info *thread) { unsigned int cpu = get_cpu(); + local_bh_disable(); + if (vfp_state_in_hw(cpu, thread)) { u32 fpexc = fmrx(FPEXC); @@ -528,6 +530,7 @@ void vfp_sync_hwstate(struct thread_info *thread) fmxr(FPEXC, fpexc); } + local_bh_enable(); put_cpu(); } @@ -717,6 +720,8 @@ void kernel_neon_begin(void) unsigned int cpu; u32 fpexc; + local_bh_disable(); + /* * Kernel mode NEON is only allowed outside of interrupt context * with preemption disabled. This will make sure that the kernel @@ -739,6 +744,7 @@ void kernel_neon_begin(void) vfp_save_state(vfp_current_hw_state[cpu], fpexc); #endif vfp_current_hw_state[cpu] = NULL; + local_bh_enable(); } EXPORT_SYMBOL(kernel_neon_begin); From patchwork Wed Dec 7 10:39:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 13066960 X-Patchwork-Delegate: herbert@gondor.apana.org.au Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2084EC4708D for ; Wed, 7 Dec 2022 10:39:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229486AbiLGKj5 (ORCPT ); Wed, 7 Dec 2022 05:39:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36906 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229611AbiLGKjz (ORCPT ); Wed, 7 Dec 2022 05:39:55 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88E7EB96 for ; Wed, 7 Dec 2022 02:39:54 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 25C5A612B7 for ; Wed, 7 Dec 2022 10:39:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 39E7AC433D7; Wed, 7 Dec 2022 10:39:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1670409593; bh=36YqQnOtNbUcTuWjMaDed0znfLGCdnAv3m9xyuWbSXQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CpOntchcb/BjKuYFzbkI72PcqkLOWGTs7yWmlHb9uwAyP5eAuhImrbOILLECobrnf kmAaRTu0MPFpgM81RRY4onOsdSOVYph+clKOcWO1giiRWif17+6TMGUukd07+5ag/6 Zeg1an9p1WXRLlscvhS1a8X7CLgFX2xeWpV+KN7Dy0ysW0pYkT6Gi1CE+jbKUczirk 8xWGhn93hGxziAGpKT4ggR9FTIAktPKG+is4liHzCe2eE/PLlUSTB79iOX0nEeNWOg Ik7/bigwdF4IvVc8Ho7PRz2A6/5mgvkVbAUgLYtrRckNh2F0Y4ZPZFbNVlE4b1Gp/z Caq8pnfQnfW9A== From: Ard Biesheuvel To: linux-arm-kernel@lists.infradead.org, linux@armlinux.org.uk Cc: linux-crypto@vger.kernel.org, Ard Biesheuvel , Linus Walleij , Arnd Bergmann Subject: [PATCH v2 2/2] ARM: permit non-nested kernel mode NEON in softirq context Date: Wed, 7 Dec 2022 11:39:36 +0100 Message-Id: <20221207103936.2198407-3-ardb@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221207103936.2198407-1-ardb@kernel.org> References: <20221207103936.2198407-1-ardb@kernel.org> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2672; i=ardb@kernel.org; h=from:subject; bh=36YqQnOtNbUcTuWjMaDed0znfLGCdnAv3m9xyuWbSXQ=; b=owEB7QES/pANAwAKAcNPIjmS2Y8kAcsmYgBjkG1nbMBKGcpxvDP1xaATdLbbvNcgx0uP63Vp8/Kd oyMBLy+JAbMEAAEKAB0WIQT72WJ8QGnJQhU3VynDTyI5ktmPJAUCY5BtZwAKCRDDTyI5ktmPJFr7C/ 4zG0d1Qfu4iWKy50r3Mru2HgU59zvrANrytytJUxsNJGc3Y7Cuc17Qd++YkZRhLSRWJrDAw4MBjty4 w/yZeVXQc6FB042MvUV4+n5MZQdeLH2KUqOZmNbhQdmGz18gma779rpMkeTaPOmBB4EYtiab6tuL0T VUbR2GAWQoS+WEnE5vgoa5ERpuVqo4U2DmDVwzFH0aZDtD7cuNoQc7wDdGgkhZdIn4C4UHfzlx8Ml2 sBvGdpbfMo3h46cHDbIqAcybT1TiL4FSkDFgU0ILnATv09/5MVyDAcd46BxQCj6kJshIAFBE9af8v/ h2JuUPqF+pVOSDTHh98E7IJMCC2lriAKAnWoUETboNJfltSSIuU3e+X07Dxl9bl6qpE18NcQFt7v1W 4bHaigetdBQ6YqElgqx7PI2r8qB+d6pbgJKCqDs535Eh0YNH/I1QtnWXGLAAFQMG1W/vD0V2BWSIKm 84CXN0WRxWxqB4lWgV2+Tw4IZA78xHUguyiFrl9pJCjl8= X-Developer-Key: i=ardb@kernel.org; a=openpgp; fpr=F43D03328115A198C90016883D200E9CA6329909 Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org We currently only permit kernel mode NEON in process context, to avoid the need to preserve/restore the NEON register file when taking an exception while running in the kernel. Like we did on arm64, we can relax this restriction substantially, by permitting kernel mode NEON from softirq context, while ensuring that softirq processing is disabled when the NEON is being used in task context. This guarantees that only NEON context belonging to user space needs to be preserved and restored, which is already taken care of. This is especially relevant for network encryption, where incoming frames are typically handled in softirq context, and deferring software decryption to a kernel thread or falling back to C code are both undesirable from a performance PoV. Signed-off-by: Ard Biesheuvel Reviewed-by: Linus Walleij --- arch/arm/include/asm/simd.h | 8 ++++++++ arch/arm/vfp/vfpmodule.c | 13 ++++++------- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/arch/arm/include/asm/simd.h b/arch/arm/include/asm/simd.h new file mode 100644 index 0000000000000000..82191dbd7e78a036 --- /dev/null +++ b/arch/arm/include/asm/simd.h @@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include + +static __must_check inline bool may_use_simd(void) +{ + return IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && !in_hardirq(); +} diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c index 8f5bc672b4aac04a..4e1a786df76df157 100644 --- a/arch/arm/vfp/vfpmodule.c +++ b/arch/arm/vfp/vfpmodule.c @@ -723,12 +723,12 @@ void kernel_neon_begin(void) local_bh_disable(); /* - * Kernel mode NEON is only allowed outside of interrupt context - * with preemption disabled. This will make sure that the kernel - * mode NEON register contents never need to be preserved. + * Kernel mode NEON is only allowed outside of hardirq context with + * preemption and softirq processing disabled. This will make sure that + * the kernel mode NEON register contents never need to be preserved. */ - BUG_ON(in_interrupt()); - cpu = get_cpu(); + BUG_ON(in_hardirq()); + cpu = __smp_processor_id(); fpexc = fmrx(FPEXC) | FPEXC_EN; fmxr(FPEXC, fpexc); @@ -744,7 +744,6 @@ void kernel_neon_begin(void) vfp_save_state(vfp_current_hw_state[cpu], fpexc); #endif vfp_current_hw_state[cpu] = NULL; - local_bh_enable(); } EXPORT_SYMBOL(kernel_neon_begin); @@ -752,7 +751,7 @@ void kernel_neon_end(void) { /* Disable the NEON/VFP unit. */ fmxr(FPEXC, fmrx(FPEXC) & ~FPEXC_EN); - put_cpu(); + local_bh_enable(); } EXPORT_SYMBOL(kernel_neon_end);