From patchwork Fri Jul 21 11:28:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13321901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19007EB64DC for ; Fri, 21 Jul 2023 11:29:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0YiCps47gGMqsAsh6gmrFN1N3pyS4WMUnCfWDxJG5Yg=; b=Ledj9t4IrOrG11 vMgSa9DIy9IxI/7lMpv+xiBqt+clk6Awch86MQ1c71rJnZy/6ouNg1ov5UhOj4U/ukLnQm4K9BY0c pjmojHmsP5xQOpLhBsx2/pWHtnyXvcbwRlMNZdsQwH6lPeOoMQ38+NdwDDTBJ6/WYZoGcivqIVxVS wa/rC+EMBPAcWvmIyGwqqRRa4KssjBlRm7qAavE8NG6NuAMDBSkjzshxrkBtSqq9yYCa5kKxBE0/A 65vH28ntnMMc9ZULLyqfaddnOAFNJOW8Q2kLNOphZ0A8/0tidZ1MWyjZlgdKoIExRPe1/cVdOdMtp H5X43fUPsuNVeiXNGxdw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMoJy-00Dvbe-29; Fri, 21 Jul 2023 11:29:18 +0000 Received: from mail-pl1-x62a.google.com ([2607:f8b0:4864:20::62a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMoJv-00Dvas-1m for linux-riscv@lists.infradead.org; Fri, 21 Jul 2023 11:29:17 +0000 Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-1b8b2886364so11613615ad.0 for ; Fri, 21 Jul 2023 04:29:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689938954; x=1690543754; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=pIlcMjCWX1E0dFpG7PlIot+p0pb5aBIdeoNALbljqW8=; b=mxUqp0waZbf7mSdX1yT6sea/FPoIo3mnePwNGairdKsS4g9sEslGJkTIUmdNfW7r/b ttmA5uPyp3a0L43JqmKKaD4bxibLS2SVdAn0bZdPaK9P4mxvUIquUxzW5Lr1fGsop2P6 gWhzHgsa8zlzsYft+gBhB14Lq4uIOLICsNigyFpmvKUqHF6jzvpfXRKdvPQkr5RQH7zT H8rQgpP/bXXLWH90behHa7a3ltLwcAeh4p6Jfr6tHYl+h+G9z1hl7l5v2RVkWu/hzwo0 Ie+z/Qz6X2xFzj2kGng+nM9HLhmjHbVwapxsbZgwcS05cdz3dnMwBIMgQis+4htVv5GA oEkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689938954; x=1690543754; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pIlcMjCWX1E0dFpG7PlIot+p0pb5aBIdeoNALbljqW8=; b=XTemt+68k0KCTFB+6/btXwfySh40q14nGUyPkEOBiQ57fkeb/BeJAI6f6ZW8VkAB40 WusNF4iXP+vpAMfClQX0TLG0xcxyEunvC3I+z6MbAvvm39aIPby9ECSSNTy/bwOrLh4+ nPJnzlzMqaB4zOBoC+VLOzSFHxgm+MMvy6TVLAwt2fWDkmPqjr8ydUWtOL/z4cicdjWv W72FB5id+MA2xs2gUwMunvTTtD9nTNrTHgKKhVCAw8Sh0i9UNE7P6INjeiLdCwrlGvRk rMS/xkESny86pjNgMnIkxClA9AXOWC07/OBy2piHexDdqvYYO7Gi+m6zBx18Z2vRjt/m VC3w== X-Gm-Message-State: ABy/qLaiHY+BVnncGOmAMxShmyQhqciiPGTyRZDJOO0A6hxQ0ImCes57 SHz19IJmgXINKJWiBNdkgLGSmBmYLOC/mvdHURvI8JJzQ3iIF5izwlH2dImx0JXrhFKxYtVwiLb WQBvmB1hrC40KF/OQaoqPm9voz42uC4cB/ljdvn1pWSVJZcAtd0gjyDy+9E9p7MYId9Qxde2rOc wHAJi/B2SYhNiJ X-Google-Smtp-Source: APBJJlFNHSxnpeC7NP25M4tGkB/cid/MNXzsabfkLVmehACLopNpmCNeb1fN/hYkSt23X6kEFWrz+A== X-Received: by 2002:a17:902:d352:b0:1b7:d0b3:1678 with SMTP id l18-20020a170902d35200b001b7d0b31678mr1199110plk.17.1689938953653; Fri, 21 Jul 2023 04:29:13 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id c1-20020a170903234100b001b8953365aesm3243121plh.22.2023.07.21.04.29.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 04:29:12 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Andy Chiu , Albert Ou , Guo Ren , Conor Dooley , Yipeng Zou , Jisheng Zhang , Vincent Chen , Heiko Stuebner , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Peter Zijlstra , Al Viro , Mathis Salmen , Andrew Bresticker Subject: [v2, 1/5] riscv: sched: defer restoring Vector context for user Date: Fri, 21 Jul 2023 11:28:51 +0000 Message-Id: <20230721112855.1006-2-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230721112855.1006-1-andy.chiu@sifive.com> References: <20230721112855.1006-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230721_042915_591023_5EAA7FD6 X-CRM114-Status: GOOD ( 16.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches. Signed-off-by: Andy Chiu Acked-by: Conor Dooley Reviewed-by: Björn Töpel --- Changelog v2: - rename and add comment for the new thread flag (Conor) --- arch/riscv/include/asm/entry-common.h | 13 +++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/signal.c | 2 +- arch/riscv/kernel/vector.c | 2 +- 6 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 6e4dee49d84b..52926f4d8d7c 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,19 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..b182f2d03e25 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -93,12 +93,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 3d78930cab51..a4f3705fd144 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -183,6 +183,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -190,7 +199,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index e32d737e039f..ec89e7edb6fd 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -153,6 +153,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -169,6 +170,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 180d951d3624..0fca2c128b5f 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -134,7 +134,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 8d92fb6c522c..9d583b760db4 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Fri Jul 21 11:28:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13321902 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 29344EB64DC for ; Fri, 21 Jul 2023 11:29:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oQJvBIyKTVN/1hd3hlyaZuOdcfgreFCbycPLMwfDhDQ=; b=DU81bh1mVYWJhn lZTPqavSijkzSIZUr9u+uJvp0qNNU+bnvvBImHvfVGVIMuQ/bVNQ9A2DFoCpa6KYPOhka4A2mH3AT dnCLjACb7GEB8XGhVyXuKRtifwCnzv0nm8+7TXKImMbiwU9ILduwl3qawEMshdvxiVamfj0WlNjsf 3VH6q//w2j14S8DEDvDHpbbiHkx+MpC5zVSUPro8k2jSAbJ1ZMcpPeeU/St3DlnZu1RIposdI7nks NP5tWaKmF5FPMr6NjwJAd6n/EZJPM24Ag9m621jfAaVSUWb6NDjH/tv4vMjDJkjJ6UtfgdWcSt+Jd +aIBpqvmWuiYiMy9lDPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKA-00Dvel-1p; Fri, 21 Jul 2023 11:29:30 +0000 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMoK6-00Dvd8-3C for linux-riscv@lists.infradead.org; Fri, 21 Jul 2023 11:29:28 +0000 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1b8ad907ba4so11098265ad.0 for ; Fri, 21 Jul 2023 04:29:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689938965; x=1690543765; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=xaxT72bLPg81xAbv2lFbqPbe5HQSZki7btL2KEJHTXk=; b=HhqdGMe6EvmQ1XaYtLlT1ByS4n6SKUUnmTimJcaGiUT9OqfdjCCOmoeGU5zgZGfh/E q84IZDC5h+9y4jgoY41KcNnh4SfXZR3dmlkpSD9ndseNCkKIec1j1Fgu1bYxfQ2cycUT GKP/pYgyU2hkETxmzC4YlI0TLuMnk7INVaV2Ldq3qNu8ess0bf0zYTWfprbJmBeLqkey QpwIroYAyJeu/HhNySplQIBiJu3Uvh35FbZshu1dYrcA+4a+YLtu8a3oXeybBTaMzEG9 /5aEatm68NhSPE0A6VYb0hHAPk8Y3nhrqUPnHxnuCWxwD2xEepxPW9GBbRRHSaUsGLnz 3zdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689938965; x=1690543765; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xaxT72bLPg81xAbv2lFbqPbe5HQSZki7btL2KEJHTXk=; b=kQZYMPmTLSUSUVo/X7UXTIWtqUZ7/otEYAMb09z5r1X3UaBPiQQMemSLJojJLPjWLS 02DHBAknU1kPT9/B703NqpDCRbfR5djl+gedBjN0lsCB6RdzgbA5GUyJctC3hetmGnt4 w1Cnh/X9oDe2cECvh2K9Xjvki1hDKvX3xOxfF8HvquBZnr7aR33G8kuDTwuJSQ8fr3Jn eIlf2tNxESV/slzQ1M6iv9gT+a4OGffEjpkr1VbJvt6J8vvZ3QofZwIFIVkVAjJNopQO mNwl6UhHiGZTOFTpC915CoTKHwUj/YFx29+jBffF0u+UXR5yhYXiyxLr4OsEBgGjU5rn Ve8w== X-Gm-Message-State: ABy/qLYxwhOU1EBdL6QgTDMsONQ2cBmD7iQFfD64hy18zFFv2IcnVZXE Fk5Sd74FE3aTMuLo61zYV0XbG2GnasISGhzFx60omtnWdQzBV8dD1EBfSsR+vW/1su6uTCxiBJO Zh41CFbXwQz7DwuxYuuSHnXrcnBO6AKo2BJTBjrv+FnAqOOza5hzQrtBgezMYJKUlq9PwMtLsHe OooQFADvnsgGXd X-Google-Smtp-Source: APBJJlH0oc3sMlUMXaV7WwhNdBlT9X2+5ZaY03PVBJVV1r71KD5MLHrnJ+wz5HCSD+J7s5H6aX0p0A== X-Received: by 2002:a17:903:41cb:b0:1b8:ab0d:cd5 with SMTP id u11-20020a17090341cb00b001b8ab0d0cd5mr1763945ple.49.1689938964811; Fri, 21 Jul 2023 04:29:24 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id c1-20020a170903234100b001b8953365aesm3243121plh.22.2023.07.21.04.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 04:29:24 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Vincent Chen , Andy Chiu , Albert Ou , Heiko Stuebner , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Alexandre Ghiti , Xianting Tian , Sia Jee Heng , Anup Patel , Jisheng Zhang , Masahiro Yamada Subject: [v2, 2/5] riscv: Add support for kernel mode vector Date: Fri, 21 Jul 2023 11:28:52 +0000 Message-Id: <20230721112855.1006-3-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230721112855.1006-1-andy.chiu@sifive.com> References: <20230721112855.1006-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230721_042927_030289_5AB92DDD X-CRM114-Status: GOOD ( 19.51 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- Changelog v2: - 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor) - export may_use_simd to include/asm/simd.h --- arch/riscv/include/asm/simd.h | 50 ++++++++++++ arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 101 +++++++++++++++++++++++++ 4 files changed, 154 insertions(+) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h new file mode 100644 index 000000000000..ef70af78005d --- /dev/null +++ b/arch/riscv/include/asm/simd.h @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2023 SiFive + */ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include +#include +#include +#include +#include + +#ifdef CONFIG_RISCV_ISA_V + +DECLARE_PER_CPU(bool, vector_context_busy); + +/* + * may_use_simd - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_simd(void) +{ + /* + * vector_context_busy is only set while preemption is disabled, + * and is clear whenever preemption is enabled. Since + * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy + * cannot change under our feet -- if it's set we cannot be + * migrated, and if it's clear we cannot be migrated to a CPU + * where it is set. + */ + return !in_irq() && !irqs_disabled() && !in_nmi() && + !this_cpu_read(vector_context_busy); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd(void) +{ + return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index a4f3705fd144..b46b8f3261fa 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,8 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +void kernel_vector_begin(void); +void kernel_vector_end(void); static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index 506cc4a9a45a..3f4435746af7 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -61,6 +61,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..1c3b32d2b340 --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,101 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include +#include + +DEFINE_PER_CPU(bool, vector_context_busy); + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +static void get_cpu_vector_context(void) +{ + bool busy; + + preempt_disable(); + busy = __this_cpu_xchg(vector_context_busy, true); + + WARN_ON(busy); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +static void put_cpu_vector_context(void) +{ + bool busy = __this_cpu_xchg(vector_context_busy, false); + + WARN_ON(!busy); + preempt_enable(); +} + +/* + * kernel_vector_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_simd() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_vector_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_vector_end() is + * called. + */ +void kernel_vector_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + BUG_ON(!may_use_simd()); + + riscv_v_vstate_save(current, task_pt_regs(current)); + + get_cpu_vector_context(); + + riscv_v_enable(); + + return 0; +} +EXPORT_SYMBOL_GPL(kernel_vector_begin); + +/* + * kernel_vector_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_vector_begin() was previously + * called, with no call to kernel_vector_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_vector_begin() is called again in the meantime. + */ +void kernel_vector_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + riscv_v_vstate_set_restore(current, task_pt_regs(current)); + + riscv_v_disable(); + + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_vector_end); From patchwork Fri Jul 21 11:28:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13321903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C56ABEB64DD for ; Fri, 21 Jul 2023 11:29:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=RzSaAbDo3PGEgJZaDD9iduIltugwEHjg8yVMP+z3O+A=; b=qLhbPw2W1pisfJ B42ARTssXDK7Y4emeTHAmJ5Yibv8Gi0UjkfUmzjIFkQNOs0eWxxiOwj8kgBHKoAsIUI1SH0rIbVpU oojCo0gkQAXoNKTRLvTkKgLYM/tPzVbrJ2vbD1XaWDwyR9vlRBpvj402NCvynVQyMl8l4iSuvyYLk YFYo2YTY/AlQ7IgW8VIa6S9HUBGj0/wW02RidqGnGaWJWdtlnz/UJcsjC32F73d1DwraZk8DDocBb to4dNUZYMp9B/Raci/LZvPM8PN5Yzokn0HQ3uPuiOQ6kxV/4FqBNTv6Fd/jLqpyTdOHdfT1YLVhaB Z1pYAGQoJDUBdShQZCwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKI-00DvhN-1A; Fri, 21 Jul 2023 11:29:38 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKF-00DvgL-2f for linux-riscv@lists.infradead.org; Fri, 21 Jul 2023 11:29:37 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1b852785a65so13153705ad.0 for ; Fri, 21 Jul 2023 04:29:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689938974; x=1690543774; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=n/dQQT78NiCGg+HnQh/L41o2qCUd4gXvPmUFJt519QU=; b=Rf8k5wxj03zve5ghmqobqtnsyZOVM5/QCD5DmBYLKlMWKvGLRDl3SLUcMllvR3Af2F tYfuZWB+njx0W5r+VA/BxqPo8wCcJct8GJOk1U69r5XV4VGgoWXRXcyNJ7O8xzGhsn/c xABKEFbfLwTqTIddU7jh1YvxRYEq5ttNIJN+K4mWyaFJCQzfQDzzFuVDLRd5qH7i5rLX 6fQo5ao0zg7AJwH7ybEY/A30RjbRp1pNd3wxZU8nj9/5TgvA9mcXbcmGzKBZvOov9eFm S28L07uys+DPSO0/WMhvdVNo2m+kx2UHGYDSjuUuiM3c+25fTeOy4Wmd6zJ8iEFjFpne DOqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689938974; x=1690543774; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n/dQQT78NiCGg+HnQh/L41o2qCUd4gXvPmUFJt519QU=; b=RMPBLWVfGqhegfyhco6if6uVX1lJmd8e/lzQ+gecuVfP6ycQAbynCr7afbeSCgtsxg iyPgvy/dV63K9YLke0W+c63/HgnN4d/W/i+MRuNncwKBuLW8Bx4GuCZGKtEd3vy4wjXF WFcYWtGuSL+B8jXcm2rrGgmh6dQIH/fB02HiRFWErj9iWWO3r5yMp4x+alUrLAdDVzR7 7EroqlH23bl9linPj6AKxIDKJ10xemz6f2lEOHDoKNJ9cB6PE56HJsX91LY05j8gejoE 4B3JW7vmC3BbE/Un7ATExecrDC6SN+DHxlzl2CBeESMD1igY3AIKur143d1ZIr6GO8BR dBNw== X-Gm-Message-State: ABy/qLZx1/ihN5+j/3pGD320K6dnFOuTVj0boqH/qi1ggS3SBwMJsE3T ZiW0e0il+WTjOG5tdWmaP8IzBSQNwHSI4PTtJX3Z/64m8F76awcAIB+gQK9ROz5Wf3LMy4XNB4Y MRBPDnZqB/KtGmpfH/hM5sw6lIMR8lz1E0tYLa0/BNNh/zmTAwH/huh8R1acDSbJ+ZcF3/wzIAn T2kp0d4MDuvtsm X-Google-Smtp-Source: APBJJlH8GXIrKFP4W1EEhyGXEpTAOKZM8UJU+L9lqAb1LLkSd0Xzo8yMTDY2fBQ+oMdn4PuGmQxIQw== X-Received: by 2002:a17:903:2282:b0:1b8:c8bc:c81b with SMTP id b2-20020a170903228200b001b8c8bcc81bmr2254495plh.21.1689938973708; Fri, 21 Jul 2023 04:29:33 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id c1-20020a170903234100b001b8953365aesm3243121plh.22.2023.07.21.04.29.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 04:29:32 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Han-Kuan Chen , Andy Chiu , Albert Ou , Conor Dooley , Andrew Jones , Heiko Stuebner Subject: [v2, 3/5] riscv: Add vector extension XOR implementation Date: Fri, 21 Jul 2023 11:28:53 +0000 Message-Id: <20230721112855.1006-4-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230721112855.1006-1-andy.chiu@sifive.com> References: <20230721112855.1006-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230721_042935_865460_31975586 X-CRM114-Status: GOOD ( 15.13 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v2: - 's/rvv/vector/' (Conor) --- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..903c3275f8d0 --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_vector_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_vector_end(); +} + +static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_vector_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_vector_end(); +} + +static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_vector_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_vector_end(); +} + +static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_vector_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_vector_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..3bc059e18171 --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +ENTRY(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +ENTRY(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +ENTRY(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +ENTRY(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Fri Jul 21 11:28:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13321904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5383EB64DD for ; Fri, 21 Jul 2023 11:29:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JocAyNlml9yT69M2CrwXRzYeaXTlC+pmy3bmNa/yhmU=; b=SCHg3kryeldIF2 WUmX2ahxMIKEEFxKziBKV/S6ixKgG7P8s6prYIbIihR/jgeKpJHytZMUTXl4KlgcJ6Fj1VGiQioXY 2fYg9hhFgHlFkAgOL5azRlVf3Wg8P++Z8gvSPVQn+5CyJx6xS7nFiFK+qsUccaeNllYVPSAk1eOAh mdJ/i+oYIXHXiNdOGey7tpQuAvKm0DJ8lprScG7Yktt05+YdJ29snEH/m1cTKqiwxcujgtbhYNFPw E/M+SFt5kBW4q+6/oBl0ILDYEAwyBSRrOegwvEowQYGFNp4Aa1SdPze8UpS+BM8B30VsDL4DEs5uA 91FT+IGfpxSXcSRdJiGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKS-00Dvl9-0X; Fri, 21 Jul 2023 11:29:48 +0000 Received: from mail-pl1-x635.google.com ([2607:f8b0:4864:20::635]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKP-00Dvk5-2S for linux-riscv@lists.infradead.org; Fri, 21 Jul 2023 11:29:47 +0000 Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1b8bbcfd89aso11007025ad.1 for ; Fri, 21 Jul 2023 04:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689938984; x=1690543784; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=uZILjIbAwnrKcvzn2vcEaPpCGw9AS7QD5tUprtRHXrI=; b=j+HXiPbceMC2s/jQ2pMYHvFOtUshjcTG5tDc8H116vzuXK05CuRuV3Q8mAO7U2MVuV Xemf/CN93YhpzoP3A58I6Y2HYMonRnQpM0pQwqPgKUpTYqhI9+54wY0izLJ0xh3/9J7A WmkKxeoVstDS9JDRxK7gFR2dQ+7wXfB3KwgQzwPFtm2S7uwBHjg1RPxalbMHs3r3kGGq N41GoQYsdgPARp66VwhFfffxSsOFIVgx+BxhBErk4WeJg0bUKyWLCzB+pkShnEvcCMSA azsOf6Ve2R7xIR0GLK9LJGiQ+MqvRgu7jzaBbA0AKQBuBRsCsAyqwJev8cvNe9a3aYw0 kg8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689938984; x=1690543784; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uZILjIbAwnrKcvzn2vcEaPpCGw9AS7QD5tUprtRHXrI=; b=VM2l4VM7JGqJmenbRi4xjXCx52YlBhaoxCPSgmPoKaipcV6XmDiNkxDVXPJA9sVunA 5wGoutDYF4eAa1uA0OFjH02xK3b1tS0IeOYvtbQzYx2Jh1mmV2sYwMNDKxZM+XJjd9/L A7rW4Gvh6bVlk5Yr14cDR/IL/fT4Sqh1k39nOC2V8xN7oG5I0uKNgeYJwVolqXv8jDNG thj5WvHy4eumePN5OoFa0sCfDSak0Vxml7yEt4NjfOe6yeDV9kpZnP/TYnBD2pBPQCl2 MWSkmpQ/3UYA7dHOQ/puy4+7U6sX/P5KajGFNQlXRCfqvN6ppyOC7EuDAtz4HlWX2m9d GrDQ== X-Gm-Message-State: ABy/qLbGeszLIkXyxXMyzzfxoSbpsfS1fwnteOci1jP6fdZbG3vAgkNH ImxBxbnqxWk+H3nA4wFFb1srMQZRGBJo9KTUQs+GWDGZQHQ3RLAJcWsFalglZIN2hna9qHz+wjI 1+VEUR+2Sk/3J0d5xd/dt7BNuvK+SBp3qytR13+m/X1hIubxPM+a9bnoXuaoUloYl6P4fPv+1wm aXL1Coy2fpvwch X-Google-Smtp-Source: APBJJlGpzbZorACpQuQBOR4OCmsQFrZMkzpaqIdrJekkeajtxsVfKnxTps1bQ5N8LjfNUG78BkP02Q== X-Received: by 2002:a17:902:d342:b0:1b8:c6aa:85c with SMTP id l2-20020a170902d34200b001b8c6aa085cmr1211380plk.45.1689938983678; Fri, 21 Jul 2023 04:29:43 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id c1-20020a170903234100b001b8953365aesm3243121plh.22.2023.07.21.04.29.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 04:29:42 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Andy Chiu , Albert Ou , Oleg Nesterov , Guo Ren , Conor Dooley , Yipeng Zou , Vincent Chen , Heiko Stuebner , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Andrew Bresticker , Mathis Salmen , Al Viro Subject: [v2, 4/5] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Fri, 21 Jul 2023 11:28:54 +0000 Message-Id: <20230721112855.1006-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230721112855.1006-1-andy.chiu@sifive.com> References: <20230721112855.1006-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230721_042945_799771_5F3A8DEE X-CRM114-Status: GOOD ( 12.49 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu --- Changelog v2: - fix build fail that get caught on this patch (Conor) --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 52926f4d8d7c..aa1b9e50d6c8 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, { if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index b46b8f3261fa..3b783b317112 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -163,23 +163,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -200,7 +196,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(&prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -218,8 +214,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 1c3b32d2b340..d9e097e68937 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -68,7 +68,7 @@ void kernel_vector_begin(void) BUG_ON(!may_use_simd()); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); get_cpu_vector_context(); diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 1d572cf3140f..85e7167245cc 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,7 +99,7 @@ static int riscv_vr_get(struct task_struct *target, * copying them to membuf. */ if (target == current) - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); /* Copy vector header from vstate. */ membuf_write(&to, vstate, offsetof(struct __riscv_v_ext_state, datap)); diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 0fca2c128b5f..75fd8cc05e10 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); From patchwork Fri Jul 21 11:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13321905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 83897C0015E for ; Fri, 21 Jul 2023 11:30:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KdZIX9SFyGYp5OuhOwdpQ3YFVdnpXlnjXbJ41HTuIiI=; b=uiBiX1kWQBHuzg aTlg0jqq9xnMaKWBsmvHhL1CgNaYd5TYhLCIvOw9vtq3aKQdR4hIjHSvgDNXEekWEt+dRN4KAH405 gxBRafF6J8po3IffP7bcsyJVIYj5sQxt3LS+ogyovTdowMgEJtw3p/k2YMe6d3dKCDcXzi4TsThYa stK8DGV+UDDtqBGRA7gIiFeArXTAJxk3A8MLveJCqZ5QIlX3H1HM1BCyY0yIkCXxJHfIzByOwgJPe HRcQLVJcupNuGN40n4dTLUMhBzwYcBaiW9d7qH82ZFjahSYVxmn+K1J4+SWvhN/5ugEikJfn6rcwF PuAFz6bY4iQ7fKAojsgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKi-00DvqW-0S; Fri, 21 Jul 2023 11:30:04 +0000 Received: from mail-pf1-x430.google.com ([2607:f8b0:4864:20::430]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qMoKf-00Dvp1-15 for linux-riscv@lists.infradead.org; Fri, 21 Jul 2023 11:30:03 +0000 Received: by mail-pf1-x430.google.com with SMTP id d2e1a72fcca58-6682909acadso1208867b3a.3 for ; Fri, 21 Jul 2023 04:30:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689938999; x=1690543799; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=/uyVAhGDh2zBFrgpJDJSo6VCu+QppJIYZj5NaU/WW3U=; b=gOj5HDP1nAB/H7HAQngWBy6S/5Mu7dQhagVYsIUjeNZnvfcu7iFzhz7v9csTylDPQC 7TzxoFJyYWOtq/D2InQvEme6Y/w1R8vx1gFQGXkimatk9Yg+HA4ipy+BfNTRqRUTtG76 VXYL6BpLJueA+N22EgXgIkudO/etew8E4Ma8LzhS44gWD7wnKLzOBZRma6Ni452RzC1d 1bBBOAEM4q4QQWfPLRl/Dl1lsZ/LXpUZ2PchQ1R7Fcmh0yPEGvWosX1N/skXWbQtPAdR iuZH3oY723SBFMLtmmaxSenj8qd07U2KsFQN4n3hHX536Y0PX/w7lgQl9il2RD8LOY8c 9R4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689938999; x=1690543799; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/uyVAhGDh2zBFrgpJDJSo6VCu+QppJIYZj5NaU/WW3U=; b=LsVW17iDoM8cOgNmfsqe99bwLkNod/Q2mN/lLGXSkdH69EIEz9fi3QgvpbxPo8DErl staDQG7TfBMq/2by3/Q2jF2EFp6iGrJRGkbHdZO3XH1XzdGgkafWgX4Bx9/nzjGEC1+U oj5EDrzNrHLyRHlmBUEKc2GoDeCnQM5taFceWee8iVhvXpZrUX0YHYDy6TCN4fElY3Fl ITcVpa/2KVrAnyRFa2Kv6Vf+M/ZdnS+NkIPCGFMH0tH4t3oDsD3h82Oyc4ZTIsw5OJGJ lesUrjBZuLpDA0tAEK+TlxcsLsmXT6oFyzNy65v6xx+8CVz5T9Wx5DVHDn50h7HX1RG5 kPQg== X-Gm-Message-State: ABy/qLaX+dmjjz4uv5Wndg0k3lLdIYNcy/q8Inw8FQvSn5b20Hu3XwLs /zX7WVRHMQAkHu9oAkqMnhgTyQ5TGsTSVovemi4HAoGnVi81R7OaA1JziLaXqWOgCeyTd9+dkCI oub3ja7KWh/IRU7FJ4dCE1RDVOQnYZwQxW05RyyBEjrTWKZyvE6Meq+SHwQmwpwIH8DmAMJXyIb HiQC8c8URPiWdM X-Google-Smtp-Source: APBJJlH7bEZW+YalWVnxq0XpKdD6pZUenshSavTAbIq6yqRuamW/dyC2Er9RAmQF1crxZN4cTOaPCw== X-Received: by 2002:a05:6a20:54a4:b0:137:a9d7:d8d3 with SMTP id i36-20020a056a2054a400b00137a9d7d8d3mr1406784pzk.31.1689938999185; Fri, 21 Jul 2023 04:29:59 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id c1-20020a170903234100b001b8953365aesm3243121plh.22.2023.07.21.04.29.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jul 2023 04:29:58 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Andy Chiu , Albert Ou , Guo Ren , Vincent Chen , Heiko Stuebner , Conor Dooley , Kefeng Wang , Jisheng Zhang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Sia Jee Heng , Mason Huo , Andrew Bresticker , Fangrui Song , Peter Zijlstra Subject: [v2, 5/5] riscv: vector: allow kernel-mode Vector with preemption Date: Fri, 21 Jul 2023 11:28:55 +0000 Message-Id: <20230721112855.1006-6-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230721112855.1006-1-andy.chiu@sifive.com> References: <20230721112855.1006-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230721_043001_375851_ECE82146 X-CRM114-Status: GOOD ( 27.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set. Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead. Signed-off-by: Andy Chiu --- Changelog v2: - fix build fail when compiling without RISCV_ISA_V (Conor) - 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor) - merge Kconfig patch into this oine (Conor). - 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor) - fix some typos (Conor) - enclose assembly with RISCV_ISA_V_PREEMPTIVE. - change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor) - 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/' --- arch/riscv/Kconfig | 10 +++++ arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/simd.h | 4 +- arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 +++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 45 ++++++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 10 files changed, 148 insertions(+), 10 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 4c07b9189c86..0622951b15dd 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -507,6 +507,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Ordinarily the kernel disables preemption before running in-kernel + Vector code. This config frees the kernel from disabling preemption + by adding memory on demand for tracking kernel's V-context. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index c950a8d9edef..497c0dd30b2a 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -42,6 +42,8 @@ struct thread_struct { unsigned long bad_cause; unsigned long vstate_ctrl; struct __riscv_v_ext_state vstate; + struct pt_regs *trap_pt_regs; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index ef70af78005d..a54a0ce58f4d 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_RISCV_ISA_V @@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void) * where it is set. */ return !in_irq() && !irqs_disabled() && !in_nmi() && - !this_cpu_read(vector_context_busy); + !this_cpu_read(vector_context_busy) && + !test_thread_flag(TIF_RISCV_V_KERNEL_MODE); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index b182f2d03e25..8797d520e8ef 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ +#define TIF_RISCV_V_KERNEL_MODE 13 /* kernel-mode Vector run with preemption-on */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) @@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KERNEL_MODE (1 << TIF_RISCV_V_KERNEL_MODE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE 0x20 + #endif /* _ASM_RISCV_THREAD_INFO_H */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 3b783b317112..c2776851d50d 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -195,9 +195,24 @@ static inline void __switch_to_vector(struct task_struct *prev, { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(&prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && + test_tsk_thread_flag(prev, TIF_RISCV_V_KERNEL_MODE)) { + regs = prev->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_save(&prev->thread.kernel_vstate, regs); + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && + test_tsk_thread_flag(next, TIF_RISCV_V_KERNEL_MODE)) { + regs = next->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_restore(&next->thread.kernel_vstate, regs); + } else { + riscv_v_vstate_set_restore(next, task_pt_regs(next)); + } } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); @@ -222,4 +237,10 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #endif /* CONFIG_RISCV_ISA_V */ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +void kernel_vector_allow_preemption(void); +#else +#define kernel_vector_allow_preemption() do {} while (0) +#endif + #endif /* ! __ASM_RISCV_VECTOR_H */ diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..4b062f7741b2 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -38,6 +38,8 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_THREAD_TRAP_REGP, task_struct, thread.trap_pt_regs); + OFFSET(TASK_THREAD_VSTATE_CTRL, task_struct, thread.vstate_ctrl); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..b6a7d4e9f526 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -66,6 +66,29 @@ _save_context: REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp) +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + /* + * Record the register set at the frame where in-kernel V registers are + * last alive. + */ + REG_L s0, TASK_TI_FLAGS(tp) + li s1, 1 << TIF_RISCV_V_KERNEL_MODE + and s0, s0, s1 + beqz s0, 1f + li s0, TASK_THREAD_TRAP_REGP + add s0, s0, tp + REG_L s1, (s0) + bnez s1, 1f + REG_S sp, (s0) + li s0, TASK_THREAD_VSTATE_CTRL + add s0, s0, tp + REG_L s1, (s0) + li s2, ~RISCV_V_VSTATE_CTRL_PREEMPTIBLE + and s1, s1, s2 + REG_S s1, (s0) +1: +#endif + /* * Set the scratch register to 0, so that if a recursive exception * occurs, the exception vector knows it came from the kernel @@ -129,6 +152,28 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + /* + * Clear tracking of the trap registers when we return to the frame + * that uses kernel mode Vector. + */ + REG_L s0, TASK_TI_FLAGS(tp) + li s1, 1 << TIF_RISCV_V_KERNEL_MODE + and s0, s0, s1 + beqz s0, 1f + li s0, TASK_THREAD_TRAP_REGP + add s0, s0, tp + REG_L s1, (s0) + bne s1, sp, 1f + REG_S x0, (s0) + li s0, TASK_THREAD_VSTATE_CTRL + add s0, s0, tp + REG_L s1, (s0) + ori s1, s1, RISCV_V_VSTATE_CTRL_PREEMPTIBLE + REG_S s1, (s0) +1: +#endif + REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index d9e097e68937..5c64f2034cdc 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -48,6 +49,44 @@ static void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +void kernel_vector_allow_preemption(void) +{ + current->thread.vstate_ctrl |= RISCV_V_VSTATE_CTRL_PREEMPTIBLE; +} + +static bool kernel_vector_preemptible(void) +{ + return !!(current->thread.vstate_ctrl & RISCV_V_VSTATE_CTRL_PREEMPTIBLE); +} + +static int riscv_v_start_kernel_context(void) +{ + struct __riscv_v_ext_state *vstate; + + vstate = ¤t->thread.kernel_vstate; + if (!vstate->datap) { + vstate->datap = kmalloc(riscv_v_vsize, GFP_KERNEL); + if (!vstate->datap) + return -ENOMEM; + } + + current->thread.trap_pt_regs = NULL; + WARN_ON(test_and_set_thread_flag(TIF_RISCV_V_KERNEL_MODE)); + return 0; +} + +static void riscv_v_stop_kernel_context(void) +{ + WARN_ON(!test_and_clear_thread_flag(TIF_RISCV_V_KERNEL_MODE)); + current->thread.trap_pt_regs = NULL; +} +#else +#define kernel_vector_preemptible() (false) +#define riscv_v_start_kernel_context() (0) +#define riscv_v_stop_kernel_context() do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context @@ -70,11 +109,14 @@ void kernel_vector_begin(void) riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); - get_cpu_vector_context(); + if (!preemptible() || !kernel_vector_preemptible()) { + get_cpu_vector_context(); + } else { + if (riscv_v_start_kernel_context()) + get_cpu_vector_context(); + } riscv_v_enable(); - - return 0; } EXPORT_SYMBOL_GPL(kernel_vector_begin); @@ -96,6 +138,9 @@ void kernel_vector_end(void) riscv_v_disable(); - put_cpu_vector_context(); + if (!test_thread_flag(TIF_RISCV_V_KERNEL_MODE)) + put_cpu_vector_context(); + else + riscv_v_stop_kernel_context(); } EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index ec89e7edb6fd..18cb37c305ab 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -160,8 +160,11 @@ void flush_thread(void) void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ - if (has_vector()) + if (has_vector()) { kfree(tsk->thread.vstate.datap); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE)) + kfree(tsk->thread.kernel_vstate.datap); + } } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -170,7 +173,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); + clear_tsk_thread_flag(dst, TIF_RISCV_V_KERNEL_MODE); return 0; } @@ -205,6 +210,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + kernel_vector_allow_preemption(); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 9d583b760db4..42f227077ee5 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void)