From patchwork Sat Jul 15 15:00:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13314550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9BE79C001B0 for ; Sat, 15 Jul 2023 15:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Xe+a6xIh7RPkiyLSF22q0SK4iiAREU1WlMcQlEEz3JQ=; b=wZrwf4dcrm1Y19 +aX0Ly93JPQ4bbeOJx5zHP4YnTi7+ON2tx36V5yHnwaYrOkGul5q5EtMdUdVEcJWQgciVFIMWwOvA hRyren7tCpdyeikUgm4Ycjx18AeXIfPByp1QKV5Sg2kyvxaLUOnv2T5u2nUAqA2p1gDFKNoVAvVzG uy07+alHDMOCRb6vJaE+gGxw/54chefru0qpzKPmgp1aGB4N/dwSwMT+Bqv3vAPjVBgBHuZVHP+Zj NGg96NqeTASvQMb9VLrvV2sQTFttzAKwWBN9y8VHTeU2vswWGPRL3aXV49Fcci7ep4PPPYMNDYSYK wGW2DSfsi7oJLP+Mk/qQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qKglS-008zaY-2h; Sat, 15 Jul 2023 15:00:54 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qKglP-008zYu-1L for linux-riscv@lists.infradead.org; Sat, 15 Jul 2023 15:00:53 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-666ecb21f86so2987310b3a.3 for ; Sat, 15 Jul 2023 08:00:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689433249; x=1692025249; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=F/225EHRyI/wX2MBgvShULrHjoLuMo1tn59JTCRbcX8=; b=gGbPrH3KzePQ2Nmhu764l8Dhr1SLQ+z5D0bmfzFY7FiJXLlOqUSxJXuYckXBpvQncJ 0DatUnqS9QSRG7kC8xLqkVscPlr3KTVfYAK1vxh6Z2epFFRd71AkAtfTbD5uGoW8lkkQ 5YCv13Op9CW4c0GjLa5A9IsGCcRSiRwMc3gj1aXfKS5PxmOyUJmYpobSUajVt2Iomn0M XfbyhsFcZF4Q7e+vpGQis3JNMeDUEQ/Jz9JtI8wMKy0ajFiSPm2g+r0NqvlwHlUrV0lz 1Kt4us7LZQDaKVw+FGWwPpT8ich6DwWN5L3UBWtjbS442Kf8brGPrTIVKD2CzWjr83gu iH0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689433249; x=1692025249; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=F/225EHRyI/wX2MBgvShULrHjoLuMo1tn59JTCRbcX8=; b=BjIRqyS06Lh7EMOJSZvHTmcVDHMT8cF+Q6PGgmZsDen+9jNkIcH/uClpY0LjeEim20 dKmSUkOdnBfkuas7O6c1eU05qGSKUD77KQ4higM/nnBzaHVZzwc90PlFtqGcUVmgtzao wiBCr38H6H4pWS/GVyvxJb2SN6tdNU5n4kU1GKrKnQVbbOt6s1CqdKjVOTVkTh1+GhTA K5sOIq83+wCWpohRTrAL5+9BHPtic2/m628pEGzWTqCSTxSQiBiByfVDNTXb/73md5cG jUqtM3LfMZ6hx2fEgibFsUJBptvZNnpKFr9lkI157VoeiOrLzDu0WYrKQb7QB2jXoHuD 4XLQ== X-Gm-Message-State: ABy/qLavoJ0/0iQc6+9qzN2PKvzu51UP0WOZ2Sgf9k4pjOk0cHezHoNX t9ECkI36ABvG3x/lZ8OPM48106SYX69LmgvaHMAfu0RJEyZ3HfmUNJ/iV0m11Sjds6I2g+OkrR2 1nBE3b9JmO3SFMexiQR8f4M/E+wNFosRyXjnja2E19C+6Sf64BOBWKpxcweKWC/KGaKS2zVT/ea rwFlNdBwr8y5Lj X-Google-Smtp-Source: APBJJlGQgAjLg2F86dxXasGz2GONxfjtWW4ZSQeLxCJ5Z1Z0ajPI7fsg+Q59uzvjQctFmwOt57szew== X-Received: by 2002:a05:6a20:3d1d:b0:132:bdba:5500 with SMTP id y29-20020a056a203d1d00b00132bdba5500mr9983066pzi.39.1689433249286; Sat, 15 Jul 2023 08:00:49 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id a28-20020a63705c000000b00528513c6bbcsm9356535pgn.28.2023.07.15.08.00.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jul 2023 08:00:48 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Andy Chiu , Albert Ou , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Jisheng Zhang , Huacai Chen , Conor Dooley , Vincent Chen , Peter Zijlstra , Andrew Bresticker Subject: [v1, 1/6] riscv: sched: defer restoring Vector context for user Date: Sat, 15 Jul 2023 15:00:27 +0000 Message-Id: <20230715150032.6917-2-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230715150032.6917-1-andy.chiu@sifive.com> References: <20230715150032.6917-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230715_080051_459367_B4949E23 X-CRM114-Status: GOOD ( 14.88 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User's will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs context swicth from A->B->C, and returns to C's userspace, then there is no need for restoring B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- arch/riscv/include/asm/entry-common.h | 13 +++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/signal.c | 2 +- arch/riscv/kernel/vector.c | 2 +- 6 files changed, 29 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 6e4dee49d84b..52926f4d8d7c 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,19 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 97e6f65ec176..d83975efe866 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -101,12 +101,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 3d78930cab51..a4f3705fd144 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -183,6 +183,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -190,7 +199,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index e32d737e039f..ec89e7edb6fd 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -153,6 +153,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -169,6 +170,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 180d951d3624..0fca2c128b5f 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -134,7 +134,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 8d92fb6c522c..9d583b760db4 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Sat Jul 15 15:00:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13314549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B29E0C001E0 for ; Sat, 15 Jul 2023 15:01:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=qDawcthkVV5SDsxO63e6MMKeomJjA3+170bSLE532lM=; b=l/xggM3pFepTNTHN/qkX1RpVkz PV16CBNPpLtsevNXSBMN31yGziaW6c1HR2u8+PcA2nryjufZ8eLxZxzDPPZUlSAoi35UqhpKsmGua i3OiGptYxckORRDlUKKzHYnyDQET6cAFDW67cbhp1oxwQniaTT9Lqi7DnJl53WadOdsjRVQDs3bHJ NbyylWu2EFAetmSAeY4KhSfCYg9x8jt2GktvMINzGZZousv6q80pNaRHSh6bBk0oeXkNboTghCNuy kfOd3hHwYBr75Edeu5qTRmrlSfxwqrquXqgKJtee2W8Z/ecr2xoHPZGMspNhM8tw8EIaVoxSvI0V2 T6/vD1+Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qKgla-008zbz-1q; Sat, 15 Jul 2023 15:01:06 +0000 Received: from mail-pf1-x431.google.com ([2607:f8b0:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qKglX-008zbA-25 for linux-riscv@lists.infradead.org; Sat, 15 Jul 2023 15:01:01 +0000 Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-666ecf9a0ceso2103084b3a.2 for ; Sat, 15 Jul 2023 08:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689433259; x=1692025259; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=zV+n68z8zwXiYMhTDIWC1NXHhRag9MR6zl1GpKdQwv0=; b=W9u1Rh7FIMBncTNbzBPs5uTGKq2Z8+gAFBxCTcEF7A9Asouw4/CMUjolJ5olWBDEqS 4Vs/Vg1so8bplsqYMxfGkwXtvPPjDYMu0e9fIuaS9CjzqsN8hvgSlD0WmXt7Td8XS2+p 7E17PKwF3fxgRIRb8cOBkSjzegqlVtieE/88INo1akPm6GxLqp0dnJYGbcPznJvOAo6F FSVCDM4HtX+AD5N4YGEG5CTP8YHHuEhcpv67HmJIRgCOlPhNC3Y/GTMfUQ+AMM8hQ8p1 1etqclN8yNV5CG0pSM5ZCoOuAXCA44MJBKcYZDjzEgMHSSmg/XxM8oNfIHmWY7VN8/hc n5/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689433259; x=1692025259; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zV+n68z8zwXiYMhTDIWC1NXHhRag9MR6zl1GpKdQwv0=; b=gvM7xHZBt1EbxAg9gHLwdTF4bqVhz1/DGwatXLF89Ryz0V8jGmzYFUfP3J+K6HTw0T Ub25HmBOsKVdwo65V8BF6b71ezK61p/2afBYnzN8PGwxsi61lC7SbYmJJX7/SMcrEPam QlGIg2pjy1AeM1D6RRor9nc4wbd13dUP1yH4+AArloCirWp2jXRhSxyfvIlb7sIygxOn 4P3kWkkAM7bsnj1COtECPBMt6l+Aq07vUnhs4eqPVnDQd+f1BjyIAciujbh2UMpwEon0 YAnNfmJGASbWsOEIew3otzcsU638o5JDDaqI98W5N+cd/l0LP7twAXkBIRaFNFMclWzW wNsg== X-Gm-Message-State: ABy/qLaNmHxwdHfR30PAhIuT7B4QNaOVw3ncaskVmkmxSHNxhgX5pEU0 OxlYud8MpVTZh7GU5TT7hj74w9FrA/5Tv2a+F/7ASJAEKxb6paev6DnhRLmktizW/zjn7qdCu78 B1RitonEFamAaqHJqkX79uGNFHgxKAuRFbVF7MyhF4Rsb+6FvESaVTCAEmwf854dBulPz1qOLAZ GsV2L8xBQ05SSb X-Google-Smtp-Source: APBJJlFGbgPSGyYp2zWN4S5JHSwJrlyR0E8ct/c2ZsjyFB0YUYArIyt6fJ/SYJe9ZI0705LRiimd0g== X-Received: by 2002:a05:6a20:430f:b0:134:3013:cdb5 with SMTP id h15-20020a056a20430f00b001343013cdb5mr2701315pzk.35.1689433258301; Sat, 15 Jul 2023 08:00:58 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id a28-20020a63705c000000b00528513c6bbcsm9356535pgn.28.2023.07.15.08.00.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jul 2023 08:00:57 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Subject: [v1, 2/6] riscv: Add support for kernel mode vector Date: Sat, 15 Jul 2023 15:00:28 +0000 Message-Id: <20230715150032.6917-3-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230715150032.6917-1-andy.chiu@sifive.com> References: <20230715150032.6917-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230715_080059_686991_50D3233E X-CRM114-Status: GOOD ( 18.89 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anup Patel , Conor Dooley , guoren@linux.alibaba.com, Alexandre Ghiti , Jisheng Zhang , Sia Jee Heng , Xianting Tian , anup@brainfault.org, Masahiro Yamada , atishp@atishpatra.org, vineetg@rivosinc.com, =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Vincent Chen , bjorn@kernel.org, Albert Ou , Guo Ren , Andy Chiu , paul.walmsley@sifive.com, greentime.hu@sifive.com, heiko.stuebner@vrull.eu MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 129 +++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index a4f3705fd144..9831b19153ae 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,8 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +int kernel_rvv_begin(void); +void kernel_rvv_end(void); static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index a42951911067..b954bbf17c84 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -57,6 +57,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..c0c152c501a5 --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include + +DECLARE_PER_CPU(bool, vector_context_busy); +DEFINE_PER_CPU(bool, vector_context_busy); + +/* + * may_use_vector - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_vector(void) +{ + /* + * vector_context_busy is only set while preemption is disabled, + * and is clear whenever preemption is enabled. Since + * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy + * cannot change under our feet -- if it's set we cannot be + * migrated, and if it's clear we cannot be migrated to a CPU + * where it is set. + */ + return !in_irq() && !irqs_disabled() && !in_nmi() && + !this_cpu_read(vector_context_busy); +} + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +static void get_cpu_vector_context(void) +{ + bool busy; + + preempt_disable(); + busy = __this_cpu_xchg(vector_context_busy, true); + + WARN_ON(busy); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +static void put_cpu_vector_context(void) +{ + bool busy = __this_cpu_xchg(vector_context_busy, false); + + WARN_ON(!busy); + preempt_enable(); +} + +/* + * kernel_rvv_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_vector() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_rvv_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_rvv_end() is + * called. + */ +int kernel_rvv_begin(void) +{ + if (!has_vector()) + return -EOPNOTSUPP; + + if (!may_use_vector()) + return -EPERM; + + /* Save vector state, if any */ + riscv_v_vstate_save(current, task_pt_regs(current)); + + /* Acquire kernel mode vector */ + get_cpu_vector_context(); + + /* Enable vector */ + riscv_v_enable(); + + return 0; +} +EXPORT_SYMBOL_GPL(kernel_rvv_begin); + +/* + * kernel_rvv_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_rvv_begin() was previously + * called, with no call to kernel_rvv_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_rvv_begin() is called again in the meantime. + */ +void kernel_rvv_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + /* Restore vector state, if any */ + riscv_v_vstate_set_restore(current, task_pt_regs(current)); + + /* disable vector */ + riscv_v_disable(); + + /* release kernel mode vector */ + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_rvv_end); From patchwork Sat Jul 15 15:00:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13314551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 288A7C001B0 for ; Sat, 15 Jul 2023 15:01:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yQmlCoNoNBMgiByf9QSeuKVkUbIFjUN/W9IQnWcNihc=; b=4HoX1oh+eMa0uk Q8BgEho8ibJsJ/LNIpjbHOZnywVnHsoAqa+LSOJrJivA6JA0MAwc2uhWCIQQ5qX9l/G4f6J2tWGlv 7bOIB0Dm3Fpp2o9f+4JJAtmT5S7wHwDRLsDt7US86k2jJIhokrubJ+4zxiZ7Caw3vNBbUPrlwrB1p J5+/M2M+B74GRtMphyiu5N1Sf3xGRFXRlgZVN9UNz8BBjPnwonovq1aSQ+mQg20EW4y5yC9E3f/GX 9dt2MS5VdkRenvKX3KyqNJPgxQtrrtM1P7SROs6JcQc/CEIPbFmyrBy5GcrsgP2GBYuoQnh89btRP aOl+QtHLyVv+Rdh5fDOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qKglj-008zdq-21; Sat, 15 Jul 2023 15:01:11 +0000 Received: from mail-pf1-x42a.google.com ([2607:f8b0:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qKglg-008zcq-0s for linux-riscv@lists.infradead.org; Sat, 15 Jul 2023 15:01:09 +0000 Received: by mail-pf1-x42a.google.com with SMTP id d2e1a72fcca58-666edfc50deso1814474b3a.0 for ; Sat, 15 Jul 2023 08:01:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689433266; x=1692025266; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=+km7VpAaR7aHNOC6jttnFLPGVHZYCDtVJYd/spo1/P8=; b=OBjNNQGEwB1iuhHQLflfF0ssRBwollQ0UOiHp2W50BNKJBBzwuteNEsR+hWc9+zxPl +oM6NNX+1yTeibVPtgC9OJRW2OrA+fcCBfcXHCfJimJgj7iVCKzsJr2g2roxdPZjhrVj 3EKQe9X8s/PabL3gH+E0J5Ql4rak/scCkYpOH6bj4jLxFKgOkPaf9g7oQOEF+Dm0Rnib WN/ZC4S/qbxGjoqgmCfPe7GDcGUHRkCNtw4OHA9nifNqmEL0NLYQ8QetWRaYrn89Q6ct l0koygYWxbHRCiIMzfMM7OXl6FMPvFOaKEa6vAJnEX+upySJ/JHnqH8WKW5oclgJyfdI cZLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689433266; x=1692025266; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+km7VpAaR7aHNOC6jttnFLPGVHZYCDtVJYd/spo1/P8=; b=aYjx/gI1B4Sw4M7O7dhyPu6UL2KnM21LJfxOoQU0irb2L30xeL1+FemeIwJ2dsTIvl ED9ODTzh61Pd77V3GekQRy3xFaxPngIE+FIGhdDD8Ssc/A4dl8yMtA03LBBeLkec1Zlz 0gpxyJYLgoJwAtyclltb29zD20qJiVcHZ78uYx0U7M1p5ZJNFgNBJ1ktxOynk8T8PIR/ b4tZsls6PX3WWFm+yC8XH5IpOansREFJlee+1zLmSq770lRpuQrhjtRtcRDKwBlwz7P9 47YXJ9UzQFbP8DtR7b3oWB9s/d0GK6unWh/X/Fo1Pdh9flVvLX05oWxfgT8L3vWm0Fcx 2cTQ== X-Gm-Message-State: ABy/qLadVOg1dUVLntS7wJ/NL5/FsZCkYINcADIJCaoymzdYUgZGJa10 +GSkgS/6jKzG0r5I2GFhCiVw7yN3bbKr/yA8TXCM4faS3C7eH8OAV2EQKlHmecS9WSTmWBmviX4 bOFN8XJQtXMpxPqSOYf/dSpBI9ECKZ4P+RxVP0hE0W3bAHOu2GzJ5X2LjM6JIfNv9p1Wzw0cFyT v/yL9T42/J3WlY X-Google-Smtp-Source: APBJJlEo6tfhpeWSveX0qSiMjipd2GDcZDYTR3KmhsxO4GyI20r0Q0stNft2lFfCkPwal8gDo2YpBw== X-Received: by 2002:a05:6a20:6a23:b0:134:135c:5a23 with SMTP id p35-20020a056a206a2300b00134135c5a23mr2878051pzk.18.1689433266374; Sat, 15 Jul 2023 08:01:06 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id a28-20020a63705c000000b00528513c6bbcsm9356535pgn.28.2023.07.15.08.01.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jul 2023 08:01:05 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Han-Kuan Chen , Andy Chiu , Albert Ou , Andrew Jones , Conor Dooley Subject: [v1, 3/6] riscv: Add vector extension XOR implementation Date: Sat, 15 Jul 2023 15:00:29 +0000 Message-Id: <20230715150032.6917-4-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230715150032.6917-1-andy.chiu@sifive.com> References: <20230715150032.6917-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230715_080108_318213_51D4E1DF X-CRM114-Status: GOOD ( 14.63 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..81b8837fa161 --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +static void xor_rvv_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_rvv_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_rvv_end(); +} + +static void xor_rvv_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_rvv_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_rvv_end(); +} + +static void xor_rvv_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_rvv_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_rvv_end(); +} + +static void xor_rvv_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_rvv_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_rvv_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_rvv_2, + .do_3 = xor_rvv_3, + .do_4 = xor_rvv_4, + .do_5 = xor_rvv_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..3bc059e18171 --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +ENTRY(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +ENTRY(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +ENTRY(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +ENTRY(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Sat Jul 15 15:00:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13314552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6CDD9C001DF for ; Sat, 15 Jul 2023 15:01:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dsv6LX2F4AfYHbt/f/EK8j9eQ5+ApzGQ2iHuXC8Cipc=; b=FeCdHRNKQeJibJ 9HqcG7ScUGC+y0sYXXqQr9NUicvgDTWo+BeXen3p9UClRPMxw60svKZyTrZL0B/NMGb5l5Pf5+9QW mCBrYsCkrzTQywAA5Z/ySQzmIg/g5pLEU+qr7Fb82v6rZobhfsGQGAipMQU00+YDk5d6uOSk+x0Xl R08vKRSLs+vTzBlo306klK4gdsC1eLMrvxn17F3ZWrRjjh2+lpcpsM584Wr9tbAXMk+U6LgEHdeB9 /vTcWK310OzQxHzYSPVqBB9n2lteORDFQK9AYUjm3tWLnsK0wE9i9tJgyM+cCb+pLr5M7qCp9TTuv 0CuOUDmZIR8QwHk8tu6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qKglr-008zfs-1D; Sat, 15 Jul 2023 15:01:19 +0000 Received: from mail-pj1-x1033.google.com ([2607:f8b0:4864:20::1033]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qKglo-008zf3-2r for linux-riscv@lists.infradead.org; Sat, 15 Jul 2023 15:01:18 +0000 Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-2633fe9b6c0so2718045a91.1 for ; Sat, 15 Jul 2023 08:01:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689433275; x=1692025275; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Hu6CPBVAfibHHMvDyHkRBdDxemljL2h5/cP72Uob0eE=; b=Cf69fvkbHrKpWPAXt02du4sv2bXva4sCsxQYWIY30+giQguTPlgfdMe85yKZ4nGUMz 9Me+f4Dw/j243mMpNMmlaS3bw9PD7Bsu6PqgbRfXK8zYpLEGPvrFlos55uf2Ujrb+Lf1 1BvGbbF9iOeZhgH+M6xJTYPuZtlS6f+3NVBY0zD/CZIl47nNriNFmuXPiUf/wRG+f9No Xj81SidDxewdaLqhQA+WFLJdZ25+tjEe+BJfQYMRHvB2Dwd6YEAdaUnNHMmf2JWsq6PO h5YnQIKimVTgmJP4nlvTT0F2ZrNRDCSUQ5OQKVK3reUCykr0j/TFby/78Hf3OSpNFaok /qVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689433275; x=1692025275; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hu6CPBVAfibHHMvDyHkRBdDxemljL2h5/cP72Uob0eE=; b=WftPKcm5eu9hbkMDNZRPKU3g0iCvOtCIBOsz4nFHzaJzkqVi4UrG/flivRhpJ8MFN+ ki9TX/cVyBDdCAIodGsUCOEyg58YAsapVSmcjqvrIWl5wxp8/mguIIq9y0AbsNMnSnWH IgBpq71+iwMfAAApxaqIaSs1dPljXzo1N9qWGzYF1S0Ek48IlSFP7cPUl1+2IcKngx+I ffyihxbFDZWpa4fyaljOdRyrydA3mqdLe/N4xOovSX+PLccJzRZ7UTUNbhcuFuPxXVbG 7AdT/3kthSO1K1gMdYAlDE6nf9MplIay33jncg9gve69lGlSjTUEFzkKJYcf0HVJSU5A ++tw== X-Gm-Message-State: ABy/qLaTJlyBNvRd7AeCqIRn03NMDnWOwteXV/l7CKIdSN9dJOX20QMP KTZCXfiGHUey5oKfnQMBsPh5hmCUtZLeXxqYjlY/dYslVmxD8I9FMMlKMskeGK6BGaQ7glVUfLT pOMZ7SLOI0YAem4eEAedvq1soZ/tdWYG8KKrp3kdjyYJR3XUCHWhtmXyKSyQQlsrFgMgpI01nZp JXQbI6UaEJjqyG X-Google-Smtp-Source: APBJJlGnJ0AiWaooxuA0jWOFbnR4NHLM8lyKrS0lMelG6TZpM5q4QlfQmZ6ryWrh37dWQ8QdxB6mtA== X-Received: by 2002:a17:90a:ea07:b0:262:d9b7:13ce with SMTP id w7-20020a17090aea0700b00262d9b713cemr7260494pjy.5.1689433275436; Sat, 15 Jul 2023 08:01:15 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id a28-20020a63705c000000b00528513c6bbcsm9356535pgn.28.2023.07.15.08.01.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jul 2023 08:01:14 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Andy Chiu , Albert Ou , Oleg Nesterov , Guo Ren , Yipeng Zou , Huacai Chen , Vincent Chen , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Mathis Salmen , Andrew Bresticker Subject: [v1, 4/6] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Sat, 15 Jul 2023 15:00:30 +0000 Message-Id: <20230715150032.6917-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230715150032.6917-1-andy.chiu@sifive.com> References: <20230715150032.6917-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230715_080116_926837_C3CD770B X-CRM114-Status: GOOD ( 11.67 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 52926f4d8d7c..aa1b9e50d6c8 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, { if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 9831b19153ae..50c556afd95a 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -163,23 +163,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -200,7 +196,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -218,8 +214,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index c0c152c501a5..30f1b861cac0 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -91,7 +91,7 @@ int kernel_rvv_begin(void) return -EPERM; /* Save vector state, if any */ - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); /* Acquire kernel mode vector */ get_cpu_vector_context(); diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 1d572cf3140f..85e7167245cc 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,7 +99,7 @@ static int riscv_vr_get(struct task_struct *target, * copying them to membuf. */ if (target == current) - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); /* Copy vector header from vstate. */ membuf_write(&to, vstate, offsetof(struct __riscv_v_ext_state, datap)); diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 0fca2c128b5f..75fd8cc05e10 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); From patchwork Sat Jul 15 15:00:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13314553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 38F62C001B0 for ; Sat, 15 Jul 2023 15:01:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=nJu5OptHTNE0yG/yiph5BhB+sFWOiICAvkVQj1Fp9zQ=; b=KRkrcSF4k72konDI99BNAHZIYs +47UP2a6B4Qwx+msnGQ1fVhWLwddq3vLjcvw0g2V/EZKz006hNvx04MYO92sN6s/wIp3H+xlBQop1 5TrrU7lCd9B0pEja43Lt3FdLMqBQLrgW6iC47Yp2pAEyzE32AuwvZcaH7zflhne15JjfB0beTkh49 03x8PGHifnpipKeIj8zPozng6aojigSOU5CU5mt6CRtIVZKP1tqDVwY4tHN4vMsCVc8BSi1CPnsnn OPxkrAqcN7I4we/s4Fds3aN0H01jamVKlUUNe7jW+7u6+ikRjY4+fdwwl8t7U7YT09R9AGyiUFCfL YXtzEn1g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qKgm5-008zkc-1k; Sat, 15 Jul 2023 15:01:33 +0000 Received: from mail-pj1-x1035.google.com ([2607:f8b0:4864:20::1035]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qKgm2-008zio-0B for linux-riscv@lists.infradead.org; Sat, 15 Jul 2023 15:01:32 +0000 Received: by mail-pj1-x1035.google.com with SMTP id 98e67ed59e1d1-2633fe9b6c0so2718171a91.1 for ; Sat, 15 Jul 2023 08:01:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689433288; x=1692025288; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=PzuRUXLhg+WKW9eqLFnFGYE2/PT3yhwmi5DQKcTLxkY=; b=Sg4KGADrYWXxWXw7+hSFWnup1ebY2c9nkgwi//vZ7v4UmFuayYlE0iuiIC1AsRUAWC EkFtZZ5tIM80uZRdfPrc77sus8ieaLnMFluJ9qIc55115+/lqRGWJ3I6uTPxue0WJlkJ ++Y8+I2+ubdGkPNtZnGDg1B/d/llW/QJ29Ic2wo84R9EW8EhCBc452t21rsGwzxc97ht w39UPgn9u6/ZVZwHVtE0tWvrcBCYvgEkfZQZAwqXzK2kfG0zusQsT4jDKPwM7n9FzE0H rHmnXzuvCvz2uIm3MWmxgddnQ4CuWuks73LKQAY+j6sf8sw4bjHtsp7TQMhh4JhSIvd8 4h8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689433288; x=1692025288; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PzuRUXLhg+WKW9eqLFnFGYE2/PT3yhwmi5DQKcTLxkY=; b=fW10C38lQTzySJu7IlylyajmZTqNtltXNr0CnWbL377JDsryWhultFgQBa1s3Pb05N uFYACMl95ObJt0SP5Ul7EhMbwO7DazEcblDByDhmUIPEBEWI7GoTOOG+WRgxgOn1OfgD VKww86WC8NwoN4TTGGJO1d7+Sb9jNJw2RXZZMJ7eO5M3c+ehxLPlGeg9Smz8gnYXslhe NZDbRyEMWdiQmPpF5Jczncpl6kVQYM2kXdQMLmV/W4Tp/jnitcDkmd7zcwdbibiw1xTf MIGK9T+iEtZtFu5OUxcm9W0EnS8leLCjamKoq+0UN/x4jVNOhQwBgvnvNdZImtaMu8oU YbMg== X-Gm-Message-State: ABy/qLbjcBHR2MJKLeStReIOvXuMB6QFqMreVhOhV5pb5FFNH7cC2W/J Pk0YdMBBobSfAJ10GRmAyD8BWiMVA3K5TlOUUXaOq9W3LQ5oarH85bSKJwnlorm4xRqguWYLcDy ce3Bq3VNma4XWukzA4+zZdr5peS7lEfHgUbCaHnGwL3HP35tDd4YC+6Vq2QaxOTTN+KgwC894+N Z4r16pfa25UmPP X-Google-Smtp-Source: APBJJlHzIpMq5gK5hcGUuObsWWRjSEs83a3oP925DRaZ6Eg9cccOeZuwgPEXiI8+/EsZvNVTnEasNw== X-Received: by 2002:a17:90a:5d07:b0:263:f73d:9f50 with SMTP id s7-20020a17090a5d0700b00263f73d9f50mr7342682pji.19.1689433288098; Sat, 15 Jul 2023 08:01:28 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id a28-20020a63705c000000b00528513c6bbcsm9356535pgn.28.2023.07.15.08.01.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jul 2023 08:01:27 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Subject: [v1, 5/6] riscv: vector: allow kernel-mode Vector with preemption Date: Sat, 15 Jul 2023 15:00:31 +0000 Message-Id: <20230715150032.6917-6-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230715150032.6917-1-andy.chiu@sifive.com> References: <20230715150032.6917-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230715_080130_093087_8F677AF3 X-CRM114-Status: GOOD ( 23.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kefeng Wang , guoren@linux.alibaba.com, Peter Zijlstra , Andrew Bresticker , paul.walmsley@sifive.com, =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Guo Ren , Jisheng Zhang , Fangrui Song , Vincent Chen , Sia Jee Heng , anup@brainfault.org, greentime.hu@sifive.com, Albert Ou , Ley Foon Tan , vineetg@rivosinc.com, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Nick Knight , bjorn@kernel.org, Andy Chiu MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KMV indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set. Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code. Signed-off-by: Andy Chiu --- arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 ++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 41 ++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 57 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 8 files changed, 136 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index e82af1097e26..d337b750f2ec 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -42,6 +42,8 @@ struct thread_struct { unsigned long bad_cause; unsigned long vstate_ctrl; struct __riscv_v_ext_state vstate; + struct pt_regs *trap_pt_regs; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index d83975efe866..59d88adfc4de 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -102,6 +102,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12 +#define TIF_RISCV_V_KMV 13 #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) @@ -109,9 +110,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KMV (1 << TIF_RISCV_V_KMV_TASK) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE 0x20 + #endif /* _ASM_RISCV_THREAD_INFO_H */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 50c556afd95a..d004c9fa6a57 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -25,6 +25,12 @@ bool riscv_v_first_use_handler(struct pt_regs *regs); int kernel_rvv_begin(void); void kernel_rvv_end(void); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv(bool preemptive_kmv); +#else +#define riscv_v_vstate_ctrl_config_kmv(preemptive_kmv) do {} while (0) +#endif + static __always_inline bool has_vector(void) { return riscv_has_extension_unlikely(RISCV_ISA_EXT_v); @@ -195,9 +201,24 @@ static inline void __switch_to_vector(struct task_struct *prev, { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV) && + test_tsk_thread_flag(prev, TIF_RISCV_V_KMV)) { + regs = prev->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_save(&prev->thread.kernel_vstate, regs); + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV) && + test_tsk_thread_flag(next, TIF_RISCV_V_KMV)) { + regs = next->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_restore(&next->thread.kernel_vstate, regs); + } else { + riscv_v_vstate_set_restore(next, task_pt_regs(next)); + } } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..4b062f7741b2 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -38,6 +38,8 @@ void asm_offsets(void) OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_THREAD_TRAP_REGP, task_struct, thread.trap_pt_regs); + OFFSET(TASK_THREAD_VSTATE_CTRL, task_struct, thread.vstate_ctrl); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..42b80b90626a 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -66,6 +66,27 @@ _save_context: REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp) + /* + * Reocrd the register set at the frame where in-kernel V registers are + * last alive. + */ + REG_L s0, TASK_TI_FLAGS(tp) + li s1, 1 << TIF_RISCV_V_KMV + and s0, s0, s1 + beqz s0, 1f + li s0, TASK_THREAD_TRAP_REGP + add s0, s0, tp + REG_L s1, (s0) + bnez s1, 1f + REG_S sp, (s0) + li s0, TASK_THREAD_VSTATE_CTRL + add s0, s0, tp + REG_L s1, (s0) + li s2, ~RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE + and s1, s1, s2 + REG_S s1, (s0) +1: + /* * Set the scratch register to 0, so that if a recursive exception * occurs, the exception vector knows it came from the kernel @@ -129,6 +150,26 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: + /* + * Clear tracking of the trap registers when we return to the frame + * that uses kernel mode Vector. + */ + REG_L s0, TASK_TI_FLAGS(tp) + li s1, 1 << TIF_RISCV_V_KMV + and s0, s0, s1 + beqz s0, 1f + li s0, TASK_THREAD_TRAP_REGP + add s0, s0, tp + REG_L s1, (s0) + bne s1, sp, 1f + REG_S x0, (s0) + li s0, TASK_THREAD_VSTATE_CTRL + add s0, s0, tp + REG_L s1, (s0) + ori s1, s1, RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE + REG_S s1, (s0) +1: + REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 30f1b861cac0..bcd6a69a5266 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -35,7 +36,8 @@ static __must_check inline bool may_use_vector(void) * where it is set. */ return !in_irq() && !irqs_disabled() && !in_nmi() && - !this_cpu_read(vector_context_busy); + !this_cpu_read(vector_context_busy) && + !test_thread_flag(TIF_RISCV_V_KMV); } /* @@ -69,6 +71,47 @@ static void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv(bool preemptive_kmv) +{ + if (preemptive_kmv) + current->thread.vstate_ctrl |= RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE; + else + current->thread.vstate_ctrl &= ~RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE; +} + +static bool riscv_v_kmv_preempitble(void) +{ + return !!(current->thread.vstate_ctrl & RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE); +} + +static int riscv_v_start_kernel_context(void) +{ + struct __riscv_v_ext_state *vstate; + + vstate = ¤t->thread.kernel_vstate; + if (!vstate->datap) { + vstate->datap = kmalloc(riscv_v_vsize, GFP_KERNEL); + if (!vstate->datap) + return -ENOMEM; + } + + current->thread.trap_pt_regs = NULL; + WARN_ON(test_and_set_thread_flag(TIF_RISCV_V_KMV)); + return 0; +} + +static void riscv_v_stop_kernel_context(void) +{ + WARN_ON(!test_and_clear_thread_flag(TIF_RISCV_V_KMV)); + current->thread.trap_pt_regs = NULL; +} +#else +#define riscv_v_kmv_preempitble() (false) +#define riscv_v_start_kernel_context() (0) +#define riscv_v_stop_kernel_context() do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV */ + /* * kernel_rvv_begin(): obtain the CPU vector registers for use by the calling * context @@ -94,7 +137,12 @@ int kernel_rvv_begin(void) riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); /* Acquire kernel mode vector */ - get_cpu_vector_context(); + if (!preemptible() || !riscv_v_kmv_preempitble()) { + get_cpu_vector_context(); + } else { + if (riscv_v_start_kernel_context()) + get_cpu_vector_context(); + } /* Enable vector */ riscv_v_enable(); @@ -124,6 +172,9 @@ void kernel_rvv_end(void) riscv_v_disable(); /* release kernel mode vector */ - put_cpu_vector_context(); + if (!test_thread_flag(TIF_RISCV_V_KMV)) + put_cpu_vector_context(); + else + riscv_v_stop_kernel_context(); } EXPORT_SYMBOL_GPL(kernel_rvv_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index ec89e7edb6fd..4db8cbc8abe9 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -160,8 +160,11 @@ void flush_thread(void) void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ - if (has_vector()) + if (has_vector()) { kfree(tsk->thread.vstate.datap); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV)) + kfree(tsk->thread.kernel_vstate.datap); + } } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -170,7 +173,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); + clear_tsk_thread_flag(dst, TIF_RISCV_V_KMV); return 0; } @@ -205,6 +210,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + riscv_v_vstate_ctrl_config_kmv(true); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 9d583b760db4..42f227077ee5 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void) From patchwork Sat Jul 15 15:00:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13314554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D1EBC001DF for ; Sat, 15 Jul 2023 15:01:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=i4MuWmT+8D013mtYXNbdPw/e/Mhk3vyViCepRYzJqls=; b=0lQGx8UvdkdY4e H/ptdWN0UbH7UXkzBDDEsMnLazxk/lBAkU8HDjfTE075bjJhhi7QF9tFu5iIJfyHeeYLc/deKYKDa qRCOhJncSlebQWlhZmOqq3Tk6ORmURSA9J1KO8JEf38IeqMOpP4/rK1b/B7ggiNg9VFNpyfbgSIBD Dl0saKKC/fYN7qZ+xWLG5K2mgBsV3xPyFYyovxCRB+o+6J8hAlCnp4Q3Lk7LqpnHb7cgzJD7+RQ03 bniV2miKHuiyNY2uK9pmvui6CyoTeQucmHnNPc+LpJKm+0BZNPxpc/JOoXMb3ZaZdYYmZ9LRa0X8+ 46e+oUdkvsA5DH86zhKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qKgm8-008zlf-0b; Sat, 15 Jul 2023 15:01:36 +0000 Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qKgm4-008zk1-34 for linux-riscv@lists.infradead.org; Sat, 15 Jul 2023 15:01:34 +0000 Received: by mail-pj1-x1031.google.com with SMTP id 98e67ed59e1d1-2632336f75fso1991586a91.3 for ; Sat, 15 Jul 2023 08:01:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1689433291; x=1692025291; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=2UCzvviqLaht1enidoVBHlz8q/RmZyOnwjkKU4rZlbI=; b=BsP/kZmUBC6ajblBI5TjQlGD5OKHfRM3e5RwrlxKX4qW4HY4F9+RDL7TJP+E/ZpoRw MMb1kJm8le+kbvyTXGgdDH2/NT2lrBZ/RUNr7CgkxHSJTOMQKdmDMHFcuy8GON3nC4Ry otYdeVbZLAvXm5/1z8qUSfhwnC38dJOWP8zFdOSVLnelRQYFJy+bgld1nU5NJ4dYkYwL V1K6c6culGNDsCecIeuHvyW8IXewBY9Bj4bjeU9o+BnHJaSkWXmtWJbMt8+MuVpImios zKjjGLdYsD3x/v4FV3UhWFhHTLaTIHFNTW50pPcJZwY/8rD6zvr5PjIer+X1ILBgBLE3 8NVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689433291; x=1692025291; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2UCzvviqLaht1enidoVBHlz8q/RmZyOnwjkKU4rZlbI=; b=Z0fDj7F19aVCmZYB0YrZ7X96fL2C25RJew2lwysRC9r0CL6dnz10zmyQDf6BVq6U9M /6S8MYEKJHrqgcpKs+rU31eu6Kb+l3VpDCNJn1he8ylvA+4N7aH1DtQ+ZQYjagvNYZAd 21RyY+q2/r7sLTsd6smkJHNKUkDN/jO1TzeuHafbATe6MGsfwtjLif9XMck7oW1qK7wB 84f4CeeglJnnSpn5YjU7/RGxxKU3QYNKzI3PgtO851k7yAKmmAFkT7OyA/3NWA6w76YZ KVmf3+vHW/5xhvGPQ2BFlyGs8fLHnOySTlQ95hhmUcNoMl9ZSsEjAkNH4dPjAfUt2V2c v6NA== X-Gm-Message-State: ABy/qLYg4WFtjgzMVRsmYvnxjAhuPMOVbFIqihhDHZYdimQW8MhFfAjk 8hGvqlo1SFA7vjPFlFcEY0B3vBcIIpgv1oNfLdCSw6y0ifpV+bu06P6Jct2HNMnnl0k7Jjdmfap xGLvtMXZI8iXRewYdjUS5fEiF0BCSgqWpaG6BEkEI9CIUbl///OH43J0T5eIqHv4Vpc7ui1/XZQ wBF4Cjy+YHRD4c X-Google-Smtp-Source: APBJJlFYPKocV2zT5SEHFTeQcJcJM0iyiUO6rDqMYENRz0C1B0Dd1bZNm+TpjYTSv7u413kjLaCHCA== X-Received: by 2002:a17:90a:13c6:b0:262:cef9:84f6 with SMTP id s6-20020a17090a13c600b00262cef984f6mr6575423pjf.22.1689433291445; Sat, 15 Jul 2023 08:01:31 -0700 (PDT) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id a28-20020a63705c000000b00528513c6bbcsm9356535pgn.28.2023.07.15.08.01.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 15 Jul 2023 08:01:30 -0700 (PDT) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: vineetg@rivosinc.com, bjorn@kernel.org, greentime.hu@sifive.com, paul.walmsley@sifive.com, guoren@linux.alibaba.com, anup@brainfault.org, atishp@atishpatra.org, heiko.stuebner@vrull.eu, Andy Chiu , Albert Ou Subject: [v1, 6/6] riscv: vector: enable preemptive kernel-mode Vector to be built Date: Sat, 15 Jul 2023 15:00:32 +0000 Message-Id: <20230715150032.6917-7-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230715150032.6917-1-andy.chiu@sifive.com> References: <20230715150032.6917-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230715_080132_988747_601889BF X-CRM114-Status: UNSURE ( 9.82 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add a Kconfig to let user decides whether kernel-mode Vector in a preemptive kernel should also run with preemption. If the config is 'N', then all kernel-mode Vector code are run with preemption disabled. Signed-off-by: Andy Chiu --- arch/riscv/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index a9e8b697fefb..da6a45ea42ec 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -500,6 +500,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_PREEMPTIVE_KMV + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Ordinarily the kernel disables preemption before running in-kernel + Vector code. This config frees the kernel from disabling preemption + by adding meory on demand for tracking kernel's V-context. + config TOOLCHAIN_HAS_ZBB bool default y