From patchwork Wed Dec 20 07:54:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6F3E9C3DA6E for ; Wed, 20 Dec 2023 07:57:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AeAITSF4oTNhUs/jUf3583Rfsl+spN8G/pIxoMyGDGE=; b=t0F442pBW8NptK 3vj+s3XNWURIRJPJdSkzyddbcAXUbIl37EvK5JD9YMFmL9UPKXDCMxlmWZVkpktPWUbApDkLw6MGr O+gw/TQKOpOBY02NFw6VL0l/KPgWcEpYjsT0CLwR4FQc3WfeC1Ekotu/aunH3jGVI5ljetffdcDRd yVMVVlfZeQ/zXiCqjGxUiPe702qsfQPTdenB1ra4UChdbX8pwc8xGmwIoSwfau6tWcAJhCE8xiGQF uyP/wC9YbeUfIWEdqLIJdstmfnSm9rSC4btyYFs5NbNKI+MCMXVeMwQqoiumHY4mtxzUAdhy+pqvt GIjCyataqv3lrGv9sF0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrS9-00GUJI-0v; Wed, 20 Dec 2023 07:57:17 +0000 Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrS6-00GUIQ-1Y for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:57:16 +0000 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-5cdbc7bebecso627097a12.1 for ; Tue, 19 Dec 2023 23:57:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059033; x=1703663833; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AC7hK0pJhSYV6xaA5paIpr1GMZMUIJvUjm2sS08sGdA=; b=NGmCOw1dIC/EqYUB9WPz329pQaCs+8U5zK4tdNFnfbHEsY9duSvHPClgavgTz58xa9 1jEBTL8M3tPGBqGNjLhc/Na9HKxpS1Eeiaqz3bVzXe+X9373k6oat4OIPJa2LGfzzu1Z QfNGROM6cJnOXcv3r8neKTrCQAoTLxubvVvdDxYF2ZuwmelEpxs7/eH22ayt+dKAzVvH 5vHl7JZJUgHHDjxbJEqcfuwdoNV7aiRDa30yLFAsNy2Q5pn5sxtqbQCjSuNOogBBUj7p B6YLBbqBuRODeIvLstz2riztqL7uzyA6PR229owR2X9JmCXhYBxboZvnElnh8N6yKOAS 9T+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059033; x=1703663833; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AC7hK0pJhSYV6xaA5paIpr1GMZMUIJvUjm2sS08sGdA=; b=IKV+JpAx4kdZKdM2gy6CWdIebEMZ0eIWz4AJZzS2/LG4g6pACLcg/UhN36gv13C3fZ s8RNR8WqNLeGgoUYMHHnNOKFPR/+CFksP7HJuAWWA8Flcn130d58YNvJRbc8KQQOvEGm KGVz8RBtq9Z+wO6hmKFtJFiV9qsJD+a1kVvyOgBBT9Wqta2sdDh+rQdBdWqJEX++0oU5 HMuSjgaCxQfDfd6o/UImpHcYvFaRRmW9UqqWfca+F36qvmade9zUA/x/+A4TqNhNajDQ rHGxhaNVGUG2NMaq0Sxbsbim5eMT4JTAIOap/8StxRoMRZhcYPDmwrWQj5R1MyJ+2OA4 5USQ== X-Gm-Message-State: AOJu0YwlcsCbyWe4ITUiOvdDY2msCg/3RsOxmnILTN6zPJ7UGuw9n9b6 NToZ2MnnrjG+G9LCoR46q9nm4TC84v5mCUlg5wtONpsOk8O25Cj4RWS11jBNbOOxith0IXz89KE OOA59FUEvQNaAadIN4h62rTQZqTX4xDLuEYPJjlYobHUMJqVmpKwweNHRd5MA/Q3Pq4yjXTf7Tj SLJ0tq/rQ0CY7s X-Google-Smtp-Source: AGHT+IEtrAgz+HSRaIV+JxWu/uYfZ3TNUehwuLYXmNAZWwnt1PYIKv0GlwgQ1nkFvKZm0BCoM7h+bw== X-Received: by 2002:a05:6a20:1012:b0:18a:b5c3:55db with SMTP id gs18-20020a056a20101200b0018ab5c355dbmr8261917pzc.50.1703059032577; Tue, 19 Dec 2023 23:57:12 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:57:10 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Vincent Chen , Andy Chiu , Albert Ou , Heiko Stuebner , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Xiao Wang , Conor Dooley , Alexandre Ghiti , Evan Green , Anup Patel , Sia Jee Heng , Sami Tolvanen , Jisheng Zhang Subject: [v6, 01/10] riscv: Add support for kernel mode vector Date: Wed, 20 Dec 2023 07:54:03 +0000 Message-Id: <20231220075412.24084-2-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235714_521266_7CE76993 X-CRM114-Status: GOOD ( 25.82 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- Changelog v6: - Use 8 bits to track non-preemptible vector context to provide better WARN coverage. Changelog v4: - Use kernel_v_flags and helpers to track vector context. Changelog v3: - Reorder patch 1 to patch 3 to make use of {get,put}_cpu_vector_context later. - Export {get,put}_cpu_vector_context. - Save V context after disabling preemption. (Guo) - Fix a build fail. (Conor) - Remove irqs_disabled() check as it is not needed, fix styling. (Björn) Changelog v2: - 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor) - export may_use_simd to include/asm/simd.h --- arch/riscv/include/asm/processor.h | 17 ++++- arch/riscv/include/asm/simd.h | 42 ++++++++++++ arch/riscv/include/asm/vector.h | 21 ++++++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 95 ++++++++++++++++++++++++++ arch/riscv/kernel/process.c | 2 +- 6 files changed, 176 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index f19f861cda54..15781e2232e0 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -73,6 +73,20 @@ struct task_struct; struct pt_regs; +/* + * We use a flag to track in-kernel Vector context. Currently the flag has the + * following meaning: + * + * - bit 0-7 indicates whether the in-kernel Vector context is active. The + * activation of this state disables the preemption. On a non-RT kernel, it + * also disable bh. Currently only 0 and 1 are valid value for this field. + * Other values are reserved for future uses. + */ + +#define RISCV_KERNEL_MODE_V_MASK 0xff + +#define RISCV_KERNEL_MODE_V 0x1 + /* CPU-specific state of a task */ struct thread_struct { /* Callee-saved registers */ @@ -81,7 +95,8 @@ struct thread_struct { unsigned long s[12]; /* s[0]: frame pointer */ struct __riscv_d_ext_state fstate; unsigned long bad_cause; - unsigned long vstate_ctrl; + u32 riscv_v_flags; + u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; }; diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h new file mode 100644 index 000000000000..269752bfa2cc --- /dev/null +++ b/arch/riscv/include/asm/simd.h @@ -0,0 +1,42 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2023 SiFive + */ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include +#include +#include +#include +#include + +#ifdef CONFIG_RISCV_ISA_V +/* + * may_use_simd - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_simd(void) +{ + /* + * RISCV_KERNEL_MODE_V is only set while preemption is disabled, + * and is clear whenever preemption is enabled. + */ + return !in_hardirq() && !in_nmi() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd(void) +{ + return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 87aaef656257..6254830c0668 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,27 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +void kernel_vector_begin(void); +void kernel_vector_end(void); +void get_cpu_vector_context(void); +void put_cpu_vector_context(void); + +static inline void riscv_v_ctx_cnt_add(u32 offset) +{ + current->thread.riscv_v_flags += offset; + barrier(); +} + +static inline void riscv_v_ctx_cnt_sub(u32 offset) +{ + barrier(); + current->thread.riscv_v_flags -= offset; +} + +static inline u32 riscv_v_ctx_cnt(void) +{ + return READ_ONCE(current->thread.riscv_v_flags); +} static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index fee22a3d1b53..8c58595696b3 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..105147c7d2da --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include +#include + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +void get_cpu_vector_context(void) +{ + preempt_disable(); + + WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != 0); + riscv_v_ctx_cnt_add(RISCV_KERNEL_MODE_V); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +void put_cpu_vector_context(void) +{ + WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != RISCV_KERNEL_MODE_V); + riscv_v_ctx_cnt_sub(RISCV_KERNEL_MODE_V); + + preempt_enable(); +} + +/* + * kernel_vector_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_simd() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_vector_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_vector_end() is + * called. + */ +void kernel_vector_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + BUG_ON(!may_use_simd()); + + get_cpu_vector_context(); + + riscv_v_vstate_save(current, task_pt_regs(current)); + + riscv_v_enable(); +} +EXPORT_SYMBOL_GPL(kernel_vector_begin); + +/* + * kernel_vector_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_vector_begin() was previously + * called, with no call to kernel_vector_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_vector_begin() is called again in the meantime. + */ +void kernel_vector_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + riscv_v_vstate_restore(current, task_pt_regs(current)); + + riscv_v_disable(); + + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4f21d970a129..5c4dcf518684 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -187,7 +187,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); - return 0; } @@ -221,6 +220,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + p->thread.riscv_v_flags = 0; p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; From patchwork Wed Dec 20 07:54:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499632 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5646C3DA6E for ; Wed, 20 Dec 2023 07:57:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=0z7jSwwfgDAJGFpiUlr+5uUQp6Fi3T2GULt8vTY8gs8=; b=ZotBDphgvxvH0x hRnPpuBfsMhn3r1nANugZI50AYCWeWwsc1CMd5qgKPFqQnqNmM7PKAwOYGIIdfH6wWBboBbCgFyeI 5bPOtPSQaMNCLmgnOECRHyf4sZkVGiJuVnKCm1KRAo52iky/7Z2hBN+x3K3+cU0tygKDUZcZ1ufmi PxyFoEAL+aUIgOF6JU9kEeyaEFUixNv7MZ6mLuLdnbdCdZ7YEfXivxdrwelfKCTmP0BsQsXdocN4B hOgYik7yVZvDRghtDMBWy/eIvf2lSKZiHxgJzg/rr02R71AwbsWqyK2Qb3EIMo9n5Mxi2g58htkNX 72pdibWhkLDaTlTztNjQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSI-00GUM4-0V; Wed, 20 Dec 2023 07:57:26 +0000 Received: from mail-pg1-x533.google.com ([2607:f8b0:4864:20::533]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSE-00GUKe-28 for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:57:24 +0000 Received: by mail-pg1-x533.google.com with SMTP id 41be03b00d2f7-5cdbc4334edso660230a12.3 for ; Tue, 19 Dec 2023 23:57:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059041; x=1703663841; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=V1lsxw5x/Z60q61VBpRIFECLxPvRWOMvCnylmkFtswE=; b=IpKJJYMQVSl5Ql35wNZWTXvEOXVCicLcRyBcn1I7kWzu4Dg+GWa7IfmiQ0N4uTP5ka MzKBskvJ37yh63NqQBoRAKWKeaLCUqAkPm2IHy6Lq4JA+za2Ozwt2My2c8ugK/fvpmL6 JiiZImpgRVV/JBdcasInVM3CjgrYMiZtbPyCHwkh5INmockbK8J7tfG03Lmqt8D5YF5j alxu3svI+t1/+p3tn//OqNKF5TvS51d2i7x24jOreyasJgg2xOcOwJNrvu8ZzW42copJ s89zSSJMAVGXUCdShaHx2L9ajyr9VGUrajKLisOqzvwwqGnHqcmzjMrBFntkLYA3+Wbx xRsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059041; x=1703663841; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=V1lsxw5x/Z60q61VBpRIFECLxPvRWOMvCnylmkFtswE=; b=gqxTwKf/9JbyQsUM9ylTtNn9LKf7ok9ZhgqBAPG/S+pgM050fq8EQNUuzA6DUppSYr 62+xYdHOgFm+WAbia3nGoOX75G3Rf8cMI5XsOj18msb1V3vi7+R4R1Eg9/+szBAX3LDt maHsTAabWoiH+UIIrQtDGWKycEG5k5zKMtCzZ3tn+FvEmR5CuHrMYRLIgBVl1jerihqA IUwhH6PlnfHWFCBM2OkP9WTxs+2DmCQz8VJyKxTWDt2KP9MZTyyazMuoRe2QrHd3gwHP gGu6bhWVO9QWZLDdJILSzBpYTezOBeVvtyF916EavJf755jQYiZzTkAtfUDFeB7wvcoG BETg== X-Gm-Message-State: AOJu0YyLUlWNr0ZdEmb7AUJAJ1BNx1F+k8iesruDVijIpkwSFys3Vl3O OBR1SJp9leDBGajX2ZZfWOBjiF/K4sC7j348vWw7mzd97pE6qyR/nPZGw8NWimwGCk3k229Uv6h 0+NjkcA60s5I5EBKV02/w4/27qm+qD6wPc5GPyHg1UEJ1dgEBfiPLSil6yDUNMIhq4d6sgBwdyM s1sDsP7Nvvg5DQ X-Google-Smtp-Source: AGHT+IEc+Rrp/P611JUnV/wfs5lD5XhiOm1v6EyQkUulvMtVU3ExoWKS+72g2+zSK73sQUHWonGDsQ== X-Received: by 2002:a05:6a20:3ca6:b0:18a:b5c3:55c1 with SMTP id b38-20020a056a203ca600b0018ab5c355c1mr11796784pzj.57.1703059040784; Tue, 19 Dec 2023 23:57:20 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.57.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:57:18 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Conor Dooley Subject: [v6, 02/10] riscv: vector: make Vector always available for softirq context Date: Wed, 20 Dec 2023 07:54:04 +0000 Message-Id: <20231220075412.24084-3-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235722_700308_BF2A2E3E X-CRM114-Status: GOOD ( 12.41 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org By disabling bottom halves in active kerne-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu --- Changelog v4: - new patch since v4 --- arch/riscv/include/asm/simd.h | 6 +++++- arch/riscv/kernel/kernel_mode_vector.c | 10 ++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index 269752bfa2cc..cd6180fe37c0 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -26,8 +26,12 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. + * + * Kernel-mode Vector temperarily disables bh. So we must not return + * true on irq_disabled(). Otherwise we would fail the lockdep check + * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); + return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 105147c7d2da..db0cf06f2abf 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -23,7 +23,10 @@ */ void get_cpu_vector_context(void) { - preempt_disable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_disable(); + else + preempt_disable(); WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != 0); riscv_v_ctx_cnt_add(RISCV_KERNEL_MODE_V); @@ -41,7 +44,10 @@ void put_cpu_vector_context(void) WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != RISCV_KERNEL_MODE_V); riscv_v_ctx_cnt_sub(RISCV_KERNEL_MODE_V); - preempt_enable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_enable(); + else + preempt_enable(); } /* From patchwork Wed Dec 20 07:54:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 170A8C3DA6E for ; Wed, 20 Dec 2023 07:57:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=EJSSAYNPmACBCn2sE4TEEv371QLzeS3Q/r8lXWNNqMs=; b=p5RArrqnk4ExAy UyWHpfWHnlOdyky5ICLIhHq+KnA97qnToSaOAiFpjXTdUdwA1FLp6oMNdpKBb2hE9p0zyVw6uzlhL hpBZVQr1ccg9QOFz62ZxyOTSeoK/f0ApGanpuZjsWRWvW2S9MkI17aXl843a//nNerloFjuE3zgnG zvNc9BFP2yAMC63aQvOoW1fRbA06nClQD60rrqycZYIykxHv0xtMdqFu/roKdu6H+SLv7KKBoGMVt 4lohtF+Pn4SDwqL47Kh6Fl2G/ZaJnH+dI1KYikGE0al5v3O48M6MqQj3tosHXA6iR5N00SKu6PQoz 6GxGFA/HJfzOCTmt69eA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSV-00GUSU-0U; Wed, 20 Dec 2023 07:57:39 +0000 Received: from mail-pf1-x436.google.com ([2607:f8b0:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSR-00GUPM-13 for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:57:36 +0000 Received: by mail-pf1-x436.google.com with SMTP id d2e1a72fcca58-6d267461249so2997784b3a.3 for ; Tue, 19 Dec 2023 23:57:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059051; x=1703663851; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=i9NRPfuDspExspcewG3zKAebEn0P1bg8Tb7NM+tntho=; b=emy6w6iwn5zRb+VJ38panrSORvBI5zwSgVW3uAIwmUasdLanFKeNoChmpq+CNDZ/+o NagzxIleirC5rqUzGxNnJidJ6lYBQqBY0GL3MgougBvDQJyiORBLVh21+kK51sfUYeQE Lb3Sg9gm4MGn31jt5PhoLNJQAyjLKUpA82/9b+b5FRwPi9hNCcwY+N2nXOwZfQFGuZV7 ZERWyOep+K2K1G1vH2vPg+PTAll5Xp1wk8OvIt7KwTBoWRRPYRmcZDBbXXlPS6zSP/4J ft+ebgIGD9OXrM5CVJd1kRm0LPhhLb67zOkZJLC2AWr3qUxNqJcXYbkQSGbvcdOUvBIv zfhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059051; x=1703663851; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=i9NRPfuDspExspcewG3zKAebEn0P1bg8Tb7NM+tntho=; b=n5EGv3zcN6TY/CQR+coTS9AYGCQmbikkUD1tkdqLaaFvUqlCnnnXf0ZyDpe3xx0GNG 8ZNJv//u5D0vLtW90mkuCK460WoG3ko+51EcRZhdBkO+FcPmKF/6latk03qCkdCsmZZr DwcNxvEH22SwoAaX3z+gv5Qnzd+zx7Rij40ZPkGfXwh6pcgHdsGZnu/byW/+ffbvKjdE 8TtLD2vLoe4KagWLGz6PMr9MbNdWKZhvTaSQQgcWOfBLTaIIgsnkEUqCluuIIuceNOBT rmzUxiig8a1cOkXxyfUyEIeDfl4cSWvJgl+VCbV0muFLqXSNZzvF/HAZWGsxroKxuSbw 6MBg== X-Gm-Message-State: AOJu0Yx5Qt3aw7gt3zUtDrOXnkVe7VtdlLdTxA+3Exh9daqz/7qCl4Py xPTdXQ9yfXiOeAgoyfc3H2EPzxyYIRBljEnhNXV2KzpnDchJTZBkfZhFMlyIjhOoKD9riJ7nusI SU8YpS2Zfc1EExktV2a/4GXZmcubM/BwLHWb+d4ko5n+9Pr0S/7zUOaPpGCFbwmQNqIvmrMWzFA NIUKlrHfcLRTcf X-Google-Smtp-Source: AGHT+IE9oYLDJ5bCsCbOhAXyUHCScR3+k6b8mSh525upe28RPQFVODu3nBTYeFAoi2IDDhBnDjx1Vw== X-Received: by 2002:a05:6a20:8783:b0:194:958c:6a39 with SMTP id g3-20020a056a20878300b00194958c6a39mr1290376pzf.93.1703059051113; Tue, 19 Dec 2023 23:57:31 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.57.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:57:28 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Han-Kuan Chen , Andy Chiu , Albert Ou , Conor Dooley , Andrew Jones , Heiko Stuebner Subject: [v6, 03/10] riscv: Add vector extension XOR implementation Date: Wed, 20 Dec 2023 07:54:05 +0000 Message-Id: <20231220075412.24084-4-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235735_364836_E93CA843 X-CRM114-Status: GOOD ( 14.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v2: - 's/rvv/vector/' (Conor) --- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..903c3275f8d0 --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include + +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); + +static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_vector_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_vector_end(); +} + +static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_vector_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_vector_end(); +} + +static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_vector_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_vector_end(); +} + +static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_vector_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_vector_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..3bc059e18171 --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +ENTRY(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +ENTRY(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +ENTRY(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +ENTRY(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Wed Dec 20 07:54:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499634 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B7821C3DA6E for ; Wed, 20 Dec 2023 07:58:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bvmWtn+omo1hkN1LE5+OU9uMXkvdc3PJqb0JzaZ8CtA=; b=GdyLGYsKI1adOr /3y2zd5VEYB/pvxstelJgnF0WBpW238k+tJKbGE4heG8GCwMbDLgjPWUahbs3On81wmyr4chdaxDR 47nnmVjpMaplqkyOYGCXr4PuZ2i2V3TFQIo3b/snNyrt0jCApst+oSGe2G7nRfqNZPgOiDH2DOIu3 zKK5n/BksG/380wpCVsGlziRUg25RYC1imC5zEYGKN71CisBNEVDlt3n11/2fT18pYeijw2rfDKrD PuPnOQEloqhIvwMEiEtf5MZPAEp+nKwI6ie1K+THyXb6NLSwOQXbKjIYoicqmP3evHA5ruxGDyHTk c7bSh1A/pJoJYf0ryJuA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSm-00GUaG-10; Wed, 20 Dec 2023 07:57:56 +0000 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSh-00GUWV-2G for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:57:54 +0000 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1d3d0faf262so19348625ad.3 for ; Tue, 19 Dec 2023 23:57:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059066; x=1703663866; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3jKdEEX1QxKneuBo1QXMeqJj0gX6w38gsAihdX7GUMI=; b=IB+pQGMBcgKsyIhXiMTDx2a2eFzxHPsyxrAem8Y5wyHbb3qVhRCvGjU25dIgNFa4f9 Y6AwY1wgoUOkcKzoP1ltFfTvc05S927+nveg8tr2PSjJJEq1SnwRr4vf0EOgtrkhzdTy jv0dj4vSvW+j3F8hDAwgZDPIvJUzl9qK0FMryZVYmqdMkwyQ1haw2kVrfxD9kf5Bf9tp 3rq6iXP7vfNvpjFDfwdd+cLcxZ6U/ccLUfawq25fnnDeTpuBkkM/ZL2ic/1fEEBNqfxe P8Gy62Qqzqigc2RWGYeYJbbMgeqWAiUh4Gq6V6cfN6Xz/+kCqF18kiXxM8kfFNasAtUt zlmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059066; x=1703663866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3jKdEEX1QxKneuBo1QXMeqJj0gX6w38gsAihdX7GUMI=; b=XH73W4bv+o//pq10qQKblDvXY17sx+javm521t7inMTvmuqqEKfN10MppdlVoAvST3 r6qCrSLztNO/zOykf2A6eOKOEG+lILygbOsqvNMxBbrN4dpLfsOXLxf8rVhPMXHnr57H vRBf5wu2hesMWks2DoW6hYoBKwgCIvVDy+HSONiLdWy/Mn5lEL7AadR9a/hy1DoE5gBH atx6Cx/lAjNAZ5g01LMlBCStw26t2HwoX+oYmJmIzaeJgmsu23qCA5aEdzBFQlVwv8+x P+AH+4rDWY96KGAvo+WWcCaIFEpK3y1FAP3QRu1MVnHE2iOcLXq8r1KbgyOjADmc3oRK nsnA== X-Gm-Message-State: AOJu0YzpiOlM5+CNBBlnVHiZfgi2m9fOmq3Aaz6YVkSKoQjyOtIBcW7S y4UyzMO0DIfe0OAGDhd4vgXEnMRm8k+oUEA9INnPvnEgXaqKqXhCAup4aCeGWqBCNKhUiWO7yD4 OrWJ63KZYMw4UYtzeI+W0TBGlqxgQOtUF5jnYOIYoy85B1Mf/7U82g3wMl1NKt7CuTlFhtKilTZ PaX8nORBkodPAY X-Google-Smtp-Source: AGHT+IEAKp5RvCU2obY/TteVr/tlY+NHKy+sUKSCmJxrSSBaQ6unVYal76KAjWgGu/crrbJnPcDjYA== X-Received: by 2002:a17:903:28f:b0:1d3:d8e5:9e38 with SMTP id j15-20020a170903028f00b001d3d8e59e38mr4046441plr.2.1703059065344; Tue, 19 Dec 2023 23:57:45 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.57.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:57:43 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Oleg Nesterov , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Sami Tolvanen , Jisheng Zhang , Deepak Gupta , Vincent Chen , Heiko Stuebner , Xiao Wang , Haorong Lu , Joel Granados Subject: [v6, 04/10] riscv: sched: defer restoring Vector context for user Date: Wed, 20 Dec 2023 07:54:06 +0000 Message-Id: <20231220075412.24084-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235751_779387_7148B77A X-CRM114-Status: GOOD ( 21.43 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v4: - fix typos and re-add Conor's A-b. Changelog v3: - Guard {get,put}_cpu_vector_context between vstate_* operation and explain it in the commit msg. - Drop R-b from Björn and A-b from Conor. Changelog v2: - rename and add comment for the new thread flag (Conor) --- arch/riscv/include/asm/entry-common.h | 17 +++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/ptrace.c | 5 ++++- arch/riscv/kernel/signal.c | 5 ++++- arch/riscv/kernel/vector.c | 2 +- 8 files changed, 41 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 7ab5e34318c8..6361a8488642 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,23 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + /* + * We are already called with irq disabled, so go without + * keeping track of vector_context_busy. + */ + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 574779900bfb..1047a97ddbc8 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -103,12 +103,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 6254830c0668..e706613aae2c 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -205,6 +205,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -212,7 +221,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index db0cf06f2abf..3f1d67109e5a 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -92,7 +92,7 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_restore(current, task_pt_regs(current)); + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_disable(); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 5c4dcf518684..58127b1c6c71 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -171,6 +171,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -187,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 2afe460de16a..7b93bcbdf9fa 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,8 +99,11 @@ static int riscv_vr_get(struct task_struct *target, * Ensure the vector registers have been saved to the memory before * copying them to membuf. */ - if (target == current) + if (target == current) { + get_cpu_vector_context(); riscv_v_vstate_save(current, task_pt_regs(current)); + put_cpu_vector_context(); + } ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl; diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 88b6220b2608..aca4a12c8416 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,10 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); + get_cpu_vector_context(); riscv_v_vstate_save(current, regs); + put_cpu_vector_context(); + /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); @@ -134,7 +137,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 578b6292487e..66e8c6ab09d2 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Wed Dec 20 07:54:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E958C3DA6E for ; Wed, 20 Dec 2023 07:58:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qZe6AD/YWVqXQ/gKrX7aata/mRZ1UQ4iHP0SjhcapQI=; b=2tFQo+pBfyD6eq UrEOLVbAcid74o8XNLKDhEEKBw99iUrRdWPn5N0SLANZp9s5uEGiRcyuuQR9mUjjkm86YPWLvljRI 4U65uQhYIVYY35xXba9M9Bcuy95aZh0WlxHIMXJp6ihXhrvyM0ih4pAQwSo1Z5NjeKwpr3Y7cHtEr 9sqEIszp5wK8U9jIpqOaYTbOi/A3c3sOnK2HX8kYEQw7+2ubbQgIO3Xt91CHpLV4OAzJmJn7pRy2+ l9BjoMVr6omSaMJqgScZS27mC7EUXQSVqdFABqpMmeGrjBkudW8CoRhlpS/xrjIVTd7kVU8tJto33 ef9tqPGqAMigUsrj70Rg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSt-00GUfK-1C; Wed, 20 Dec 2023 07:58:03 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrSp-00GUbO-1h for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:58:01 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1d3e8a51e6bso7380705ad.3 for ; Tue, 19 Dec 2023 23:57:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059076; x=1703663876; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=+5y5mXY/GjTuMPoMs/eCuiabcFqtKPrJHseD8GV4oj8=; b=XCnEZhi5uNjVzldJGOtylmc+meQb2HTbbROVgKUeBfdQTDI9AqknhBdW5LjSgvqglo bGEahksjDgrYE+3qi3vLGLWXQ9LUwvXdLAlZ7jJlLzhHOKmYbvKI1DjN0oF9eoN5GCe0 Xp7gY1LQCiSuqieL6Nz+8iT/P247+0jcUSCtls0yDFAr8NrPgM27zhnMgGi9SxGT87Ba XMh2p6+T0VIh6kuX79yvZw8IpTLuo+LazEs7O7Ge+qsSKf0UiVec7h4HpwAJVCHjHCYE UUc5wwcCsqkAAgbjqH1ZPuu/N5z1OBPDAujxYqvJLsGFoLt4IEDPLQ2zuQLkkphKQRtc /8iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059076; x=1703663876; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+5y5mXY/GjTuMPoMs/eCuiabcFqtKPrJHseD8GV4oj8=; b=Uhad1UR5NXGz5q31uLuAvLamiOCDQ1oKsveVuzTjVisTLVt0nnxblCVhcwCi5LMXtb 25t5fPMrGoWAY5klKWIgIQ53A/6IdgqLQbAPWlvtjkpMgj4Ixf0sIIPWpIQUV0smFOLk fSq9GQT/3DZ5vyl5AeiA77kcwMt3LPZV3cMiAuTzKpSziln6rOSgNuS/2aC4A+CFpUBh tjp640SrZ5OjCcrKkF+7oWVN/7dBVgNpAT59VbNDpbKoxQnkqr1J1uai8eiqRAME6ezY 3o6Nb7rNrhIUvYTyBzRCiudruCkbC7OCGkdHTYQTLUIfFgqb+pZMvZXnHhf4lX6HUvlT 1T2g== X-Gm-Message-State: AOJu0Ywee/SDSM6yYR2XpTECn63cwQkviVvdUcesdoqITOnVuQxZuLV+ JN6Rzmru3pjW6WnlqG2ZNY/pJ5lzXF/YRDIGamk8+w0xPlCH3DQrmai/+9W2HPEwBts1cN/MyxX KBEsZxI7dbEGcbccJhaOmOGhUKhLIS8k+oMCbqpaJ3h8WdfomYmdX4SyAkkw2Oxhc3YuXCHRIcE husqKQEmJWapZC X-Google-Smtp-Source: AGHT+IGiciNCrz7IJZBh89xO3JgJdIWf8ReQ7b0YXHaKA7drz6+cfTNsAWzv27t793xAMJkwcEkfhQ== X-Received: by 2002:a17:902:e885:b0:1ca:7f91:aa5d with SMTP id w5-20020a170902e88500b001ca7f91aa5dmr22870702plg.16.1703059076008; Tue, 19 Dec 2023 23:57:56 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.57.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:57:54 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Conor Dooley , Andrew Jones , Han-Kuan Chen , Heiko Stuebner , Aurelien Jarno , Alexandre Ghiti , Bo YU , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= Subject: [v6, 05/10] riscv: lib: vectorize copy_to_user/copy_from_user Date: Wed, 20 Dec 2023 07:54:07 +0000 Message-Id: <20231220075412.24084-6-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235759_569735_49D2FB38 X-CRM114-Status: GOOD ( 21.95 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch utilizes Vector to perform copy_to_user/copy_from_user. If Vector is available and the size of copy is large enough for Vector to perform better than scalar, then direct the kernel to do Vector copies for userspace. Though the best programming practice for users is to reduce the copy, this provides a faster variant when copies are inevitable. The optimal size for using Vector, copy_to_user_thres, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it. The exception fixup code of the __asm_vector_usercopy must fallback to the scalar one because accessing user pages might fault, and must be sleepable. Current kernel-mode Vector does not allow tasks to be preemptible, so we must disactivate Vector and perform a scalar fallback in such case. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Signed-off-by: Andy Chiu --- Changelog v6: - Add a kconfig entry to configure threshold values (Charlie) - Refine assembly code (Charlie) Changelog v4: - new patch since v4 --- arch/riscv/Kconfig | 8 +++++ arch/riscv/lib/Makefile | 2 ++ arch/riscv/lib/riscv_v_helpers.c | 38 ++++++++++++++++++++++++ arch/riscv/lib/uaccess.S | 10 +++++++ arch/riscv/lib/uaccess_vector.S | 50 ++++++++++++++++++++++++++++++++ 5 files changed, 108 insertions(+) create mode 100644 arch/riscv/lib/riscv_v_helpers.c create mode 100644 arch/riscv/lib/uaccess_vector.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 95a2a06acc6a..3c5ba05e8a2d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -525,6 +525,14 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_UCOPY_THRESHOLD + int "Threshold size for vectorized user copies" + depends on RISCV_ISA_V + default 768 + help + Prefer using vectorized copy_to_user()/copy_from_user() when the + workload size exceeds this value. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 494f9cd1a00c..1fe8d797e0f2 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -12,3 +12,5 @@ lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o +lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o +lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o diff --git a/arch/riscv/lib/riscv_v_helpers.c b/arch/riscv/lib/riscv_v_helpers.c new file mode 100644 index 000000000000..139e5de1b793 --- /dev/null +++ b/arch/riscv/lib/riscv_v_helpers.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2023 SiFive + * Author: Andy Chiu + */ +#include +#include + +#include +#include + +size_t riscv_v_usercopy_threshold = CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD; +int __asm_vector_usercopy(void *dst, void *src, size_t n); +int fallback_scalar_usercopy(void *dst, void *src, size_t n); +asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n) +{ + size_t remain, copied; + + /* skip has_vector() check because it has been done by the asm */ + if (!may_use_simd()) + goto fallback; + + kernel_vector_begin(); + remain = __asm_vector_usercopy(dst, src, n); + kernel_vector_end(); + + if (remain) { + copied = n - remain; + dst += copied; + src += copied; + goto fallback; + } + + return remain; + +fallback: + return fallback_scalar_usercopy(dst, src, n); +} diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index 3ab438f30d13..a1e4a3c42925 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -3,6 +3,8 @@ #include #include #include +#include +#include .macro fixup op reg addr lbl 100: @@ -11,6 +13,13 @@ .endm SYM_FUNC_START(__asm_copy_to_user) +#ifdef CONFIG_RISCV_ISA_V + ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_v, CONFIG_RISCV_ISA_V) + REG_L t0, riscv_v_usercopy_threshold + bltu a2, t0, fallback_scalar_usercopy + tail enter_vector_usercopy +#endif +SYM_FUNC_START(fallback_scalar_usercopy) /* Enable access to user memory */ li t6, SR_SUM @@ -181,6 +190,7 @@ SYM_FUNC_START(__asm_copy_to_user) sub a0, t5, a0 ret SYM_FUNC_END(__asm_copy_to_user) +SYM_FUNC_END(fallback_scalar_usercopy) EXPORT_SYMBOL(__asm_copy_to_user) SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user) EXPORT_SYMBOL(__asm_copy_from_user) diff --git a/arch/riscv/lib/uaccess_vector.S b/arch/riscv/lib/uaccess_vector.S new file mode 100644 index 000000000000..7bd96cee39e4 --- /dev/null +++ b/arch/riscv/lib/uaccess_vector.S @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include +#include +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + + .macro fixup op reg addr lbl +100: + \op \reg, \addr + _asm_extable 100b, \lbl + .endm + +SYM_FUNC_START(__asm_vector_usercopy) + /* Enable access to user memory */ + li t6, SR_SUM + csrs CSR_STATUS, t6 + +loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + fixup vle8.v vData, (pSrc), 10f + fixup vse8.v vData, (pDst), 10f + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + add pDst, pDst, iVL + bnez iNum, loop + +.Lout_copy_user: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + li a0, 0 + ret + + /* Exception fixup code */ +10: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + mv a0, iNum + ret +SYM_FUNC_END(__asm_vector_usercopy) From patchwork Wed Dec 20 07:54:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C306FC46CD8 for ; Wed, 20 Dec 2023 07:58:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XEk/QqnqhX8ZoFxmGrbRutbbfOwccT8QZsLZma5oEkY=; b=uW9+03P+fVncmD zi7bE9pGtR8BNCIL6YJP1KXLYfv72dFxXb5WLrlzVxiBNttHyYO390KIboxXst5D1Le5/EXL3LVjW gLvuitbz23CrOnaaUVxa0ZFsGkDCfYzK9FJ6eD25BrvLzGNiQInsO+YnZ/hGDEce96e7HYVn2gpsG esyuN66EwT55ZmGbVbHTkfG0baetFQ3g0SHlyEQOcJfpVGiR3ajKdOCSih4SNjIcw9MhWwNSMml4Q rocafk14gguq0TLmF34P3eiP5qrrT84+LJXSCbNh3uHv6kuj60Dj+XXS5m5qJcnm3AIduYSkHbz6J EACw54N3Oh56TaDSuZcg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrT3-00GUks-12; Wed, 20 Dec 2023 07:58:13 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrT0-00GUjA-09 for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:58:11 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1d3ef33e68dso1964855ad.1 for ; Tue, 19 Dec 2023 23:58:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059088; x=1703663888; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=503CjaG/B7x1b1AikMZQqWs8/BfTT3y9kpJ7D16KU0k=; b=FFMjT7MaievqmFb16QWPmcsITgB+yhIG4BRdGSJ6Fb6FqTRgezNlij5fhtaN6ml+mJ 8CMTBYhwrjoRCWeaFylYKcZ4vMIohNwrcKORuRL9jMFRqvXee2/Rh5Znfhij2/2C7fTq qUHY9uYs+W+RqGZsAS950QJ2vWK4XaNay2xBJJ+K/4WSIjjo0EQdyL2zxDkJG5wIVbw1 ZTabROri75nijhdGfW+RcIxe7geygYaRUSaP0bfuLP7nxiIKV3iLGrB9YoJjnDAwsvc1 eCmiWdUCUelquWX9EvtxXKQa6YU9jYrv1m8l8bU2myOj9fbv8RY2j6v7AciPu2jYvySc 1bFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059088; x=1703663888; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=503CjaG/B7x1b1AikMZQqWs8/BfTT3y9kpJ7D16KU0k=; b=b9yjXAwq/7t74WdanXrwjLOAZwwuP5Pj5wW6gLTZAJ/xdnZAddulY5LZoK52tNvVLC oA16AqSdcyZYuT1rJ1tbvq3F1K7dxOhgB6x/zR8Yz9lsEdUprQpQb7IomLox9W0FeVr0 Bzf3Zdz2wCISJXI7WB/6YjeZZ+6isUPbtdKVrSYFuwsiwSoFElubq6vD59pdHQXKGLAr DVWybwLdnHixg1bn/gwIKnpWKuZQrH4Es93suPNShAINgY4dcUgS9rNoTYE0BEJJR94x mrUThWY+qYybKR9VLlr5PRYmZ+WMKFuVXsLWgLc95Dg7Xs6URiuGZauzmWjTxmLyPwLo Sj9A== X-Gm-Message-State: AOJu0YywCAz6ZepxBiIVf2bhzKgvaot1DmDVMc9QyZauFdCmMv9kpqWJ 26iQQOe1HKhynighgBL9Lk0xMIx7Dz9jrkOCG+MakvkoiBsW4/0CnXwgRnXYZ+5wcS5jh9Nd0bg /YNawHgfLIi4kyo3BLG4L4Ei05hqFIAobyyrllmkgD1bzIa9++olfg6RBpXyez8/PcwmJJsJ2/Y 5wn/WGnxmJkyz2CIYsZkVs X-Google-Smtp-Source: AGHT+IG+R1pDScKLWVc5PYFWbJNHc+b7F3OwZ3UDvqnkMRZoAeuNeE1rn07Ee7f4lNtWJaePGVdtjw== X-Received: by 2002:a17:902:db10:b0:1d3:d7d1:fc68 with SMTP id m16-20020a170902db1000b001d3d7d1fc68mr3339200plx.32.1703059088305; Tue, 19 Dec 2023 23:58:08 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.58.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:58:06 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Conor Dooley , Andrew Jones , Han-Kuan Chen , Heiko Stuebner Subject: [v6, 06/10] riscv: lib: add vectorized mem* routines Date: Wed, 20 Dec 2023 07:54:08 +0000 Message-Id: <20231220075412.24084-7-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235810_109140_0DE02D78 X-CRM114-Status: GOOD ( 15.32 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Provide vectorized memcpy/memset/memmove to accelerate common memory operations. Also, group them into V_OPT_TEMPLATE3 macro because their setup/tear-down and fallback logics are the same. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Signed-off-by: Andy Chiu --- Changelog v6: - provide kconfig to set threshold for vectorized functions (Charlie) - rename *thres to *threshold (Charlie) Changelog v4: - new patch since v4 --- arch/riscv/Kconfig | 24 ++++++++++++++++ arch/riscv/lib/Makefile | 3 ++ arch/riscv/lib/memcpy_vector.S | 29 +++++++++++++++++++ arch/riscv/lib/memmove_vector.S | 49 ++++++++++++++++++++++++++++++++ arch/riscv/lib/memset_vector.S | 33 +++++++++++++++++++++ arch/riscv/lib/riscv_v_helpers.c | 22 ++++++++++++++ 6 files changed, 160 insertions(+) create mode 100644 arch/riscv/lib/memcpy_vector.S create mode 100644 arch/riscv/lib/memmove_vector.S create mode 100644 arch/riscv/lib/memset_vector.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3c5ba05e8a2d..cba53dcc2ae0 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -533,6 +533,30 @@ config RISCV_ISA_V_UCOPY_THRESHOLD Prefer using vectorized copy_to_user()/copy_from_user() when the workload size exceeds this value. +config RISCV_ISA_V_MEMSET_THRESHOLD + int "Threshold size for vectorized memset()" + depends on RISCV_ISA_V + default 1280 + help + Prefer using vectorized memset() when the workload size exceeds this + value. + +config RISCV_ISA_V_MEMCPY_THRESHOLD + int "Threshold size for vectorized memcpy()" + depends on RISCV_ISA_V + default 768 + help + Prefer using vectorized memcpy() when the workload size exceeds this + value. + +config RISCV_ISA_V_MEMMOVE_THRESHOLD + int "Threshold size for vectorized memmove()" + depends on RISCV_ISA_V + default 512 + help + Prefer using vectorized memmove() when the workload size exceeds this + value. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 1fe8d797e0f2..3111863afd2e 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -14,3 +14,6 @@ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o +lib-$(CONFIG_RISCV_ISA_V) += memset_vector.o +lib-$(CONFIG_RISCV_ISA_V) += memcpy_vector.o +lib-$(CONFIG_RISCV_ISA_V) += memmove_vector.o diff --git a/arch/riscv/lib/memcpy_vector.S b/arch/riscv/lib/memcpy_vector.S new file mode 100644 index 000000000000..4176b6e0a53c --- /dev/null +++ b/arch/riscv/lib/memcpy_vector.S @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 +#define pDstPtr a4 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + + +/* void *memcpy(void *, const void *, size_t) */ +SYM_FUNC_START(__asm_memcpy_vector) + mv pDstPtr, pDst +loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + vle8.v vData, (pSrc) + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + vse8.v vData, (pDstPtr) + add pDstPtr, pDstPtr, iVL + bnez iNum, loop + ret +SYM_FUNC_END(__asm_memcpy_vector) diff --git a/arch/riscv/lib/memmove_vector.S b/arch/riscv/lib/memmove_vector.S new file mode 100644 index 000000000000..4cea9d244dc9 --- /dev/null +++ b/arch/riscv/lib/memmove_vector.S @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 +#define pDstPtr a4 +#define pSrcBackwardPtr a5 +#define pDstBackwardPtr a6 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + +SYM_FUNC_START(__asm_memmove_vector) + + mv pDstPtr, pDst + + bgeu pSrc, pDst, forward_copy_loop + add pSrcBackwardPtr, pSrc, iNum + add pDstBackwardPtr, pDst, iNum + bltu pDst, pSrcBackwardPtr, backward_copy_loop + +forward_copy_loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + + vle8.v vData, (pSrc) + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + vse8.v vData, (pDstPtr) + add pDstPtr, pDstPtr, iVL + + bnez iNum, forward_copy_loop + ret + +backward_copy_loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + + sub pSrcBackwardPtr, pSrcBackwardPtr, iVL + vle8.v vData, (pSrcBackwardPtr) + sub iNum, iNum, iVL + sub pDstBackwardPtr, pDstBackwardPtr, iVL + vse8.v vData, (pDstBackwardPtr) + bnez iNum, backward_copy_loop + ret + +SYM_FUNC_END(__asm_memmove_vector) diff --git a/arch/riscv/lib/memset_vector.S b/arch/riscv/lib/memset_vector.S new file mode 100644 index 000000000000..4611feed72ac --- /dev/null +++ b/arch/riscv/lib/memset_vector.S @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include +#include + +#define pDst a0 +#define iValue a1 +#define iNum a2 + +#define iVL a3 +#define iTemp a4 +#define pDstPtr a5 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + +/* void *memset(void *, int, size_t) */ +SYM_FUNC_START(__asm_memset_vector) + + mv pDstPtr, pDst + + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + vmv.v.x vData, iValue + +loop: + vse8.v vData, (pDstPtr) + sub iNum, iNum, iVL + add pDstPtr, pDstPtr, iVL + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + bnez iNum, loop + + ret + +SYM_FUNC_END(__asm_memset_vector) diff --git a/arch/riscv/lib/riscv_v_helpers.c b/arch/riscv/lib/riscv_v_helpers.c index 139e5de1b793..75615998078d 100644 --- a/arch/riscv/lib/riscv_v_helpers.c +++ b/arch/riscv/lib/riscv_v_helpers.c @@ -36,3 +36,25 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n) fallback: return fallback_scalar_usercopy(dst, src, n); } + +#define V_OPT_TEMPLATE3(prefix, type_r, type_0, type_1) \ +extern type_r __asm_##prefix##_vector(type_0, type_1, size_t n); \ +type_r prefix(type_0 a0, type_1 a1, size_t n) \ +{ \ + type_r ret; \ + if (has_vector() && may_use_simd() && \ + n > riscv_v_##prefix##_threshold) { \ + kernel_vector_begin(); \ + ret = __asm_##prefix##_vector(a0, a1, n); \ + kernel_vector_end(); \ + return ret; \ + } \ + return __##prefix(a0, a1, n); \ +} + +static size_t riscv_v_memset_threshold = CONFIG_RISCV_ISA_V_MEMSET_THRESHOLD; +V_OPT_TEMPLATE3(memset, void *, void*, int) +static size_t riscv_v_memcpy_threshold = CONFIG_RISCV_ISA_V_MEMCPY_THRESHOLD; +V_OPT_TEMPLATE3(memcpy, void *, void*, const void *) +static size_t riscv_v_memmove_threshold = CONFIG_RISCV_ISA_V_MEMMOVE_THRESHOLD; +V_OPT_TEMPLATE3(memmove, void *, void*, const void *) From patchwork Wed Dec 20 07:54:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2BEDCC3DA6E for ; Wed, 20 Dec 2023 07:58:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pHgVDHjIhi7iHX57jDgV7aUCenbUnDmKM359swtKcLI=; b=wdmmlaoiYSNlIi QzC14EsTIDon4wkrinZGfnIFhDc/lR9l+/2NTjF7Rr0IIiTHIN/vMK/mCoc0Qj0yS3aoJMle/AQWe 3J7CCg1lhdWMfrbIP4yIS5JCVKBBccp4DAn/z523ap7TmOOD7mto8JxmQrViZD4RcidpvUGnG8Cat EDjWxckOXDV6jkOZuPu9ePbJ1fcjVIPSMdYwco02mNYAf3ygv+TMl5CUJ282xrleTRf9eFAre2hAR 1YhutkmkgVNbmbi6Gg66CAggof6AXEfOjtfkH7YRcaqmg2B4ZYaTfbY0tM2CIagOchhpIfI2ER3ka 04clG4FcV5WPAN3NjAew==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTG-00GUrB-13; Wed, 20 Dec 2023 07:58:26 +0000 Received: from mail-pf1-x42c.google.com ([2607:f8b0:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTD-00GUo3-0B for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:58:24 +0000 Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-6d84b68a866so1743833b3a.0 for ; Tue, 19 Dec 2023 23:58:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059099; x=1703663899; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0PStOdX0bJ0JRt1kGvqh4Fd4K0oqMPuH8sOnQ/vCigs=; b=j5gUP9jYmAKnW51vYoHGFBw8eZiVhc4A+k6hAPizXDKKWursYztU/1A9GbEVy9k3KO snVPRVecMuvWXo27QSh0g9Y2bNmd2ld8Kvyhd9uiXjBV2+g30bKTWyWck4Z/jLPxe5nr P5GoHO0Yqj+kQIdyrbTahPI4n8PVvwInArZfhIl8OwhiuXULkPfKanZ42p0yTKIGkRZP Aij8R+DJX1W5D7UbRhWA20PwevTnFQiHhtyHK4CqPWZp2FUKZkw9SrVBOo53PzRnI+qA iI1mLMNLoWbxtzUyFQKHk1R3L67OM6e8yJluIUKpsLQ6WPncXOZeokBmjR46gjl2m0mM Z6nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059099; x=1703663899; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0PStOdX0bJ0JRt1kGvqh4Fd4K0oqMPuH8sOnQ/vCigs=; b=Zli29D7AIyBu8rzE6wj/0GznCwel3oHbdCTpqIOrRnkf9Jo9gGdJdbGpaC41MKyTrO Hq0rM1pVSegxgWB5pSQ1KgKjinM/mfmb8AD6HLL366TxPQjc8AfcqsoMo43B8SuAztJ1 /LAyi7AsTsvVJyaLN0xyYRnxsdE467ldLSmgL2Edw7CsCqjHznXo7bue9Gt/OmcYGaHF IKnYPHBsuAEpXtwNGdTkswvZQWbAXAJkfaiJJWPysUkI5Du87Ene8MC8fr1+hkhooQWq tOTx8tjjk03TKZ8sO8TEo9d/08oX+aIiilS7rTxVpqGqwYjsEzRwU1DymsXT0+gzBkEC 6IHg== X-Gm-Message-State: AOJu0YzgDUpORrLdGsKWFCzJUx1hr+YJo6T9Afc2rpRRAG1FtNEFmaYB Rt3egFMt/B0+5wWwJZfP3VjomAhHfFuW0UUxOeZiIZioCKcxeee5pbe1S6T0hdQIMUoXxD8ix8e sN114gRNqJDuVw7rVIomRl+iMnT4yy5Z4HfyjEd0TnvXAdCBicja+q/YpVe6KGnAm5UklV4V44p B1Ie/fDz9t8qcC X-Google-Smtp-Source: AGHT+IH5h/HccS31pjZUAoSEzdkyMR6zp0RIo7HJTu5uye8X0F14SrH5/JM0qOVxnUmE8T6oueRIew== X-Received: by 2002:a05:6a20:429f:b0:18f:97c:4f4c with SMTP id o31-20020a056a20429f00b0018f097c4f4cmr10981264pzj.88.1703059098902; Tue, 19 Dec 2023 23:58:18 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:58:17 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Oleg Nesterov , Conor Dooley , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , =?utf-8?b?Q2zDqW1l?= =?utf-8?b?bnQgTMOpZ2Vy?= , Vincent Chen , Heiko Stuebner , Xiao Wang , Haorong Lu , Mathis Salmen Subject: [v6, 07/10] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Wed, 20 Dec 2023 07:54:09 +0000 Message-Id: <20231220075412.24084-8-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235823_119214_180CE62D X-CRM114-Status: GOOD ( 12.04 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v6: - re-added for v6 Changelog v3: - save V context after get_cpu_vector_context Changelog v2: - fix build fail that get caught on this patch (Conor) --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 6361a8488642..08fe8cdbf33e 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -16,7 +16,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, * We are already called with irq disabled, so go without * keeping track of vector_context_busy. */ - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index e706613aae2c..c5a83c277583 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -183,23 +183,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -220,7 +216,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(&prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -238,8 +234,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 3f1d67109e5a..238154cb4fce 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -72,7 +72,7 @@ void kernel_vector_begin(void) get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); riscv_v_enable(); } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 7b93bcbdf9fa..e8515aa9d80b 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -101,7 +101,7 @@ static int riscv_vr_get(struct task_struct *target, */ if (target == current) { get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); put_cpu_vector_context(); } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index aca4a12c8416..5d69f4db9e8f 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -87,7 +87,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); get_cpu_vector_context(); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); put_cpu_vector_context(); /* Copy everything of vstate but datap. */ From patchwork Wed Dec 20 07:54:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499638 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 33550C3DA6E for ; Wed, 20 Dec 2023 07:58:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DBoi3TludzXQKB4y5mbYXIWz1RaUEj0UxOrD0WXy9GU=; b=Y+D7xRbFztDOuz VZ7P1S8WZou5uIS/Agl2EA6oQmGHslEBmUv7DjrPHxdwwaUhwrgJkAjLuaRenRTWGTTnDC2p8owIf FmYCkHRqoxtaCgE6wYNXdVABC8BjBIIyu5zvFG/3nyci7rpeXvAjELmGHfbCQCBGoLf50QjW8eO7+ loM1xe/CTsv/5UBH/UXsHVhYkOEiBe5xHHVIZmol+wm202+bm2Oa00vT+fWSRtiUe/ELAcJWgUZwa VdTX+YupWytfHRCh1R2Dl/ob9MsYt8UBPeDSJB5o+MSEUH0CS02cyJFFvKtVxs92LV8n5KaxbmZbm nSBLzTwXz0hBYVDpvI3w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTL-00GUu3-0p; Wed, 20 Dec 2023 07:58:31 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTI-00GUr8-2a for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:58:30 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1d3ddccfaffso11356425ad.1 for ; Tue, 19 Dec 2023 23:58:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059105; x=1703663905; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=P5Cxj4sYn/DUqYRmscS0Hi8QTiyLCMucfAQpAUL42TnZFrnExtv89wAHyWaKA6K9lS JSSnP7J6jAbDpLyyroOXGhj/L24/DcH2gQUbErSwVAbG6Z3kmAweahbLXv/Lzrno7hsC mOAU/GBI1nCjf4zZui7dPcSYWau9i82HcLEJo93EkahVjQMHpP12VhKatTplgFN2Vp23 0u0SR9VkmeLX79QZuRoJl8Zm2oytlcOTZZvU9vbOUY+w6BYkU/pLcdq6uUcMdv2f6ulm STdaO6my3avU9o5y9F+IRx5eD5+hz++tIpCup93mmCstJmvpx2zXICHKbVSsxcq3YKH8 aUvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059105; x=1703663905; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=FcnGV7UblCxWu0yPmt5286YnlIQqkkLn4u7l2K34mVhX2DuIn5N+NHJXupiKTZFmVP 16/Va3N3FOFdTI7E5jdZuNynLCkZ56Fq2UaXPBi1BjNOfdju/v1JX/Z5h6VsGnJyx7yS anD8fXuW8/o/3J34O1yq+6a/EXNfc6DJV48cNKcLnDuFNCr1tT1Gf0Zvvn73hrHvfDZ8 n54HULEqCqXqLliXVNPyvGeX2Cqmj0IAuqTFWIhqL9eRqaVfP7TUAlfUCnjqpTe3a8Bu 8QZic2jum0QRwR942/sYJINo+xuSwW0P66u28keKq/8kTwsfq7tbwQ5WYx2phZPIls+/ mGPA== X-Gm-Message-State: AOJu0YwERRKwfee84HUMcg2yH6lX6ysIfCKJvtoxwIuLsDiCMfftrIu7 4C9pYSOKXP1YfxXMmIq9EDH4resJjrzuq1Jruwy0tfGW8LqAXg4p1RFK6KWgJWNI+dCYBMHyxfX kricxJFIh4pMZZxcboYejVcFn+ldqABINCNZk6SO5EuiGRoQB63BrohXo1KHE1B/Vj3gi5tkio+ f4KTvMzU4EDTQq X-Google-Smtp-Source: AGHT+IEqLdd4681a2XlLts0MEEprcq7bkwekiWYtJkmDWco/5+ierGWMWhhE5el7ppd9a/m4JCQCiA== X-Received: by 2002:a17:902:6547:b0:1d3:eca1:91b8 with SMTP id d7-20020a170902654700b001d3eca191b8mr730511pln.46.1703059104684; Tue, 19 Dec 2023 23:58:24 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.58.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:58:23 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Conor Dooley , Joel Granados Subject: [v6, 08/10] riscv: vector: use a mask to write vstate_ctrl Date: Wed, 20 Dec 2023 07:54:10 +0000 Message-Id: <20231220075412.24084-9-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235828_838036_B5B618DE X-CRM114-Status: UNSURE ( 8.48 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_ctrl_set() should only touch bits within PR_RISCV_V_VSTATE_CTRL_MASK. So, use the mask when we really set task's vstate_ctrl. Signed-off-by: Andy Chiu --- Changelog v6: - splitted out from v3 --- arch/riscv/kernel/vector.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 66e8c6ab09d2..c1f28bc89ec6 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void) From patchwork Wed Dec 20 07:54:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 359C0C46CCD for ; Wed, 20 Dec 2023 07:58:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FZDpG9eOj/DqNq4PDJHmw5uH4Vq2oOK0A3nzNPwphyo=; b=eZyxl305rVkByT OohlH44HlaJ3A2TuUxxHsiknTpVCfMCQG7dnsqDddh0CgwBYvpcNsFc/arZTmeLQZ2hvBVfxtJDyK H0WNteEHqxV5aeJGANIv/thXVB2xwZTnc1aFEeKX5wbEB0S32TchPzRbnJRGwqoeI4FElmIMagDZf wBdhyBlNEz8mIJAnFbSRvsv6i1b0PxCFChPHOCd1rDu8VDM03qz+j0p61NbDGJ/hWavNEMgv5jTXK 2abZD0SzlDVKh4p2Cey+JUfJpTl3zKrmxpjmcMai10Dlzedyi7jZCT/XqU6GsFc642isFv7kphF2l NvNHF+iHKVJHm134gd5A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTX-00GV0J-0f; Wed, 20 Dec 2023 07:58:43 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTS-00GUwS-0n for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:58:41 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1d3ce28ac3cso18223635ad.0 for ; Tue, 19 Dec 2023 23:58:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059114; x=1703663914; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=gyaAcQXJB9Pn9Hs/a7PNDEHYRp3nfW2OLz+IfLnmB1A=; b=MEOroDvB/NpJxxRsz3kfLXAJpdrli/5Yh2NCp1ICb5/I6CiUAJttFm27iV5C0PfqVV QJx9OcCePfKN8pnm96MbSeO3FhFoR1jmKEgpnLp5mq10iBknPkc/gLXzYE8kEQ8kfpX6 AKow9114e06WZPzc+6eOkF+GlKD822q4V+fc91iUkgyEvO3xYx87iMIqkbUnoPbaX6N7 Dzb5z/8HBLRHkufe4/226ziWtvXwO2AxgrfTOU9MkQQbVUDYVuq+pTcEiMT8c84d3gLR Lrc+lY1M3Y3eyW8UdmYqVuf2edAeso1DtCAHbXwfqRWCmWdc+YtDrFuq8del14w3Kh8c C0fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059114; x=1703663914; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gyaAcQXJB9Pn9Hs/a7PNDEHYRp3nfW2OLz+IfLnmB1A=; b=dYpdn2uq8HYmtC9F0z3RKSJrpaIePKj1w6P59V8alcl4fb0sEyojRkS+MYoq1RkjyJ MiZsfJYNpeFLsB9Z8tVQucYV+BEozZsY1BpOaYxZoJ+2XiPbfdS1cHNgSR+s/w4UgSRS O+bIwU7eZ4BxMBxuRe1o+JLbFADjelKj0AhVGWYjFQJe9j0fam+VE+bYUJhSjRw9uuah 3X3IccOY73E5WYnIj33j5t6QzroRMQMEiDtcK8yWSv9sk7KIvgmSMJzayruIgkj6zmYs 5DunwC/DOjCncm6Q9SzRpyqil1orz0GsBnOJcnHz1Q0IjtMmsYN06SJZLvgXmAgbgU42 6M6A== X-Gm-Message-State: AOJu0YwPajL/2ueHGp4S569i37Wipytp5T8QGl84ezhHI10PVCD/rZNp 5gMYpyxK5zdk23Q6bH6+p8+bLZPRfrRV2vqf1n6KA7/Tak81/AMsi+DFJFxNL2809MXHWdSfE1+ SgFMt+/aF29DGSuK9w3ogEx5NjaHhSG6TVHNG2MXhnMyLE/zhLqdRc5lZH8D0arKx8Ng5XPe/fe GCBeC/eFh1ICRl X-Google-Smtp-Source: AGHT+IGalfOT82tCup2UHErKLlWkzx0UVPBobwmFKOmxD7EOSJHU34YA8VpM5UHiePwNjwLNKFZSyA== X-Received: by 2002:a17:903:2450:b0:1d0:7407:3557 with SMTP id l16-20020a170903245000b001d074073557mr11070566pls.15.1703059114177; Tue, 19 Dec 2023 23:58:34 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.58.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:58:32 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Guo Ren , Xiao Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Jisheng Zhang , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Conor Dooley , Joel Granados Subject: [v6, 09/10] riscv: vector: use kmem_cache to manage vector context Date: Wed, 20 Dec 2023 07:54:11 +0000 Message-Id: <20231220075412.24084-10-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235838_300731_E910B007 X-CRM114-Status: GOOD ( 12.29 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu --- Changelog v6: - new patch since v6 --- arch/riscv/include/asm/vector.h | 4 ++++ arch/riscv/kernel/process.c | 7 ++++++- arch/riscv/kernel/vector.c | 16 +++++++++++++++- 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index c5a83c277583..0e6741dd9ef3 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -26,6 +26,8 @@ void kernel_vector_begin(void); void kernel_vector_end(void); void get_cpu_vector_context(void); void put_cpu_vector_context(void); +void riscv_v_thread_free(struct task_struct *tsk); +void __init riscv_v_setup_ctx_cache(void); static inline void riscv_v_ctx_cnt_add(u32 offset) { @@ -239,6 +241,8 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) +#define riscv_v_thread_free(tsk) do {} while (0) +#define riscv_v_setup_ctx_cache() do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 58127b1c6c71..38bdbcf9b81d 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -179,7 +179,7 @@ void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ if (has_vector()) - kfree(tsk->thread.vstate.datap); + riscv_v_thread_free(tsk); } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -227,3 +227,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; } + +void __init arch_task_cache_init(void) +{ + riscv_v_setup_ctx_cache(); +} diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index c1f28bc89ec6..1fe140e34557 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -21,6 +21,7 @@ #include static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); +static struct kmem_cache *riscv_v_user_cachep; unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -47,6 +48,13 @@ int riscv_v_setup_vsize(void) return 0; } +void __init riscv_v_setup_ctx_cache(void) +{ + riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", + riscv_v_vsize, 16, SLAB_PANIC, + 0, riscv_v_vsize, NULL); +} + static bool insn_is_vector(u32 insn_buf) { u32 opcode = insn_buf & __INSN_OPCODE_MASK; @@ -84,7 +92,7 @@ static int riscv_v_thread_zalloc(void) { void *datap; - datap = kzalloc(riscv_v_vsize, GFP_KERNEL); + datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); if (!datap) return -ENOMEM; @@ -94,6 +102,12 @@ static int riscv_v_thread_zalloc(void) return 0; } +void riscv_v_thread_free(struct task_struct *tsk) +{ + if (tsk->thread.vstate.datap) + kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +} + #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) #define VSTATE_CTRL_GET_NEXT(x) (((x) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2) #define VSTATE_CTRL_MAKE_NEXT(x) (((x) << 2) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) From patchwork Wed Dec 20 07:54:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13499640 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01F8DC3DA6E for ; Wed, 20 Dec 2023 07:59:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4FqPqAY1rQpDOUrL+qT1vy2NVIZiBbVgtDZ8OrVopm0=; b=QRK6mXWzjsirZQ RL5HTMkrcelypH9yM5OvkHJ5jgeoGqWeliyl/XKQwKywG3W/ntVVqJpdzNywC9v9j+ILaYtp524un GdE/m5XwIvkldmSPVbcm36nHe09xuZ5emGvtL1SzfYpXZoyr/xYnTOgvyNG2HgL+kLzLA5AungZXv +GfTtkbk63kgkPyDilQQCqDbeaLJrTy4W/vYqFTWfBoMrq2a5rDvgzxcqfeh/nEgi+EkwSQ2JwZu6 810TLZTfNRO0Ti0oRQelTmJ/FUlGJkVhy6QenJFgB4FEMhyCX85idr08bKbfdiXSeIpXkbkjaNjES 9WhnZ0tfPNl6fNFiGtgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTo-00GV9M-1u; Wed, 20 Dec 2023 07:59:00 +0000 Received: from mail-pl1-x634.google.com ([2607:f8b0:4864:20::634]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rFrTi-00GV4z-1i for linux-riscv@lists.infradead.org; Wed, 20 Dec 2023 07:58:58 +0000 Received: by mail-pl1-x634.google.com with SMTP id d9443c01a7336-1d3dee5f534so3424465ad.1 for ; Tue, 19 Dec 2023 23:58:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703059130; x=1703663930; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yEIlxQYdoTfBtAtUGxe3vkmN12G9f4frdyjbyxekBAA=; b=IpdJTtp03QH5nIJLSN1sAGg5KGqoG5hLMeCt94jk43MThxHCaQGWrCB79ICbhn/1aR iT5VG1jLxnRpwZaQlLigHKdfxgHu+24cuq7q/LzA0wzYP580YJraqEKrKISIqIIR8raL fGfHpAru4t/JqCDVt0b0wNC6aGBPmAFD+AUJSBiNOHQXOkIDQ0A41QJEJlZmFQwb9gzB Tt+Yt1qaG6UoCMCWGY7fVKiKticPOCYFJKXkZDxxjzxZwJjc1oiCxcLjfVY4y+vMY7mK VEppoLhA1su3p/Teiohg5Glc5aRsM7SAzSFV78DwZbEaWhhddaOApIUkBMIeC7dVvKdc HtHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703059130; x=1703663930; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yEIlxQYdoTfBtAtUGxe3vkmN12G9f4frdyjbyxekBAA=; b=THDK1zSQxVzViYmW2/xG6V+98dT0+rfamI5YokLfPGF8gJhqfewme+darqIcMDyIPx 1bOs+hi1NMaQJ1+mso6TS7Ch2Se3ZA7FxOWvDQeZ5jf/pPG+is9Gx0EThM0n0P43urrW +w5yJr8qOrmvhl89+pgUiidjrF+VDXK8XA0nWy8AW1+NaX8D1YLsLTKhpYViajxT+bi8 OShtc2Z02LKtsH8T7UFbm8Ov/KYPybQ7GGOlIBjmxMjfSHqz5ee11PaQw+xmFiXM+Fya EGYrB4hKbxumRXFNpVU0GC66OweNJ4mxUOin+yIXaZe484rViEMsMqDApqjItzpgVOD1 zh9g== X-Gm-Message-State: AOJu0Yyr8MuzXfC5iw4NsYVQcDcZohTYRXoNLp8tk3PZeEMKRG3LbybD 6x1A+COlTQDyeyEVrdQv0zqapltGQq8dPk3eJ8BhWsKs0y6cDhV+g4naly61f2IcTnw6B9O2GXz Q2Qo4wU7rjgtsM/tzgJ/l+SSg0Vfbff7vXWg0ytTNBa4watqIBgtfu0fY/Bn8kXW9jNkSJ0OlWy 3IPnhJxQXiSsODQzd0FOu+ X-Google-Smtp-Source: AGHT+IEWsrqWPrzzb47w2invj5UXqL/DFqTpuvL9MtR5x2RMz5rG7XdXMks+WhDUqk5DQyQhNUc3Fw== X-Received: by 2002:a17:903:1107:b0:1d3:bceb:ba62 with SMTP id n7-20020a170903110700b001d3bcebba62mr2733841plh.45.1703059130082; Tue, 19 Dec 2023 23:58:50 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id q20-20020a170902789400b001cf8546335fsm3441453pll.5.2023.12.19.23.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 Dec 2023 23:58:46 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Conor Dooley , Baoquan He , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Xiao Wang , Sami Tolvanen , Nathan Chancellor , Jisheng Zhang , Deepak Gupta , Joel Granados Subject: [v6, 10/10] riscv: vector: allow kernel-mode Vector with preemption Date: Wed, 20 Dec 2023 07:54:12 +0000 Message-Id: <20231220075412.24084-11-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231220075412.24084-1-andy.chiu@sifive.com> References: <20231220075412.24084-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231219_235854_588042_036D889A X-CRM114-Status: GOOD ( 31.00 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu --- Changelog v6: - re-write patch to handle context nesting for softirqs - drop thread flag and track context instead in riscv_v_flags - refine some asm code and constraint it into C functions - preallocate v context for preempt_v - Return non-zero in riscv_v_start_kernel_context with non-preemptible kernel-mode Vector Changelog v4: - dropped from v4 Changelog v3: - Guard vstate_save with {get,set}_cpu_vector_context - Add comments on preventions of nesting V contexts - remove warnings in context switch when trap's reg is not pressent (Conor) - refactor code (Björn) Changelog v2: - fix build fail when compiling without RISCV_ISA_V (Conor) - 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor) - merge Kconfig patch into this oine (Conor). - 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor) - fix some typos (Conor) - enclose assembly with RISCV_ISA_V_PREEMPTIVE. - change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor) - 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/' --- arch/riscv/Kconfig | 14 +++ arch/riscv/include/asm/processor.h | 26 +++++- arch/riscv/include/asm/simd.h | 26 +++++- arch/riscv/include/asm/vector.h | 57 +++++++++++- arch/riscv/kernel/entry.S | 8 ++ arch/riscv/kernel/kernel_mode_vector.c | 121 ++++++++++++++++++++++++- arch/riscv/kernel/process.c | 3 + arch/riscv/kernel/vector.c | 31 +++++-- 8 files changed, 265 insertions(+), 21 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index cba53dcc2ae0..70603c486593 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -557,6 +557,20 @@ config RISCV_ISA_V_MEMMOVE_THRESHOLD Prefer using vectorized memmove() when the workload size exceeds this value. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Usually, in-kernel SIMD routines are run with preemption disabled. + Functions which envoke long running SIMD thus must yield core's + vector unit to prevent blocking other tasks for too long. + + This config allows kernel to run SIMD without explicitly disable + preemption. Enabling this config will result in higher memory + consumption due to the allocation of per-task's kernel Vector context. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 15781e2232e0..4de9124bcf4f 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -81,11 +81,32 @@ struct pt_regs; * activation of this state disables the preemption. On a non-RT kernel, it * also disable bh. Currently only 0 and 1 are valid value for this field. * Other values are reserved for future uses. + * - bits 8-15 are used for tracking preemptible kernel-mode Vector, when + * RISCV_ISA_V_PREEMPTIVE is set. Calling kernel_vector_begin() does not + * disable the preemption if the thread's kernel_vstate.datap is allocated. + * Instead, the kernel adds 1 into this field. Then the trap entry/exit code + * knows if we are entering/exiting the context that owns preempt_v. + * - 0: the task is not using preempt_v + * - 1: the task is actively using, and owns preempt_v + * - >1: the task was using preempt_v, but then took a trap within. Thus, + * the task does not own preempt_v. Any use of Vector will have to save + * preempt_v, if dirty, and fallback to non-preemptible kernel-mode + * Vector. + * - bit 30: The in-kernel preempt_v context is saved, and requries to be + * restored when returning to the context that owns the preempt_v. + * - bit 31: The in-kernel preempt_v context is dirty, as signaled by the + * trap entry code. Any context switches out-of current task need to save + * it to the task's in-kernel V context. Also, any traps nesting on-top-of + * preempt_v requesting to use V needs a save. */ -#define RISCV_KERNEL_MODE_V_MASK 0xff +#define RISCV_KERNEL_MODE_V_MASK 0x000000ff +#define RISCV_PREEMPT_V_MASK 0x0000ff00 -#define RISCV_KERNEL_MODE_V 0x1 +#define RISCV_KERNEL_MODE_V 0x00000001 +#define RISCV_PREEMPT_V 0x00000100 +#define RISCV_PREEMPT_V_DIRTY 0x80000000 +#define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000 /* CPU-specific state of a task */ struct thread_struct { @@ -99,6 +120,7 @@ struct thread_struct { u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index cd6180fe37c0..ecfc3b11ef13 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -12,6 +12,7 @@ #include #include #include +#include #ifdef CONFIG_RISCV_ISA_V /* @@ -26,12 +27,27 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. - * - * Kernel-mode Vector temperarily disables bh. So we must not return - * true on irq_disabled(). Otherwise we would fail the lockdep check - * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); + if (in_hardirq() || in_nmi()) + return false; + + /* + * Nesting is acheived in preempt_v by spreading the control for + * preemptible and non-preemptible kernel-mode Vector into two fields. + * Always try to match with prempt_v if kernel V-context exists. Then, + * fallback to check non preempt_v if nesting happens, or if the config + * is not set. + */ + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) { + if (!riscv_preempt_v_started(current)) + return true; + } + /* + * Non-preemptible kernel-mode Vector temperarily disables bh. So we + * must not return true on irq_disabled(). Otherwise we would fail the + * lockdep check calling local_bh_enable() + */ + return !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 0e6741dd9ef3..542eaf9227c3 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -28,6 +28,7 @@ void get_cpu_vector_context(void); void put_cpu_vector_context(void); void riscv_v_thread_free(struct task_struct *tsk); void __init riscv_v_setup_ctx_cache(void); +void riscv_v_thread_alloc(struct task_struct *tsk); static inline void riscv_v_ctx_cnt_add(u32 offset) { @@ -212,14 +213,63 @@ static inline void riscv_v_vstate_set_restore(struct task_struct *task, } } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline bool riscv_preempt_v_dirty(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + return !!(val & RISCV_PREEMPT_V_DIRTY); +} + +static inline bool riscv_preempt_v_restore(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + return !!(val & RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_preempt_v_clear_dirty(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags &= ~RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_set_restore(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags |= RISCV_PREEMPT_V_NEED_RESTORE; +} + +static inline bool riscv_preempt_v_started(struct task_struct *task) +{ + return !!(READ_ONCE(task->thread.riscv_v_flags) & RISCV_PREEMPT_V_MASK); +} +#else /* !CONFIG_RISCV_ISA_V_PREEMPTIVE */ +static inline bool riscv_preempt_v_dirty(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_restore(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_started(struct task_struct *task) { return false; } +#define riscv_preempt_v_clear_dirty(tsk) do {} while (0) +#define riscv_preempt_v_set_restore(tsk) do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(&prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (riscv_preempt_v_dirty(prev)) { + __riscv_v_vstate_save(&prev->thread.kernel_vstate, + prev->thread.kernel_vstate.datap); + riscv_preempt_v_clear_dirty(prev); + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (riscv_preempt_v_started(next)) + riscv_preempt_v_set_restore(next); + else + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); @@ -243,6 +293,7 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vstate_on(regs) do {} while (0) #define riscv_v_thread_free(tsk) do {} while (0) #define riscv_v_setup_ctx_cache() do {} while (0) +#define riscv_v_thread_alloc(tsk) do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 54ca4564a926..9d1a305d5508 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -83,6 +83,10 @@ SYM_CODE_START(handle_exception) /* Load the kernel shadow call stack pointer if coming from userspace */ scs_load_current_if_task_changed s5 +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_start +#endif move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -138,6 +142,10 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_end +#endif REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 238154cb4fce..6a7df511ccdc 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -50,6 +50,111 @@ void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline void riscv_preempt_v_set_dirty(void) +{ + current->thread.riscv_v_flags |= RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_reset_flags(void) +{ + current->thread.riscv_v_flags &= ~(RISCV_PREEMPT_V_DIRTY | RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_preempt_v_depth_inc(void) +{ + riscv_v_ctx_cnt_add(RISCV_PREEMPT_V); +} + +static inline void riscv_preempt_v_depth_dec(void) +{ + riscv_v_ctx_cnt_sub(RISCV_PREEMPT_V); +} + +static inline u32 riscv_preempt_v_get_depth(void) +{ + return riscv_v_ctx_cnt() & RISCV_PREEMPT_V_MASK; +} + +#define PREEMPT_V_FIRST_DEPTH RISCV_PREEMPT_V +static int riscv_v_stop_kernel_context(void) +{ + if (riscv_preempt_v_get_depth() != PREEMPT_V_FIRST_DEPTH) + return 1; + + riscv_preempt_v_depth_dec(); + return 0; +} + +static int riscv_v_start_kernel_context(bool *is_nested) +{ + struct __riscv_v_ext_state *vstate = ¤t->thread.kernel_vstate; + + if (!vstate->datap) + return -ENOENT; + + if (riscv_preempt_v_started(current)) { + WARN_ON(riscv_preempt_v_get_depth() == PREEMPT_V_FIRST_DEPTH); + if (riscv_preempt_v_dirty(current)) { + get_cpu_vector_context(); + __riscv_v_vstate_save(vstate, vstate->datap); + riscv_preempt_v_clear_dirty(current); + put_cpu_vector_context(); + } + get_cpu_vector_context(); + riscv_preempt_v_set_restore(current); + *is_nested = true; + return 0; + } + + get_cpu_vector_context(); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + put_cpu_vector_context(); + + riscv_preempt_v_depth_inc(); + return 0; +} + +/* low-level V context handling code, called with irq disabled */ +asmlinkage void riscv_v_context_nesting_start(struct pt_regs *regs) +{ + int depth; + + if (!riscv_preempt_v_started(current)) + return; + + depth = riscv_preempt_v_get_depth(); + if (depth == PREEMPT_V_FIRST_DEPTH && (regs->status & SR_VS) == SR_VS_DIRTY) + riscv_preempt_v_set_dirty(); + + riscv_preempt_v_depth_inc(); +} + +asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs) +{ + struct __riscv_v_ext_state *vstate = ¤t->thread.kernel_vstate; + u32 depth; + + lockdep_assert_irqs_disabled(); + + if (!riscv_preempt_v_started(current)) + return; + + riscv_preempt_v_depth_dec(); + depth = riscv_preempt_v_get_depth(); + if (depth == PREEMPT_V_FIRST_DEPTH) { + if (riscv_preempt_v_restore(current)) { + __riscv_v_vstate_restore(vstate, vstate->datap); + __riscv_v_vstate_clean(regs); + } + riscv_preempt_v_reset_flags(); + } +} +#else +#define riscv_v_start_kernel_context(nested) (-ENOENT) +#define riscv_v_stop_kernel_context() (-ENOENT) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context @@ -65,14 +170,20 @@ void put_cpu_vector_context(void) */ void kernel_vector_begin(void) { + bool nested = false; + if (WARN_ON(!has_vector())) return; BUG_ON(!may_use_simd()); - get_cpu_vector_context(); + if (riscv_v_start_kernel_context(&nested)) { + get_cpu_vector_context(); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + } - riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + if (!nested) + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_enable(); } @@ -92,10 +203,10 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_set_restore(current, task_pt_regs(current)); - riscv_v_disable(); - put_cpu_vector_context(); + if (riscv_v_stop_kernel_context()) {// we should call this early + put_cpu_vector_context(); + } } EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 38bdbcf9b81d..1afdec4aeda6 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -188,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } @@ -223,6 +224,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.s[0] = 0; } p->thread.riscv_v_flags = 0; + if (has_vector()) + riscv_v_thread_alloc(p); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 1fe140e34557..f9769703fd39 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -22,6 +22,9 @@ static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); static struct kmem_cache *riscv_v_user_cachep; +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static struct kmem_cache *riscv_v_kernel_cachep; +#endif unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -53,6 +56,11 @@ void __init riscv_v_setup_ctx_cache(void) riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", riscv_v_vsize, 16, SLAB_PANIC, 0, riscv_v_vsize, NULL); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_kernel_cachep = kmem_cache_create("riscv_vector_kctx", + riscv_v_vsize, 16, + SLAB_PANIC, NULL); +#endif } static bool insn_is_vector(u32 insn_buf) @@ -88,24 +96,35 @@ static bool insn_is_vector(u32 insn_buf) return false; } -static int riscv_v_thread_zalloc(void) +static int riscv_v_thread_zalloc(struct kmem_cache *cache, + struct __riscv_v_ext_state *ctx) { void *datap; - datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); + datap = kmem_cache_zalloc(cache, GFP_KERNEL); if (!datap) return -ENOMEM; - current->thread.vstate.datap = datap; - memset(¤t->thread.vstate, 0, offsetof(struct __riscv_v_ext_state, - datap)); + ctx->datap = datap; + memset(ctx, 0, offsetof(struct __riscv_v_ext_state, datap)); return 0; } +void riscv_v_thread_alloc(struct task_struct *tsk) +{ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_thread_zalloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate); +#endif +} + void riscv_v_thread_free(struct task_struct *tsk) { if (tsk->thread.vstate.datap) kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + if (tsk->thread.kernel_vstate.datap) + kmem_cache_free(riscv_v_kernel_cachep, tsk->thread.kernel_vstate.datap); +#endif } #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) @@ -177,7 +196,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) * context where VS has been off. So, try to allocate the user's V * context and resume execution. */ - if (riscv_v_thread_zalloc()) { + if (riscv_v_thread_zalloc(riscv_v_user_cachep, ¤t->thread.vstate)) { force_sig(SIGBUS); return true; }