From patchwork Thu Dec 21 13:43:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1C620C35274 for ; Thu, 21 Dec 2023 13:43:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Rzz4wP0fspjsKTnmY24kvZ4JrNr9b4okuX0FgpCxGDA=; b=PPDipxqM2wDMAP qar6juaGMQnzGKyKmc1PInioT8VcyuSpcWjTj1r8K8+3jdy+jyYa9/rKzbEa/FkXAA3S1iJDDAJnq iDneAJvG+GwHl70W8GWBYl98GM3fJ6RqLT8pMT6Rkfq5J7GKFTMhzOGdRXFwyEDYckqzy94cd/D0n iZgS2vKlphKTAh6J5UBPa+KEVX2gmr8EWv3mOE/bjBJpeEB7tgsf32otD32xbTdQh6WFR59szZhjL VRYcaNGQTFqV7v0I9nECFPhuLRZyb+A5hTFAMHYD5Ty0fOvNARPR12sDmXuKAZ8OOhNNaJCj8WpBn Fmg4A9BC7xx3kxZS3HIg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJKz-0031ag-05; Thu, 21 Dec 2023 13:43:45 +0000 Received: from mail-pl1-x631.google.com ([2607:f8b0:4864:20::631]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJKw-0031Z7-00 for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:43:43 +0000 Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1d3f3ee00a2so4274165ad.3 for ; Thu, 21 Dec 2023 05:43:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166218; x=1703771018; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KC+oYSteuJ9QTRiXSDG1fgUO8aa5CYBbSm+dPx6z0O4=; b=HtwkdMK8Tf1Cb8VdgIfjzR/M0NxTrrai7krSNRDFO2L+dtUkrJD39+bEFGlO6Iwd5p 05xQX916a6VvHCGRPEfw24tWQ6sDLwaOiecdvYV5pXvbKrBVA+zeVjX1SHjOasnfGmwf j5AjRyiCpDTAE8Pzw+f0z1jKSlgb8bUtWAvUhYuI79hoG6i1LtiTHurTuMU2F9P4U9A4 xsIPtZz4cP6/JQOViFLnXg+6CHQqb/futjXrqiyysRsLelBDdYDmH2rVvPkV3CcK9G1P v3Z7REIfxssYuhwex8xk3FDkVczdiGzT3rA3x01j39Z7m2G5gtL1lELfa6wlRmvhWon6 7GnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166218; x=1703771018; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KC+oYSteuJ9QTRiXSDG1fgUO8aa5CYBbSm+dPx6z0O4=; b=WLmYdSGpmqodEUjR1kKvLXkEZE86iXBRQ+cPiRkdPJDeJmLzCo6prLw00tur1zoep7 3IlLsRSekRXJglq7x5tiNK29MBHqg6n78GrvlOBNc7QyWy12oNyoc905u+o+N+pi/2oI q6CbvXhVvyHc6EE2WOgJCuFCGQg+4FDyKVRSUjZsThs6dx7/5UHmY9jpW5CUBcV4CJIi VGrU9+XLTSMIJhQ8lu4CTsmZo68WRV0ym7xN5/xQfqDdvj02cirvn8du4B8yX8KQP/IS OPz4d3gqloDjm+xThTC1Xe3mgleOvyFI/EH6L2LV/5Sg5oMl24W66bcMZuykGPS8T21N peLg== X-Gm-Message-State: AOJu0Yz4JiAxH5HnK0Nap2LP0ttz9Za56rqIk5n4czm0S7qJ9R3Xz9Bh iHVz7DwvPgET7FLWF4QRX8Y98zu8gdJn/mmCE6EImiepbq0mVaHMUpBz1TV+20StzPRx2gGgZUq o9f94mMvzb2K2iX8IylPRqMWmDEuIJET5bDCAabwitcGG++41Qh4TpTRvh8TLm9VxhUg5s7fLeT 5ayY12l/Yy8C2P X-Google-Smtp-Source: AGHT+IEzKaaQiiiYusyZdC7BGIMbni3oLT2Nqvu7Nef4UIcvXgQMumvpb37/NkPiTTRNBXtoMQ/hQg== X-Received: by 2002:a17:903:40c6:b0:1d3:3b89:210f with SMTP id t6-20020a17090340c600b001d33b89210fmr10845190pld.1.1703166217723; Thu, 21 Dec 2023 05:43:37 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.43.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:43:36 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Vincent Chen , Andy Chiu , Albert Ou , Heiko Stuebner , Conor Dooley , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Xiao Wang , Alexandre Ghiti , Sami Tolvanen , Sia Jee Heng , Jisheng Zhang Subject: [v7, 01/10] riscv: Add support for kernel mode vector Date: Thu, 21 Dec 2023 13:43:08 +0000 Message-Id: <20231221134318.28105-2-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054342_037957_588BE606 X-CRM114-Status: GOOD ( 25.75 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen Signed-off-by: Vincent Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- Changelog v7: - fix build fail for allmodconfig Changelog v6: - Use 8 bits to track non-preemptible vector context to provide better WARN coverage. Changelog v4: - Use kernel_v_flags and helpers to track vector context. Changelog v3: - Reorder patch 1 to patch 3 to make use of {get,put}_cpu_vector_context later. - Export {get,put}_cpu_vector_context. - Save V context after disabling preemption. (Guo) - Fix a build fail. (Conor) - Remove irqs_disabled() check as it is not needed, fix styling. (Björn) Changelog v2: - 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor) - export may_use_simd to include/asm/simd.h --- arch/riscv/include/asm/processor.h | 17 ++++- arch/riscv/include/asm/simd.h | 44 ++++++++++++ arch/riscv/include/asm/vector.h | 21 ++++++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 95 ++++++++++++++++++++++++++ arch/riscv/kernel/process.c | 2 +- 6 files changed, 178 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index f19f861cda54..15781e2232e0 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -73,6 +73,20 @@ struct task_struct; struct pt_regs; +/* + * We use a flag to track in-kernel Vector context. Currently the flag has the + * following meaning: + * + * - bit 0-7 indicates whether the in-kernel Vector context is active. The + * activation of this state disables the preemption. On a non-RT kernel, it + * also disable bh. Currently only 0 and 1 are valid value for this field. + * Other values are reserved for future uses. + */ + +#define RISCV_KERNEL_MODE_V_MASK 0xff + +#define RISCV_KERNEL_MODE_V 0x1 + /* CPU-specific state of a task */ struct thread_struct { /* Callee-saved registers */ @@ -81,7 +95,8 @@ struct thread_struct { unsigned long s[12]; /* s[0]: frame pointer */ struct __riscv_d_ext_state fstate; unsigned long bad_cause; - unsigned long vstate_ctrl; + u32 riscv_v_flags; + u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; }; diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h new file mode 100644 index 000000000000..3b603e47c5d8 --- /dev/null +++ b/arch/riscv/include/asm/simd.h @@ -0,0 +1,44 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2023 SiFive + */ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include +#include +#include +#include +#include + +#include + +#ifdef CONFIG_RISCV_ISA_V +/* + * may_use_simd - whether it is allowable at this time to issue vector + * instructions or access the vector register file + * + * Callers must not assume that the result remains true beyond the next + * preempt_enable() or return from softirq context. + */ +static __must_check inline bool may_use_simd(void) +{ + /* + * RISCV_KERNEL_MODE_V is only set while preemption is disabled, + * and is clear whenever preemption is enabled. + */ + return !in_hardirq() && !in_nmi() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd(void) +{ + return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 87aaef656257..6254830c0668 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -22,6 +22,27 @@ extern unsigned long riscv_v_vsize; int riscv_v_setup_vsize(void); bool riscv_v_first_use_handler(struct pt_regs *regs); +void kernel_vector_begin(void); +void kernel_vector_end(void); +void get_cpu_vector_context(void); +void put_cpu_vector_context(void); + +static inline void riscv_v_ctx_cnt_add(u32 offset) +{ + current->thread.riscv_v_flags += offset; + barrier(); +} + +static inline void riscv_v_ctx_cnt_sub(u32 offset) +{ + barrier(); + current->thread.riscv_v_flags -= offset; +} + +static inline u32 riscv_v_ctx_cnt(void) +{ + return READ_ONCE(current->thread.riscv_v_flags); +} static __always_inline bool has_vector(void) { diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index fee22a3d1b53..8c58595696b3 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -63,6 +63,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..105147c7d2da --- /dev/null +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2012 ARM Ltd. + * Author: Catalin Marinas + * Copyright (C) 2017 Linaro Ltd. + * Copyright (C) 2021 SiFive + */ +#include +#include +#include +#include +#include + +#include +#include +#include + +/* + * Claim ownership of the CPU vector context for use by the calling context. + * + * The caller may freely manipulate the vector context metadata until + * put_cpu_vector_context() is called. + */ +void get_cpu_vector_context(void) +{ + preempt_disable(); + + WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != 0); + riscv_v_ctx_cnt_add(RISCV_KERNEL_MODE_V); +} + +/* + * Release the CPU vector context. + * + * Must be called from a context in which get_cpu_vector_context() was + * previously called, with no call to put_cpu_vector_context() in the + * meantime. + */ +void put_cpu_vector_context(void) +{ + WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != RISCV_KERNEL_MODE_V); + riscv_v_ctx_cnt_sub(RISCV_KERNEL_MODE_V); + + preempt_enable(); +} + +/* + * kernel_vector_begin(): obtain the CPU vector registers for use by the calling + * context + * + * Must not be called unless may_use_simd() returns true. + * Task context in the vector registers is saved back to memory as necessary. + * + * A matching call to kernel_vector_end() must be made before returning from the + * calling context. + * + * The caller may freely use the vector registers until kernel_vector_end() is + * called. + */ +void kernel_vector_begin(void) +{ + if (WARN_ON(!has_vector())) + return; + + BUG_ON(!may_use_simd()); + + get_cpu_vector_context(); + + riscv_v_vstate_save(current, task_pt_regs(current)); + + riscv_v_enable(); +} +EXPORT_SYMBOL_GPL(kernel_vector_begin); + +/* + * kernel_vector_end(): give the CPU vector registers back to the current task + * + * Must be called from a context in which kernel_vector_begin() was previously + * called, with no call to kernel_vector_end() in the meantime. + * + * The caller must not use the vector registers after this function is called, + * unless kernel_vector_begin() is called again in the meantime. + */ +void kernel_vector_end(void) +{ + if (WARN_ON(!has_vector())) + return; + + riscv_v_vstate_restore(current, task_pt_regs(current)); + + riscv_v_disable(); + + put_cpu_vector_context(); +} +EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 4f21d970a129..5c4dcf518684 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -187,7 +187,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); - return 0; } @@ -221,6 +220,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 = 0; /* Return value of fork() */ p->thread.s[0] = 0; } + p->thread.riscv_v_flags = 0; p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; From patchwork Thu Dec 21 13:43:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 556CAC46CD8 for ; Thu, 21 Dec 2023 13:43:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ePbB9hv4zEEFx9fIvbAJHGVmLU4/w3DiGNdCbgHMSEo=; b=d1cT96Rcg+/+Hs 2RyzHCh+zV0S/ydOx8Nv8cBapErdbw5HraNPJwn/I3YNH4y4d8wmRjgn/OXiW2OFfxtYc1FqzzOEV lvl8OShWnb7e9yRYCkZmsHuTR76Nz3Ia6eeYBbWl48NxUmWBO/jzlHABLLU0ne0K8oEgnd7tRlx3u iRBQ/b9GWyq3mX0UlMwSLNRyElatF1Rwmi/LjOtQJW00uhQ3Yj2waPmuc0c51csB8JlIwmrgnQMBO DfYMQfmtmQraMmBEsDz75d6p6uMSrWZa9AFGcB1aoWUXqvN/vwkalefJ2CMT9XGOFTxEiV4oQWgaj RFz8CC+xZQK+v3R2LOsA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJL3-0031dL-2f; Thu, 21 Dec 2023 13:43:49 +0000 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJL1-0031bR-1r for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:43:48 +0000 Received: by mail-pj1-x1034.google.com with SMTP id 98e67ed59e1d1-28bf1410e37so495047a91.2 for ; Thu, 21 Dec 2023 05:43:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166225; x=1703771025; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=Okech3HlgIWagSpdZ6UWmUk2XTG35ZUmXiF4qccU1pg=; b=EmCc8c1A2lvDXKpLnGAx9s+LBl/S4uhYs9+ULP3wvH5ww3f3T93dsWtAjMVNCHfOBU B7JcyI76xKTw3w4BD6kbACYNzaS7Md7Lpd4L48cP97L573ryKeJG9VuqIXA2EbIygP4x aLwZz59z+t2yTY8+OBy1Kfb4ruOHQeATaA4N+5bi1yoB1iZxuB2VzlozjZUbPPn/Rjk2 oWOgc+QhpbPlJ0atCRtoHazvzxOJEtzSTSyGhjMRW7jCkJdm/qYhKGDYKyVax7BOKHOf bcO7a3G9sXX4eScz8ODIaYUk5D/QkbTqq0K+MZoKeQQcCwLVhGQ0ebxNEmL9jyQCFc/b zTLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166225; x=1703771025; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Okech3HlgIWagSpdZ6UWmUk2XTG35ZUmXiF4qccU1pg=; b=f1FvV9lmuW52t0zXx57v6N578y7jPgpfIgKKkfAKYLefj1pUPX8XHJiH1MaEbCjGYQ UGTdQCvI84BLH2zd1VdptdtfCpFvGaL6nMNCCyXVO79QhqY1a41exDIGdCPWsWN7UY1L qRBP/4jZT5Zqct9GtF5EwSIsYSJRO2Dww9eGCJgOmU0aCy2+t8cFgYF9Hu86qpf3Rpa0 0xOJR3yVqsy/lQnzzji++9tW07H16yo3uN6J3fF8eObKVTR6Vsbdb3ugYsd4SGVa8Efw 2XuIeb9jYgMEXImLi8nfEF+raAHEP4IIWSubtu76802eD74k4ewHuOEXASZaiiuH00hM Th5w== X-Gm-Message-State: AOJu0YzxUuDn0Xv9waxz4L7aLpe1fwEOhKorvjsglc3g9VOW072xM1/P 8v9RYqPlnBYtxtbHPTC7GCRlqr8KoMDS5NzfjoRoRa5682yr6Yu1gZOzrERECQftoMpYAhA492Y FQo9IlZmAi8IVYPmlh6nN9haLOhZWHz0f7P99gnVl3PiZxs/dmpF8AlINAnvOpOgm5fK3V6Rd/x fR0X43VTuM9sMF X-Google-Smtp-Source: AGHT+IEF/jUm0cNA4KicQ5YSVHG03ki4VbwFRBT30U0JChfOf/MtKoONi+0dI/mScAeEjMhK0JpvAg== X-Received: by 2002:a17:90a:d715:b0:28b:c1f5:ab15 with SMTP id y21-20020a17090ad71500b0028bc1f5ab15mr3186574pju.34.1703166224655; Thu, 21 Dec 2023 05:43:44 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.43.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:43:43 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Conor Dooley Subject: [v7, 02/10] riscv: vector: make Vector always available for softirq context Date: Thu, 21 Dec 2023 13:43:09 +0000 Message-Id: <20231221134318.28105-3-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054347_611399_FDA53D94 X-CRM114-Status: GOOD ( 12.05 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org By disabling bottom halves in active kerne-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector. After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context. Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT. Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly") Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled") Signed-off-by: Andy Chiu --- Changelog v4: - new patch since v4 --- arch/riscv/include/asm/simd.h | 6 +++++- arch/riscv/kernel/kernel_mode_vector.c | 10 ++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index 3b603e47c5d8..7df5a976a80a 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -28,8 +28,12 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. + * + * Kernel-mode Vector temperarily disables bh. So we must not return + * true on irq_disabled(). Otherwise we would fail the lockdep check + * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); + return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 105147c7d2da..db0cf06f2abf 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -23,7 +23,10 @@ */ void get_cpu_vector_context(void) { - preempt_disable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_disable(); + else + preempt_disable(); WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != 0); riscv_v_ctx_cnt_add(RISCV_KERNEL_MODE_V); @@ -41,7 +44,10 @@ void put_cpu_vector_context(void) WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != RISCV_KERNEL_MODE_V); riscv_v_ctx_cnt_sub(RISCV_KERNEL_MODE_V); - preempt_enable(); + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) + local_bh_enable(); + else + preempt_enable(); } /* From patchwork Thu Dec 21 13:43:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 85BC2C35274 for ; Thu, 21 Dec 2023 13:44:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DV8xRL7tEjykz4cKWnFtTLNKKrx8DErQgcUfaLcD/9s=; b=m0jjXR4X/l85lt Wg9G4cj1RiYHKjzSl52UZMYW/zxbcZIACPS4C1Iy8iejIf3wGqZvcs4sA5anwtVQ6V+v1a/dL1CWd 9SN+Lq46qM1p4QuF3jwq4EWKxg4e494VxKSYirJ6qbtlZmft2SUWavqaveVVbMS5yVkaxe5YDySuH wMRwfg6nr1JuYgtQ5v8+P1o1DSWLzyORlQ6s2edMfNyVsv7u+l9wM/NvAQ0hmzj2xI4qWrBwKo4I0 CzyU1y311xg2fGRAj/bHXx5KcuNOpU2b1CpyW5pw/3XJk/usBLSahWpG019j2uhGgJdCHHl8eJHnI xwdBLbDxJT8ER5mYJQ8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLG-0031jk-2Z; Thu, 21 Dec 2023 13:44:02 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLD-0031gn-0N for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:01 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-1d3fc184b6dso4474755ad.2 for ; Thu, 21 Dec 2023 05:43:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166235; x=1703771035; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=AqcMhZ6hnnDAvLBGLmTurXkp9t0sE5KdvWpv8IRkAao=; b=FMwrxDSWnLab0bEoZUO8kQCARgAbG6Fw5qCcVwVxg9G5hWsNviGkvROEdqvVCG3QnD BjqWAilOIY8mEhUiq2Le6jPWOT0AT8pYkTnjB3aIysloh1QYo2/Lu1Hrb1o1G3RvNM/i f10iUBbPFJ8smk4M8eCpIzP9yZtlFAvlt8VOVcsK+KbSOv2Id7CVZGHqfcqDSUKSxUwl w4cEqaP4IWQQf69A0xV/HU3fq23/Wkzzz5Ijoyl28XwR1VNnQTuYFuAfL01Iz7NxRxVz heleuIOhfy+27C5Ftn7aXEvBOF4/ZCodKJwdhDNwWwfFLRzjZwfnncshEwqsN0NoIcAl I2Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166235; x=1703771035; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AqcMhZ6hnnDAvLBGLmTurXkp9t0sE5KdvWpv8IRkAao=; b=DCvDEFTXRER/vHvOGQt456h0y8s0mbCFiTIi6ECx0e0P3zfAJxCgS3O/wO9I+IBt8E ogMsfXPVvkqXu7qeLjDSgKxOWzpHSRMDp3iOtAtDAxX1zn5fqtpCIMpCK+zBdyXsZw1c P13A8izu6HVI7TPIlagBq7AIB2yrtYVc/salAFkV0uo3QjM97Xrf2ei3DgBpoRu9T1dt COq3ovJDXNeGVHdi7yqzWzhdM/zLJF7T4mOmHTd2gfpUnbYJ0IlzRYAbNuSPY2DKRebk ufGnH4QZ60aZfhNXaWqt9esf4xPcoUDZ27HHCgw0GZVKqhTGLJss4jN3pzLID4CO2ZjG 6/OQ== X-Gm-Message-State: AOJu0YxZerC9NX/DD+vzrumOOD/DpUKZkSN7o6fmy5CsfHb0F5TIpVBz 6NoRMbjl8Yokhb3uWTCUWvZ/KwfIN0wMk/o60D4S+88wwAtU2Gx9oNIXD7VU1zSWAs+efCZ5LZ9 f0oYVvOStHbETpKxlku06CpMwv9+q0SdPYcROf8v6E5HaIhorCWy9n8bpQXE1g9uLlwOVQvA0uV vbqEbWSwU5+dQZ X-Google-Smtp-Source: AGHT+IHMxBxtOLeATxuL0RHWmg09uEvKnnZrqSLjhd03HMW9WzMmyvjgdPYJm7idTGzyszEuuTP+og== X-Received: by 2002:a17:902:bb10:b0:1d3:ef79:4a2e with SMTP id im16-20020a170902bb1000b001d3ef794a2emr1317096plb.87.1703166234922; Thu, 21 Dec 2023 05:43:54 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.43.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:43:54 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Han-Kuan Chen , Andy Chiu , Albert Ou , Guo Ren , Sami Tolvanen , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Deepak Gupta , Andrew Jones , Conor Dooley , Heiko Stuebner Subject: [v7, 03/10] riscv: Add vector extension XOR implementation Date: Thu, 21 Dec 2023 13:43:10 +0000 Message-Id: <20231221134318.28105-4-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054359_155682_15B6D845 X-CRM114-Status: GOOD ( 15.12 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Greentime Hu This patch adds support for vector optimized XOR and it is tested in qemu. Co-developed-by: Han-Kuan Chen Signed-off-by: Han-Kuan Chen Signed-off-by: Greentime Hu Signed-off-by: Andy Chiu --- Changelog v7: - fix build warning message and use proper entry/exit macro for assembly. Drop Conor's A-b Changelog v2: - 's/rvv/vector/' (Conor) --- arch/riscv/include/asm/asm-prototypes.h | 14 +++++ arch/riscv/include/asm/xor.h | 68 +++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++ 4 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h index 36b955c762ba..b34b68a99855 100644 --- a/arch/riscv/include/asm/asm-prototypes.h +++ b/arch/riscv/include/asm/asm-prototypes.h @@ -9,6 +9,20 @@ long long __lshrti3(long long a, int b); long long __ashrti3(long long a, int b); long long __ashlti3(long long a, int b); +void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2); +void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3); +void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4); +void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5); #define DECLARE_DO_ERROR_INFO(name) asmlinkage void name(struct pt_regs *regs) diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h new file mode 100644 index 000000000000..96011861e46b --- /dev/null +++ b/arch/riscv/include/asm/xor.h @@ -0,0 +1,68 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ + +#include +#include +#ifdef CONFIG_RISCV_ISA_V +#include +#include +#include + +static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +{ + kernel_vector_begin(); + xor_regs_2_(bytes, p1, p2); + kernel_vector_end(); +} + +static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +{ + kernel_vector_begin(); + xor_regs_3_(bytes, p1, p2, p3); + kernel_vector_end(); +} + +static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) +{ + kernel_vector_begin(); + xor_regs_4_(bytes, p1, p2, p3, p4); + kernel_vector_end(); +} + +static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5) +{ + kernel_vector_begin(); + xor_regs_5_(bytes, p1, p2, p3, p4, p5); + kernel_vector_end(); +} + +static struct xor_block_template xor_block_rvv = { + .name = "rvv", + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 26cb2502ecf8..494f9cd1a00c 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o diff --git a/arch/riscv/lib/xor.S b/arch/riscv/lib/xor.S new file mode 100644 index 000000000000..b28f2430e52f --- /dev/null +++ b/arch/riscv/lib/xor.S @@ -0,0 +1,81 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* + * Copyright (C) 2021 SiFive + */ +#include +#include +#include + +SYM_FUNC_START(xor_regs_2_) + vsetvli a3, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a3 + vxor.vv v16, v0, v8 + add a2, a2, a3 + vse8.v v16, (a1) + add a1, a1, a3 + bnez a0, xor_regs_2_ + ret +SYM_FUNC_END(xor_regs_2_) +EXPORT_SYMBOL(xor_regs_2_) + +SYM_FUNC_START(xor_regs_3_) + vsetvli a4, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a4 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a4 + vxor.vv v16, v0, v16 + add a3, a3, a4 + vse8.v v16, (a1) + add a1, a1, a4 + bnez a0, xor_regs_3_ + ret +SYM_FUNC_END(xor_regs_3_) +EXPORT_SYMBOL(xor_regs_3_) + +SYM_FUNC_START(xor_regs_4_) + vsetvli a5, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a5 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a5 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a5 + vxor.vv v16, v0, v24 + add a4, a4, a5 + vse8.v v16, (a1) + add a1, a1, a5 + bnez a0, xor_regs_4_ + ret +SYM_FUNC_END(xor_regs_4_) +EXPORT_SYMBOL(xor_regs_4_) + +SYM_FUNC_START(xor_regs_5_) + vsetvli a6, a0, e8, m8, ta, ma + vle8.v v0, (a1) + vle8.v v8, (a2) + sub a0, a0, a6 + vxor.vv v0, v0, v8 + vle8.v v16, (a3) + add a2, a2, a6 + vxor.vv v0, v0, v16 + vle8.v v24, (a4) + add a3, a3, a6 + vxor.vv v0, v0, v24 + vle8.v v8, (a5) + add a4, a4, a6 + vxor.vv v16, v0, v8 + add a5, a5, a6 + vse8.v v16, (a1) + add a1, a1, a6 + bnez a0, xor_regs_5_ + ret +SYM_FUNC_END(xor_regs_5_) +EXPORT_SYMBOL(xor_regs_5_) From patchwork Thu Dec 21 13:43:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 40ED0C35274 for ; Thu, 21 Dec 2023 13:44:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bvmWtn+omo1hkN1LE5+OU9uMXkvdc3PJqb0JzaZ8CtA=; b=M5HCkKHdOIlBub GExgjSxDiXnTshvy2UxMUnYYE/zll8emKsmHjpE2uselKm9L9kfXg8u01QL0An6TVBH9Cfb7OtIrA 3Rl9lvM2tNs2m3MZakAi1fPiLvo5tmB866PBLvfXsHV3j6Lb9f5TWrRarBLzmNpQ9rQaTweZnDJed 6D3uweiHay3fWYDL+pcA1Xio3b73TQYsaAnbUSAX7RmFKbkvDkZDrBX0ozJ3+A9Qy4kM+DcG9l/MB XwH7TQbeEIp2cf/hhi/W8Y/quR6qwejFIVMkjB13J16tnyExdaHFVhN05mWcWnCFHZmSJ1cEC7abi CMHnM5cgv5E0wF8sotNQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLV-0031qd-0B; Thu, 21 Dec 2023 13:44:17 +0000 Received: from mail-pl1-x629.google.com ([2607:f8b0:4864:20::629]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLQ-0031nC-28 for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:14 +0000 Received: by mail-pl1-x629.google.com with SMTP id d9443c01a7336-1d2f1cecf89so4537345ad.1 for ; Thu, 21 Dec 2023 05:44:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166249; x=1703771049; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3jKdEEX1QxKneuBo1QXMeqJj0gX6w38gsAihdX7GUMI=; b=hHDVNAOfnCNcgGFMKCVTc0ZZIsnrV1J1M3fMCA0vPjNVBi8Sn2oZ8zGwhwgF/RFe8a VuPQbfvRkX3o8uPcX4nzW6phszXBpqi/bxxSEf4wQm8QJl8CpEMbMtxR6YWAxnxZ8Cjt l+3rr7v4F/VHttsKC45lhgNiRW0pKyZMi4Y0dvuQXWkS7wN4QK9Vszbeb3uLn4tamjni l4KvJRoSDn4Ql+ItY2wJWtxHRPMmAZPQyo5d/b9pyjy6mh7TT/wEirPwGfp+WgB+tksD ikXPwrj7HODqn39w80UejsMl84NEDL2C9jyc8F4JGowWl4Qiamc8RBWd+hBF6bjseuDL eC9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166249; x=1703771049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3jKdEEX1QxKneuBo1QXMeqJj0gX6w38gsAihdX7GUMI=; b=ZXHES+xtTjiIDmuw0fkgPQ8rdQ5+qNv5GBToy4vyCJ+KijbGMrEQTJLrPxDwh7/mLU J5u1FeKAvWllFxi8PiGHgYFXhLB+XbJ6CAj00QxX/zikCMheatLe6EL5mexbZao1ZFOi uBdGCWPUHfNiiBg0J3RGF49LAB9FNeVt8/z3NwBUbgJlK/hLfih+xUq7heOXmNeP6Gj5 TEaAdAN5iOkkVKK6f+pesO0dYOBshHqNfEI9e1ANwu0QtAqkVo37HHJ2tCQ2uoPEQ7Xy TgqrTHo/sViA4YhTaoYKCXzkovjX389qTCqqvuw2MooaYpUmhVL25Kd0Lr9/uXWyK6a+ m4LQ== X-Gm-Message-State: AOJu0Yx+Pq93pUaJnMXU2sb2SR1RGLwCi5B1h/yHVCj2jiFkk9dHZeFo yoQzXHtezXVh4Q8hYncqNJUtBhC989iO6eDN5W4AyjnqkZZEFTneUqLzyN3W1Um58Bny7d4riQJ 11n7Y6lV93b/y4m1RhSSQx1gYzOD6FHpm991VTZ3fnqj4Pt30nPfm7AtkorBZl71Db5eObU9qFr wglWI+Ulo5NNMm X-Google-Smtp-Source: AGHT+IFrvo5hkVAGwDo1kmMoW2jSWCzLED8JXw2V6a5RRi2wRpkk5nczo8KgJ/SB05cwptXqXadwLg== X-Received: by 2002:a17:902:c451:b0:1d3:e9f5:d3aa with SMTP id m17-20020a170902c45100b001d3e9f5d3aamr1834914plm.45.1703166248495; Thu, 21 Dec 2023 05:44:08 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.44.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:44:07 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Oleg Nesterov , Conor Dooley , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Guo Ren , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Sami Tolvanen , Jisheng Zhang , Deepak Gupta , Vincent Chen , Heiko Stuebner , Xiao Wang , Haorong Lu , Joel Granados Subject: [v7, 04/10] riscv: sched: defer restoring Vector context for user Date: Thu, 21 Dec 2023 13:43:11 +0000 Message-Id: <20231221134318.28105-5-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054412_703562_1CB9E9C3 X-CRM114-Status: GOOD ( 21.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v4: - fix typos and re-add Conor's A-b. Changelog v3: - Guard {get,put}_cpu_vector_context between vstate_* operation and explain it in the commit msg. - Drop R-b from Björn and A-b from Conor. Changelog v2: - rename and add comment for the new thread flag (Conor) --- arch/riscv/include/asm/entry-common.h | 17 +++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/ptrace.c | 5 ++++- arch/riscv/kernel/signal.c | 5 ++++- arch/riscv/kernel/vector.c | 2 +- 8 files changed, 41 insertions(+), 5 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 7ab5e34318c8..6361a8488642 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -4,6 +4,23 @@ #define _ASM_RISCV_ENTRY_COMMON_H #include +#include +#include + +static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, + unsigned long ti_work) +{ + if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { + clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE); + /* + * We are already called with irq disabled, so go without + * keeping track of vector_context_busy. + */ + riscv_v_vstate_restore(current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare void handle_page_fault(struct pt_regs *regs); void handle_break(struct pt_regs *regs); diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 574779900bfb..1047a97ddbc8 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -103,12 +103,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); #define TIF_NOTIFY_SIGNAL 9 /* signal notifications exist */ #define TIF_UPROBE 10 /* uprobe breakpoint or singlestep */ #define TIF_32BIT 11 /* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12 /* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) #define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) #define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 6254830c0668..e706613aae2c 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -205,6 +205,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore(struct task_struct *task, + struct pt_regs *regs) +{ + if ((regs->status & SR_VS) != SR_VS_OFF) { + set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE); + riscv_v_vstate_on(regs); + } +} + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { @@ -212,7 +221,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs = task_pt_regs(prev); riscv_v_vstate_save(prev, regs); - riscv_v_vstate_restore(next, task_pt_regs(next)); + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index db0cf06f2abf..3f1d67109e5a 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -92,7 +92,7 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_restore(current, task_pt_regs(current)); + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_disable(); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 5c4dcf518684..58127b1c6c71 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -171,6 +171,7 @@ void flush_thread(void) riscv_v_vstate_off(task_pt_regs(current)); kfree(current->thread.vstate.datap); memset(¤t->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE); #endif } @@ -187,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 2afe460de16a..7b93bcbdf9fa 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -99,8 +99,11 @@ static int riscv_vr_get(struct task_struct *target, * Ensure the vector registers have been saved to the memory before * copying them to membuf. */ - if (target == current) + if (target == current) { + get_cpu_vector_context(); riscv_v_vstate_save(current, task_pt_regs(current)); + put_cpu_vector_context(); + } ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl; diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index 88b6220b2608..aca4a12c8416 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -86,7 +86,10 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) /* datap is designed to be 16 byte aligned for better performance */ WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); + get_cpu_vector_context(); riscv_v_vstate_save(current, regs); + put_cpu_vector_context(); + /* Copy everything of vstate but datap. */ err = __copy_to_user(&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state, datap)); @@ -134,7 +137,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec) if (unlikely(err)) return err; - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return err; } diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 578b6292487e..66e8c6ab09d2 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) return true; } riscv_v_vstate_on(regs); - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_set_restore(current, regs); return true; } From patchwork Thu Dec 21 13:43:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6EA06C35274 for ; Thu, 21 Dec 2023 13:44:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qZe6AD/YWVqXQ/gKrX7aata/mRZ1UQ4iHP0SjhcapQI=; b=IJqPJjlO66bbLM r+rKONHDWSd0t2gUodj9ofJhaEI0Rq1qhwA6CGCqzJg8egEleeIgKd4/PrB9Eeu24TKelZ+L2Wsk7 UGdl1gIVokezeGw0Vblyg5s6CBLDIEPuwY5zotRa0T9UySri1bRlcZrOEPwh/0rL6XJz1UbaMJhEw Avx1NhcwYQxGhYLOstE9BNnEqauiHbZpS4mp+NFTysZ5dssj58lsHl3lck4ZNZztZk50RBvPxVE0r 3M2rdxji/kXXLyvSKcx/+L1pCeLYfqP/wPsq3FKHmqcGid7g6ZAo2nfgeIALXkrJza04K0asZx9jO WINtR1pCFVZ8GjkU2hkQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLd-0031wH-16; Thu, 21 Dec 2023 13:44:25 +0000 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLa-0031s4-1Q for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:24 +0000 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1d3dee5f534so13881585ad.1 for ; Thu, 21 Dec 2023 05:44:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166258; x=1703771058; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=+5y5mXY/GjTuMPoMs/eCuiabcFqtKPrJHseD8GV4oj8=; b=ltkdJklRbfg3XTgVnUIqoQQn+MxO68LGPSHZFPUH5nAwJYGIl8qh7Tsg4RGaMFn7VR yNRsI6yxDRTbZoVyh9LgywTf0/0EyHowri1g021PmoR6SYu5LLoD7FKpldAwFhm4DvXe TFA252/PbP5Nsp7moUaMW7ZS/QkBICnpO6QwtkkfY5HPSb/uZimQh6rVOyQOHrBnsSeM SVQz6M0ghEYFXILKrLMOk3hUhgAhhhPDw1r74bdQiJGJCerSJt0QF8pXmnmCSquM6Hfp MP3F0UOg6PwR95xKvjy3KaQxtsYQPgiqIHVwnIOM+bZmR9bdsvKArUcq1XcWVC5FLR8h aMRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166258; x=1703771058; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+5y5mXY/GjTuMPoMs/eCuiabcFqtKPrJHseD8GV4oj8=; b=DZG2riy4o3Qipif+qVhJE1/WUeW1aVzfZq3y55fGFZHNdvflwxx8HxJt9bQC6GAlDY vKUGPG5uLaAtuADmxFus0PCez/StfD2xNjVuGYSd9TYC6AIkVZz+hj2I4Ua6BVPEbn4s sMF8NTSgvkRDGY9X0C/fWpqL0ROcEFH6/s7mdQRWqGlEO36e/TgQKz/AF0z7UDj5PQCB 1kFCzjz6f1fSILmjK4QKFSx1Pf/NmDGJS3klpUBszw/VVSg4Wi0FDmW0uzZnat/6+kBy jcu6i42OxjCNntBCRN8Q+7xjYwbkFp3wqWYn5nvvhUPC1PHpN54ldOwdcxIBRud16Oh6 cwsw== X-Gm-Message-State: AOJu0YxvwOk9kt1W7wsPSzo1huYadz9DgWN5qPLr2swxoXFvUoSoFD4X V2Drnsrus7QGyepmwB8LzmMR6Iq8ZclW3bhveJLTD6rjY+sGjyhWs4XaJfWYN6ErhLVR/m4XW56 2iQnNESgphuR3XSA/Flat/bkGYC+kjX3IJFBpu0LTSn5ENHXj4mDIqcbmzYUkyRfAiyxdkEG9Gp cb3H360NNZpLiY X-Google-Smtp-Source: AGHT+IGNiZA6N0QSyHhAb79lTcUyla5b2o0CNqobBRMvxiMOC18DjJyk2UFTevHQYb5awhb5BDS+PQ== X-Received: by 2002:a17:902:d385:b0:1d3:ff24:b3bb with SMTP id e5-20020a170902d38500b001d3ff24b3bbmr860957pld.49.1703166258203; Thu, 21 Dec 2023 05:44:18 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:44:17 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Conor Dooley , Han-Kuan Chen , Andrew Jones , Heiko Stuebner , Aurelien Jarno , Alexandre Ghiti , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= Subject: [v7, 05/10] riscv: lib: vectorize copy_to_user/copy_from_user Date: Thu, 21 Dec 2023 13:43:12 +0000 Message-Id: <20231221134318.28105-6-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054422_479775_5289ABE6 X-CRM114-Status: GOOD ( 21.65 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch utilizes Vector to perform copy_to_user/copy_from_user. If Vector is available and the size of copy is large enough for Vector to perform better than scalar, then direct the kernel to do Vector copies for userspace. Though the best programming practice for users is to reduce the copy, this provides a faster variant when copies are inevitable. The optimal size for using Vector, copy_to_user_thres, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it. The exception fixup code of the __asm_vector_usercopy must fallback to the scalar one because accessing user pages might fault, and must be sleepable. Current kernel-mode Vector does not allow tasks to be preemptible, so we must disactivate Vector and perform a scalar fallback in such case. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Signed-off-by: Andy Chiu --- Changelog v6: - Add a kconfig entry to configure threshold values (Charlie) - Refine assembly code (Charlie) Changelog v4: - new patch since v4 --- arch/riscv/Kconfig | 8 +++++ arch/riscv/lib/Makefile | 2 ++ arch/riscv/lib/riscv_v_helpers.c | 38 ++++++++++++++++++++++++ arch/riscv/lib/uaccess.S | 10 +++++++ arch/riscv/lib/uaccess_vector.S | 50 ++++++++++++++++++++++++++++++++ 5 files changed, 108 insertions(+) create mode 100644 arch/riscv/lib/riscv_v_helpers.c create mode 100644 arch/riscv/lib/uaccess_vector.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 95a2a06acc6a..3c5ba05e8a2d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -525,6 +525,14 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_UCOPY_THRESHOLD + int "Threshold size for vectorized user copies" + depends on RISCV_ISA_V + default 768 + help + Prefer using vectorized copy_to_user()/copy_from_user() when the + workload size exceeds this value. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 494f9cd1a00c..1fe8d797e0f2 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -12,3 +12,5 @@ lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o +lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o +lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o diff --git a/arch/riscv/lib/riscv_v_helpers.c b/arch/riscv/lib/riscv_v_helpers.c new file mode 100644 index 000000000000..139e5de1b793 --- /dev/null +++ b/arch/riscv/lib/riscv_v_helpers.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (C) 2023 SiFive + * Author: Andy Chiu + */ +#include +#include + +#include +#include + +size_t riscv_v_usercopy_threshold = CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD; +int __asm_vector_usercopy(void *dst, void *src, size_t n); +int fallback_scalar_usercopy(void *dst, void *src, size_t n); +asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n) +{ + size_t remain, copied; + + /* skip has_vector() check because it has been done by the asm */ + if (!may_use_simd()) + goto fallback; + + kernel_vector_begin(); + remain = __asm_vector_usercopy(dst, src, n); + kernel_vector_end(); + + if (remain) { + copied = n - remain; + dst += copied; + src += copied; + goto fallback; + } + + return remain; + +fallback: + return fallback_scalar_usercopy(dst, src, n); +} diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index 3ab438f30d13..a1e4a3c42925 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -3,6 +3,8 @@ #include #include #include +#include +#include .macro fixup op reg addr lbl 100: @@ -11,6 +13,13 @@ .endm SYM_FUNC_START(__asm_copy_to_user) +#ifdef CONFIG_RISCV_ISA_V + ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_v, CONFIG_RISCV_ISA_V) + REG_L t0, riscv_v_usercopy_threshold + bltu a2, t0, fallback_scalar_usercopy + tail enter_vector_usercopy +#endif +SYM_FUNC_START(fallback_scalar_usercopy) /* Enable access to user memory */ li t6, SR_SUM @@ -181,6 +190,7 @@ SYM_FUNC_START(__asm_copy_to_user) sub a0, t5, a0 ret SYM_FUNC_END(__asm_copy_to_user) +SYM_FUNC_END(fallback_scalar_usercopy) EXPORT_SYMBOL(__asm_copy_to_user) SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user) EXPORT_SYMBOL(__asm_copy_from_user) diff --git a/arch/riscv/lib/uaccess_vector.S b/arch/riscv/lib/uaccess_vector.S new file mode 100644 index 000000000000..7bd96cee39e4 --- /dev/null +++ b/arch/riscv/lib/uaccess_vector.S @@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include +#include +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + + .macro fixup op reg addr lbl +100: + \op \reg, \addr + _asm_extable 100b, \lbl + .endm + +SYM_FUNC_START(__asm_vector_usercopy) + /* Enable access to user memory */ + li t6, SR_SUM + csrs CSR_STATUS, t6 + +loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + fixup vle8.v vData, (pSrc), 10f + fixup vse8.v vData, (pDst), 10f + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + add pDst, pDst, iVL + bnez iNum, loop + +.Lout_copy_user: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + li a0, 0 + ret + + /* Exception fixup code */ +10: + /* Disable access to user memory */ + csrc CSR_STATUS, t6 + mv a0, iNum + ret +SYM_FUNC_END(__asm_vector_usercopy) From patchwork Thu Dec 21 13:43:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6DB4DC35274 for ; Thu, 21 Dec 2023 13:44:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dzhyEtGaXaTm7QHCx6aU+nrKbpEhoKsy5k5jmhcl+CA=; b=IzM5HIg6jHu6K0 zoVjOR1FDhmZNPk45Oicyrv67CWK9hhnWDW53CzhZcNuIbwdKBBUsKSZIHt4fg7mJxA6A/CNhDU1N EYAF/9LghYgXz/ai+nHL7WQ2/+en7MDmfh4H+wDPlCuJsMRqU4b8o8HhzgNpzMljt60uAKs7NKfYn kVIqCqXq7MAMuZvEdmWM6WxFFsjK2PsG+r65QeqkZemF35xXmBdIXxX/HzAFNpReN9szlVW19A73A CSI6MxWRMatwkQK1IuwWjQF6sBIMJMQJCbIHEwws12/UZL1wQ31hM+9voD8IkBz9d1vZT7Fms5KE3 NLboL2GP9SZUK7GM6Dng==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLn-00322A-29; Thu, 21 Dec 2023 13:44:35 +0000 Received: from mail-pj1-x1030.google.com ([2607:f8b0:4864:20::1030]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLk-0031zi-07 for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:34 +0000 Received: by mail-pj1-x1030.google.com with SMTP id 98e67ed59e1d1-28bfb64e746so173868a91.0 for ; Thu, 21 Dec 2023 05:44:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166270; x=1703771070; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=l1jZcVBDRaflxDGHuWgpJUzAXe4cvvRKG5aPkIKHmck=; b=liyN+l9PS4ZMC32sdWygRzQXsVx9HiO8Oj7zavEKzqKBnIqRqdV/5AFJr8eOJ8FQDw ai7UWjY3403hY2A1s/7veHRKnd6M3IrCUQpSNlBkPQBPyYSuGx/PQFu75a9nF/90lhZa it5kwjH4mQBRNn7hJiqsdmC55q46nxy3Na4OV7g9ZUMl0Tmk2xKDoCOIb0Z8ncSSaJrp 9YHfoSzqLCuMT+Di1ITlRrgxOXqLOVGiwjsvIPwESXAPq2+Urz2x0w1tm2aJgrkJguFi nVEgwnQkA0T1ZN1aO+vdJ2iXf6i7rRhNwxHEF8mNwN8dr/N/GNtTwgFbY4NhV+uOvXBM 6Yog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166270; x=1703771070; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=l1jZcVBDRaflxDGHuWgpJUzAXe4cvvRKG5aPkIKHmck=; b=NTCAHUwOgChhzzDF44jeJbWMMYeCalmqqAasD/K5vEs4ovMGAp0z+49YUu64dVl9Eg E6NIU5fWKQIYwubOpCNHBnNwE7i7HSZL7F2sgbwAhy3SiInUOcAr4dB9p7yifzhALpOP G5hmyY+kaPrZi3PGeHxlGov59Q9GbGCfIcKPt9L/O5ufnE3UFSiBVp5i8CkjtEd9rtWh brs2CyV2a8qta+U6idHH4MJpX7MT/tHU1TVpV14skl7XR41SLCVdt20oVRyc6oivKJHh tVqw7em3xsGLIZ8YnIFkkhKcTQp2nPfRnmn3YYIYl244lv5EgsDTpuENpbdBvVZR8r2H dSIA== X-Gm-Message-State: AOJu0YyBATxIi82pCfluvTYl8S50k7vDMVUWqO2i4p4PaCZdhOQyL7eq LdVbCljU3D1NM4DvDZwOAzvFZ+1AHj2JXPDSbmM3d9dhAUsGrl6/YKxA6NdZ96H62SZb8QGKnRN KA6dNuURPUokefKi5N3JdoGriwR4Aj3+jYj0Zybz+Z3pBjIf/d6gD1AKXP1NA9qFc4kc26mtURk YBh/nIVo6Q7m69 X-Google-Smtp-Source: AGHT+IFTLCrwrXYM5pIn3dAgoGLhtzxTtp+vg+N4GYjR/fut4cytVyvkBT91N5pBvdfSxohbBBaQyg== X-Received: by 2002:a17:90a:d710:b0:28b:8bbb:32cd with SMTP id y16-20020a17090ad71000b0028b8bbb32cdmr3870198pju.78.1703166269927; Thu, 21 Dec 2023 05:44:29 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:44:29 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Kees Cook , Conor Dooley , Andrew Jones , Han-Kuan Chen , Heiko Stuebner Subject: [v7, 06/10] riscv: lib: add vectorized mem* routines Date: Thu, 21 Dec 2023 13:43:13 +0000 Message-Id: <20231221134318.28105-7-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054432_077646_3FAA785E X-CRM114-Status: GOOD ( 16.33 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Provide vectorized memcpy/memset/memmove to accelerate common memory operations. Also, group them into V_OPT_TEMPLATE3 macro because their setup/tear-down and fallback logics are the same. The optimal size for the kernel to preference Vector over scalar, riscv_v_mem*_threshold, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it. The original implementation of Vector operations comes from https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel. Signed-off-by: Andy Chiu --- Changelog v7: - add __NO_FORTIFY to prevent conflicting function declaration with macro for mem* functions. Changelog v6: - provide kconfig to set threshold for vectorized functions (Charlie) - rename *thres to *threshold (Charlie) Changelog v4: - new patch since v4 --- arch/riscv/Kconfig | 24 ++++++++++++++++ arch/riscv/lib/Makefile | 3 ++ arch/riscv/lib/memcpy_vector.S | 29 +++++++++++++++++++ arch/riscv/lib/memmove_vector.S | 49 ++++++++++++++++++++++++++++++++ arch/riscv/lib/memset_vector.S | 33 +++++++++++++++++++++ arch/riscv/lib/riscv_v_helpers.c | 26 +++++++++++++++++ 6 files changed, 164 insertions(+) create mode 100644 arch/riscv/lib/memcpy_vector.S create mode 100644 arch/riscv/lib/memmove_vector.S create mode 100644 arch/riscv/lib/memset_vector.S diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3c5ba05e8a2d..cba53dcc2ae0 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -533,6 +533,30 @@ config RISCV_ISA_V_UCOPY_THRESHOLD Prefer using vectorized copy_to_user()/copy_from_user() when the workload size exceeds this value. +config RISCV_ISA_V_MEMSET_THRESHOLD + int "Threshold size for vectorized memset()" + depends on RISCV_ISA_V + default 1280 + help + Prefer using vectorized memset() when the workload size exceeds this + value. + +config RISCV_ISA_V_MEMCPY_THRESHOLD + int "Threshold size for vectorized memcpy()" + depends on RISCV_ISA_V + default 768 + help + Prefer using vectorized memcpy() when the workload size exceeds this + value. + +config RISCV_ISA_V_MEMMOVE_THRESHOLD + int "Threshold size for vectorized memmove()" + depends on RISCV_ISA_V + default 512 + help + Prefer using vectorized memmove() when the workload size exceeds this + value. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile index 1fe8d797e0f2..3111863afd2e 100644 --- a/arch/riscv/lib/Makefile +++ b/arch/riscv/lib/Makefile @@ -14,3 +14,6 @@ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o lib-$(CONFIG_RISCV_ISA_V) += xor.o lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o +lib-$(CONFIG_RISCV_ISA_V) += memset_vector.o +lib-$(CONFIG_RISCV_ISA_V) += memcpy_vector.o +lib-$(CONFIG_RISCV_ISA_V) += memmove_vector.o diff --git a/arch/riscv/lib/memcpy_vector.S b/arch/riscv/lib/memcpy_vector.S new file mode 100644 index 000000000000..4176b6e0a53c --- /dev/null +++ b/arch/riscv/lib/memcpy_vector.S @@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 +#define pDstPtr a4 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + + +/* void *memcpy(void *, const void *, size_t) */ +SYM_FUNC_START(__asm_memcpy_vector) + mv pDstPtr, pDst +loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + vle8.v vData, (pSrc) + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + vse8.v vData, (pDstPtr) + add pDstPtr, pDstPtr, iVL + bnez iNum, loop + ret +SYM_FUNC_END(__asm_memcpy_vector) diff --git a/arch/riscv/lib/memmove_vector.S b/arch/riscv/lib/memmove_vector.S new file mode 100644 index 000000000000..4cea9d244dc9 --- /dev/null +++ b/arch/riscv/lib/memmove_vector.S @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include +#include + +#define pDst a0 +#define pSrc a1 +#define iNum a2 + +#define iVL a3 +#define pDstPtr a4 +#define pSrcBackwardPtr a5 +#define pDstBackwardPtr a6 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + +SYM_FUNC_START(__asm_memmove_vector) + + mv pDstPtr, pDst + + bgeu pSrc, pDst, forward_copy_loop + add pSrcBackwardPtr, pSrc, iNum + add pDstBackwardPtr, pDst, iNum + bltu pDst, pSrcBackwardPtr, backward_copy_loop + +forward_copy_loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + + vle8.v vData, (pSrc) + sub iNum, iNum, iVL + add pSrc, pSrc, iVL + vse8.v vData, (pDstPtr) + add pDstPtr, pDstPtr, iVL + + bnez iNum, forward_copy_loop + ret + +backward_copy_loop: + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + + sub pSrcBackwardPtr, pSrcBackwardPtr, iVL + vle8.v vData, (pSrcBackwardPtr) + sub iNum, iNum, iVL + sub pDstBackwardPtr, pDstBackwardPtr, iVL + vse8.v vData, (pDstBackwardPtr) + bnez iNum, backward_copy_loop + ret + +SYM_FUNC_END(__asm_memmove_vector) diff --git a/arch/riscv/lib/memset_vector.S b/arch/riscv/lib/memset_vector.S new file mode 100644 index 000000000000..4611feed72ac --- /dev/null +++ b/arch/riscv/lib/memset_vector.S @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#include +#include + +#define pDst a0 +#define iValue a1 +#define iNum a2 + +#define iVL a3 +#define iTemp a4 +#define pDstPtr a5 + +#define ELEM_LMUL_SETTING m8 +#define vData v0 + +/* void *memset(void *, int, size_t) */ +SYM_FUNC_START(__asm_memset_vector) + + mv pDstPtr, pDst + + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + vmv.v.x vData, iValue + +loop: + vse8.v vData, (pDstPtr) + sub iNum, iNum, iVL + add pDstPtr, pDstPtr, iVL + vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma + bnez iNum, loop + + ret + +SYM_FUNC_END(__asm_memset_vector) diff --git a/arch/riscv/lib/riscv_v_helpers.c b/arch/riscv/lib/riscv_v_helpers.c index 139e5de1b793..28467d737faf 100644 --- a/arch/riscv/lib/riscv_v_helpers.c +++ b/arch/riscv/lib/riscv_v_helpers.c @@ -3,9 +3,13 @@ * Copyright (C) 2023 SiFive * Author: Andy Chiu */ +#ifndef __NO_FORTIFY +# define __NO_FORTIFY +#endif #include #include +#include #include #include @@ -36,3 +40,25 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n) fallback: return fallback_scalar_usercopy(dst, src, n); } + +#define V_OPT_TEMPLATE3(prefix, type_r, type_0, type_1) \ +extern type_r __asm_##prefix##_vector(type_0, type_1, size_t n); \ +type_r prefix(type_0 a0, type_1 a1, size_t n) \ +{ \ + type_r ret; \ + if (has_vector() && may_use_simd() && \ + n > riscv_v_##prefix##_threshold) { \ + kernel_vector_begin(); \ + ret = __asm_##prefix##_vector(a0, a1, n); \ + kernel_vector_end(); \ + return ret; \ + } \ + return __##prefix(a0, a1, n); \ +} + +static size_t riscv_v_memset_threshold = CONFIG_RISCV_ISA_V_MEMSET_THRESHOLD; +V_OPT_TEMPLATE3(memset, void *, void*, int) +static size_t riscv_v_memcpy_threshold = CONFIG_RISCV_ISA_V_MEMCPY_THRESHOLD; +V_OPT_TEMPLATE3(memcpy, void *, void*, const void *) +static size_t riscv_v_memmove_threshold = CONFIG_RISCV_ISA_V_MEMMOVE_THRESHOLD; +V_OPT_TEMPLATE3(memmove, void *, void*, const void *) From patchwork Thu Dec 21 13:43:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id ABA62C35274 for ; Thu, 21 Dec 2023 13:44:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pHgVDHjIhi7iHX57jDgV7aUCenbUnDmKM359swtKcLI=; b=NB0cv3IZnFUKRK n0HEEvyZhmsAl25YuU8umJIA1Q7Az/reydbs44bH2Z35EfWwZsbUq3bPGYCrPnevMd1S5lqNFOQ5P Krhv2PgfdPImMgzIcO7JJ+Xis/h1caqlXcuWjM+Aim5G1MQF4SajFcmUhLZv+b7TbgxfM/KQl136J NDiy9uG8mpHmn3MHYFezXa7gqlP+Xf3M7nyO7hikP1m2K0kfuaXWkrQ3NBHRP+2MA0nc7mIxvL1B4 Zml3megQ+cHB/qJDk0mHGKpZXQW0kFTney0aU32d/NHd2FeeSF4YMTzMo7hL0fOXsEbxbp+DAQIP5 obVrttt7G3WmsncPxAmg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLw-00327Y-2F; Thu, 21 Dec 2023 13:44:44 +0000 Received: from mail-pl1-x62d.google.com ([2607:f8b0:4864:20::62d]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJLt-003254-1f for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:43 +0000 Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1d3536cd414so6753075ad.2 for ; Thu, 21 Dec 2023 05:44:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166280; x=1703771080; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0PStOdX0bJ0JRt1kGvqh4Fd4K0oqMPuH8sOnQ/vCigs=; b=A9H3EZwFBiKrlbVBq+z/d3fo8951cB3yZa5Xft3QHfY7FTdfcK7j0kt42Qw5iZUVRM lNI+wQ5sQmc+5ywl2TFSiUrfwT2uJI5oboSvugzXZg/oRPTQFXqdrzLZmnHIcLtPHKvQ CF5wk4iAU4UckxnhTbmCw9ibcl9P6NsWd3+2JLZcsNZRVyts70CwV9fxdMxBXZsf/xv7 U/JX3IZ2TpURaneG5QMfuAH0CkaDC2248fzDQED3zgomIverwoUG5Yjw80hWUAv8tJpJ LPmmX6wO3alX8s7j/C6zF0NqP/4cP+zA05lJT/jYeNphr4pH0okYloqcVwLn1xr9Deq3 MSNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166280; x=1703771080; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0PStOdX0bJ0JRt1kGvqh4Fd4K0oqMPuH8sOnQ/vCigs=; b=sjmP+8Da6Q/BT6P3WKUOGCt6gmOXqDIqzCpk+kz/ksQW4IWR2KCAVo2Vw8kyKcQLsw 2cZHUn9wWWyTNQ/Wlae7ZOiBGsZXXpU/uvv0h+BF1fUWvbjSaz2A3Bb4BaJLUVqD+d9w zjhnuKGGY96Pb3xAasSbiC+mFR3ZWA7pS5WNO80hjejipQeVhR5uJ0VEFHGNee6w9LMx vef90IqNAFs1lwuOExEhuGKgqyWOsvjdrAZ12BBovpmOntFi42aqIYdl/IEcMe2Dud1L IZPb12Kk3xzQg1a2OK1VvnQAqMmhhKkS++ENrmxR/ZRtVJnsuCOklvib474zYO0o6LLx P1gQ== X-Gm-Message-State: AOJu0YxefAfvkMInjUf3/kVm6sC/wWn0dIEKq7lJbo005llIKL4FMtuM P4L519ceF1sn3XIkTa2nM7B8SLG2R94wxsw74e76HSbPnVpuCEPKnOLtA0I87WrSsNTYDEAxUzO 1pyErWkLhC5EKiPzg6SlJEAfhsvSlQVhlfvRuj+RSfqJAd++Yw+reglgedjlKsLxZvPYIqPo1cg wyA+P0ovOJE838 X-Google-Smtp-Source: AGHT+IEkZt8VDcEGMVh1chITcWavhfZDrRiMNvK4lYYlh9afNQNDZcoqpMQKA7D9FUzN7bAPwZIkGQ== X-Received: by 2002:a17:902:be17:b0:1d0:9a64:e511 with SMTP id r23-20020a170902be1700b001d09a64e511mr19093642pls.73.1703166279918; Thu, 21 Dec 2023 05:44:39 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.44.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:44:39 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Oleg Nesterov , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Conor Dooley , Guo Ren , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Vincent Chen , Heiko Stuebner , Xiao Wang , Haorong Lu , Mathis Salmen Subject: [v7, 07/10] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}() Date: Thu, 21 Dec 2023 13:43:14 +0000 Message-Id: <20231221134318.28105-8-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054441_559497_5C1464E5 X-CRM114-Status: GOOD ( 11.65 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused. Signed-off-by: Andy Chiu Acked-by: Conor Dooley --- Changelog v6: - re-added for v6 Changelog v3: - save V context after get_cpu_vector_context Changelog v2: - fix build fail that get caught on this patch (Conor) --- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-) diff --git a/arch/riscv/include/asm/entry-common.h b/arch/riscv/include/asm/entry-common.h index 6361a8488642..08fe8cdbf33e 100644 --- a/arch/riscv/include/asm/entry-common.h +++ b/arch/riscv/include/asm/entry-common.h @@ -16,7 +16,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, * We are already called with irq disabled, so go without * keeping track of vector_context_busy. */ - riscv_v_vstate_restore(current, regs); + riscv_v_vstate_restore(¤t->thread.vstate, regs); } } diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index e706613aae2c..c5a83c277583 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -183,23 +183,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs) __riscv_v_vstate_dirty(regs); } -static inline void riscv_v_vstate_save(struct task_struct *task, +static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) == SR_VS_DIRTY) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_save(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } } -static inline void riscv_v_vstate_restore(struct task_struct *task, +static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate, struct pt_regs *regs) { if ((regs->status & SR_VS) != SR_VS_OFF) { - struct __riscv_v_ext_state *vstate = &task->thread.vstate; - __riscv_v_vstate_restore(vstate, vstate->datap); __riscv_v_vstate_clean(regs); } @@ -220,7 +216,7 @@ static inline void __switch_to_vector(struct task_struct *prev, struct pt_regs *regs; regs = task_pt_regs(prev); - riscv_v_vstate_save(prev, regs); + riscv_v_vstate_save(&prev->thread.vstate, regs); riscv_v_vstate_set_restore(next, task_pt_regs(next)); } @@ -238,8 +234,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; } static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vsize (0) #define riscv_v_vstate_discard(regs) do {} while (0) -#define riscv_v_vstate_save(task, regs) do {} while (0) -#define riscv_v_vstate_restore(task, regs) do {} while (0) +#define riscv_v_vstate_save(vstate, regs) do {} while (0) +#define riscv_v_vstate_restore(vstate, regs) do {} while (0) #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 3f1d67109e5a..238154cb4fce 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -72,7 +72,7 @@ void kernel_vector_begin(void) get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); riscv_v_enable(); } diff --git a/arch/riscv/kernel/ptrace.c b/arch/riscv/kernel/ptrace.c index 7b93bcbdf9fa..e8515aa9d80b 100644 --- a/arch/riscv/kernel/ptrace.c +++ b/arch/riscv/kernel/ptrace.c @@ -101,7 +101,7 @@ static int riscv_vr_get(struct task_struct *target, */ if (target == current) { get_cpu_vector_context(); - riscv_v_vstate_save(current, task_pt_regs(current)); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); put_cpu_vector_context(); } diff --git a/arch/riscv/kernel/signal.c b/arch/riscv/kernel/signal.c index aca4a12c8416..5d69f4db9e8f 100644 --- a/arch/riscv/kernel/signal.c +++ b/arch/riscv/kernel/signal.c @@ -87,7 +87,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec) WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16))); get_cpu_vector_context(); - riscv_v_vstate_save(current, regs); + riscv_v_vstate_save(¤t->thread.vstate, regs); put_cpu_vector_context(); /* Copy everything of vstate but datap. */ From patchwork Thu Dec 21 13:43:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26403C35274 for ; Thu, 21 Dec 2023 13:45:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DBoi3TludzXQKB4y5mbYXIWz1RaUEj0UxOrD0WXy9GU=; b=kBH6DqoT6VkQQe uTgo+fQo9SWULbfa9pNGha9+ae6l5c9TInbhqxCZZkyWmo9LhKYkEEKFW8wrO/flzLowKZUmcPMg/ nycKO/5lQEEcHRqcP2YqeDGVNEwJ9X3wGJ7vZlR3peYLCruxQDAjwmQnwOa100QucFv+Q5E8lNqdz 4epDqaJ/E6GRJgl9S9vsgBgcVj22Fqgfl65pG3aN51h1H+m+rH4z6CWcjlNcUXQn5p1RBUvX9F8vH A8tm9cFGRbpUB6ssBJOduvHlPGFRVYcznfRTJvQa/1EJJHOIBmJ98v7wUNH7RuEnb9jptPv6ay8ZW MEZprX0sAsU66HY20hMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJM6-0032EK-0I; Thu, 21 Dec 2023 13:44:54 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJM1-00328j-0F for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:51 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1d408c2908cso1975415ad.1 for ; Thu, 21 Dec 2023 05:44:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166285; x=1703771085; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=kJ2wDGjNNg1QF4hz5KEXY+FdPMKrii0UKXFqQU9L1Ct6DP3V3EmrQ1yhET4nDP0kP9 tmfODtuz96hztv8WYPN7MqCYrgP1csqWeyg9DhHVxBKNUcbC2GNrGFNyCvABC3d7BunN 5HPgmsJmhNTHCqNat4xQMdSYXQMzpax1eYQK4rorV2z7JI/gU+bhxSBWREl3JaVhJl4O nJLnp1OdLtoZ9TYEehFEsM57Xioh3vcjaPJwbSburRAzjUPtimpZhnli4eiBnAhkpG+2 jI+dYBIBK+Eynq8xrLFjrczTGSenatbuE8ypCbWe9cbUvvqY9VWk4XnPNLJEkOeGBiIc Mexg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166285; x=1703771085; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0MSU8pZMg99K/bCQ2FBfCGwDj7GvbS21JTvN92FV/sE=; b=VQJFDpLWibG/s3tDWS9Ag/SyYgM3g1206HGXteQWkkh8GO81lfYnjsxj3KZ/8oo5RB WIr3UdsIJqtUmDOrHLRA1Wroz2Y0Wz8idHsIJZkI14X+so+mIJz+7mqQiycjI1Wdf9X1 6GnYheztnrx232xmZSzh1M15wcdTCv3deJXOdlTJPDQPK1uXxwpuInq366L0d4SE/kWu PMF4anaQTKAzuh7HxDwyI5a4C8hvBKVthQWqmxGrF1XmsCOIRkfwTg5LXm2LDVlQEXSq ucgLeChgw3s7OSQNtbG37GcDgR2sLJIdkOczc3RMDH28S/uadmAYDXKLQZao2ZnZe2Zy AFqA== X-Gm-Message-State: AOJu0Yz8EuG/Z+sOWXqKRXGvOLpjAWn/EyYHGTA450Ca0C45w1TaHUiw DXmS6mCQKYOc4S/Sjz1eLXpqNr2upTnBi9zMEFOIDYfCTepGMyZk7K6sPw4bHmjCaZ9FWSnvuQp 3SRxzdfP8oCUKBByqU/OIeMMVzdHE/DQE8Jr5imnWozW3vrfu0lQ8NacTqnmZlz9Eo2a2IqsZ1a yGDtzDL+nBeePd X-Google-Smtp-Source: AGHT+IGdEEDxX9XiN91ul6kDEy9PC3LCHGUGQUp/Fa1/M4436qZCkmQl4zofx+MeelR8gw4cIorBjw== X-Received: by 2002:a17:902:74c3:b0:1d4:5f9:d3d7 with SMTP id f3-20020a17090274c300b001d405f9d3d7mr669796plt.127.1703166284909; Thu, 21 Dec 2023 05:44:44 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.44.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:44:44 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Conor Dooley , Joel Granados Subject: [v7, 08/10] riscv: vector: use a mask to write vstate_ctrl Date: Thu, 21 Dec 2023 13:43:15 +0000 Message-Id: <20231221134318.28105-9-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054449_112669_56C936AF X-CRM114-Status: UNSURE ( 8.07 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv_v_ctrl_set() should only touch bits within PR_RISCV_V_VSTATE_CTRL_MASK. So, use the mask when we really set task's vstate_ctrl. Signed-off-by: Andy Chiu --- Changelog v6: - splitted out from v3 --- arch/riscv/kernel/vector.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 66e8c6ab09d2..c1f28bc89ec6 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt); if (inherit) ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK; + tsk->thread.vstate_ctrl |= ctrl; } bool riscv_v_vstate_ctrl_user_allowed(void) From patchwork Thu Dec 21 13:43:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502201 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E810AC35274 for ; Thu, 21 Dec 2023 13:45:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FZDpG9eOj/DqNq4PDJHmw5uH4Vq2oOK0A3nzNPwphyo=; b=p9/pJCa1U7jzdL /6JgTjS2qUQ/o/C8mq/rTAjKdcogQ+3+JSr3exB1g97qOVHk/Jar62G1WVSEPIZXPqNwm1pTq9gdh RGVzF/ZEb0dWJ1FzbGvGX+ZHEK6exkwhKln51kjX58W3DFunQNg/rCkxSCHFH8OgK/Xrq+LfyOPfP M39bs+UmJ6rWteu78lAOa9HxFTUnrPRCYq1jvL9ErjaSPqsbs5VqhwOeZr9IJXk7Mdf3jXLk64xKV SJGUqZhBcQ3qmqXtin1jRVNtvX5E1Pn3Qp9OM7G377PA5/zUEZm9hga/PWxhnceZLHQEPB309UpOT m4eqIP6mqUkxP2tmD7hw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJMA-0032HL-1E; Thu, 21 Dec 2023 13:44:58 +0000 Received: from mail-pl1-x636.google.com ([2607:f8b0:4864:20::636]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJM6-0032EJ-0b for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:44:56 +0000 Received: by mail-pl1-x636.google.com with SMTP id d9443c01a7336-1d3ef33e68dso5816975ad.1 for ; Thu, 21 Dec 2023 05:44:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166293; x=1703771093; darn=lists.infradead.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=gyaAcQXJB9Pn9Hs/a7PNDEHYRp3nfW2OLz+IfLnmB1A=; b=V2ukY3RBO2BycCFSqVA8PmkHHYAhCGnOcGIwinUR17BidoSur50Z15fnd67YHEKmeS qpRKF1m7HeY/m8s9Bdq//SpVicsygkPiAyw+1C2FIyj9t+/bsG3wZEv4ikDWqfQp319W w9Uw6ZqxBCbgbcfEjbZscf4hk2f2QisjsiBRfKf1TBir8Kg40n6N+3yPfSphA8irpDHW M39YmabbscAy3y8e/qbn9KeMbdx+H8RhaWQtN7auc+IVhZeMGZHXB3C8DVk4FKwZ/7Bn ckaqa7Ul1khLHrBZQ6ryvQHBa7F8w87evTQroXxSmkRG/w0EliKbxG/VAdEXClxcck0j SNbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166293; x=1703771093; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gyaAcQXJB9Pn9Hs/a7PNDEHYRp3nfW2OLz+IfLnmB1A=; b=o4Yz5e1uNGVKuAm46npqdml3YJT5iHrC10papUnpdQOXrqWP+fvwMZbXxoOmUS0Q4D 9zqCxJrcxD0/uDsBXCamk0MXUIiV5Ng8oJ4K65IdZRa5JFTGIDDye+SnZFL2sNzujMIy rWQ77HkI1/Tm3HagAKUZcLua/7dFbwdCW2jYjq80sIzQNAB1Noain0+auUF4mGQJFpgE 3jXHAScRIJYDvYEtGuo19L0Jig/NNp/0pXtt4H1eMXKzxX2BKC9v3K8HGrasqI8k+dTi GsayUCDj+/9vLpH1oRWnVe1Xf7Drg7U04FbSIZ/jjroOclwCM8nDo3Nofam6WY+Szha4 kNIg== X-Gm-Message-State: AOJu0Yxw2VR1lpJ1raXFcQQSPuLW2aTbV5o2dREeOtefk9u1q5Cplkn7 hvd8W8K/UUGCg/qWxShbGHnDfQgG+HgscsY7zJSvAESE8qdDWiEst+0aq8dw8KzR/T+WGG+RXsp vyScXYByoFwfmkgfnK/wV/x1YekKy76Qy0fbaYZwtat/IUYYen6VKysPLjQhthtVW9plydQ4hMk seXwZogDqzCSlg X-Google-Smtp-Source: AGHT+IHSsaze0fHydA06P2yYiQwpg6QYZhfYxXcDxSWgTU0dG1XNZjYDUGVmFJ5iALoqa23joDWj7g== X-Received: by 2002:a17:902:e892:b0:1cf:b4bb:9bdc with SMTP id w18-20020a170902e89200b001cfb4bb9bdcmr1012113plg.9.1703166292813; Thu, 21 Dec 2023 05:44:52 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.44.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:44:52 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Guo Ren , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Xiao Wang , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Jisheng Zhang , Conor Dooley , Joel Granados Subject: [v7, 09/10] riscv: vector: use kmem_cache to manage vector context Date: Thu, 21 Dec 2023 13:43:16 +0000 Message-Id: <20231221134318.28105-10-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054454_269760_7A33B3AA X-CRM114-Status: GOOD ( 11.90 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu --- Changelog v6: - new patch since v6 --- arch/riscv/include/asm/vector.h | 4 ++++ arch/riscv/kernel/process.c | 7 ++++++- arch/riscv/kernel/vector.c | 16 +++++++++++++++- 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index c5a83c277583..0e6741dd9ef3 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -26,6 +26,8 @@ void kernel_vector_begin(void); void kernel_vector_end(void); void get_cpu_vector_context(void); void put_cpu_vector_context(void); +void riscv_v_thread_free(struct task_struct *tsk); +void __init riscv_v_setup_ctx_cache(void); static inline void riscv_v_ctx_cnt_add(u32 offset) { @@ -239,6 +241,8 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define __switch_to_vector(__prev, __next) do {} while (0) #define riscv_v_vstate_off(regs) do {} while (0) #define riscv_v_vstate_on(regs) do {} while (0) +#define riscv_v_thread_free(tsk) do {} while (0) +#define riscv_v_setup_ctx_cache() do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 58127b1c6c71..38bdbcf9b81d 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -179,7 +179,7 @@ void arch_release_task_struct(struct task_struct *tsk) { /* Free the vector context of datap. */ if (has_vector()) - kfree(tsk->thread.vstate.datap); + riscv_v_thread_free(tsk); } int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) @@ -227,3 +227,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; } + +void __init arch_task_cache_init(void) +{ + riscv_v_setup_ctx_cache(); +} diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index c1f28bc89ec6..1fe140e34557 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -21,6 +21,7 @@ #include static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); +static struct kmem_cache *riscv_v_user_cachep; unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -47,6 +48,13 @@ int riscv_v_setup_vsize(void) return 0; } +void __init riscv_v_setup_ctx_cache(void) +{ + riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", + riscv_v_vsize, 16, SLAB_PANIC, + 0, riscv_v_vsize, NULL); +} + static bool insn_is_vector(u32 insn_buf) { u32 opcode = insn_buf & __INSN_OPCODE_MASK; @@ -84,7 +92,7 @@ static int riscv_v_thread_zalloc(void) { void *datap; - datap = kzalloc(riscv_v_vsize, GFP_KERNEL); + datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); if (!datap) return -ENOMEM; @@ -94,6 +102,12 @@ static int riscv_v_thread_zalloc(void) return 0; } +void riscv_v_thread_free(struct task_struct *tsk) +{ + if (tsk->thread.vstate.datap) + kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +} + #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) #define VSTATE_CTRL_GET_NEXT(x) (((x) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2) #define VSTATE_CTRL_MAKE_NEXT(x) (((x) << 2) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) From patchwork Thu Dec 21 13:43:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andy Chiu X-Patchwork-Id: 13502202 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E00B9C35274 for ; Thu, 21 Dec 2023 13:45:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=rrVRP/Vu44wsVcgtDDyxxHyJYkSws8IDRpWbLD113e8=; b=tkGWmn1qYrYo6N hr8qJ+w0LeNPCGaL9TSfSEO3abyXvfAbGdBsuE5uWvhnsdRA2kR0HpvSGnUva13bxHhjoy+EgqhjF Q0Qk7aU9fcVrKaKw9zEJs5cDzeVwjWu/tI6g52eVeRiI6Sb0yMXjsEdWKCtgT0KYlkj0YMpU+aGgv mYJBba2g13yCOUDsffnHBYJhJtJ0gDTmN4EbzTXGTgsw8e69lBZuhPQ4OnQ0UEtL4zzSAMMK5mSXj cp9VcD0w4rKog8Vz5ry4+gm9igfEOapk0wFxKgUkUuOijd/M0MG9d0HqNCC95npltt5JIURZYDb6t svC+gz7mZdPwUKP0RWTA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rGJMP-0032Rk-2P; Thu, 21 Dec 2023 13:45:13 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rGJMK-0032Oq-2k for linux-riscv@lists.infradead.org; Thu, 21 Dec 2023 13:45:12 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1d3e416f303so3632695ad.0 for ; Thu, 21 Dec 2023 05:45:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1703166307; x=1703771107; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I+xHVE2QJP9Exewue73SftDKtilcu+ifQusK0L+nlas=; b=N9WL7A6Z1rygKv+7kLiU/ZYozlL9SYtvNQwnR153UrrgBhtU3AI1Q3d3HgUrQW+tlE yAI8s8FTwAtxpyDu1Yqfko2VcRhoEsysEi+7Z4dvp7nYMt8P7Q3lZfty51yuFhVr+abW CB625fjpWqhHrKk35XAHLIyEeFXJeJWoWiVZQMo6ypIt6Y9bJxh6xXXAaz2tL8U1seeJ rj9PHOswmFtEMjXqQKnY0YDaNUSFidtfL7EJ+/OR6mO8ZV39QMOesAnvWmdOD+HCtYkQ NdrVtU205DZCuwC3owUXiw/WCt3QyTb9uE9Vli6U++eAfFuntubRvET/VqOJBP1Yt5th VGGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703166307; x=1703771107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I+xHVE2QJP9Exewue73SftDKtilcu+ifQusK0L+nlas=; b=qemsT8IMcX2dg6ArU5566GFEmiZjYwgKHLb9cnQ2TSL11DScjBYUUgAUCY8Ejz0CBL ZmatgEZV9NhphN/Kl97y+y9advmyDCRVEYvwthla3pB/UkOmObnNCPEG5WUgU6mJLxwl xUI/FItHJcoOsmMI61ttgbYKwHJBQ7n+eYFnWqIL1oCUxSuhHLuq8Y6RMFIFfZhZJroT M3txNIv4V2IOQ9oES2LShEKUBnZrE7HtFYF9r4Gr26a3dpidPmEyf/IPEIINcRrGXNkO e6fzu53+R49QCs66N4zSbGuMtywF81cqrjXSCE3N00CxkcXb621XZsiMb4fpUKlsxZNd zVQg== X-Gm-Message-State: AOJu0YwOJZTFiLdC/hIgwBM/3ig3kWZ8i67D43Eik/z9UW1nb2Pc0v4a 3W0egO8EHEU64GhH0EE0OIV/Mzzggtdcp0YExIqdrIBvg3yyzUMthAVx2X+QMhPT5cVUiHs7U0x Char6d6puICQ2ophvNg6po1PB75mcslG9xx2Uac8R3n10ghA0ItFYBO+9XFZMQmarrIOPC0/03c VGnmu5ZpYEPDjd X-Google-Smtp-Source: AGHT+IEQcAWzC5gqxmAe+dXyBFM0VGozaCehLIuu3XWLQMyNn9261nV+x5mZXmEyD/jgIZ8EKlP+Iw== X-Received: by 2002:a17:903:2311:b0:1d3:6400:5532 with SMTP id d17-20020a170903231100b001d364005532mr9508726plh.96.1703166306869; Thu, 21 Dec 2023 05:45:06 -0800 (PST) Received: from hsinchu26.internal.sifive.com (59-124-168-89.hinet-ip.hinet.net. [59.124.168.89]) by smtp.gmail.com with ESMTPSA id iw3-20020a170903044300b001c72d5e16acsm1646001plb.57.2023.12.21.05.45.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Dec 2023 05:45:06 -0800 (PST) From: Andy Chiu To: linux-riscv@lists.infradead.org, palmer@dabbelt.com Cc: paul.walmsley@sifive.com, greentime.hu@sifive.com, guoren@linux.alibaba.com, bjorn@kernel.org, charlie@rivosinc.com, ardb@kernel.org, arnd@arndb.de, peterz@infradead.org, tglx@linutronix.de, Andy Chiu , Albert Ou , Vincent Chen , Heiko Stuebner , Baoquan He , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Guo Ren , Xiao Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Sami Tolvanen , Nathan Chancellor , Jisheng Zhang , Fangrui Song , Deepak Gupta , Conor Dooley , Joel Granados Subject: [v7, 10/10] riscv: vector: allow kernel-mode Vector with preemption Date: Thu, 21 Dec 2023 13:43:17 +0000 Message-Id: <20231221134318.28105-11-andy.chiu@sifive.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20231221134318.28105-1-andy.chiu@sifive.com> References: <20231221134318.28105-1-andy.chiu@sifive.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231221_054508_919633_FD27C5A6 X-CRM114-Status: GOOD ( 30.81 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu --- Changelog v6: - re-write patch to handle context nesting for softirqs - drop thread flag and track context instead in riscv_v_flags - refine some asm code and constraint it into C functions - preallocate v context for preempt_v - Return non-zero in riscv_v_start_kernel_context with non-preemptible kernel-mode Vector Changelog v4: - dropped from v4 Changelog v3: - Guard vstate_save with {get,set}_cpu_vector_context - Add comments on preventions of nesting V contexts - remove warnings in context switch when trap's reg is not pressent (Conor) - refactor code (Björn) Changelog v2: - fix build fail when compiling without RISCV_ISA_V (Conor) - 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor) - merge Kconfig patch into this oine (Conor). - 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor) - fix some typos (Conor) - enclose assembly with RISCV_ISA_V_PREEMPTIVE. - change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor) - 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/' --- arch/riscv/Kconfig | 14 +++ arch/riscv/include/asm/processor.h | 26 +++++- arch/riscv/include/asm/simd.h | 26 +++++- arch/riscv/include/asm/vector.h | 57 +++++++++++- arch/riscv/kernel/entry.S | 8 ++ arch/riscv/kernel/kernel_mode_vector.c | 121 ++++++++++++++++++++++++- arch/riscv/kernel/process.c | 3 + arch/riscv/kernel/vector.c | 31 +++++-- 8 files changed, 265 insertions(+), 21 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index cba53dcc2ae0..70603c486593 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -557,6 +557,20 @@ config RISCV_ISA_V_MEMMOVE_THRESHOLD Prefer using vectorized memmove() when the workload size exceeds this value. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Usually, in-kernel SIMD routines are run with preemption disabled. + Functions which envoke long running SIMD thus must yield core's + vector unit to prevent blocking other tasks for too long. + + This config allows kernel to run SIMD without explicitly disable + preemption. Enabling this config will result in higher memory + consumption due to the allocation of per-task's kernel Vector context. + config TOOLCHAIN_HAS_ZBB bool default y diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 15781e2232e0..4de9124bcf4f 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -81,11 +81,32 @@ struct pt_regs; * activation of this state disables the preemption. On a non-RT kernel, it * also disable bh. Currently only 0 and 1 are valid value for this field. * Other values are reserved for future uses. + * - bits 8-15 are used for tracking preemptible kernel-mode Vector, when + * RISCV_ISA_V_PREEMPTIVE is set. Calling kernel_vector_begin() does not + * disable the preemption if the thread's kernel_vstate.datap is allocated. + * Instead, the kernel adds 1 into this field. Then the trap entry/exit code + * knows if we are entering/exiting the context that owns preempt_v. + * - 0: the task is not using preempt_v + * - 1: the task is actively using, and owns preempt_v + * - >1: the task was using preempt_v, but then took a trap within. Thus, + * the task does not own preempt_v. Any use of Vector will have to save + * preempt_v, if dirty, and fallback to non-preemptible kernel-mode + * Vector. + * - bit 30: The in-kernel preempt_v context is saved, and requries to be + * restored when returning to the context that owns the preempt_v. + * - bit 31: The in-kernel preempt_v context is dirty, as signaled by the + * trap entry code. Any context switches out-of current task need to save + * it to the task's in-kernel V context. Also, any traps nesting on-top-of + * preempt_v requesting to use V needs a save. */ -#define RISCV_KERNEL_MODE_V_MASK 0xff +#define RISCV_KERNEL_MODE_V_MASK 0x000000ff +#define RISCV_PREEMPT_V_MASK 0x0000ff00 -#define RISCV_KERNEL_MODE_V 0x1 +#define RISCV_KERNEL_MODE_V 0x00000001 +#define RISCV_PREEMPT_V 0x00000100 +#define RISCV_PREEMPT_V_DIRTY 0x80000000 +#define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000 /* CPU-specific state of a task */ struct thread_struct { @@ -99,6 +120,7 @@ struct thread_struct { u32 vstate_ctrl; struct __riscv_v_ext_state vstate; unsigned long align_ctl; + struct __riscv_v_ext_state kernel_vstate; }; /* Whitelist the fstate from the task_struct for hardened usercopy */ diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index 7df5a976a80a..3fe6f201ee6e 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -12,6 +12,7 @@ #include #include #include +#include #include @@ -28,12 +29,27 @@ static __must_check inline bool may_use_simd(void) /* * RISCV_KERNEL_MODE_V is only set while preemption is disabled, * and is clear whenever preemption is enabled. - * - * Kernel-mode Vector temperarily disables bh. So we must not return - * true on irq_disabled(). Otherwise we would fail the lockdep check - * calling local_bh_enable() */ - return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); + if (in_hardirq() || in_nmi()) + return false; + + /* + * Nesting is acheived in preempt_v by spreading the control for + * preemptible and non-preemptible kernel-mode Vector into two fields. + * Always try to match with prempt_v if kernel V-context exists. Then, + * fallback to check non preempt_v if nesting happens, or if the config + * is not set. + */ + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) { + if (!riscv_preempt_v_started(current)) + return true; + } + /* + * Non-preemptible kernel-mode Vector temperarily disables bh. So we + * must not return true on irq_disabled(). Otherwise we would fail the + * lockdep check calling local_bh_enable() + */ + return !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 0e6741dd9ef3..542eaf9227c3 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -28,6 +28,7 @@ void get_cpu_vector_context(void); void put_cpu_vector_context(void); void riscv_v_thread_free(struct task_struct *tsk); void __init riscv_v_setup_ctx_cache(void); +void riscv_v_thread_alloc(struct task_struct *tsk); static inline void riscv_v_ctx_cnt_add(u32 offset) { @@ -212,14 +213,63 @@ static inline void riscv_v_vstate_set_restore(struct task_struct *task, } } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline bool riscv_preempt_v_dirty(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + return !!(val & RISCV_PREEMPT_V_DIRTY); +} + +static inline bool riscv_preempt_v_restore(struct task_struct *task) +{ + u32 val = READ_ONCE(task->thread.riscv_v_flags); + + return !!(val & RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_preempt_v_clear_dirty(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags &= ~RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_set_restore(struct task_struct *task) +{ + barrier(); + task->thread.riscv_v_flags |= RISCV_PREEMPT_V_NEED_RESTORE; +} + +static inline bool riscv_preempt_v_started(struct task_struct *task) +{ + return !!(READ_ONCE(task->thread.riscv_v_flags) & RISCV_PREEMPT_V_MASK); +} +#else /* !CONFIG_RISCV_ISA_V_PREEMPTIVE */ +static inline bool riscv_preempt_v_dirty(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_restore(struct task_struct *task) { return false; } +static inline bool riscv_preempt_v_started(struct task_struct *task) { return false; } +#define riscv_preempt_v_clear_dirty(tsk) do {} while (0) +#define riscv_preempt_v_set_restore(tsk) do {} while (0) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + static inline void __switch_to_vector(struct task_struct *prev, struct task_struct *next) { struct pt_regs *regs; - regs = task_pt_regs(prev); - riscv_v_vstate_save(&prev->thread.vstate, regs); - riscv_v_vstate_set_restore(next, task_pt_regs(next)); + if (riscv_preempt_v_dirty(prev)) { + __riscv_v_vstate_save(&prev->thread.kernel_vstate, + prev->thread.kernel_vstate.datap); + riscv_preempt_v_clear_dirty(prev); + } else { + regs = task_pt_regs(prev); + riscv_v_vstate_save(&prev->thread.vstate, regs); + } + + if (riscv_preempt_v_started(next)) + riscv_preempt_v_set_restore(next); + else + riscv_v_vstate_set_restore(next, task_pt_regs(next)); } void riscv_v_vstate_ctrl_init(struct task_struct *tsk); @@ -243,6 +293,7 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } #define riscv_v_vstate_on(regs) do {} while (0) #define riscv_v_thread_free(tsk) do {} while (0) #define riscv_v_setup_ctx_cache() do {} while (0) +#define riscv_v_thread_alloc(tsk) do {} while (0) #endif /* CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 54ca4564a926..9d1a305d5508 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -83,6 +83,10 @@ SYM_CODE_START(handle_exception) /* Load the kernel shadow call stack pointer if coming from userspace */ scs_load_current_if_task_changed s5 +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_start +#endif move a0, sp /* pt_regs */ la ra, ret_from_exception @@ -138,6 +142,10 @@ SYM_CODE_START_NOALIGN(ret_from_exception) */ csrw CSR_SCRATCH, tp 1: +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + move a0, sp + call riscv_v_context_nesting_end +#endif REG_L a0, PT_STATUS(sp) /* * The current load reservation is effectively part of the processor's diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 238154cb4fce..6a7df511ccdc 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -50,6 +50,111 @@ void put_cpu_vector_context(void) preempt_enable(); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static inline void riscv_preempt_v_set_dirty(void) +{ + current->thread.riscv_v_flags |= RISCV_PREEMPT_V_DIRTY; +} + +static inline void riscv_preempt_v_reset_flags(void) +{ + current->thread.riscv_v_flags &= ~(RISCV_PREEMPT_V_DIRTY | RISCV_PREEMPT_V_NEED_RESTORE); +} + +static inline void riscv_preempt_v_depth_inc(void) +{ + riscv_v_ctx_cnt_add(RISCV_PREEMPT_V); +} + +static inline void riscv_preempt_v_depth_dec(void) +{ + riscv_v_ctx_cnt_sub(RISCV_PREEMPT_V); +} + +static inline u32 riscv_preempt_v_get_depth(void) +{ + return riscv_v_ctx_cnt() & RISCV_PREEMPT_V_MASK; +} + +#define PREEMPT_V_FIRST_DEPTH RISCV_PREEMPT_V +static int riscv_v_stop_kernel_context(void) +{ + if (riscv_preempt_v_get_depth() != PREEMPT_V_FIRST_DEPTH) + return 1; + + riscv_preempt_v_depth_dec(); + return 0; +} + +static int riscv_v_start_kernel_context(bool *is_nested) +{ + struct __riscv_v_ext_state *vstate = ¤t->thread.kernel_vstate; + + if (!vstate->datap) + return -ENOENT; + + if (riscv_preempt_v_started(current)) { + WARN_ON(riscv_preempt_v_get_depth() == PREEMPT_V_FIRST_DEPTH); + if (riscv_preempt_v_dirty(current)) { + get_cpu_vector_context(); + __riscv_v_vstate_save(vstate, vstate->datap); + riscv_preempt_v_clear_dirty(current); + put_cpu_vector_context(); + } + get_cpu_vector_context(); + riscv_preempt_v_set_restore(current); + *is_nested = true; + return 0; + } + + get_cpu_vector_context(); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + put_cpu_vector_context(); + + riscv_preempt_v_depth_inc(); + return 0; +} + +/* low-level V context handling code, called with irq disabled */ +asmlinkage void riscv_v_context_nesting_start(struct pt_regs *regs) +{ + int depth; + + if (!riscv_preempt_v_started(current)) + return; + + depth = riscv_preempt_v_get_depth(); + if (depth == PREEMPT_V_FIRST_DEPTH && (regs->status & SR_VS) == SR_VS_DIRTY) + riscv_preempt_v_set_dirty(); + + riscv_preempt_v_depth_inc(); +} + +asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs) +{ + struct __riscv_v_ext_state *vstate = ¤t->thread.kernel_vstate; + u32 depth; + + lockdep_assert_irqs_disabled(); + + if (!riscv_preempt_v_started(current)) + return; + + riscv_preempt_v_depth_dec(); + depth = riscv_preempt_v_get_depth(); + if (depth == PREEMPT_V_FIRST_DEPTH) { + if (riscv_preempt_v_restore(current)) { + __riscv_v_vstate_restore(vstate, vstate->datap); + __riscv_v_vstate_clean(regs); + } + riscv_preempt_v_reset_flags(); + } +} +#else +#define riscv_v_start_kernel_context(nested) (-ENOENT) +#define riscv_v_stop_kernel_context() (-ENOENT) +#endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ + /* * kernel_vector_begin(): obtain the CPU vector registers for use by the calling * context @@ -65,14 +170,20 @@ void put_cpu_vector_context(void) */ void kernel_vector_begin(void) { + bool nested = false; + if (WARN_ON(!has_vector())) return; BUG_ON(!may_use_simd()); - get_cpu_vector_context(); + if (riscv_v_start_kernel_context(&nested)) { + get_cpu_vector_context(); + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + } - riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + if (!nested) + riscv_v_vstate_set_restore(current, task_pt_regs(current)); riscv_v_enable(); } @@ -92,10 +203,10 @@ void kernel_vector_end(void) if (WARN_ON(!has_vector())) return; - riscv_v_vstate_set_restore(current, task_pt_regs(current)); - riscv_v_disable(); - put_cpu_vector_context(); + if (riscv_v_stop_kernel_context()) {// we should call this early + put_cpu_vector_context(); + } } EXPORT_SYMBOL_GPL(kernel_vector_end); diff --git a/arch/riscv/kernel/process.c b/arch/riscv/kernel/process.c index 38bdbcf9b81d..1afdec4aeda6 100644 --- a/arch/riscv/kernel/process.c +++ b/arch/riscv/kernel/process.c @@ -188,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src; /* clear entire V context, including datap for a new task */ memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state)); + memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state)); clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE); return 0; } @@ -223,6 +224,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.s[0] = 0; } p->thread.riscv_v_flags = 0; + if (has_vector()) + riscv_v_thread_alloc(p); p->thread.ra = (unsigned long)ret_from_fork; p->thread.sp = (unsigned long)childregs; /* kernel sp */ return 0; diff --git a/arch/riscv/kernel/vector.c b/arch/riscv/kernel/vector.c index 1fe140e34557..f9769703fd39 100644 --- a/arch/riscv/kernel/vector.c +++ b/arch/riscv/kernel/vector.c @@ -22,6 +22,9 @@ static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE); static struct kmem_cache *riscv_v_user_cachep; +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +static struct kmem_cache *riscv_v_kernel_cachep; +#endif unsigned long riscv_v_vsize __read_mostly; EXPORT_SYMBOL_GPL(riscv_v_vsize); @@ -53,6 +56,11 @@ void __init riscv_v_setup_ctx_cache(void) riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx", riscv_v_vsize, 16, SLAB_PANIC, 0, riscv_v_vsize, NULL); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_kernel_cachep = kmem_cache_create("riscv_vector_kctx", + riscv_v_vsize, 16, + SLAB_PANIC, NULL); +#endif } static bool insn_is_vector(u32 insn_buf) @@ -88,24 +96,35 @@ static bool insn_is_vector(u32 insn_buf) return false; } -static int riscv_v_thread_zalloc(void) +static int riscv_v_thread_zalloc(struct kmem_cache *cache, + struct __riscv_v_ext_state *ctx) { void *datap; - datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL); + datap = kmem_cache_zalloc(cache, GFP_KERNEL); if (!datap) return -ENOMEM; - current->thread.vstate.datap = datap; - memset(¤t->thread.vstate, 0, offsetof(struct __riscv_v_ext_state, - datap)); + ctx->datap = datap; + memset(ctx, 0, offsetof(struct __riscv_v_ext_state, datap)); return 0; } +void riscv_v_thread_alloc(struct task_struct *tsk) +{ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + riscv_v_thread_zalloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate); +#endif +} + void riscv_v_thread_free(struct task_struct *tsk) { if (tsk->thread.vstate.datap) kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE + if (tsk->thread.kernel_vstate.datap) + kmem_cache_free(riscv_v_kernel_cachep, tsk->thread.kernel_vstate.datap); +#endif } #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK) @@ -177,7 +196,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs) * context where VS has been off. So, try to allocate the user's V * context and resume execution. */ - if (riscv_v_thread_zalloc()) { + if (riscv_v_thread_zalloc(riscv_v_user_cachep, ¤t->thread.vstate)) { force_sig(SIGBUS); return true; }