From patchwork Sun Nov 3 14:51:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD0CED11100 for ; Sun, 3 Nov 2024 14:53:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9kRKqQ5UXLGzODQfxDxI9nK7DK4jncXta0FhxXOfE70=; b=hbJWkiXyjNimeV h2dWu9tCZeylgmyIXEcH26mZlzV5m7wGf2u/kvJyQ/8/A53tWak69U6Owd67IDMbtWtcyUQOWesJn Mr9cI1GbbGeZ0bxdqpmKQhJGuT6lp1n2xYVhS2Xy5qbvo2CkybNwwNAGzQHg049DH8uge9cIAIS4r gy9TBQJztwExJIAVz3+7qxibGinJ2rB6yg7C6BzcR/7C79sA7R9wOuMdVTjftUvZXIKseylfQ/u5X laIz79mbpQDHwRWRMOFECiDZ2l1Bck2PuHnqewD1f8/ZnIhFvMrGBh8g0ADyJK2YB4rKnSgYgrV0x /Um7DSmownSu02kuOlUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7byP-0000000BZ9B-0bPm; Sun, 03 Nov 2024 14:53:01 +0000 Received: from mail-wr1-x42e.google.com ([2a00:1450:4864:20::42e]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7byM-0000000BZ8A-3XCr for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:53:00 +0000 Received: by mail-wr1-x42e.google.com with SMTP id ffacd0b85a97d-37d462c91a9so2057418f8f.2 for ; Sun, 03 Nov 2024 06:52:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645577; x=1731250377; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=2gIazIiZ5YEQ+IbfX+SjXQHhhSfEwzvX6EXLv4PA9vByW9N+Fd3HPAJK7zIl+vP38v XFB12zgbZRxYrD+LAnI68hwRqfIFmQv5+mN25v382QWeTC56UrlHBY1UQYiPC4kZDZCc BMG+vHv0Qt1ea4SrOlpU+M3gJv6VNAqsqE3O4eGn/5HaPaWpGCBadU3VlOh8O6knfoHQ 4gm0ruge+HlPdBtIyfWZ/gHWUWAWCE6TmjNBsIij6EkuU/idbX0YM6eGwgxp07IugrLM Cm0e9Q4BZMntgZdYiO9ZIeKF+rBTaqCW6A4fyN37pGxsJqTnlGpkTzTVWf30XMI+bCTq pkdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645577; x=1731250377; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=lg0p++D3gKB7nqtvb9OURwvcaZI6tBk5CMYogN6LParFwx2DjdYKPfgTLZwqFOdTmS QRZU5g7dOT3dXLuwhoeD4sk95WyfXypTUJtv+l4IJWd4LVeAVrLr6nBuR7jiwwIbhyzC q1cMoFYUKwJIJ61kK4xnlUm9fXtHBeJC4GjcogDZcgo/2LJB7D/39ZRglDguAsSbAT3N Mz1V8OpnN0FEiUeHof0ErXjTUMxs36hjEcE6gAwnpLtmIHif9zQDZY4rDX40dsQhmMlT 4wDbEHKIT3lvkcf1sFhlkW8MefLqU0jK4FVBmbuOUlcqyZVP3vzY16DrVdMfhQAl1Qtt +//w== X-Forwarded-Encrypted: i=1; AJvYcCXjb9NKlX5CmTRPgr/EnEQfYm2P4xwQIp3gfmV1TCT0M/V5/2J3z8ga6o6LzG8Yp6Zh+plWPi9b/2YN1Q==@lists.infradead.org X-Gm-Message-State: AOJu0YzMUPzUMx+ZgmMh4Y2aJLFlOmsB//OXQXs2fxNOX1HASS5KkSkh f7ycEiKv/3jlGwBoDEPjtQGJkfTt1gJijeQJjg2AsAgM5g9mxfQ0c06cLodYVBs= X-Google-Smtp-Source: AGHT+IFvsXKhMz1fjru6A+QnFRydk5FLICcqibf9uwIL5DAG/Ic4jbFEh3bsH3jcltqVkAZyHMwFyA== X-Received: by 2002:a05:6000:1b08:b0:378:89be:1825 with SMTP id ffacd0b85a97d-381b710fbb0mr11387559f8f.49.1730645577054; Sun, 03 Nov 2024 06:52:57 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c113e642sm10721990f8f.68.2024.11.03.06.52.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:52:56 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 01/13] riscv: Move cpufeature.h macros into their own header Date: Sun, 3 Nov 2024 15:51:41 +0100 Message-Id: <20241103145153.105097-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065258_909985_50090656 X-CRM114-Status: GOOD ( 14.89 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org asm/cmpxchg.h will soon need riscv_has_extension_unlikely() macros and then needs to include asm/cpufeature.h which introduces a lot of header circular dependencies. So move the riscv_has_extension_XXX() macros into their own header which prevents such circular dependencies by including a restricted number of headers. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cpufeature-macros.h | 66 ++++++++++++++++++++++ arch/riscv/include/asm/cpufeature.h | 61 ++------------------ 2 files changed, 70 insertions(+), 57 deletions(-) create mode 100644 arch/riscv/include/asm/cpufeature-macros.h diff --git a/arch/riscv/include/asm/cpufeature-macros.h b/arch/riscv/include/asm/cpufeature-macros.h new file mode 100644 index 000000000000..a8103edbf51f --- /dev/null +++ b/arch/riscv/include/asm/cpufeature-macros.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright 2022-2024 Rivos, Inc + */ + +#ifndef _ASM_CPUFEATURE_MACROS_H +#define _ASM_CPUFEATURE_MACROS_H + +#include +#include + +#define STANDARD_EXT 0 + +bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); +#define riscv_isa_extension_available(isa_bitmap, ext) \ + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) + +static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_no); + + return true; +l_no: + return false; +} + +static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_yes); + + return false; +l_yes: + return true; +} + +static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +static __always_inline bool riscv_has_extension_likely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +#endif /* _ASM_CPUFEATURE_MACROS_H */ diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 45f9c1171a48..87ed88fc950d 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -8,9 +8,11 @@ #include #include +#include +#include +#include #include -#include -#include +#include /* * These are probed via a device_initcall(), via either the SBI or directly @@ -103,61 +105,6 @@ extern const size_t riscv_isa_ext_count; extern bool riscv_isa_fallback; unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap); - -#define STANDARD_EXT 0 - -bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); -#define riscv_isa_extension_available(isa_bitmap, ext) \ - __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) - -static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_no); - - return true; -l_no: - return false; -} - -static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_yes); - - return false; -l_yes: - return true; -} - -static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_unlikely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - -static __always_inline bool riscv_has_extension_likely(const unsigned long ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_likely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const unsigned long ext) { compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); From patchwork Sun Nov 3 14:51:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 632A1D11101 for ; Sun, 3 Nov 2024 14:54:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=O9S/PE2Qe1LCKRodbuyOrqzBPbXk+c18GxGFxHKlOXo=; b=dSn5VFDal10b6x p9nuDxCwZQKZTDqzsYjGT4r8dCNjFvQ0CU42wyhj6Qtbuiuhs1Yr7WsYI9WvGk2/Ojc1OksaPQ1NY /wslcLNIm6bSEPROsLiPIa/aX9Ij6X8FOJBn6K3JboWaPz1c9dMobYdr+91Hbvnv7jJfk5csrHT/x NxFkglt8d3BNJWT6bWg7puTs2WyuwwdVvjoSvF8NADzlx+4nSlzvBRmbo3LxNNgujzc6Z1PVBOHgq 4T2caoNvbM954ppNsdjtA54i7mquThdT21wYN9j+WXntaDVbkAyrzPytQql6oBv56DmdMRU0UThNU zMhnanf+K+GWcVjrCNJA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7bzO-0000000BZLK-1VJ0; Sun, 03 Nov 2024 14:54:02 +0000 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7bzL-0000000BZKf-349H for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:54:00 +0000 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-37d47eff9acso2180711f8f.3 for ; Sun, 03 Nov 2024 06:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645638; x=1731250438; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nql3DD9359qdXdLLpVtxL8oa+5wIDas5DqswNkA3KmI=; b=sxD46vKLcOv8WbXNSOd/Ll+Ir28ZzBMxg4OW1baZn8eNq4kK4yGjFZ7lvIP3XBBAtj RGofedpfatF6TipRhEpoTGRdx4TQsvrJezvYpY6jvdnyq8YkED78RmorFdaquNPzE6bn xFeclBXV/BWnUtXpS8phbE7ZnznCOjHNn9Ji0aPJpVpyWl4JmQKYEN3Gga230UiC13AY PY5k6PMTINubAtlH3T4bttJhxWaC+U0c1c+YywRKoC9m+9NXA08Zwo1RE0HeXAgvhiOv /8W44v4WkZx/caFwKbocuBRhwESmm/Lj4dIvVJCk8yGFdKkDmDgqvXkffaaAWqQtP+PA uelQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645638; x=1731250438; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nql3DD9359qdXdLLpVtxL8oa+5wIDas5DqswNkA3KmI=; b=X58q1eOSU97JEgNSBL4Q1xJtmUVCb8CTuQ3Ch0nMcgCt2I40guMhT91OKT6ij+ZdPV wRAdrEqTPUWPqAXfKRQCoTgyUXUtyTNBBdCNYWwITWViJ6DzAhW9QumxBiIj7YNzt/WT htz8NBBOOuyN/TWM0kbtRaKT1QinHRsPhvAA0HPOrFWGW6vbwFPp9gRHmb6lhEc2EyTt UXDh1q2iYMbh3OZGXKJNX/FkMlEgU0WaPxLlyhmaotvsQnc8ELf5WUkHvP/kDD3tUmII AmWU1tYVETvE1lOp2Mu2QBLocUMx6SvnfQHJd52oetnRwvgTNhPX1waaK0jsN3qZ0NiA pYrw== X-Forwarded-Encrypted: i=1; AJvYcCVGEtMVdSegmbzZMXT1iLy/sgrE1NGy8k0fWqF4PblrVpgfJWs9MuwSniGLmsOAFyUM0PNeVap1MB6UrQ==@lists.infradead.org X-Gm-Message-State: AOJu0YwZVHktR7+conu3G31TpV+tDZc7EMy8dCn2TU2NYBco8cyu/5So mtf3peSbVOdWgcqvoihkHJCcjFiJTwF0DxBVNrvHk4YIVVNAwZZze30tIPsr0wQ= X-Google-Smtp-Source: AGHT+IEzqXU+n6yFRFqvDrou02tN/4/eCcsT/sjowi0L/ehc2kcbHr+rCGMDYfv6CZMwRO5vLoFRDw== X-Received: by 2002:a5d:47ac:0:b0:371:9360:c4a8 with SMTP id ffacd0b85a97d-381c7a472ecmr8333590f8f.6.1730645638302; Sun, 03 Nov 2024 06:53:58 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10d7e0fsm10710932f8f.47.2024.11.03.06.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:53:57 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 02/13] riscv: Do not fail to build on byte/halfword operations with Zawrs Date: Sun, 3 Nov 2024 15:51:42 +0100 Message-Id: <20241103145153.105097-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065359_793805_38BD6FAE X-CRM114-Status: UNSURE ( 9.78 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv does not have lr instructions on byte and halfword but the qspinlock implementation actually uses such atomics provided by the Zabha extension, so those sizes are legitimate. Then instead of failing to build, just fallback to the !Zawrs path. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ebbce134917c..ac1d7df898ef 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -245,6 +245,11 @@ static __always_inline void __cmpwait(volatile void *ptr, : : : : no_zawrs); switch (size) { + case 1: + fallthrough; + case 2: + /* RISC-V doesn't have lr instructions on byte and half-word. */ + goto no_zawrs; case 4: asm volatile( " lr.w %0, %1\n" From patchwork Sun Nov 3 14:51:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0C34BD11101 for ; Sun, 3 Nov 2024 14:55:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UgmyfHYhhkYV+cs/689A8mKtbQ2K0+EM31IhhQSRlto=; b=GdtU7O3hkO9vua gpOo6YcQ/gkQgfKquEznrl8C7dg/NCSW3btDFTnBaR3Y2A3LDsM0yF3jBiGyfFs5FSb7KyjTxBoFx lch5uHXi1Ytu4YF5WXu1zo3eiMEb4YMNpaoxDYaQyUUULYCccaySibdlybH46U7DFBC9riDJgGgKl 7C5q3Vn6EbMIgieMBw2sxzNyG+ScXS9IEn+HF/cTV9Ab6pS9SSYrOHwWgrpiYtHL4n1F+DQ2hDXW3 VVumRag13ZRGN2LpiJZGLu/J8cMtiYWKVgFnZIuShHOjjWT7c+YAn8abUGHF6FvKlyCVa9tiTzbR0 SjbQjhn267cLjMS1PcVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c0O-0000000BZWf-2dXy; Sun, 03 Nov 2024 14:55:04 +0000 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c0L-0000000BZVa-3zGA for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:55:03 +0000 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-37d49a7207cso2132917f8f.0 for ; Sun, 03 Nov 2024 06:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645700; x=1731250500; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j2RpMtAPESbZvfWQgyCWL6emNu2SZ3U7eDtJoRXd/Zk=; b=PVFKBH5siDC5VxgedbeZyqHX97e1iiLMNtzDx18UPPNvt/rFrrsnva/1TA+WrqFWCd +/qylw+voJoksZQb2L1F6aADT9pdNYrf+PBOomlUSX4kmP0glmHbCAE1IYPKXOWxyp+L sQVP1xhhOSmrNJI/D9nbbMc5b4VIokWUyIFlcZqktXOdrgsj/3rI2atFwQh54xJ1R3Ae el+MVw5MEko/h3kmBE0CKbU0l4DW+OcwfOmv44LeXi7jU4odh+V/E5g5TsbONhJIM+vQ x5VhxIZyUeyhQVG0JOQo/rB478ogfgeLtCRdNqVKUB/7qgZUWyalgCSY0Re58AReo0Su VFCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645700; x=1731250500; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j2RpMtAPESbZvfWQgyCWL6emNu2SZ3U7eDtJoRXd/Zk=; b=XG13BBQ4lUmqCsnMFxbPh/Lqsm42qkkCfsnzhjbIjwLILhhtisWRYylf6W9tu6DUxx Rs0rShfuu7tN8O0sSkqb9DJ77kAXZw1/QzaK+v/65xwWxbffcufNZbLUvF7p27ERSOdU KobzJWjEr4RPG68YeWINJpjfe3CCFQR2cnu9tWYtZWzcS3cAjvqPJ6lmZW+MwgG26FMG NPu/9PcedzU9fTEw8sR2+CaPvLD3mXkUquOXZq1Sc7mU1ONESQ0h0Mv1kcT75yMGZjLG W41Rh6MFxK2GiTawuJx5d6yvucliPVW6zIMLc3YDdXD8+5sNm/fJDw0iiUgyA6/Sltd6 wbuA== X-Forwarded-Encrypted: i=1; AJvYcCVVrvDLSjdYU36EKAHrLa/5pGObkv5/7L7il0tplZULteMgFU5JZXqrpE1Gx3tU3yXoGdx5mXpko6iCYQ==@lists.infradead.org X-Gm-Message-State: AOJu0YyUOLF1a0J7k53UgTsVya5Fo/nSkh+Q53B220iPXfCy0ZWmmJOh 3PE6zh8cgBQ5XDpTn2gnwACmkZ8CU+eSnO7RCDCP1RHrFW0P91u2HtOVQ+EuW+o= X-Google-Smtp-Source: AGHT+IEoz3UUuA4gFnje/hBBB1Z82zuI7puFcHPYtiCXU5cLSTtZW75S2vlz7SH9FLr/9gF8I3PV6g== X-Received: by 2002:a5d:584b:0:b0:37d:4a2d:6948 with SMTP id ffacd0b85a97d-381c7a5f380mr7629990f8f.33.1730645699563; Sun, 03 Nov 2024 06:54:59 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10d4342sm10785612f8f.32.2024.11.03.06.54.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:54:59 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 03/13] riscv: Implement cmpxchg32/64() using Zacas Date: Sun, 3 Nov 2024 15:51:43 +0100 Message-Id: <20241103145153.105097-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065502_014868_1613908C X-CRM114-Status: GOOD ( 13.93 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 16 +++++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 48 +++++++++++++++++++++----------- 3 files changed, 50 insertions(+), 17 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 62545946ecf4..3542efe3088b 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -632,6 +632,22 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + If you don't know what to do here, say Y. config TOOLCHAIN_HAS_ZBB diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index d469db9f46f4..3700a1574413 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -82,6 +82,9 @@ else riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei endif +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ac1d7df898ef..39c1daf39f6a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -137,24 +138,37 @@ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ ({ \ - register unsigned int __rc; \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r = o; \ \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr" lr_sfx " %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ - : "rJ" (co o), "rJ" (n) \ - : "memory"); \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ + } \ }) -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -164,15 +178,15 @@ switch (sizeof(*__ptr)) { \ case 1: \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ __ret, __ptr, /**/, __old, __new); \ break; \ default: \ From patchwork Sun Nov 3 14:51:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A30CD11101 for ; Sun, 3 Nov 2024 14:56:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WfX/Ah556/Ah8ZMdrByD2befs0yjr1KQCmAUhoZYtrI=; b=K7tr3w3RuggZ5H LNkcncJX18L1YNtyVADMSsv8jcySrZmMFpa9/TvkKAcUfVOwjJRaMLb53xNPs02rI+M2oQF0kb3xQ TIl0ekOFW169gBqm9ftW/TB9Jg1c82wQwVi7fEuNnELkviNqglg7nECSsGUFGNpO56JhSd0+BKyu5 AxYGPsdWpdpRbYCeQ6TwjQNgmH1lVuG+FiwgQ+52ku0X5mUH0IpAeIm348RseZGDihzmTmq9fr0Nk te//dHf0Ze3sls4vJcnZBPT5uT4NxZmij1Yp6vQmURgsVCrffdt9ROrInOfjKetHQFP6bTcxbU0sS N4bpFhp5CLiWWDkW5/AA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c1M-0000000BZhb-33Jy; Sun, 03 Nov 2024 14:56:04 +0000 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c1K-0000000BZgu-34C5 for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:56:03 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-43158625112so29019475e9.3 for ; Sun, 03 Nov 2024 06:56:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645761; x=1731250561; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/lAswjGp3xf0DHvJRryPePimKYUd7lF/oN62wU2lB88=; b=U5G84sT4SDp9kDNGuySDO8rvdx6vQPdMvfROnJZ2aYAhE2AwFTIfJeJ1f+9UeN7tOP 5YJEDZ7C15c4+DW/0TRAIXPLqYFJXY4L5BjahSPtX7BYlsUTHugg+uG1l4q61GSLmqhB q3ni/752FDVziAUGsDYY83AuRk3cjZzFyaUXI03f4KmkQOFLYtNSl1W5O6WB8+UzVaWE kf7lIhYpEJJOhfkZUDO8u1vQyiXnaCr5vxLyRp38ZdD46XQhDHPJmlcvxfddvvTq3fWN FgrewTcJeIHEzeV8XjJqTMcIW5Z4VclsUd9A9RbCfMR8rejcOd5BenvIX580w7S+UMU6 xFiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645761; x=1731250561; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/lAswjGp3xf0DHvJRryPePimKYUd7lF/oN62wU2lB88=; b=nPiMi17BVLzu4OFFhgzTQmsVSvz1ZZVNUgEajQPn6SIWDzYE4ZJuAHkgk/J5d6i8nx Nhc4Dsu8gZUqv3eMA9lT1t0wK1C/3EgI2WDSEg6jq2MtLcmphRBTESCJldiESo09hcUR eDSgfa7t+Go5lMKbyCZQ9rL1SIsvXv69VY9JCexb9ILbYe+X9vC8+RoA89VBDi2KQCQO T/gBbU2tydlbLWWVp1rjHsEysEOtMm6axdNAXkaIRUwHSpkoMIdbvZ82cHaKEX3rY6OE nSyY7eTa9xYUCxtzukKyOY3nauWjY/G4wlH1KKkt8J7cYdRussuTph8MVF8MJO9qR+Ps hPxg== X-Forwarded-Encrypted: i=1; AJvYcCX0YXrAZwVUEjHKnZ8vu1sLLKrkPjlMd6nXOAUEij3mkrCoiN/+TJEq2AVL0KbYXWqp30ZJ39nZeksoNw==@lists.infradead.org X-Gm-Message-State: AOJu0Yxc38keva79HKWaZRBj0Xvp4NRJEl071JozQamIV0bq0naolWPG scEIFQMYiFL2lgTuJbBV0qj7vR8ha4Iqeu7aoBmEep1STlMsM8PdxW0vfQU27C8= X-Google-Smtp-Source: AGHT+IFXcE4yQOkwclhHZgMus+XU93hkWLBdlkgkjwLbfRe2UgNWPomwyzpyojRGnPIQGSxet5Zozw== X-Received: by 2002:a05:600c:3b04:b0:431:6083:cd38 with SMTP id 5b1f17b1804b1-43283242a52mr72748635e9.6.1730645760870; Sun, 03 Nov 2024 06:56:00 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c116ad6fsm10730912f8f.98.2024.11.03.06.55.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:56:00 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Conor Dooley , Andrew Jones Subject: [PATCH v6 04/13] dt-bindings: riscv: Add Zabha ISA extension description Date: Sun, 3 Nov 2024 15:51:44 +0100 Message-Id: <20241103145153.105097-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065602_795704_3AEA30B2 X-CRM114-Status: UNSURE ( 8.50 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index 2cf2026cff57..db062107823b 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -178,6 +178,12 @@ properties: as ratified at commit 4a69197e5617 ("Update to ratified state") of riscv-svvptc. + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operations + as ratified at commit 49f49c842ff9 ("Update to Rafified state") of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instructions From patchwork Sun Nov 3 14:51:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC79CD11100 for ; Sun, 3 Nov 2024 14:57:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NNWke5xO+VWGSSb67AL8GWsIQ7dfZLJImYZEcOa4xhM=; b=Tx6CEt149O6Id+ SK+x/8dfysAT3nge25Kw8w56HcfhOFWydkbTKdonbrzp24JOjoz4FO4HwFIE3W8MLkwL4OxXRnPbw XrVpQSVuB85fK8jziOE/hXDt1GI/+0XcfjTfvd0YU2xseYpZac3iVfPWPG6toPitrqsRX9ocwhfTE cNBHOXfZuWSWh3kMR3U+qEPFSPLziOkKb9d33KVda4blCYqxh+7IfM1rQ8ZMPTIC4ymQoDoyLgxmt rbREIXJzZ0HlbrLzMXWEURp+rHHx3FtTCqFVItSneNH4bunOn7zZBhSnulxt+hg1n3n9WHHCkXb6x HtRpUCz2UT8Gz1v3z53Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c2N-0000000BZsa-0Ohg; Sun, 03 Nov 2024 14:57:07 +0000 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c2K-0000000BZro-1KCB for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:57:06 +0000 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-37d63a79bb6so2268955f8f.0 for ; Sun, 03 Nov 2024 06:57:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645822; x=1731250622; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LcjpaetCkzwKl6qLE/ePzZTB5CzyEYuJ8vIVpAvXySA=; b=SOThoJ99/423vy2dK2Q0j1tRasf3KjR5Ih30KbHCXwlwWugPF3O9wKhoPa/Gw7Kwbn Pe0MdRSvE89YcrchTGZFPvbMAwgwCA0QKb8Exp8WYvN+HVzA6wPEo2W7D32LdnqLhbhy Ypnobss8v93iXqUJVe0wTz5JYDJUHKRvvXDm8NRuh2nGZGSW+jqfhanHC2TDsEkXh/nI AmMNcKttVcWdM9/oFe3CxMyiCxiCPCDMbBA2oYvhFHu0i5tEQmCpOgPJs2Epn6/1mbiK lxAunpc6me9nKN5SYiiBWxKRO2Yq0KKPxveIWI6hslCDIF5IRIRxiqNeD5qnFlsuLJnR 3N0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645822; x=1731250622; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LcjpaetCkzwKl6qLE/ePzZTB5CzyEYuJ8vIVpAvXySA=; b=LfeMxokcwcJrWH4veHsMWsAj7VofmN2DUxyT6O0EsPjYPIs69N2hQdsR6TOw77OS94 iRwvUqZ/cF8a1tXspPbSVD7v1mFQOkh0sGMnj/5V0YqZ/LRt2Y/Qczj3IBcE+xEY9bdY icgPfDZ+TMQFcI9cFMn8R/pICPDugeMIwyMFgTtQ399BPtvN7jDHqw+pe8l552lDoChf NOEk8F1fkcb63iEF3NCvnAHEF9YHLwboUPnN2GiYBpxk9MDSvdbTV0/tCk2yVUDp06M5 daMB/OPMToILm3S+V8OA7gIM3RM6/S0jKmjLSI/FqexKFOXP89eAO5i3rEW3CMtB5Jg5 KYAA== X-Forwarded-Encrypted: i=1; AJvYcCXtqcV6/ZqyO6ox79bWTRGqVgZS9NHc25BOGKJgX8fDuImb3aURdQRYeMKSJ1GFV+Jg0o2OnfHk3VaOUg==@lists.infradead.org X-Gm-Message-State: AOJu0YyUwLQzfUug4q0In53ZPF2RbeGNI84hVaPaLB9fFewDLWw9qJN+ C7SZN+5gDYiWh72Kpt9B9F0h0wlg4f/ncO70mGbWgn18/VwjXPIVP6oAIx1k1PQ= X-Google-Smtp-Source: AGHT+IEk9gKHRdbGlFKX3OCkRNc0X43TINOjb8HRMwojR0fr3EGR3rRYNjyJ35dWqeTb53PfklFgKw== X-Received: by 2002:a05:6000:2805:b0:37d:5173:7a54 with SMTP id ffacd0b85a97d-381bea27660mr7207035f8f.52.1730645822142; Sun, 03 Nov 2024 06:57:02 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c1168afcsm10825859f8f.91.2024.11.03.06.57.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:57:01 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 05/13] riscv: Implement cmpxchg8/16() using Zabha Date: Sun, 3 Nov 2024 15:51:45 +0100 Message-Id: <20241103145153.105097-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065704_364894_A028BAD5 X-CRM114-Status: GOOD ( 15.94 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 18 ++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 78 ++++++++++++++++++++------------ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 72 insertions(+), 29 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 3542efe3088b..668be90a42e4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -632,6 +632,24 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 3700a1574413..9fe1ee740dda 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 39c1daf39f6a..1f4cd12e4664 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -108,34 +108,49 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ -({ \ - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx = (ulong)(n) << __s; \ - ulong __oldx = (ulong)(o) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z5\n" \ - " bne %1, %z3, 1f\n" \ - " and %1, %0, %z6\n" \ - " or %1, %1, %z4\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" ((long)__oldx), "rJ" (__newx), \ - "rJ" (__mask), "rJ" (~__mask) \ - : "memory"); \ - \ - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r = o; \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ @@ -177,8 +192,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 46d9de54179e..74bcb0e2bd1f 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -93,6 +93,7 @@ #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_SVVPTC 86 +#define RISCV_ISA_EXT_ZABHA 87 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 3a8eeaa9310c..5e743d8d34f5 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -322,6 +322,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), From patchwork Sun Nov 3 14:51:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860421 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13F79D11100 for ; Sun, 3 Nov 2024 14:58:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CZmva5+Tu9/zuslc+nFimYKC/KMc3X2MSufFIAOPpK4=; b=qWPe/gwbaCrbal GLO+3kMbRsjpDfSHlnUS1x0FHsE4/s8GHSHuJnrgcoPmXlbVgRPKN2nWLPJgBBq6jy6w6jtMMUhxy kByW698xjQqb1P6W4yi2MFVBDXJk17SJz2gaBZkwcI6hMYdkr6Dy31u5kMj+/MVQN/rKj6NKfXWUk fV8eMLAgY20MphNi5rUWL8KAWVrDID2PUFofad1LHO1kSpr/i6eLYNJXxNT5H/XWcPOeumhL9AcLF XcNFwKv9KYLLZdpLBUka2kM28hlOadO1TTTZj45v5oJ3Sf0tz7eIpHIVAPnDl/DY+LmpmiT4Zo/t3 QRreRk4HHz66uhTu5F5g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c3M-0000000Ba50-1LJn; Sun, 03 Nov 2024 14:58:08 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c3K-0000000Ba41-0DVv for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:58:07 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-4315baec69eso29042455e9.2 for ; Sun, 03 Nov 2024 06:58:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645884; x=1731250684; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N5AV+nOoC/tuklQRoqzKvrTd6uHJ4OaRW4xZaKd3wLE=; b=M8MJ/pv158C5ccGAoLsRPI0dQZ3FgNaLFutn8lyASRXfRcIZ0u2U/R2iH5skuPG1CP Ipe3oAiTDOTZotkXjuap7m0N+8umQrDjTDPCmnPmLSHSjFHMGswsZoqGb4Vok6EROEkj NN+pAJGhNUtuuhz9NWdi2esJa/A1g6xVsc7RC6MyQiZacgNrehMNBAoVetPMJ0yVO56M DKXzrdxEJKVEXhghHKrHiJKIbHHJsUqn3hvxx5G6tOTy1V2SLLKmj0Vyl/OOKJrWycMG VEFC7olUjRRMPDbvMpD3Yxg5Ne2HF0S9jyZo95OH4OLtf/YLEiZpFjvNbKxUCTFScG+7 0BUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645884; x=1731250684; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N5AV+nOoC/tuklQRoqzKvrTd6uHJ4OaRW4xZaKd3wLE=; b=nC0mPlpE5P8twRdVNhHvJ+031CEZw3In5mBHTdxcCrNyQQEYq8Vvsub3tYcwRLnizC 8Tk9IXfhhDggcPdu2cpr2y+LnbHpsg2mq04pfIK0DGTaWpHUgPq1iwJTy1ia8FCJQkv8 qtdFoaKRMaODaPdmvd4kzWOj8G6uzrHtQbjAnvYHZsLr0kdfVMs8tnMWB6Bbs9/jJtgt aD1iMk1+C1qjVMCBVq1I1qVuwB2crzBLOe0lRDiZ4ZyONCyk2hX/LZfhWsEWUyB2B+/X mygrQUhHjjgCAk1sZJwwKW8sdQ1Pqap40QDu4i1I6IXWFw7gwFlC3AWFraWP5nsUnqoD gbFg== X-Forwarded-Encrypted: i=1; AJvYcCWfHzygQpdY0Ax1Zz0lbubdJmy3PjpU8KQjjIG1daS+DIbfVIswC27PN9Ib3fQP+RnLsRdefjj3lzXA2A==@lists.infradead.org X-Gm-Message-State: AOJu0YxGx4fnybDQc2LtHNfZY7e1WuuSoScLdG6NVjfm/aa2TkHu50fV wqHg1kI/9x/V4WmC096XBp+qYhf5+2qo/VucD03fjW23xzft9qchnYA2NWjWGxo= X-Google-Smtp-Source: AGHT+IE55nBQ7rbTlkCqdHR3oJ0XymsMVwn2Z7Lo05//Ad0W+sZxb56nu4edxwe80UlQVGkf+GZkPg== X-Received: by 2002:a05:600c:22d3:b0:432:7c08:d121 with SMTP id 5b1f17b1804b1-432849fa03amr81532075e9.12.1730645883736; Sun, 03 Nov 2024 06:58:03 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d5c65b8sm124358355e9.18.2024.11.03.06.58.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:58:03 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri , Andrew Jones Subject: [PATCH v6 06/13] riscv: Improve zacas fully-ordered cmpxchg() Date: Sun, 3 Nov 2024 15:51:46 +0100 Message-Id: <20241103145153.105097-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065806_235611_C74C9FB5 X-CRM114-Status: GOOD ( 11.12 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 92 ++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 28 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 1f4cd12e4664..052418aba11a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -107,8 +107,10 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ - -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ @@ -117,9 +119,9 @@ r = o; \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -134,7 +136,7 @@ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -142,7 +144,7 @@ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -153,16 +155,19 @@ } \ }) -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ r = o; \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -170,12 +175,12 @@ register unsigned int __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -183,7 +188,9 @@ } \ }) -#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -192,22 +199,28 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -215,17 +228,40 @@ (__typeof__(*(__ptr)))__ret; \ }) +/* + * These macros are here to improve the readability of the arch_cmpxchg_XXX() + * macros. + */ +#define SC_SFX(x) x +#define CAS_SFX(x) x +#define SC_PREPEND(x) x +#define SC_APPEND(x) x +#define CAS_PREPEND(x) x +#define CAS_APPEND(x) x + #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(""), \ + CAS_PREPEND(""), CAS_APPEND("")) #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(RISCV_ACQUIRE_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND(RISCV_ACQUIRE_BARRIER)) #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(RISCV_RELEASE_BARRIER), SC_APPEND(""), \ + CAS_PREPEND(RISCV_RELEASE_BARRIER), CAS_APPEND("")) #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(".rl"), CAS_SFX(".aqrl"), \ + SC_PREPEND(""), SC_APPEND(RISCV_FULL_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND("")) #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) From patchwork Sun Nov 3 14:51:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 51BD8D11101 for ; Sun, 3 Nov 2024 14:59:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=w+0m1strbfPEtryxkI82JSxScycN69YCQOF6L8N2kAk=; b=1QTkW/3PD0VaEA mBfQ5iL4zSOx4S/XHraeeUKqX+OC+V62BGCxp3tG4ar6DPlkPB+7JR1lTBW/dw7/2MuU7a2q2EQEh 4bJPs+E3eObbxdDRP8/CXu9+JIdwL1i7f//5lLC11xg7PLY7wlc2gasjPTGhFQ7K7JG2ej5SpZiLL BqLgdxW5pvm4Eq2v2gq43f4pSOoA17Mt0JN72718EvldvgRjJQMUr6TopU6437tr+/ED0a7OtfTqF 9QfKrz69qiJOE3gOnFQ3iL2tX5Jg6fzkUu0H8ijkD3RNSoumh3W2fkvCiaYxVVJN/LKUzFnvogf7t s+tsDGm0fNeNqxzFHyjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c4L-0000000BaEk-38Uo; Sun, 03 Nov 2024 14:59:09 +0000 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c4J-0000000BaDx-03mJ for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 14:59:08 +0000 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-37d3ecad390so2674246f8f.1 for ; Sun, 03 Nov 2024 06:59:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730645945; x=1731250745; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2YMddV/TixlRcMkBqzVFy3RqDe67DeCntBmj39JunYc=; b=QlYGGIMtJxTdKyufCcgVoKEyDIEMWtf+WgK4eFsfY3m1Lde5HkY/++H0Ucw+3yhb2p +t3xd90kNg86wGV2aKSop0tUT7BLq6p3V+ya1CRfzjlLELl2Ew56ICfSnA4buEZ9FYPD jQxsaOFRv+3s4A0zt9HXM4zGUjxictLthuUCfv9jAKRRfxJbldcCMQSCbELhbHTpu/bp 168aFzj2p6zLl+6s9uBSKkoJE5Bt6/kfG453yFFFY5Qn7nXU22yG08ITADen2A+Synz5 3Rw7q7ZaTSlGMCaBl5+og7J8o1IsIbd3xm5yk/RIgxfpUTc8fJA6YYxFVkkvdAhSxES/ whEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730645945; x=1731250745; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2YMddV/TixlRcMkBqzVFy3RqDe67DeCntBmj39JunYc=; b=OMQGxZCOeSvXBphavEUDiVQRQHUNTa170ssMHFkhrVo/RhZoqTlJZ1dNElP3SoJTnB UgLn0zF2RKVP5VpdR4+v6rSYXU5IzQNtZ/qEVlbyDMAMFoGV4wSM7Po/X+IrW5os4kF1 mp5yE48Un2ww4cjEj/6RQYpy/j0tZe1X7Axv+2fDERJUgfGeMUvMvcS6RlmDbq1WXxdp OzykjNP5A7jqyadHquPf0zJ/7QJJ3kiVWqetyCn56SiW5+fxScHQDZxWzH6sfaAFcuZI MuLaEZk1aja80SgUiq18adQlDxuaA3obdxHQFnTOXMTFXwyAsvr55deRoPjY2lKw105R E9VQ== X-Forwarded-Encrypted: i=1; AJvYcCXWbPEHf2RmMEfMTAjN8QU1ACgxftA2+I6iBAm2wdYlEwLOJLLwDyFC6TCHRM6uHuvwnSjpszgnZXD7cA==@lists.infradead.org X-Gm-Message-State: AOJu0YwKDc9cs85Wxc/IkCe4vjs5e9eu2LlUXCShsKa261RIv7aCHuzM xlt+vfBjP3Ym6BrzlazIrI434l62QIdneuXdLfmwg739oHaxYo8dfBccNySZE9Y= X-Google-Smtp-Source: AGHT+IEwAhzLmFndg4dtG6o7eex1DY00zUsKhGi9VFC31IF8dEpvf7rYFDq+11tOCJeycg1G7RqbUQ== X-Received: by 2002:a5d:5a8c:0:b0:381:d133:d541 with SMTP id ffacd0b85a97d-381d133d9cdmr3234403f8f.11.1730645945038; Sun, 03 Nov 2024 06:59:05 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d6852d9sm126046165e9.37.2024.11.03.06.59.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 06:59:04 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 07/13] riscv: Implement arch_cmpxchg128() using Zacas Date: Sun, 3 Nov 2024 15:51:47 +0100 Message-Id: <20241103145153.105097-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_065907_060367_F3E066AC X-CRM114-Status: GOOD ( 11.99 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 668be90a42e4..093ee6537331 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -115,6 +115,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 052418aba11a..f95929f538b2 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -296,6 +296,44 @@ arch_cmpxchg_release((ptr), (o), (n)); \ }) +#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) + +#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o = (o); \ + union __u128_halves __hn = { .full = (n) }; \ + union __u128_halves __ho = { .full = (__o) }; \ + register unsigned long t1 asm ("t1") = __hn.low; \ + register unsigned long t2 asm ("t2") = __hn.high; \ + register unsigned long t3 asm ("t3") = __ho.low; \ + register unsigned long t4 asm ("t4") = __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ + : "rJ" (t1), "rJ" (t2) \ + : "memory"); \ + \ + ((u128)t4 << 64) | t3; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ + #ifdef CONFIG_RISCV_ISA_ZAWRS /* * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to From patchwork Sun Nov 3 14:51:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 452D6D11101 for ; Sun, 3 Nov 2024 15:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=tirAkl3dMM8AqneL1/oZOXYeSAPKsmU+6/bQ+/UiY8c=; b=GSVWEV5fMdJXv3 uDr396SfDd+geMv/BSdX4FahUrESnXH5c+1YY/9Fmc5LFUgskZoxv/xGC2bmEFyADqo5zb4Kg3SL3 qHM9H+LjvyavjeGkYTqcaa7qi3uSG/cj0O2/Klt10ywTuU/NEwPVvjmKXPh+TWgPyglBV1hXT3TEu qTafM2CO8ooPoxlH0Sw/Itxq+iRV8b59771LYdV0nPk6MQOlX7Bp/kU6DGqJ5ilqeO6UrWtmmtjXS zL+SUwsb1F2Z/fSwDL9AHArphTRiGLT8UlGB1IBwLdEKlMlQDTfBiHWjO7OqYQtnDnitXPY+NNYE0 MHHRBRZC+DKWSbcDK/7w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c5K-0000000BaQ6-3VnO; Sun, 03 Nov 2024 15:00:10 +0000 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c5H-0000000BaPC-499C for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 15:00:09 +0000 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-37ec4e349f4so2144579f8f.0 for ; Sun, 03 Nov 2024 07:00:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646006; x=1731250806; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C7omRwb2buD8uK45bQUSORyOPEIU9zvIfXDatLe8IP8=; b=zwn8cPbBWDQxQcDBagA+n9hHJ/Ns10N+V7Ay54I67IQpTadho3MEKUVtyEB0s6G/V8 Am0o4USu1xlRPXl+2b93CNIHbqynIWKqzKkTN/6ZU65cBc8dgrvc//+9SLNglVW/5PYv LOVcLoW9o9H5Um8ACAZX4oVIiPKVkrv+g+ioF58/hlPz9zqgAIBu92QwMcGiICxt8T3r JRbFobqCF1G4cd0B/UYtnTz1/LFsbXdIw1GCWQvqp9L7HemLXA8PJq8+b5RSE7qLa+eY 3/TJX6Ko9Ovy4y0EO37IuGjf6XmO6NattJNN6EJTwUa6kSGBL4W/plmrkqygESsfpx2j aVvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646006; x=1731250806; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C7omRwb2buD8uK45bQUSORyOPEIU9zvIfXDatLe8IP8=; b=w2Jja000hb0S7eg7j4+5JYU/k3EvxDDAOdt0nBeOYyZxD43D0wubvVT8Tdqrv+/YDD GOh2Ft+foaZdZT2gnJeZYp08jECPy477cT1ja6qCcv6hbFNEkNtP3Wzrpbmeuwz4ecST ir1CLTpaFpgzD0zLEO1vFtfUGv4c1Hq+yGCEWBc7bEvvGRIwoB+wJyCTo0uaPr4zNfRL nqT0HbJVYJLu9d3xDfmPF+ysY718aRg6KT3DDO30SPUXXM+JO5XYvW64zCLM9RJoZeVg numpdOWGnc6KGXpFRT276Sl7dnj6jql3IXjWWOu+5AwGI859xlWw/xjh+49W/d/1esni BT8w== X-Forwarded-Encrypted: i=1; AJvYcCV23R14rI4MxqYKS/TP3daUzvgdKMM/s5qJjAV4RCyUKHFCCSdeFc4hpQAk0s3jyFDyimrbfdhrNc/gAA==@lists.infradead.org X-Gm-Message-State: AOJu0YwNp7FcLws89w4JBcIUGJbPkOt+GFkfvnhDkAtXfyEVp7iPgO3o b/uRRubE91GakMjFLHQda0FZoAM6vgqs3R/3of3jepnCeFzOwI43n79A+aVK+V4= X-Google-Smtp-Source: AGHT+IErbU6/+ESL9c7D4Rkx4JGXztbjh558spmD2njQc7VJwfVBMIIjkXAoMafrlIu7KWoRdehU9Q== X-Received: by 2002:a05:6000:4029:b0:37e:d2b7:acd5 with SMTP id ffacd0b85a97d-381c7a476a8mr7741924f8f.8.1730646006376; Sun, 03 Nov 2024 07:00:06 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10e7465sm10805300f8f.53.2024.11.03.07.00.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:00:05 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 08/13] riscv: Implement xchg8/16() using Zabha Date: Sun, 3 Nov 2024 15:51:48 +0100 Message-Id: <20241103145153.105097-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_070008_057100_00C94FFA X-CRM114-Status: UNSURE ( 9.73 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 65 ++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index f95929f538b2..4cadc56220fe 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -14,29 +14,41 @@ #include #include -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ -({ \ - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx = (ulong)(n) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z4\n" \ - " or %1, %1, %z3\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ - : "memory"); \ - \ - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + sc_append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -59,8 +71,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ From patchwork Sun Nov 3 14:51:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BFC0D11100 for ; Sun, 3 Nov 2024 15:01:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=+5n1CE5RLr0yYJGTWgH8fp+XxA04Fy1MBeGqBqCJMa8=; b=eAZRIMl8MAI5jN lDAJV0eGDhjPb2iJGaK1pGktiVcClcl6ho08Lanb6etwBabn6FyNQM1PLvkCsjeKiHk45mI64k9aF rkUBg3bQK6e9BS0fwI0G4yI63Sz/JRxQEGgA4a1erV4igCS+KdQZfdQoDs5MMXx94JDJcBi41onIQ hXV4IttsgMJkcoOCf1n6faLCTCypbO+We6HUR/hQzLC5+dASxd+2GHcFMsveKZNp0UhNqUZwyfa45 D2xMlsh/pQUR4suWWm+EPifYG38H+6KEryYdCPQWYFR+phUFA2jhy2byvBYTwi9xbwHEjBTfnwfdB L2NUW0UW4qZfcETeABBQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c6L-0000000BaZw-0I2M; Sun, 03 Nov 2024 15:01:13 +0000 Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c6H-0000000BaYq-2nXu for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 15:01:11 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-4314fa33a35so26280805e9.1 for ; Sun, 03 Nov 2024 07:01:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646068; x=1731250868; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2s2g6PiZXcRkI3A8BFgYC9wzyXLIo5JanvDPO+lSWQk=; b=xycMlEB2t7OusjlukEZ7qivk3u4r2OnvhLEpmDsLNmYIxYWpOk51SzjdLFYGy3h6xi r4mNRn+68FFOh/FRBYmEskIAArzk5lE5zgZ6vT/H8AJmYE86uVNzCM5otkRSTJ7j9BH6 XRuUtU/RS1g4RItf+n/q3fORnlUMUvfcWlXnbazv3ocBDXOcitd0MB6Dm4bw7GWCRjsU N3Vr9Qzjzep6sxqUEMjzhYUoErx8/MT5ECaT3Ak1x60QJKQxhKqLF33FaOsxNuTKdQF+ 79hc15WXcRIPZUid4CMO4A6Ial6qJOQERzJZm33v+TvJv84V9Z1jj5DKh7sn35YUvFAl PT5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646068; x=1731250868; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2s2g6PiZXcRkI3A8BFgYC9wzyXLIo5JanvDPO+lSWQk=; b=Crf45e79BCXl2mRXxLDPsZdqnujDrZu3h7WMwlJoz9nMHCyX4RLkxWeCMRO0BDe70J HntcMH5m0hhL4tDX3t38hRjDN5//c8T3jBKEjxhiGq9zTKRp0xDaU5AyGnuSvRD1ThY3 tt1YhikftBdCao+a6BTixNMg3hc93CJUjyDRUE4JHwyHGvoS5dehnQsn4gZtxIaFRy21 ogG1VaYssGh7lsB6dUYuSSxHO3KMsN2LIbj1BE7HLb6+LAokNLoYYEzdtX6+2KoHumh/ K2XJDFc2ONm7fff0gQ10A9ST8EmiJ/5kPLlL/oRhwsRuQlucJpV4GJ/tnhlLX/TTJD1t qS5A== X-Forwarded-Encrypted: i=1; AJvYcCWM+IwfkaZIxO0vPXzIR6KM+yHwupCZvbbx4iqE5Ehazl0LRE3rQ9I9sp7hJsU4czdtj9vBqNKdeHFSQw==@lists.infradead.org X-Gm-Message-State: AOJu0YzsxBknJFdUuL8QGL2EFzlKy/5Jp6QfcrtZ/X+BZACZGx9yVbml kACM8YfRGRryizOKQ4PhgBWNcBOCIklGsGOXfpCMjfBx4b6NsfIpWSq1OUyHFkM= X-Google-Smtp-Source: AGHT+IHT05faWmtqnYuiNBzL9haD0/7Dkr0QGrJhaTkS4osZpzeT6/FO8hvYgpCC2wE7TfnKCk1UVw== X-Received: by 2002:a05:600c:4215:b0:431:55af:a230 with SMTP id 5b1f17b1804b1-431bb9e6031mr145075625e9.33.1730646067731; Sun, 03 Nov 2024 07:01:07 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4327d6852fdsm125977405e9.34.2024.11.03.07.01.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:01:07 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren , Andrew Jones Subject: [PATCH v6 09/13] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Sun, 3 Nov 2024 15:51:49 +0100 Message-Id: <20241103145153.105097-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_070109_740072_08938F73 X-CRM114-Status: GOOD ( 12.64 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Sun Nov 3 14:51:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CC11AD11103 for ; Sun, 3 Nov 2024 15:02:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JpD3j3wJjL5a+57faERVV8pLtClDjpqrcWqql+lm3Y4=; b=Fn0l6NNQkA+QGI gXdnkGFbFiY/InxYY2yi6rZxpFoj6Hxnvgt36/9tteHLtbUKMdTdm9332uT9TQ7zr2ZJ10Hd10hGF ldnPXEzOKEw9yXVXpB/xv0Jqncf+11HBUVZdwbFS4GSeziLAZ31CggRlUS0mHuwTZy/UZGw3//wRm b6Z1o0cfP0mRs7BEgUqfpqNf7wnZApoAIXJ/i9gdSH9JhR6aIs8m2EPepTMZ7glSxHjeFSUjYrcjO 5n/dfM5+kZ+vILm4c8UrljQApj3U8yhkrtn3nDw04voMDdAe2wJBZK5oCX695V+Tc+J2qJztnWUpR MZNz14BXP7V+GoGW86zQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c7L-0000000BapX-21VZ; Sun, 03 Nov 2024 15:02:15 +0000 Received: from mail-wr1-x434.google.com ([2a00:1450:4864:20::434]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c7H-0000000BanK-0psa for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 15:02:13 +0000 Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-37d49ffaba6so2313859f8f.0 for ; Sun, 03 Nov 2024 07:02:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646129; x=1731250929; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hiozRe0ZQqJk1GBz+SIEaweXO3Ezj4/Y82ena4X838E=; b=WxTno+aE3c1tX8qnNoEvgZUwoSokzK/EKRFTUPDK17IrA9S8beyU6YOuzI+v2qSREM pH5AmW7/S1zFAa7hzg6qxsLHfhVST1XOGj/ihaEPpzB839cVgBQalZsSnW/mNBPBgQux tgybYqDJmQOT2FTNlEwiy+dTFmzGCbeAh6tSbZtFo8Cy6oV49+ysbhZHU7RIORmmFJc4 2xL+hY9Fi+gApIVULAOtAqeF8++rMS4o/02bfU7e15cf1dxMd98S2TSQKJK0mHRlf1ri HHleBKLQk9IblQo2vXgbtZFIpW/hrd3IYtdRTJQctSRGwo71mA39LsJVe4ATNOLp1Hwo zYOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646129; x=1731250929; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hiozRe0ZQqJk1GBz+SIEaweXO3Ezj4/Y82ena4X838E=; b=hu2Q0+0jVqZvO7TgHWYLzHB2kYrohflGh92zSRxaYuqeVDL4/y39ttTw2l78tz89eO zxmY3HfeROVl6G5R8UOzOp7vPKGlwc1yQmR1RotdWtP6odbygVtMdVLkIRlsez/ksO/k pSQxAHLulimT36GOqFifC8ub09Dv+mdvq2AfR9RedhQDD4ZyYqAcHiAwMCgqwUbxqY4h vdtVdnFTijDsBGniCGy6oeuvpu5GrLZsMvD3dM1g72LZXbaK91ZITvo4Mkz36Fv4gduD VxyxNmJbhmsaaFfVg8I9mWvq/boVL0oiVaz6ANFhdQmxDBiHrIcsAARRJ5hpvRmhxPFQ Q3/A== X-Forwarded-Encrypted: i=1; AJvYcCXpXAt1NcNBMmPC8JyKjb6VQIoVYUiP5r86z6lU/v1eov1RPhnwzgS5tbDMaeVsgjbwusTZ1HFrkHqz3Q==@lists.infradead.org X-Gm-Message-State: AOJu0YxTcYJ46xg+6lZFoihMGQ7VkHvfEHtmaC7cwnhkpMxfrPGpTr0q buCgJ2n0oeebS01v6B6Kvqzi7wF4z4Ed0eQtFddmAvlVqSfI3XZvlI6k1Dg43AI= X-Google-Smtp-Source: AGHT+IEWaiYi7foaZ2KTG2kLPtUDJP2UndtLmP9dR8kSWRyDgH5SEK/ylfyL6hxCJksB0pHYX//hgw== X-Received: by 2002:a05:6000:1b08:b0:378:89be:1825 with SMTP id ffacd0b85a97d-381b710fbb0mr11405436f8f.49.1730646129165; Sun, 03 Nov 2024 07:02:09 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10d4b82sm10818939f8f.43.2024.11.03.07.02.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:02:08 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren , Andrew Jones Subject: [PATCH v6 10/13] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Sun, 3 Nov 2024 15:51:50 +0100 Message-Id: <20241103145153.105097-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_070211_271644_1D7BD2CA X-CRM114-Status: GOOD ( 21.34 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Sun Nov 3 14:51:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E0C6FD11180 for ; Sun, 3 Nov 2024 16:15:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lErWCDkJ1DF3lggUluWYjFo762dOJvVXJS7XmmCOBGI=; b=NQBE+OvLb2ul3y QxUJw2rRpbijDsLeuFzg7plhm1gQ4+P4BFSmFhoLje/k1CP0CxLiAp60FRE4mMA2m/1zDeYwC4yNh eqb6+wVllIGlPzVjJzjybZkmIzr0HktyUS8EbdaFe4T6c29nsCk3pBZ9mCDP2z4J0owtEKpKx4R0E dW3JPFRldiEbziqxIjjR25PW8YDLH0q2q6MXv+3xDoxM6zrGKDUloWBTaiPjjAcBtR2HyTbH+oYhT 1e0RJO53HyZt1uPRZG5sfi2y7ElOGuv5VWnPHd5aTScvuiQQmCZrH6vfwRvrpH4Ib0A5aZMoSxXHI sw/XY2zzXwzr50F7BqCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7dG0-0000000BgFv-196Q; Sun, 03 Nov 2024 16:15:16 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c8L-0000000Bb0f-2xea for linux-riscv@bombadil.infradead.org; Sun, 03 Nov 2024 15:03:17 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=51tYDRwP1WvxpzmCthvL0bau6hmUz8qjqFRqrLI9cd4=; b=gl7gu+hZGoCJ0EdwYDTjzKKDBj TmNVsbfUaQGjwppSkynZZE+Fbtfoah8ufiGxdM90bHTv+tMnuJF2s4E1ME1gApCv9NMproKo5Ov7p A6rc1M8wadUfSqaWgmLplkkpMgYV3LX7JEA73OXqRJkimZl2K/NYWQQc+PLUS3Nf9Sb3iWKjh8sJD ayjkmmTtT4zbojI07YYPOVVWOYbzyOs6HyRfTpn2J04XAPKw1AsCjzgfcsVk0EBGRlCC4dQXl4+VO pwTkPw+qpE1YWl/cc5ZXF8R9Bdt+tlX8J1CtnyuHJNgXlaHWWNfSkG1KbT/1F1cV+1/M+CbRNpCW7 QrUq6lrQ==; Received: from mail-wm1-x32f.google.com ([2a00:1450:4864:20::32f]) by desiato.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c8I-0000000B8UK-0RhQ for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 15:03:15 +0000 Received: by mail-wm1-x32f.google.com with SMTP id 5b1f17b1804b1-4315baa51d8so29918095e9.0 for ; Sun, 03 Nov 2024 07:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646190; x=1731250990; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=51tYDRwP1WvxpzmCthvL0bau6hmUz8qjqFRqrLI9cd4=; b=gSFmJZcwZPFR3CTPCjdDtsJEac4evplsetr5FUnoigIuB1U9GO40zgpher9HnHDtQ/ Ws78lIenhXlkkfKhZxryswHiv8Lv/SqlRV3fvMLuy6JD6Feerc+Y6X4mxxpWWEzPgn+c eXpIodpas2Uj6J9ghJHxSeqQQpHVMYduc503z6L0yymBzlkxzfQ/l9Y3CJmRRooMrVto vEfhgmmupYLrpKo3rfQUhqn8OT2q8+GMuhWwsu8qgjCF/A2CnG68iWg5LemQQ5C3f0XE El49cpYbEdQMDiOpGn8DbOxH+0zuK8V5WapBZmWpQH49cRgrvFwVFNzYfe7kvbXcj3l2 DsPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646190; x=1731250990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=51tYDRwP1WvxpzmCthvL0bau6hmUz8qjqFRqrLI9cd4=; b=FjnIsGahn0OvPX4PBH/kfjuEUos6OVQbyysmXpE3k1cIPWx5RVXkurp+eSPBKcMKam HGg1C9f2SmfdGhVqi5UHLKU8AaTHLn/ALvWIj3ehU1SNsiVWMsh4G3Bo2I0k14WH0EDo a10e1x22pS7unIui3mSaFvDCGXksDUEL93JPmqs0B2FatofIXduzPFSyryEy1MuJRwbg +9JZlHXqOfrspkDR5NI+kFr6Q1+YfWOSYzvbn5YMqsgEtNSf2bjjCcpxAji88uCdDlA7 k1p0ZwI7DTpGX5cPnKw8gp5Uq3ybOjwt58OlJ4FWkFn64rT7HewmGyzzHItCQFw6MDmt +vGg== X-Forwarded-Encrypted: i=1; AJvYcCUV4C0zy1ZDIlk3Te2iIbniw5zzIjeNEoYvSQZCnE5ow3mqlx+svKNalW68dBA0RfsUVBpA2GiyrZ4NaA==@lists.infradead.org X-Gm-Message-State: AOJu0YzbdbuHxhuN5Y0mZyd8+4oRYAiJ/Iyn8j84mYJ75GUybO1aN9dC 5Wr1FSEB+1RFIB14nUJoOM+urMrlby7oDI2kwEt/YwwC8u6UqGw3SejIZ2cLPIA= X-Google-Smtp-Source: AGHT+IFqldFYKHqBiRaB6ae6bURZkYwVuzbn5bQ+e+Hym9E5heEep9SJOfB/eoXxuGPgZenIG/xzCA== X-Received: by 2002:a05:600c:1c9e:b0:431:5632:448b with SMTP id 5b1f17b1804b1-4327b80d3c5mr115258455e9.25.1730646190486; Sun, 03 Nov 2024 07:03:10 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c113e5cbsm10816042f8f.80.2024.11.03.07.03.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:03:10 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v6 11/13] riscv: Add ISA extension parsing for Ziccrse Date: Sun, 3 Nov 2024 15:51:51 +0100 Message-Id: <20241103145153.105097-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_150314_350006_518D99F6 X-CRM114-Status: GOOD ( 10.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 74bcb0e2bd1f..0aa3c3f5e682 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -94,6 +94,7 @@ #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_SVVPTC 86 #define RISCV_ISA_EXT_ZABHA 87 +#define RISCV_ISA_EXT_ZICCRSE 88 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 5e743d8d34f5..5f453a039ec9 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -314,6 +314,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { riscv_ext_zicbom_validate), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, riscv_ext_zicboz_validate), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), From patchwork Sun Nov 3 14:51:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4CE4DD11101 for ; Sun, 3 Nov 2024 15:04:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AIqCgQ3a5MlRFTxX9TApAn0odLlqVUBZyli1CgmUCCc=; b=24c2Jhc838Z+Dt wwoLe76noknnMZRWfFOnex4WOXPzMVQ3LAnoyPFrwMxwDAgqls6KuS3ZiHHSHy4XCvdkN11Ogl5OM 711ZSSWXn+vKl/K+s0qOboyka+yvc6Q6J07FrfP9Z3Z1Z2s50ZFltBmga5B7mjTDNX+Pstuj1Q/Lm /OR6id62n2+im7ykBVIfJyhx9Kc2OTE22UX9nzUr+Oog5oSEZIhoTEOjZdXWp0viELdhmekYRWRQ+ VuTs3kvesNAFItG4ZkbuJc1OMVGpS452JxGJ+yIJRQGtf33/xmZcAbYh4Ba1ViWtPuDQ9Wt/gjY9R fbVM0ROZ9BbG3UH5Dmlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7c9N-0000000BbGQ-1Oc5; Sun, 03 Nov 2024 15:04:21 +0000 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7c9F-0000000BbFZ-2Tfi for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 15:04:20 +0000 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-4319399a411so29606805e9.2 for ; Sun, 03 Nov 2024 07:04:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646252; x=1731251052; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CYOR6ehP8ZUAtZy2SlX+P5w8jiHdus9J4uApBvZwtJU=; b=j6SnrbvDg6ZX2TNJmVPnB/QTx6u9m0mo6h7i+W2FIukgt8plgD+TTasT3hKhgkCzVO gsC4vYXhP7evwmqTnLkxn2ctYQ04mXFgrYS5RGnhqOfFe/xKht1oQwoG+qINdCWOLtp2 F9C4FE9CkbQSS2EwwwWYYrqfHsmzzhvkghChB+AHk+CCtuIL+/dPaD/lC04xvAVi4Qw7 3dgpBVACmFRqWiv3RFE78V9rwrck4yF0PBGqswy1o1zf7tYC4KsF9mlnTAjPJGEG16VY bSTRjjWX3xbXF1qZG5eUnOSH4ocanNRqc1SLyN+K4dU8fiBVFAinfmwztRimyigrTnU9 EHeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646252; x=1731251052; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CYOR6ehP8ZUAtZy2SlX+P5w8jiHdus9J4uApBvZwtJU=; b=C64kthg3qZ7e7+QaaR46eMhwC+6D6QBXzIZWaPTUmqcWUTjOpaNcB7q2Rw1P5JvaHZ EcZixftmfgoskdhlCInvmhUat7KIsLUSyRMHeVbXbakZHY1i1WtdtvXNQfDix93AhIiy q/mdmvjdaNtJXavEGzwW9CReNeOWnRJT71Ok1G+nfP1T+JWbbElBs8GRyzGNtmSstpqz hhLHz7puvbdPCCs+WSR80LKqN8/hkuznq86tJD3e43Vrgvw4SsyJS03HJUTg+Dv8yyLf A+1QcoNnlCQw0lU6fOlbIEr1XX3cDbWB2NtkiNP0aMlCpq73efFoT0X9Ec019OT52rUc 5vVQ== X-Forwarded-Encrypted: i=1; AJvYcCURdcffX+cqABvJjgWYL3uCA5dUCuPn2Jcsr3tLa5aIop+jXZVrl26iuyISMWCpcY+PDrH+EUaOfX6cXg==@lists.infradead.org X-Gm-Message-State: AOJu0YyUv0M4xofobkJn06yVvIqtOgcyjK6W7eNsAah34QIBKDgY1O8u lHuMIEADCXGxZY+/mwkSSOTL0oM1G9JCvwa1Id5M4hMihOMFk2ppGxitDwp/meA= X-Google-Smtp-Source: AGHT+IHLb+bX4JOWUSFmTs6UYPJV8WTvK/tBJRvIgcA7ae04ZGiofBjbX29h/oO76eksCbmobaHszg== X-Received: by 2002:a05:600c:4fcb:b0:426:5269:1a50 with SMTP id 5b1f17b1804b1-4319ac9c15fmr240004605e9.11.1730646251731; Sun, 03 Nov 2024 07:04:11 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c10b7b32sm10858711f8f.18.2024.11.03.07.04.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:04:11 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Conor Dooley , Andrew Jones Subject: [PATCH v6 12/13] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Sun, 3 Nov 2024 15:51:52 +0100 Message-Id: <20241103145153.105097-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_070419_042351_E0830032 X-CRM114-Status: UNSURE ( 8.19 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Ziccrse ISA extension which was ratified in the riscv profiles specification v1.0. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index db062107823b..35a9ad1d7e63 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -296,6 +296,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as ratified in commit b1d806605f87 + ("Updated to ratified state.") of the riscv profiles specification. + - const: zk description: The standard Zk Standard Scalar cryptography extension as ratified From patchwork Sun Nov 3 14:51:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13860437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F2F7D11100 for ; Sun, 3 Nov 2024 15:05:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9Nf+JKpW/VkjOgekODnQBM5wlwRl+aCo34FbszI9IIo=; b=bivXRdEyc/HT6W o+laVmE9Dri7l/xkRL6k5JL/diJ9cgKMcSvPd2j2rRTrI163sRMqpzbJZXTGg9DKSGS/PJPFscBWT pvgWpOBovzDWIAwYe/knJsuK37rWM/n9pGKiTrtm1mU8kRKgyVkUv6Gb+9zQJaEPQoIvuzFN1qMo3 X6bDH6v6qYLUqSMOyjfpnTDh12hN/feaaYUuvGkBpqJqumGqMwU5+FMjMgQlLGZd5dsVkxXDBEayr uEDMT3p4enD0imsD3tyVmMmjBykgOB+rOk0fVYJb6vHg4Wicn/i8nPTSRnwATIUZE6SLn2mK4X3rU rGMuDh0Ci/o9pHAYvfQw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t7cAI-0000000BbRR-2Yij; Sun, 03 Nov 2024 15:05:18 +0000 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t7cAF-0000000BbQG-0769 for linux-riscv@lists.infradead.org; Sun, 03 Nov 2024 15:05:16 +0000 Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-431ac30d379so27662065e9.1 for ; Sun, 03 Nov 2024 07:05:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1730646313; x=1731251113; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9aY9phw0h9f3xNjCboMfKLftvRP8BQ6/29lVmcfPzaI=; b=FaySEaYL05phpGcbgnos7A0sFrCk8CGsK5ZckzhAPyqCJGLaAUp+VKuxdAsWDgXPF1 r681GcuLNABgwnGYjNJN6IRh99tEYlcfUrcpJMkzaW0V0C9HmqxcdxQFFYhvyksyPn1n mw3m6/7Fq3JjQb8fry2iU+gMvwJH5RnOb1B8YjrUf5KQo3Hs0CtF6BXeO36LCQ6XWs11 6bpD2MJS2anKxV+PMBm3rz0KwHEN6aqRGsBtBR0KZfS0XVs/mO8tdfgtrlBHO71HSso7 BiTpNisX7GxyTApxJ8exymvptxOaz7l3aBdr0U7yjyOq221oAekwrds0mwQupJhK6QXL NOCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730646313; x=1731251113; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9aY9phw0h9f3xNjCboMfKLftvRP8BQ6/29lVmcfPzaI=; b=PL8j5LD07dP6VD9VXb6llV1XNQ2jJj+ZFVW68OMLtBfvmSUowmaCPAhZIVxQjMq178 MQVYZB4LDuyXGesBRkYLw9/3n1+T/J3IRpD5Bn026BuYLHcVuFfNL3+JEBhtA7YGIjNw umSni0nCDEo8k2ZLjsl4gXeb4aXW4RkucIOMITl8z2ZWlEik5sXCYxGowXEm2n+dSK+1 A2WwxfhKjbyt2n3pNbtknoITfM8NNFYtdT0cmxmHAlcAjjyXbAbY+DTxSs7xtwhWGhVf SzkCDj7L0bDrl5tAtgDkez3cK44lI4zXUpL7unP0THiVl/IPW2GSNgu0HEY5IxQ0wE5O HoLg== X-Forwarded-Encrypted: i=1; AJvYcCX2j3tZSVluH6efqM+zMIEgznMEvzFaDHpXDUZmAC/StgaLbTr/NUF1cpDdyegsBSGzyk6uDdhb+Nu0pg==@lists.infradead.org X-Gm-Message-State: AOJu0Yxkp/+9whiMWxVLVcj/WSfzrzOQAps5lShk1ef1tf01bLcqp/J1 WiwLFtXzfij0/WLpYKVlMzJeBFRsr9RG8BTNRBkbGsHCXkX0NKPXK9qPsiMTY38= X-Google-Smtp-Source: AGHT+IHPZiSZzcaPz06TDTdXTUesvEzSWH1SxGmB9LrKBfkdnmynnmf9paGsFdY/fWwSFbEmyLWwew== X-Received: by 2002:a05:600c:45cd:b0:431:5eeb:2214 with SMTP id 5b1f17b1804b1-4319ad363eamr237927625e9.33.1730646313170; Sun, 03 Nov 2024 07:05:13 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (lfbn-lyo-1-472-36.w2-7.abo.wanadoo.fr. [2.7.62.36]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-381c113e89csm10899063f8f.74.2024.11.03.07.05.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 03 Nov 2024 07:05:12 -0800 (PST) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v6 13/13] riscv: Add qspinlock support Date: Sun, 3 Nov 2024 15:51:53 +0100 Message-Id: <20241103145153.105097-14-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20241103145153.105097-1-alexghiti@rivosinc.com> References: <20241103145153.105097-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241103_070515_102262_4687E40F X-CRM114-Status: GOOD ( 27.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@kernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 34 ++++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 47 +++++++++++++++++++ arch/riscv/kernel/setup.c | 37 +++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 093ee6537331..f5698ecc5ccc 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -82,6 +82,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -507,6 +508,39 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU && NONPORTABLE + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms + without one of those extensions. + + If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks + when supported and otherwise ticket spinlocks. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. + +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 1461af12da6e..de13d5a234f8 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -6,10 +6,12 @@ generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h generic-y += mmzone.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h +generic-y += ticket_spinlock.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..e5121b89acea --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +/* + * TODO: Use an alternative instead of a static key when we are able to parse + * the extensions string earlier in the boot process. + */ +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS) + +#include + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..438e4f6ad2ad 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,42 @@ static void __init parse_dtb(void) #endif } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); +#endif + +static void __init riscv_spinlock_init(void) +{ + char *using_ext = NULL; + + if (IS_ENABLED(CONFIG_RISCV_TICKET_SPINLOCKS)) { + pr_info("Ticket spinlock: enabled\n"); + return; + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + riscv_isa_extension_available(NULL, ZABHA) && + riscv_isa_extension_available(NULL, ZACAS)) { + using_ext = "using Zabha"; + } else if (riscv_isa_extension_available(NULL, ZICCRSE)) { + using_ext = "using Ziccrse"; + } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) + else { + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + return; + } +#endif + + if (!using_ext) + pr_err("Queued spinlock without Zabha or Ziccrse"); + else + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -297,6 +333,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); riscv_user_isa_enable(); + riscv_spinlock_init(); } bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) } #endif +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */