From patchwork Sun Aug 18 06:35:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B5297C5320E for ; Sun, 18 Aug 2024 06:36:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9kRKqQ5UXLGzODQfxDxI9nK7DK4jncXta0FhxXOfE70=; b=4BltZWecB+VtHw bsqihNe6B3uKtUZPjWqpczAXefxb9N+aTEbwHU4kkKwJYv2KThvw+zwDd3koiWu5Y039Kx/vY6cS4 Ltw65Nag0bEy+UiXgZRPaTyirm+k+VRE7VaNVKyrwIWomVK80ri8oCyfabVs9m6KxM/iwx02GPu22 m3NqaBjueQUiuXVI1fq8wvsqgL+C2A5YAGJS/qbF39rdvXFU59KFJJi5vM7fLCXVRYIqlG/EAdWy2 Tuyat/2x6cJusc5ZlpxnP5M112xvtLOf4p3mdYpxOCYqBCuAZJJmRD8ZVQwCYVEt+NH21QLa2oYOu oSafKQt5FGJofUKMWHZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZWx-0000000GCzs-183I; Sun, 18 Aug 2024 06:36:47 +0000 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZWu-0000000GCya-2r5V for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:36:46 +0000 Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-371afae614aso585095f8f.0 for ; Sat, 17 Aug 2024 23:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963003; x=1724567803; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=sQFy1okCbHFfB/txm3JF8HiMI/S+Ce6ZEkq0uKxg1ynOYSL+jZABwUrU4WWrbEQsWu hO7yfMiqxCyY4fPoowSeme5Hqkn4B3aHNuBZjZCuSks0xoUK5SY/TormZjNs6FRiKiuk NYlp21aWYxSSMK0wuJNcMEESF6lzQlNuj2qnbOTbgsQra2ZQ13/+8igfzhb3FwcHlTBF wxVDMYNK1qlMy2QpVxuH1dAi/dGHmUsf6FdNUbz0QlgmUeHq6YELGgZr/Fr9RBVvjP3w vKL6EvPG1A/rWRUnMdmsLCFodEo4SE+sJSw44PF1EwkQ0OdUzuTyX1TwxkGnxoLpAOnc KByg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963003; x=1724567803; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ll6PO6fq7jekV3yT3oRS1DwKakMdlEGxiDtEOFhGbwU=; b=WTwe5hBkMl6IujqEO/0VeVy6jq/7EQyR5VL0R+WATKrQwrpBfey1cEGB5JNLN5n1nC W9LB9xgrcz3b/pxEp1p3RVjvck3ghCK21gB2qeVDQGLYeSUYy+yKhQvU7Uv2VkoWw3PW T7CJo62WKHek5JI+CLAjGeuInuWMU1LP08t4s9Uqji9hN5JBHn9T7OEfMyocJNJhvFbr 6w8UG2oJ3jtJo0qAWNoHkl6wNhuDgjrvRxI7R+psGoLNJ9pjVZp7GPJ3Lo49lMxJa+G6 ZNAilTWNHk5fJFoYfTOfGycY401QsE9CCyxoiev08/pyh9LghVlA9JqWGs6E+7kddHFO rsxg== X-Forwarded-Encrypted: i=1; AJvYcCUrGPgLHIHGsEINh2ck3sXZUBvjV8zMUlG0Ky051wYJjI0nsMKzjxY4bIGQkcWpbkELfSunJB17erYkecxiQk3h0h0IGZ1QLbf1UzpTPq0M X-Gm-Message-State: AOJu0YzaqxiYjkApdmg6/uh9dtX5DEnyXyzJGwh8cEHEUs2goTl61XUC KMxq0ofmDlOd0msSV3uKgpBSKtyDxIfPW15BXj+vhI6e6FIUN4YowS+BPKszpJo= X-Google-Smtp-Source: AGHT+IEMupb/eaKm5/+VPG7gQL1BT1ElhtPyyJ9e/FdDd2FVBOCIDCIy/gU0Zn16LvTEMnl2YJ5n3Q== X-Received: by 2002:a5d:4148:0:b0:368:5ba0:622 with SMTP id ffacd0b85a97d-371946a4203mr4585432f8f.44.1723963002780; Sat, 17 Aug 2024 23:36:42 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37189896de9sm7251293f8f.75.2024.08.17.23.36.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:36:42 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 01/13] riscv: Move cpufeature.h macros into their own header Date: Sun, 18 Aug 2024 08:35:26 +0200 Message-Id: <20240818063538.6651-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_233644_747582_5B288A35 X-CRM114-Status: GOOD ( 15.10 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org asm/cmpxchg.h will soon need riscv_has_extension_unlikely() macros and then needs to include asm/cpufeature.h which introduces a lot of header circular dependencies. So move the riscv_has_extension_XXX() macros into their own header which prevents such circular dependencies by including a restricted number of headers. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cpufeature-macros.h | 66 ++++++++++++++++++++++ arch/riscv/include/asm/cpufeature.h | 61 ++------------------ 2 files changed, 70 insertions(+), 57 deletions(-) create mode 100644 arch/riscv/include/asm/cpufeature-macros.h diff --git a/arch/riscv/include/asm/cpufeature-macros.h b/arch/riscv/include/asm/cpufeature-macros.h new file mode 100644 index 000000000000..a8103edbf51f --- /dev/null +++ b/arch/riscv/include/asm/cpufeature-macros.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright 2022-2024 Rivos, Inc + */ + +#ifndef _ASM_CPUFEATURE_MACROS_H +#define _ASM_CPUFEATURE_MACROS_H + +#include +#include + +#define STANDARD_EXT 0 + +bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); +#define riscv_isa_extension_available(isa_bitmap, ext) \ + __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) + +static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_no); + + return true; +l_no: + return false; +} + +static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, + const unsigned long ext) +{ + asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) + : + : [vendor] "i" (vendor), [ext] "i" (ext) + : + : l_yes); + + return false; +l_yes: + return true; +} + +static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_unlikely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +static __always_inline bool riscv_has_extension_likely(const unsigned long ext) +{ + compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); + + if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) + return __riscv_has_extension_likely(STANDARD_EXT, ext); + + return __riscv_isa_extension_available(NULL, ext); +} + +#endif /* _ASM_CPUFEATURE_MACROS_H */ diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 45f9c1171a48..87ed88fc950d 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -8,9 +8,11 @@ #include #include +#include +#include +#include #include -#include -#include +#include /* * These are probed via a device_initcall(), via either the SBI or directly @@ -103,61 +105,6 @@ extern const size_t riscv_isa_ext_count; extern bool riscv_isa_fallback; unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap); - -#define STANDARD_EXT 0 - -bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit); -#define riscv_isa_extension_available(isa_bitmap, ext) \ - __riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext) - -static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_no); - - return true; -l_no: - return false; -} - -static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor, - const unsigned long ext) -{ - asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1) - : - : [vendor] "i" (vendor), [ext] "i" (ext) - : - : l_yes); - - return false; -l_yes: - return true; -} - -static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_unlikely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - -static __always_inline bool riscv_has_extension_likely(const unsigned long ext) -{ - compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); - - if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE)) - return __riscv_has_extension_likely(STANDARD_EXT, ext); - - return __riscv_isa_extension_available(NULL, ext); -} - static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const unsigned long ext) { compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX"); From patchwork Sun Aug 18 06:35:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72C09C52D7C for ; Sun, 18 Aug 2024 06:37:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Qst7upxLzxoMskb8f4znUVZbJIRscStcS/k7BGpTetI=; b=fZGWP08fMcTqTS cLC+uzAXWHUU7d7OHr6w3oJBJd+cSvgraCqpzr0CPV59iqN7i56xqUYrWVCsAADy8G4k911M6mtqZ WEKq2YOILtGTDRqN6Iiv8qtW60P6z5V1jNK7fabSdRIYQmleDaFLdmz7k74yGxTbb6n1TXHi3TDkz n/FQYe2vr1qFm4snHvRG5Fr+mhpJUUK1+ODLuXzwEfmYM+y+NjVh3viY2A8CjchUMs+SQtyr0nsVY OJrgvXcPIcK7vSKy/BVqTin8mhmkhuKgErbEu1lABTP985f2i+9N4M7NzN1XtnZ4O1STmaMNfj10Z JJXTJwRKYjg8p7OgvbfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZXx-0000000GDEh-2irG; Sun, 18 Aug 2024 06:37:49 +0000 Received: from mail-wm1-x331.google.com ([2a00:1450:4864:20::331]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZXu-0000000GDCa-0i5A for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:37:47 +0000 Received: by mail-wm1-x331.google.com with SMTP id 5b1f17b1804b1-428178fc07eso23515655e9.3 for ; Sat, 17 Aug 2024 23:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963064; x=1724567864; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=baf+ls/kqppHRVa4XBeUzS5UcHJPfW6ACwqFNpH2vH8=; b=xZLq+UdtsDEW2WDCXq13UMja7Vl3nFdPBn9JDuOhQL4j9EUPYdDQkKCsyDnwnsMZaE eZs64MlSzrvZbMpB05ssdmxcrAWsnVYKmBGEBIQZTT2dX/aXPT185ZyC52+xJjzbuULw euVUpICIn2Wd8FUJFENCf9/sLuuKseclLBk9VJLn2TKn5SIN+kFYRaxpYTUBnUyaxWt/ qxk0JZmwu2p7lBRyj//iOJ8ZwLzGIhxp00tVaQRl56hRb3CdhpP6o2CTun4288OfniMm CdVUGt+ogWnYQh5K5yKUsUZITnLkkzjjZR5sUfR1cbRPExfb4aYS0zP5rw7FgepNzyty 60Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963064; x=1724567864; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=baf+ls/kqppHRVa4XBeUzS5UcHJPfW6ACwqFNpH2vH8=; b=v1wIzcYlP4l9YSXYZgrtvDgmzv7jWKPMFI+ktCtFVxyaWRp1wymC7bRVwu8ehQUOLu 0NKYdUojFYMuuMptpyKc6ozW7rnjuRB5RwtVOqlq3pzY8ooqr0GGntm+aUkZgPvC41cm lscI2T2ilT0bIdZtPToVXE3Xclj7G54mN1vFcjQGP2yUF6W8ITDiFJZ1XQGX2eSf/X6S qLHUfA/2SSXok4zSp4mKDHtpYRca4tOq/Dq47JemAuiyTpDQPHnnVXIT3ABHpJk/9S6O my0PcuBd/8/3f4F6esuUcplfUyrrwlAoaqeCJPSw+A1g1rebhMCrlX73L3LqkO7XIfP3 AiFw== X-Forwarded-Encrypted: i=1; AJvYcCWCkqzqVZSkgof4t+BmDh23wZzoreG6iFzSl6XLoDUcWrJvam4nag+tkzmpop3fwISQoFiqQZI3uwz/kaQxqssRabqJ/7KcWCsWS/h4FZvT X-Gm-Message-State: AOJu0Yy0mnbERwi0ume+HWBH0ZS0S3mSTiX3zdk6Qz0QXkpVvYFFV8zQ M7qOTI3deAbjpYsnjZ6ljOg34t3waZEkKVnFTSdraFv8sEc9eqQUWZosErzUk7I= X-Google-Smtp-Source: AGHT+IGOB8iW+hQOXjQLAOh7AsPotaVJZqzzaXjL4GlISP37ETZjWU8j89260z8kTNAIQ1PuXHkP4A== X-Received: by 2002:a05:600c:3b1f:b0:426:629f:1556 with SMTP id 5b1f17b1804b1-429ed7f9224mr53506085e9.31.1723963064325; Sat, 17 Aug 2024 23:37:44 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ded7c93esm119862495e9.41.2024.08.17.23.37.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:37:43 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 02/13] riscv: Do not fail to build on byte/halfword operations with Zawrs Date: Sun, 18 Aug 2024 08:35:27 +0200 Message-Id: <20240818063538.6651-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_233746_229526_0A926B1C X-CRM114-Status: GOOD ( 10.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org riscv does not have lr instructions on byte and halfword but the qspinlock implementation actually uses such atomics provided by the Zabha extension, so those sizes are legitimate. Then instead of failing to build, just fallback to the !Zawrs path. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ebbce134917c..ac1d7df898ef 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -245,6 +245,11 @@ static __always_inline void __cmpwait(volatile void *ptr, : : : : no_zawrs); switch (size) { + case 1: + fallthrough; + case 2: + /* RISC-V doesn't have lr instructions on byte and half-word. */ + goto no_zawrs; case 4: asm volatile( " lr.w %0, %1\n" From patchwork Sun Aug 18 06:35:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CFEF0C5320E for ; Sun, 18 Aug 2024 06:38:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5c7IWEm0QIm5IA9EvaJR6+WKmCJSMsrONaGxbDVWUg4=; b=3zLzpOJGZKaxwL gaPN2V3Jyc508wucpfcGWSTu3YJ6PPLL0wZ67ecfJLx3X+Rho9aqM52Ih1W3xxVY/Hc7bfW96+/Fm l0reNJ/G6su+n7g+R+Wv+1m8DaVdg+fs2U7pAqVgMujkWeKMazawWu9GolslvskhEdnViENp9gMcH sEaQbmgXBntWBFpjwHJpIhEqa/PWWIy3Xem3HnFEuVJ3KZ/Y5r4iZJnne1JDEjlRBUSA15uY0KCuN SM94YFPgpXSvBC11Cq+rI3lL6z7GSUk/imx8FqgoofwscUIKDng9g0nETPkNmkIqIw+u6nsN2f+mx zda/vDs1CsyQ84Qrig4A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZYx-0000000GDQJ-0CXv; Sun, 18 Aug 2024 06:38:51 +0000 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZYt-0000000GDPT-48Z2 for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:38:49 +0000 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-37182eee02dso1612294f8f.1 for ; Sat, 17 Aug 2024 23:38:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963126; x=1724567926; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dgpIYh7KuQW+iTbAJx3HXjQmOubS3QQ17d+o/pur+bQ=; b=BJgBl0WXEv7aMFs88RLZtVlABjs8bH2E4kepFQHUpQXicQsrDacUNM1bRi2Rv6kvdT 4OIki928B1rYKQuu/4RXnOzFvX6EcmwlEv6ti2I6/h/Zab1VMCnjMEJtTyiwMYePE/hQ e2oKKCxppgoUCtZbVqcdkUl4X3e39RfL0ilTCXSvmN/tUYIAeufg1546PO4yA8yDDsvl RNX28H1taaxgxroW7B3KXWuPEY9Ho7shujXacXsM0TToVhpwfWokHaxvTjrUUjcNYKYy dtqHRza8k6Lk/URuNL2EcOIq/bfPKtYw8U3GryColYVtP50jfFPCcp2oH1GZqZ456zl3 eiFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963126; x=1724567926; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dgpIYh7KuQW+iTbAJx3HXjQmOubS3QQ17d+o/pur+bQ=; b=kuQWSWLvVWxGSVMQk9dEOihCEGkf7iKyKwDnGC5O2k33/UXxhqbZmthDMzKar7rpvn HeNlCipr+FObBXIjjW2iZgOGB0nB+qJOcbS6i/tzA6d9U07yaG79JVbQqRG5up4/57Vv UPpef5oCfcV/0GL+zAJsdP/unybJGC+ZsVSkDbnOdRWEpJTFAURmB4e+dc0JvWxHThX0 Rh8j76Ij2kdJ7zu00edBjuwxxyV6b7CU9UkU2tP9KYHOo2DO7940tpvwp6aN+m0JX0Mj THqGo1G1VluaEf/IUsmV/7BknJdIEwBCZwf9s2BIbDnURkA8DqWtjh1xuPqhGaCYvbtE c8bg== X-Forwarded-Encrypted: i=1; AJvYcCXpTQYEn7hC9HzmCvWpMMNe672zdrs++vkQ4Jth3PdmLO8xZ5OZVupMOtLfULxhPfe1DtcPBpQukHiHfSlZ0SV17Co8cxvhm5Q6y3Z7igGp X-Gm-Message-State: AOJu0Yxj/MhzPA5eJCA1lYtyzS3uu62tK6Peu4Ai42bKw3GVn0lDs+Bq aYOJqa8zfR5gmLZXuIDNpoNf0rI1GYz2oc5nciObHqbF93nCtLUOhblt1pE6DMs= X-Google-Smtp-Source: AGHT+IH8V6P8ozp0CwQCJM5ywpee/RSEJ1cgWdpG4qWioSCMxPEkZ3gtZHRrHCz0XJWkyE2olycTYw== X-Received: by 2002:adf:f648:0:b0:366:ee84:6a77 with SMTP id ffacd0b85a97d-37186bbd203mr7255006f8f.3.1723963125945; Sat, 17 Aug 2024 23:38:45 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429eeadfafbsm62053035e9.47.2024.08.17.23.38.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:38:45 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 03/13] riscv: Implement cmpxchg32/64() using Zacas Date: Sun, 18 Aug 2024 08:35:28 +0200 Message-Id: <20240818063538.6651-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_233848_055164_8542EC72 X-CRM114-Status: GOOD ( 14.04 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zacas in cmpxchg operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 16 +++++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 48 +++++++++++++++++++++----------- 3 files changed, 50 insertions(+), 17 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 0f3cd7c3a436..d955c64d50c2 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -613,6 +613,22 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. +config TOOLCHAIN_HAS_ZACAS + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZACAS + bool "Zacas extension support for atomic CAS" + depends on TOOLCHAIN_HAS_ZACAS + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zacas ISA-extension to implement kernel atomic + cmpxchg operations when it is detected at boot. + If you don't know what to do here, say Y. config TOOLCHAIN_HAS_ZBB diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index 6fe682139d2e..f1788131d5fe 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -82,6 +82,9 @@ else riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei endif +# Check if the toolchain supports Zacas +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index ac1d7df898ef..39c1daf39f6a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -12,6 +12,7 @@ #include #include #include +#include #define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ ({ \ @@ -137,24 +138,37 @@ r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ }) -#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ ({ \ - register unsigned int __rc; \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r = o; \ \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr" lr_sfx " %0, %2\n" \ - " bne %0, %z3, 1f\n" \ - " sc" sc_sfx " %1, %z4, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ - : "rJ" (co o), "rJ" (n) \ - : "memory"); \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" sc_cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + register unsigned int __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr" lr_sfx " %0, %2\n" \ + " bne %0, %z3, 1f\n" \ + " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ + : "rJ" (co o), "rJ" (n) \ + : "memory"); \ + } \ }) -#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -164,15 +178,15 @@ switch (sizeof(*__ptr)) { \ case 1: \ case 2: \ - __arch_cmpxchg_masked(sc_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \ + __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \ + __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ __ret, __ptr, /**/, __old, __new); \ break; \ default: \ From patchwork Sun Aug 18 06:35:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 85D87C5320E for ; Sun, 18 Aug 2024 06:39:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=HuOOrd4YhOM9efxmvvIf+C76Fz+R4RnIkKWVsFQrqBY=; b=sUs1Wns4E1htvM CghzTpf0iDH/B1IyexKi/CusErd0yKmBbRGGLXCvL4G1x4EqwsZWHjKqFS4Ls3Cd1wUOOzdGUzG6A UAxZBtcwX+ZTIEfiy8RdKK+80V8bpdFjhS8rhAvM0drm5oSrwfd4m7Yeq8Jjet6IvKmRPfy2J5jw/ yb3C+vjXxklKHu/11fwi7xtVkTX3unFOceONyUrkJFOvdo6j2cgeSHxdbbrKcCtDHaVLeUrj6k2+q YIPwM4JTbGKSgVz7dkrYgl5zSSX8HWDsDb21x8lBbLVHRVJ7zH9fW90rcv08TadGGEKkwJ/1Voa9Y UJqv7SJDRc0BlUbSDETQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZZw-0000000GDck-1SCM; Sun, 18 Aug 2024 06:39:52 +0000 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZZt-0000000GDbs-14Gp for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:39:50 +0000 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-428e3129851so25485675e9.3 for ; Sat, 17 Aug 2024 23:39:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963188; x=1724567988; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5sYtVuKxVaNEjuRyviwIAJgcsawGj7aKXy8R1Bii4eU=; b=sUtTAlTjppcjIV8udbjQF/yYXHCnRKnI86d26RkrMX3cJjycVJxeXrV2xfqaP0alGQ 9daJA722BfJeOnwUFcqqxqnFBCsCINOgHGkn0H0h68wCfEB/m/PW7GAjlXuvfLicQoC9 r2uu6vPM8Z52YETq+2N0k7XJZMxqiJsiW4CNtZcx4T/szA0eNf1T2mYBw/ehDZVWqpjK V7IP68QxY//hUp/d8yHP4rVhM4HxNiWZPCJidq+584fXPguhG9U68p6SH2rcULf03N6K VrY0n3SBJd9RLrmcnEWgOScnRXJkH9EHQBz6P43G28KIUnm/27CDGb5UDM4gnT54J41a DKLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963188; x=1724567988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5sYtVuKxVaNEjuRyviwIAJgcsawGj7aKXy8R1Bii4eU=; b=QvA0WBq24KQZrBfHcJhwsKtss4hrlawS/+tyzHgixkN+0jVyRtKCTmqM5mo9Nw8doa Ujq+ikjW5KCH6Je5W80x+rt7UbCbt1R9fCIinfq3gerREwyA4ti6+Y6Vrapd8ckcDXPA Ctx4VLaL7f5swvWsk+HPOq8J+Y8/g0H0vQJPbL5DXe+jZP3MRSYeP03nLf1QxWEV1KD4 ztUKTCcNQwiXZQglGD8L4L8tkLKxjGNN54vk+Va450DMPOU7124vwft1m4qeGuRSUR79 ipTrp2WLFYwP0OadUs+1yVixVujOY8Z13FkjGTeOWU2DtOV2TsuapSMcsxbt8VJMkQTK K3BQ== X-Forwarded-Encrypted: i=1; AJvYcCWxQEKvq+LJYG4qj3olEsWDusf8MvIdlVQED51wYsLNd0gFygwVbVa2MLFFG4JJfPxjbakO8iAG8WwWoiycDtQNx14V0DFjuxVmwH2GUyXr X-Gm-Message-State: AOJu0Yy1g2S1XumddgWRddaHwEhJLpLsNN4Lv/gKid3rUBi0Kky32gZx ZDWa+jiEJSsuHYYgIycKU+H/2CTvLb7Eu4Rh6YudEnxzVqC5uxuISj1PRR0/DlQ= X-Google-Smtp-Source: AGHT+IGGAMdr2P1EdP5GWl4akDsUdOuePbLIYOG470k8Ym7dIUmRU5AiPcQ2o3G7Nw7Sy6NLXGgTQw== X-Received: by 2002:a05:600c:3c9e:b0:427:98b4:624b with SMTP id 5b1f17b1804b1-429ed7e4314mr53197535e9.24.1723963187547; Sat, 17 Aug 2024 23:39:47 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed658e9esm66744295e9.27.2024.08.17.23.39.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:39:47 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Conor Dooley Subject: [PATCH v5 04/13] dt-bindings: riscv: Add Zabha ISA extension description Date: Sun, 18 Aug 2024 08:35:29 +0200 Message-Id: <20240818063538.6651-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_233949_321519_1A065908 X-CRM114-Status: UNSURE ( 8.85 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Zabha ISA extension which was ratified in April 2024. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index a06dbc6b4928..a63578b95c4a 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -171,6 +171,12 @@ properties: memory types as ratified in the 20191213 version of the privileged ISA specification. + - const: zabha + description: | + The Zabha extension for Byte and Halfword Atomic Memory Operations + as ratified at commit 49f49c842ff9 ("Update to Rafified state") of + riscv-zabha. + - const: zacas description: | The Zacas extension for Atomic Compare-and-Swap (CAS) instructions From patchwork Sun Aug 18 06:35:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CAEE5C52D7C for ; Sun, 18 Aug 2024 06:40:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vOW2jVR3KadH+Hgwu1Oj7PPCp1Fdh3HluazL/4xhEWY=; b=zCEd17gBnYhk7N HDxMoL8PrNT9Cb/2MeBhON4t6FvYcJ6vzrVcCPRCTz3xsrdCoAu9OCjmZSVe597AAKbu1GKvIktxf ItzSIYVBsKBnh8jVtjrZ/cClMXgj+dsJV2Byd2zGQ4jwc8rtgkZdur572p9dtvp0c95TRbGtUXiBf GwubekLjcMCVOA74MseSJZjl7drhQu0FBBrrlfJCrIYcH2hdv0ZJJuNj9rR6YZCIBFbuf6cviMw3a J3YpJxc0oizMownB4YB1uPnI6R+eYHOPmkh79n1EPFBjaOYIQ+a33hDR43dEMHngqV7rBnJ9T/wU7 dedKL4hx1EwDKtJVcmeQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZaw-0000000GDmu-2yqd; Sun, 18 Aug 2024 06:40:54 +0000 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZas-0000000GDlz-3bDJ for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:40:52 +0000 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-37189d8e637so1903105f8f.3 for ; Sat, 17 Aug 2024 23:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963249; x=1724568049; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/ZGnwjJ9zA50deMtasLoWGJlTqCTlw/AR/rhuAE8IZk=; b=ba0ThMJvjf53RAmDemLLnCgVv8wk39txduoltw26DqN7f7fSGFtiRHmFyQZQ6XcmnN Q5voTu17fXKf9Z95MvLKQAiBTbHRpZP9Y4vodEmSRO4j9SXlqEpMhf51NrPZGH353x9A QPes//GtHSXX5rVao1iZnUs1RBchOx4wEE0tSJPDbw6dhdCe2dwyjJdsUFZnVSkqc/kn dJ7L/jqjCEcnLvWIBvC1OB8tgv70AQOR1Nn48xnenbuXxUdLEmwcaayIjRgGOstTFRTX umBJmV8/nQRlecy5rlfUjsQylQj8QalDu/w409L3duzwRH3j0EUVAYHHWG+xnnX74WB5 OSfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963249; x=1724568049; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/ZGnwjJ9zA50deMtasLoWGJlTqCTlw/AR/rhuAE8IZk=; b=dynoQ37e2qDiK00te6g3uK2SC8MYb5wf1G0F+QmqhP6Ouzt5R6P4CW+tgnBePyTqI6 lDYNuakETXEdjsKbWSGk3jIBcaSomiABJ4vObAaAorg9an9obUb3sHp6HFntSjyFN9gj KhhYqmaBaQG1KM7HfRKQZAdR8o/cKoRdmeReHDCVt2yclzXRB/0oE9N3teOj3C9spvsS 4jWJYuAmhSWXserXt3Vtozw/9ATQYxOmNiEWe9WpmidzfJR3KPnkCJw0JZLoDworvZC6 0q3qGKD0mP/+AhJTJgO208qyQE5BBiafDXFR8ASS1675Y/ShuOr9HHi2UkR6tbKIoiCL zeEA== X-Forwarded-Encrypted: i=1; AJvYcCXytW5TBaBqh5604Gk2+kgAD0ucXk8d94IzOTbVxCCfso8XNcpna16YrmUeV6TnYxS8kGMziTkqnYoQ6V/TWLnN1KRCUzFkjMr56Ktkp5mI X-Gm-Message-State: AOJu0YxQ5ook3vBoO2cESL7mdIOUjJbBKKW5sxnO4S0JwqIfC9MZIeHY NgbZfTqI7jQfj5I1MarRQ3zBg8/Qzci4BFMOB5c4H+3Rhc/orqQcHoyVrJk/PUs= X-Google-Smtp-Source: AGHT+IH5q0bFu/E9vYl6O9ntKbWhfQ/kqCOKwUu/eU0CrpOWwaixsLysU9gopA/iIxfwBIo3S9Ykcw== X-Received: by 2002:a5d:53c5:0:b0:371:7d84:9bef with SMTP id ffacd0b85a97d-37194651551mr4876064f8f.28.1723963249067; Sat, 17 Aug 2024 23:40:49 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-371898aabdesm7286232f8f.99.2024.08.17.23.40.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:40:48 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 05/13] riscv: Implement cmpxchg8/16() using Zabha Date: Sun, 18 Aug 2024 08:35:30 +0200 Message-Id: <20240818063538.6651-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234050_928082_85B095F3 X-CRM114-Status: GOOD ( 16.25 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in cmpxchg8/16() operations. Note that in the absence of Zacas support in the toolchain, CAS instructions from Zabha won't be used. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 18 ++++++++ arch/riscv/Makefile | 3 ++ arch/riscv/include/asm/cmpxchg.h | 78 ++++++++++++++++++++------------ arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 5 files changed, 72 insertions(+), 29 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d955c64d50c2..212ec2aab389 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -613,6 +613,24 @@ config RISCV_ISA_ZAWRS use of these instructions in the kernel when the Zawrs extension is detected at boot. +config TOOLCHAIN_HAS_ZABHA + bool + default y + depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha) + depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha) + depends on AS_HAS_OPTION_ARCH + +config RISCV_ISA_ZABHA + bool "Zabha extension support for atomic byte/halfword operations" + depends on TOOLCHAIN_HAS_ZABHA + depends on RISCV_ALTERNATIVE + default y + help + Enable the use of the Zabha ISA-extension to implement kernel + byte/halfword atomic memory operations when it is detected at boot. + + If you don't know what to do here, say Y. + config TOOLCHAIN_HAS_ZACAS bool default y diff --git a/arch/riscv/Makefile b/arch/riscv/Makefile index f1788131d5fe..f6dc5ba7c526 100644 --- a/arch/riscv/Makefile +++ b/arch/riscv/Makefile @@ -85,6 +85,9 @@ endif # Check if the toolchain supports Zacas riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas +# Check if the toolchain supports Zabha +riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha + # Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by # matching non-v and non-multi-letter extensions out with the filter ([^v_]*) KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/') diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 39c1daf39f6a..1f4cd12e4664 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -108,34 +108,49 @@ * indicated by comparing RETURN with OLD. */ -#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \ -({ \ - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx = (ulong)(n) << __s; \ - ulong __oldx = (ulong)(o) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z5\n" \ - " bne %1, %z3, 1f\n" \ - " and %1, %0, %z6\n" \ - " or %1, %1, %z4\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - "1:\n" \ - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" ((long)__oldx), "rJ" (__newx), \ - "rJ" (__mask), "rJ" (~__mask) \ - : "memory"); \ - \ - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ + r = o; \ + \ + __asm__ __volatile__ ( \ + prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + append \ + : "+&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __oldx = (ulong)(o) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z5\n" \ + " bne %1, %z3, 1f\n" \ + " and %1, %0, %z6\n" \ + " or %1, %1, %z4\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + append \ + "1:\n" \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" ((long)__oldx), "rJ" (__newx), \ + "rJ" (__mask), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) #define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ @@ -177,8 +192,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ + prepend, append, \ + __ret, __ptr, __old, __new); \ + break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, prepend, append, \ + __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ + prepend, append, \ __ret, __ptr, __old, __new); \ break; \ case 4: \ diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index 5a0bd27fd11a..f5d53251c947 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -92,6 +92,7 @@ #define RISCV_ISA_EXT_ZCF 83 #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 +#define RISCV_ISA_EXT_ZABHA 86 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index b427188b28fc..67ebcc4c9424 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -322,6 +322,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { __RISCV_ISA_EXT_DATA(zihintpause, RISCV_ISA_EXT_ZIHINTPAUSE), __RISCV_ISA_EXT_DATA(zihpm, RISCV_ISA_EXT_ZIHPM), __RISCV_ISA_EXT_DATA(zimop, RISCV_ISA_EXT_ZIMOP), + __RISCV_ISA_EXT_DATA(zabha, RISCV_ISA_EXT_ZABHA), __RISCV_ISA_EXT_DATA(zacas, RISCV_ISA_EXT_ZACAS), __RISCV_ISA_EXT_DATA(zawrs, RISCV_ISA_EXT_ZAWRS), __RISCV_ISA_EXT_DATA(zfa, RISCV_ISA_EXT_ZFA), From patchwork Sun Aug 18 06:35:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 65863C52D7C for ; Sun, 18 Aug 2024 06:41:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TZTezrnkMsECmb+yT2Pf1vbkUfVF2i6zCFMIdNBnt5U=; b=2VMPnv6N0AKduX vndhzKLAKPBhFwbhE2o15JXODMFKFcwNa/t1cXJVK3mzap6SeSvJ7RXS3u0HE0mDs7fMCEM0wUe0n AQXAQEFAZdG8iA/lL5pZ5fpwk2G1o7Q4RJROcm3pG4nGrQrMl3gAdZhs6hkktF7B/QAPNLmFRW7nI orHf/aQ7IRtDltIr2VUSlxS3NxNZioGKnfefvr2Zdg0FeIpTrhAAHQFWE5AeT/CAys100pe6NTdGc 6lUDJ60nT3ulK3LgMnqNzRkHSPAtUnLLG7nJSuSzNC9Xfk75h4xLuyKOeWhIjzKc0iYSlR64FfAez ark+BnxZLetfs/svRHwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZbv-0000000GE0G-40qT; Sun, 18 Aug 2024 06:41:55 +0000 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZbt-0000000GDzS-0lZd for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:41:54 +0000 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-3719398eafcso1419737f8f.1 for ; Sat, 17 Aug 2024 23:41:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963311; x=1724568111; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EzrQ84GOc79Cvd4pehyYMEiToRaVIXWpNcTPRgBQZOo=; b=lsH3BED8aSnHex2WZ9/6bLiU9f85uZGEspZH2fcHJA3JkGD3lsr3Ml+quKarkH+C+j eyd1u9hbxCByH4r2abh13B40a9ojHHnZSIjnEXG13cltFTSv/OaS3AvOpoGI7OPoHNOp y1Az0dKHLCFpA9FLa+7+OM6a0s6r6q0F1VNa+9SXY/MSOfuSEk6ac/zL2T4D2YouZU5C yw+lxIuQDE6uQYlScKMFMkTVfA9p8uW0NIOjkAb8Eyr11qBXPrIz4KBLU3Oaqvvjqg1j 23FbKvz/oHzkmcAMnxGrWU6l8TaCxlFMYLLtKfaEhMtF2pZJx/4xD+4at5oMksHo/tvV +Zdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963311; x=1724568111; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EzrQ84GOc79Cvd4pehyYMEiToRaVIXWpNcTPRgBQZOo=; b=PW9pFj4RGMRhPz5Y0L0/WgL7RLkW2wvQdAWtWMjxBKeIPg5z9U3/NZpVSH9FGvxRkZ ulc6vCD/7Nzi5rQK3vhS0YhUQt64EBVdcLHal4BFUEcpvE2mZj+4oImjjTZPOlnEaM+M fkt4Xb1ukEK0gw3Z3q2HxHXBatKSjz5aU1yjkPoBr3OQLpfOGtxqvueFu4RQwFNxVffX vJOzht/LIHSMuDJT2JnFv4T0D2fu4/iqv9AjIWAah7jCmpe+a7X5P1Udd2IHtTiUPEeq Swp4sO1K+7CspJ3JfGgqn4Hf4ERmJWTs46hYRH5OrXjZELAgVih7Nten2k6dTh7EXf34 HuGg== X-Forwarded-Encrypted: i=1; AJvYcCUSJNGzZry1XbZBLa6w2qJtSiUNKsugaUjhax6aMiroHSpCznKD89SveG+5ZEtivIdK9QGg0fKemCT+OLcSLTrc8i51fTyoJRMa5SqtwwbF X-Gm-Message-State: AOJu0YxLx5bhzeqMKwzAPFaN/AKCJkHfz3ZuoyOO8Rojoj8D9Ftr7Fra FCDRjdYaUhTc6B5P6q0oGVl58fYCSbXcCFdBcOETCKbXfMb4tjHgcnN5l90wRf4= X-Google-Smtp-Source: AGHT+IH7jXMI8vQlmJu9PzoH6e8IpPUsf9yyX42mqd9sjDsguutH4AFv/sh4mfXo/xTrWqv9pY4RtQ== X-Received: by 2002:adf:a1ce:0:b0:368:7fbc:4062 with SMTP id ffacd0b85a97d-371946531e9mr4631837f8f.33.1723963310553; Sat, 17 Aug 2024 23:41:50 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37189849606sm7362530f8f.32.2024.08.17.23.41.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:41:50 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrea Parri Subject: [PATCH v5 06/13] riscv: Improve zacas fully-ordered cmpxchg() Date: Sun, 18 Aug 2024 08:35:31 +0200 Message-Id: <20240818063538.6651-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234153_231468_9E49B581 X-CRM114-Status: GOOD ( 11.57 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The current fully-ordered cmpxchgXX() implementation results in: amocas.X.rl a5,a4,(s1) fence rw,rw This provides enough sync but we can actually use the following better mapping instead: amocas.X.aqrl a5,a4,(s1) Suggested-by: Andrea Parri Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 92 ++++++++++++++++++++++---------- 1 file changed, 64 insertions(+), 28 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 1f4cd12e4664..5b2f95f7f310 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -107,8 +107,10 @@ * store NEW in MEM. Return the initial value in MEM. Success is * indicated by comparing RETURN with OLD. */ - -#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, prepend, append, r, p, o, n) \ +#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ @@ -117,9 +119,9 @@ r = o; \ \ __asm__ __volatile__ ( \ - prepend \ + cas_prepend \ " amocas" cas_sfx " %0, %z2, %1\n" \ - append \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -134,7 +136,7 @@ ulong __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr.w %0, %2\n" \ " and %1, %0, %z5\n" \ " bne %1, %z3, 1f\n" \ @@ -142,7 +144,7 @@ " or %1, %1, %z4\n" \ " sc.w" sc_sfx " %1, %1, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ : "rJ" ((long)__oldx), "rJ" (__newx), \ @@ -153,16 +155,19 @@ } \ }) -#define __arch_cmpxchg(lr_sfx, sc_cas_sfx, prepend, append, r, p, co, o, n) \ +#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + r, p, co, o, n) \ ({ \ if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \ riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \ r = o; \ \ __asm__ __volatile__ ( \ - prepend \ - " amocas" sc_cas_sfx " %0, %z2, %1\n" \ - append \ + cas_prepend \ + " amocas" cas_sfx " %0, %z2, %1\n" \ + cas_append \ : "+&r" (r), "+A" (*(p)) \ : "rJ" (n) \ : "memory"); \ @@ -170,12 +175,12 @@ register unsigned int __rc; \ \ __asm__ __volatile__ ( \ - prepend \ + sc_prepend \ "0: lr" lr_sfx " %0, %2\n" \ " bne %0, %z3, 1f\n" \ - " sc" sc_cas_sfx " %1, %z4, %2\n" \ + " sc" sc_sfx " %1, %z4, %2\n" \ " bnez %1, 0b\n" \ - append \ + sc_append \ "1:\n" \ : "=&r" (r), "=&r" (__rc), "+A" (*(p)) \ : "rJ" (co o), "rJ" (n) \ @@ -183,7 +188,9 @@ } \ }) -#define _arch_cmpxchg(ptr, old, new, sc_cas_sfx, prepend, append) \ +#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append) \ ({ \ __typeof__(ptr) __ptr = (ptr); \ __typeof__(*(__ptr)) __old = (old); \ @@ -192,22 +199,28 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".b" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 2: \ - __arch_cmpxchg_masked(sc_cas_sfx, ".h" sc_cas_sfx, \ - prepend, append, \ - __ret, __ptr, __old, __new); \ + __arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, __old, __new); \ break; \ case 4: \ - __arch_cmpxchg(".w", ".w" sc_cas_sfx, prepend, append, \ - __ret, __ptr, (long), __old, __new); \ + __arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, (long), __old, __new); \ break; \ case 8: \ - __arch_cmpxchg(".d", ".d" sc_cas_sfx, prepend, append, \ - __ret, __ptr, /**/, __old, __new); \ + __arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \ + sc_prepend, sc_append, \ + cas_prepend, cas_append, \ + __ret, __ptr, /**/, __old, __new); \ break; \ default: \ BUILD_BUG(); \ @@ -215,17 +228,40 @@ (__typeof__(*(__ptr)))__ret; \ }) +/* + * Those macros are there only to make the arch_cmpxchg_XXX() macros + * more readable. + */ +#define SC_SFX(x) x +#define CAS_SFX(x) x +#define SC_PREPEND(x) x +#define SC_APPEND(x) x +#define CAS_PREPEND(x) x +#define CAS_APPEND(x) x + #define arch_cmpxchg_relaxed(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(""), \ + CAS_PREPEND(""), CAS_APPEND("")) #define arch_cmpxchg_acquire(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER) + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(""), SC_APPEND(RISCV_ACQUIRE_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND(RISCV_ACQUIRE_BARRIER)) #define arch_cmpxchg_release(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(""), CAS_SFX(""), \ + SC_PREPEND(RISCV_RELEASE_BARRIER), SC_APPEND(""), \ + CAS_PREPEND(RISCV_RELEASE_BARRIER), CAS_APPEND("")) #define arch_cmpxchg(ptr, o, n) \ - _arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n") + _arch_cmpxchg((ptr), (o), (n), \ + SC_SFX(".rl"), CAS_SFX(".aqrl"), \ + SC_PREPEND(""), SC_APPEND(RISCV_FULL_BARRIER), \ + CAS_PREPEND(""), CAS_APPEND("")) #define arch_cmpxchg_local(ptr, o, n) \ arch_cmpxchg_relaxed((ptr), (o), (n)) From patchwork Sun Aug 18 06:35:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D6A0FC52D7C for ; Sun, 18 Aug 2024 06:42:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CWx+RHf/asleKfO7mdq7EfxxO/+gsxlvgGPRa50Ak0c=; b=ZV1n/0JdOjDGgS sw+RQ1Q2jgHwJWROvKrTV3XBHrKRRrl7E1diep/i34XGSNHiWEpwYodScJ5f1Q10nJy1/dwzDWst5 RpjOMG8Xyi6oj9rmtUFIOZa/dfNhzZIPWK0F9B+8uHGZqXNQ7iFLDLTMrgBhpHH3ik2rQo8spyjZb xzaFEibwfN9ZvTAw8ywo1sTTL7mb5UF1ZqbHM/vwL9iZTSJl5W1YQuov1s8guuOiZq98Rstotc6QL +TMe6i1HtAVF+MmzIcSoY1SaYgf/+50rVudxqBNti/nHg7rq3+3mTsUwUrYIWJjXC4I1cz5pYxIIw ISrtOgOD3vy07N5lcDfA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZcv-0000000GECR-0emb; Sun, 18 Aug 2024 06:42:57 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZcs-0000000GEBA-2Yo2 for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:42:56 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-4280bca3960so27868325e9.3 for ; Sat, 17 Aug 2024 23:42:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963372; x=1724568172; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=quUzKJl3l9W/iH7huB/Ox/rGh7MBHVzUurNoeTrSXJY=; b=HZWCtucWfP8IWa2UmUklPzmGf2dJdecT133+rLqimY3oybrnwHNUXSwvaL5qUoiUHO dMsx1fl0BJnvhrHdyH/+K8PtBePKW5In5nWLq66/LoLgJegqBHqR4U0zxgza8ZArRtwn ddi1tWe+Z9i3Cl/xpi+Msa+7FcJzZ32qEYhEQLL0/VSQ0WFtmCjlwfZXzi4dgMwoveOk YPB2/NAD+gWMxT0DBt35nFD69Xwe0QZ/d/BRS87opGOhe31ymbdiTKQnDcjcCVFZZQD1 IAv7nYtbwIvk1XTusJEhT1G+yEKQ39+oft9l77rv4+X/s76nJHNFMLG63Hh7gNcH3hks 62Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963372; x=1724568172; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=quUzKJl3l9W/iH7huB/Ox/rGh7MBHVzUurNoeTrSXJY=; b=UgM4MivHEbB+9MAmfufC+8wPAq+EGWZg6HU6OkaQlsJYxThRZtqlqd+oNkMfGaHCEH dimWZVMbmio3YEQoPwEuytBTxZOxGczYO79xZrWX8xLGlq3AQd6iGRCMH93Wh0fBZVUS KxtQ5RRPZmVXVZH0O/zlPUpM0GeRbF43SGBzBcxFHiC+llMmTN0x5RfoHllPFsiGMew7 DPnZbIkM2U8xXRD2PEthUtcnvWa/HmnVDOCMA4JAIW/l+Kn2ZKx/YeWhxACbXSYzjPPC /VPZmuG/WAvntD2T1th8SblgMmCwagXv0Uz3JLc/bgv6jybd5cCapZ2GWHd23FvqAeHG xqhQ== X-Forwarded-Encrypted: i=1; AJvYcCVD+ZNkRCcQfYYYSCpA1j8QEmheeFBDnSSI5uxgMfB/Di245uXHX75UlGR15DWr4+gpRwsyO7PzbG8Pcw6iR7XRPHz9nXCyhlNU/grUgyk9 X-Gm-Message-State: AOJu0YxCR56LSxLalOqSxMaYUaOw1EjHBikpY0fbCB+VigKxayBbeaqc 2mcDdT2v9GOb6Ur9s7JRXoT+xMS4gC2mXi0sloYtiqW0/iOGjwS1pGecjAaai3Y= X-Google-Smtp-Source: AGHT+IEe0ieY97eTK1clBc3HGi9QZxo6cf+OA5t5L0JQLmZOTEmF4/dAe6bc3mWD2TnCj1Dfw0jsiA== X-Received: by 2002:a05:600c:3509:b0:426:6b92:387d with SMTP id 5b1f17b1804b1-429ed7b8860mr53677765e9.21.1723963372179; Sat, 17 Aug 2024 23:42:52 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed7945b8sm66605265e9.44.2024.08.17.23.42.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:42:51 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 07/13] riscv: Implement arch_cmpxchg128() using Zacas Date: Sun, 18 Aug 2024 08:35:32 +0200 Message-Id: <20240818063538.6651-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234254_680682_206ED3A1 X-CRM114-Status: GOOD ( 12.09 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Now that Zacas is supported in the kernel, let's use the double word atomic version of amocas to improve the SLUB allocator. Note that we have to select fixed registers, otherwise gcc fails to pick even registers and then produces a reserved encoding which fails to assemble. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cmpxchg.h | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 212ec2aab389..ef55ab94027e 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -111,6 +111,7 @@ config RISCV select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO select HARDIRQS_SW_RESEND select HAS_IOPORT if MMU + select HAVE_ALIGNED_STRUCT_PAGE select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 5b2f95f7f310..05ba8a8e2ef5 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -296,6 +296,44 @@ arch_cmpxchg_release((ptr), (o), (n)); \ }) +#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS) + +#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS) + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(p, o, n, cas_sfx) \ +({ \ + __typeof__(*(p)) __o = (o); \ + union __u128_halves __hn = { .full = (n) }; \ + union __u128_halves __ho = { .full = (__o) }; \ + register unsigned long t1 asm ("t1") = __hn.low; \ + register unsigned long t2 asm ("t2") = __hn.high; \ + register unsigned long t3 asm ("t3") = __ho.low; \ + register unsigned long t4 asm ("t4") = __ho.high; \ + \ + __asm__ __volatile__ ( \ + " amocas.q" cas_sfx " %0, %z3, %2" \ + : "+&r" (t3), "+&r" (t4), "+A" (*(p)) \ + : "rJ" (t1), "rJ" (t2) \ + : "memory"); \ + \ + ((u128)t4 << 64) | t3; \ +}) + +#define arch_cmpxchg128(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), ".aqrl") + +#define arch_cmpxchg128_local(ptr, o, n) \ + __arch_cmpxchg128((ptr), (o), (n), "") + +#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */ + #ifdef CONFIG_RISCV_ISA_ZAWRS /* * Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to From patchwork Sun Aug 18 06:35:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 721B7C5320E for ; Sun, 18 Aug 2024 06:44:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kIYKzYAy9htyicTO37z/htPyr9TDIPQNPJqxANxy+yY=; b=ejVqSY5PmCzagC XugV8QpL6/59r/ySKwRaTOzE8EEH4wDJd8Z9J2BIBD3SCe5BlhDyOW/uI7H4rQo4zsiLCOik4IYxt asiN1PI9uRed0ckUoLoA9gT3I1hfB7Q9M9nzTZCQYNqMZvpCKc7Id7Efigj3rzdunKAr5XC5NxVvf LrSAz5uwQJDW0LdVdclBEZH1oSlDPtSPF+vvzebgJqhNSBQpohX43TYxB0rO0h95s1a7kxL0mXgNz JM/OydEHuW7/T986LdAKtX+pUJx5f4xFiDYU9Ro7/MOY5fjKzfphokR28yv1wmT9Pjg1n9j5FyjxI GDHxZ4Wpbu4Tvo5wQg5A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZdu-0000000GELi-19pR; Sun, 18 Aug 2024 06:43:58 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZdr-0000000GEKw-1pCh for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:43:56 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-428178fc07eso23532085e9.3 for ; Sat, 17 Aug 2024 23:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963434; x=1724568234; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H3v3ACeRnW6fQEuje9seVAm+KH7JNORlBJR9qvXfha0=; b=DEQu9nBSeZE29ZXsCGkC7i9maXOHlkVb4YmzV4BgNGHbl5AjcsZEHD/PvQhxi2RR90 oIGEH7o4YWd2WDRqNSvfsXhfOnFUa2T4qLfhZbV2/W09tNryakQ4ebDBAupwqpiw0WyN ZrIXyvmiIZH4gWFZ2JrVdepiPR2jqqtLdH8Otcy8QjCT7jlHo/3t1/NngloNsxWvOCuU qLOV5i3pfqP4IXckBnY9wLWCqLCQTXy/EAyYxXk1m5qEc4bWYX3l6Pt7hu1OFdpBXr9E qAS1thQiwGQz8rEzPr2mxHdd9gqVGyb5vkI4OBt09C7YvTuteLw7chmoLf8DEeuQ/PwJ 7Myg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963434; x=1724568234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H3v3ACeRnW6fQEuje9seVAm+KH7JNORlBJR9qvXfha0=; b=uUVrsOLgxn9TTvgvBpaDXua0AgcHLZeSfMvioE6D1CukmDZ2cNgrUbdvBdcI6WBCRV KvZv8KJ8BLGRdTMaR4DpLuKDbrz/0MtO1aN0WjvfgKdjhjVWMaL8Id0tQVEO1xVceIb7 dKkPl0EH+I8ZK14wm0T2mPpoVAHL0zLiLyUNA/VkF6UdoGcTBLtJM6j7IxgdkluGQBp4 yOle2Cg9PVS2WT+ejL/tvH2WIkWVsQeD8z7ASIrE6Vc+/l7+NzVRUB0lkKCX5cUzfSnW Ws+up9QSf6P3g92quNcW6d0VXW2weIR03JV4F0hZJNZ7xWitv1PvM9eTOymIij3eZlst Z2vQ== X-Forwarded-Encrypted: i=1; AJvYcCWfJeZFSmOsdKL1XeTInBPrMd4uQb+VJ6q6xTMdtDFQXqnpnR8De9jA1areq5upDULJCMfUcEWXRl4h6lnHzPV81C3p5zXQusDjxITD7nGk X-Gm-Message-State: AOJu0Ywu5HGRy2GVT1YStyk72g7DSOeLwrPZiQaSV4gLCtcn7lrqGQCT JC6JUmqN9i0vzxwCMQ1LDjV92i7o5wtdDdwhn7Ql1mIKvpq3/KPNMWL774+MVcA= X-Google-Smtp-Source: AGHT+IGdpiZerNesAs6xzBEap+g5NepzOjC8cdpysjNsTdJ9hLtJ74EBvU/xnjt3uajkBUnG6SvmcQ== X-Received: by 2002:a05:600c:3509:b0:426:64a2:5362 with SMTP id 5b1f17b1804b1-429ed7a6100mr51627275e9.8.1723963433892; Sat, 17 Aug 2024 23:43:53 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed4a857bsm66904395e9.0.2024.08.17.23.43.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:43:53 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti , Andrew Jones Subject: [PATCH v5 08/13] riscv: Implement xchg8/16() using Zabha Date: Sun, 18 Aug 2024 08:35:33 +0200 Message-Id: <20240818063538.6651-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234355_497576_97D42BFC X-CRM114-Status: GOOD ( 10.02 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This adds runtime support for Zabha in xchg8/16() operations. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/cmpxchg.h | 65 ++++++++++++++++++++------------ 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h index 05ba8a8e2ef5..19779d4d134a 100644 --- a/arch/riscv/include/asm/cmpxchg.h +++ b/arch/riscv/include/asm/cmpxchg.h @@ -14,29 +14,41 @@ #include #include -#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \ -({ \ - u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ - ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ - ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ - << __s; \ - ulong __newx = (ulong)(n) << __s; \ - ulong __retx; \ - ulong __rc; \ - \ - __asm__ __volatile__ ( \ - prepend \ - "0: lr.w %0, %2\n" \ - " and %1, %0, %z4\n" \ - " or %1, %1, %z3\n" \ - " sc.w" sc_sfx " %1, %1, %2\n" \ - " bnez %1, 0b\n" \ - append \ - : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ - : "rJ" (__newx), "rJ" (~__mask) \ - : "memory"); \ - \ - r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ +#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \ + swap_append, r, p, n) \ +({ \ + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \ + riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \ + __asm__ __volatile__ ( \ + prepend \ + " amoswap" swap_sfx " %0, %z2, %1\n" \ + swap_append \ + : "=&r" (r), "+A" (*(p)) \ + : "rJ" (n) \ + : "memory"); \ + } else { \ + u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \ + ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \ + ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \ + << __s; \ + ulong __newx = (ulong)(n) << __s; \ + ulong __retx; \ + ulong __rc; \ + \ + __asm__ __volatile__ ( \ + prepend \ + "0: lr.w %0, %2\n" \ + " and %1, %0, %z4\n" \ + " or %1, %1, %z3\n" \ + " sc.w" sc_sfx " %1, %1, %2\n" \ + " bnez %1, 0b\n" \ + sc_append \ + : "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \ + : "rJ" (__newx), "rJ" (~__mask) \ + : "memory"); \ + \ + r = (__typeof__(*(p)))((__retx & __mask) >> __s); \ + } \ }) #define __arch_xchg(sfx, prepend, append, r, p, n) \ @@ -59,8 +71,13 @@ \ switch (sizeof(*__ptr)) { \ case 1: \ + __arch_xchg_masked(sc_sfx, ".b" swap_sfx, \ + prepend, sc_append, swap_append, \ + __ret, __ptr, __new); \ + break; \ case 2: \ - __arch_xchg_masked(sc_sfx, prepend, sc_append, \ + __arch_xchg_masked(sc_sfx, ".h" swap_sfx, \ + prepend, sc_append, swap_append, \ __ret, __ptr, __new); \ break; \ case 4: \ From patchwork Sun Aug 18 06:35:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1956FC52D7C for ; Sun, 18 Aug 2024 06:45:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cbsxV22WuluSG2D5qmTTGZPXT7B7VeNOXpZs3rTejvk=; b=Wda1v3dyh+bArx enJrDmSRlYMYIY5g+Nxw9tpK5yc/VwU0O+kg6LH2fsRSyY3ZqzO8bf15Ib8H9+u7KqdrW2k1wZD3C envMAvv512M+iJAdVlC/CpRhUpRm70/06jH9innwchTkMhXE4if0UrI9FxE76VhKdjge5doWA0rGd zmbIhpQhRgpX7BnPukARoZ9jZWa+NbxjIaE6gYzXLDNS2zce0CuMPvaUDIQ/f1L8SmaN5nHy4Ho5R 0hOxabB+hrwoQ6vFSc1J6+moXIHEHfHlnGXjeqqbQ+mhCjBnKbI9YnZV0NFSDZiqZ/W5bzgKjoA9p xcf+lcMnaO7SbXF8NA6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZeu-0000000GEUy-2HXs; Sun, 18 Aug 2024 06:45:00 +0000 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZer-0000000GEU8-3aNB for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:44:59 +0000 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-4281d812d3eso34770085e9.3 for ; Sat, 17 Aug 2024 23:44:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963496; x=1724568296; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RFL/O3acOIAnvUQyUxNex19UAi4aS/YbFByq84YgWQ0=; b=GV2BIG/5ohnKWXUqXIo74lXoncs/1iK7UF9XU5n0IhmFpeZU05XBRqkXS8wuJ8bIRY 7+Ij+OUzMmatrvOvT4Xqp7oTYrZuAcOtLNgng92rG6SQYisPEkoWzBNOugdPOWoVQPB+ KgappgvCbJ+2nnT9pbtnkxJZ6XBLANcp9Y8PDpz/xzg+dn09Aqz2Z4wFZDMwTunmEM1c IKvBmZLUMq6rUyqtSSKMh7fcfUbHrvAy+4j9j3joXK1ECtrp4JMxaMBVXjUQqL+ReYVQ YstB5M8t676npCXUklUshQaMASo3RLGI2yZkz1wzNKx6kVq/8girALPMKDqYt1FhY7R/ YjDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963496; x=1724568296; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RFL/O3acOIAnvUQyUxNex19UAi4aS/YbFByq84YgWQ0=; b=OmqT2CESgwObLwdhz6nXcmz6O1pRqhN9ksLQ5mEKebP2T/KzCeh2S73FJ3KPvV2thb 0X6vlNb6pLtZuA7vDnlWO+wyWEVpMJeDpvNPEiUK0cznCUlCncIHXDwNBYREKYB3Fy0H lRTHRwgCNkLUQ3ale7OOyZzPdo7GNwK6sh/l+DKsvcKSH+YLsMjwwdkKIYPjiZOyyHvH u0ebV5bRWMdhyjxS74eA3HVO8dcx5zAhS2m5/hvNtmJoARhTt4uZHAoSZmDuUCKnijyB un3PtWyaVCV2eTwKaxYwqR//rsp4XhYmhnoWF8pjtlexxOx1aBac11aE6Sj6fr4RWmRJ 6/jg== X-Forwarded-Encrypted: i=1; AJvYcCU9UAWbuGQaN7aS+VH/E4qjz21PlStWHgHA3i+A3zOvZB7q2eTX2UCVmW/ZsN3Nlc9aj/FAYDNXvDs52OgItt/UwErKUYuTPF+H5dbaK6T+ X-Gm-Message-State: AOJu0Yxf2+3UfRBW0D091ZHFa1guaf8G44QN7UE7SuT9rnjzwfrWO5mA a3p1LlVd2h+8oSHLl5jNqxjWZlBL5bvrJxsz2O61ylxHvd6/BXnTvPjqlMvZlMM= X-Google-Smtp-Source: AGHT+IFPBW1mn5KeYnZcFZpLT4fLpEaOfFIUvQMa/GHHDd1nNhnkQYgHstBKdHhKt1mlaNsMcxCEDw== X-Received: by 2002:a05:600c:4ed2:b0:426:5b44:2be7 with SMTP id 5b1f17b1804b1-429ed79568amr68241915e9.10.1723963495566; Sat, 17 Aug 2024 23:44:55 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ded180dcsm120623885e9.4.2024.08.17.23.44.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:44:55 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v5 09/13] asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock Date: Sun, 18 Aug 2024 08:35:34 +0200 Message-Id: <20240818063538.6651-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234457_927876_D72B8052 X-CRM114-Status: GOOD ( 12.85 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren The arch_spinlock_t of qspinlock has contained the atomic_t val, which satisfies the ticket-lock requirement. Thus, unify the arch_spinlock_t into qspinlock_types.h. This is the preparation for the next combo spinlock. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones --- include/asm-generic/spinlock.h | 14 +++++++------- include/asm-generic/spinlock_types.h | 12 ++---------- 2 files changed, 9 insertions(+), 17 deletions(-) diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 90803a826ba0..4773334ee638 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -32,7 +32,7 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) { - u32 val = atomic_fetch_add(1<<16, lock); + u32 val = atomic_fetch_add(1<<16, &lock->val); u16 ticket = val >> 16; if (ticket == (u16)val) @@ -46,31 +46,31 @@ static __always_inline void arch_spin_lock(arch_spinlock_t *lock) * have no outstanding writes due to the atomic_fetch_add() the extra * orderings are free. */ - atomic_cond_read_acquire(lock, ticket == (u16)VAL); + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); smp_mb(); } static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) { - u32 old = atomic_read(lock); + u32 old = atomic_read(&lock->val); if ((old >> 16) != (old & 0xffff)) return false; - return atomic_try_cmpxchg(lock, &old, old + (1<<16)); /* SC, for RCsc */ + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ } static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) { u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); smp_store_release(ptr, (u16)val + 1); } static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) { - u32 val = lock.counter; + u32 val = lock.val.counter; return ((val >> 16) == (val & 0xffff)); } @@ -84,7 +84,7 @@ static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) { - u32 val = atomic_read(lock); + u32 val = atomic_read(&lock->val); return (s16)((val >> 16) - (val & 0xffff)) > 1; } diff --git a/include/asm-generic/spinlock_types.h b/include/asm-generic/spinlock_types.h index 8962bb730945..f534aa5de394 100644 --- a/include/asm-generic/spinlock_types.h +++ b/include/asm-generic/spinlock_types.h @@ -3,15 +3,7 @@ #ifndef __ASM_GENERIC_SPINLOCK_TYPES_H #define __ASM_GENERIC_SPINLOCK_TYPES_H -#include -typedef atomic_t arch_spinlock_t; - -/* - * qrwlock_types depends on arch_spinlock_t, so we must typedef that before the - * include. - */ -#include - -#define __ARCH_SPIN_LOCK_UNLOCKED ATOMIC_INIT(0) +#include +#include #endif /* __ASM_GENERIC_SPINLOCK_TYPES_H */ From patchwork Sun Aug 18 06:35:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E54BC52D7C for ; Sun, 18 Aug 2024 06:46:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6gOW+9DFDfvc59tscnOEuJAhRvEpKvrcN19n2IIDvRI=; b=vkRFtJWRRbYeSI V4e7058JOaCZMhxtn7Gl+Y5XaQvsUZz9ScpJXB8pyh6sP4gynX2w8vMSPZmdj05C5364yb2vDYan3 lQFSox87QrF2BogIYWVf05Cs8vHgakANTvQcE2kWLnKfIxk1j2NGt7cN3iRDl5FGMOaIPhzc1Yri+ NQm24UYWXPzratnEJ7cOmMQ82hYZ474tzUnTGCZNb6pQOQxA2XaAdgjflLbBgvywJznHz/IYLwz/t 72Cnx1ewEnWaFa0DiRn2o/Zh4jXpg0UCFX4JJkhoNaG1/yBxAIInxDgxKRIvvjNyj50GhqF/Sh2TF n8sWYomDen8uUO3RjYXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZfv-0000000GEfl-07OQ; Sun, 18 Aug 2024 06:46:03 +0000 Received: from mail-wr1-x432.google.com ([2a00:1450:4864:20::432]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZfr-0000000GEev-2Cl6 for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:46:01 +0000 Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-3719896b7c8so1004433f8f.3 for ; Sat, 17 Aug 2024 23:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963557; x=1724568357; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J1DJp3g8ecAC/DPl4n2lm0ysWSqC430kG0AnG9TGYTU=; b=UskcowtEZoI01xqjDhET94SZfkPyppizYyTilpM6q5+FN2veoMUy+lnIrtyHk4nIO3 /+Jg5bC/IqbfzvWYl8QUn854AwW5/CDTB6hdAFTu6hqjC7WQj6JAlHZ/YQfSstamSfFP NUWtpNrFg9khsRMXMkisH4av2gKh18E+vl3pZcFPF08GgYsLv9ldWjWtf8Mbpi8Oldtj 2xlQ0ErNy4WbyxEHQO59cy14j+rBqZEDa8wzuEmJXf4sgGRy27Wk885F3708/Z3RZ+RD 2hGVDLDjThUMrcMlSyThoT/JQKARSDtz3pTsAXZAJ0VUSAAEjecF81QWAlZBScynkeb8 b/ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963557; x=1724568357; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J1DJp3g8ecAC/DPl4n2lm0ysWSqC430kG0AnG9TGYTU=; b=aKToenzanzf3pBvqIKDG8SVi5SqK1jypjFcXvScRx7id5dwoW1VpOSSustYMJv/P4Z hB5MSEx1ACT7TzuMBpvqJkkZHhx1WYgWwWCxO5m0ceTqmBC9wXUr8RkosF99hGl31/Ph LroHN+68p4gVs+SBoS/rSjWQV+3CIqy6OHoZO/XXZ1cU8bgHSxXz1Iue5Nqf9ePJJAb9 34n6+MuaV1lL+kdhkJy6Liy01KN2+BW7m/qEsK05zfC02nliKCDOSrXC9GpKqia+oJEE M1JVvwEbmJkdB4KqiCBNhbu887MG+Gl4omdR/oJMIGEes2UpNtvWOdPvsmSUp4HR9pdU roLw== X-Forwarded-Encrypted: i=1; AJvYcCViZbt80xTOk6QJwXGpOvXLT7dNZNwa7wf1oDI/yznenEQf0SPFxIPhXCibkcnhF9ZccM40eFZTughKIsd9nenKIHiQygGNXgPcXSPh50Nm X-Gm-Message-State: AOJu0YzRH0XoT7KtHEH9d79uy7FPjoJxKh7zZHsQHRSEg8Ii8UOlqXgW qGhBr3JlRo5scSWU4IwxK3Ch+G1DDU4lCanwmYBbVeg7omtDISsyTGpIYAe8Chc= X-Google-Smtp-Source: AGHT+IHPeK5VmE03PU4h9/i27orpAxZrVCijc8uGk6SFEjR1/wG06pKyXQz/+RBNMzRG1dmIOL6ROQ== X-Received: by 2002:a5d:6911:0:b0:368:3717:10aa with SMTP id ffacd0b85a97d-371a73dbb2fmr2311786f8f.11.1723963557033; Sat, 17 Aug 2024 23:45:57 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-37189849cdbsm7338955f8f.28.2024.08.17.23.45.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:45:56 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Guo Ren Subject: [PATCH v5 10/13] asm-generic: ticket-lock: Add separate ticket-lock.h Date: Sun, 18 Aug 2024 08:35:35 +0200 Message-Id: <20240818063538.6651-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234559_618310_203B974F X-CRM114-Status: GOOD ( 21.56 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren Add a separate ticket-lock.h to include multiple spinlock versions and select one at compile time or runtime. Reviewed-by: Leonardo Bras Suggested-by: Arnd Bergmann Link: https://lore.kernel.org/linux-riscv/CAK8P3a2rnz9mQqhN6-e0CGUUv9rntRELFdxt_weiD7FxH7fkfQ@mail.gmail.com/ Signed-off-by: Guo Ren Signed-off-by: Guo Ren Acked-by: Peter Zijlstra (Intel) Reviewed-by: Andrew Jones --- include/asm-generic/spinlock.h | 87 +--------------------- include/asm-generic/ticket_spinlock.h | 103 ++++++++++++++++++++++++++ 2 files changed, 104 insertions(+), 86 deletions(-) create mode 100644 include/asm-generic/ticket_spinlock.h diff --git a/include/asm-generic/spinlock.h b/include/asm-generic/spinlock.h index 4773334ee638..970590baf61b 100644 --- a/include/asm-generic/spinlock.h +++ b/include/asm-generic/spinlock.h @@ -1,94 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * 'Generic' ticket-lock implementation. - * - * It relies on atomic_fetch_add() having well defined forward progress - * guarantees under contention. If your architecture cannot provide this, stick - * to a test-and-set lock. - * - * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a - * sub-word of the value. This is generally true for anything LL/SC although - * you'd be hard pressed to find anything useful in architecture specifications - * about this. If your architecture cannot do this you might be better off with - * a test-and-set. - * - * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence - * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with - * a full fence after the spin to upgrade the otherwise-RCpc - * atomic_cond_read_acquire(). - * - * The implementation uses smp_cond_load_acquire() to spin, so if the - * architecture has WFE like instructions to sleep instead of poll for word - * modifications be sure to implement that (see ARM64 for example). - * - */ - #ifndef __ASM_GENERIC_SPINLOCK_H #define __ASM_GENERIC_SPINLOCK_H -#include -#include - -static __always_inline void arch_spin_lock(arch_spinlock_t *lock) -{ - u32 val = atomic_fetch_add(1<<16, &lock->val); - u16 ticket = val >> 16; - - if (ticket == (u16)val) - return; - - /* - * atomic_cond_read_acquire() is RCpc, but rather than defining a - * custom cond_read_rcsc() here we just emit a full fence. We only - * need the prior reads before subsequent writes ordering from - * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we - * have no outstanding writes due to the atomic_fetch_add() the extra - * orderings are free. - */ - atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); - smp_mb(); -} - -static __always_inline bool arch_spin_trylock(arch_spinlock_t *lock) -{ - u32 old = atomic_read(&lock->val); - - if ((old >> 16) != (old & 0xffff)) - return false; - - return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ -} - -static __always_inline void arch_spin_unlock(arch_spinlock_t *lock) -{ - u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); - u32 val = atomic_read(&lock->val); - - smp_store_release(ptr, (u16)val + 1); -} - -static __always_inline int arch_spin_value_unlocked(arch_spinlock_t lock) -{ - u32 val = lock.val.counter; - - return ((val >> 16) == (val & 0xffff)); -} - -static __always_inline int arch_spin_is_locked(arch_spinlock_t *lock) -{ - arch_spinlock_t val = READ_ONCE(*lock); - - return !arch_spin_value_unlocked(val); -} - -static __always_inline int arch_spin_is_contended(arch_spinlock_t *lock) -{ - u32 val = atomic_read(&lock->val); - - return (s16)((val >> 16) - (val & 0xffff)) > 1; -} - +#include #include #endif /* __ASM_GENERIC_SPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h new file mode 100644 index 000000000000..cfcff22b37b3 --- /dev/null +++ b/include/asm-generic/ticket_spinlock.h @@ -0,0 +1,103 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * 'Generic' ticket-lock implementation. + * + * It relies on atomic_fetch_add() having well defined forward progress + * guarantees under contention. If your architecture cannot provide this, stick + * to a test-and-set lock. + * + * It also relies on atomic_fetch_add() being safe vs smp_store_release() on a + * sub-word of the value. This is generally true for anything LL/SC although + * you'd be hard pressed to find anything useful in architecture specifications + * about this. If your architecture cannot do this you might be better off with + * a test-and-set. + * + * It further assumes atomic_*_release() + atomic_*_acquire() is RCpc and hence + * uses atomic_fetch_add() which is RCsc to create an RCsc hot path, along with + * a full fence after the spin to upgrade the otherwise-RCpc + * atomic_cond_read_acquire(). + * + * The implementation uses smp_cond_load_acquire() to spin, so if the + * architecture has WFE like instructions to sleep instead of poll for word + * modifications be sure to implement that (see ARM64 for example). + * + */ + +#ifndef __ASM_GENERIC_TICKET_SPINLOCK_H +#define __ASM_GENERIC_TICKET_SPINLOCK_H + +#include +#include + +static __always_inline void ticket_spin_lock(arch_spinlock_t *lock) +{ + u32 val = atomic_fetch_add(1<<16, &lock->val); + u16 ticket = val >> 16; + + if (ticket == (u16)val) + return; + + /* + * atomic_cond_read_acquire() is RCpc, but rather than defining a + * custom cond_read_rcsc() here we just emit a full fence. We only + * need the prior reads before subsequent writes ordering from + * smb_mb(), but as atomic_cond_read_acquire() just emits reads and we + * have no outstanding writes due to the atomic_fetch_add() the extra + * orderings are free. + */ + atomic_cond_read_acquire(&lock->val, ticket == (u16)VAL); + smp_mb(); +} + +static __always_inline bool ticket_spin_trylock(arch_spinlock_t *lock) +{ + u32 old = atomic_read(&lock->val); + + if ((old >> 16) != (old & 0xffff)) + return false; + + return atomic_try_cmpxchg(&lock->val, &old, old + (1<<16)); /* SC, for RCsc */ +} + +static __always_inline void ticket_spin_unlock(arch_spinlock_t *lock) +{ + u16 *ptr = (u16 *)lock + IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + u32 val = atomic_read(&lock->val); + + smp_store_release(ptr, (u16)val + 1); +} + +static __always_inline int ticket_spin_value_unlocked(arch_spinlock_t lock) +{ + u32 val = lock.val.counter; + + return ((val >> 16) == (val & 0xffff)); +} + +static __always_inline int ticket_spin_is_locked(arch_spinlock_t *lock) +{ + arch_spinlock_t val = READ_ONCE(*lock); + + return !ticket_spin_value_unlocked(val); +} + +static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) +{ + u32 val = atomic_read(&lock->val); + + return (s16)((val >> 16) - (val & 0xffff)) > 1; +} + +/* + * Remapping spinlock architecture specific functions to the corresponding + * ticket spinlock functions. + */ +#define arch_spin_is_locked(l) ticket_spin_is_locked(l) +#define arch_spin_is_contended(l) ticket_spin_is_contended(l) +#define arch_spin_value_unlocked(l) ticket_spin_value_unlocked(l) +#define arch_spin_lock(l) ticket_spin_lock(l) +#define arch_spin_trylock(l) ticket_spin_trylock(l) +#define arch_spin_unlock(l) ticket_spin_unlock(l) + +#endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */ From patchwork Sun Aug 18 06:35:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55F82C52D7C for ; Sun, 18 Aug 2024 06:47:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Pdpi0iF+85MUbNtYw/j4psbCATe1VeHpcQu2RpdIiu0=; b=A8ph7owhwzX2ft SP+6AqnCtaqReBn6X4ICr7a4SwOrLYvvJnpsQr6yhdNr1XK/ktfUdXgy/vX2Vg9mJ2uAGZafyGzI2 9nDrl7vAmWg81+5K5zEu7f+uLlH34Cu4oXWolF0jdau43EZGeskWsOsWQS0+sCv74e3F0KQyxkiSe mGKGq98RaZGLRKbsWa/eWOb5X6AR7EZRlpnUft5Vh4PkMc3ym9BBGRPzXn0Mlz4+DZp+pR0+rLY/R fzVFqY/mu3QslBHUNjeVRFYWfg7C3HDTZHgjB2vEobPnKX5qGyiO/MPBNuISY5/ymP2NIGf+kIlMX +7uH3HGwUON9Mo3hsuFw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZgu-0000000GEoc-0Qrg; Sun, 18 Aug 2024 06:47:04 +0000 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZgq-0000000GEnq-3OoJ for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:47:02 +0000 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-428243f928fso34977405e9.0 for ; Sat, 17 Aug 2024 23:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963619; x=1724568419; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vlqxfa1Dgyr/RCZKCOdmB4jO+WlSQVOulpijR1tIjck=; b=tp8m7w4CE8yXrxcSM9imCLoH+sVDA9gm7PewEFS+ykEt+AxmY15+DWCL9v7hZE7G7Q 4YOjmMWp0S3r3srg+RxU0Pw2GOslRT16HLzGZC3T17Vg6f4QrK2TGyyWc27LRrNvGwUI WcPc94SRDZ86Q4/MPxkZoBH56HOE+bt6f6pZUVHMEggaBSAzQjSmGusqKANvpD5+jYP9 n6z41jleGXCX8MIU/mzaD7yhPDBw8YjvHkp0VGrw8vh2m5OujB7X1GWXWh4d2Qc6z59D tYP05jFO6uuE/VDVXSRKltgY7dZ54ZCtvRoycWJVuos3/hAcrtf+Ar04xoHi5pI16mnW 2+Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963619; x=1724568419; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vlqxfa1Dgyr/RCZKCOdmB4jO+WlSQVOulpijR1tIjck=; b=qr4IETB5c5wtMuCXG2gpF/ayuYekTg39PfSQJVrRZiwudRSjSMcC3whBmEtcpxKl5D ZjXoMg7nKehFOmG1CraMwAh3E7EoEDcSbots2cFRp3mQ6z3ZCWobg7XgjK6yI9PXoFMH gPTQu8JWADoG0JnJovawQ3InB+SLL3k3Km5RMHDm13PmaOKnBlNAMG+/BzOmOmXbgRrI GlDGwwBDbNBJZdos8ZrCw8BFBrrHYjdHd6D1FK25a2SQq/J67c0C8jwcFKiMbi7Iurn/ hUQTF2EIODoAjOHrLZZttktfGsFm/+gQZ8tK6ai/H/nkPtv4eO5I9uUNYUP7mknWDs7b gRtA== X-Forwarded-Encrypted: i=1; AJvYcCUymY419WoFiHV2CaGWgYvdRfbIxYa3tM+zJN0hHBruvUKjB8VF/d6EJtos4VeZXU+6GV9stUadKEL7kdXN/jo08sBNvMrfXV1I4HtDt+gD X-Gm-Message-State: AOJu0YzhVQePD2eDOXovb31k8WT1nOLFDauucoOdU8wNJsYSMR4Ox5XS 9ZElxfkjXcu3+VqYONophjXEClVAvkm/o27QFWbRqUNiYwdEzK/Phz6P85VGXWk= X-Google-Smtp-Source: AGHT+IFwS3MzzU8TURVJ4Ap0a8/3o7r3oh1yyMOzNik5dKrEar+UZ6Tut2SPaHafSLHiL9m2WkM9gg== X-Received: by 2002:a5d:694a:0:b0:367:dc45:55ab with SMTP id ffacd0b85a97d-371944561f9mr6249467f8f.25.1723963618917; Sat, 17 Aug 2024 23:46:58 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-371898b89bdsm7269358f8f.112.2024.08.17.23.46.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:46:58 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 11/13] riscv: Add ISA extension parsing for Ziccrse Date: Sun, 18 Aug 2024 08:35:36 +0200 Message-Id: <20240818063538.6651-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240817_234700_870391_B59542AB X-CRM114-Status: UNSURE ( 9.97 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add support to parse the Ziccrse string in the riscv,isa string. Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- arch/riscv/include/asm/hwcap.h | 1 + arch/riscv/kernel/cpufeature.c | 1 + 2 files changed, 2 insertions(+) diff --git a/arch/riscv/include/asm/hwcap.h b/arch/riscv/include/asm/hwcap.h index f5d53251c947..9e228b079a6d 100644 --- a/arch/riscv/include/asm/hwcap.h +++ b/arch/riscv/include/asm/hwcap.h @@ -93,6 +93,7 @@ #define RISCV_ISA_EXT_ZCMOP 84 #define RISCV_ISA_EXT_ZAWRS 85 #define RISCV_ISA_EXT_ZABHA 86 +#define RISCV_ISA_EXT_ZICCRSE 87 #define RISCV_ISA_EXT_XLINUXENVCFG 127 diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 67ebcc4c9424..ea9c255bbe3d 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -314,6 +314,7 @@ const struct riscv_isa_ext_data riscv_isa_ext[] = { riscv_ext_zicbom_validate), __RISCV_ISA_EXT_SUPERSET_VALIDATE(zicboz, RISCV_ISA_EXT_ZICBOZ, riscv_xlinuxenvcfg_exts, riscv_ext_zicboz_validate), + __RISCV_ISA_EXT_DATA(ziccrse, RISCV_ISA_EXT_ZICCRSE), __RISCV_ISA_EXT_DATA(zicntr, RISCV_ISA_EXT_ZICNTR), __RISCV_ISA_EXT_DATA(zicond, RISCV_ISA_EXT_ZICOND), __RISCV_ISA_EXT_DATA(zicsr, RISCV_ISA_EXT_ZICSR), From patchwork Sun Aug 18 06:35:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BCF42C52D7C for ; Sun, 18 Aug 2024 08:00:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9d6e4TppBsxCPspoHJ2T56P9zoZYUvO5M9jEWurh1+8=; b=4VEHn1nFre1mfr LADj5NX+P2u975dR7t3CWpI9l8+l5vQ7205d8fvU9P6z9aHR4Yu+L+9lUzS9fqyAOR8+NYqYj24Vl 4lgWVyER6lTID939q95/75GGCN9gd6F5vk6YmT++I2te6i3XstJ1h273zHbYhU9ClE4gzrHZKjE+l qQyPr1faCPoQ3e8j+GL2bGDOBO0uq/RU39LLYketX+aVcr52DEvoSdqVlAukoi4EO4qiFAifKPzEB OPokS+PnKuS28ZZijltpbxlH84YQqmDoRjvbUe74IGLE0IHiwus+Q22ITju5yJVsyuWoNE0kJbmHA m3ltBUSehoxXA/OmKAUQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfapi-0000000GKfI-08A3; Sun, 18 Aug 2024 08:00:14 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZhw-0000000GExO-1mtW for linux-riscv@bombadil.infradead.org; Sun, 18 Aug 2024 06:48:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=AUJVvfa71x+iwF5DF3rrBv6vNd03sOzmOgv8ZqY/vyI=; b=Qzn+4afgCwqo5IgC3Tt1VPDc68 nW9ZUMkjmFVZMu+5ylq4U5X7Orzj2D437c1Cl2Sr8Bwb9GNLo6SElNlDGUdaig+UGA7qNyvH1YLMT +LtX2IwxYxqIL/pfSEXCzDG6UE5fDPtnoOTRanPV6tpsB5zmg/cq4yqtu1tP7j3gQ9WxIzTF3p0SX 6V+flcObyhx8L2w2pPTuCTXHniKKEMmi5PpLAjuzvEB4A2xpCUpEHqJWi3Sslo51SUC6VqJGJ3BMR ml32br4FLh/4RH9ciBysbx+MYQDM16bAbuhpuiQPH6SyBmuMZljPMbxIxSH842Tf7IZVX7BgsHeeV J6KAaHEw==; Received: from mail-wr1-x430.google.com ([2a00:1450:4864:20::430]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZhs-00000008xTx-3HeZ for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:48:07 +0000 Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-37196229343so1258951f8f.0 for ; Sat, 17 Aug 2024 23:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963680; x=1724568480; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AUJVvfa71x+iwF5DF3rrBv6vNd03sOzmOgv8ZqY/vyI=; b=GuaL/yM61guvyBRqNTOeCdbt2ZKpR7rfN4jaAuz4WdkGvPAT5cTkGPAVnAC0C12ATS 8i4VfqsGrdZAZHvt21rf5BF7rFJI+7e1IG/pR9byW0z55nXBo1z6dQ/+V/gCCGC6ySTw g8yhON/5/IKWl4OBMtqmB5AkgIAJ3fLg9Ku01s7oxF7zv/zZkJWbJ3r38WVMQ1T7p5ld VC3gQmRFWP4KK7Z+Aqrd35LK6ugUZdBRjp6VuzUh+9jXKTBjWDWwzcVjGlcuTqfpwiVb oZqLLITHF1JvqG6DbWyXCFqU5W7TsTrMu6rj2ndTE6LRw3Fd9vqnjZXc60wNMIk7SNTu KMTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963680; x=1724568480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AUJVvfa71x+iwF5DF3rrBv6vNd03sOzmOgv8ZqY/vyI=; b=V6IMp2AzUQflbGNS52b/LtT2+/IhNR+pC806LD5mWNtThCsXJxfu3KnpYMq0+/lkMJ a6pqkq3edF8BJ0G1sMTicElfJYXQAJ00x3JGIEybH8T3kv7Z15T8YA/cR1aerq3oGKXw BjiO/gOSYWPdCdP7WpYMNHexHHNiog74KR/EZrouBF0tQhPhcTyx/3f2+nJwStWKqjxE OX7z6dQ4G3IP3yZE2CVbZb8+5JNNzvVR8/W5nA3rTgPjgZmg+y8lKWG5/chVVVre8MlA EWWOsoc2+Cz3Tm2+ZzQheKWHSgez7SJFgYlP8hA4uDoyosJtu8QlInckgMwYDfN1/bhh FLDg== X-Forwarded-Encrypted: i=1; AJvYcCUEu3gynDkSFBsx8tukmzZZwNGhAHFYYPcuOZVIY+H7UtDOSkq1Ud2GqAVrZlCS5Sux9H0W1o54/9OlYHLKculsz3YY6fSwuL9sAeqMCjnD X-Gm-Message-State: AOJu0YwBokxHhlF5grla3JU7K8x/l291W81irnT9NQ02SB1Y3wolAtpM OpW4/bmHPa/nxhvBxOHzC5wR0sZsxNdE3sXutvSLWvHpCbwI5Hyn6HqBfjO3sco= X-Google-Smtp-Source: AGHT+IHGhKQ874lL1J7nsjPh6FSfKHuvYCqLot3YTtCsdD7yaCkb97X7rPbK0sQldMNV0hrVqhCG+Q== X-Received: by 2002:a5d:53c5:0:b0:367:8a00:fac3 with SMTP id ffacd0b85a97d-371946514e9mr5069861f8f.30.1723963680234; Sat, 17 Aug 2024 23:48:00 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3718983a1b8sm7336054f8f.5.2024.08.17.23.47.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:47:59 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 12/13] dt-bindings: riscv: Add Ziccrse ISA extension description Date: Sun, 18 Aug 2024 08:35:37 +0200 Message-Id: <20240818063538.6651-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240818_074805_470675_7B3B521A X-CRM114-Status: UNSURE ( 8.47 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Add description for the Ziccrse ISA extension which was ratified in the riscv profiles specification v1.0. Signed-off-by: Alexandre Ghiti Reviewed-by: Guo Ren Acked-by: Conor Dooley Reviewed-by: Andrew Jones --- Documentation/devicetree/bindings/riscv/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Documentation/devicetree/bindings/riscv/extensions.yaml b/Documentation/devicetree/bindings/riscv/extensions.yaml index a63578b95c4a..4f174c4c08ff 100644 --- a/Documentation/devicetree/bindings/riscv/extensions.yaml +++ b/Documentation/devicetree/bindings/riscv/extensions.yaml @@ -289,6 +289,12 @@ properties: in commit 64074bc ("Update version numbers for Zfh/Zfinx") of riscv-isa-manual. + - const: ziccrse + description: + The standard Ziccrse extension which provides forward progress + guarantee on LR/SC sequences, as ratified in commit b1d806605f87 + ("Updated to ratified state.") of the riscv profiles specification. + - const: zk description: The standard Zk Standard Scalar cryptography extension as ratified From patchwork Sun Aug 18 06:35:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13767365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2531AC5320E for ; Sun, 18 Aug 2024 08:00:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BtEECD+GxIsS6xb89XnfjhXCUme+PLH+v2e4yLoZip4=; b=KrqXqMQhLK45aE CF+Pe2jVr+F+jOHalu/laEd/lC36VDm9kv1FEyKF/ycm1SRvc6+Xu8l+4ebNT62p9lL03CnRzMisV e/1pvWWusOjK5FmUl+xCN50T54QoTeP0mpxcBEUkDZrRjOnoN5DKzLRz01DIZjRZW/xTANacHgzTS 9SOEAeln+myF0cfrnORrA7ToDXwe0FwiEFa28cbYE18YkW/GGyW4/2jyNDQcovUXzISvX35ihRsab mog3iLGr8SoQp2M2SqOPvgA7kmvRs6djFIJojmcrNPKZrFRNJZJP2WHCIcW8BGAYRBHfy/szvLV0k xqoHiSSpNGujsgr/bRwg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfapi-0000000GKfN-3IFc; Sun, 18 Aug 2024 08:00:14 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZiu-0000000GF0L-1f6N for linux-riscv@bombadil.infradead.org; Sun, 18 Aug 2024 06:49:08 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=luYpQp4OVqMmof3eSGbnJOpDEcugh3MQdyInZ+zIklA=; b=R91Q6FJQvA/mj6HjM1LEiN0Ee6 eUvkn7E3/ebicSvxuUnSi4JtRNnsFgxZH2HTNgJmVXIfMrA7xWk+gT5U66kmPLyKBza2hSqK7aL9r ST89MvRVmdLJcjaCmBRyBEQ+KhH8eOpN1hQNQgK4dr3A2IhUnBX3yWHYI0UjAqOhKLNWxvLoTl5rQ R2xGdonxV0KJkuHhX+0frvlPP628dlmWoCGZNpzuR1EtPWnNHbVEOSm3112J2KRyjedP3JcEahGlF SOVOGJJM7+9cnZO/P7L9A2Gqzfwe58m81BVUOU/YwlPjO5j0VVArU5yFWKRqpodvsZXyIgQt59GIx woI7NNYA==; Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sfZiq-00000008xYP-1ykU for linux-riscv@lists.infradead.org; Sun, 18 Aug 2024 06:49:07 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-429d2d7be1eso16609345e9.1 for ; Sat, 17 Aug 2024 23:49:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1723963742; x=1724568542; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=luYpQp4OVqMmof3eSGbnJOpDEcugh3MQdyInZ+zIklA=; b=V03k+AvRjGRmWP0SP2ExXBpH2Oe3zCupa79YnblV3ffnxBa3ycRQk1LP6iiTVUGtAx LxaDM8P7CbSILtPqTr24D3xqmaO//FZhXdJZpU7RHmoSPAYQiymCHLddHhHDkixyJW7/ 5vGRgxSUmlXPzmYbZjMWT1aYwEFtVhzyxuL+uQFzS5PXdUrW+uXv0sK4JpYE9ILy2MGK GuNnfmoz1Q2XbCuS0/1JwGq8IYtBIL7lqymoqjYHLtWwhJni4gUEYP5RYj2iTPlNHczc VF/CRHec7DiDX0NOuFT391nFk35yt1z6z1WQXBGQV0rAq0w/TAFG07N+5JqKDP/evwN5 lnyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723963742; x=1724568542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=luYpQp4OVqMmof3eSGbnJOpDEcugh3MQdyInZ+zIklA=; b=vIgTnFkMXMOVakno1oPtwtiaV7JnNE0LuAVcupPjVy3XGklcKFOzEgS6Q5dZjRu0UH tNbUoT/kvjgqgGZoCIS0bSSNfD6QO2sjgh5oH+FhEjQxzLF79aV3l1PIKm9q3ns8OcJX pvi6fI+10Z4svGi5wSI4Tz34bBF4y0EYcQeD0thKTiYDX1IZC3P6LcbJpWeDzaPKhLl3 DKHRCrCVyQj6LbdyDVFbnRXX2uSLntMXNrxj7oOU0iH0OrnuoJ6+6cvlxmN4fsF2Cadk 62fbTcmzk/ublEkJ/cegj1aeyYuY01JuE4/sBC0aJcnUxtEWlfdzw8cU5EmMyZb1VcFP ovPg== X-Forwarded-Encrypted: i=1; AJvYcCUuXrOQ3D1sGu0EveYhXq1totdOjFm7QT0TkqqMZ0a/pjWs1PtJgEeeG1e84NX1iwSMfSKUCe2dLP4UIDK9Hf55GcjdJYOCYL9z8pBCiQ46 X-Gm-Message-State: AOJu0Yy832m2ko1yB1K5R8RYfX/IHAGsHOw0p2QakzjXmYf7Zbke62CT c7l0uPoyZZZhL4bDXcSztrGU5D9nh7y0yHEp9i08qRNj0dLrR6XDWMfniYSXfWU= X-Google-Smtp-Source: AGHT+IEsvkhvyT/0ituMtpPmeaM6FNA/kWbI3AHcBR+0rlpsw+92Yz8DA6J8n1o3o1wi0guN4MhOfg== X-Received: by 2002:a05:600c:4e91:b0:426:64c1:8388 with SMTP id 5b1f17b1804b1-429eda23e74mr44379405e9.17.1723963741667; Sat, 17 Aug 2024 23:49:01 -0700 (PDT) Received: from alex-rivos.guest.squarehotel.net ([130.93.157.50]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-429ed648f0csm66770755e9.9.2024.08.17.23.49.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 17 Aug 2024 23:49:01 -0700 (PDT) From: Alexandre Ghiti To: Jonathan Corbet , Paul Walmsley , Palmer Dabbelt , Albert Ou , Conor Dooley , Rob Herring , Krzysztof Kozlowski , Andrea Parri , Nathan Chancellor , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng , Arnd Bergmann , Leonardo Bras , Guo Ren , linux-doc@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-arch@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH v5 13/13] riscv: Add qspinlock support Date: Sun, 18 Aug 2024 08:35:38 +0200 Message-Id: <20240818063538.6651-14-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240818063538.6651-1-alexghiti@rivosinc.com> References: <20240818063538.6651-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240818_074905_145926_CD3204E4 X-CRM114-Status: GOOD ( 27.43 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org In order to produce a generic kernel, a user can select CONFIG_COMBO_SPINLOCKS which will fallback at runtime to the ticket spinlock implementation if Zabha or Ziccrse are not present. Note that we can't use alternatives here because the discovery of extensions is done too late and we need to start with the qspinlock implementation because the ticket spinlock implementation would pollute the spinlock value, so let's use static keys. This is largely based on Guo's work and Leonardo reviews at [1]. Link: https://lore.kernel.org/linux-riscv/20231225125847.2778638-1-guoren@kernel.org/ [1] Signed-off-by: Guo Ren Signed-off-by: Alexandre Ghiti Reviewed-by: Andrew Jones --- .../locking/queued-spinlocks/arch-support.txt | 2 +- arch/riscv/Kconfig | 34 ++++++++++++++ arch/riscv/include/asm/Kbuild | 4 +- arch/riscv/include/asm/spinlock.h | 47 +++++++++++++++++++ arch/riscv/kernel/setup.c | 37 +++++++++++++++ include/asm-generic/qspinlock.h | 2 + include/asm-generic/ticket_spinlock.h | 2 + 7 files changed, 126 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/spinlock.h diff --git a/Documentation/features/locking/queued-spinlocks/arch-support.txt b/Documentation/features/locking/queued-spinlocks/arch-support.txt index 22f2990392ff..cf26042480e2 100644 --- a/Documentation/features/locking/queued-spinlocks/arch-support.txt +++ b/Documentation/features/locking/queued-spinlocks/arch-support.txt @@ -20,7 +20,7 @@ | openrisc: | ok | | parisc: | TODO | | powerpc: | ok | - | riscv: | TODO | + | riscv: | ok | | s390: | TODO | | sh: | TODO | | sparc: | ok | diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index ef55ab94027e..201d0669db7f 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -79,6 +79,7 @@ config RISCV select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE + select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU select BUILDTIME_TABLE_SORT if MMU select CLINT_TIMER if RISCV_M_MODE @@ -488,6 +489,39 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +choice + prompt "RISC-V spinlock type" + default RISCV_COMBO_SPINLOCKS + +config RISCV_TICKET_SPINLOCKS + bool "Using ticket spinlock" + +config RISCV_QUEUED_SPINLOCKS + bool "Using queued spinlock" + depends on SMP && MMU && NONPORTABLE + select ARCH_USE_QUEUED_SPINLOCKS + help + The queued spinlock implementation requires the forward progress + guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or + LR/SC with Ziccrse provide such guarantee. + + Select this if and only if Zabha or Ziccrse is available on your + platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms + without one of those extensions. + + If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks + when supported and otherwise ticket spinlocks. + +config RISCV_COMBO_SPINLOCKS + bool "Using combo spinlock" + depends on SMP && MMU + select ARCH_USE_QUEUED_SPINLOCKS + help + Embed both queued spinlock and ticket lock so that the spinlock + implementation can be chosen at runtime. + +endchoice + config RISCV_ALTERNATIVE bool depends on !XIP_KERNEL diff --git a/arch/riscv/include/asm/Kbuild b/arch/riscv/include/asm/Kbuild index 5c589770f2a8..1c2618c964f0 100644 --- a/arch/riscv/include/asm/Kbuild +++ b/arch/riscv/include/asm/Kbuild @@ -5,10 +5,12 @@ syscall-y += syscall_table_64.h generic-y += early_ioremap.h generic-y += flat.h generic-y += kvm_para.h +generic-y += mcs_spinlock.h generic-y += parport.h -generic-y += spinlock.h generic-y += spinlock_types.h +generic-y += ticket_spinlock.h generic-y += qrwlock.h generic-y += qrwlock_types.h +generic-y += qspinlock.h generic-y += user.h generic-y += vmlinux.lds.h diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h new file mode 100644 index 000000000000..e5121b89acea --- /dev/null +++ b/arch/riscv/include/asm/spinlock.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef __ASM_RISCV_SPINLOCK_H +#define __ASM_RISCV_SPINLOCK_H + +#ifdef CONFIG_RISCV_COMBO_SPINLOCKS +#define _Q_PENDING_LOOPS (1 << 9) + +#define __no_arch_spinlock_redefine +#include +#include +#include + +/* + * TODO: Use an alternative instead of a static key when we are able to parse + * the extensions string earlier in the boot process. + */ +DECLARE_STATIC_KEY_TRUE(qspinlock_key); + +#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \ +static __always_inline type arch_spin_##op(type_lock lock) \ +{ \ + if (static_branch_unlikely(&qspinlock_key)) \ + return queued_spin_##op(lock); \ + return ticket_spin_##op(lock); \ +} + +SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *) +SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t) + +#elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS) + +#include + +#else + +#include + +#endif + +#include + +#endif /* __ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c index a2cde65b69e9..438e4f6ad2ad 100644 --- a/arch/riscv/kernel/setup.c +++ b/arch/riscv/kernel/setup.c @@ -244,6 +244,42 @@ static void __init parse_dtb(void) #endif } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) +DEFINE_STATIC_KEY_TRUE(qspinlock_key); +EXPORT_SYMBOL(qspinlock_key); +#endif + +static void __init riscv_spinlock_init(void) +{ + char *using_ext = NULL; + + if (IS_ENABLED(CONFIG_RISCV_TICKET_SPINLOCKS)) { + pr_info("Ticket spinlock: enabled\n"); + return; + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && + IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && + riscv_isa_extension_available(NULL, ZABHA) && + riscv_isa_extension_available(NULL, ZACAS)) { + using_ext = "using Zabha"; + } else if (riscv_isa_extension_available(NULL, ZICCRSE)) { + using_ext = "using Ziccrse"; + } +#if defined(CONFIG_RISCV_COMBO_SPINLOCKS) + else { + static_branch_disable(&qspinlock_key); + pr_info("Ticket spinlock: enabled\n"); + return; + } +#endif + + if (!using_ext) + pr_err("Queued spinlock without Zabha or Ziccrse"); + else + pr_info("Queued spinlock %s: enabled\n", using_ext); +} + extern void __init init_rt_signal_env(void); void __init setup_arch(char **cmdline_p) @@ -297,6 +333,7 @@ void __init setup_arch(char **cmdline_p) riscv_set_dma_cache_alignment(); riscv_user_isa_enable(); + riscv_spinlock_init(); } bool arch_cpu_is_hotpluggable(int cpu) diff --git a/include/asm-generic/qspinlock.h b/include/asm-generic/qspinlock.h index 0655aa5b57b2..bf47cca2c375 100644 --- a/include/asm-generic/qspinlock.h +++ b/include/asm-generic/qspinlock.h @@ -136,6 +136,7 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) } #endif +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * queued spinlock functions. @@ -146,5 +147,6 @@ static __always_inline bool virt_spin_lock(struct qspinlock *lock) #define arch_spin_lock(l) queued_spin_lock(l) #define arch_spin_trylock(l) queued_spin_trylock(l) #define arch_spin_unlock(l) queued_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_QSPINLOCK_H */ diff --git a/include/asm-generic/ticket_spinlock.h b/include/asm-generic/ticket_spinlock.h index cfcff22b37b3..325779970d8a 100644 --- a/include/asm-generic/ticket_spinlock.h +++ b/include/asm-generic/ticket_spinlock.h @@ -89,6 +89,7 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) return (s16)((val >> 16) - (val & 0xffff)) > 1; } +#ifndef __no_arch_spinlock_redefine /* * Remapping spinlock architecture specific functions to the corresponding * ticket spinlock functions. @@ -99,5 +100,6 @@ static __always_inline int ticket_spin_is_contended(arch_spinlock_t *lock) #define arch_spin_lock(l) ticket_spin_lock(l) #define arch_spin_trylock(l) ticket_spin_trylock(l) #define arch_spin_unlock(l) ticket_spin_unlock(l) +#endif #endif /* __ASM_GENERIC_TICKET_SPINLOCK_H */