From patchwork Wed Mar 31 14:30:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guo Ren X-Patchwork-Id: 12175533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83132C433ED for ; Wed, 31 Mar 2021 14:32:29 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 34D4F60FF1 for ; Wed, 31 Mar 2021 14:32:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34D4F60FF1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:MIME-Version:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:References:In-Reply-To:Message-Id:Date:Subject:Cc:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=zbBSaRoW82f3hPJmMXqIakYOcI17WmTOa0Jt+0BU6W8=; b=J5AObR3llBSysu0oUNak+UH2XZ eMFiGiEFuq05+4OjXmRYbAQ63gW4B0zc1nGWniBdPVLfvcZG/ANfo74B5ULeroSFRCAeQ2OvXopJy VkEHZjhtLTdO/xfjACmqudrZ5QPhm3WgFODv5P1nBDEPuwuXEFf1qifMeFwwhy2wwBTRPRdXLcZZ3 mThV22NjEIh7/JanJtPvMVb0ft9uj+yShKpT4D9Zl/cIkhAlzGxyhvoKHG4dx9dpcNpb8KYU3/I9M 04fqg39pw0AwAGPVpSTfyh+zPz9pMVT7emmv4hBHGMrijU4yfidDcJzICTXEOJ2RmSgJt9j9gfW3d tYpKXpbA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lRbtJ-006oSK-32; Wed, 31 Mar 2021 14:32:17 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lRbtE-006oRZ-Um for linux-riscv@lists.infradead.org; Wed, 31 Mar 2021 14:32:15 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 714AF60FF3; Wed, 31 Mar 2021 14:32:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617201131; bh=1nsbIPHoYnp+y4hhU/ERIAox7V175+dXhzkr9rkZCgY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cW2e8TK/MVdFBPif56BK0z+HaiRyKiplVLsKPXZhc9cKMUoNS8THGYKAIdPHTdLPI VqBImBnGowMYxlacVGWlXCBvo5toHf6RttaKq5Gf/nV1Qx1TYuEnnNoWs3wWvyd9Xt wCLPwjmQWHGAhfCL7RGY6EMyqnIs//JUSG6UQ4dQT9zbKIO6D0YYqqyc2+KeAiKq7y opQ4Tf9BJxIbW0LZ54ZcP5KKy8yCmY+XClnU+dvMp8exURVpqAUL3TkxZ0hN674Myl Gd/5gyC8+LdG6hmlXseJDVkvkjvzURTJBCe3fY+B5pcta+OW+EV5FfVNBCwzwYe9ex YjoHB5tQnrqDA== From: guoren@kernel.org To: guoren@kernel.org Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-csky@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-xtensa@linux-xtensa.org, openrisc@lists.librecores.org, sparclinux@vger.kernel.org, Guo Ren , Peter Zijlstra , Anup Patel , Arnd Bergmann Subject: [PATCH v6 3/9] riscv: locks: Introduce ticket-based spinlock implementation Date: Wed, 31 Mar 2021 14:30:34 +0000 Message-Id: <1617201040-83905-4-git-send-email-guoren@kernel.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1617201040-83905-1-git-send-email-guoren@kernel.org> References: <1617201040-83905-1-git-send-email-guoren@kernel.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210331_153213_332403_1D92BEEA X-CRM114-Status: GOOD ( 13.25 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Guo Ren This patch introduces a ticket lock implementation for riscv, along the same lines as the implementation for arch/arm & arch/csky. We still use qspinlock as default. Signed-off-by: Guo Ren Cc: Peter Zijlstra Cc: Anup Patel Cc: Arnd Bergmann --- arch/riscv/Kconfig | 7 ++- arch/riscv/include/asm/spinlock.h | 84 +++++++++++++++++++++++++ arch/riscv/include/asm/spinlock_types.h | 17 +++++ 3 files changed, 107 insertions(+), 1 deletion(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 67cc65ba1ea1..34d0276f01d5 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -34,7 +34,7 @@ config RISCV select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_USE_QUEUED_RWLOCKS - select ARCH_USE_QUEUED_SPINLOCKS + select ARCH_USE_QUEUED_SPINLOCKS if !RISCV_TICKET_LOCK select ARCH_USE_QUEUED_SPINLOCKS_XCHG32 select CLONE_BACKWARDS select CLINT_TIMER if !MMU @@ -344,6 +344,11 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK def_bool y depends on NUMA +config RISCV_TICKET_LOCK + bool "Ticket-based spin-locking" + help + Say Y here to use ticket-based spin-locking. + config RISCV_ISA_C bool "Emit compressed instructions when building Linux" default y diff --git a/arch/riscv/include/asm/spinlock.h b/arch/riscv/include/asm/spinlock.h index a557de67a425..90b7eaa950cf 100644 --- a/arch/riscv/include/asm/spinlock.h +++ b/arch/riscv/include/asm/spinlock.h @@ -7,7 +7,91 @@ #ifndef _ASM_RISCV_SPINLOCK_H #define _ASM_RISCV_SPINLOCK_H +#ifdef CONFIG_RISCV_TICKET_LOCK +#ifdef CONFIG_32BIT +#define __ASM_SLLIW "slli\t" +#define __ASM_SRLIW "srli\t" +#else +#define __ASM_SLLIW "slliw\t" +#define __ASM_SRLIW "srliw\t" +#endif + +/* + * Ticket-based spin-locking. + */ +static inline void arch_spin_lock(arch_spinlock_t *lock) +{ + arch_spinlock_t lockval; + u32 tmp; + + asm volatile ( + "1: lr.w %0, %2 \n" + " mv %1, %0 \n" + " addw %0, %0, %3 \n" + " sc.w %0, %0, %2 \n" + " bnez %0, 1b \n" + : "=&r" (tmp), "=&r" (lockval), "+A" (lock->lock) + : "r" (1 << TICKET_NEXT) + : "memory"); + + smp_cond_load_acquire(&lock->tickets.owner, + VAL == lockval.tickets.next); +} + +static inline int arch_spin_trylock(arch_spinlock_t *lock) +{ + u32 tmp, contended, res; + + do { + asm volatile ( + " lr.w %0, %3 \n" + __ASM_SRLIW "%1, %0, %5 \n" + __ASM_SLLIW "%2, %0, %5 \n" + " or %1, %2, %1 \n" + " li %2, 0 \n" + " sub %1, %1, %0 \n" + " bnez %1, 1f \n" + " addw %0, %0, %4 \n" + " sc.w %2, %0, %3 \n" + "1: \n" + : "=&r" (tmp), "=&r" (contended), "=&r" (res), + "+A" (lock->lock) + : "r" (1 << TICKET_NEXT), "I" (TICKET_NEXT) + : "memory"); + } while (res); + + if (!contended) + __atomic_acquire_fence(); + + return !contended; +} + +static inline void arch_spin_unlock(arch_spinlock_t *lock) +{ + smp_store_release(&lock->tickets.owner, lock->tickets.owner + 1); +} + +static inline int arch_spin_value_unlocked(arch_spinlock_t lock) +{ + return lock.tickets.owner == lock.tickets.next; +} + +static inline int arch_spin_is_locked(arch_spinlock_t *lock) +{ + return !arch_spin_value_unlocked(READ_ONCE(*lock)); +} + +static inline int arch_spin_is_contended(arch_spinlock_t *lock) +{ + struct __raw_tickets tickets = READ_ONCE(lock->tickets); + + return (tickets.next - tickets.owner) > 1; +} +#define arch_spin_is_contended arch_spin_is_contended +#else /* CONFIG_RISCV_TICKET_LOCK */ #include +#endif /* CONFIG_RISCV_TICKET_LOCK */ + #include #endif /* _ASM_RISCV_SPINLOCK_H */ diff --git a/arch/riscv/include/asm/spinlock_types.h b/arch/riscv/include/asm/spinlock_types.h index d033a973f287..afbb19841d0f 100644 --- a/arch/riscv/include/asm/spinlock_types.h +++ b/arch/riscv/include/asm/spinlock_types.h @@ -10,7 +10,24 @@ # error "please don't include this file directly" #endif +#ifdef CONFIG_RISCV_TICKET_LOCK +#define TICKET_NEXT 16 + +typedef struct { + union { + u32 lock; + struct __raw_tickets { + /* little endian */ + u16 owner; + u16 next; + } tickets; + }; +} arch_spinlock_t; + +#define __ARCH_SPIN_LOCK_UNLOCKED { { 0 } } +#else #include +#endif #include #endif /* _ASM_RISCV_SPINLOCK_TYPES_H */