From patchwork Sat Dec 16 03:26:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 13495385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB8FCC4332F for ; Sat, 16 Dec 2023 03:27:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 586D08D0146; Fri, 15 Dec 2023 22:27:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5121F8D0143; Fri, 15 Dec 2023 22:27:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 361888D0146; Fri, 15 Dec 2023 22:27:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 214DD8D0143 for ; Fri, 15 Dec 2023 22:27:24 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F39711201A8 for ; Sat, 16 Dec 2023 03:27:23 +0000 (UTC) X-FDA: 81571245966.06.93598D8 Received: from out-177.mta0.migadu.com (out-177.mta0.migadu.com [91.218.175.177]) by imf16.hostedemail.com (Postfix) with ESMTP id 40F0A180006 for ; Sat, 16 Dec 2023 03:27:22 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=aaiz38pz; spf=pass (imf16.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702697242; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ERZYSmZBhuvvvnoYNYpkA3Vl+n3Jximiv1YGZ7OitLQ=; b=m8VpDP5ODdmodg/sgxA5JPosSSwqA7JHL7p2dTp0AJjdj66EzNAaIQvaS/fTOMBRHWLjv5 yyDyBp/YEtc5za+UsWH7Z8fROcPgV8JU6OUuo5wjha+Gv0TuxFBy+jWnj16NyPw3VlsT+O svtB+qpv/6yHV2PZh98ocGoOylxUHU8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702697242; a=rsa-sha256; cv=none; b=BBCGjmPc+6TO595ogHL2ZNJS1Y1BaugmylFNtfknUhlSnZ2O0P7t5hn2Vvx+W7gRYPz+GD PrbEzG3uKspilzbgXJFz657PJyKgwxh2BsLraS+iiyNora3eGb/0ARrhhUumhxFzCQAW9S 7wtOk6s2wCadA6futo+XLll/Ig6cDSY= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=aaiz38pz; spf=pass (imf16.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.177 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1702697240; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ERZYSmZBhuvvvnoYNYpkA3Vl+n3Jximiv1YGZ7OitLQ=; b=aaiz38pzKiugC7Tlg40b87BRR7qNJ7SLJQXvUkpdJM+Lutdum8RxgQdiWqRBF8d9+kXBci 8dAv7dkfFinK2AYiUgdDki2xwVSVyOnDuDr6pLssATxbY1vRtr0MkZvomYziXMZRlSCGfx xau0WAmrv1ShMUk4AXdY6FF6ZxeArLk= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: Kent Overstreet , tglx@linutronix.de, x86@kernel.org, tj@kernel.org, peterz@infradead.org, mathieu.desnoyers@efficios.com, paulmck@kernel.org, keescook@chromium.org, dave.hansen@linux.intel.com, mingo@redhat.com, will@kernel.org, longman@redhat.com, boqun.feng@gmail.com, brauner@kernel.org Subject: [PATCH 21/50] locking/seqlock: Split out seqlock_types.h Date: Fri, 15 Dec 2023 22:26:20 -0500 Message-ID: <20231216032651.3553101-11-kent.overstreet@linux.dev> In-Reply-To: <20231216032651.3553101-1-kent.overstreet@linux.dev> References: <20231216024834.3510073-1-kent.overstreet@linux.dev> <20231216032651.3553101-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Stat-Signature: aquk3jcq4ruh8zmfofu4x7ff7q7bwtjg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 40F0A180006 X-Rspam-User: X-HE-Tag: 1702697242-484181 X-HE-Meta: U2FsdGVkX1/hSBVqx047NpgD0g5Zqgn2fa1YqB7Atonw7gq8vlqrMfqFoEQkdyKaP4XZWDyJyxNtzb/8uP1Q2YiEFDVy6Px7UoL8/uZYR5oZdC3eLUu6qHBpqKtiJasL1Hml0u0VcIPiawLzJnTRc2zkfvp+rXV3vSQdUAqjPuhVOgwDRVlG3Zf0t4u56e4f/f8Pru8VUPnbtVBRtPVbBz/Ri2GAtlmcenpExH6MDdZrmYXlu82exMnUBezSVsuCjr56p/oIe8L9NdUrG4a82F7Jezknb4eIyDOinUz6KwHlMhf6WU/CJyb57O8Rg4GSTeDFs8oEnoTkPZV8q9CpPZzPGLj7dlWpmpxR5xKgUSF9zd5mrhtq2Wp3uQLlOuvim6dkB3juQrdwrOEwiw7NGureMOIluGGQUhTisPrZxClZRtFRgvurdFShFu8+sIRgtoqrGIPPXVNEfmuw9mHW6gG//O6/HG4qxXDYnMAz3ZWl3iT7aQycdAnPeuyuvfqQQwxoGVGaht/QqQrSkiJbpcNGMVV8QERT/8Wl8MwBvkZfNmMfbZD9e16rC1X6UDjOfQH53q+agVEl+RKIUXOffmAev/fvQQAKHhXatEkK5xidRC3MzQe9rMFCOWUUSWMmLfoOgvEnxX5GSmtT5IKVXoxgUqjSBWMt3HhGrsOaVgXVOJd2Xb53w3HKKq3kxxs3kgtaNDvHkdLNzij2BGXgBFtPJJ66N5Y62fOAY1WqA9T2sT7LOD1TcY4othzVq3h+xgCa9dh8j0wr3UZvD6CJimZH1tAcaxc24SHwc5vwlLhHKpDE25MgEV+j+AWHsESVD9usMiTrOq4DhgpkQ275GjMtq0dNOdOb+YbDrJR9Iq1aRyP+sXAgNmd0LRHhRBeLu5cLqAq389FUAwKtEey79Xk1FwdthRhdeu8iUKXQwYfIgIG/AMaOYF8Op+NAP12e2KezWVq3JBAHAyjDIFE mNlqToHO ZSaOT24lpG62rJtCJtWDRAhR/YJNNoxgMtVWRjp/oP6LlsJubj16xXdT+3UTeyEfFUQq4YbBL+mV4v7TzT86jK6m6mINW4u0BYdyodWObs6Owqw82TTN+Uxeg7MgNTjtOB0UHNbZChhxEpimKoIUwyCzUq8m5ASF7hyK3LftFkTeVxDAuxAznq4cUrQJLQYBA+jJNBo7EfgdDqSSlOHNlCMaDC/rVr2v5yrRmwMtmTW+PnazwDD4/BfqsAiQ0p1q1dl6oiM42eorJjjSW1V9sz1VM3ff/IAuzqleKkgSaGRN3GHhz3XjnKWgJXQ7ereJoSgPSl4CPK04bRkhSn679pT8GTo9hofSbVipxdYkdMittdpRyQebgKNkgZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Trimming down sched.h dependencies: we don't want to include more than the base types. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Signed-off-by: Kent Overstreet Acked-by: Waiman Long --- include/linux/sched.h | 2 +- include/linux/seqlock.h | 79 +---------------------------- include/linux/seqlock_types.h | 93 +++++++++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 78 deletions(-) create mode 100644 include/linux/seqlock_types.h diff --git a/include/linux/sched.h b/include/linux/sched.h index 6d803d0904d9..436f7ce1450a 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -33,7 +33,7 @@ #include #include #include -#include +#include #include #include #include diff --git a/include/linux/seqlock.h b/include/linux/seqlock.h index e92f9d5577ba..d90d8ee29d81 100644 --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -18,6 +18,7 @@ #include #include #include +#include #include #include @@ -37,37 +38,6 @@ */ #define KCSAN_SEQLOCK_REGION_MAX 1000 -/* - * Sequence counters (seqcount_t) - * - * This is the raw counting mechanism, without any writer protection. - * - * Write side critical sections must be serialized and non-preemptible. - * - * If readers can be invoked from hardirq or softirq contexts, - * interrupts or bottom halves must also be respectively disabled before - * entering the write section. - * - * This mechanism can't be used if the protected data contains pointers, - * as the writer can invalidate a pointer that a reader is following. - * - * If the write serialization mechanism is one of the common kernel - * locking primitives, use a sequence counter with associated lock - * (seqcount_LOCKNAME_t) instead. - * - * If it's desired to automatically handle the sequence counter writer - * serialization and non-preemptibility requirements, use a sequential - * lock (seqlock_t) instead. - * - * See Documentation/locking/seqlock.rst - */ -typedef struct seqcount { - unsigned sequence; -#ifdef CONFIG_DEBUG_LOCK_ALLOC - struct lockdep_map dep_map; -#endif -} seqcount_t; - static inline void __seqcount_init(seqcount_t *s, const char *name, struct lock_class_key *key) { @@ -131,28 +101,6 @@ static inline void seqcount_lockdep_reader_access(const seqcount_t *s) * See Documentation/locking/seqlock.rst */ -/* - * For PREEMPT_RT, seqcount_LOCKNAME_t write side critical sections cannot - * disable preemption. It can lead to higher latencies, and the write side - * sections will not be able to acquire locks which become sleeping locks - * (e.g. spinlock_t). - * - * To remain preemptible while avoiding a possible livelock caused by the - * reader preempting the writer, use a different technique: let the reader - * detect if a seqcount_LOCKNAME_t writer is in progress. If that is the - * case, acquire then release the associated LOCKNAME writer serialization - * lock. This will allow any possibly-preempted writer to make progress - * until the end of its writer serialization lock critical section. - * - * This lock-unlock technique must be implemented for all of PREEMPT_RT - * sleeping locks. See Documentation/locking/locktypes.rst - */ -#if defined(CONFIG_LOCKDEP) || defined(CONFIG_PREEMPT_RT) -#define __SEQ_LOCK(expr) expr -#else -#define __SEQ_LOCK(expr) -#endif - /* * typedef seqcount_LOCKNAME_t - sequence counter with LOCKNAME associated * @seqcount: The real sequence counter @@ -194,11 +142,6 @@ static inline void seqcount_lockdep_reader_access(const seqcount_t *s) * @lockbase: prefix for associated lock/unlock */ #define SEQCOUNT_LOCKNAME(lockname, locktype, preemptible, lockbase) \ -typedef struct seqcount_##lockname { \ - seqcount_t seqcount; \ - __SEQ_LOCK(locktype *lock); \ -} seqcount_##lockname##_t; \ - \ static __always_inline seqcount_t * \ __seqprop_##lockname##_ptr(seqcount_##lockname##_t *s) \ { \ @@ -284,6 +227,7 @@ SEQCOUNT_LOCKNAME(raw_spinlock, raw_spinlock_t, false, raw_spin) SEQCOUNT_LOCKNAME(spinlock, spinlock_t, __SEQ_RT, spin) SEQCOUNT_LOCKNAME(rwlock, rwlock_t, __SEQ_RT, read) SEQCOUNT_LOCKNAME(mutex, struct mutex, true, mutex) +#undef SEQCOUNT_LOCKNAME /* * SEQCNT_LOCKNAME_ZERO - static initializer for seqcount_LOCKNAME_t @@ -794,25 +738,6 @@ static inline void raw_write_seqcount_latch(seqcount_latch_t *s) smp_wmb(); /* increment "sequence" before following stores */ } -/* - * Sequential locks (seqlock_t) - * - * Sequence counters with an embedded spinlock for writer serialization - * and non-preemptibility. - * - * For more info, see: - * - Comments on top of seqcount_t - * - Documentation/locking/seqlock.rst - */ -typedef struct { - /* - * Make sure that readers don't starve writers on PREEMPT_RT: use - * seqcount_spinlock_t instead of seqcount_t. Check __SEQ_LOCK(). - */ - seqcount_spinlock_t seqcount; - spinlock_t lock; -} seqlock_t; - #define __SEQLOCK_UNLOCKED(lockname) \ { \ .seqcount = SEQCNT_SPINLOCK_ZERO(lockname, &(lockname).lock), \ diff --git a/include/linux/seqlock_types.h b/include/linux/seqlock_types.h new file mode 100644 index 000000000000..dfdf43e3fa3d --- /dev/null +++ b/include/linux/seqlock_types.h @@ -0,0 +1,93 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __LINUX_SEQLOCK_TYPES_H +#define __LINUX_SEQLOCK_TYPES_H + +#include +#include +#include + +/* + * Sequence counters (seqcount_t) + * + * This is the raw counting mechanism, without any writer protection. + * + * Write side critical sections must be serialized and non-preemptible. + * + * If readers can be invoked from hardirq or softirq contexts, + * interrupts or bottom halves must also be respectively disabled before + * entering the write section. + * + * This mechanism can't be used if the protected data contains pointers, + * as the writer can invalidate a pointer that a reader is following. + * + * If the write serialization mechanism is one of the common kernel + * locking primitives, use a sequence counter with associated lock + * (seqcount_LOCKNAME_t) instead. + * + * If it's desired to automatically handle the sequence counter writer + * serialization and non-preemptibility requirements, use a sequential + * lock (seqlock_t) instead. + * + * See Documentation/locking/seqlock.rst + */ +typedef struct seqcount { + unsigned sequence; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +} seqcount_t; + +/* + * For PREEMPT_RT, seqcount_LOCKNAME_t write side critical sections cannot + * disable preemption. It can lead to higher latencies, and the write side + * sections will not be able to acquire locks which become sleeping locks + * (e.g. spinlock_t). + * + * To remain preemptible while avoiding a possible livelock caused by the + * reader preempting the writer, use a different technique: let the reader + * detect if a seqcount_LOCKNAME_t writer is in progress. If that is the + * case, acquire then release the associated LOCKNAME writer serialization + * lock. This will allow any possibly-preempted writer to make progress + * until the end of its writer serialization lock critical section. + * + * This lock-unlock technique must be implemented for all of PREEMPT_RT + * sleeping locks. See Documentation/locking/locktypes.rst + */ +#if defined(CONFIG_LOCKDEP) || defined(CONFIG_PREEMPT_RT) +#define __SEQ_LOCK(expr) expr +#else +#define __SEQ_LOCK(expr) +#endif + +#define SEQCOUNT_LOCKNAME(lockname, locktype, preemptible, lockbase) \ +typedef struct seqcount_##lockname { \ + seqcount_t seqcount; \ + __SEQ_LOCK(locktype *lock); \ +} seqcount_##lockname##_t; + +SEQCOUNT_LOCKNAME(raw_spinlock, raw_spinlock_t, false, raw_spin) +SEQCOUNT_LOCKNAME(spinlock, spinlock_t, __SEQ_RT, spin) +SEQCOUNT_LOCKNAME(rwlock, rwlock_t, __SEQ_RT, read) +SEQCOUNT_LOCKNAME(mutex, struct mutex, true, mutex) +#undef SEQCOUNT_LOCKNAME + +/* + * Sequential locks (seqlock_t) + * + * Sequence counters with an embedded spinlock for writer serialization + * and non-preemptibility. + * + * For more info, see: + * - Comments on top of seqcount_t + * - Documentation/locking/seqlock.rst + */ +typedef struct { + /* + * Make sure that readers don't starve writers on PREEMPT_RT: use + * seqcount_spinlock_t instead of seqcount_t. Check __SEQ_LOCK(). + */ + seqcount_spinlock_t seqcount; + spinlock_t lock; +} seqlock_t; + +#endif /* __LINUX_SEQLOCK_TYPES_H */