From patchwork Wed Dec 18 03:07:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 13912976 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C0DEE77183 for ; Wed, 18 Dec 2024 03:07:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 197B36B008C; Tue, 17 Dec 2024 22:07:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1484C6B0092; Tue, 17 Dec 2024 22:07:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29FF6B0093; Tue, 17 Dec 2024 22:07:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D48E96B008C for ; Tue, 17 Dec 2024 22:07:42 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 92882120AC2 for ; Wed, 18 Dec 2024 03:07:42 +0000 (UTC) X-FDA: 82906594050.05.195CD33 Received: from mail-oo1-f41.google.com (mail-oo1-f41.google.com [209.85.161.41]) by imf06.hostedemail.com (Postfix) with ESMTP id B476118000E for ; Wed, 18 Dec 2024 03:07:18 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PRbomrfq; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734491239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fcHIP1+8Pwyc2xkUuGVXiolOQkisEg3F+R5j1Yt+uzY=; b=PhuRiMMCGaYIhnXmRsWj6/NPmO8CEqc0Ha24Mm16eBUuOwv1VqN883dmy8ThOIVLBHPWXD qsU18MC8Pv4+e5bGel4qS9vsBK5pDvHB55gct10UTdGcloyNLfO91DihNeS7QF1nyxle3F 0E4xR5fAWzKWyW9o+0mKP16/BENUk+k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734491239; a=rsa-sha256; cv=none; b=VT+PsvHZFwM5oGOppbAaJ30a6LJcWfv+B4RTYVZRUgsTxsx0AeEov+jmAmTvE/JwH/QwJz sr8KJ/LKNfaqRed46Ho1+LzHGAzU9Nfti0H8UA/DkC+lgjYVftWxLF/IyjW+2U8wL19Lzt G/16C9sGsZ7tL6dW020l0iWTPGJYupo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=PRbomrfq; spf=pass (imf06.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.161.41 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oo1-f41.google.com with SMTP id 006d021491bc7-5f4c111991bso185328eaf.0 for ; Tue, 17 Dec 2024 19:07:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734491260; x=1735096060; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fcHIP1+8Pwyc2xkUuGVXiolOQkisEg3F+R5j1Yt+uzY=; b=PRbomrfqMw9wkMkgVgNx9CIIXX7m2D+Pf6dcJsTIjVMhreYIM19UbrZXixyn1assfb am238bacuqDGJS7uTQzJ6qxmUttk55FZKIMEHWUM8bNH1DkZYXhVTLIBzsathHsOOfwF P4FPTtGxeO+4sHQGx2JfOxR+2CkgO7BFY8af2qZAdmvG1sopSVLn0wXPCrRcPtMLayXx GwsrAl8sCAPI+lNVuuEq7C33KaBFVzk9FCgpnydjfBHvLe2/SzbZriTyOnf2sEt9msaR z88zQbnhzGR4OPYO8RYpCfgwcgUgp8Gee4fDacTBom+V95W2yFJr769Mr6H7sBT9wxC5 rXaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734491260; x=1735096060; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fcHIP1+8Pwyc2xkUuGVXiolOQkisEg3F+R5j1Yt+uzY=; b=J3819hoLrpWqqId3yu8NYQHHhR6baAKBOfhw1gsNYnOZIaD+O20LH734+pkjF2A7wg bgFpm+WoP7rNsYlw7hjFQ3+wm4LDOqVnoRfsAGL3vDFMU35ogSxYZ6CxE2aIR3UICuJh wz8Fc0Ea+KlEKSJYmPz2Q76mX95MD2ctsLEzeVDGHIeBtj9PPMRN2p2lroLxKO2KzSzt Mfo8NiKGvsbaCl/4HeQzT/APE8vkdPM1xWMv9I/BSePvldIay0/USY/o4d/POzPMhE3J Jjk41EohdjN+4UqGa4Bp55MJPUEv3Oo0iYHCnYdLk2KEoNSLxO8rmjppllFGuVqi/C4B lUJA== X-Forwarded-Encrypted: i=1; AJvYcCUUkAWwUDmDx30U5DNIfy5J7FiL0gGKUxMgoEGcFofhTIFLyhqJMZwaoId5S+YEgHwJPnrXSj37lQ==@kvack.org X-Gm-Message-State: AOJu0YzxAel234V9CnQUjwCE6GKmhhtlD4X7GPt4GmzmDeAh4IeyKeJl GuvB2Io9+RY2DxatE1XDjh6CTz1VWTv3RtdegBZ79kI5bjnIA0/D X-Gm-Gg: ASbGncuIQkjzno7xcceoWGXLenKvGJzjkfoUnyx8b6ep46Mx7EIY/p/Yv3qgIToH6oQ AHnXR7KcLHJpBeN+qvmY/KELAlWr+3oloP3SWoj25CoYKJth4wS3SuWh7N9eIk8EJiuwRx3Bq6s 3u66fuhWNrxOKuNuXfB78hAbY8YkX7I+NL84s4dp+5mkNUxcAGKmLjUQ7Kg/kZ/N3gFC86BFbqE nEC7QQ5AnGYV1QhsKcBK+sGf8ESC9bIxXAau7juik9T5PgPUF3q0B7RTsleEQ== X-Google-Smtp-Source: AGHT+IGAOWU8WTtpajWGghl6QstJosNCQ5UcWaUN0/NUcMXEsAsN5z1Ap2Hz23QuLwZvMvbxMXfsRw== X-Received: by 2002:a4a:d74c:0:b0:5f2:b6d5:bf53 with SMTP id 006d021491bc7-5f45c71a68cmr2978615eaf.1.1734491259593; Tue, 17 Dec 2024 19:07:39 -0800 (PST) Received: from localhost ([2a03:2880:12ff:4::]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5f33a98a3f7sm2526797eaf.36.2024.12.17.19.07.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Dec 2024 19:07:38 -0800 (PST) From: alexei.starovoitov@gmail.com To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v3 3/6] locking/local_lock: Introduce local_trylock_irqsave() Date: Tue, 17 Dec 2024 19:07:16 -0800 Message-ID: <20241218030720.1602449-4-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> References: <20241218030720.1602449-1-alexei.starovoitov@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B476118000E X-Stat-Signature: b3gynxygkj5gcdxsaicf97c7j9mah3ck X-Rspam-User: X-HE-Tag: 1734491238-569817 X-HE-Meta: U2FsdGVkX1/0l4WTHdoLBTdZBmbtJ0EWOgDzsFt4OEFCds3+x5+MpLbuTSIpXQqioR4LogZuKUvt+eKepARPTzOKUsCgtpHdVfjaFeeQOFWshxU+OUopFS3amL2bka+6oZntthBFAxKCxuAX2v6q5RNV+io/bTL8flzrzpXhWcZgvhsH2skcrN4ZhbQzffeH8I1JKCngtj1Vr4gnW71S1n4ClXZdMnY3X8iApwGEjK8UgEQfYrQAoK9+BkTOTggMlXG1Dbi/kMIBVQAU7VJiARh3ch1kxyHlMEK8ZKkmYqJUVyflyuBb0CQgPllr3TLmBN4Azd7gdWmpy9VE22wm0yywAbIAP7XYHxb6lnqwwlku3YkLy/O8nhmO2VIa8aNk2MglqpMQulOLgh2fEDapdDTbbu8agACzcQ9eZbybqHF9SbJueEso5a7XHjck+4ETHItmeFVAyf47cQAqIURQdcaGo8LG+ygoDABG+nLl953FvtYOBRSXd9LL8q2txSqX9HY32QUGyh80P11lrwyLUC5qjHGqgV6AeAMfYoyGpvV4IGbH7biFnwWB2Q/sZ2rkhz4/4LISwQ0j89hdj+vUmgUfyDo9ZAwJPIx25Zvc634DojM8CRTzUIxGKJ63jtZ5+D8QSzWluX3B7VvOdDAHRt4WvY02lMeebyvc1QxCm2oZj6pD5oL/v/VnMrSNuJzd1ZZxNY+P76mAsxNphHVUMq6lkkXt233HxyZwXfEaPpFXdNnk+EZg1XK0RkO9SPU1k+daCwAfYTLX4GkqX1Raa0yIRfiq1ky29jGJ54WFtHyokxqbTCkaB8i97Dip2j6gM22i/QiqXGVfXxTq4QIzt9h8SAbB9lDAJ6xUIviQ3IIGQkkx4P1hs5URAwMvjQqJql0ALgG7KCg3Jxi6SWJzMrKkikat9WdEjkFIDY7NswJMyOwkUokSjlt3QiVU5HZHTM8INHMVNwlZdKv3Kay AMUVjnOK zCa5yNnEVqI+8CxsEtEYkhT6n29W4ZPEgB6sjuaQF+d4t51RR94jq3kHwA94agOfzgACNpd8+kdvS8vjuMyte10lbtMCym5v+xi+rCu3aGSt/xAo7C76dyVn8jBwj3NDhNgfGueon+FiTc6IDhjM6K0nSg+dCWuwAhM3KnjclKJJ7zGZ8ButEZta8FRiWOZxgMSPq0wmYd0pE0gNLc+ljJDYofaArrDAZF2/OoU1Aor34epwHB92Cnwf3Pkd0Z38wJWADJ0FvZ0DTgKFNofZyIDzhq8t8xqSB/Tuh9hbmxgjkoYBlaJrcJPjvpsA3kCTMvRfOR62ioKIGRE/Svmj9Jsoo/U49jihg/qKVJ1cceaYzEc/iAwbuUURT0QElESD+j59niJoXsA54W03YARD1aPzDEtj5tWyqYy8uO7lBaCSNPNUCdHp0f2slyudcCFUWOfWtAdNbB/WeR0qIEbaZqFecEg0ew8se8NIjYtN292ze3EyNNdI+JR7OKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov This is inspired by 'struct local_tryirq_lock' in: https://lore.kernel.org/all/20241112-slub-percpu-caches-v1-5-ddc0bdc27e05@suse.cz/ Similar to local_lock_irqsave() introduce local_trylock_irqsave(). It uses spin_trylock in PREEMPT_RT when not in hard IRQ and not in NMI and instantly fails otherwise. In !RT it uses simple active flag that prevents IRQs or NMIs reentering locked region. Note there is no need to use local_inc for active flag. If IRQ handler grabs the same local_lock after READ_ONCE(lock->active) already completed it has to unlock it before returning. Similar with NMI handler. So there is a strict nesting of scopes. It's a per cpu lock, so multiple cpus do not access it in parallel. Signed-off-by: Alexei Starovoitov --- include/linux/local_lock.h | 9 ++++ include/linux/local_lock_internal.h | 76 ++++++++++++++++++++++++++--- 2 files changed, 78 insertions(+), 7 deletions(-) diff --git a/include/linux/local_lock.h b/include/linux/local_lock.h index 091dc0b6bdfb..84ee560c4f51 100644 --- a/include/linux/local_lock.h +++ b/include/linux/local_lock.h @@ -30,6 +30,15 @@ #define local_lock_irqsave(lock, flags) \ __local_lock_irqsave(lock, flags) +/** + * local_trylock_irqsave - Try to acquire a per CPU local lock, save and disable + * interrupts. Always fails in RT when in_hardirq or NMI. + * @lock: The lock variable + * @flags: Storage for interrupt flags + */ +#define local_trylock_irqsave(lock, flags) \ + __local_trylock_irqsave(lock, flags) + /** * local_unlock - Release a per CPU local lock * @lock: The lock variable diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h index 8dd71fbbb6d2..93672127c73d 100644 --- a/include/linux/local_lock_internal.h +++ b/include/linux/local_lock_internal.h @@ -9,6 +9,7 @@ #ifndef CONFIG_PREEMPT_RT typedef struct { + int active; #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; struct task_struct *owner; @@ -22,7 +23,7 @@ typedef struct { .wait_type_inner = LD_WAIT_CONFIG, \ .lock_type = LD_LOCK_PERCPU, \ }, \ - .owner = NULL, + .owner = NULL, .active = 0 static inline void local_lock_acquire(local_lock_t *l) { @@ -31,6 +32,13 @@ static inline void local_lock_acquire(local_lock_t *l) l->owner = current; } +static inline void local_trylock_acquire(local_lock_t *l) +{ + lock_map_acquire_try(&l->dep_map); + DEBUG_LOCKS_WARN_ON(l->owner); + l->owner = current; +} + static inline void local_lock_release(local_lock_t *l) { DEBUG_LOCKS_WARN_ON(l->owner != current); @@ -45,6 +53,7 @@ static inline void local_lock_debug_init(local_lock_t *l) #else /* CONFIG_DEBUG_LOCK_ALLOC */ # define LOCAL_LOCK_DEBUG_INIT(lockname) static inline void local_lock_acquire(local_lock_t *l) { } +static inline void local_trylock_acquire(local_lock_t *l) { } static inline void local_lock_release(local_lock_t *l) { } static inline void local_lock_debug_init(local_lock_t *l) { } #endif /* !CONFIG_DEBUG_LOCK_ALLOC */ @@ -60,6 +69,7 @@ do { \ 0, LD_WAIT_CONFIG, LD_WAIT_INV, \ LD_LOCK_PERCPU); \ local_lock_debug_init(lock); \ + (lock)->active = 0; \ } while (0) #define __spinlock_nested_bh_init(lock) \ @@ -75,37 +85,73 @@ do { \ #define __local_lock(lock) \ do { \ + local_lock_t *l; \ preempt_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irq(lock) \ do { \ + local_lock_t *l; \ local_irq_disable(); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) #define __local_lock_irqsave(lock, flags) \ do { \ + local_lock_t *l; \ local_irq_save(flags); \ - local_lock_acquire(this_cpu_ptr(lock)); \ + l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 0); \ + WRITE_ONCE(l->active, 1); \ + local_lock_acquire(l); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + local_lock_t *l; \ + local_irq_save(flags); \ + l = this_cpu_ptr(lock); \ + if (READ_ONCE(l->active) == 1) { \ + local_irq_restore(flags); \ + l = NULL; \ + } else { \ + WRITE_ONCE(l->active, 1); \ + local_trylock_acquire(l); \ + } \ + !!l; \ + }) + #define __local_unlock(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ preempt_enable(); \ } while (0) #define __local_unlock_irq(lock) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_enable(); \ } while (0) #define __local_unlock_irqrestore(lock, flags) \ do { \ - local_lock_release(this_cpu_ptr(lock)); \ + local_lock_t *l = this_cpu_ptr(lock); \ + lockdep_assert(l->active == 1); \ + WRITE_ONCE(l->active, 0); \ + local_lock_release(l); \ local_irq_restore(flags); \ } while (0) @@ -148,6 +194,22 @@ typedef spinlock_t local_lock_t; __local_lock(lock); \ } while (0) +#define __local_trylock_irqsave(lock, flags) \ + ({ \ + __label__ out; \ + int ret = 0; \ + typecheck(unsigned long, flags); \ + flags = 0; \ + if (in_nmi() || in_hardirq()) \ + goto out; \ + migrate_disable(); \ + ret = spin_trylock(this_cpu_ptr((lock))); \ + if (!ret) \ + migrate_enable(); \ + out: \ + ret; \ + }) + #define __local_unlock(__lock) \ do { \ spin_unlock(this_cpu_ptr((__lock))); \