From patchwork Mon Aug 19 18:30:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoph Lameter via B4 Relay X-Patchwork-Id: 13768822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BEA2C3DA4A for ; Mon, 19 Aug 2024 18:30:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9B416B007B; Mon, 19 Aug 2024 14:30:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4C646B0082; Mon, 19 Aug 2024 14:30:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CEBE26B0083; Mon, 19 Aug 2024 14:30:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B10626B007B for ; Mon, 19 Aug 2024 14:30:32 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0182D161489 for ; Mon, 19 Aug 2024 18:30:31 +0000 (UTC) X-FDA: 82469835504.03.3BE8C52 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf18.hostedemail.com (Postfix) with ESMTP id 5899C1C001C for ; Mon, 19 Aug 2024 18:30:28 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CisaNtag; spf=pass (imf18.hostedemail.com: domain of devnull+cl.gentwo.org@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=devnull+cl.gentwo.org@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724092152; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=m5fESDmgUulHYhvam2KqbMt2sJ+bjwH383zyCt0W3Ms=; b=FVWaY8/+SJySpE1PEseu1fCTlXawgG5BaeO9gI25jNLXTm8LHIMYXRELpAiRH4zb3krUHV +eyUfJaVw9L78EzCY3b/Z8LAceD+5/csPKa7Muw0QqJ1hzRfyWVuaLnfsdZmMOEsDcrOdu E2qTLQm++7qPOauM8DCIM87WOLXNGCU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724092152; a=rsa-sha256; cv=none; b=M8hU0FMuriDyj13d9KfgGXKvkQ1qdFu3/Al/HvKgipI980haZI0oB+0IV3+05YKpbf0GGm tbHPXL4onJgDIuTiOwkQdAfvupLMxcHpEJ/vQvHq5Uq6wUx3Lr+V+J9ZSushhHNqwr0BLZ PEZefx1GJ54UvR68WwoAJeAl/ikxNUk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=CisaNtag; spf=pass (imf18.hostedemail.com: domain of devnull+cl.gentwo.org@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=devnull+cl.gentwo.org@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id B506CCE0B13; Mon, 19 Aug 2024 18:30:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPS id D269CC4AF0C; Mon, 19 Aug 2024 18:30:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1724092223; bh=tSCee26JGqJNW5fxQxtynl0HcRSR8D1JgiXpOYtmgQE=; h=From:Date:Subject:To:Cc:Reply-To:From; b=CisaNtagHd7kzmJV3mDWmcmMsgFTgfs+tmHl8P4hdEaoMPYBULdPAkWsHEZYWMEda Iakc6ZACdJudRxVuGArYNN4cP1ELKOjvPYuQpa3AQTl2Z/jp1APCMx0qGvV4yr+QNV 4hgKfLp8qglnaLCaoKW1PLTGGzKvQWg8Mlb9g49F6nQKNzrFfIXCgNATvK/0gVlatK dkrIfUS7tNHWCesy3AI2AzQy7a/Hyr2hfFnbNHe0hWOqkI7vBjg8cG6yHzQ5PkGfRY BM6i40sEcVBpG1JH+s4rC50exOg4TLN9m9RVHw0eIEqMtf20BBy5/3T6Tr8VlztScE 0+85NVFrK3c1w== Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD985C52D7C; Mon, 19 Aug 2024 18:30:23 +0000 (UTC) From: Christoph Lameter via B4 Relay Date: Mon, 19 Aug 2024 11:30:15 -0700 Subject: [PATCH v2] Avoid memory barrier in read_seqcount() through load acquire MIME-Version: 1.0 Message-Id: <20240819-seq_optimize-v2-1-9d0da82b022f@gentwo.org> X-B4-Tracking: v=1; b=H4sIADaPw2YC/6tWKk4tykwtVrJSqFYqSi3LLM7MzwNyjHQUlJIzE vPSU3UzU4B8JSMDIxMDC0Nj3eLUwvj8gpLM3MyqVF0zi2QTCzNLs2RzSwsloJaCotS0zAqwcdG xtbUA0UUVFF4AAAA= To: Catalin Marinas , Will Deacon , Peter Zijlstra , Ingo Molnar , Waiman Long , Boqun Feng Cc: Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-arch@vger.kernel.org, "Christoph Lameter (Ampere)" X-Mailer: b4 0.15-dev-37811 X-Developer-Signature: v=1; a=ed25519-sha256; t=1724092223; l=5831; i=cl@gentwo.org; s=20240811; h=from:subject:message-id; bh=33u86Z1NtTj5I1hzjsT/8J5bCWwIeort9/FFXzrHRO8=; b=xjoqKpuJtKmVY/p7XqcrNDj6VvwsP/litDEErvD8DiwRs7UAnTdP/zYFWneATI5cWwAHPBQAe ONJwSIhLrBvD2MOb674/6mx5xNjsJr7TwjTFsCE/PjBfuy4IL4vKzRA X-Developer-Key: i=cl@gentwo.org; a=ed25519; pk=I7gqGwDi9drzCReFIuf2k9de1FI1BGibsshXI0DIvq8= X-Endpoint-Received: by B4 Relay for cl@gentwo.org/20240811 with auth_id=194 X-Original-From: "Christoph Lameter (Ampere)" Reply-To: cl@gentwo.org X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5899C1C001C X-Stat-Signature: hnnmy4y337athfy8if5t8n4d4c949ci7 X-HE-Tag: 1724092228-614973 X-HE-Meta: U2FsdGVkX19ZhwIne0sw+tZ2hKuSGCTGbsLFy8dqtthhV7T4scd4PhQSiXfJ8TX+s8qfV3cwcXwR0hYNWveV2V7B/hGbOsU5SELX8vGaR70rOsieHSsz3JhlhUM9EqRO8DdyueTiaYXCpFu95Ctf/o9HhwC/pUxYy592wpHwh4qn4nVofNLllJHt9+WYIbmrllJqoRQeo1yWymZg+1BrjMOJpc8C2LWFf8fa+A6wKBi1uWxTt+pabZ2TaKemk7CasD/H6R91XytCFO56ZKGR/sIVfKyWorXoFuOeps1/43dH9rzZAlsGgsNY0qw+3H+ygsVR5OlyZgs/l8n+ydlzCpG5DI0iXVhTmpyaX4RqEtV9hILQzgwneByMnsMecPT5UCgcSDNO9QaftRT48iJmAYQnmwJdAM+EvVWp/Ckwu3FZF6Zkm/D456d9tDUqURxvolB804OMph/OvhMTY0tSFf07mb/TBdQ7nAYUW9E5hf9cxTA31tLPigpr/f+SmAk2kLxhoeTSRQU9dBzE+1xchmMI2pgeOeDafXIOADp2MfMF1fuXj/9ppiV8xYzSBrCQ+UKBWpCAmU2EC0abhqq0d5X9aW1gA65PtJm7k86wOIF6E16zrkI/0kFrV6pzumxyYVE3dOOq6oVHUjczxurvgiZ4CwPegT0+dBH44NKQFBL+y4cJOYM0WXUp7x1ncHG2taG5+3KZaeylTn1TMkO1w4PHpc6McZG9ptj7xXz+V2O6jQMuo7OV+iFLEA1OUGzVdvbScElgUCBvyaljBvGxMiHiMZcGNuZcqda8JKBTEM+4SG4WIgOWIsViK2FMA7a/aGpjO9zQMppYQfuBnimEW+vStLtHsYNwk54UTYu1+RsNihyMOmkZ6hXPrVT+OesKNEJT1DdokM8Hl7jTWTlFADDXBbc8h5caI6d0J7QOB13g7DoGjQ5HokMtLA/WYKCL9NKmy8jOiprMBlI1Jzh f9IiOhJQ gFNDvKTetPQiADa5G3ZgOtuuATnvlZ/dyZNd67By2uoc7WPV/1LuW1T58+FDDTa3G18Ijtul8I1DwQ6aoaI30NlQFXIuVIRUce7CJQY5rkA4Z0AsabhiVokS+FFP1ajVbF8AxupEEm7TRQzMSIm324nK9zVN0bZOHnH0idJyE6LCFkw62Rqrj7aa0uAAH2BUntNp1CUbNdwDaZdDsbJxoC4Zozp3O+p9yQ4Jr0Ly/QtlTmnaA6pwh8cAVRBqif0EoCWo7Hi4psHPz/i6U9Ageah5D7ft+lSau28s+3yTrZzHLd5XmfNHUmMHdyrrzUoA+rg8dyS3sYkuxR3T3e5p8/SPiwC7Y0t68ic53hok41ZxBzq2b9n6tkOclD8OPBfmhuPa3U5JDJhyNUetjmD4ci37uQ/ZWtjW7/l/FPFQhA+VLin1GdYwu3q1A4/BE8DblzW69qW7Fbn4DCMDUmaypbKhkYWV6D3hEEHMqy5IzmreYDJs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: "Christoph Lameter (Ampere)" Some architectures support load acquire which can save us a memory barrier and save some cycles. A typical sequence do { seq = read_seqcount_begin(&s); } while (read_seqcount_retry(&s, seq); requires 13 cycles on ARM64 for an empty loop. Two read memory barriers are needed. One for each of the seqcount_* functions. We can replace the first read barrier with a load acquire of the seqcount which saves us one barrier. On ARM64 doing so reduces the cycle count from 13 to 8. This is a general improvement for the ARM64 architecture and not specific to a certain processor. The cycle count here was obtained on a Neoverse N1 (Ampere Altra). The ARM documentation states that load acquire is more effective than a load plus barrier. In general that tends to be true on all compute platforms that support both. See (as quoted by Linus Torvalds): https://developer.arm.com/documentation/102336/0100/Load-Acquire-and-Store-Release-instructions "Weaker ordering requirements that are imposed by Load-Acquire and Store-Release instructions allow for micro-architectural optimizations, which could reduce some of the performance impacts that are otherwise imposed by an explicit memory barrier. If the ordering requirement is satisfied using either a Load-Acquire or Store-Release, then it would be preferable to use these instructions instead of a DMB" Signed-off-by: Christoph Lameter (Ampere) Acked-by: Will Deacon --- V1->V2 - Describe the benefit of load acquire vs barriers - Explain the CONFIG_ARCH_HAS_ACQUIRE_RELEASE option better --- arch/Kconfig | 8 ++++++++ arch/arm64/Kconfig | 1 + include/linux/seqlock.h | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+) --- base-commit: b0da640826ba3b6506b4996a6b23a429235e6923 change-id: 20240813-seq_optimize-68c48696c798 Best regards, diff --git a/arch/Kconfig b/arch/Kconfig index 975dd22a2dbd..3c270f496231 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1600,6 +1600,14 @@ config ARCH_HAS_KERNEL_FPU_SUPPORT Architectures that select this option can run floating-point code in the kernel, as described in Documentation/core-api/floating-point.rst. +config ARCH_HAS_ACQUIRE_RELEASE + bool + help + Setting ARCH_HAS_ACQUIRE_RELEASE indicates that the architecture + supports load acquire and release. Typically these are more effective + than memory barriers. Code will prefer the use of load acquire and + store release over memory barriers if this option is enabled. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig" diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a2f8ff354ca6..19e34fff145f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -39,6 +39,7 @@ config ARM64 select ARCH_HAS_PTE_DEVMAP select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_HW_PTE_YOUNG + select ARCH_HAS_ACQUIRE_RELEASE select ARCH_HAS_SETUP_DMA_OPS select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_SET_MEMORY diff --git a/include/linux/seqlock.h b/include/linux/seqlock.h index d90d8ee29d81..353fcf32b800 100644 --- a/include/linux/seqlock.h +++ b/include/linux/seqlock.h @@ -176,6 +176,28 @@ __seqprop_##lockname##_sequence(const seqcount_##lockname##_t *s) \ return seq; \ } \ \ +static __always_inline unsigned \ +__seqprop_##lockname##_sequence_acquire(const seqcount_##lockname##_t *s) \ +{ \ + unsigned seq = smp_load_acquire(&s->seqcount.sequence); \ + \ + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) \ + return seq; \ + \ + if (preemptible && unlikely(seq & 1)) { \ + __SEQ_LOCK(lockbase##_lock(s->lock)); \ + __SEQ_LOCK(lockbase##_unlock(s->lock)); \ + \ + /* \ + * Re-read the sequence counter since the (possibly \ + * preempted) writer made progress. \ + */ \ + seq = smp_load_acquire(&s->seqcount.sequence); \ + } \ + \ + return seq; \ +} \ + \ static __always_inline bool \ __seqprop_##lockname##_preemptible(const seqcount_##lockname##_t *s) \ { \ @@ -211,6 +233,11 @@ static inline unsigned __seqprop_sequence(const seqcount_t *s) return READ_ONCE(s->sequence); } +static inline unsigned __seqprop_sequence_acquire(const seqcount_t *s) +{ + return smp_load_acquire(&s->sequence); +} + static inline bool __seqprop_preemptible(const seqcount_t *s) { return false; @@ -259,6 +286,7 @@ SEQCOUNT_LOCKNAME(mutex, struct mutex, true, mutex) #define seqprop_ptr(s) __seqprop(s, ptr)(s) #define seqprop_const_ptr(s) __seqprop(s, const_ptr)(s) #define seqprop_sequence(s) __seqprop(s, sequence)(s) +#define seqprop_sequence_acquire(s) __seqprop(s, sequence_acquire)(s) #define seqprop_preemptible(s) __seqprop(s, preemptible)(s) #define seqprop_assert(s) __seqprop(s, assert)(s) @@ -293,6 +321,18 @@ SEQCOUNT_LOCKNAME(mutex, struct mutex, true, mutex) * * Return: count to be passed to read_seqcount_retry() */ +#ifdef CONFIG_ARCH_HAS_ACQUIRE_RELEASE +#define raw_read_seqcount_begin(s) \ +({ \ + unsigned _seq; \ + \ + while ((_seq = seqprop_sequence_acquire(s)) & 1) \ + cpu_relax(); \ + \ + kcsan_atomic_next(KCSAN_SEQLOCK_REGION_MAX); \ + _seq; \ +}) +#else #define raw_read_seqcount_begin(s) \ ({ \ unsigned _seq = __read_seqcount_begin(s); \ @@ -300,6 +340,7 @@ SEQCOUNT_LOCKNAME(mutex, struct mutex, true, mutex) smp_rmb(); \ _seq; \ }) +#endif /** * read_seqcount_begin() - begin a seqcount_t read critical section