From patchwork Thu Feb 2 14:50:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B87DC05027 for ; Thu, 2 Feb 2023 15:29:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FD6A6B0081; Thu, 2 Feb 2023 10:29:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 45CB26B0083; Thu, 2 Feb 2023 10:29:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C55E6B0087; Thu, 2 Feb 2023 10:29:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 08BAA6B0081 for ; Thu, 2 Feb 2023 10:29:25 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BEF24120F61 for ; Thu, 2 Feb 2023 15:29:24 +0000 (UTC) X-FDA: 80422735848.03.815E82C Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf13.hostedemail.com (Postfix) with ESMTP id C3F1C2000C for ; Thu, 2 Feb 2023 15:29:21 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=Ql+cyqR5; spf=none (imf13.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=sduTuNa9Y93QC8FtEhMkgsBeoFYULf3DBAkiVbSNSHc=; b=1vEAWlb78a+MA1d0FvhbjPqE0ws9Ko1qxx8btgnQ1h6SqaX07fiwlARrXUEjvrJ8nUEtdM FCPCHoDGwSHCMpJsrmQOs8gyWJ/DiJJpoBteDNpamg2dwSNCN+okWek2AamNOPZSmNzHrt iv/CUZfNPsJsR8h/QkjHMNoIvSY6T9g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=Ql+cyqR5; spf=none (imf13.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351762; a=rsa-sha256; cv=none; b=0S3Ng/dwmUBq55rP/RtrrJFk6JBDueyIoovZ699D2gzsFdearPL/CueB83j0hF5zwaaptP 62y2xpMto3qcRjF+UjyjYgn/Oy86X4V63xCUMU/vBiwAUhUxgQ+bXczuCmGqg6Sn6zoCaa OmADUYJjg/XzqtUgIvubO6RiCjORXE8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=sduTuNa9Y93QC8FtEhMkgsBeoFYULf3DBAkiVbSNSHc=; b=Ql+cyqR51WKsltXZ5zb2xdtiWO l8ltfVfe3HF1kXMg3+syulZDERIZ071AJ59oVxg/E/ywg9uykQ07KTmbyZgpZJSaK+luBotas/BE/ E4ONu9TlXb4PR6kfN72XUPocrOt5KRQl0z2UATFAW0O8iNzGguxzS8vTSgYk7mGYI2B5E2xD1mtIE j+dU1aDeLic8DmDJ/tRtKYdpyLH0biZQtc4ZRnML9WbKIjz62UAnOVFdTa7gJkxanOrIYbbWmjIXL o+OWTB8xyNWH5gXiU/UIxEMoI7nqiJ39nJ+ejT/m+wVG6ouhh9fedoBJ+By2n6WOZoHChcjblobbU hi2nLMSg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVT-005CFf-1o; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00F29300446; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8ADCD23F31FB4; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.373335780@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:33 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 03/10] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C3F1C2000C X-Rspam-User: X-Stat-Signature: ygbe9byjcfi1y7iez54prkkp4rjdcjf4 X-HE-Tag: 1675351761-464499 X-HE-Meta: U2FsdGVkX18aXEmERRENxaDirMtr00Bn6N04jZaJ5YKMkH4fCl3qOx3UElVgBEKiPbqeq5o1GxBdhOI/UA1YfzXgl+VNOokwDU9W7f4aDM7luLuy5Cax/g8s5q1KLA4Y8ERKvk5muLBdQAmxIt2LlqYoD+fB0HEjFOUrpjv2FepMXjiqTaxvX3YFIyApSgb9A0fZUxVDCGJ6Fw9DogmEDB6zPfUbRZTqwe7n41xHigYwo08bYcy4rWyL/oujpyAQX1ULte+4PluxS3WMFJ8rQif697810t4764KyUe16C5QJtJpqrIpWXwB4ut/6qijaGvdBec3NTVvNhzpdi70E9fsQLCx3Ow5SiVxQZ5rVl6tbF9bf9ljBAIHthPV6Lco9NGWqsulp1oJ+ElU43KyFj/7OWj769xzG8JNtxxs4uLtPkdcXak8FG5LVdA/4v1a+TRiEG43b+bFUOtKUPzYLacZqOEODywSVT/MMuk1IIxPrq8oujYA7FGRC8jOWkRI3ND+EA1atcaJqOp63CJJja8b0gYyU2JoV6MqH/RrYIjcP3DL57RaW250UliNSH2qmuCIej/FMfTMZmkSns9RfJmiOk43ezl7p5PWrBiB7AG/6SlwkxdQ/TIADmtcnVIFyWenb5vr13r10aCEXYFTaXPfX6CktfhIxVcRs7YzjkXf6qIMzoCsz6sAzrrj4LDxn3UwQ6K8I8YKazzFM1lrFeRE9uqU24C8c+ll5uuymKH8XjjtukJBM7bbYd1OEdAOXf/kkxFiV6VamjNOafSYufsXr2i9YzTBVxV4JdoDFQSPuN+sPbLRReg8nZsNlSgkrXeKPUl6aqWYqvuST3J2sGY/Y7/wj/84KAx/+EsO/G9DJxUSo9sUeay3skHcsCz47j2IAA26wZznhX+7GtyAqMU2WSZRGw5Cy5PTFuL0zR9IKJP79fegVCG7CiGe/RwEBzjPXcc8Xbx1Q9TKmFEU x93VAmY+ XBiv5SEEpBbah+vAvIRyKj7tJe57Cnk4zlO8DyzlThXNSOIvIg1vg41A1M6JmbcZvQARrBrqE2cl33phpVgTLNas80n7nAuxj+olIAOt52sbFfKFLu+lMWWA05rMygZ5YH8KsaoHrToyXNHFOOGfTr3Py9MreqjlMVc/+/tL8YzZnZZmBa3Ug33MOn9AFJZDC1fDloWU+qnPUlRRw94N+vH8i0KXqVubV00bOwuh/Ejqs3hOujrHN6NNz/b8bxguiouscd53j4cbPvE0+KEdiGicdF1AkiHZBa14cKioDduCnS6NEJsWSDVgk6U9Z1o9CDHF4D7Sra5iQqiPimstVkfjY2WAu9GDX/EpizYPzAAW9MJ4zrrgX9f3w9hhJNcM7mGH3797sFRJ1GZYvDL5MzF9C90SHc+YHdDml X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For all architectures that currently support cmpxchg_double() implement the cmpxchg128() family of functions that is basically the same but with a saner interface. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens Acked-by: Mark Rutland --- arch/arm64/include/asm/atomic_ll_sc.h | 41 +++++++++++++++++++++++++ arch/arm64/include/asm/atomic_lse.h | 31 +++++++++++++++++++ arch/arm64/include/asm/cmpxchg.h | 26 ++++++++++++++++ arch/s390/include/asm/cmpxchg.h | 14 ++++++++ arch/x86/include/asm/cmpxchg_32.h | 3 + arch/x86/include/asm/cmpxchg_64.h | 55 +++++++++++++++++++++++++++++++++- 6 files changed, 168 insertions(+), 2 deletions(-) --- a/arch/arm64/include/asm/atomic_ll_sc.h +++ b/arch/arm64/include/asm/atomic_ll_sc.h @@ -326,6 +326,47 @@ __CMPXCHG_DBL( , , , ) __CMPXCHG_DBL(_mb, dmb ish, l, "memory") #undef __CMPXCHG_DBL + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __CMPXCHG128(name, mb, rel, cl...) \ +static __always_inline u128 \ +__ll_sc__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ +{ \ + union __u128_halves r, o = { .full = (old) }, \ + n = { .full = (new) }; \ + unsigned int tmp; \ + \ + asm volatile("// __cmpxchg128" #name "\n" \ + " prfm pstl1strm, %[v]\n" \ + "1: ldxp %[rl], %[rh], %[v]\n" \ + " cmp %[rl], %[ol]\n" \ + " ccmp %[rh], %[oh], 0, eq\n" \ + " b.ne 2f\n" \ + " st" #rel "xp %w[tmp], %[nl], %[nh], %[v]\n" \ + " cbnz %w[tmp], 1b\n" \ + " " #mb "\n" \ + "2:" \ + : [v] "+Q" (*(u128 *)ptr), \ + [rl] "=&r" (r.low), [rh] "=&r" (r.high), \ + [tmp] "=&r" (tmp) \ + : [ol] "r" (o.low), [oh] "r" (o.high), \ + [nl] "r" (n.low), [nh] "r" (n.high) \ + : "cc", ##cl); \ + \ + return r.full; \ +} + +__CMPXCHG128( , , ) +__CMPXCHG128(_mb, dmb ish, l, "memory") + +#undef __CMPXCHG128 + #undef K #endif /* __ASM_ATOMIC_LL_SC_H */ --- a/arch/arm64/include/asm/atomic_lse.h +++ b/arch/arm64/include/asm/atomic_lse.h @@ -324,4 +324,35 @@ __CMPXCHG_DBL(_mb, al, "memory") #undef __CMPXCHG_DBL +#define __CMPXCHG128(name, mb, cl...) \ +static __always_inline u128 \ +__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ +{ \ + union __u128_halves r, o = { .full = (old) }, \ + n = { .full = (new) }; \ + register unsigned long x0 asm ("x0") = o.low; \ + register unsigned long x1 asm ("x1") = o.high; \ + register unsigned long x2 asm ("x2") = n.low; \ + register unsigned long x3 asm ("x3") = n.high; \ + register unsigned long x4 asm ("x4") = (unsigned long)ptr; \ + \ + asm volatile( \ + __LSE_PREAMBLE \ + " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\ + : [old1] "+&r" (x0), [old2] "+&r" (x1), \ + [v] "+Q" (*(u128 *)ptr) \ + : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \ + [oldval1] "r" (o.low), [oldval2] "r" (o.high) \ + : cl); \ + \ + r.low = x0; r.high = x1; \ + \ + return r.full; \ +} + +__CMPXCHG128( , ) +__CMPXCHG128(_mb, al, "memory") + +#undef __CMPXCHG128 + #endif /* __ASM_ATOMIC_LSE_H */ --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -147,6 +147,19 @@ __CMPXCHG_DBL(_mb) #undef __CMPXCHG_DBL +#define __CMPXCHG128(name) \ +static inline u128 __cmpxchg128##name(volatile u128 *ptr, \ + u128 old, u128 new) \ +{ \ + return __lse_ll_sc_body(_cmpxchg128##name, \ + ptr, old, new); \ +} + +__CMPXCHG128( ) +__CMPXCHG128(_mb) + +#undef __CMPXCHG128 + #define __CMPXCHG_GEN(sfx) \ static __always_inline unsigned long __cmpxchg##sfx(volatile void *ptr, \ unsigned long old, \ @@ -229,6 +242,19 @@ __CMPXCHG_GEN(_mb) __ret; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128() 1 + +#define arch_cmpxchg128(ptr, o, n) \ +({ \ + __cmpxchg128_mb((ptr), (o), (n)); \ +}) + +#define arch_cmpxchg128_local(ptr, o, n) \ +({ \ + __cmpxchg128((ptr), (o), (n)); \ +}) + #define __CMPWAIT_CASE(w, sfx, sz) \ static inline void __cmpwait_case_##sz(volatile void *ptr, \ unsigned long val) \ --- a/arch/s390/include/asm/cmpxchg.h +++ b/arch/s390/include/asm/cmpxchg.h @@ -201,4 +201,18 @@ static __always_inline int __cmpxchg_dou (unsigned long)(n1), (unsigned long)(n2)); \ }) +#define system_has_cmpxchg128() 1 + +static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) +{ + asm volatile( + " cdsg %[old],%[new],%[ptr]\n" + : [old] "+d" (old), [ptr] "+QS" (*ptr) + : [new] "d" (new) + : "memory", "cc"); + return old; +} + +#define arch_cmpxchg128 arch_cmpxchg128 + #endif /* __ASM_CMPXCHG_H */ --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -103,6 +103,7 @@ static inline bool __try_cmpxchg64(volat #endif -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) +#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) +#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8) #endif /* _ASM_X86_CMPXCHG_32_H */ --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -20,6 +20,59 @@ arch_try_cmpxchg((ptr), (po), (n)); \ }) -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) +{ + union __u128_halves o = { .full = old, }, n = { .full = new, }; + + asm volatile(LOCK_PREFIX "cmpxchg16b %[ptr]" + : [ptr] "+m" (*ptr), + "+a" (o.low), "+d" (o.high) + : "b" (n.low), "c" (n.high) + : "memory"); + + return o.full; +} + +static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new) +{ + union __u128_halves o = { .full = old, }, n = { .full = new, }; + + asm volatile("cmpxchg16b %[ptr]" + : [ptr] "+m" (*ptr), + "+a" (o.low), "+d" (o.high) + : "b" (n.low), "c" (n.high) + : "memory"); + + return o.full; +} + +static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *old, u128 new) +{ + union __u128_halves o = { .full = *old, }, n = { .full = new, }; + bool ret; + + asm volatile(LOCK_PREFIX "cmpxchg16b %[ptr]" + CC_SET(e) + : CC_OUT(e) (ret), + [ptr] "+m" (*ptr), + "+a" (o.low), "+d" (o.high) + : "b" (n.low), "c" (n.high) + : "memory"); + + if (unlikely(!ret)) + *old = o.full; + + return likely(ret); +} + +#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) +#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16) #endif /* _ASM_X86_CMPXCHG_64_H */