From patchwork Mon May 15 07:57:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82D09C77B7D for ; Mon, 15 May 2023 08:06:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8685900007; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE89C900004; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C6D9900005; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 70F4A900003 for ; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 383BD1A1367 for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) X-FDA: 80791758264.20.708D669 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf20.hostedemail.com (Postfix) with ESMTP id C50BB1C000B for ; Mon, 15 May 2023 08:06:49 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=C7Q+9vEV; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138010; a=rsa-sha256; cv=none; b=ZJDFHEGt4fx7PMxEOooi1Rrkb5peJSWmxE4596oibTPzohttm3PXrQ0Col6Lfx010Mv/nG jDoass0FNBhtF1V0S6LNTe1QQk0vACeyzphYBuQ1tXoxTfr6cvYcputf2L9MeJlSqxoTf1 4xM7PWYZAu6zD9PzS1+On7Wts/OwFLY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=C7Q+9vEV; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=3RRGLs3vZsBA3WquAm9CYBrs3TiBBk1z4f/JSTl06ko=; b=G/VOrhVrqP7wLG4zgtDr66ww9BuD8jT6cyxUC3vSLI/kkKbckg1VdscYcWVX4ZkY3ZCBQY Toj/+7mH4aMD27FevChUb2exuMaRp1pY7TM64kz/MDTGvbCDYo6L03GQQpBNTmAxi2wefL BIfr6CqKwS6QapPYoQRtIa+d5+iD0Ug= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=3RRGLs3vZsBA3WquAm9CYBrs3TiBBk1z4f/JSTl06ko=; b=C7Q+9vEV1QZqlrWrdiRS/4ZaOo C4QaD94dYLKM4rVSpBMuM4MPCUEwWG2wlZb7z7L+ytDDmEzthuIXFw5s4xhl1Zhwa8PHyegz3LsWv sX9yk6m2RhCvKI4eawfr4xg6NlwXbIHLgql/4Hh2IAyjJ5EKkfn282tlp6HqkTBBJgH+es2+j28ap mwmpcCi64hREGLbEv+l39wIXLA29adUS/A2oGsp2yAeCyreAL19bzS9x+aYuSwanUcEPwpYAb2/WQ Q6Uzz57SiaFlBid1/0NAHDSF0uzCiEKR6fHXhBcPEjn8/4auGrU3RiX6Ur+Re+rAmPdmvbZq7AL9R kV+s3owA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDl-00BQN3-1H; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8708C300ECD; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BBF7D202FCEA5; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.248739380@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:04 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 05/11] percpu: Wire up cmpxchg128 References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C50BB1C000B X-Stat-Signature: xrftro1q9g83n9rhg3e3hcddwebehb1x X-HE-Tag: 1684138009-3163 X-HE-Meta: U2FsdGVkX18po87AhZVbNInM07KnNV/z2uOZuMAlUXjeJX8xkiUqdZptZst4c3xxMcVWydVBHY+u+LkvOZTRYuSyulU83Fdm2abOCNp1tEThSrpv2WHP29ZUmDYUSDpuZ2Fvb/L7uUZm2kimH7M26o1u7TYu1AJLQkLOZhmJEpjQZ/aRU2eONenBN6/qohKBhvO95t5Ns4sM2I0YscACIdxEsl+iDyvnKLxdFNMMatETUXXj/9flbPzSOdtSyUmOXi5FeJoqHE7fHlQ1qMXJr9ZJGTSH085l8CloXdyayC76XlkeIBH2GwqErHOzATLBb+s9n4f+sHWUo8LQr6cURPdK+EyUdGOMLwgiJY9TXqZU6oUH2EgsBxESMqc9EvCbXDSLE4MuesM34O2i8T3rywFz43GRBaDlHybSG7kkFQDvjYlTwZKdQTChlhuflMfx0Tu58/DonhTJ20JB1bQSw+lwiXj2NeJbrnuJ6k8yEFTlRMCNBsUisze3WoVk1ksGJjSKZrXJthNnr2fWe8G+9qlTAiXoVTlaIjHqjnCJFRHh8qfMbUnxIRBY7nwOgU+1eYNX0Gg1E/gxP7EvRuZlXf3fp5bv+/tj6Xap4xanNqeVkSkXALiqJk8GUlbDRFQUy46XJ5Zs8Nbmg/CDa540aWMkhgJSxj4GbsPGmf8JpqePnMRDwx7WV46muuKsEkcYDzzdbo5rf+ha1/xZ7v6idTjxAZOBzh7R8HAjFg9Eg/7557984MbhhKnhEeUS/z/+zcO2E20orODf9NAstJ6qxrY13VjQXsukYbTj/LWV6l5FSEX8edGB5wT4KIgAvejz7CFaFA9PWFnwgTzop5yk3agKigo4SI+wYIxtQS1A1biJeEoasPR8ZbLPB+nhpAnPgrXAVU5Ne+mpHnQXi2+N5+6aTAmxlL+crV/H5lg4YkAMTdFZPWQc9sHD64dZvMfnqVzJfBnB85WdJdnoBTM 6AWloqJ6 /cHBWrGX3FnF4FbH+8pV2fl19CA/L6oZlLIfLW6jkQXBIHClkgCj1tkx6LMjETAWrq/4pjVazxse/Pp53oh5kSfBXwvT6t42F3zmAwEvPJX/UPru0Bp6xUXmSUb8xDauvi6qsxUw2FXr5AJL3KwsRx/i20+zKwVobMyDF3aNL/C1vkETENJNz9UQk3NjxuMu5DAuMYO4C+qGL9qCDAAcRocYhri1tEr+0Bacsk90tK6ZNX6ysyV12k20pNLPE5cJ4P90smlK1Qh6JbJI21YhqQ/ssHHvBjSiAqVIT02XPyrLQmX4Z/QiJ9EoedzWTFYUPC/j2Dto80b0jr8JOv0eyp3de9nciL2J0d4RK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to replace cmpxchg_double() with the newly minted cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg(). Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/percpu.h | 20 +++++++++++++ arch/s390/include/asm/percpu.h | 16 ++++++++++ arch/x86/include/asm/percpu.h | 59 ++++++++++++++++++++++++++++++++++++++++ include/asm-generic/percpu.h | 16 ++++++++++ 4 files changed, 111 insertions(+) --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd) * re-enabling preemption for preemptible kernels, but doing that in a way * which builds inside a module would mean messing directly with the preempt * count. If you do this, peterz and tglx will hunt you down. + * + * Not to mention it'll break the actual preemption model for missing a + * preemption point when TIF_NEED_RESCHED gets set while preemption is + * disabled. */ #define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ ({ \ @@ -240,6 +244,22 @@ PERCPU_RET_OP(add, add, ldadd) #define this_cpu_cmpxchg_8(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) + +#define this_cpu_cmpxchg128(pcp, o, n) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ = o; \ + new__ = n; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + ret__; \ +}) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); #define __per_cpu_offset --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -148,6 +148,22 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) + +#define this_cpu_cmpxchg128(pcp, oval, nval) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ = oval; \ + new__ = nval; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + ret__; \ +}) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -210,6 +210,65 @@ do { \ (typeof(_var))(unsigned long) pco_old__; \ }) +#if defined(CONFIG_X86_32) && defined(CONFIG_X86_CMPXCHG64) +#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + u64 var; \ + struct { \ + u32 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg8b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval) +#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval) +#endif + +#ifdef CONFIG_X86_64 +#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval); +#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval); + +#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + u128 var; \ + struct { \ + u64 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg16b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval) +#define this_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval) +#endif + /* * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -298,6 +298,14 @@ do { \ #define raw_cpu_cmpxchg_8(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_cmpxchg64 +#define raw_cpu_cmpxchg64(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef raw_cpu_cmpxchg128 +#define raw_cpu_cmpxchg128(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef raw_cpu_cmpxchg_double_1 #define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ @@ -423,6 +431,14 @@ do { \ #define this_cpu_cmpxchg_8(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg64 +#define this_cpu_cmpxchg64(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg128 +#define this_cpu_cmpxchg128(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \