From patchwork Thu Feb 2 14:50:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ADA1C61DA4 for ; Thu, 2 Feb 2023 15:29:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC2B66B0075; Thu, 2 Feb 2023 10:29:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C71776B0078; Thu, 2 Feb 2023 10:29:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC4486B007B; Thu, 2 Feb 2023 10:29:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9D0F66B0075 for ; Thu, 2 Feb 2023 10:29:19 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 64B491C6000 for ; Thu, 2 Feb 2023 15:29:19 +0000 (UTC) X-FDA: 80422735638.17.A983B2F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 98DCA4000E for ; Thu, 2 Feb 2023 15:29:16 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h4vsppu3; dmarc=none; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351756; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=SmAZMxmhVzZFpMcgEk2xjJ9SlIW4an+wqq1Ku13zd8I=; b=UsYKvvjSK3qzytdtLp/62AxH5Eq2nNEaB/9wvi+wIqePbjKgmWwGS0ntTVHGFVdtG//y7/ 2r4kaGnvv+Qp72z282YqWdUyFg2QmQ6WnpY07hEZghSozD7D916Tj06mRtmOh+CavTiAnF 9neN9VGX2JjYwuLLZVVDypL/OjMRoG8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h4vsppu3; dmarc=none; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351756; a=rsa-sha256; cv=none; b=CbGcdz8GRwAMEapkGDttY48ZD9TLInKvZNHi4mGYTOzQrKa0bf/befIJXRSZc1rBzlPSr4 1GczSaVuNPl0Irg44H0YuZOxOqLMLqrSXNVeTLaAj01uHPVtudxwsv+aMuVY3nz+yYBy2f sxDjb1lORBA9Be1AZc36eTVZHE84lbI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=SmAZMxmhVzZFpMcgEk2xjJ9SlIW4an+wqq1Ku13zd8I=; b=h4vsppu3IFznffLowHKrwukDM3 slWQ9lk2uwAZd19UeQ8IiP6R82RdPP/jl2f/J2Bp9b69mzYayvhTYAstyqu4YM7w/Qm7aRexqb8Fi fQKtWH0QhG0NPqA0vENbIPMnhqB/F+DW1yTsK/SNLBxsZ+l2DuNJkFsE0R/fqfkzGDR1VH/zOyBr/ kbyhWQHhXJ87Z+BFddWkPvPlDnD0qTMm+8fW+KHQw4EYkzITpu27WO8OztfpCgecLHKf25790kYjx VZRNEjKgQFWHaeZeahaKB/xNAXWZznALwCoy0s/Ne5hgd7KBKy/fdMMjB/r6uKcIQbSkov2P0s+5k KGDjPJfg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pNbW4-00DV2d-2S; Thu, 02 Feb 2023 15:28:48 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3BA2B3021E2; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 93E3223F31FBB; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.494373332@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:35 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 05/10] percpu: Wire up cmpxchg128 References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 98DCA4000E X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: p1effww6yfpjeiioaz4tqzkn17txrsmi X-HE-Tag: 1675351756-844924 X-HE-Meta: U2FsdGVkX18iQi2SiMJpjkd40vWZGQHSwCV6sJaBj0qm8rzc9X8kS8x17gaTbV+K32GWybxA+ovCEHKg9tCEPoAJMeD0frUqznVKki1y/1BmGksxpUc9uY1GG3uEpd54B7Cz4SK2qpRd7ohgBlFaORnVbAJjc9Ndk74FjiWYnDxo65cOY4LEfoAi63OTYe89/StIMjgdYXi2olPgSBEp1CF6VhqZBEZvPUSO0eFOWn0XLMyTrPm1M69gB4JV8/96Ik5O9R+Inbf+GpXm+3d65KWlmzHJgnWYThXgoYfCds+snPUoBERTZIFTE/24ea9W78QMB0r1Z6lZ0Xpa58VsPlEGPsRGdxe0wOX1aFJ/rD/kAImsicx/6TLcA3q7w7oP478nZyvmnc/8lHqSM/eTpTiRhfPKpNbGVPMw00NpHLihDi6lSxtp9xBWV19PxlKCnydfAJiSiYuDhwHFe6oZpCohS3/y8UvUe5sF3LrpVgJ85hA6k1dHDmCeuS5SHiU2LrVXk0rK6vmvGWpQI5pcCLyaiBD9CYYp3WEyV+jvpFxE+5FN7s6+FVFvzLHwWAac05eejNMpCFh4g2dO65hdzBygmzE36xMS0OwZlHEJCOGly8RCm2nepLOwVLFSDdHjA4B3IuA8h0Bhg8sbDSUTi99lgFf9ZnehrPr4cCEWsYqw5Djo4YUW91Hp3bwdXjZegas+8Ky+gDyk1Sp0+KfkObu9Jjfn/IgXtmLmRMpBtL/CQ2a4doJ929x65aAvlwMwwltotzBJSmg73oHvzwmx5S1GEb8g3Z9RF5WmvuwD6BimZcbRlV54LxPBgXLfiLnAIoNBxfga13OpeqOuYzQaZfli8qDuAQ30Dr1aQekX7pPY4Phjl3zrvFQXQyhw0py2X9j0cJ0IlzVHTIns0mdpFhBWF2E4pU0xZKTsrVu0QJTXxbzWKMwe6SrXXBcmsb8ktoy2pdlTfPoRCHwq1Kr ypWYH8Sm lUN9V7vugjnZww9YW9lWlRuHbUnvDgeBPeloAB9TsYJP6i5YltSL2qLsN4D9uOfAgssP9hNazYqblMWqho42Lt1U0v7H4E/iiLVFbLCBnt6f8UNDb/BsiQB1ao76yKY1Lli6LdRoyqpLc6STrTxnuJCuUTJSpRPecdlTgkXtIcEFol4E3P68NLcjD1Qp1bR+awsc62u5VP5nWNGWZbZsn0IzZ5nG7Ta1EcsxjMwIB6M0+IplEYLuGcdvcbLRyROStiTdQZ9fBBkqGuG4XUCpaBnVHgqKUMYsrxIpbi3TXr9cKJ/h9pl29BhMp2CJmmh01igaLkl3joJeWLtcAN0pAwWQzm3ZGbwEn4HSttp02sAzgHYE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to replace cmpxchg_double() with the newly minted cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg(). Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens Acked-by: Mark Rutland --- arch/arm64/include/asm/percpu.h | 21 +++++++++++++++ arch/s390/include/asm/percpu.h | 17 ++++++++++++ arch/x86/include/asm/percpu.h | 56 ++++++++++++++++++++++++++++++++++++++++ include/asm-generic/percpu.h | 8 +++++ include/linux/percpu-defs.h | 20 ++++++++++++-- 5 files changed, 120 insertions(+), 2 deletions(-) --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd) * re-enabling preemption for preemptible kernels, but doing that in a way * which builds inside a module would mean messing directly with the preempt * count. If you do this, peterz and tglx will hunt you down. + * + * Not to mention it'll break the actual preemption model for missing a + * preemption point when TIF_NEED_RESCHED gets set while preemption is + * disabled. */ #define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ ({ \ @@ -240,6 +244,23 @@ PERCPU_RET_OP(add, add, ldadd) #define this_cpu_cmpxchg_8(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_16(pcp, o, n) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + union { \ + pcp_op_T__ pot; \ + u128 val; \ + } old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__.pot = o; \ + new__.pot = n; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__.val = cmpxchg128_local((void *)ptr__, old__.val, new__.val); \ + preempt_enable_notrace(); \ + ret__.pot; \ +}) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); #define __per_cpu_offset --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -148,6 +148,23 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_16(pcp, oval, nval) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + union { \ + pcp_op_T__ pot; \ + u128 val; \ + } old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__.pot = oval; \ + new__.pot = nval; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__.val = cmpxchg128((void *)ptr__, old__.val, new__.val); \ + preempt_enable_notrace(); \ + ret__.pot; \ +}) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -210,6 +210,62 @@ do { \ (typeof(_var))(unsigned long) pco_old__; \ }) +#if defined(CONFIG_X86_32) && defined(CONFIG_X86_CMPXCHG64) +#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + typeof(_var) var; \ + struct { \ + u32 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg8b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval) +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval) +#endif + +#ifdef CONFIG_X86_64 +#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + typeof(_var) var; \ + struct { \ + u64 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg16b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg_16(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval) +#define this_cpu_cmpxchg_16(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval) +#endif + /* * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -298,6 +298,10 @@ do { \ #define raw_cpu_cmpxchg_8(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_cmpxchg_16 +#define raw_cpu_cmpxchg_16(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef raw_cpu_cmpxchg_double_1 #define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ @@ -423,6 +427,10 @@ do { \ #define this_cpu_cmpxchg_8(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_16 +#define this_cpu_cmpxchg_16(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -343,6 +343,22 @@ static inline void __this_cpu_preempt_ch pscr2_ret__; \ }) +#define __pcpu_size16_call_return2(stem, variable, ...) \ +({ \ + typeof(variable) pscr2_ret__; \ + __verify_pcpu_ptr(&(variable)); \ + switch(sizeof(variable)) { \ + case 1: pscr2_ret__ = stem##1(variable, __VA_ARGS__); break; \ + case 2: pscr2_ret__ = stem##2(variable, __VA_ARGS__); break; \ + case 4: pscr2_ret__ = stem##4(variable, __VA_ARGS__); break; \ + case 8: pscr2_ret__ = stem##8(variable, __VA_ARGS__); break; \ + case 16: pscr2_ret__ = stem##16(variable, __VA_ARGS__); break; \ + default: \ + __bad_size_call_parameter(); break; \ + } \ + pscr2_ret__; \ +}) + /* * Special handling for cmpxchg_double. cmpxchg_double is passed two * percpu variables. The first has to be aligned to a double word @@ -425,7 +441,7 @@ do { \ #define raw_cpu_add_return(pcp, val) __pcpu_size_call_return2(raw_cpu_add_return_, pcp, val) #define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval) #define raw_cpu_cmpxchg(pcp, oval, nval) \ - __pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) + __pcpu_size16_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) #define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) @@ -512,7 +528,7 @@ do { \ #define this_cpu_add_return(pcp, val) __pcpu_size_call_return2(this_cpu_add_return_, pcp, val) #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ - __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) + __pcpu_size16_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2)