From patchwork Wed May 31 13:08:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13262287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5734CC7EE2E for ; Wed, 31 May 2023 13:28:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DF6E8E0001; Wed, 31 May 2023 09:28:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 22C388E000B; Wed, 31 May 2023 09:28:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E13418E0008; Wed, 31 May 2023 09:28:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C43018E0006 for ; Wed, 31 May 2023 09:28:01 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6D1861A0163 for ; Wed, 31 May 2023 13:28:01 +0000 (UTC) X-FDA: 80850628362.08.6A39F3A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 2E1C914001D for ; Wed, 31 May 2023 13:27:58 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LIi2st65; dmarc=none; spf=none (imf09.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685539679; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=OYq3F8aZwPPC/Y8UsexFdal3aiucmNsqQObmCMz9zjE=; b=yIs7BDupvt8P6h5hwoprTVUj3zYX2VcY/QdMLnu/2gKWh32VO7Fa6ToPGysAhyoNLIt60P UNFTAy5lrdsIcI+0w8cyjzznQQAsYIaTCh8WBRJODvieZWGx8lM9E4RL+gFzwaHC/q7A+5 BK9aqIZWJgZnE5Bs2/QtFLbJAwXAJLs= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=LIi2st65; dmarc=none; spf=none (imf09.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685539679; a=rsa-sha256; cv=none; b=UBXtU06eXfs+BTjdvHxt+oZWoA5IgP3xemyUWv5b0aS/h7Oey0FMdfA35riIpNKACHfiCE Z6dKyhkvleI76To5jkaICrj0QZ38STiNoKZs6O/Jv8T5LkQN6TBjnFTkGTiFRJZIyN35x+ hO/JsMCu4Gu2YLPNfGzKJooiN67I/o8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=OYq3F8aZwPPC/Y8UsexFdal3aiucmNsqQObmCMz9zjE=; b=LIi2st653mrq+qr82jZjSZoSMu b4IdGvoFjlZSfW2/JQDfIQgkhRmizDc+3i69fP7F75A4WAkEYw/JPtYzsJYXvDjOlO3lYqV3CTQG2 W5QHSm3lF3LRIXC6DORpHYE0GY/d/aV+oN7w/SrBOka2MUxnFUNUBPn6uIzvLTkgZCknrMs9/URJT jHAuSX05RfqOvsyIyF/kFoW7hbFpLP/c+WXZHzxrpS37v59bTdQPKeQCwIDOzbrtY2xMJrtSDtp8c jQSyhSMxk2M6L9+FSzaQWKUsmX61W6UzXzdaMQ0wKEbO8TRWRg4MO80O6dbE/9Vv3BMgIz9xTSMYB XK7W6Mcg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1q4LrH-007IJW-3w; Wed, 31 May 2023 13:27:23 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 55A95300C0E; Wed, 31 May 2023 15:27:21 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 95BD0243B856F; Wed, 31 May 2023 15:27:16 +0200 (CEST) Message-ID: <20230531132323.722039569@infradead.org> User-Agent: quilt/0.66 Date: Wed, 31 May 2023 15:08:40 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org, sfr@canb.auug.org.au, mpe@ellerman.id.au, James.Bottomley@hansenpartnership.com, deller@gmx.de, linux-parisc@vger.kernel.org Subject: [PATCH 07/12] percpu: #ifndef __SIZEOF_INT128__ References: <20230531130833.635651916@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2E1C914001D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: co1jtwstq3ksju9btu3jg1ip6rjw7wti X-HE-Tag: 1685539678-869674 X-HE-Meta: U2FsdGVkX1/52chemtu4IMGla9cea4mBCE0q7+ndoryD7qc/mb0C/BLqqP4nmwxzVKoYBkfDV9JkzLhsQIsztVj8qDRYfj25ySDLajZXpha5IsjU4dicJ6XZWpeDCO+ktjO7JST7c1/urSzt0JNHvCo4T9H6A4m8ETjf2bU0PWrZTUYxTHDnqM8pcmNivJF7I+LUsSqfHluMIw9byPaT9C6xq/uRKQcqwrRMXpBE+3xm97q9mqNBqWVJfEyx8W3AyL+6r4XyDL45iwZAapmRuubMCIyC4S1GhFqjhCPHCnqCdJ28FDFT743kVhjUvr33/8y+Rxh+N0v+H2/wZ3TsiPu0ljStzagE3lme/phB2WcXJbm8WmnD816wrIWB9S1ZU/XlkDNX5xdus/CszIm1DoecfidZiUgOLeJfMPWLTn5qHGBBNfFa8UssmFJbjClY9B8+BrVFoNKPwVOKSxlMG+mngNNXkRcW0r+HMjqVsMLYsbsMZsTH9yLDS0S1fn+4BlLn9pDf4qsA98ZNRwfffKCimd64ic3BUavhy+xRLZ9dhfNeTfsyogygwqMkDGei90DYXvbv9MCmUPca+2KgNl9aVVCnHsV7PZPkCbBM4b97Plu7otFzQH+vhsSkoIuAD8NY5eo4bNiNQzA8mGvdNck5TzBiAUBNosBzTbl1FDVkW/JVCTgO96XQDsY7JUmwUFh9+ix8qbygXWAXa0OcbPS04rBHyCTIkFo4L2fE2D+0Vjps/yXi6MtZCU+rbS96+jxLTVNj3Y5nilb1dQbuycEfzTAHf24NpYr0YbMLbfo9IMXw2OqzkZRe836fsaNx0JOQNrnOCsQTf8ZRw6/mKmpZXet/HwsS9Irzc3uzAzFsgYIsOISJe9qr0FjfKUxxxToQQyHCnrIhFF1k9uNrN6c3p7LsJDYd8Pq5IpkTNeli81LmgHrJjR6Zou5lEOGlPDXCrtVqXnhkrGEcqQd 3BJJLREw MvIpp2xs7f2v05EvnBpjRJzphl4V22hMBS7ipeZTg8NzxM8kOmiNyU2n+gQuGsJk3G3TjeQ7nOwdmEtLx96ddbebBCX9mR/6yhwzrUFQKQobIq4Fsq5haMLfXcB/QLGwEyGz4d015LIHKXlxhYWpGGe4ThNPzb7PPXlcPzIjjPak6VyT9+5VbPBiV+up5XF8H5lWNcQ3u3IPd89r27cfCyK28H+a8rHvDAMr2fkpBoQclw05gMlkaW6aq6LSklpT1F/05gAZeQbRVqgzYrhQFrGm9nWhFhbG3NtfWr6H5McCbq4jdS3BJ5XB+Kquz3iuxKJ5uLN8m8OqgXL6DOYHRxExAMBbzAcZZeO+eg/XmAb3s3Mv/vU5F/ctvVlcI4waZQ2cROVUj18Qte2A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some 64bit architectures do not advertise __SIZEOF_INT128__ on all supported compiler versions. Notably the HPPA64 only started doing with GCC-11. Since the per-cpu ops are universally availably, and this_cpu_{,try_}cmpxchg128() is expected to be available on all 64bit architectures a wee bodge is in order. Sadly, while C reverts to memcpy() for assignment of POD types, it does not revert to memcmp() for for equality. Therefore frob that manually. Signed-off-by: Peter Zijlstra (Intel) --- include/asm-generic/percpu.h | 56 +++++++++++++++++++++++++++++++++++++++++++ include/linux/types.h | 7 +++++ 2 files changed, 63 insertions(+) --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -313,6 +313,35 @@ do { \ #define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval) #endif +#ifndef __SIZEOF_INT128__ +#define raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \ +({ \ + typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \ + typeof(pcp) __val = *__p, __old = *(ovalp); \ + bool __ret; \ + if (!__builtin_memcmp(&__val, &__old, sizeof(pcp))) { \ + *__p = nval; \ + __ret = true; \ + } else { \ + *(ovalp) = __val; \ + __ret = false; \ + } \ + __ret; \ +}) + +#define raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \ +({ \ + typeof(pcp) __old = (oval); \ + raw_cpu_generic_try_cmpxchg_memcpy(pcp, &__old, nval); \ + __old; \ +}) + +#define raw_cpu_cmpxchg128(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) +#define raw_cpu_try_cmpxchg128(pcp, ovalp, nval) \ + raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) +#endif + #ifndef raw_cpu_try_cmpxchg_1 #ifdef raw_cpu_cmpxchg_1 #define raw_cpu_try_cmpxchg_1(pcp, ovalp, nval) \ @@ -503,6 +532,33 @@ do { \ #define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval) #endif +#ifndef __SIZEOF_INT128__ +#define this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) \ +({ \ + bool __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = raw_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) \ +({ \ + typeof(pcp) __ret; \ + unsigned long __flags; \ + raw_local_irq_save(__flags); \ + __ret = raw_cpu_generic_cmpxchg_memcmp(pcp, oval, nval); \ + raw_local_irq_restore(__flags); \ + __ret; \ +}) + +#define this_cpu_cmpxchg128(pcp, oval, nval) \ + this_cpu_generic_cmpxchg_memcmp(pcp, oval, nval) +#define this_cpu_try_cmpxchg128(pcp, ovalp, nval) \ + this_cpu_generic_try_cmpxchg_memcmp(pcp, ovalp, nval) +#endif + #ifndef this_cpu_try_cmpxchg_1 #ifdef this_cpu_cmpxchg_1 #define this_cpu_try_cmpxchg_1(pcp, ovalp, nval) \ --- a/include/linux/types.h +++ b/include/linux/types.h @@ -13,6 +13,13 @@ #ifdef __SIZEOF_INT128__ typedef __s128 s128; typedef __u128 u128; +#else +#ifdef CONFIG_64BIT +/* hack for this_cpu_cmpxchg128 */ +typedef struct { + u64 a, b; +} u128 __attribute__((aligned(16))); +#endif #endif typedef u32 __kernel_dev_t;