From patchwork Thu Feb 2 14:50:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126376 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F354CC636D6 for ; Thu, 2 Feb 2023 15:29:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE6986B007B; Thu, 2 Feb 2023 10:29:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E47766B007D; Thu, 2 Feb 2023 10:29:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4AD66B0081; Thu, 2 Feb 2023 10:29:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A3F3D6B007D for ; Thu, 2 Feb 2023 10:29:22 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 767C8160301 for ; Thu, 2 Feb 2023 15:29:22 +0000 (UTC) X-FDA: 80422735764.17.B59B063 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf02.hostedemail.com (Postfix) with ESMTP id 1CAD58001C for ; Thu, 2 Feb 2023 15:29:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=f3v2Mpvq; spf=none (imf02.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351760; a=rsa-sha256; cv=none; b=IgUmSmHKj98z/oYhjTrRejztx3q02wO4bfsBQF2o5Db1MG6Pw2P1VGb3PcRDlRoRPEf7gz 3RY2EpciPCVrpa4xQoTwUh6tJG9yLkfqU7Pum92DueHCRRqLso8v5SgEgYTpRK/6fL5eog MhIgLFy18CLFeOUx3c9BOGUQltmqOrQ= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=f3v2Mpvq; spf=none (imf02.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=HwKpSq8Syzt9wYk5/qyQZy9JqJElqdY1XVTNG2HpXaA=; b=AFQjxL6FdWue3VoSO9GOqL1Z3JAaC/3umC2uphU3EuaQ8zJRDiPZ8i6owsL78vDbZ5jGhY P1Rv9VPU6+iCffwflkwrXtPuOmtumQGDzX4DAeKtvh0ywZsexWs1rr/WLNs/uSW9Z9wMtR dyiF2EVbx5Bblh7ynC2VduRXoP9wj2c= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=HwKpSq8Syzt9wYk5/qyQZy9JqJElqdY1XVTNG2HpXaA=; b=f3v2Mpvql11MUtfS2akSYxDAG8 Ac5da3MJnPyezReQuN4422epSgtiTYlOzqFuLziYo5jZeTrm9iZPT5QJe+fFKqJwYW56ZUpGr6HnN Jw/4Md5ktVKhlOKxTGSjiWtdNcPeZvnsfgFEC8IsK3XWjhnC1R0p3zqW529oo6T1L+wdRSNvX9STe 449/8ucwS1cVcvCHuhBOmCM9ldEQcPLoS3Q5cA6/BR6V5oNZGr2xuoUNuwbYsv6IopHfJEcW0eLuT rgfknVDH6HJm2BPg+cXG0OAiLn51lYPVxMn5CAgHbJsHd7GF1ZtruDek7QZeeDh5qqO1xYt7zWrEI /6jtD8KQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVT-005CFh-1o; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 077F13010E0; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8330A23F31FB5; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.250913242@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:31 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 01/10] cyrpto/b128ops: Remove struct u128 References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 1CAD58001C X-Rspamd-Server: rspam01 X-Stat-Signature: 94mahfw7p8dkyfdnkr1n8dejabsbx373 X-HE-Tag: 1675351759-962675 X-HE-Meta: U2FsdGVkX1/Q8hJbdNSYkXr1PTqvYHKQIxmmHIQdW7fxb4d7+teE7okZYaagZLsilnh8im0pViX/+PEPhYWXrhZHgZpFnamzZRirMvWcNw9qxcQo88uetGziqus0oYIPxz7Y2K4ms5DYEotF/bf4+aL6VDF+RvwrNrhqZZxJ0hu503fheSbatTAIPHyawOo0ruV6hNc6nxiTgkWVp7by2nU2XYwAsaTPiGzMpzjmvqcWoP8lbDlZRxb8916njynRjS/Bw/KGVFZWT05acgwM5dwH1zHHJSpQ9f8syYxCy9amZ3i6H2aqEROCxEsZjWPL6ecJjmcw3Yy6Itmqz+fuOH8pzuZJYogGZqFycuwyKoMBbn4Uaga14P/INYuzGUho+dGM3TWUZID9efTNmtslaB7nFFTUuGqOFwu2On8+JLp4nKKatjs49nYUYTGY8ZwjpOMHGzGQRYv3jmo79Grx+boOpguVgt/WP8JZghbZ0zEwsEyNKk3uyhbjhUwuACXjzAmdmzoedbZVOSeskhUkOaWv92gI2leUkal8D74tBt/rDUg9d/k9WyQxqanrPmMBUjrQzSYu1vSJJmzR1mCij4qO6nhdAHD6DHUo597/+xIE7yT1RjBMMXj4SeZ7RFq9CQ9gqeLxUDUUmczqXLI6GfYqwP+034lgEJC8W/3cMTlNlyHiOiVRGshDyw65mXWYKa8I+NpAr0OgKDPtmyQagjE73SUTcCYF981Q7XGUWHqD7SnzhDTOnSr0XIAbRTr3/6r7oXWo9MA13YBMm+Pgpdw3x1+lMdZojvFnxXMniKsvavjWTedUUpFWXwmYRNb466halLSHWsIRhzPqqxNr1Fg4GlBJQ2YgDCFrBi6X5+i7uOvK3EAGRPs0/att8fZxKMFIrb2fGQnbbZQ9igbKYT7tov1BA/siZbVoz8gthche9OQB0kMh6TaBImeScUeR0oHjw0JxpWmhIjNUTnh 7l7+8qoa /TyAiggWAzytchu0SV4mqVJ9A8Rqp0sffiLHO/v1yvgdtidL8a/+GC2IdhFmEoedcuaMZVOROcPjMs8Ja7kzIoq4c7Rfco9W2KrZ+7FYUEpq1vZuoscQPOT4X+SZNRs5EW/Yi2zBnTIZ6EhRGQjehmDAyrMWNLqhizvzXmKJ26xzMTP8bF7CuEth8DCV1wBx/fUvHTZOV8JiaMP8sFWDz8GOQLmQaHvMWN+02vBF9bvmKkGOb/FRkdb+I4SCbVrFYwS1c57MIJk7+e/HwuwyOAZ1NyuV5v0w0MgZ4Jz217rciAhtSOcHmwI2mzxkNk8DOgegGRH2i4POpA0JPI9351jCPGC5ONEAp2foXfhCS0rHWehU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per git-grep u128_xor() and its related struct u128 are unused except to implement {be,le}128_xor(). Remove them to free up the namespace. Signed-off-by: Peter Zijlstra (Intel) --- include/crypto/b128ops.h | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) --- a/include/crypto/b128ops.h +++ b/include/crypto/b128ops.h @@ -50,10 +50,6 @@ #include typedef struct { - u64 a, b; -} u128; - -typedef struct { __be64 a, b; } be128; @@ -61,20 +57,16 @@ typedef struct { __le64 b, a; } le128; -static inline void u128_xor(u128 *r, const u128 *p, const u128 *q) +static inline void be128_xor(be128 *r, const be128 *p, const be128 *q) { r->a = p->a ^ q->a; r->b = p->b ^ q->b; } -static inline void be128_xor(be128 *r, const be128 *p, const be128 *q) -{ - u128_xor((u128 *)r, (u128 *)p, (u128 *)q); -} - static inline void le128_xor(le128 *r, const le128 *p, const le128 *q) { - u128_xor((u128 *)r, (u128 *)p, (u128 *)q); + r->a = p->a ^ q->a; + r->b = p->b ^ q->b; } #endif /* _CRYPTO_B128OPS_H */ From patchwork Thu Feb 2 14:50:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AFFBC61DA4 for ; Thu, 2 Feb 2023 15:29:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4EAC6B0085; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BFCCD6B0083; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93DE66B0085; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 819D46B0080 for ; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5EEF81A05B9 for ; Thu, 2 Feb 2023 15:29:23 +0000 (UTC) X-FDA: 80422735806.07.2DB14BC Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf15.hostedemail.com (Postfix) with ESMTP id 95135A0016 for ; Thu, 2 Feb 2023 15:29:21 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=ZUZ1EWs+; spf=none (imf15.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=4C3XvlEzMOWDLYDQ8XunR/mwzv7K56/vPk2XJ/CM7ag=; b=uRg+jRdFBD2WwUhOAMmYjT2k/H2Lrr4OocYmJ4yVS1D7/7xkzQK3jyCDOO2Lg4G+8hRAKy sA4Q3IemrPA2f39kg8ezo0TjYYtnDtVtsVD7yOkiSf2mUOgZVh/128180pYkPuxknKQtfp WzH0zFfHrsp0d8ohpbNQ3Ae7BX8ZDvc= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=ZUZ1EWs+; spf=none (imf15.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351761; a=rsa-sha256; cv=none; b=Ewz/63ewXZR/JULljwtZIjIHWuUx9cbje2cM7aI9LeYO9tEIRAArFBBSBBT+7qbOQL2UvA sPszh/ANuvjQPaqEdIHdkcHCT3lX/9tSFDbdFb6fzg1SUpfWsTJzfsXtXQsHUmCELGRY1+ UHYnI87stG4dP5jvIJKpNMedyr2xFy0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=4C3XvlEzMOWDLYDQ8XunR/mwzv7K56/vPk2XJ/CM7ag=; b=ZUZ1EWs+4c6vvynJYe49Tmql6x E9Hwyf28DeRTT6vZ+IaEtJqSkdzqvVmas2ZpgoVLuZkqPEKd3ve/jHYhAKkHBhSE1Vruzdjv9NFAg sg0Ucbnq82V8ZGLKw74u4lh5Andr5WgxZ+l7vsftcEhZsa0tiiLRuIwKokYbHWpFDD3Z0bMsXQi5a BiZifR9/6fYGl3k8uVPVTlhK5cuHqpARfsBWDyuaKd9ptgf8fyagi0kS64lLeh/PurgpcMdHjY7n7 mIXERrd2lrAuy5AeTjEMyW8XGqNrNKGWJEjt3SIwCVKjHIcXLjpiPYUY2tz2OrZsdsVzHgkt9cEsl USOkEkfQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVT-005CFg-1p; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0552A300ECD; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8796E23F305BD; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.313053173@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:32 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 02/10] types: Introduce [us]128 References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: c866nhbdadmiqu9rzxoopyc86aj48cwj X-Rspamd-Queue-Id: 95135A0016 X-HE-Tag: 1675351761-822186 X-HE-Meta: U2FsdGVkX1/zcErIfam1HGHcUIkPFP+828rM5qw78jQvQrZ35cDC8Vol+29SFXozu0zzQLnuw5aJMnKOsix/KDg2Zka0rKMPvuVeYqL3eAKitzVOiyJk6u7BugyryDmtZYYmhibkFWY3P4Qc8Sq24DjZXAOhFnofHKNFwvlQ1aP20Bn4RGGPLGWJWJdUrAIhcreee6tEnxxNXD49SJnBigxSRx/NRThFgB6vKn4urlQ7+4yeffch07R4tLetrvUIoqx81ROvuSpnmQjcKmyqUNcXSP4e9LXa/S9JY3LELwW/8rE48zzJZ3+DpSI1k9XFD2HP8uXrLV8s/PgaWzPVRJMIDe0JWvEGWknp6uaDAGedj+2wfd892yo6xk14p2efbA1tMHHl0GXc16D+TiUOMs0uQ8Efuk6+XVf5C5jaT1NcOhQG1hnBw030cvVl08VWsi5utXFhE3G4+4KNw2xIY4BdEfJJRk5Col7E0Zu4856VnpBSeZydTQm/WqrPtubm6N3ON3W5VoD/8YolCdw6iGRGkIJR1GqbFiVgmWuND7VTEAyrQ2swB839OlPhkeoDfivMISI7FDy8A1lbifzvPCelQclmJBwie2yN1CI0tYhoDKExEoduffi+hEJt/JuY90yH8aCu0AmW7Xz2xp2JWUf16THwiBejkv05a3yqUQeudWHCQcHbNtusndkC71OIOS1ka6k0IAflwvbcZ00qxjI/fAv62ewQsczbxo0sPxMgQxr4BtM0j2yd1os1f2uI9rBYepssQiMWh7Ka+/Jk7LzEUQmohL9QIakF65qV6oSbtw5O9QoppOvmNKOhf5vlpUjaZFqZKFdSDPTo0NFJYxM6UYmPIJTG+Yp9oJpM5cVFH0znn/2LCSh6ACsV4d6F8X7OQzOgqDf58iDjz1/MoxE8YJW4ecir2ApFhjPBK2HfcWVe76CKdz+Z69+7c4CxgqrCdg05RR9yRst+XfB xEvPh2sL 5AKQQWEVlrreZXUxnfQBV1ChaKRbzhoLXGwJ5ez61AyxboIJlBhGvQfovO14gA7tn+ckCZeYLiagv64KSnhDJ0VfbHmIAYNuv5DHwdvSkaJGhfF1heDCW/2jMLdwS61yVboyk0ZNSz4WAzEIGKhiod3x5CFHvRFxQa14cNMYCcROzDmfSqblNmWLwKEHob2GLQmC0tDWG3ZUnk8KK1uQGJNFCN/zmd4a1CVHqNOS41YBSuuLZ8qDI7eUYxND0dO+3xvDqmbnbWDk+p0QQhl6ERs/50I+cRhKCNyXZj5LSkwDfpgbwsoWIql14YN/vkUYocRVn2ja5e3nveT1NyE7X6LHXVMJmFVtbsMElsmk3DGYOynlLMoSwl//5OWnfhXDmTPHkOTz2Dsj31rM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce [us]128 (when available). Unlike [us]64, ensure they are always naturally aligned. This also enables 128bit wide atomics (which require natural alignment) such as cmpxchg128(). Signed-off-by: Peter Zijlstra (Intel) --- include/linux/types.h | 5 +++++ include/uapi/linux/types.h | 4 ++++ 2 files changed, 9 insertions(+) --- a/include/linux/types.h +++ b/include/linux/types.h @@ -10,6 +10,11 @@ #define DECLARE_BITMAP(name,bits) \ unsigned long name[BITS_TO_LONGS(bits)] +#ifdef __SIZEOF_INT128__ +typedef __s128 s128; +typedef __u128 u128; +#endif + typedef u32 __kernel_dev_t; typedef __kernel_fd_set fd_set; --- a/include/uapi/linux/types.h +++ b/include/uapi/linux/types.h @@ -13,6 +13,10 @@ #include +#ifdef __SIZEOF_INT128__ +typedef __signed__ __int128 __s128 __attribute__((aligned(16))); +typedef unsigned __int128 __u128 __attribute__((aligned(16))); +#endif /* * Below are truly Linux-specific types that should never collide with --- a/lib/crypto/curve25519-hacl64.c +++ b/lib/crypto/curve25519-hacl64.c @@ -14,8 +14,6 @@ #include #include -typedef __uint128_t u128; - static __always_inline u64 u64_eq_mask(u64 a, u64 b) { u64 x = a ^ b; --- a/lib/crypto/poly1305-donna64.c +++ b/lib/crypto/poly1305-donna64.c @@ -10,8 +10,6 @@ #include #include -typedef __uint128_t u128; - void poly1305_core_setkey(struct poly1305_core_key *key, const u8 raw_key[POLY1305_BLOCK_SIZE]) { From patchwork Thu Feb 2 14:50:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B87DC05027 for ; Thu, 2 Feb 2023 15:29:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FD6A6B0081; Thu, 2 Feb 2023 10:29:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 45CB26B0083; Thu, 2 Feb 2023 10:29:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C55E6B0087; Thu, 2 Feb 2023 10:29:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 08BAA6B0081 for ; Thu, 2 Feb 2023 10:29:25 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BEF24120F61 for ; Thu, 2 Feb 2023 15:29:24 +0000 (UTC) X-FDA: 80422735848.03.815E82C Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf13.hostedemail.com (Postfix) with ESMTP id C3F1C2000C for ; Thu, 2 Feb 2023 15:29:21 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=Ql+cyqR5; spf=none (imf13.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=sduTuNa9Y93QC8FtEhMkgsBeoFYULf3DBAkiVbSNSHc=; b=1vEAWlb78a+MA1d0FvhbjPqE0ws9Ko1qxx8btgnQ1h6SqaX07fiwlARrXUEjvrJ8nUEtdM FCPCHoDGwSHCMpJsrmQOs8gyWJ/DiJJpoBteDNpamg2dwSNCN+okWek2AamNOPZSmNzHrt iv/CUZfNPsJsR8h/QkjHMNoIvSY6T9g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=Ql+cyqR5; spf=none (imf13.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351762; a=rsa-sha256; cv=none; b=0S3Ng/dwmUBq55rP/RtrrJFk6JBDueyIoovZ699D2gzsFdearPL/CueB83j0hF5zwaaptP 62y2xpMto3qcRjF+UjyjYgn/Oy86X4V63xCUMU/vBiwAUhUxgQ+bXczuCmGqg6Sn6zoCaa OmADUYJjg/XzqtUgIvubO6RiCjORXE8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=sduTuNa9Y93QC8FtEhMkgsBeoFYULf3DBAkiVbSNSHc=; b=Ql+cyqR51WKsltXZ5zb2xdtiWO l8ltfVfe3HF1kXMg3+syulZDERIZ071AJ59oVxg/E/ywg9uykQ07KTmbyZgpZJSaK+luBotas/BE/ E4ONu9TlXb4PR6kfN72XUPocrOt5KRQl0z2UATFAW0O8iNzGguxzS8vTSgYk7mGYI2B5E2xD1mtIE j+dU1aDeLic8DmDJ/tRtKYdpyLH0biZQtc4ZRnML9WbKIjz62UAnOVFdTa7gJkxanOrIYbbWmjIXL o+OWTB8xyNWH5gXiU/UIxEMoI7nqiJ39nJ+ejT/m+wVG6ouhh9fedoBJ+By2n6WOZoHChcjblobbU hi2nLMSg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVT-005CFf-1o; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 00F29300446; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8ADCD23F31FB4; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.373335780@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:33 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 03/10] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C3F1C2000C X-Rspam-User: X-Stat-Signature: ygbe9byjcfi1y7iez54prkkp4rjdcjf4 X-HE-Tag: 1675351761-464499 X-HE-Meta: U2FsdGVkX18aXEmERRENxaDirMtr00Bn6N04jZaJ5YKMkH4fCl3qOx3UElVgBEKiPbqeq5o1GxBdhOI/UA1YfzXgl+VNOokwDU9W7f4aDM7luLuy5Cax/g8s5q1KLA4Y8ERKvk5muLBdQAmxIt2LlqYoD+fB0HEjFOUrpjv2FepMXjiqTaxvX3YFIyApSgb9A0fZUxVDCGJ6Fw9DogmEDB6zPfUbRZTqwe7n41xHigYwo08bYcy4rWyL/oujpyAQX1ULte+4PluxS3WMFJ8rQif697810t4764KyUe16C5QJtJpqrIpWXwB4ut/6qijaGvdBec3NTVvNhzpdi70E9fsQLCx3Ow5SiVxQZ5rVl6tbF9bf9ljBAIHthPV6Lco9NGWqsulp1oJ+ElU43KyFj/7OWj769xzG8JNtxxs4uLtPkdcXak8FG5LVdA/4v1a+TRiEG43b+bFUOtKUPzYLacZqOEODywSVT/MMuk1IIxPrq8oujYA7FGRC8jOWkRI3ND+EA1atcaJqOp63CJJja8b0gYyU2JoV6MqH/RrYIjcP3DL57RaW250UliNSH2qmuCIej/FMfTMZmkSns9RfJmiOk43ezl7p5PWrBiB7AG/6SlwkxdQ/TIADmtcnVIFyWenb5vr13r10aCEXYFTaXPfX6CktfhIxVcRs7YzjkXf6qIMzoCsz6sAzrrj4LDxn3UwQ6K8I8YKazzFM1lrFeRE9uqU24C8c+ll5uuymKH8XjjtukJBM7bbYd1OEdAOXf/kkxFiV6VamjNOafSYufsXr2i9YzTBVxV4JdoDFQSPuN+sPbLRReg8nZsNlSgkrXeKPUl6aqWYqvuST3J2sGY/Y7/wj/84KAx/+EsO/G9DJxUSo9sUeay3skHcsCz47j2IAA26wZznhX+7GtyAqMU2WSZRGw5Cy5PTFuL0zR9IKJP79fegVCG7CiGe/RwEBzjPXcc8Xbx1Q9TKmFEU x93VAmY+ XBiv5SEEpBbah+vAvIRyKj7tJe57Cnk4zlO8DyzlThXNSOIvIg1vg41A1M6JmbcZvQARrBrqE2cl33phpVgTLNas80n7nAuxj+olIAOt52sbFfKFLu+lMWWA05rMygZ5YH8KsaoHrToyXNHFOOGfTr3Py9MreqjlMVc/+/tL8YzZnZZmBa3Ug33MOn9AFJZDC1fDloWU+qnPUlRRw94N+vH8i0KXqVubV00bOwuh/Ejqs3hOujrHN6NNz/b8bxguiouscd53j4cbPvE0+KEdiGicdF1AkiHZBa14cKioDduCnS6NEJsWSDVgk6U9Z1o9CDHF4D7Sra5iQqiPimstVkfjY2WAu9GDX/EpizYPzAAW9MJ4zrrgX9f3w9hhJNcM7mGH3797sFRJ1GZYvDL5MzF9C90SHc+YHdDml X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For all architectures that currently support cmpxchg_double() implement the cmpxchg128() family of functions that is basically the same but with a saner interface. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens Acked-by: Mark Rutland --- arch/arm64/include/asm/atomic_ll_sc.h | 41 +++++++++++++++++++++++++ arch/arm64/include/asm/atomic_lse.h | 31 +++++++++++++++++++ arch/arm64/include/asm/cmpxchg.h | 26 ++++++++++++++++ arch/s390/include/asm/cmpxchg.h | 14 ++++++++ arch/x86/include/asm/cmpxchg_32.h | 3 + arch/x86/include/asm/cmpxchg_64.h | 55 +++++++++++++++++++++++++++++++++- 6 files changed, 168 insertions(+), 2 deletions(-) --- a/arch/arm64/include/asm/atomic_ll_sc.h +++ b/arch/arm64/include/asm/atomic_ll_sc.h @@ -326,6 +326,47 @@ __CMPXCHG_DBL( , , , ) __CMPXCHG_DBL(_mb, dmb ish, l, "memory") #undef __CMPXCHG_DBL + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __CMPXCHG128(name, mb, rel, cl...) \ +static __always_inline u128 \ +__ll_sc__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ +{ \ + union __u128_halves r, o = { .full = (old) }, \ + n = { .full = (new) }; \ + unsigned int tmp; \ + \ + asm volatile("// __cmpxchg128" #name "\n" \ + " prfm pstl1strm, %[v]\n" \ + "1: ldxp %[rl], %[rh], %[v]\n" \ + " cmp %[rl], %[ol]\n" \ + " ccmp %[rh], %[oh], 0, eq\n" \ + " b.ne 2f\n" \ + " st" #rel "xp %w[tmp], %[nl], %[nh], %[v]\n" \ + " cbnz %w[tmp], 1b\n" \ + " " #mb "\n" \ + "2:" \ + : [v] "+Q" (*(u128 *)ptr), \ + [rl] "=&r" (r.low), [rh] "=&r" (r.high), \ + [tmp] "=&r" (tmp) \ + : [ol] "r" (o.low), [oh] "r" (o.high), \ + [nl] "r" (n.low), [nh] "r" (n.high) \ + : "cc", ##cl); \ + \ + return r.full; \ +} + +__CMPXCHG128( , , ) +__CMPXCHG128(_mb, dmb ish, l, "memory") + +#undef __CMPXCHG128 + #undef K #endif /* __ASM_ATOMIC_LL_SC_H */ --- a/arch/arm64/include/asm/atomic_lse.h +++ b/arch/arm64/include/asm/atomic_lse.h @@ -324,4 +324,35 @@ __CMPXCHG_DBL(_mb, al, "memory") #undef __CMPXCHG_DBL +#define __CMPXCHG128(name, mb, cl...) \ +static __always_inline u128 \ +__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ +{ \ + union __u128_halves r, o = { .full = (old) }, \ + n = { .full = (new) }; \ + register unsigned long x0 asm ("x0") = o.low; \ + register unsigned long x1 asm ("x1") = o.high; \ + register unsigned long x2 asm ("x2") = n.low; \ + register unsigned long x3 asm ("x3") = n.high; \ + register unsigned long x4 asm ("x4") = (unsigned long)ptr; \ + \ + asm volatile( \ + __LSE_PREAMBLE \ + " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\ + : [old1] "+&r" (x0), [old2] "+&r" (x1), \ + [v] "+Q" (*(u128 *)ptr) \ + : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \ + [oldval1] "r" (o.low), [oldval2] "r" (o.high) \ + : cl); \ + \ + r.low = x0; r.high = x1; \ + \ + return r.full; \ +} + +__CMPXCHG128( , ) +__CMPXCHG128(_mb, al, "memory") + +#undef __CMPXCHG128 + #endif /* __ASM_ATOMIC_LSE_H */ --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -147,6 +147,19 @@ __CMPXCHG_DBL(_mb) #undef __CMPXCHG_DBL +#define __CMPXCHG128(name) \ +static inline u128 __cmpxchg128##name(volatile u128 *ptr, \ + u128 old, u128 new) \ +{ \ + return __lse_ll_sc_body(_cmpxchg128##name, \ + ptr, old, new); \ +} + +__CMPXCHG128( ) +__CMPXCHG128(_mb) + +#undef __CMPXCHG128 + #define __CMPXCHG_GEN(sfx) \ static __always_inline unsigned long __cmpxchg##sfx(volatile void *ptr, \ unsigned long old, \ @@ -229,6 +242,19 @@ __CMPXCHG_GEN(_mb) __ret; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128() 1 + +#define arch_cmpxchg128(ptr, o, n) \ +({ \ + __cmpxchg128_mb((ptr), (o), (n)); \ +}) + +#define arch_cmpxchg128_local(ptr, o, n) \ +({ \ + __cmpxchg128((ptr), (o), (n)); \ +}) + #define __CMPWAIT_CASE(w, sfx, sz) \ static inline void __cmpwait_case_##sz(volatile void *ptr, \ unsigned long val) \ --- a/arch/s390/include/asm/cmpxchg.h +++ b/arch/s390/include/asm/cmpxchg.h @@ -201,4 +201,18 @@ static __always_inline int __cmpxchg_dou (unsigned long)(n1), (unsigned long)(n2)); \ }) +#define system_has_cmpxchg128() 1 + +static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) +{ + asm volatile( + " cdsg %[old],%[new],%[ptr]\n" + : [old] "+d" (old), [ptr] "+QS" (*ptr) + : [new] "d" (new) + : "memory", "cc"); + return old; +} + +#define arch_cmpxchg128 arch_cmpxchg128 + #endif /* __ASM_CMPXCHG_H */ --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -103,6 +103,7 @@ static inline bool __try_cmpxchg64(volat #endif -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) +#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) +#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8) #endif /* _ASM_X86_CMPXCHG_32_H */ --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -20,6 +20,59 @@ arch_try_cmpxchg((ptr), (po), (n)); \ }) -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) +{ + union __u128_halves o = { .full = old, }, n = { .full = new, }; + + asm volatile(LOCK_PREFIX "cmpxchg16b %[ptr]" + : [ptr] "+m" (*ptr), + "+a" (o.low), "+d" (o.high) + : "b" (n.low), "c" (n.high) + : "memory"); + + return o.full; +} + +static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new) +{ + union __u128_halves o = { .full = old, }, n = { .full = new, }; + + asm volatile("cmpxchg16b %[ptr]" + : [ptr] "+m" (*ptr), + "+a" (o.low), "+d" (o.high) + : "b" (n.low), "c" (n.high) + : "memory"); + + return o.full; +} + +static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *old, u128 new) +{ + union __u128_halves o = { .full = *old, }, n = { .full = new, }; + bool ret; + + asm volatile(LOCK_PREFIX "cmpxchg16b %[ptr]" + CC_SET(e) + : CC_OUT(e) (ret), + [ptr] "+m" (*ptr), + "+a" (o.low), "+d" (o.high) + : "b" (n.low), "c" (n.high) + : "memory"); + + if (unlikely(!ret)) + *old = o.full; + + return likely(ret); +} + +#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) +#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16) #endif /* _ASM_X86_CMPXCHG_64_H */ From patchwork Thu Feb 2 14:50:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126377 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DF9BC05027 for ; Thu, 2 Feb 2023 15:29:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 293C06B007D; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19FB76B0082; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE74E6B0080; Thu, 2 Feb 2023 10:29:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 92F766B007E for ; Thu, 2 Feb 2023 10:29:22 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 51BF180F2F for ; Thu, 2 Feb 2023 15:29:22 +0000 (UTC) X-FDA: 80422735764.17.04A522F Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf12.hostedemail.com (Postfix) with ESMTP id 1589340023 for ; Thu, 2 Feb 2023 15:29:19 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=joKm4qjS; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=VyG6+KQHCAeEAEZCgwKflnVjYNYIFWnZVnfAizA2/bE=; b=RYqhgmwf4U3xYDYdxQsxJbHQx+caQbQp3NGf347ng2gKhndDEJPfHlBaV/+tAQj63nD8e4 Ya6GhKqPTf5YNjdODufPNLyDg8bj5XZKpYISjWM3Ei5zB1KYHdIej9kYuXwDkfiV6XKKVt nuo6awbIsz0W6UbeU+95KahAi7oK7bM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=joKm4qjS; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351760; a=rsa-sha256; cv=none; b=ib8D7pOC8CT/W4Myb+byEYU55L4PSmBErCKQSDtTzR1Fa7JNa3wJqfxeUfDF4N1anEEE7x RuprjNKs9U6ynmpJjWaSFeml1RK+ychyzCwcRBr6pAGc9ry90pRd9rfZuI/jDN9gDm1ZBr ClyNFUmyfNR5rTGWKz9Q+yj2571xj0M= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=VyG6+KQHCAeEAEZCgwKflnVjYNYIFWnZVnfAizA2/bE=; b=joKm4qjSNFbo4hFpEXGNuPRr+M x6C5AUB7KuB7mYmWZ8TfjRhnW9H0blPwPiJnykDLW6v51sQfgfyaxBFX3KpHYtJrLS8sqTxZIuCXw qdjov4449SEqRvitwAf8Sd7TSv329PRG4oug2ZxlILrwyM0Jh4AqX1+EVIl2yr9QKd5UpOgx7dTyq lgK0tlJ3jkAs/BdwoWpjxH873GJEWPzTbwwJfM0ow40bR8Qfwr4+gbXL9vpVs1ZAenAWLuoq5aGdx lVVWKHojJbD5Z11OlT50xTw1DOeEcnjjGX6jZgdN/9qHW9cfJnjUj0kA9GbtgZNG4bs5rEzayP5TR ZJ0qDe3Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVT-005CFe-1p; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 03242300E3F; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8DFE523F31FBA; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.433050731@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:34 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 04/10] instrumentation: Wire up cmpxchg128() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1589340023 X-Rspam-User: X-Stat-Signature: k6ex3oopdzuxohh3txr87a8tm8sdoxxm X-HE-Tag: 1675351759-901314 X-HE-Meta: U2FsdGVkX18U2rnI2khFkaGcxS2kqdJ9/C1ykeEUeN205PrRjZEE2Ycijh5kPhNa2T+TVYQduAUHyzCfP9LtowyJjHS0XOJlN6y5rDUi8pvN7Dfjg/vyfXsKt4rlDkdz3xNNc0cQ2thsmDLK7KdcD4xzIb/zS2a0F26ikSbFpop57kBfhI11oanFtKsWzmIAomQg096V1PmNeEictq675HMPYV8wejASDfS6siyZA8i1vi4FEx9TWMKIng92tT0ApD6UdQWyfazDR6qkKS2V9uvx5OXEzFQF1OYPGfgvBvMsOyo7qInc8jQjmL/c7pxzLIbSoTsmTYuCcMe843062MKdCY1qJK3vAVHd7Iqif513NrCdmj7w0Bl6VDgT7TUXN3e3XDSw8GBrc1JhPx+L42hSd1+sFVLXbIAkYcBA/jBJmj/5iJwqXAAmUk/VxEPmMtFf+ofdJiSaMH21FF2kj8Yfx+X32GDzp/3hXg+EKRd8csQ3ARl84e57ZpEHtPeAdQkNuo9XD0W7ct5QUKD0Rtf+wxF76/mHJwe6tqhnWfwj2+YvHLGOTQSqag25qpmjgav63/H6MB42UcqDhHyWPQeKa3i4Wf19+ngShYdnantsDluObYPZNaahlH23oUZM2XShkaqOEgWeQCf9XTJ/ml6DZPvmXacU0B0PS8S54gF0mVvfR78VgzedAE43CRtrdRmw61RQbThMcbOu2Q5gkeNhJXV8m99fSen7xGqVRqnGuCD+UhjnQI3swGCA3HnV/0xjLt9xvKOgHAw92AieOk2wl11BM2rfKbHx4JnIpUVyKuy+qN03NnMUWqMbSFtqufnsCWvyHT1OCezT5vGKqMS0gFWf+hnsOHVPbNFmXauTJfmlQUkWPi8BCLbgvQ8nI8Xyn22lLTzBqPMhyDmaFwaSTZaXJOZ1GxLNai8GjWfTxSkWOsKdtqfBsGuZcqGH3/l4VSccN1WXZy0Vlvx qXOJusvD /kgLvsdvayd6aQiJ52xWUDExpUkyCco7StjRsYZuwZFj0lbATb6dxDqb83tMMBhrJVEYdID3+ksytVaRgs8iKbm2Nt9XpnTj0hHcqkaMmuJw6WJq3bpdvWilM2hCegi+gnaMVtuWjd7Yq4yh3UPeUti5c6SBNqN0HrH4wtqNUppD6tzvE1eLs2wU7W7GzDnNI07zLx/0yuzA42bAGL9iksdZ1LaAPvHVYK1CwbWH3Puk1cGZq3ZgAex5npS6huIW+DCh8obAY5dON4KGdozzaZU9oACwtp/9cikLRCGwz3vz2FBs38E56a/qOnw5ELTO5Bw5Jdma7qA8ug2tFMUz++W3+pF/dKTW10Q7jKu6QYWmv5pU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Wire up the cmpxchg128 familty in the atomic wrappery scripts. These provide the generic cmpxchg128 family of functions from the arch_ prefixed version, adding explicit instrumentation where needed. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Mark Rutland --- include/linux/atomic/atomic-arch-fallback.h | 95 +++++++++++++++++++++++++++- include/linux/atomic/atomic-instrumented.h | 77 ++++++++++++++++++++++ scripts/atomic/gen-atomic-fallback.sh | 4 - scripts/atomic/gen-atomic-instrumented.sh | 4 - 4 files changed, 174 insertions(+), 6 deletions(-) --- a/include/linux/atomic/atomic-arch-fallback.h +++ b/include/linux/atomic/atomic-arch-fallback.h @@ -77,6 +77,29 @@ #endif /* arch_cmpxchg64_relaxed */ +#ifndef arch_cmpxchg128_relaxed +#define arch_cmpxchg128_acquire arch_cmpxchg128 +#define arch_cmpxchg128_release arch_cmpxchg128 +#define arch_cmpxchg128_relaxed arch_cmpxchg128 +#else /* arch_cmpxchg128_relaxed */ + +#ifndef arch_cmpxchg128_acquire +#define arch_cmpxchg128_acquire(...) \ + __atomic_op_acquire(arch_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_cmpxchg128_release +#define arch_cmpxchg128_release(...) \ + __atomic_op_release(arch_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_cmpxchg128 +#define arch_cmpxchg128(...) \ + __atomic_op_fence(arch_cmpxchg128, __VA_ARGS__) +#endif + +#endif /* arch_cmpxchg128_relaxed */ + #ifndef arch_try_cmpxchg_relaxed #ifdef arch_try_cmpxchg #define arch_try_cmpxchg_acquire arch_try_cmpxchg @@ -217,6 +240,76 @@ #endif /* arch_try_cmpxchg64_relaxed */ +#ifndef arch_try_cmpxchg128_relaxed +#ifdef arch_try_cmpxchg128 +#define arch_try_cmpxchg128_acquire arch_try_cmpxchg128 +#define arch_try_cmpxchg128_release arch_try_cmpxchg128 +#define arch_try_cmpxchg128_relaxed arch_try_cmpxchg128 +#endif /* arch_try_cmpxchg128 */ + +#ifndef arch_try_cmpxchg128 +#define arch_try_cmpxchg128(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128 */ + +#ifndef arch_try_cmpxchg128_acquire +#define arch_try_cmpxchg128_acquire(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128_acquire((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128_acquire */ + +#ifndef arch_try_cmpxchg128_release +#define arch_try_cmpxchg128_release(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128_release((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128_release */ + +#ifndef arch_try_cmpxchg128_relaxed +#define arch_try_cmpxchg128_relaxed(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128_relaxed((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128_relaxed */ + +#else /* arch_try_cmpxchg128_relaxed */ + +#ifndef arch_try_cmpxchg128_acquire +#define arch_try_cmpxchg128_acquire(...) \ + __atomic_op_acquire(arch_try_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_try_cmpxchg128_release +#define arch_try_cmpxchg128_release(...) \ + __atomic_op_release(arch_try_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_try_cmpxchg128 +#define arch_try_cmpxchg128(...) \ + __atomic_op_fence(arch_try_cmpxchg128, __VA_ARGS__) +#endif + +#endif /* arch_try_cmpxchg128_relaxed */ + #ifndef arch_atomic_read_acquire static __always_inline int arch_atomic_read_acquire(const atomic_t *v) @@ -2456,4 +2549,4 @@ arch_atomic64_dec_if_positive(atomic64_t #endif #endif /* _LINUX_ATOMIC_FALLBACK_H */ -// b5e87bdd5ede61470c29f7a7e4de781af3770f09 +// 46357a526de89c762d30fb238f35a7d5950a670b --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -1968,6 +1968,36 @@ atomic_long_dec_if_positive(atomic_long_ arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \ }) +#define cmpxchg128(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128(__ai_ptr, __VA_ARGS__); \ +}) + +#define cmpxchg128_acquire(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_acquire(__ai_ptr, __VA_ARGS__); \ +}) + +#define cmpxchg128_release(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + kcsan_release(); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_release(__ai_ptr, __VA_ARGS__); \ +}) + +#define cmpxchg128_relaxed(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_relaxed(__ai_ptr, __VA_ARGS__); \ +}) + #define try_cmpxchg(ptr, oldp, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2044,6 +2074,44 @@ atomic_long_dec_if_positive(atomic_long_ arch_try_cmpxchg64_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ }) +#define try_cmpxchg128(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + kcsan_mb(); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg128_acquire(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg128_release(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + kcsan_release(); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg128_relaxed(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_atomic_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + #define cmpxchg_local(ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2058,6 +2126,13 @@ atomic_long_dec_if_positive(atomic_long_ arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \ }) +#define cmpxchg128_local(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + instrument_atomic_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_local(__ai_ptr, __VA_ARGS__); \ +}) + #define sync_cmpxchg(ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2083,4 +2158,4 @@ atomic_long_dec_if_positive(atomic_long_ }) #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 764f741eb77a7ad565dc8d99ce2837d5542e8aee +// 27320c1ec2bf2878ecb9df3ea4816a7bc0c57a52 --- a/scripts/atomic/gen-atomic-fallback.sh +++ b/scripts/atomic/gen-atomic-fallback.sh @@ -217,11 +217,11 @@ cat << EOF EOF -for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64"; do +for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64" "arch_cmpxchg128"; do gen_xchg_fallbacks "${xchg}" done -for cmpxchg in "cmpxchg" "cmpxchg64"; do +for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do gen_try_cmpxchg_fallbacks "${cmpxchg}" done --- a/scripts/atomic/gen-atomic-instrumented.sh +++ b/scripts/atomic/gen-atomic-instrumented.sh @@ -166,14 +166,14 @@ grep '^[a-z]' "$1" | while read name met done -for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg" "try_cmpxchg64"; do +for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do for order in "" "_acquire" "_release" "_relaxed"; do gen_xchg "${xchg}" "${order}" "" printf "\n" done done -for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg"; do +for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg"; do gen_xchg "${xchg}" "" "" printf "\n" done From patchwork Thu Feb 2 14:50:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126374 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ADA1C61DA4 for ; Thu, 2 Feb 2023 15:29:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC2B66B0075; Thu, 2 Feb 2023 10:29:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C71776B0078; Thu, 2 Feb 2023 10:29:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC4486B007B; Thu, 2 Feb 2023 10:29:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9D0F66B0075 for ; Thu, 2 Feb 2023 10:29:19 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 64B491C6000 for ; Thu, 2 Feb 2023 15:29:19 +0000 (UTC) X-FDA: 80422735638.17.A983B2F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id 98DCA4000E for ; Thu, 2 Feb 2023 15:29:16 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h4vsppu3; dmarc=none; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351756; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=SmAZMxmhVzZFpMcgEk2xjJ9SlIW4an+wqq1Ku13zd8I=; b=UsYKvvjSK3qzytdtLp/62AxH5Eq2nNEaB/9wvi+wIqePbjKgmWwGS0ntTVHGFVdtG//y7/ 2r4kaGnvv+Qp72z282YqWdUyFg2QmQ6WnpY07hEZghSozD7D916Tj06mRtmOh+CavTiAnF 9neN9VGX2JjYwuLLZVVDypL/OjMRoG8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=h4vsppu3; dmarc=none; spf=none (imf12.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351756; a=rsa-sha256; cv=none; b=CbGcdz8GRwAMEapkGDttY48ZD9TLInKvZNHi4mGYTOzQrKa0bf/befIJXRSZc1rBzlPSr4 1GczSaVuNPl0Irg44H0YuZOxOqLMLqrSXNVeTLaAj01uHPVtudxwsv+aMuVY3nz+yYBy2f sxDjb1lORBA9Be1AZc36eTVZHE84lbI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=SmAZMxmhVzZFpMcgEk2xjJ9SlIW4an+wqq1Ku13zd8I=; b=h4vsppu3IFznffLowHKrwukDM3 slWQ9lk2uwAZd19UeQ8IiP6R82RdPP/jl2f/J2Bp9b69mzYayvhTYAstyqu4YM7w/Qm7aRexqb8Fi fQKtWH0QhG0NPqA0vENbIPMnhqB/F+DW1yTsK/SNLBxsZ+l2DuNJkFsE0R/fqfkzGDR1VH/zOyBr/ kbyhWQHhXJ87Z+BFddWkPvPlDnD0qTMm+8fW+KHQw4EYkzITpu27WO8OztfpCgecLHKf25790kYjx VZRNEjKgQFWHaeZeahaKB/xNAXWZznALwCoy0s/Ne5hgd7KBKy/fdMMjB/r6uKcIQbSkov2P0s+5k KGDjPJfg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pNbW4-00DV2d-2S; Thu, 02 Feb 2023 15:28:48 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3BA2B3021E2; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 93E3223F31FBB; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.494373332@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:35 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 05/10] percpu: Wire up cmpxchg128 References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 98DCA4000E X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: p1effww6yfpjeiioaz4tqzkn17txrsmi X-HE-Tag: 1675351756-844924 X-HE-Meta: U2FsdGVkX18iQi2SiMJpjkd40vWZGQHSwCV6sJaBj0qm8rzc9X8kS8x17gaTbV+K32GWybxA+ovCEHKg9tCEPoAJMeD0frUqznVKki1y/1BmGksxpUc9uY1GG3uEpd54B7Cz4SK2qpRd7ohgBlFaORnVbAJjc9Ndk74FjiWYnDxo65cOY4LEfoAi63OTYe89/StIMjgdYXi2olPgSBEp1CF6VhqZBEZvPUSO0eFOWn0XLMyTrPm1M69gB4JV8/96Ik5O9R+Inbf+GpXm+3d65KWlmzHJgnWYThXgoYfCds+snPUoBERTZIFTE/24ea9W78QMB0r1Z6lZ0Xpa58VsPlEGPsRGdxe0wOX1aFJ/rD/kAImsicx/6TLcA3q7w7oP478nZyvmnc/8lHqSM/eTpTiRhfPKpNbGVPMw00NpHLihDi6lSxtp9xBWV19PxlKCnydfAJiSiYuDhwHFe6oZpCohS3/y8UvUe5sF3LrpVgJ85hA6k1dHDmCeuS5SHiU2LrVXk0rK6vmvGWpQI5pcCLyaiBD9CYYp3WEyV+jvpFxE+5FN7s6+FVFvzLHwWAac05eejNMpCFh4g2dO65hdzBygmzE36xMS0OwZlHEJCOGly8RCm2nepLOwVLFSDdHjA4B3IuA8h0Bhg8sbDSUTi99lgFf9ZnehrPr4cCEWsYqw5Djo4YUW91Hp3bwdXjZegas+8Ky+gDyk1Sp0+KfkObu9Jjfn/IgXtmLmRMpBtL/CQ2a4doJ929x65aAvlwMwwltotzBJSmg73oHvzwmx5S1GEb8g3Z9RF5WmvuwD6BimZcbRlV54LxPBgXLfiLnAIoNBxfga13OpeqOuYzQaZfli8qDuAQ30Dr1aQekX7pPY4Phjl3zrvFQXQyhw0py2X9j0cJ0IlzVHTIns0mdpFhBWF2E4pU0xZKTsrVu0QJTXxbzWKMwe6SrXXBcmsb8ktoy2pdlTfPoRCHwq1Kr ypWYH8Sm lUN9V7vugjnZww9YW9lWlRuHbUnvDgeBPeloAB9TsYJP6i5YltSL2qLsN4D9uOfAgssP9hNazYqblMWqho42Lt1U0v7H4E/iiLVFbLCBnt6f8UNDb/BsiQB1ao76yKY1Lli6LdRoyqpLc6STrTxnuJCuUTJSpRPecdlTgkXtIcEFol4E3P68NLcjD1Qp1bR+awsc62u5VP5nWNGWZbZsn0IzZ5nG7Ta1EcsxjMwIB6M0+IplEYLuGcdvcbLRyROStiTdQZ9fBBkqGuG4XUCpaBnVHgqKUMYsrxIpbi3TXr9cKJ/h9pl29BhMp2CJmmh01igaLkl3joJeWLtcAN0pAwWQzm3ZGbwEn4HSttp02sAzgHYE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to replace cmpxchg_double() with the newly minted cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg(). Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens Acked-by: Mark Rutland --- arch/arm64/include/asm/percpu.h | 21 +++++++++++++++ arch/s390/include/asm/percpu.h | 17 ++++++++++++ arch/x86/include/asm/percpu.h | 56 ++++++++++++++++++++++++++++++++++++++++ include/asm-generic/percpu.h | 8 +++++ include/linux/percpu-defs.h | 20 ++++++++++++-- 5 files changed, 120 insertions(+), 2 deletions(-) --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd) * re-enabling preemption for preemptible kernels, but doing that in a way * which builds inside a module would mean messing directly with the preempt * count. If you do this, peterz and tglx will hunt you down. + * + * Not to mention it'll break the actual preemption model for missing a + * preemption point when TIF_NEED_RESCHED gets set while preemption is + * disabled. */ #define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ ({ \ @@ -240,6 +244,23 @@ PERCPU_RET_OP(add, add, ldadd) #define this_cpu_cmpxchg_8(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg_16(pcp, o, n) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + union { \ + pcp_op_T__ pot; \ + u128 val; \ + } old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__.pot = o; \ + new__.pot = n; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__.val = cmpxchg128_local((void *)ptr__, old__.val, new__.val); \ + preempt_enable_notrace(); \ + ret__.pot; \ +}) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); #define __per_cpu_offset --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -148,6 +148,23 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg_16(pcp, oval, nval) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + union { \ + pcp_op_T__ pot; \ + u128 val; \ + } old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__.pot = oval; \ + new__.pot = nval; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__.val = cmpxchg128((void *)ptr__, old__.val, new__.val); \ + preempt_enable_notrace(); \ + ret__.pot; \ +}) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -210,6 +210,62 @@ do { \ (typeof(_var))(unsigned long) pco_old__; \ }) +#if defined(CONFIG_X86_32) && defined(CONFIG_X86_CMPXCHG64) +#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + typeof(_var) var; \ + struct { \ + u32 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg8b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval) +#define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval) +#endif + +#ifdef CONFIG_X86_64 +#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + typeof(_var) var; \ + struct { \ + u64 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg16b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg_16(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval) +#define this_cpu_cmpxchg_16(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval) +#endif + /* * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -298,6 +298,10 @@ do { \ #define raw_cpu_cmpxchg_8(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_cmpxchg_16 +#define raw_cpu_cmpxchg_16(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef raw_cpu_cmpxchg_double_1 #define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ @@ -423,6 +427,10 @@ do { \ #define this_cpu_cmpxchg_8(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg_16 +#define this_cpu_cmpxchg_16(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -343,6 +343,22 @@ static inline void __this_cpu_preempt_ch pscr2_ret__; \ }) +#define __pcpu_size16_call_return2(stem, variable, ...) \ +({ \ + typeof(variable) pscr2_ret__; \ + __verify_pcpu_ptr(&(variable)); \ + switch(sizeof(variable)) { \ + case 1: pscr2_ret__ = stem##1(variable, __VA_ARGS__); break; \ + case 2: pscr2_ret__ = stem##2(variable, __VA_ARGS__); break; \ + case 4: pscr2_ret__ = stem##4(variable, __VA_ARGS__); break; \ + case 8: pscr2_ret__ = stem##8(variable, __VA_ARGS__); break; \ + case 16: pscr2_ret__ = stem##16(variable, __VA_ARGS__); break; \ + default: \ + __bad_size_call_parameter(); break; \ + } \ + pscr2_ret__; \ +}) + /* * Special handling for cmpxchg_double. cmpxchg_double is passed two * percpu variables. The first has to be aligned to a double word @@ -425,7 +441,7 @@ do { \ #define raw_cpu_add_return(pcp, val) __pcpu_size_call_return2(raw_cpu_add_return_, pcp, val) #define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval) #define raw_cpu_cmpxchg(pcp, oval, nval) \ - __pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) + __pcpu_size16_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) #define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) @@ -512,7 +528,7 @@ do { \ #define this_cpu_add_return(pcp, val) __pcpu_size_call_return2(this_cpu_add_return_, pcp, val) #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ - __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) + __pcpu_size16_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) #define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) From patchwork Thu Feb 2 14:50:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49B12C05027 for ; Thu, 2 Feb 2023 16:02:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C334A6B0071; Thu, 2 Feb 2023 11:02:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE3BA6B0073; Thu, 2 Feb 2023 11:02:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A849F6B0074; Thu, 2 Feb 2023 11:02:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 974A36B0071 for ; Thu, 2 Feb 2023 11:02:46 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5AACDC0F79 for ; Thu, 2 Feb 2023 16:02:46 +0000 (UTC) X-FDA: 80422819932.27.EE4D2B8 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf23.hostedemail.com (Postfix) with ESMTP id 7E7DE14001D for ; Thu, 2 Feb 2023 16:02:42 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=FGER7ivR; spf=none (imf23.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675353762; a=rsa-sha256; cv=none; b=IP7e0S8bFe8+Ckw9N970csTc3uXohQsngivC+a069EaH/Y+L8lt5hsnJvUtHn6xfuvFSoJ o8gR4O/PUzkvlo5yOz7NDIwWdlxRLBCBXRhfv2B8hxvpI7YSUGp0+Iqb2bXNaoxs5s44G5 Ic3yJdewSMYzW27h09z4Uao+pZt/Dd0= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=FGER7ivR; spf=none (imf23.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675353762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=ojAeZWZTS/q5DGsRDsr4pNfeq1PMMFo32HAk+mO3778=; b=VeiB/2obZPn75fM6bPanDN1Qjns6lNW2VVP/WDCBYtp+RYrIUgb7V+ORDn8He3gqeF7kjv CiNpe+AhsieZKGqzmtQHjxxKnioaW/1v9EnCRWoweTjNS286QEgwqRV4dBH6moJroiT0YL 2ZltlRZJH9JHCbg7CCcxupSyxXUNn+w= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ojAeZWZTS/q5DGsRDsr4pNfeq1PMMFo32HAk+mO3778=; b=FGER7ivRI9nqOjWL8qnZWxwDNb 39pm5f8BWqpO7fYq6QibN6ZMQNOeNsT4Tg9w7Vo+6gECc8/1oeRV2ABFtChGZ49limW+d/1pIgGMI 6wZgZ5jkMGHagoUdy5N0VpZWWjsJ3arChLUNp8epccoypGY5Q5iFN52IwwNXOvdwaeXRq9dSRv67L jexzaT3xActH30W/+i3QbicxHZqQOv425gL5Sur5FXyTLpHPwvrtDjg8n9dBZqrcjfi6sgZ4wuAdR HndfRFQO3mBLVi9gwLq1o2+rRTAz/XzyLKaQsYHafQ62KU2RwBgaEtKPlqLleGgm1mDqKHiB31COp qIdnlelA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVU-005CFv-21; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3AE643021D4; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 96E8D23F31FB6; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.564329626@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:36 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org, Vasant Hegde Subject: [PATCH v2 06/10] x86,amd_iommu: Replace cmpxchg_double() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 7E7DE14001D X-Rspamd-Server: rspam01 X-Stat-Signature: ezkagao95kd3kg47je4gthtas8ag8cih X-HE-Tag: 1675353762-923385 X-HE-Meta: U2FsdGVkX19ISJB6mnjGhf8ggmbxMqobUDIvx01a9zhv3528jam8szoU3p2aYBaCfkjumb61LG8elQXCg53aDn7cy29dktAGpIudbAPUMVM8X958lK2mIeayac1MKr9uO58wWLCtpKSEO0XZo5170Xk9YJDnXgihqktREpBJEgxgY/OBaJ+XNEtKO4hqLl/j+3J1tpJDk2/cleucxUEtcyUzpFonr3Xh0Ia+Md6cxoAN/CL3b2LKy0AaD3ROAo0+87xbxuDXSdZCEhJGgyrfMFWVSf2RhukF39JYKC7ug5N+FBzmpnQLddDp3s0wZNzjp4IcrJwkBAE4nFvuI+BGOVUdWrlcU7U5kLeUnSoxBpsRrLxVkcbzuUgkcWpPtDDQrY+B4bGwPwb3Y/XI5gSE7DRwdo1HguRS5xnsSsqPdwQRcuNoMpfGsCJd1lfD8kYs2HlIsSL79FKoL12P5mM+7gNlJd/t/cU5PeOYS+FEgTrgD39/0E/Tnz/jjsTSv32oCl6THsoPBO8uOQ2X02VybY9t5Dc1OVRD61vEo1dSuBmAmvw4/kxnM5M0NUPJnwWkWUPbbkkZ537elbOqzMkFLDO/sGKpn9aUNo/epSZqrMhQY2ZCIcQZEeu6quCGexkD7WcFJl15zAIJ03YvLl6vpABHagSWpGIcZNttKkRO4P3NCuWU5Vv7fb7+eEQ69lApKtQ8cZbSETJMT+ihULCqfRzWxZtKKo2iPI83idTF5a8qh92g1MPD/v3B8i5jKtb+SPbYuPFue+TKfmfHg0kbJ3+QT0HFC0X76LrCHvh8bXkpLqYp3YK2RgblW4Fkg37ahaPgJnDprZ1U7OHRRGasHi3LTqnvQv4selQT7r0GnkDZe2J36xIw4zyGWHOT/Kk7ilMBefhluTciViXAKIT++p8hO8YgzIN2PEHa830eY3qiVaXEpC3EnFFHuQGMY+A6mbvRTXaf3a3LEkBWQPv yWfu34eA z/dP6gVNzmAquK/TCNDvJdfN62n/xt8FNmonM9+oq82e4yyQ2/QRiZUhV3kBOpHgOQdw9tWcVxnrB+FsbJcsVxUCGOCv5uvyuufjAhyyWdf0pgJorK8X7UAb0vt5hrTk/scuekf4PQKxtgQDWQYmuV30PkqVZ7/QslSIfrV5bm//j+qGkKyU3cEukqyR24RhwhZaQXTX1NddnRHKSi7mXS+9wl5KSDcZpCj/aTZQPef9a3JHUr7oJztGdirVMZNBBPDK/qwCTaqhIIr19pqhJkkg7iurlRvJzbR5Pfps0WwGVCsE3E55yhigHCmiV6BLKgdfhbm98OCqfqhBTZyR8g2vxO3sXAfUGKho5QYwnHLv23B9O904yE6IvCKf+j+6YtdM7w1ergnga+SlKOiTlMI3S+4mdSUFEnS50J0NWv4J0MzK2fvT+wBJgMg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) Tested-by: Vasant Hegde --- drivers/iommu/amd/amd_iommu_types.h | 9 +++++++-- drivers/iommu/amd/iommu.c | 10 ++++------ 2 files changed, 11 insertions(+), 8 deletions(-) --- a/drivers/iommu/amd/amd_iommu_types.h +++ b/drivers/iommu/amd/amd_iommu_types.h @@ -979,8 +979,13 @@ union irte_ga_hi { }; struct irte_ga { - union irte_ga_lo lo; - union irte_ga_hi hi; + union { + struct { + union irte_ga_lo lo; + union irte_ga_hi hi; + }; + u128 irte; + }; }; struct irq_2_irte { --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -2992,10 +2992,10 @@ static int alloc_irq_index(struct amd_io static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index, struct irte_ga *irte, struct amd_ir_data *data) { - bool ret; struct irq_remap_table *table; - unsigned long flags; struct irte_ga *entry; + unsigned long flags; + u128 old; table = get_irq_table(iommu, devid); if (!table) @@ -3006,16 +3006,14 @@ static int modify_irte_ga(struct amd_iom entry = (struct irte_ga *)table->table; entry = &entry[index]; - ret = cmpxchg_double(&entry->lo.val, &entry->hi.val, - entry->lo.val, entry->hi.val, - irte->lo.val, irte->hi.val); /* * We use cmpxchg16 to atomically update the 128-bit IRTE, * and it cannot be updated by the hardware or other processors * behind us, so the return value of cmpxchg16 should be the * same as the old value. */ - WARN_ON(!ret); + old = entry->irte; + WARN_ON(!try_cmpxchg128(&entry->irte, &old, irte->irte)); if (data) data->ref = entry; From patchwork Thu Feb 2 14:50:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126378 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDC38C636D4 for ; Thu, 2 Feb 2023 15:29:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56F356B0082; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AA2F6B0083; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD0D56B007E; Thu, 2 Feb 2023 10:29:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B82586B007B for ; Thu, 2 Feb 2023 10:29:22 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 770F980F1E for ; Thu, 2 Feb 2023 15:29:22 +0000 (UTC) X-FDA: 80422735764.20.8E7A0F1 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf06.hostedemail.com (Postfix) with ESMTP id 2754118000B for ; Thu, 2 Feb 2023 15:29:19 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=Huq1Bblp; spf=none (imf06.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=3ubHXmjMHuO7YFbxrF9R90Y+YVnse9xBuEbHAitlUas=; b=IrH8OF/DFHdkY2NUgmjAmckW6VITGpH9pNQTYgpx+Jy2e+4bQqkPboqujcBn37pfMjdtrg pN512+nFuJdxvMYz4FzAGfRtw2jfYYijX8mfqeRJ4L+goVwhx6gSjra32uyBfoyS+rSzSp xh2//nxqAAxhB8VymzDFoT8hFiWMYWo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=Huq1Bblp; spf=none (imf06.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351760; a=rsa-sha256; cv=none; b=HCQqqIEjkhCTp5BEBGgWfjI1xcGDE43gONA2oRIXFmg6+JnveGAc1uq/B+CU9RRQDm8G6m sgOmuVGbfhHYtFKaj0Huh86W/BZlNKWFyklhRli7CQvSM2btdEzZtw3ifnekka4Spk7lWr zblM8cOggNMVsEPrpkiyssEHUBk2mF8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=3ubHXmjMHuO7YFbxrF9R90Y+YVnse9xBuEbHAitlUas=; b=Huq1BblpHoH+3Obm8iACci5gq3 megzhmNOz8SY2lHByGtJBiuzwCZGxcMReZzqMQ4nIME6rS6KZPdBpp1dHuZ00OwNftd1HbhvcySE3 68GXZUzOIxjxl/PwuKG0UW9kglxjDzk22ad/LD6UJv9zU4ZgdkrBGLgtw+Cz/xSyQt3S1hhz17MSd 2NH8seRyMtxOnwvDrB2Z74x1Ewfp5FBvRH95zDsTzpjJKo5MKphmgIskhBW4cEQnuCCQ7WcDOGGtJ goq/K07AkOWFmKlEHfneue1iwLtuIO6jiCeUIAifHqjRKGFxknHNxxnuLwpxLHmhgTgElUSqJXVoQ hqTg8AmQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVU-005CFw-21; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3AE0D3007C5; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 9AAE923F31FBD; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.624998774@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:37 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 07/10] x86,intel_iommu: Replace cmpxchg_double() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2754118000B X-Stat-Signature: eaazumrprq8dg4duzfeq6g18wopmp84f X-Rspam-User: X-HE-Tag: 1675351759-193980 X-HE-Meta: U2FsdGVkX188T2b71GfStdRNSQoOKkVLwb2M+NFMLJnILZVgTfxBUt8tKllehamr0Ezb6p3w/UTKbj7z6gjpGqeOjDeP8yp9JI29aq3g5+6UzY3Cad9ag+v6HChGIZg9wfHdZh7PTmxdkX8OzbnErIZpqPNHRcEZx4/y2hvGhXtD+QTyioFRhlSp1akk3My2Cy023VMu6cXmogTpuCtFyptU47uhF6+9a89UoXTpZld883DbDjtuuf7wdQLTtc7WUkSUOg5YsKeZeiEHIEBcKVvM3PKJTX/x0SA59Ym8eAbIYiDDcwYMIe+b2A5FW2IPwydt65d/c2AiIUU1lZH46gdju4paDrvR8ownP9QhfJQyd9FgG5PLZp+SU+MfZRPxASkr7ANWKN5kR+ps4zwoesANZJcCd7/1sNbp1Ef6anSUN5PoNR1g8EHWjDgblbRSD/EaD/ajWdy6wL8PjUolbAK2pR4d9GPfORvM4/JAWWeDB17SSdvFhzZdGU8eWnZ0/JG1SnhwCbZbup3/CyKQXus8Bvsk9rwlTbq63pkWNnJwubjzTjnZCBU8guexikeZ0fOTJqh7oi/FINSA0ilbw40xpk48WCJ6lRBuSy5TFBrPy1crcw8NPVkRdvt+1DMVtcEN1TuP9l+ncZ37vg3de9W22ykCpByDqYOp/G/hpmUfIx5TdYv2uS2xgQUAZxIQ0QoofnhiYanoTrmIhV948mehya6z36NMY8MheQRKPQbEzt9M5KYcfFLDJ7Futs4m0qM8z1LrxywDswAI9OOvl74+dPf4oU1noPsxP29ZdVb8o/9SH4AQbgbUWiHy3hK+VuJsKp9X2uNfn55L/Mzu5caTuP0pCrIRX5g9WLs4sZrazNKGshZcUvwZiSOBKeZQii2dHggNToKgMalYhoSSH5PI6p+zN3IiKOO8TOcQDyvarKd3u41in2grf8MugoYTCVNgP93347Wvo2iiWUO Fqfcb3G2 y+cRXHIg3buIMSSwBvhGR9BPTUV5TvXaclTPj+UcaZUzH4EdWbu41oTA3GYBJQE8dRA8grRgUcszYmtkyd6DhT8yrHLC/tBzOW5E1l7GrcW0dkSab//RXKmNsJvbJzvpAxMtFc7U/INeITPQAM0y9u9+Poezp5BlvfyTRvOgzBRU8Y+ANSZqlcu4/r+gJWY7KkBxxuR66oHujwqqc378XFKe57fRVeM0ywZwmZkWDgrluF81EUyRW+VzSriiqWIyBhbszHi7l1lg3LGJfi0G+sLb51SAd4iYWtZ9KYq+YHT8Y6CIHCScM1mAvogGpiodgazUCRa87hoNgWkQCRKJvaA5QVzAQPOkdurpez4R6erUkD2YcAPMYsFJIh0sTOh6Khq9ty5RfiJnPsjEY6Pc9B2iNZUhQYUPgToOa X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lu Baolu --- drivers/iommu/intel/irq_remapping.c | 8 -- include/linux/dmar.h | 125 +++++++++++++++++++----------------- 2 files changed, 68 insertions(+), 65 deletions(-) --- a/drivers/iommu/intel/irq_remapping.c +++ b/drivers/iommu/intel/irq_remapping.c @@ -174,18 +174,14 @@ static int modify_irte(struct irq_2_iomm irte = &iommu->ir_table->base[index]; if ((irte->pst == 1) || (irte_modified->pst == 1)) { - bool ret; - - ret = cmpxchg_double(&irte->low, &irte->high, - irte->low, irte->high, - irte_modified->low, irte_modified->high); /* * We use cmpxchg16 to atomically update the 128-bit IRTE, * and it cannot be updated by the hardware or other processors * behind us, so the return value of cmpxchg16 should be the * same as the old value. */ - WARN_ON(!ret); + u128 old = irte->irte; + WARN_ON(!try_cmpxchg128(&irte->irte, &old, irte_modified->irte)); } else { WRITE_ONCE(irte->low, irte_modified->low); WRITE_ONCE(irte->high, irte_modified->high); --- a/include/linux/dmar.h +++ b/include/linux/dmar.h @@ -201,67 +201,74 @@ static inline void detect_intel_iommu(vo struct irte { union { - /* Shared between remapped and posted mode*/ struct { - __u64 present : 1, /* 0 */ - fpd : 1, /* 1 */ - __res0 : 6, /* 2 - 6 */ - avail : 4, /* 8 - 11 */ - __res1 : 3, /* 12 - 14 */ - pst : 1, /* 15 */ - vector : 8, /* 16 - 23 */ - __res2 : 40; /* 24 - 63 */ + union { + /* Shared between remapped and posted mode*/ + struct { + __u64 present : 1, /* 0 */ + fpd : 1, /* 1 */ + __res0 : 6, /* 2 - 6 */ + avail : 4, /* 8 - 11 */ + __res1 : 3, /* 12 - 14 */ + pst : 1, /* 15 */ + vector : 8, /* 16 - 23 */ + __res2 : 40; /* 24 - 63 */ + }; + + /* Remapped mode */ + struct { + __u64 r_present : 1, /* 0 */ + r_fpd : 1, /* 1 */ + dst_mode : 1, /* 2 */ + redir_hint : 1, /* 3 */ + trigger_mode : 1, /* 4 */ + dlvry_mode : 3, /* 5 - 7 */ + r_avail : 4, /* 8 - 11 */ + r_res0 : 4, /* 12 - 15 */ + r_vector : 8, /* 16 - 23 */ + r_res1 : 8, /* 24 - 31 */ + dest_id : 32; /* 32 - 63 */ + }; + + /* Posted mode */ + struct { + __u64 p_present : 1, /* 0 */ + p_fpd : 1, /* 1 */ + p_res0 : 6, /* 2 - 7 */ + p_avail : 4, /* 8 - 11 */ + p_res1 : 2, /* 12 - 13 */ + p_urgent : 1, /* 14 */ + p_pst : 1, /* 15 */ + p_vector : 8, /* 16 - 23 */ + p_res2 : 14, /* 24 - 37 */ + pda_l : 26; /* 38 - 63 */ + }; + __u64 low; + }; + + union { + /* Shared between remapped and posted mode*/ + struct { + __u64 sid : 16, /* 64 - 79 */ + sq : 2, /* 80 - 81 */ + svt : 2, /* 82 - 83 */ + __res3 : 44; /* 84 - 127 */ + }; + + /* Posted mode*/ + struct { + __u64 p_sid : 16, /* 64 - 79 */ + p_sq : 2, /* 80 - 81 */ + p_svt : 2, /* 82 - 83 */ + p_res3 : 12, /* 84 - 95 */ + pda_h : 32; /* 96 - 127 */ + }; + __u64 high; + }; }; - - /* Remapped mode */ - struct { - __u64 r_present : 1, /* 0 */ - r_fpd : 1, /* 1 */ - dst_mode : 1, /* 2 */ - redir_hint : 1, /* 3 */ - trigger_mode : 1, /* 4 */ - dlvry_mode : 3, /* 5 - 7 */ - r_avail : 4, /* 8 - 11 */ - r_res0 : 4, /* 12 - 15 */ - r_vector : 8, /* 16 - 23 */ - r_res1 : 8, /* 24 - 31 */ - dest_id : 32; /* 32 - 63 */ - }; - - /* Posted mode */ - struct { - __u64 p_present : 1, /* 0 */ - p_fpd : 1, /* 1 */ - p_res0 : 6, /* 2 - 7 */ - p_avail : 4, /* 8 - 11 */ - p_res1 : 2, /* 12 - 13 */ - p_urgent : 1, /* 14 */ - p_pst : 1, /* 15 */ - p_vector : 8, /* 16 - 23 */ - p_res2 : 14, /* 24 - 37 */ - pda_l : 26; /* 38 - 63 */ - }; - __u64 low; - }; - - union { - /* Shared between remapped and posted mode*/ - struct { - __u64 sid : 16, /* 64 - 79 */ - sq : 2, /* 80 - 81 */ - svt : 2, /* 82 - 83 */ - __res3 : 44; /* 84 - 127 */ - }; - - /* Posted mode*/ - struct { - __u64 p_sid : 16, /* 64 - 79 */ - p_sq : 2, /* 80 - 81 */ - p_svt : 2, /* 82 - 83 */ - p_res3 : 12, /* 84 - 95 */ - pda_h : 32; /* 96 - 127 */ - }; - __u64 high; +#ifdef CONFIG_IRQ_REMAP + __u128 irte; +#endif }; }; From patchwork Thu Feb 2 14:50:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FACFC636D4 for ; Thu, 2 Feb 2023 15:29:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE50C6B0080; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EBF656B0081; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA2E26B0087; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B054A6B0080 for ; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 90E7B407D0 for ; Thu, 2 Feb 2023 15:29:23 +0000 (UTC) X-FDA: 80422735806.07.D5715DE Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf14.hostedemail.com (Postfix) with ESMTP id A2EF4100002 for ; Thu, 2 Feb 2023 15:29:21 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=AdjAk7Aa; dmarc=none; spf=none (imf14.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351761; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=7LJNI547mLVzITOTkRSnwSNDR+QIZCwHcxn6Kje3Jnw=; b=dBAnMWCEweDMMkk/4sIB211WsuUEuaI08ES0mW57mmJI1wuYpXGyD2XTRomRcr8BOPTYnN FiVxevciEvYkhRY/4YBzGNmAZFRlRF/Rjrfdhh39hN/d/frKDeM549VRqwQliJf3nxQLRl eO2FCvwIR/T7d07qhoTqnnOijyHagu0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=AdjAk7Aa; dmarc=none; spf=none (imf14.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351761; a=rsa-sha256; cv=none; b=gciLoyh4cHpRrqkgNd0Kx0fcPHznacHy2HcLELB5ZlqKAHvlY0OfRiwVhYVhiHLEj104fv QHobxxrJW/BbDjzVXgoaDzaCtpnvqUEwWX4ohakDxVnc6NoYUVbYKpOovnrVJ6h7d2T6Zc 3H/4S/MPVMcXc0HRCd98cK9HOcnzg1M= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=7LJNI547mLVzITOTkRSnwSNDR+QIZCwHcxn6Kje3Jnw=; b=AdjAk7AaBkcmzPYhGVosKJ4+8w u/+3KJ3kuOmQbqc61ESwTvAmbfHPi718PrDB2flKC9stiZKyv18e2HTQyWjiVkWMeOeDp5NoedT1J eUX68mxfRgV+2TpoaAtsxJeHXnu4ejKq1Bvk1SQzrpe+C6ULIcHywXaHeVhYfrVvLGlyvlVXdXpZz S997RqQxKM/SoFP56GFQGn7Hblp3oRfm9L+xg+MlrA2xY4hAdiDAL48fv1e5OOXC1nQKTC1DYiP+E Hm3CXQ2CsqSNxONoQX/0o4ADP1mXkr7enqw6PdLqxv4y9pqZDHqr9t2B7lQ2niSKD6P6KKByFA/E7 MwD9ufNg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVU-005CFy-21; Thu, 02 Feb 2023 15:28:12 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3ADEF30012F; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 9FEF623F31FBE; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.684926740@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:38 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 08/10] slub: Replace cmpxchg_double() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A2EF4100002 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: rk4358kbegipagieiyyas5ruik1chr1e X-HE-Tag: 1675351761-148468 X-HE-Meta: U2FsdGVkX1/VaGFQQ6sukHR1hsLZW/Z05htHLUMY70KfJ+tZe2Hnkern2N1XHOlIuX+HC1wL0h/ySKWzcKSZelQzGqeLzhofFWya/W7fkdUF7hIp4s8YQc8NZyYC+TBNKJXOU0NaeW1pxEJVXeJ2cGfdHOuwdLPUT67rqeoFOp/uIcb8GMRKid4Te5Nt3DgZCRn4NpHZ260p7VZarQVixocjUaBWyXoQEgdKoWD5sCT8VzogbG6CLgVjL/SNC71AG0Aw7Rbha0P+3lqLEEGrREtHYZmCwEqY1SUbkeA3dmh0wDtLHNj9szkTzQdi5ENKu18BhHzpjY8i1TddjjJnVfQ7Ap6L1ZZB6KF4YP6QiWxbqn2sxNKz9ff/RoLKICylMVUCl74Z/9cscXaOj6wVkTUXP3nOETASqIYIvKlJGxZjb4piT3dSU9MLQPJV7niuD5AQ9C+NtEk0WfqsF2kDBBDknvg2XL4RIwzwOKfo2hczMVCuiieFEMB/JWgpMZYMJD3YokU+UOkM4mFkxJ0v34QISstfAN2/ahIf2YZRN+Rx4/g/Ix2iefCDOyrQ6/QVgFNqeKVCdfu74C4KyOelmZ0s64+j5sfnNeV/HRNkXgcUjokS1eDsdZQnkPTVF0YSHOeic8jUoSZZAgjPIu6QP7gTmmD4DPkrdi3yl60bbihwkeUP1oOWEOmRJwdbuwfmGfI/uEPWp/LMPO9qaYhbSceNNTiPE4yKCtU17tx4VQXSG5wMr+gedZkjvlq7NXW8gUuafNtCMDWw4ucd0wT3Cof2imGSrMf9/67VX9f6nzIWUrQ83UrVl5XbQvAfVpNxKNdNT22J0IbDLQFnjNK5zuGfkDeLbSF/eXyILNAiD/DZgEFjWQ6yTq+/RUk0I79BM8YYhuvm7/GAwfcV78+GOQCwxgmAlztZoNJQSM34+1Fie7WJYWZsVhku5DG7u2i8B/+XE2U9oB2RXwC5lGX 1fIkKf7W e8wqtb3KsBU8DMQOx7zsYji+ib7JAZjfMLBm1f8OZ/KNpo60NdapkS7WA6OYj3x3XnrWipV4f5G/DdAEbx3QPGUg877s+Cg5fcLEtN339qh0lNliG05jqOuCmdCA2rcdkULEs+ioRJBy/hwtdnR3b5UvzJ480yqE0YlDsVEp5RkEsdxHULiAzdR+6g085tkip8nnIjhJdTXeKR6KnahVHJE4IAwbEKNnGY+saEw9EtRl8UJY7so8vXZmnISEdkcDUMTlY3NVI02kqwBMkGY7Ove9WI9Vqk+Hd+udr15NuVQ9vPHxM4dh9BPZ6AKfJ7dniEBVPAqVZ4R8WuVL+Abl+8L3oBjR5dWY0zQ2zNijUPi4/4lLaiALmbfAeDsq5GBEpUSMSDnPheAMZUVanbp/SJ+c51NTb9loTrWf0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) Acked-by: Vlastimil Babka Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slub_def.h | 12 ++- mm/slab.h | 45 +++++++++++++- mm/slub.c | 142 ++++++++++++++++++++++++++++------------------- 3 files changed, 135 insertions(+), 64 deletions(-) --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -39,7 +39,8 @@ enum stat_item { CPU_PARTIAL_FREE, /* Refill cpu partial on free */ CPU_PARTIAL_NODE, /* Refill cpu partial from node partial */ CPU_PARTIAL_DRAIN, /* Drain cpu partial to node partial */ - NR_SLUB_STAT_ITEMS }; + NR_SLUB_STAT_ITEMS +}; #ifndef CONFIG_SLUB_TINY /* @@ -47,8 +48,13 @@ enum stat_item { * with this_cpu_cmpxchg_double() alignment requirements. */ struct kmem_cache_cpu { - void **freelist; /* Pointer to next available object */ - unsigned long tid; /* Globally unique transaction id */ + union { + struct { + void **freelist; /* Pointer to next available object */ + unsigned long tid; /* Globally unique transaction id */ + }; + freelist_aba_t freelist_tid; + }; struct slab *slab; /* The slab from which we are allocating */ #ifdef CONFIG_SLUB_CPU_PARTIAL struct slab *partial; /* Partially allocated frozen slabs */ --- a/mm/slab.h +++ b/mm/slab.h @@ -5,6 +5,34 @@ * Internal slab definitions */ +/* + * Freelist pointer and counter to cmpxchg together, avoids the typical ABA + * problems with cmpxchg of just a pointer. + */ +typedef union { + struct { + void *freelist; + unsigned long counter; + }; +#ifdef CONFIG_64BIT + u128 full; +#else + u64 full; +#endif +} freelist_aba_t; + +#ifdef CONFIG_64BIT +# ifdef system_has_cmpxchg128 +# define system_has_freelist_aba() system_has_cmpxchg128() +# define try_cmpxchg_freelist try_cmpxchg128 +# endif +#else /* CONFIG_64BIT */ +# ifdef system_has_cmpxchg64 +# define system_has_freelist_aba() system_has_cmpxchg64() +# define try_cmpxchg_freelist try_cmpxchg64 +# endif +#endif /* CONFIG_64BIT */ + /* Reuses the bits in struct page */ struct slab { unsigned long __page_flags; @@ -37,14 +65,21 @@ struct slab { #endif }; /* Double-word boundary */ - void *freelist; /* first free object */ union { - unsigned long counters; struct { - unsigned inuse:16; - unsigned objects:15; - unsigned frozen:1; + void *freelist; /* first free object */ + union { + unsigned long counters; + struct { + unsigned inuse:16; + unsigned objects:15; + unsigned frozen:1; + }; + }; }; +#ifdef system_has_freelist_aba + freelist_aba_t freelist_counter; +#endif }; }; struct rcu_head rcu_head; --- a/mm/slub.c +++ b/mm/slub.c @@ -292,7 +292,13 @@ static inline bool kmem_cache_has_cpu_pa /* Poison object */ #define __OBJECT_POISON ((slab_flags_t __force)0x80000000U) /* Use cmpxchg_double */ + +#if defined(system_has_freelist_aba) && \ + defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) #define __CMPXCHG_DOUBLE ((slab_flags_t __force)0x40000000U) +#else +#define __CMPXCHG_DOUBLE ((slab_flags_t __force)0U) +#endif /* * Tracking user of a slab. @@ -512,6 +518,43 @@ static __always_inline void slab_unlock( __bit_spin_unlock(PG_locked, &page->flags); } +static inline bool +__update_freelist_fast(struct slab *slab, + void *freelist_old, unsigned long counters_old, + void *freelist_new, unsigned long counters_new) +{ + + bool ret = false; + +#ifdef system_has_freelist_aba + freelist_aba_t old = { .freelist = freelist_old, .counter = counters_old }; + freelist_aba_t new = { .freelist = freelist_new, .counter = counters_new }; + + ret = try_cmpxchg_freelist(&slab->freelist_counter.full, &old.full, new.full); +#endif /* system_has_freelist_aba */ + + return ret; +} + +static inline bool +__update_freelist_slow(struct slab *slab, + void *freelist_old, unsigned long counters_old, + void *freelist_new, unsigned long counters_new) +{ + bool ret = false; + + slab_lock(slab); + if (slab->freelist == freelist_old && + slab->counters == counters_old) { + slab->freelist = freelist_new; + slab->counters = counters_new; + ret = true; + } + slab_unlock(slab); + + return ret; +} + /* * Interrupts must be disabled (for the fallback code to work right), typically * by an _irqsave() lock variant. On PREEMPT_RT the preempt_disable(), which is @@ -519,33 +562,25 @@ static __always_inline void slab_unlock( * allocation/ free operation in hardirq context. Therefore nothing can * interrupt the operation. */ -static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab, +static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slab, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, const char *n) { + bool ret; + if (USE_LOCKLESS_FAST_PATH()) lockdep_assert_irqs_disabled(); -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ - defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) + if (s->flags & __CMPXCHG_DOUBLE) { - if (cmpxchg_double(&slab->freelist, &slab->counters, - freelist_old, counters_old, - freelist_new, counters_new)) - return true; - } else -#endif - { - slab_lock(slab); - if (slab->freelist == freelist_old && - slab->counters == counters_old) { - slab->freelist = freelist_new; - slab->counters = counters_new; - slab_unlock(slab); - return true; - } - slab_unlock(slab); + ret = __update_freelist_fast(slab, freelist_old, counters_old, + freelist_new, counters_new); + } else { + ret = __update_freelist_slow(slab, freelist_old, counters_old, + freelist_new, counters_new); } + if (likely(ret)) + return true; cpu_relax(); stat(s, CMPXCHG_DOUBLE_FAIL); @@ -557,36 +592,26 @@ static inline bool __cmpxchg_double_slab return false; } -static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab, +static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, const char *n) { -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ - defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) + bool ret; + if (s->flags & __CMPXCHG_DOUBLE) { - if (cmpxchg_double(&slab->freelist, &slab->counters, - freelist_old, counters_old, - freelist_new, counters_new)) - return true; - } else -#endif - { + ret = __update_freelist_fast(slab, freelist_old, counters_old, + freelist_new, counters_new); + } else { unsigned long flags; local_irq_save(flags); - slab_lock(slab); - if (slab->freelist == freelist_old && - slab->counters == counters_old) { - slab->freelist = freelist_new; - slab->counters = counters_new; - slab_unlock(slab); - local_irq_restore(flags); - return true; - } - slab_unlock(slab); + ret = __update_freelist_slow(slab, freelist_old, counters_old, + freelist_new, counters_new); local_irq_restore(flags); } + if (likely(ret)) + return true; cpu_relax(); stat(s, CMPXCHG_DOUBLE_FAIL); @@ -2229,7 +2254,7 @@ static inline void *acquire_slab(struct VM_BUG_ON(new.frozen); new.frozen = 1; - if (!__cmpxchg_double_slab(s, slab, + if (!__slab_update_freelist(s, slab, freelist, counters, new.freelist, new.counters, "acquire_slab")) @@ -2555,7 +2580,7 @@ static void deactivate_slab(struct kmem_ } - if (!cmpxchg_double_slab(s, slab, + if (!slab_update_freelist(s, slab, old.freelist, old.counters, new.freelist, new.counters, "unfreezing slab")) { @@ -2612,7 +2637,7 @@ static void __unfreeze_partials(struct k new.frozen = 0; - } while (!__cmpxchg_double_slab(s, slab, + } while (!__slab_update_freelist(s, slab, old.freelist, old.counters, new.freelist, new.counters, "unfreezing slab")); @@ -3009,6 +3034,18 @@ static inline bool pfmemalloc_match(stru } #ifndef CONFIG_SLUB_TINY +static inline bool +__update_cpu_freelist_fast(struct kmem_cache *s, + void *freelist_old, void *freelist_new, + unsigned long tid) +{ + freelist_aba_t old = { .freelist = freelist_old, .counter = tid }; + freelist_aba_t new = { .freelist = freelist_new, .counter = next_tid(tid) }; + + return this_cpu_cmpxchg(s->cpu_slab->freelist_tid.full, + old.full, new.full) == old.full; +} + /* * Check the slab->freelist and either transfer the freelist to the * per cpu freelist or deactivate the slab. @@ -3035,7 +3072,7 @@ static inline void *get_freelist(struct new.inuse = slab->objects; new.frozen = freelist != NULL; - } while (!__cmpxchg_double_slab(s, slab, + } while (!__slab_update_freelist(s, slab, freelist, counters, NULL, new.counters, "get_freelist")); @@ -3360,11 +3397,7 @@ static __always_inline void *__slab_allo * against code executing on this cpu *not* from access by * other cpus. */ - if (unlikely(!this_cpu_cmpxchg_double( - s->cpu_slab->freelist, s->cpu_slab->tid, - object, tid, - next_object, next_tid(tid)))) { - + if (unlikely(!__update_cpu_freelist_fast(s, object, next_object, tid))) { note_cmpxchg_failure("slab_alloc", s, tid); goto redo; } @@ -3632,7 +3665,7 @@ static void __slab_free(struct kmem_cach } } - } while (!cmpxchg_double_slab(s, slab, + } while (!slab_update_freelist(s, slab, prior, counters, head, new.counters, "__slab_free")); @@ -3737,11 +3770,7 @@ static __always_inline void do_slab_free set_freepointer(s, tail_obj, freelist); - if (unlikely(!this_cpu_cmpxchg_double( - s->cpu_slab->freelist, s->cpu_slab->tid, - freelist, tid, - head, next_tid(tid)))) { - + if (unlikely(!__update_cpu_freelist_fast(s, freelist, head, tid))) { note_cmpxchg_failure("slab_free", s, tid); goto redo; } @@ -4505,11 +4534,12 @@ static int kmem_cache_open(struct kmem_c } } -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ +#if defined(system_has_freelist_aba) && \ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) - if (system_has_cmpxchg_double() && (s->flags & SLAB_NO_CMPXCHG) == 0) + if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) { /* Enable fast mode */ s->flags |= __CMPXCHG_DOUBLE; + } #endif /* From patchwork Thu Feb 2 14:50:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13126379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C7E3C636D6 for ; Thu, 2 Feb 2023 15:29:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 918906B007E; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C98E6B0083; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56E256B0081; Thu, 2 Feb 2023 10:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 377646B0080 for ; Thu, 2 Feb 2023 10:29:23 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 868A5140259 for ; Thu, 2 Feb 2023 15:29:22 +0000 (UTC) X-FDA: 80422735764.19.300864E Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf03.hostedemail.com (Postfix) with ESMTP id 9244420010 for ; Thu, 2 Feb 2023 15:29:20 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=pZLBuYaO; dmarc=none; spf=none (imf03.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=rBSL+GYZTolG4BkmZCuUbB8ch62f+1wBaXoRcDgmADY=; b=MURwAO/mdD57QHkY/tVa8KmxA6YHyqNI6YKWOZVESkfiVVL4GjRlln7jH9PdHbAu55Fhyw WUWq74NOhTqIJ/nH1HNg3M1zRI5cZeQwwYCtC+pveRtGWCzSezW7a5w164xTGUQxonlTOB H+a5c2WTepPrT6YM0xUyUGJ5gz/YRnM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=pZLBuYaO; dmarc=none; spf=none (imf03.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351760; a=rsa-sha256; cv=none; b=ufeZPQPJ9RFzAJgL9JMunaLZ1uJTcCuVubvyIvyRB4yW1D1Jabg42yLuC/F/HqN3t9Ddjb I5jHEuZaMtEnITqi51OjlFbPjLof0P4D8BJ3SE0mB+Ef6KOMbNsKQIT9dS+wOhzJtmZMdJ JPSN0tBr44fwBnPwL0q70E5dWpczMJg= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=rBSL+GYZTolG4BkmZCuUbB8ch62f+1wBaXoRcDgmADY=; b=pZLBuYaOOcLCZpIZXdbs4/z29l Ru65O+4O60MMvGftzBQo2laIfm5ZNV8OQYEfOja4QTivVl2/qVEBR6Z3fXkif6LktnEODVvHNAk59 1TpHqmiet2vnoTra7lOMUZ7cSDwQ1QoHfxLsofpzRfE62RBG054ijogDuNUT8zn3qBpTXt1DZzVuk uAg28JmhgjkN9Bwp8ZquABnCm4cFVfBiJcrhKMCiCYsyfVLHb7sryMr+s0EM0Ud0zeqYPRGRYrRUh 3oZO43uOB8Xyo9F9eVtNpxyW9elMVvD3FdtlgX7C5hMXnDihAVoywSnBmxJtQMRFvrpbWcXf1FAPT 3H3e8yKg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pNbVU-005CFx-22; Thu, 02 Feb 2023 15:28:13 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 3BA83302D60; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id A72B423F31FBF; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.746130134@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:39 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 09/10] arch: Remove cmpxchg_double References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9244420010 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xstfitpgper1ipdfodt4shott6rsjwri X-HE-Tag: 1675351760-119543 X-HE-Meta: U2FsdGVkX18gNo7kgsRpWeBKI98Q1QxgIJe3TstcoXf7KAJyui51HseeufgiH1rjsM1moGCqx+OOHeY9Ylj77qa71W3BQNHkkETBrsB650HT9mFjMYng/TTOIUZ/TYxFqHATbS8nbJALgVGC6TtCkba/XoGKbwg7U/VxV4V4e0DSXOToRoriDJE6T62U8vaVtyLyCX2JkZNd0n2KcomcfVUzN/hIjrXV+W1c8TDdu9c395pMIwV+fNYuxsMq3vOX+hhhSItK/81KqzQXkfcbPnxY8zu466eXmeTnJyMv5rr8Nh5VoXcs5nTIMAUMhlbJVLhNKFjscPscdRnN+W8ib3Dto2EiqrhjowzmhNxOGRULce5fzNk9in/bZPZOfEUkIKjF58yO/fLWw2qfuOJ1OXVHAmD9SJNfZx3MDGf77i3yIFWyqV2MRFky7+8hNF2dUV4VJYeWkWgYhNjrNEUt6ASgfZoZbp33IKxgMQkAUyR7DZ+RqqXhNBJ+BvJf2FqUiEuuiI8XIakWMHrEDH3IA/Bq2HWDF6VeiVuBYR/n4yNgby0/JqX6W1DH7HPs/LskLXRJYyOVaDfH0p2161Ec6PlNKgu4LHsTKahFhuBz8WoXRbRpXAfK8PGTcd22LUP+CmowG096/aSssNPQm+JizcPyagP2ctY+IEDmUID0b2oHmQhVKlmyWQ3lv87tt6h+DYrsM0lFF+KXO1oj8OC1uyDKfhY6wulQz1b4uA/Na4Oc1NPxnW9wcnpuJWCfgS3f23NPyzYSfemKWWwaJNhgp37wcOcQ8M2ddRGrkhpizerpsYjuayOLU+haeX2oJ5yOW8y6Pt0tMt9xxQfeFQnmoVbtSiRbraIQ4h3LMLQr+Tl7a+ZvIEm9z/UbcIIgakyItJyEJqppg0w3OHUuvXtl8uE9YZ4lPVMmLjY766qab6rfOni365WqikZSppR5qIt2F6zd67CVCCga3qSmiz3 1cbBjqiu Xgbl8t+WJU1yRD3Jfs3+3XtJnravSSXeekA2xGwehGZTcbRmFaNPYBtGB8X+7zRuW7m/XJShfkxLgZjVlzNR4GKD+bIlLff7Q652bZRFY0u98GDwUOoGiqiRZBYtHuMOx/ugQbge7q7aP44KrIFcDkLkH2Cz5O9x5CzhSomnN6tN1j3uOowLzQoXgQewweaV4SZje2SnJDXpBnNlsU5t4rjIKkn4GvedjkeDFGqMonNBwyDrTTvXI3iLWVDTPpLvrXQJz1w3BiJEkYE0Gba+q1TL3KcLYzIDXBJifOztsUvyOsdRsC9cRztw4giC2IQbfFH4/fN2qgQ9ZWvDPr6qRQG4k9/4XHJZ6TsgyvLor9r2IFFHtdSJxDWveRDdgcJhrFPDK2Ax7LOjgiHLUAk0zphSBWihXSEMiZbO9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No moar users, remove the monster. Signed-off-by: Peter Zijlstra (Intel) --- Documentation/core-api/this_cpu_ops.rst | 2 - arch/arm64/include/asm/atomic_ll_sc.h | 33 ---------------- arch/arm64/include/asm/atomic_lse.h | 36 ------------------ arch/arm64/include/asm/cmpxchg.h | 46 ----------------------- arch/arm64/include/asm/percpu.h | 10 ----- arch/s390/include/asm/cmpxchg.h | 34 ----------------- arch/s390/include/asm/percpu.h | 18 --------- arch/x86/include/asm/cmpxchg.h | 25 ------------ arch/x86/include/asm/cmpxchg_32.h | 1 arch/x86/include/asm/cmpxchg_64.h | 1 arch/x86/include/asm/percpu.h | 41 -------------------- include/asm-generic/percpu.h | 58 ----------------------------- include/linux/atomic/atomic-instrumented.h | 17 -------- include/linux/percpu-defs.h | 38 ------------------- scripts/atomic/gen-atomic-instrumented.sh | 17 ++------ 15 files changed, 6 insertions(+), 371 deletions(-) --- a/Documentation/core-api/this_cpu_ops.rst +++ b/Documentation/core-api/this_cpu_ops.rst @@ -53,7 +53,6 @@ are defined. These operations can be use this_cpu_add_return(pcp, val) this_cpu_xchg(pcp, nval) this_cpu_cmpxchg(pcp, oval, nval) - this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) this_cpu_sub(pcp, val) this_cpu_inc(pcp) this_cpu_dec(pcp) @@ -242,7 +241,6 @@ modifies the variable, then RMW actions __this_cpu_add_return(pcp, val) __this_cpu_xchg(pcp, nval) __this_cpu_cmpxchg(pcp, oval, nval) - __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) __this_cpu_sub(pcp, val) __this_cpu_inc(pcp) __this_cpu_dec(pcp) --- a/arch/arm64/include/asm/atomic_ll_sc.h +++ b/arch/arm64/include/asm/atomic_ll_sc.h @@ -294,39 +294,6 @@ __CMPXCHG_CASE( , , mb_, 64, dmb ish, #undef __CMPXCHG_CASE -#define __CMPXCHG_DBL(name, mb, rel, cl) \ -static __always_inline long \ -__ll_sc__cmpxchg_double##name(unsigned long old1, \ - unsigned long old2, \ - unsigned long new1, \ - unsigned long new2, \ - volatile void *ptr) \ -{ \ - unsigned long tmp, ret; \ - \ - asm volatile("// __cmpxchg_double" #name "\n" \ - " prfm pstl1strm, %2\n" \ - "1: ldxp %0, %1, %2\n" \ - " eor %0, %0, %3\n" \ - " eor %1, %1, %4\n" \ - " orr %1, %0, %1\n" \ - " cbnz %1, 2f\n" \ - " st" #rel "xp %w0, %5, %6, %2\n" \ - " cbnz %w0, 1b\n" \ - " " #mb "\n" \ - "2:" \ - : "=&r" (tmp), "=&r" (ret), "+Q" (*(__uint128_t *)ptr) \ - : "r" (old1), "r" (old2), "r" (new1), "r" (new2) \ - : cl); \ - \ - return ret; \ -} - -__CMPXCHG_DBL( , , , ) -__CMPXCHG_DBL(_mb, dmb ish, l, "memory") - -#undef __CMPXCHG_DBL - union __u128_halves { u128 full; struct { --- a/arch/arm64/include/asm/atomic_lse.h +++ b/arch/arm64/include/asm/atomic_lse.h @@ -288,42 +288,6 @@ __CMPXCHG_CASE(x, , mb_, 64, al, "memo #undef __CMPXCHG_CASE -#define __CMPXCHG_DBL(name, mb, cl...) \ -static __always_inline long \ -__lse__cmpxchg_double##name(unsigned long old1, \ - unsigned long old2, \ - unsigned long new1, \ - unsigned long new2, \ - volatile void *ptr) \ -{ \ - unsigned long oldval1 = old1; \ - unsigned long oldval2 = old2; \ - register unsigned long x0 asm ("x0") = old1; \ - register unsigned long x1 asm ("x1") = old2; \ - register unsigned long x2 asm ("x2") = new1; \ - register unsigned long x3 asm ("x3") = new2; \ - register unsigned long x4 asm ("x4") = (unsigned long)ptr; \ - \ - asm volatile( \ - __LSE_PREAMBLE \ - " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\ - " eor %[old1], %[old1], %[oldval1]\n" \ - " eor %[old2], %[old2], %[oldval2]\n" \ - " orr %[old1], %[old1], %[old2]" \ - : [old1] "+&r" (x0), [old2] "+&r" (x1), \ - [v] "+Q" (*(__uint128_t *)ptr) \ - : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \ - [oldval1] "r" (oldval1), [oldval2] "r" (oldval2) \ - : cl); \ - \ - return x0; \ -} - -__CMPXCHG_DBL( , ) -__CMPXCHG_DBL(_mb, al, "memory") - -#undef __CMPXCHG_DBL - #define __CMPXCHG128(name, mb, cl...) \ static __always_inline u128 \ __lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -131,22 +131,6 @@ __CMPXCHG_CASE(mb_, 64) #undef __CMPXCHG_CASE -#define __CMPXCHG_DBL(name) \ -static inline long __cmpxchg_double##name(unsigned long old1, \ - unsigned long old2, \ - unsigned long new1, \ - unsigned long new2, \ - volatile void *ptr) \ -{ \ - return __lse_ll_sc_body(_cmpxchg_double##name, \ - old1, old2, new1, new2, ptr); \ -} - -__CMPXCHG_DBL( ) -__CMPXCHG_DBL(_mb) - -#undef __CMPXCHG_DBL - #define __CMPXCHG128(name) \ static inline u128 __cmpxchg128##name(volatile u128 *ptr, \ u128 old, u128 new) \ @@ -212,36 +196,6 @@ __CMPXCHG_GEN(_mb) #define arch_cmpxchg64 arch_cmpxchg #define arch_cmpxchg64_local arch_cmpxchg_local -/* cmpxchg_double */ -#define system_has_cmpxchg_double() 1 - -#define __cmpxchg_double_check(ptr1, ptr2) \ -({ \ - if (sizeof(*(ptr1)) != 8) \ - BUILD_BUG(); \ - VM_BUG_ON((unsigned long *)(ptr2) - (unsigned long *)(ptr1) != 1); \ -}) - -#define arch_cmpxchg_double(ptr1, ptr2, o1, o2, n1, n2) \ -({ \ - int __ret; \ - __cmpxchg_double_check(ptr1, ptr2); \ - __ret = !__cmpxchg_double_mb((unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2), \ - ptr1); \ - __ret; \ -}) - -#define arch_cmpxchg_double_local(ptr1, ptr2, o1, o2, n1, n2) \ -({ \ - int __ret; \ - __cmpxchg_double_check(ptr1, ptr2); \ - __ret = !__cmpxchg_double((unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2), \ - ptr1); \ - __ret; \ -}) - /* cmpxchg128 */ #define system_has_cmpxchg128() 1 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -145,16 +145,6 @@ PERCPU_RET_OP(add, add, ldadd) * preemption point when TIF_NEED_RESCHED gets set while preemption is * disabled. */ -#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ -({ \ - int __ret; \ - preempt_disable_notrace(); \ - __ret = cmpxchg_double_local( raw_cpu_ptr(&(ptr1)), \ - raw_cpu_ptr(&(ptr2)), \ - o1, o2, n1, n2); \ - preempt_enable_notrace(); \ - __ret; \ -}) #define _pcp_protect(op, pcp, ...) \ ({ \ --- a/arch/s390/include/asm/cmpxchg.h +++ b/arch/s390/include/asm/cmpxchg.h @@ -167,40 +167,6 @@ static __always_inline unsigned long __c #define arch_cmpxchg_local arch_cmpxchg #define arch_cmpxchg64_local arch_cmpxchg -#define system_has_cmpxchg_double() 1 - -static __always_inline int __cmpxchg_double(unsigned long p1, unsigned long p2, - unsigned long o1, unsigned long o2, - unsigned long n1, unsigned long n2) -{ - union register_pair old = { .even = o1, .odd = o2, }; - union register_pair new = { .even = n1, .odd = n2, }; - int cc; - - asm volatile( - " cdsg %[old],%[new],%[ptr]\n" - " ipm %[cc]\n" - " srl %[cc],28\n" - : [cc] "=&d" (cc), [old] "+&d" (old.pair) - : [new] "d" (new.pair), - [ptr] "QS" (*(unsigned long *)p1), "Q" (*(unsigned long *)p2) - : "memory", "cc"); - return !cc; -} - -#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ -({ \ - typeof(p1) __p1 = (p1); \ - typeof(p2) __p2 = (p2); \ - \ - BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \ - BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \ - VM_BUG_ON((unsigned long)((__p1) + 1) != (unsigned long)(__p2));\ - __cmpxchg_double((unsigned long)__p1, (unsigned long)__p2, \ - (unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2)); \ -}) - #define system_has_cmpxchg128() 1 static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -181,24 +181,6 @@ #define this_cpu_xchg_4(pcp, nval) arch_this_cpu_xchg(pcp, nval) #define this_cpu_xchg_8(pcp, nval) arch_this_cpu_xchg(pcp, nval) -#define arch_this_cpu_cmpxchg_double(pcp1, pcp2, o1, o2, n1, n2) \ -({ \ - typeof(pcp1) *p1__; \ - typeof(pcp2) *p2__; \ - int ret__; \ - \ - preempt_disable_notrace(); \ - p1__ = raw_cpu_ptr(&(pcp1)); \ - p2__ = raw_cpu_ptr(&(pcp2)); \ - ret__ = __cmpxchg_double((unsigned long)p1__, (unsigned long)p2__, \ - (unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2)); \ - preempt_enable_notrace(); \ - ret__; \ -}) - -#define this_cpu_cmpxchg_double_8 arch_this_cpu_cmpxchg_double - #include #endif /* __ARCH_S390_PERCPU__ */ --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -233,29 +233,4 @@ extern void __add_wrong_size(void) #define __xadd(ptr, inc, lock) __xchg_op((ptr), (inc), xadd, lock) #define xadd(ptr, inc) __xadd((ptr), (inc), LOCK_PREFIX) -#define __cmpxchg_double(pfx, p1, p2, o1, o2, n1, n2) \ -({ \ - bool __ret; \ - __typeof__(*(p1)) __old1 = (o1), __new1 = (n1); \ - __typeof__(*(p2)) __old2 = (o2), __new2 = (n2); \ - BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \ - BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \ - VM_BUG_ON((unsigned long)(p1) % (2 * sizeof(long))); \ - VM_BUG_ON((unsigned long)((p1) + 1) != (unsigned long)(p2)); \ - asm volatile(pfx "cmpxchg%c5b %1" \ - CC_SET(e) \ - : CC_OUT(e) (__ret), \ - "+m" (*(p1)), "+m" (*(p2)), \ - "+a" (__old1), "+d" (__old2) \ - : "i" (2 * sizeof(long)), \ - "b" (__new1), "c" (__new2)); \ - __ret; \ -}) - -#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ - __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) - -#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ - __cmpxchg_double(, p1, p2, o1, o2, n1, n2) - #endif /* ASM_X86_CMPXCHG_H */ --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -103,7 +103,6 @@ static inline bool __try_cmpxchg64(volat #endif -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) #define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8) #endif /* _ASM_X86_CMPXCHG_32_H */ --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -72,7 +72,6 @@ static __always_inline bool arch_try_cmp return likely(ret); } -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) #define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16) #endif /* _ASM_X86_CMPXCHG_64_H */ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -346,23 +346,6 @@ do { \ #define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) #define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) -#ifdef CONFIG_X86_CMPXCHG64 -#define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ -({ \ - bool __ret; \ - typeof(pcp1) __o1 = (o1), __n1 = (n1); \ - typeof(pcp2) __o2 = (o2), __n2 = (n2); \ - asm volatile("cmpxchg8b "__percpu_arg(1) \ - CC_SET(z) \ - : CC_OUT(z) (__ret), "+m" (pcp1), "+m" (pcp2), "+a" (__o1), "+d" (__o2) \ - : "b" (__n1), "c" (__n2)); \ - __ret; \ -}) - -#define raw_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double -#define this_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double -#endif /* CONFIG_X86_CMPXCHG64 */ - /* * Per cpu atomic 64 bit operations are only available under 64 bit. * 32 bit must fall back to generic operations. @@ -385,30 +368,6 @@ do { \ #define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) #define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) - -/* - * Pretty complex macro to generate cmpxchg16 instruction. The instruction - * is not supported on early AMD64 processors so we must be able to emulate - * it in software. The address used in the cmpxchg16 instruction must be - * aligned to a 16 byte boundary. - */ -#define percpu_cmpxchg16b_double(pcp1, pcp2, o1, o2, n1, n2) \ -({ \ - bool __ret; \ - typeof(pcp1) __o1 = (o1), __n1 = (n1); \ - typeof(pcp2) __o2 = (o2), __n2 = (n2); \ - alternative_io("leaq %P1,%%rsi\n\tcall this_cpu_cmpxchg16b_emu\n\t", \ - "cmpxchg16b " __percpu_arg(1) "\n\tsetz %0\n\t", \ - X86_FEATURE_CX16, \ - ASM_OUTPUT2("=a" (__ret), "+m" (pcp1), \ - "+m" (pcp2), "+d" (__o2)), \ - "b" (__n1), "c" (__n2), "a" (__o1) : "rsi"); \ - __ret; \ -}) - -#define raw_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double -#define this_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double - #endif static __always_inline bool x86_this_cpu_constant_test_bit(unsigned int nr, --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -99,19 +99,6 @@ do { \ __ret; \ }) -#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ -({ \ - typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \ - typeof(pcp2) *__p2 = raw_cpu_ptr(&(pcp2)); \ - int __ret = 0; \ - if (*__p1 == (oval1) && *__p2 == (oval2)) { \ - *__p1 = nval1; \ - *__p2 = nval2; \ - __ret = 1; \ - } \ - (__ret); \ -}) - #define __this_cpu_generic_read_nopreempt(pcp) \ ({ \ typeof(pcp) ___ret; \ @@ -180,17 +167,6 @@ do { \ __ret; \ }) -#define this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ -({ \ - int __ret; \ - unsigned long __flags; \ - raw_local_irq_save(__flags); \ - __ret = raw_cpu_generic_cmpxchg_double(pcp1, pcp2, \ - oval1, oval2, nval1, nval2); \ - raw_local_irq_restore(__flags); \ - __ret; \ -}) - #ifndef raw_cpu_read_1 #define raw_cpu_read_1(pcp) raw_cpu_generic_read(pcp) #endif @@ -303,23 +279,6 @@ do { \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif -#ifndef raw_cpu_cmpxchg_double_1 -#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef raw_cpu_cmpxchg_double_2 -#define raw_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef raw_cpu_cmpxchg_double_4 -#define raw_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef raw_cpu_cmpxchg_double_8 -#define raw_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif - #ifndef this_cpu_read_1 #define this_cpu_read_1(pcp) this_cpu_generic_read(pcp) #endif @@ -432,21 +391,4 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif -#ifndef this_cpu_cmpxchg_double_1 -#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef this_cpu_cmpxchg_double_2 -#define this_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef this_cpu_cmpxchg_double_4 -#define this_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef this_cpu_cmpxchg_double_8 -#define this_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif - #endif /* _ASM_GENERIC_PERCPU_H_ */ --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -2141,21 +2141,6 @@ atomic_long_dec_if_positive(atomic_long_ arch_sync_cmpxchg(__ai_ptr, __VA_ARGS__); \ }) -#define cmpxchg_double(ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - kcsan_mb(); \ - instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ - arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \ -}) - - -#define cmpxchg_double_local(ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ - arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \ -}) #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 27320c1ec2bf2878ecb9df3ea4816a7bc0c57a52 +// 416a741acbd4d28dbfa45f1b2a2c1b714454229f --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -359,33 +359,6 @@ static inline void __this_cpu_preempt_ch pscr2_ret__; \ }) -/* - * Special handling for cmpxchg_double. cmpxchg_double is passed two - * percpu variables. The first has to be aligned to a double word - * boundary and the second has to follow directly thereafter. - * We enforce this on all architectures even if they don't support - * a double cmpxchg instruction, since it's a cheap requirement, and it - * avoids breaking the requirement for architectures with the instruction. - */ -#define __pcpu_double_call_return_bool(stem, pcp1, pcp2, ...) \ -({ \ - bool pdcrb_ret__; \ - __verify_pcpu_ptr(&(pcp1)); \ - BUILD_BUG_ON(sizeof(pcp1) != sizeof(pcp2)); \ - VM_BUG_ON((unsigned long)(&(pcp1)) % (2 * sizeof(pcp1))); \ - VM_BUG_ON((unsigned long)(&(pcp2)) != \ - (unsigned long)(&(pcp1)) + sizeof(pcp1)); \ - switch(sizeof(pcp1)) { \ - case 1: pdcrb_ret__ = stem##1(pcp1, pcp2, __VA_ARGS__); break; \ - case 2: pdcrb_ret__ = stem##2(pcp1, pcp2, __VA_ARGS__); break; \ - case 4: pdcrb_ret__ = stem##4(pcp1, pcp2, __VA_ARGS__); break; \ - case 8: pdcrb_ret__ = stem##8(pcp1, pcp2, __VA_ARGS__); break; \ - default: \ - __bad_size_call_parameter(); break; \ - } \ - pdcrb_ret__; \ -}) - #define __pcpu_size_call(stem, variable, ...) \ do { \ __verify_pcpu_ptr(&(variable)); \ @@ -442,9 +415,6 @@ do { \ #define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval) #define raw_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size16_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) -#define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) - #define raw_cpu_sub(pcp, val) raw_cpu_add(pcp, -(val)) #define raw_cpu_inc(pcp) raw_cpu_add(pcp, 1) #define raw_cpu_dec(pcp) raw_cpu_sub(pcp, 1) @@ -504,11 +474,6 @@ do { \ raw_cpu_cmpxchg(pcp, oval, nval); \ }) -#define __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ -({ __this_cpu_preempt_check("cmpxchg_double"); \ - raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2); \ -}) - #define __this_cpu_sub(pcp, val) __this_cpu_add(pcp, -(typeof(pcp))(val)) #define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1) #define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1) @@ -529,9 +494,6 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size16_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) -#define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) - #define this_cpu_sub(pcp, val) this_cpu_add(pcp, -(typeof(pcp))(val)) #define this_cpu_inc(pcp) this_cpu_add(pcp, 1) #define this_cpu_dec(pcp) this_cpu_sub(pcp, 1) --- a/scripts/atomic/gen-atomic-instrumented.sh +++ b/scripts/atomic/gen-atomic-instrumented.sh @@ -84,7 +84,6 @@ gen_xchg() { local xchg="$1"; shift local order="$1"; shift - local mult="$1"; shift kcsan_barrier="" if [ "${xchg%_local}" = "${xchg}" ]; then @@ -104,8 +103,8 @@ cat < X-Patchwork-Id: 13126373 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE0BEC636D4 for ; Thu, 2 Feb 2023 15:29:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23A546B0074; Thu, 2 Feb 2023 10:29:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EA106B0075; Thu, 2 Feb 2023 10:29:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B2356B0078; Thu, 2 Feb 2023 10:29:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EEA656B0074 for ; Thu, 2 Feb 2023 10:29:18 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9FC44160757 for ; Thu, 2 Feb 2023 15:29:18 +0000 (UTC) X-FDA: 80422735596.24.77951A4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf20.hostedemail.com (Postfix) with ESMTP id E656E1C001D for ; Thu, 2 Feb 2023 15:29:16 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SutQsn5H; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675351757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=LNTskqo5roGsovH+JFQKAebOFNRW3Kky7MgwbIWmzaM=; b=BiDql02lj2zJzMGjitkewpZjBgC+2XJRMLvb3N573m96zLhNn/r7wd21hzUcFeE2qC7Vyg u0b1RqyfSdrfoKQVb+BbhvsUoCpTANoZdqYr66ereNP0lAvu4Dg7X0/QZ65q5SBBEIUSwn avAp9W3byxrPUv1UuQDhgGxGIwinCp0= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=SutQsn5H; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675351757; a=rsa-sha256; cv=none; b=HtqlBxXnUAiXlj6m43TlHCHSpJCgMRE/R2AJeA+Mn9IKl/Q1JxNxdHkKSlW/AA/0WJ0333 alaw5lGbt6RsylJvmoMbUl/0sUAqTo+/WjxFZaRbYUC07htBxULJQtynqjIUHu8pZ78wjD zLZlGMnaqp7UBUvtyboCkyObMJ0LRv0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=LNTskqo5roGsovH+JFQKAebOFNRW3Kky7MgwbIWmzaM=; b=SutQsn5HCDPd7RqYYJOqaRBAAw ghza+gVMb7VvZ8S5mXEBYv9fP8q8Y9wXALUK5q0pzocuZuyXrPepuT1+wCOd/OzB+ntZth6DaMT6B S8zOdGU0FmbWx9fMkVMCos7RUSpLps4sna/LO0IsKeshnyabCK7cNceYm4e+4Fmn11Z+jL+MTpNzN I6xv1Uhkv4ss9oJhXYo64MzEkqJxVqSn1hWToVBJlsAr9MjKCkQ2hdLEGzmvnX+l4DLFC6cT1hP4d 2/1/3QPc9YolaFPwoZcHS+Sn+GoiEBzfYylecr9r98dtv7EMTfckM+5a0Ow5UAFEBYeCLaBAG2mQU PcOP0H6w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pNbW4-00DV2e-2G; Thu, 02 Feb 2023 15:28:48 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4121B302E1F; Thu, 2 Feb 2023 16:28:45 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id AB09623F326C1; Thu, 2 Feb 2023 16:28:40 +0100 (CET) Message-ID: <20230202152655.805747571@infradead.org> User-Agent: quilt/0.66 Date: Thu, 02 Feb 2023 15:50:40 +0100 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v2 10/10] s390/cpum_sf: Convert to cmpxchg128() References: <20230202145030.223740842@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E656E1C001D X-Rspam-User: X-Stat-Signature: a5pcz34ef7u1smwipogkw78kg5yp5xpz X-HE-Tag: 1675351756-910153 X-HE-Meta: U2FsdGVkX1+9aBANX0vKYqNwrlIo/cXfdR59M0RHm+pOoM+hbpj/K83ofTgWfS+rPf0WkLvI5Qjq/V9t3ISYeXO7ZyPZXplu9GO2XdgSh6ISi6EHskD51E22efqCafYR3Mw7oPsiI3AshWuGYhwIzYej1u6iy9Z/W2/1lct4auqe0e8KzBpulqXEOdMparYbg9yL4So1tf6XBsqGlx1nUFDgS3I2J52QN0xSNl72XRJ29KZjw9YO99s5Tsl2V5t0gMuN19jiS5GopLWa9xBm0Hr4cX1oz9PhefUNExFl8y1Jr1XKYBfwCuUI166iytcoLhKpjpkgIUdSpSqSr9kbMM+suZ5aKepYnPumrcvJoWYfXSHsCxs/Y40YSTrRxbEF8ofvHpNNzdX3mdXhJHquU2YyyP/nv/VNhjAEtymeagzSdTVJ4MrlGx55baHxru9qZXoHinGgV+dBCduKLDn8VYPqgs8N9rS0WeoFoDhGNrNt+5Gsjh6VCq6stMqn3NYvchsi2bSVrGQEyp1hY2k4vSe1C2H1VpbIL66YejjqE5FsctPCZvOznt6j3p4wfinoeeZxjuHfXuSP4pNY3Zv3byYrX/kVhKnpDJWijvCPu2Jr5CJ4a182sL1LecV3H866geQ3qVL5o9ma7wTUUZRqFmBFcj54hhQMVpOQpvn958ljzLxPvcCfEe0Nsnbn4tAhqszd7Ym/RpQEI6X8t3Pl4wFry3dL6JvLVAYXu9CVE7OoZA7nkH7iO7Sp4gx+qd3po/sY84TtrOz63Tw1dEWHsEsC62CWNY6I2eDrxxLd2RkIr912cFZThv8bil+LHPg8+FCaUbJqfvB94Wrte6kQGsQSYg18NrRwGh06B95Ks6WI1gLfQZT1K/25uxbUD+29DMkEZsU3Q6ZkmKH7hh/sNvfVxaNn/1vJ3lTSLQQYRZj3DwZndYQsz1NafciI/6U8agsxUPNgyU1mGzDWgbT xgS4UtyW /fYml+RIOfR32CSSrlglwY/uCwzu+Xo3IEXqLUs5fvKebQn6SJ3YkvmHJYc0PgdQ/BCkzNkUWUuelNWS/mHcYmqTq9XY+PbeogyBS+lPydsItv+Z4D2rW+BQ8Ug+s6g94BkjgtgzgvGX47lUEUTitqt42wc2+Qwrgu5yyTlBITZ3v7HcRlYqL9//7hBJ3uqTEIuS523yKty3DlVBpx9sTg8DZje9QcKUw1hM7bh8I0MVL0+7fCV9yINp2AG0I7qlNm7rrh1volH7V4KkCgxBepOuZRUd85XslOEmnap7biSOx+tIv1+i9V2AoZNi+2VpTQHnDAzK2JyLpMGjgbvQs6B5uej/53e+wDfaO7NcWmkjl39XMt9e6lJ7nxuHEa0eZd0XAZ/RHx/hYejJDjHORSWLFTEPiEoYRA7dky9SdUbcjLF5d4ctQsPFWq7FSe5qE10MN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that there is a cross arch u128 and cmpxchg128(), use those instead of the custom CDSG helper. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens --- arch/s390/include/asm/cpu_mf.h | 2 +- arch/s390/kernel/perf_cpum_sf.c | 22 ++++++---------------- 2 files changed, 7 insertions(+), 17 deletions(-) --- a/arch/s390/include/asm/cpu_mf.h +++ b/arch/s390/include/asm/cpu_mf.h @@ -141,7 +141,7 @@ union hws_trailer_header { unsigned int dsdes:16; /* 48-63: size of diagnostic SDE */ unsigned long long overflow; /* 64 - Overflow Count */ }; - __uint128_t val; + u128 val; }; struct hws_trailer_entry { --- a/arch/s390/kernel/perf_cpum_sf.c +++ b/arch/s390/kernel/perf_cpum_sf.c @@ -1228,16 +1228,6 @@ static void hw_collect_samples(struct pe } } -static inline __uint128_t __cdsg(__uint128_t *ptr, __uint128_t old, __uint128_t new) -{ - asm volatile( - " cdsg %[old],%[new],%[ptr]\n" - : [old] "+d" (old), [ptr] "+QS" (*ptr) - : [new] "d" (new) - : "memory", "cc"); - return old; -} - /* hw_perf_event_update() - Process sampling buffer * @event: The perf event * @flush_all: Flag to also flush partially filled sample-data-blocks @@ -1307,14 +1297,14 @@ static void hw_perf_event_update(struct /* Reset trailer (using compare-double-and-swap) */ /* READ_ONCE() 16 byte header */ - prev.val = __cdsg(&te->header.val, 0, 0); + prev.val = cmpxchg128(&te->header.val, 0, 0); do { old.val = prev.val; new.val = prev.val; new.f = 0; new.a = 1; new.overflow = 0; - prev.val = __cdsg(&te->header.val, old.val, new.val); + prev.val = cmpxchg128(&te->header.val, old.val, new.val); } while (prev.val != old.val); /* Advance to next sample-data-block */ @@ -1496,7 +1486,7 @@ static bool aux_set_alert(struct aux_buf te = aux_sdb_trailer(aux, alert_index); /* READ_ONCE() 16 byte header */ - prev.val = __cdsg(&te->header.val, 0, 0); + prev.val = cmpxchg128(&te->header.val, 0, 0); do { old.val = prev.val; new.val = prev.val; @@ -1511,7 +1501,7 @@ static bool aux_set_alert(struct aux_buf } new.a = 1; new.overflow = 0; - prev.val = __cdsg(&te->header.val, old.val, new.val); + prev.val = cmpxchg128(&te->header.val, old.val, new.val); } while (prev.val != old.val); return true; } @@ -1575,7 +1565,7 @@ static bool aux_reset_buffer(struct aux_ for (i = 0; i < range_scan; i++, idx++) { te = aux_sdb_trailer(aux, idx); /* READ_ONCE() 16 byte header */ - prev.val = __cdsg(&te->header.val, 0, 0); + prev.val = cmpxchg128(&te->header.val, 0, 0); do { old.val = prev.val; new.val = prev.val; @@ -1586,7 +1576,7 @@ static bool aux_reset_buffer(struct aux_ new.a = 1; else new.a = 0; - prev.val = __cdsg(&te->header.val, old.val, new.val); + prev.val = cmpxchg128(&te->header.val, old.val, new.val); } while (prev.val != old.val); *overflow += orig_overflow; }