From patchwork Mon May 15 07:57:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90940C77B75 for ; Mon, 15 May 2023 08:06:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA84A900003; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0B95280001; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9898A900003; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6FE10900002 for ; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 340D51A1366 for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) X-FDA: 80791758264.18.6235D50 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf18.hostedemail.com (Postfix) with ESMTP id 2D1A91C0007 for ; Mon, 15 May 2023 08:06:49 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b="mfr/Lee6"; spf=none (imf18.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=gp7OCKaBYjIIkPcuPMybL+ixdp/AdwhFq0MWXNBGUCo=; b=WuYuxdDGcD0HzcH2ME8DhnRDYHMP5+6L6tnsjAQe2umnK4hHyroKXHruMVZQgqQFLIUBGl wNaVg+u4zqW4bI34nohHUEz5STgJU8gkDAhD9CuUdj5c65ypDzbN6s27cTGsjXUSh/1iyA QKHPbMjDD2Tw4ixdega/SZ88rW52zJQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138010; a=rsa-sha256; cv=none; b=LmIwFvnsDBF49g5Qsgwi6rD7ZVvprRvkJa8DyqZTFDxEpAnqMc371KkFRkQyCU4VWFat+T lUvWbn3JWG4rkrLPchrYJcIbD/KVAYLPUJLFngdgqQodsjKuIZSR48Z1PKZWrGWxR+G75F k6swpQBx3NnrH17Rpgz+2x6fdWZdst8= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b="mfr/Lee6"; spf=none (imf18.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=gp7OCKaBYjIIkPcuPMybL+ixdp/AdwhFq0MWXNBGUCo=; b=mfr/Lee6q6wb3hW9VLMJP6RMas nTl9Qe1+9Iu48xOQT4aXYKIo3YdHu4pVixS6ep5klWjn2Hleinv1VLcJZJw+35Dak53xO4M9X9KIK FpMiCIIAsi7/QgdKTHO+THopVLnjgnmza7zDZfiP5yT3vka2IQVSR4N/m/pdNqYPnTzsA8lIVgWqD P5m3mwC26ExmfD/meFO5f1ezwrEtuIohzrkIWoLo3mQANg3Hx3wLE8RBN4yr9HOkOwiYyEuNQ3/1W PLL/LI+1uLThBRAeCCPMwP+WHJ7vOYl/0orY1PoeN18xp4Y80oyboD7BypdwlKtzQ00oQnrhgh54F yIuqhB5w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDk-00BQMy-1T; Mon, 15 May 2023 08:06:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0C5C8300786; Mon, 15 May 2023 10:06:10 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id AB37F202674A5; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080553.979680310@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:00 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 01/11] cyrpto/b128ops: Remove struct u128 References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2D1A91C0007 X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: 1qpanabfwa5939f5ctg6g3ott8nuyxxq X-HE-Tag: 1684138009-756905 X-HE-Meta: U2FsdGVkX1+gVGd4UfjJMea6iqm+f+aSYPLYhVd1aSIiU2UD4ORtpxzOMM9Xlh8JEWDPaPoCdB7/1e7sFTIQ+f+wTmj5N6E9llB9mNDttqbA2GC0Mu5NBOZZ5s+MCOGeLkvIjxvRVg3pK/5KLXY8W8GPzAVtuttICKJam6BYMpXe1IUdjkgupEjRBNko2TEpt07nyTyB/UWZKN4Vjy40af0kTEpSgGglo/jM9XElJgCn9B9UiTDx1XbPD39479d52iNgi4I6TyDhzEBKQPR+wx/SrsUxykixtKeZ3sUXmBbbJet8zy3CZ7jGlmVoTfep7cJv58/Og3sog05T3E2Uil6qSpSi7xELtk7uke7r5jI69SsFEjajK8GME1HiKS+NUU45D2uKwZ8tq2PG8OUuq4XvpXIwjm8vDZWq6iFmIEkBj61qaQP/CdV2BFSj/tLMho2P5n/iRNOTwpSrHJ39m7VsZHfPp2EWFOnjJF+FCsF7634Fo4VyIVFH2OwOAxjFCWaAj7riUC8ZP0Vt3WWG84Ot4XIUPwWpWrzL8sz8qyFLxARP32NC3qYNksRVhsGkV1zXvJpkV6gQn5YLSzXb/tQQ1z6nNiFATGNFcv6KcwiOowKLEzuW1zQxAxbWEvZivkQCNDf2Puq7ICHsp/nq2PiQ+SEW+osKFt7HyHP0PBX3G7GfIJhLiaycXvkO2u1vNHIOa5FB2kkhJldLTGcfSsWNiweqgqMu5BnJiOQ+1bsxxjDzshw2S65/ACAogiZu5E5+ZVW5ZFJeUnokcVh1zlg/w2Q/fxEHQ2Q1bo7X76FMxLvFdYZluarVexBq/G+miMvXHea2HqhJFYYVxFlTjmuNze8z11nS9R8TPfQJKmUeOM+PBw+ihC009Z6Gz//O+HYURM62vi/ehGQ729/8WA/Y6/zM4i2riqZ213ZxM1wT8Gbk7/4h9CNDAZxqCNlzt4PWEMsJpP572FY91gp zxmS4IyM H6NR4MTk6TzBCs0mwjho7DHkYjfw12xj6oFlAkujOV99fVuIAX+48jAdwYJe+SMamVDIMlzCfn5xy7lWevlKsfJu7FwpOTSo0praqWkqmnGZn6lg6BtMUKHw5nWoW+Ik67CbwzfGu01XrAZfcGEzcCWC+z30qlcfCdU5LwMyDgo/+5FQFx2Cn0+3vJmT9tIII9ANx2tZmRo685v4WqAbi579UFRk4vZuMMpeXTV4yhY69fxFj+M6MBBfNpG7unwtAK970VisKVnf7opWJ93axYFIRh/TnwCCb5v5ceZT/v//rNnAFYAFWvsYUUCvPX6v0RjkZL8PerhKzupMGUiuMsmFW+FMKr1zJnDVfTIBe9qEAizwmzGLTZ4Ztes6fClsocbzj5jylPksSckkdBKGx8zz/AQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Per git-grep u128_xor() and its related struct u128 are unused except to implement {be,le}128_xor(). Remove them to free up the namespace. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Herbert Xu --- include/crypto/b128ops.h | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) --- a/include/crypto/b128ops.h +++ b/include/crypto/b128ops.h @@ -50,10 +50,6 @@ #include typedef struct { - u64 a, b; -} u128; - -typedef struct { __be64 a, b; } be128; @@ -61,20 +57,16 @@ typedef struct { __le64 b, a; } le128; -static inline void u128_xor(u128 *r, const u128 *p, const u128 *q) +static inline void be128_xor(be128 *r, const be128 *p, const be128 *q) { r->a = p->a ^ q->a; r->b = p->b ^ q->b; } -static inline void be128_xor(be128 *r, const be128 *p, const be128 *q) -{ - u128_xor((u128 *)r, (u128 *)p, (u128 *)q); -} - static inline void le128_xor(le128 *r, const le128 *p, const le128 *q) { - u128_xor((u128 *)r, (u128 *)p, (u128 *)q); + r->a = p->a ^ q->a; + r->b = p->b ^ q->b; } #endif /* _CRYPTO_B128OPS_H */ From patchwork Mon May 15 07:57:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B8D3C7EE26 for ; Mon, 15 May 2023 08:06:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39877900004; Mon, 15 May 2023 04:06:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F934280001; Mon, 15 May 2023 04:06:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8642900005; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8C9A0900006 for ; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2ACBCA13C9 for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) X-FDA: 80791758264.17.5BB6446 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf28.hostedemail.com (Postfix) with ESMTP id 42C18C0008 for ; Mon, 15 May 2023 08:06:50 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=J+RYi2Ol; spf=none (imf28.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=axnHoGjIL7t9C+QVLq6pAPgEZLVVerXkwPR51K17BxQ=; b=5lo2uJ5N7D3DKFNHuA5AuqFKMWGvAlCJcsCtgS+yZ1+wZB/k5+XUasxTIULROMxd5YNRMI TmQv4+bKLzolXPLejUmGRISMgYgxKdw8+SrThrE+oCx4hlnmBJiyaTZxrIyrJ+8y0Gb2ed 0joBPP+6Exjjy+o4+4SX+GGqC23ll+A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138010; a=rsa-sha256; cv=none; b=b05M+QPCdhbftYlQjj0zXQd20pb9ggXhAe0hGQvY67GlxZAENSrhBYTH/MkbgP8IP5P/5t i11HLjblRgQcjjD+iYcTjrbqtOzUicodcfxxJBmsblHsLN0lSvKfprJyDEY+ORvkhX+Th6 hwgW+8PR5l2J3byDzLqsOt9SO11k1dg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=J+RYi2Ol; spf=none (imf28.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=axnHoGjIL7t9C+QVLq6pAPgEZLVVerXkwPR51K17BxQ=; b=J+RYi2Olcu/GDwkb7uyoMxOowZ gdnnCNimNXMzjHfiHbHeyq75lJdwzqja8okQUmitQ1ua7iWanYtjpUVa32iVsf63pmox4vvEN+Ejz 1wpagut3YpzUSwNPpwpsl3jsBlJSSesN7UlPVAxsCXGiTi2d8aMeHHRkqQGQKv/P2cwZEs4JCNCfS xBlhWHGUBVitTRjLQwLm3qFy1fFLZshzr1Jf7x5B4z/zbEO4fjLkYFu6HtcQ+JbhA2M4mY6B7vLd7 0ZMD0sEUiA88G6VeDUxFw1EIBsMuAOuqnZrusc8ZVstoP2bAOXDwpltv/Y/kKX6d4HXBKVWWAlia0 mx0Oe8lw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDk-00BQMz-1T; Mon, 15 May 2023 08:06:16 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 09F29300244; Mon, 15 May 2023 10:06:10 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id AEDEA202FCE90; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.047618712@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:01 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 02/11] types: Introduce [us]128 References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: ma9tqenazed8ogz7eg5fgpyppin1sfou X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 42C18C0008 X-Rspam-User: X-HE-Tag: 1684138009-101608 X-HE-Meta: U2FsdGVkX1812PP9pKAmoxmypcXjipw+Lr1OgRx5HsRliCGhuBNCkhx1HOTSCF2I7ry5txvf+swoc3yZd9qCPCNKG4r76ryGBuI6AARkMivZWPVHSPgFyPCFIsA0DrcZc27R4P+gwI7Ab8YfrKVaBUohtlidPnQ9sbhqtKkv0Dk8IETHiQCr9RJkFa4L9UkSRc36D0rdBwAaMv/5OJMQB+AS7Y/+rTIG4sl5ldeZHwksI3+sjzxVH/rXVdbBsrPeN+vrVQcX1LkfVx+tturxMEAcIKXIZKnMrFE2mpFXN7etVcfdWL5Pg5a+M9DzgVz0qIKeVfrqSNVF9NBGiXUaJaSU2xquq7YM1tROaG6NbPOOlDVsIJGtdaIou9R/ShAVc0l3hPetIAuEkY5MbnG4h5bBp8S2bVr//0wldBPNan5ITc7YSTJ2NGtrC/bWnctumafwzai6XsFxrOMgXq0mt+QQ7HDRWXR/h/uSiDonsfKmw9MWJAuMgtG2GUWF0DrEBiiwquuFGFuwakjV/gOCn099CcTVYCnpqCmR5uTMcRvT56onB1XlTJ82Y3olLWPTN4KKxwqxRPoLXANPVhhtyBmCphlDP7uSGx9bO4Y1dlRC6TaAN/OrlIDRz6Rm0Xs/9Rste5Gclf5iUfQrYkbQWDVOd2QPGXbRfBMQcZr+euzQWu0qJgdfT5qGraNTThkKtZ8bATSnPW/LVsD4/KFH0x39vV9Vs2VD8k7NKz8fGwIH5VlckxHwFX+usfpk9Srh5VuVWdombQ5YPKhUvlo8s6IpJQDRdvdwPGCVrW3MGcdo4kMVyiwniHGC0/vqqJtxDuR6nHHeU9G0xZlnU9tyZ6qyAp4h3uhpUtj1ATiZ2v13NkMczPlKbNK5jTjHFKE/KQrJ8jFAff80KN+ei+weNnC5q/+CDBmYcR18yaf4dwiAIOe3Nh1Stf7DQGvIYINgYwPx1z9kaRpDiRwZ/Ys 7RIXQR7j zJpxMRAJIySC7cXELZzHnES/7OH4H5/XS6RxPsji7lbsLFF4bNxF0aXkUsL3QoMYq6na9z75+ZQa2UhAEBO9YRwJW7zTjqN+I0oJfiJTPEzmw69I+uo8wXDRdR0q2qB4OZoLcnQWpUFwQ5x5sQQrQVOI1OPLyp77El43Ok9DesbBQhzfmGKgskWOvhTNoH7GaIUM+5zi6hJ2W++QWR9btMrmj1gdZ3R6BrKiTbL8K+ulDn+0/ZLvR8sCXDgzwDISUch07tQPaPpC4oshFwoJ8fS+1SWozjV3+Q2F+h0l4wDLY9i/N1YfMRiZR/sOMLEcAiCrFd1ScIX4KPUl1nLjOB0pLbfSWr/IEEiMOWYvzee5oawkOXezQqHjCWHAM1/tqIup79qIewS7T/JjxJXqeqsZVUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce [us]128 (when available). Unlike [us]64, ensure they are always naturally aligned. This also enables 128bit wide atomics (which require natural alignment) such as cmpxchg128(). Signed-off-by: Peter Zijlstra (Intel) Acked-by: Herbert Xu --- include/linux/types.h | 5 +++++ include/uapi/linux/types.h | 4 ++++ 2 files changed, 9 insertions(+) --- a/include/linux/types.h +++ b/include/linux/types.h @@ -10,6 +10,11 @@ #define DECLARE_BITMAP(name,bits) \ unsigned long name[BITS_TO_LONGS(bits)] +#ifdef __SIZEOF_INT128__ +typedef __s128 s128; +typedef __u128 u128; +#endif + typedef u32 __kernel_dev_t; typedef __kernel_fd_set fd_set; --- a/include/uapi/linux/types.h +++ b/include/uapi/linux/types.h @@ -13,6 +13,10 @@ #include +#ifdef __SIZEOF_INT128__ +typedef __signed__ __int128 __s128 __attribute__((aligned(16))); +typedef unsigned __int128 __u128 __attribute__((aligned(16))); +#endif /* * Below are truly Linux-specific types that should never collide with --- a/lib/crypto/curve25519-hacl64.c +++ b/lib/crypto/curve25519-hacl64.c @@ -14,8 +14,6 @@ #include #include -typedef __uint128_t u128; - static __always_inline u64 u64_eq_mask(u64 a, u64 b) { u64 x = a ^ b; --- a/lib/crypto/poly1305-donna64.c +++ b/lib/crypto/poly1305-donna64.c @@ -10,8 +10,6 @@ #include #include -typedef __uint128_t u128; - void poly1305_core_setkey(struct poly1305_core_key *key, const u8 raw_key[POLY1305_BLOCK_SIZE]) { From patchwork Mon May 15 07:57:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 115B0C77B75 for ; Mon, 15 May 2023 08:07:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFD15280009; Mon, 15 May 2023 04:06:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B85DE280008; Mon, 15 May 2023 04:06:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B042280009; Mon, 15 May 2023 04:06:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7E5DE280008 for ; Mon, 15 May 2023 04:06:57 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 579C6C13A1 for ; Mon, 15 May 2023 08:06:57 +0000 (UTC) X-FDA: 80791758474.08.15261BA Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 819EB4000A for ; Mon, 15 May 2023 08:06:55 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nCa92XFn; spf=none (imf11.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=1dpaKFw/ImnX3ivBkF4/OTirLfnK12XzoUFqOemVS9M=; b=Jo7ulvLony+HBQC89Um/v5Qt5hIdTJts8jQPBpjxIgGV8BgbBOhWD2IYBcMxEzUqWGvbej d0I0H1KBJjbBKCcFpmFaeJPWcwrgMRwxQnVnajfZ+qoHE65m6Idpc5fcqAMV2jkIQAFdL2 wOimIc/QCd8LNN8Z/fPPWE4MsH3UNsg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138015; a=rsa-sha256; cv=none; b=QLHfwO+LKkbtrY+fNw98sVEaCqSnIyCV69dX4XB007joUo10F44sGHs6gsi6xle6lg5gnG oFMUusK/pDmr75vSS4TfAktQIbjbeEw3MtlPxQJ9R37mIK73Vq2xVYK0SI1S76NdBTFA4k SkJXvf2ho6DenKNZByYqbQHxf/bRdCE= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=nCa92XFn; spf=none (imf11.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=1dpaKFw/ImnX3ivBkF4/OTirLfnK12XzoUFqOemVS9M=; b=nCa92XFnMhdkelbE+tRZRrXIYS kOcZdCKGEMW4KSKB+6B4aivSqYkUHANT/UlD4Aj6tqOcvWV54dCTn3Zgoan/4wXM6P8uTariGluRA dK6MON9zNhNVc6sBNEvJzPx2w9m+W7e4I4aCZpl5DVMtACFO18Ch9JPZ1jyUpU83EMRKXHtV8Iz1D XfR1l9t+Il6O7f3x5b8EmWQM90cSGueNT4YgQOkrbRReRJORMk8uuKxUx66lTPr2d1C5h5y33LUiC TgvfbRNNsp0dRP9g9qP2uK7HzZtalOk93SWeVN83x38l4Z+We3DOMEzC6VzL2KU4lEfmaiaAB2JGw HPP9047A==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pyTDk-003HUP-En; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 10D4F300912; Mon, 15 May 2023 10:06:10 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B1D27202FCE9F; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.114813040@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:02 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 03/11] arch: Introduce arch_{,try_}_cmpxchg128{,_local}() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: oh6tyikm6nqzsq7n8rg8bu6bqfojbo97 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 819EB4000A X-Rspam-User: X-HE-Tag: 1684138015-825838 X-HE-Meta: U2FsdGVkX1/034IrSJ1xlAxGzU2eqKvlAp3qW3y4cI7P4zhNA7yXP6Skw3yGtEy3gIS5nPbtADEarRPRocILpKKGA+1ksbx3yzH4g4lfdZNDKHXDEgz4R9Jqaa74NNmBeUp5iUawlg8uekPwuwH5f4CRXJnwFTxl3jPkWs2yJPngJgCkvrrII2GjHL/aq4XaDSww3l5EwqdVTe34IBwmVrwa29qEK54+vzxWgTcgUnq8JnyM8oL2m/kF7mZ+JdtX5czJ+OF9CiNNxT4HjXujpYiFwM4h7Ljpvtr7N2HaQS1lpq0d+8tQEib4kfzcd7AEaqkhkw7a72lum09aXb1H12t7acarTvrp9uM1L5iP7K4oc6cYaceRF9NGVALGXLNahDr0Q0szWNSouAih7utgeLx46alG9Vh/tQ0d0FRaRjXmPWUtKrPSqb87coJWdGBUh5e0fFGJ4FDrK0PGgqq7OFyB4VuM71/b9dwp5ITFMMse89xvHJqQIPkS1Jh2YZihHofjA07wpdobSOCrZgfOesDyIGdNKZZkSSwl+5QiHjS4RGEenHhU9wMi47E7AY1nw/WvHDRoxU5mlWTkviD4wIIlUDM7skvFjMH2DcTyksZdhmV275LembJrWKGo8vmhvbNz85btCynRnZO4dth2PQidbKj95p/M/Qex73b6z4oiCUq4TiwFBv5Al1DdMlaVdQ/YpwMpxvqMtngpWY4UBU9sEIu3kCGquEyqY1yAACGZNaDzHVI4tjaODTViKi8RasoEg6O/K+begq4LDM3uruwO6ZebXoACyMRuyf5EGXzqSlHhJHwM9/lLB4zH1HB4oaOhetrQZz5ZXnQHnxZXpR/2E1elKAhBk+9cd/fCOIwkEyZyPMoZVHoxSP/siklaG5+qusgiFsqS2EzYzWBG0a430hyRdsnpx3cCCMtc9ojJEQtBH1+NmewxdatHBO0YS9Gr8dmX2/Hgtui5i3L mYilKbuT bpqqi7MaYZ6jsHr2TfVDx4q5XVHF/srhTDYArcU9PsFJ5U9LYfLV962QPZmm28KFrMn5liBtDCE8d+JCpMR0uQNDyhohwUBTmK6SzOCi6BH4xoKiqXfZRpiwmd4WwmV/XlR+/U+kIYH071qeKpQM9IdB5psYlVY7oMy5fxvi5XbbhGrALZshQOprKhjHwXbVFhMt6O+aDhxAlH+BsRcknf44+3QPwSDODOvKgXgsOLN7zMaJhMpURhcBdOwmvfhCgoOmCLzRBcz0aufiYAOa6KIOFNazgWhSDRmQfVE+Yl3cAO12UgR1K74IXSNbKAUzXx7xnYTNK2sQUp9J56jDEpFxhzKWLYfwPXSqRxjWALCuSQ3bZShM6xOyqilFJTDfnzPuN34Aa4UdsgpqTLPqY7cwqsQtMrDlP43LwsGOmiunDeTYiordEd87Gb2dUyPcwNmH8VFBsPTzdDGVB06RZlqJNbchuU1LWakNT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For all architectures that currently support cmpxchg_double() implement the cmpxchg128() family of functions that is basically the same but with a saner interface. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens Acked-by: Mark Rutland --- arch/arm64/include/asm/atomic_ll_sc.h | 41 +++++++++++++++++++++ arch/arm64/include/asm/atomic_lse.h | 31 ++++++++++++++++ arch/arm64/include/asm/cmpxchg.h | 26 +++++++++++++ arch/s390/include/asm/cmpxchg.h | 14 +++++++ arch/x86/include/asm/cmpxchg_32.h | 3 + arch/x86/include/asm/cmpxchg_64.h | 64 +++++++++++++++++++++++++++++++++- 6 files changed, 177 insertions(+), 2 deletions(-) --- a/arch/arm64/include/asm/atomic_ll_sc.h +++ b/arch/arm64/include/asm/atomic_ll_sc.h @@ -326,6 +326,47 @@ __CMPXCHG_DBL( , , , ) __CMPXCHG_DBL(_mb, dmb ish, l, "memory") #undef __CMPXCHG_DBL + +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __CMPXCHG128(name, mb, rel, cl...) \ +static __always_inline u128 \ +__ll_sc__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ +{ \ + union __u128_halves r, o = { .full = (old) }, \ + n = { .full = (new) }; \ + unsigned int tmp; \ + \ + asm volatile("// __cmpxchg128" #name "\n" \ + " prfm pstl1strm, %[v]\n" \ + "1: ldxp %[rl], %[rh], %[v]\n" \ + " cmp %[rl], %[ol]\n" \ + " ccmp %[rh], %[oh], 0, eq\n" \ + " b.ne 2f\n" \ + " st" #rel "xp %w[tmp], %[nl], %[nh], %[v]\n" \ + " cbnz %w[tmp], 1b\n" \ + " " #mb "\n" \ + "2:" \ + : [v] "+Q" (*(u128 *)ptr), \ + [rl] "=&r" (r.low), [rh] "=&r" (r.high), \ + [tmp] "=&r" (tmp) \ + : [ol] "r" (o.low), [oh] "r" (o.high), \ + [nl] "r" (n.low), [nh] "r" (n.high) \ + : "cc", ##cl); \ + \ + return r.full; \ +} + +__CMPXCHG128( , , ) +__CMPXCHG128(_mb, dmb ish, l, "memory") + +#undef __CMPXCHG128 + #undef K #endif /* __ASM_ATOMIC_LL_SC_H */ --- a/arch/arm64/include/asm/atomic_lse.h +++ b/arch/arm64/include/asm/atomic_lse.h @@ -317,4 +317,35 @@ __CMPXCHG_DBL(_mb, al, "memory") #undef __CMPXCHG_DBL +#define __CMPXCHG128(name, mb, cl...) \ +static __always_inline u128 \ +__lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ +{ \ + union __u128_halves r, o = { .full = (old) }, \ + n = { .full = (new) }; \ + register unsigned long x0 asm ("x0") = o.low; \ + register unsigned long x1 asm ("x1") = o.high; \ + register unsigned long x2 asm ("x2") = n.low; \ + register unsigned long x3 asm ("x3") = n.high; \ + register unsigned long x4 asm ("x4") = (unsigned long)ptr; \ + \ + asm volatile( \ + __LSE_PREAMBLE \ + " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\ + : [old1] "+&r" (x0), [old2] "+&r" (x1), \ + [v] "+Q" (*(u128 *)ptr) \ + : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \ + [oldval1] "r" (o.low), [oldval2] "r" (o.high) \ + : cl); \ + \ + r.low = x0; r.high = x1; \ + \ + return r.full; \ +} + +__CMPXCHG128( , ) +__CMPXCHG128(_mb, al, "memory") + +#undef __CMPXCHG128 + #endif /* __ASM_ATOMIC_LSE_H */ --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -146,6 +146,19 @@ __CMPXCHG_DBL(_mb) #undef __CMPXCHG_DBL +#define __CMPXCHG128(name) \ +static inline u128 __cmpxchg128##name(volatile u128 *ptr, \ + u128 old, u128 new) \ +{ \ + return __lse_ll_sc_body(_cmpxchg128##name, \ + ptr, old, new); \ +} + +__CMPXCHG128( ) +__CMPXCHG128(_mb) + +#undef __CMPXCHG128 + #define __CMPXCHG_GEN(sfx) \ static __always_inline unsigned long __cmpxchg##sfx(volatile void *ptr, \ unsigned long old, \ @@ -228,6 +241,19 @@ __CMPXCHG_GEN(_mb) __ret; \ }) +/* cmpxchg128 */ +#define system_has_cmpxchg128() 1 + +#define arch_cmpxchg128(ptr, o, n) \ +({ \ + __cmpxchg128_mb((ptr), (o), (n)); \ +}) + +#define arch_cmpxchg128_local(ptr, o, n) \ +({ \ + __cmpxchg128((ptr), (o), (n)); \ +}) + #define __CMPWAIT_CASE(w, sfx, sz) \ static inline void __cmpwait_case_##sz(volatile void *ptr, \ unsigned long val) \ --- a/arch/s390/include/asm/cmpxchg.h +++ b/arch/s390/include/asm/cmpxchg.h @@ -224,4 +224,18 @@ static __always_inline int __cmpxchg_dou (unsigned long)(n1), (unsigned long)(n2)); \ }) +#define system_has_cmpxchg128() 1 + +static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) +{ + asm volatile( + " cdsg %[old],%[new],%[ptr]\n" + : [old] "+d" (old), [ptr] "+QS" (*ptr) + : [new] "d" (new) + : "memory", "cc"); + return old; +} + +#define arch_cmpxchg128 arch_cmpxchg128 + #endif /* __ASM_CMPXCHG_H */ --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -103,6 +103,7 @@ static inline bool __try_cmpxchg64(volat #endif -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) +#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) +#define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8) #endif /* _ASM_X86_CMPXCHG_32_H */ --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -20,6 +20,68 @@ arch_try_cmpxchg((ptr), (po), (n)); \ }) -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) +union __u128_halves { + u128 full; + struct { + u64 low, high; + }; +}; + +#define __arch_cmpxchg128(_ptr, _old, _new, _lock) \ +({ \ + union __u128_halves o = { .full = (_old), }, \ + n = { .full = (_new), }; \ + \ + asm volatile(_lock "cmpxchg16b %[ptr]" \ + : [ptr] "+m" (*(_ptr)), \ + "+a" (o.low), "+d" (o.high) \ + : "b" (n.low), "c" (n.high) \ + : "memory"); \ + \ + o.full; \ +}) + +static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) +{ + return __arch_cmpxchg128(ptr, old, new, LOCK_PREFIX); +} + +static __always_inline u128 arch_cmpxchg128_local(volatile u128 *ptr, u128 old, u128 new) +{ + return __arch_cmpxchg128(ptr, old, new,); +} + +#define __arch_try_cmpxchg128(_ptr, _oldp, _new, _lock) \ +({ \ + union __u128_halves o = { .full = *(_oldp), }, \ + n = { .full = (_new), }; \ + bool ret; \ + \ + asm volatile(_lock "cmpxchg16b %[ptr]" \ + CC_SET(e) \ + : CC_OUT(e) (ret), \ + [ptr] "+m" (*ptr), \ + "+a" (o.low), "+d" (o.high) \ + : "b" (n.low), "c" (n.high) \ + : "memory"); \ + \ + if (unlikely(!ret)) \ + *(_oldp) = o.full; \ + \ + likely(ret); \ +}) + +static __always_inline bool arch_try_cmpxchg128(volatile u128 *ptr, u128 *oldp, u128 new) +{ + return __arch_try_cmpxchg128(ptr, oldp, new, LOCK_PREFIX); +} + +static __always_inline bool arch_try_cmpxchg128_local(volatile u128 *ptr, u128 *oldp, u128 new) +{ + return __arch_try_cmpxchg128(ptr, oldp, new,); +} + +#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) +#define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16) #endif /* _ASM_X86_CMPXCHG_64_H */ From patchwork Mon May 15 07:57:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F458C7EE24 for ; Mon, 15 May 2023 08:07:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 498A1280005; Mon, 15 May 2023 04:06:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 39421280009; Mon, 15 May 2023 04:06:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4CA8280005; Mon, 15 May 2023 04:06:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 426ED280004 for ; Mon, 15 May 2023 04:06:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1B4B5413D3 for ; Mon, 15 May 2023 08:06:55 +0000 (UTC) X-FDA: 80791758390.28.053DA44 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 0EBFE1C000A for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=d+Q3WNOa; spf=none (imf21.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=RCr9Xn0mKbWganhsRQfOR6jVFCjDlLx/rSxLfiFKpAk=; b=dHmqX8LqQBhvGP/HMEfA6kabq4Hkz3Q0QdND8LcyZel4MCC5hWwWZTlXmTcJ143Hhx+nrp SizRawv3aQJW2hflP+MuRjt+1/6W8re9fuAw+2TUCaqlMJhnNxa81b6UTfucaOqMNWY1OL z6AIVqrtqaVqpddy2124vYe6q8P84zQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138013; a=rsa-sha256; cv=none; b=1Mc3YRPgCpCBqCJjL6qLUvrXdLothzTspANomtE4G6AOcOx0rFygn4+PZumW4Mxld8mGLR LnywuxZb4kgQqUkqczU1csGDptfAbMduy7cTrfthasmYxc7b6Au9BiOyEo9trEvOAu4iqY 63/RR6GQ1nynz+Rc2jQdzDtW5uU+XM4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=d+Q3WNOa; spf=none (imf21.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=RCr9Xn0mKbWganhsRQfOR6jVFCjDlLx/rSxLfiFKpAk=; b=d+Q3WNOaRhK0mj1LuRITzeVvaf IUGC/nzRDUh6SubVu5LMHXgslcA+Kt+w9WLxc1CJgrI/DvmbjYTbVGjYoXL4/7oHpiiRhtZ1ltogj Ai4KCHKJzkavv9NelV+OYkzOPvTUvWyNYNhW3a95xNkNqyXJG5aOP4h2Y/4WSrkkOhu+RwFm5Eiqi k+dC8bMZ7KL0Dw9NG0LJbxdLqFErv1WaMC426/uXywDkDIMDgMr3SEU65fJxW4ItF5lXQl2/zrgQI O3UoOHKOCNf+8ZYv/SqyRtrWEytfLqwmt3qDqBRzsaf6TUaInYa/0xBEkXbiOB4cFhqmra4E5jnbT DTUpnFTA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pyTDk-003HUR-Ep; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 0E9FF30080C; Mon, 15 May 2023 10:06:10 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id B721D202FCEA3; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.181725437@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:03 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 04/11] instrumentation: Wire up cmpxchg128() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: y338phtr55hug3u3xwtf994kokracdfp X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0EBFE1C000A X-Rspam-User: X-HE-Tag: 1684138012-534160 X-HE-Meta: U2FsdGVkX1/Aj7EHSZoML8bYQvcCKtBR/vKZPeqVi72xtak0x2ChVnd581cpbbQR2rjMvolwkkbms8jhdsISSv2wtVkWIc+kjBtYN68dndbwst4Px6Xe9anwyxV65dO98rPcNOTPuaMsAKS+H41N5JxOWlV+1JIsQPHtW0uqfOcjXFB1IOzLazlLRwydGoUXjZGJWxnGbBtVUPg2ghnRdliBkm7DbVFlmyDlBkv3ISc1m8VxwHPJbvV1lG6mF5miWpzag/2Gb1d8WY1ytP8Idm+jxOy2t34EFCXmzNNFv85FvPJEOD28+9V/HSSCKfAv35E/iC6YKJ9lWjfXRzxhWmSPYj5QuTXIP6hKLqZLGAue1ufRz4keelGVCEz5eSfhme/rQ25OjTX977xOOOlxOuAp5vY2Z3NaWHmaGRzfpcefAl/jyikUQhkn48pLPAAdWwk3tOTfaT0DMTVmdzqNedCI1SuMCOqBSYENTaTXTishUc34RpoqiMSTxWm1fZHtiJNrhBW3JOKIWeoanENiyrbxxmkIS2/NBoWkwuXIfMiIOevZ0293RtujBukGw0jBD3QjcQByYDb1ikZdcHUxQz0k4YL5MfYiYLCqcO7TFogFFB/wZ0M4+NQTP9Ru3SzZQZS0AgXU0DooMe6pBdG8AE6nfQ9+y+8tzZ6pzWcE5LkJQTw3FdOwPRwOAseL4sUZTqsieIhxShthkdJN23XoF47LAUE73P0OdhEzFO/YxlzBEG6zbKe2bFsj3efehV7xZKP22v1kb+3kySrpFl5DJ3yqq9MEBpwPJ56oxtm5RIQiUzqxarqwOimTaHjuab3W9SPL/Zy667XdoCGIuk1OcWeJ2j3/9M+fCJDVW46YV+/AUctq0PBOqtfYmY7YZvLc1g6DSHo8MFTgDmoyMR4rXyWvXFO998y2dVVS65qxQY26nIKbb/5H+8BPhgHk3d8fEfGqYzuRoMx0/MZRru2 ZAqkETt5 Rwtdp4OIQeXWLRrW/9aPVH1+L0mopFV9Z7kgkQv4HAFAFzqzlcivGW4x4WyoplZJNYd+aaeVT9iLm6icPUL/+gFemM9FhFDE5VG7cMm5C0ppPMB9zvgvjVsLp9m6l7kiW8odrzOQJitJBLmIRXckyhfNjispnemPsA3SMXHiXggy6y+BvgNzsnFMpXZ3Aqf5tOl9TUG48mrfIIQhs8UUntNBx96qtMXj/5O/P9rpbTJqLlTIqGgIzB4Y3l7OgM+kYKQfvrM406aKvLfCmbY/xO816rTjZC8q+9SVqEDyJyhjq3lzvGxU50s3gEF3y6kNSUoFsMJ37OqnQkya1bfa/Oytq/cVLVV+exMFBSo740FTfND+iiGg+4kLRo4wVHhpGTgO+RN/rmST592Gvv37sNPxC/A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Wire up the cmpxchg128 family in the atomic wrapper scripts. These provide the generic cmpxchg128 family of functions from the arch_ prefixed version, adding explicit instrumentation where needed. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Mark Rutland --- include/linux/atomic/atomic-arch-fallback.h | 95 +++++++++++++++++++++++++++- include/linux/atomic/atomic-instrumented.h | 86 +++++++++++++++++++++++++ scripts/atomic/gen-atomic-fallback.sh | 4 - scripts/atomic/gen-atomic-instrumented.sh | 4 - 4 files changed, 183 insertions(+), 6 deletions(-) --- a/include/linux/atomic/atomic-arch-fallback.h +++ b/include/linux/atomic/atomic-arch-fallback.h @@ -77,6 +77,29 @@ #endif /* arch_cmpxchg64_relaxed */ +#ifndef arch_cmpxchg128_relaxed +#define arch_cmpxchg128_acquire arch_cmpxchg128 +#define arch_cmpxchg128_release arch_cmpxchg128 +#define arch_cmpxchg128_relaxed arch_cmpxchg128 +#else /* arch_cmpxchg128_relaxed */ + +#ifndef arch_cmpxchg128_acquire +#define arch_cmpxchg128_acquire(...) \ + __atomic_op_acquire(arch_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_cmpxchg128_release +#define arch_cmpxchg128_release(...) \ + __atomic_op_release(arch_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_cmpxchg128 +#define arch_cmpxchg128(...) \ + __atomic_op_fence(arch_cmpxchg128, __VA_ARGS__) +#endif + +#endif /* arch_cmpxchg128_relaxed */ + #ifndef arch_try_cmpxchg_relaxed #ifdef arch_try_cmpxchg #define arch_try_cmpxchg_acquire arch_try_cmpxchg @@ -217,6 +240,76 @@ #endif /* arch_try_cmpxchg64_relaxed */ +#ifndef arch_try_cmpxchg128_relaxed +#ifdef arch_try_cmpxchg128 +#define arch_try_cmpxchg128_acquire arch_try_cmpxchg128 +#define arch_try_cmpxchg128_release arch_try_cmpxchg128 +#define arch_try_cmpxchg128_relaxed arch_try_cmpxchg128 +#endif /* arch_try_cmpxchg128 */ + +#ifndef arch_try_cmpxchg128 +#define arch_try_cmpxchg128(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128 */ + +#ifndef arch_try_cmpxchg128_acquire +#define arch_try_cmpxchg128_acquire(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128_acquire((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128_acquire */ + +#ifndef arch_try_cmpxchg128_release +#define arch_try_cmpxchg128_release(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128_release((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128_release */ + +#ifndef arch_try_cmpxchg128_relaxed +#define arch_try_cmpxchg128_relaxed(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = arch_cmpxchg128_relaxed((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif /* arch_try_cmpxchg128_relaxed */ + +#else /* arch_try_cmpxchg128_relaxed */ + +#ifndef arch_try_cmpxchg128_acquire +#define arch_try_cmpxchg128_acquire(...) \ + __atomic_op_acquire(arch_try_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_try_cmpxchg128_release +#define arch_try_cmpxchg128_release(...) \ + __atomic_op_release(arch_try_cmpxchg128, __VA_ARGS__) +#endif + +#ifndef arch_try_cmpxchg128 +#define arch_try_cmpxchg128(...) \ + __atomic_op_fence(arch_try_cmpxchg128, __VA_ARGS__) +#endif + +#endif /* arch_try_cmpxchg128_relaxed */ + #ifndef arch_try_cmpxchg_local #define arch_try_cmpxchg_local(_ptr, _oldp, _new) \ ({ \ @@ -2668,4 +2761,4 @@ arch_atomic64_dec_if_positive(atomic64_t #endif #endif /* _LINUX_ATOMIC_FALLBACK_H */ -// ad2e2b4d168dbc60a73922616047a9bfa446af36 +// 52dfc6fe4a2e7234bbd2aa3e16a377c1db793a53 --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -2034,6 +2034,36 @@ atomic_long_dec_if_positive(atomic_long_ arch_cmpxchg64_relaxed(__ai_ptr, __VA_ARGS__); \ }) +#define cmpxchg128(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128(__ai_ptr, __VA_ARGS__); \ +}) + +#define cmpxchg128_acquire(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_acquire(__ai_ptr, __VA_ARGS__); \ +}) + +#define cmpxchg128_release(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + kcsan_release(); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_release(__ai_ptr, __VA_ARGS__); \ +}) + +#define cmpxchg128_relaxed(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_relaxed(__ai_ptr, __VA_ARGS__); \ +}) + #define try_cmpxchg(ptr, oldp, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2110,6 +2140,44 @@ atomic_long_dec_if_positive(atomic_long_ arch_try_cmpxchg64_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ }) +#define try_cmpxchg128(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + kcsan_mb(); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg128_acquire(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_acquire(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg128_release(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + kcsan_release(); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_release(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + +#define try_cmpxchg128_relaxed(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_relaxed(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + #define cmpxchg_local(ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2124,6 +2192,13 @@ atomic_long_dec_if_positive(atomic_long_ arch_cmpxchg64_local(__ai_ptr, __VA_ARGS__); \ }) +#define cmpxchg128_local(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + arch_cmpxchg128_local(__ai_ptr, __VA_ARGS__); \ +}) + #define sync_cmpxchg(ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2150,6 +2225,15 @@ atomic_long_dec_if_positive(atomic_long_ arch_try_cmpxchg64_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ }) +#define try_cmpxchg128_local(ptr, oldp, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + typeof(oldp) __ai_oldp = (oldp); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + instrument_read_write(__ai_oldp, sizeof(*__ai_oldp)); \ + arch_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ +}) + #define cmpxchg_double(ptr, ...) \ ({ \ typeof(ptr) __ai_ptr = (ptr); \ @@ -2167,4 +2251,4 @@ atomic_long_dec_if_positive(atomic_long_ }) #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 6b513a42e1a1b5962532a019b7fc91eaa044ad5e +// 82d1be694fab30414527d0877c29fa75ed5a0b74 --- a/scripts/atomic/gen-atomic-fallback.sh +++ b/scripts/atomic/gen-atomic-fallback.sh @@ -217,11 +217,11 @@ cat << EOF EOF -for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64"; do +for xchg in "arch_xchg" "arch_cmpxchg" "arch_cmpxchg64" "arch_cmpxchg128"; do gen_xchg_fallbacks "${xchg}" done -for cmpxchg in "cmpxchg" "cmpxchg64"; do +for cmpxchg in "cmpxchg" "cmpxchg64" "cmpxchg128"; do gen_try_cmpxchg_fallbacks "${cmpxchg}" done --- a/scripts/atomic/gen-atomic-instrumented.sh +++ b/scripts/atomic/gen-atomic-instrumented.sh @@ -166,14 +166,14 @@ grep '^[a-z]' "$1" | while read name met done -for xchg in "xchg" "cmpxchg" "cmpxchg64" "try_cmpxchg" "try_cmpxchg64"; do +for xchg in "xchg" "cmpxchg" "cmpxchg64" "cmpxchg128" "try_cmpxchg" "try_cmpxchg64" "try_cmpxchg128"; do for order in "" "_acquire" "_release" "_relaxed"; do gen_xchg "${xchg}" "${order}" "" printf "\n" done done -for xchg in "cmpxchg_local" "cmpxchg64_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" ; do +for xchg in "cmpxchg_local" "cmpxchg64_local" "cmpxchg128_local" "sync_cmpxchg" "try_cmpxchg_local" "try_cmpxchg64_local" "try_cmpxchg128_local"; do gen_xchg "${xchg}" "" "" printf "\n" done From patchwork Mon May 15 07:57:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82D09C77B7D for ; Mon, 15 May 2023 08:06:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B8685900007; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE89C900004; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C6D9900005; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 70F4A900003 for ; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 383BD1A1367 for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) X-FDA: 80791758264.20.708D669 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf20.hostedemail.com (Postfix) with ESMTP id C50BB1C000B for ; Mon, 15 May 2023 08:06:49 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=C7Q+9vEV; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138010; a=rsa-sha256; cv=none; b=ZJDFHEGt4fx7PMxEOooi1Rrkb5peJSWmxE4596oibTPzohttm3PXrQ0Col6Lfx010Mv/nG jDoass0FNBhtF1V0S6LNTe1QQk0vACeyzphYBuQ1tXoxTfr6cvYcputf2L9MeJlSqxoTf1 4xM7PWYZAu6zD9PzS1+On7Wts/OwFLY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=C7Q+9vEV; spf=none (imf20.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=3RRGLs3vZsBA3WquAm9CYBrs3TiBBk1z4f/JSTl06ko=; b=G/VOrhVrqP7wLG4zgtDr66ww9BuD8jT6cyxUC3vSLI/kkKbckg1VdscYcWVX4ZkY3ZCBQY Toj/+7mH4aMD27FevChUb2exuMaRp1pY7TM64kz/MDTGvbCDYo6L03GQQpBNTmAxi2wefL BIfr6CqKwS6QapPYoQRtIa+d5+iD0Ug= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=3RRGLs3vZsBA3WquAm9CYBrs3TiBBk1z4f/JSTl06ko=; b=C7Q+9vEV1QZqlrWrdiRS/4ZaOo C4QaD94dYLKM4rVSpBMuM4MPCUEwWG2wlZb7z7L+ytDDmEzthuIXFw5s4xhl1Zhwa8PHyegz3LsWv sX9yk6m2RhCvKI4eawfr4xg6NlwXbIHLgql/4Hh2IAyjJ5EKkfn282tlp6HqkTBBJgH+es2+j28ap mwmpcCi64hREGLbEv+l39wIXLA29adUS/A2oGsp2yAeCyreAL19bzS9x+aYuSwanUcEPwpYAb2/WQ Q6Uzz57SiaFlBid1/0NAHDSF0uzCiEKR6fHXhBcPEjn8/4auGrU3RiX6Ur+Re+rAmPdmvbZq7AL9R kV+s3owA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDl-00BQN3-1H; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8708C300ECD; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id BBF7D202FCEA5; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.248739380@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:04 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 05/11] percpu: Wire up cmpxchg128 References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C50BB1C000B X-Stat-Signature: xrftro1q9g83n9rhg3e3hcddwebehb1x X-HE-Tag: 1684138009-3163 X-HE-Meta: U2FsdGVkX18po87AhZVbNInM07KnNV/z2uOZuMAlUXjeJX8xkiUqdZptZst4c3xxMcVWydVBHY+u+LkvOZTRYuSyulU83Fdm2abOCNp1tEThSrpv2WHP29ZUmDYUSDpuZ2Fvb/L7uUZm2kimH7M26o1u7TYu1AJLQkLOZhmJEpjQZ/aRU2eONenBN6/qohKBhvO95t5Ns4sM2I0YscACIdxEsl+iDyvnKLxdFNMMatETUXXj/9flbPzSOdtSyUmOXi5FeJoqHE7fHlQ1qMXJr9ZJGTSH085l8CloXdyayC76XlkeIBH2GwqErHOzATLBb+s9n4f+sHWUo8LQr6cURPdK+EyUdGOMLwgiJY9TXqZU6oUH2EgsBxESMqc9EvCbXDSLE4MuesM34O2i8T3rywFz43GRBaDlHybSG7kkFQDvjYlTwZKdQTChlhuflMfx0Tu58/DonhTJ20JB1bQSw+lwiXj2NeJbrnuJ6k8yEFTlRMCNBsUisze3WoVk1ksGJjSKZrXJthNnr2fWe8G+9qlTAiXoVTlaIjHqjnCJFRHh8qfMbUnxIRBY7nwOgU+1eYNX0Gg1E/gxP7EvRuZlXf3fp5bv+/tj6Xap4xanNqeVkSkXALiqJk8GUlbDRFQUy46XJ5Zs8Nbmg/CDa540aWMkhgJSxj4GbsPGmf8JpqePnMRDwx7WV46muuKsEkcYDzzdbo5rf+ha1/xZ7v6idTjxAZOBzh7R8HAjFg9Eg/7557984MbhhKnhEeUS/z/+zcO2E20orODf9NAstJ6qxrY13VjQXsukYbTj/LWV6l5FSEX8edGB5wT4KIgAvejz7CFaFA9PWFnwgTzop5yk3agKigo4SI+wYIxtQS1A1biJeEoasPR8ZbLPB+nhpAnPgrXAVU5Ne+mpHnQXi2+N5+6aTAmxlL+crV/H5lg4YkAMTdFZPWQc9sHD64dZvMfnqVzJfBnB85WdJdnoBTM 6AWloqJ6 /cHBWrGX3FnF4FbH+8pV2fl19CA/L6oZlLIfLW6jkQXBIHClkgCj1tkx6LMjETAWrq/4pjVazxse/Pp53oh5kSfBXwvT6t42F3zmAwEvPJX/UPru0Bp6xUXmSUb8xDauvi6qsxUw2FXr5AJL3KwsRx/i20+zKwVobMyDF3aNL/C1vkETENJNz9UQk3NjxuMu5DAuMYO4C+qGL9qCDAAcRocYhri1tEr+0Bacsk90tK6ZNX6ysyV12k20pNLPE5cJ4P90smlK1Qh6JbJI21YhqQ/ssHHvBjSiAqVIT02XPyrLQmX4Z/QiJ9EoedzWTFYUPC/j2Dto80b0jr8JOv0eyp3de9nciL2J0d4RK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In order to replace cmpxchg_double() with the newly minted cmpxchg128() family of functions, wire it up in this_cpu_cmpxchg(). Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/percpu.h | 20 +++++++++++++ arch/s390/include/asm/percpu.h | 16 ++++++++++ arch/x86/include/asm/percpu.h | 59 ++++++++++++++++++++++++++++++++++++++++ include/asm-generic/percpu.h | 16 ++++++++++ 4 files changed, 111 insertions(+) --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -140,6 +140,10 @@ PERCPU_RET_OP(add, add, ldadd) * re-enabling preemption for preemptible kernels, but doing that in a way * which builds inside a module would mean messing directly with the preempt * count. If you do this, peterz and tglx will hunt you down. + * + * Not to mention it'll break the actual preemption model for missing a + * preemption point when TIF_NEED_RESCHED gets set while preemption is + * disabled. */ #define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ ({ \ @@ -240,6 +244,22 @@ PERCPU_RET_OP(add, add, ldadd) #define this_cpu_cmpxchg_8(pcp, o, n) \ _pcp_protect_return(cmpxchg_relaxed, pcp, o, n) +#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) + +#define this_cpu_cmpxchg128(pcp, o, n) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ = o; \ + new__ = n; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128_local((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + ret__; \ +}) + #ifdef __KVM_NVHE_HYPERVISOR__ extern unsigned long __hyp_per_cpu_offset(unsigned int cpu); #define __per_cpu_offset --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -148,6 +148,22 @@ #define this_cpu_cmpxchg_4(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) arch_this_cpu_cmpxchg(pcp, oval, nval) +#define this_cpu_cmpxchg64(pcp, o, n) this_cpu_cmpxchg_8(pcp, o, n) + +#define this_cpu_cmpxchg128(pcp, oval, nval) \ +({ \ + typedef typeof(pcp) pcp_op_T__; \ + u128 old__, new__, ret__; \ + pcp_op_T__ *ptr__; \ + old__ = oval; \ + new__ = nval; \ + preempt_disable_notrace(); \ + ptr__ = raw_cpu_ptr(&(pcp)); \ + ret__ = cmpxchg128((void *)ptr__, old__, new__); \ + preempt_enable_notrace(); \ + ret__; \ +}) + #define arch_this_cpu_xchg(pcp, nval) \ ({ \ typeof(pcp) *ptr__; \ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -210,6 +210,65 @@ do { \ (typeof(_var))(unsigned long) pco_old__; \ }) +#if defined(CONFIG_X86_32) && defined(CONFIG_X86_CMPXCHG64) +#define percpu_cmpxchg64_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + u64 var; \ + struct { \ + u32 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg8b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, , pcp, oval, nval) +#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg64_op(8, volatile, pcp, oval, nval) +#endif + +#ifdef CONFIG_X86_64 +#define raw_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, , pcp, oval, nval); +#define this_cpu_cmpxchg64(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval); + +#define percpu_cmpxchg128_op(size, qual, _var, _oval, _nval) \ +({ \ + union { \ + u128 var; \ + struct { \ + u64 low, high; \ + }; \ + } old__, new__; \ + \ + old__.var = _oval; \ + new__.var = _nval; \ + \ + asm qual ("cmpxchg16b " __percpu_arg([var]) \ + : [var] "+m" (_var), \ + "+a" (old__.low), \ + "+d" (old__.high) \ + : "b" (new__.low), \ + "c" (new__.high) \ + : "memory"); \ + \ + old__.var; \ +}) + +#define raw_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, , pcp, oval, nval) +#define this_cpu_cmpxchg128(pcp, oval, nval) percpu_cmpxchg128_op(16, volatile, pcp, oval, nval) +#endif + /* * this_cpu_read() makes gcc load the percpu variable every time it is * accessed while this_cpu_read_stable() allows the value to be cached. --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -298,6 +298,14 @@ do { \ #define raw_cpu_cmpxchg_8(pcp, oval, nval) \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef raw_cpu_cmpxchg64 +#define raw_cpu_cmpxchg64(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef raw_cpu_cmpxchg128 +#define raw_cpu_cmpxchg128(pcp, oval, nval) \ + raw_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef raw_cpu_cmpxchg_double_1 #define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ @@ -423,6 +431,14 @@ do { \ #define this_cpu_cmpxchg_8(pcp, oval, nval) \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif +#ifndef this_cpu_cmpxchg64 +#define this_cpu_cmpxchg64(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif +#ifndef this_cpu_cmpxchg128 +#define this_cpu_cmpxchg128(pcp, oval, nval) \ + this_cpu_generic_cmpxchg(pcp, oval, nval) +#endif #ifndef this_cpu_cmpxchg_double_1 #define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ From patchwork Mon May 15 07:57:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74C0EC77B7D for ; Mon, 15 May 2023 08:07:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AEFB0280001; Mon, 15 May 2023 04:06:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7421280009; Mon, 15 May 2023 04:06:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B784280001; Mon, 15 May 2023 04:06:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 33D6A280006 for ; Mon, 15 May 2023 04:06:56 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 04147121394 for ; Mon, 15 May 2023 08:06:55 +0000 (UTC) X-FDA: 80791758432.22.B752D7D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 3B7F5A0004 for ; Mon, 15 May 2023 08:06:54 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="VA/C0Cmc"; spf=none (imf25.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138014; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=rgZ7hZWkHDtTpF84Xw00Ifyh/7yejqO/hjYHP4KrVyk=; b=DtvBGVw4DY7JsnZZeZGXkG2If7nLoMPm8HdeTm6Jgf/2YoMyHcncbx6PolOydXqcyrOInB n4cO7N/kofqruMWZ76bLe7JSt+haHXj6BFl+GxoRpBJ+pO5Ydc07vVIh9lsvlJPlyu1Lz3 4UNBjM8HnS7DmKfBdKc/hD5kmO5AneI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138014; a=rsa-sha256; cv=none; b=3sntECuwFS1myfdE8/VBRCEErAHOx++IzVQzhYYIhu8OBUWkp4/6yGaZJiaw9Ne+erpcLz ito/Vckjzv3Mq1MIF13Fn48PYYjbfi/gkSfPOe3F1R53dxT51p+PPm3FOTOVH0waQ8SraQ eGnmd39OGmm9ojsSsMaNGzEY6rgXbQE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="VA/C0Cmc"; spf=none (imf25.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=rgZ7hZWkHDtTpF84Xw00Ifyh/7yejqO/hjYHP4KrVyk=; b=VA/C0CmchcfXpbAczSsPgonOyt 2aUt/agGqPFIFpZ6l2I6Qg5nqdJuz9IoNMj9V/ozfrCxRR6fb5YEf1Ga0kiJRyftM4e6pWjqWSKkc dIb0BRa5UE1aFif3qoM2MZ9k3ki3NvKxYPbB/R6/S547Pj10WkYXT/Q4uPhD0UtHAZXSrtLCyzGlN kb1o1hAHxW8lgC6PteRUH9iYuSUB3Gmflm7Y6x74H7yhLbWGQ2MymwpDc7/LOZXgNN8aLC7xVvmQ7 r4CZEX4BQYELzBYNeKfUDIJMu3CGkBsiIghUyTmTjEZk9+BAmwahfI7LbKKQJ31UK8rF4TV1yz3Wq y8BpnERg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pyTDl-003HUW-Ez; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 872123033CC; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C2169202FCEA4; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.315901115@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:05 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org, Vasant Hegde Subject: [PATCH v3 06/11] x86,amd_iommu: Replace cmpxchg_double() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: xttj9tz4rd4o45fxk38c3ms3fqdxy6of X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 3B7F5A0004 X-HE-Tag: 1684138013-59949 X-HE-Meta: U2FsdGVkX1+8UWwyQMP7ffmSjuA+qx45n/IX6cM9F0IvGqiMpsgeVMOQoJVxln5iBIpDMNUTFI8YKcBjhTAjT/OxMt4VEI3AMxHc9h9gtyHdEAPf1BB3LBIvOsAm5Nn3p/EVv0fii5Vuh0DoiwI0lHhw9PD7XTX5E45irLMR0wjhSxWspHju9lXMD3VHAHzI7laWlvoUASD7qkuA1BYnIrZ5Lnv8Ev/spr7iE5iOdJzqO+kfuvM8SrYrvzT8pV82fBd9f3/0y/rQguamCW2PBe/LYMva7HhYvJ/sqUiJ2JjeNKQ7YCfiAdFCyx0VdUHar+vsldqH7aUnDsH6T+FEE0kt730m91dU3EvofGu8DBLZiNxOzEFS2bnwZMlLL2ilfM4DL0wZzGkbaiPFieVUn97oPI2TGaZWO0R8WxdwUlNUyrEF9AJVT20hvttPfjW3t3G0r+N3vKLnvYVk2daRuaqtJigMOGgAJ8YWWzDcW/bYxXwA3DsgswJcp+v391U0JutW3h1U1Dz1MThvrtWZVi4NJVMK0AA2VmOLXET3Vy/WqwFnK8xGReFOJrl4zM318gSt70qCuNOY+t4U7MJY60Hlc9sDKOW/4d3XMgBmzult+hCUcTLiPrnyBT4MKowjlL0Gm1ncIkrvLjPosgiMmN/KH005N4J9YisKyJCr+IZsh9nSZk0nbCdQDjERid69C3s3fNgyc1d+gf010++/ss1REXIV50uTlnFPVBUcZ+2DP5sTYnVcPat48rJy4jX3j5oYJm7jb1PSFzVQJbO6bCbeSfLP2RPgiViYXp56tIb9njnF+Dm6ci+HIvcISlIR23QwvWwcrd9x1ZR1QmslzS1r/Xgj9MVVd0fZlNAteGPWbb9QkGb2uxM0YqM34s1T2D7Kj9+9SfLc97SZBNxExAqfsrptPGcO3y2TS8QO/mbNgJHU/nKadJFPCuxkXH0yLEyLEUdEhyqdNPdB7c7 SdwvVOVJ YcESTPXKK9jR40gmeq0bRGrxyZqVo1X2w7nAdeetJDnWlYhWaR3E7REwx+z3Wf+aagVSRPKjD935ZQ/+r/wQxx4rfXbPlW5PONzayba1LUlkU5AEb2DpZ1i6Eke6BtN5lwNirhZ7BA5HHTcdoN3pitJjBrImYm1Oe3J6MCIfZGwZNmGx216PM65PpedO3GPSOlsFnvOTDR87GnSu/yrMJVmQwlFSnVxcBC+8BB99X1CpeQwkgvVYf5NcD+yMRLC+bI0v4xkaAsAOVgpZQ3SXXCx0uCPYrzJMp7Whj0n8opqtxnGqnFVcTkbDxO0ogR8/61egZYSc9J7KNFJoEyuevcud0najBbYPF+nG6/F2Bt7BTfbXY1DJ9eOUDUnKn4TP3vihUvwhBKc8YVrrR6pE8nwImZFyW3Mvk2m91 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) Tested-by: Vasant Hegde --- drivers/iommu/amd/amd_iommu_types.h | 9 +++++++-- drivers/iommu/amd/iommu.c | 10 ++++------ 2 files changed, 11 insertions(+), 8 deletions(-) --- a/drivers/iommu/amd/amd_iommu_types.h +++ b/drivers/iommu/amd/amd_iommu_types.h @@ -986,8 +986,13 @@ union irte_ga_hi { }; struct irte_ga { - union irte_ga_lo lo; - union irte_ga_hi hi; + union { + struct { + union irte_ga_lo lo; + union irte_ga_hi hi; + }; + u128 irte; + }; }; struct irq_2_irte { --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -3003,10 +3003,10 @@ static int alloc_irq_index(struct amd_io static int modify_irte_ga(struct amd_iommu *iommu, u16 devid, int index, struct irte_ga *irte, struct amd_ir_data *data) { - bool ret; struct irq_remap_table *table; - unsigned long flags; struct irte_ga *entry; + unsigned long flags; + u128 old; table = get_irq_table(iommu, devid); if (!table) @@ -3017,16 +3017,14 @@ static int modify_irte_ga(struct amd_iom entry = (struct irte_ga *)table->table; entry = &entry[index]; - ret = cmpxchg_double(&entry->lo.val, &entry->hi.val, - entry->lo.val, entry->hi.val, - irte->lo.val, irte->hi.val); /* * We use cmpxchg16 to atomically update the 128-bit IRTE, * and it cannot be updated by the hardware or other processors * behind us, so the return value of cmpxchg16 should be the * same as the old value. */ - WARN_ON(!ret); + old = entry->irte; + WARN_ON(!try_cmpxchg128(&entry->irte, &old, irte->irte)); if (data) data->ref = entry; From patchwork Mon May 15 07:57:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2BD7C77B75 for ; Mon, 15 May 2023 08:07:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BC86280006; Mon, 15 May 2023 04:06:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6810A280008; Mon, 15 May 2023 04:06:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4BF6280001; Mon, 15 May 2023 04:06:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6EE99280001 for ; Mon, 15 May 2023 04:06:55 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 40BB416130C for ; Mon, 15 May 2023 08:06:55 +0000 (UTC) X-FDA: 80791758390.06.7F935A0 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf05.hostedemail.com (Postfix) with ESMTP id 67386100011 for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=EaiXXa+J; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138012; a=rsa-sha256; cv=none; b=K0xTnVgWXnhKPHAlv+LYsWhNHqyT0NN+2MXkxNRzTqCtAxlLR0trf2kFI3Qhy9I9Mv6n9E dcg5LErjqgPh8e0I3g49/0d6Vc2NKynqqm7z8gV6XlXnnQLiWROwybff/Bm5APV70IPg65 i0FvExKVlimVpuHCPm45qLJS2RRTU8E= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=EaiXXa+J; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=/ezz9F43epJmjX+2+q2EuT6G37yn9BJ4CMV+iiE+KIk=; b=qMs40ciD4VwylVrWgLk004OgvatfJJW94N6XqeSK3WHygwqF5Nb7mvxsNkRj3zoSSgGvjG zzOBKDGuoolrkktqlgmZF5VMs9C0L9Z4ZRTw/M62HlCcajugrAq10pgfnYO3PxPEX/QYln H8ZMkc1bh1bKGJbezpCs4Si4ObO6CBA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=/ezz9F43epJmjX+2+q2EuT6G37yn9BJ4CMV+iiE+KIk=; b=EaiXXa+JLNkopxcm154uvINSoT XXhhxSNYI9Dj8TJsMePYGsmPawAk4PgxqA24LoC8ToY8T+UvgX6Mt1Iok43XBxw17lZCw5OZdQ2tQ ZaaSSGuhzCCsuKXfTSd/sC/q9v1c7d0frobCxDxfEp+AjNlpv5Qz2smjLvBY1tflTjx41hwRaKkev GYei11HqZz4Lpne0uwvABdXqI3PMUjfJYYIBLxOseh77Y/9EJ3pflT5Yk/oyQZ6FSRQ1NCjm4r6UF NHWZLaCOz+6hbMv+Kar15kRPnKaI+ssxzGQ048NySZiovPqlPKgHK6tfNCL3XXal7p8b7Oo4dmUEi /ipC8uhw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDl-00BQN4-1M; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 87106302FB8; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C75F2202FCEA7; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.382778664@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:06 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 07/11] x86,intel_iommu: Replace cmpxchg_double() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 67386100011 X-Stat-Signature: euhynbq7pdz31a4emunemgsp3cbgffbp X-HE-Tag: 1684138012-394616 X-HE-Meta: U2FsdGVkX1+gCufDXUJlJldWph5T6J0++yjhFcihBmRgkLwxOlNtOa22jNq5rv4ozblfuuTSu+GpcfkqBdjARGptGrI6SSVV+8Z8ooUX9T2uMvhfoUM1DqBWk1Wz7f0AO80+O6Y0ymOUD+HvvqY9Lu15oAZPiOmFrlW5MK6FdKWnJGlT1B5/QKHVImncprbtGI62rWKly3V4lS/xGmb0cofgpLN9fhxb0z3KJ1ZRg7C5OACquX0SxxApxRjNYtYk/4+Ld3dGrnKOuUd0SUt0wc/ouW48TBZp0tdNbHP1nPpl/4GcHICaw1UlRDnWRF0sfEj7oLf6Yl/5S31gWMknAZl4v1GEGUZ3tcL0Vw0fdh+zhUI5RygyqbSIoYZy2o4tu2cLhPxULPc6ZnpmjqFh1ZzOtsMN1jGhoGhO2n1/6C9g+tmM18KZCBWKo7Ko3gQMYmOD2T5/PHa4A8EZPhmG2eiIq3xHkpTTqDpLES8Qeh+SZqCTPPGc4U0oSPb7/0MMI9m5gxpnEcDQl6/fBmqLxw7t6P4Zw5u8jzRtjFc3kAL1VOp2m3MePIroBQFIsGpklwoQH0zRHXPVpmnfBfnMeCcj57keluZrGakpw+PP810CqAl/9WOFoXTcxrD1GDwnP+BhvWYq+pW3KxaMTc4oAhJiMXMWg8c+te9UFrwc8apMmkS0mkb5p3DvfISwtiYaAYuUMWRbHvF8ZJwex+ZA/hji0NycIu7sOkaozJ91KTHrjfcW2vlXjfg5v2zPyqngB3aAXQHahVHfp2ttHkTie/vAsVBucBYGYyaGPBIDR7KBlWIYnYDXoOVFf/b10oLsu5OvovOVMbP0YqTIMLefUqjH1j+XRxfs08IL44QaGFg+Hr3l7dfpWXtngNZ2HtUykRNkDCHU2osQFFQQbM9rN4vt6LXrj/qopC2y3vIPcSPBXQa40+jh922CR3XQsn/Xk8DhjuqSI53EOK2nck+ 9HBN8fLF pONvUjBBx1qCb3MRAa5sOCuTM9VHwEAwpN5vbN/4AvbEfZ0D+Sy/mT8f7RRk5WnMp70lyq84t/RJEJzzfndu8CzZP5PMak3j/ZopiqIgkKZ42MjajSk5aXdFyKImgoJHw3qrB070hTYWNBFCGgpJCyVT2+319+bAI9P4aHLH1yNA+qzrS7xpNhkDPlyhzXWwPyqaGfjcyYmw2ymzJ+iQJTi6SAyS6SHNIDqwDey1HnR4FnBd6yfglQybNMg0WlHff+lutBxZFCw+mIBJ7WiVlBilk8YgvBqfllHGuiZhLnvPjU1pxZBkuieolgY8g3uQWIauSvj7n/AYJeQoSJuoIrEf4Ta/hRklm8tx/TqF5VUyGaVz+7REfDmVChiof2E2E9Hn3EE4Ih5DEZ8iMEtl5IIU2Qf80lhc6U2gk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Lu Baolu --- drivers/iommu/intel/irq_remapping.c | 8 -- include/linux/dmar.h | 125 +++++++++++++++++++----------------- 2 files changed, 68 insertions(+), 65 deletions(-) --- a/drivers/iommu/intel/irq_remapping.c +++ b/drivers/iommu/intel/irq_remapping.c @@ -174,18 +174,14 @@ static int modify_irte(struct irq_2_iomm irte = &iommu->ir_table->base[index]; if ((irte->pst == 1) || (irte_modified->pst == 1)) { - bool ret; - - ret = cmpxchg_double(&irte->low, &irte->high, - irte->low, irte->high, - irte_modified->low, irte_modified->high); /* * We use cmpxchg16 to atomically update the 128-bit IRTE, * and it cannot be updated by the hardware or other processors * behind us, so the return value of cmpxchg16 should be the * same as the old value. */ - WARN_ON(!ret); + u128 old = irte->irte; + WARN_ON(!try_cmpxchg128(&irte->irte, &old, irte_modified->irte)); } else { WRITE_ONCE(irte->low, irte_modified->low); WRITE_ONCE(irte->high, irte_modified->high); --- a/include/linux/dmar.h +++ b/include/linux/dmar.h @@ -201,67 +201,74 @@ static inline void detect_intel_iommu(vo struct irte { union { - /* Shared between remapped and posted mode*/ struct { - __u64 present : 1, /* 0 */ - fpd : 1, /* 1 */ - __res0 : 6, /* 2 - 6 */ - avail : 4, /* 8 - 11 */ - __res1 : 3, /* 12 - 14 */ - pst : 1, /* 15 */ - vector : 8, /* 16 - 23 */ - __res2 : 40; /* 24 - 63 */ + union { + /* Shared between remapped and posted mode*/ + struct { + __u64 present : 1, /* 0 */ + fpd : 1, /* 1 */ + __res0 : 6, /* 2 - 6 */ + avail : 4, /* 8 - 11 */ + __res1 : 3, /* 12 - 14 */ + pst : 1, /* 15 */ + vector : 8, /* 16 - 23 */ + __res2 : 40; /* 24 - 63 */ + }; + + /* Remapped mode */ + struct { + __u64 r_present : 1, /* 0 */ + r_fpd : 1, /* 1 */ + dst_mode : 1, /* 2 */ + redir_hint : 1, /* 3 */ + trigger_mode : 1, /* 4 */ + dlvry_mode : 3, /* 5 - 7 */ + r_avail : 4, /* 8 - 11 */ + r_res0 : 4, /* 12 - 15 */ + r_vector : 8, /* 16 - 23 */ + r_res1 : 8, /* 24 - 31 */ + dest_id : 32; /* 32 - 63 */ + }; + + /* Posted mode */ + struct { + __u64 p_present : 1, /* 0 */ + p_fpd : 1, /* 1 */ + p_res0 : 6, /* 2 - 7 */ + p_avail : 4, /* 8 - 11 */ + p_res1 : 2, /* 12 - 13 */ + p_urgent : 1, /* 14 */ + p_pst : 1, /* 15 */ + p_vector : 8, /* 16 - 23 */ + p_res2 : 14, /* 24 - 37 */ + pda_l : 26; /* 38 - 63 */ + }; + __u64 low; + }; + + union { + /* Shared between remapped and posted mode*/ + struct { + __u64 sid : 16, /* 64 - 79 */ + sq : 2, /* 80 - 81 */ + svt : 2, /* 82 - 83 */ + __res3 : 44; /* 84 - 127 */ + }; + + /* Posted mode*/ + struct { + __u64 p_sid : 16, /* 64 - 79 */ + p_sq : 2, /* 80 - 81 */ + p_svt : 2, /* 82 - 83 */ + p_res3 : 12, /* 84 - 95 */ + pda_h : 32; /* 96 - 127 */ + }; + __u64 high; + }; }; - - /* Remapped mode */ - struct { - __u64 r_present : 1, /* 0 */ - r_fpd : 1, /* 1 */ - dst_mode : 1, /* 2 */ - redir_hint : 1, /* 3 */ - trigger_mode : 1, /* 4 */ - dlvry_mode : 3, /* 5 - 7 */ - r_avail : 4, /* 8 - 11 */ - r_res0 : 4, /* 12 - 15 */ - r_vector : 8, /* 16 - 23 */ - r_res1 : 8, /* 24 - 31 */ - dest_id : 32; /* 32 - 63 */ - }; - - /* Posted mode */ - struct { - __u64 p_present : 1, /* 0 */ - p_fpd : 1, /* 1 */ - p_res0 : 6, /* 2 - 7 */ - p_avail : 4, /* 8 - 11 */ - p_res1 : 2, /* 12 - 13 */ - p_urgent : 1, /* 14 */ - p_pst : 1, /* 15 */ - p_vector : 8, /* 16 - 23 */ - p_res2 : 14, /* 24 - 37 */ - pda_l : 26; /* 38 - 63 */ - }; - __u64 low; - }; - - union { - /* Shared between remapped and posted mode*/ - struct { - __u64 sid : 16, /* 64 - 79 */ - sq : 2, /* 80 - 81 */ - svt : 2, /* 82 - 83 */ - __res3 : 44; /* 84 - 127 */ - }; - - /* Posted mode*/ - struct { - __u64 p_sid : 16, /* 64 - 79 */ - p_sq : 2, /* 80 - 81 */ - p_svt : 2, /* 82 - 83 */ - p_res3 : 12, /* 84 - 95 */ - pda_h : 32; /* 96 - 127 */ - }; - __u64 high; +#ifdef CONFIG_IRQ_REMAP + __u128 irte; +#endif }; }; From patchwork Mon May 15 07:57:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240936 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D012C77B7D for ; Mon, 15 May 2023 08:07:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5553280003; Mon, 15 May 2023 04:06:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BADA9280006; Mon, 15 May 2023 04:06:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DF21280003; Mon, 15 May 2023 04:06:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3D367280001 for ; Mon, 15 May 2023 04:06:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 12DBB161335 for ; Mon, 15 May 2023 08:06:55 +0000 (UTC) X-FDA: 80791758390.28.7D6AD5D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id E13324000A for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=pfsV7Qbq; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=1Hyxt0TX1t/0uiattJSuj2ZTNM+BZCXyokp3fCNZGCs=; b=08bgc3MBjhv+c2zOkxcmQ9sy0gvdOWmC+D+Xllu3I3dCeH7NDDmK2ei/nHCGu5cpPVdArB nIlVoAyeRLDB+mFUYUL7/jxuZZcW5Wb+UjIDEYw96CdVZlG29bABHejF/olNw0ABdmgQsa RnQctatr8VgI4/CuCtmDfyyxpN5GpbU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138013; a=rsa-sha256; cv=none; b=zWjh2pafL2uKGwArDJ2CnH+EdTcpYnqbeEsoz345ZE2KwXJFuX/p/vDsjZyb7juPNtjseS AqvJbbhvBpz4egXFN47nw5OaZ2YX9CSz32CiKWLN8kBBaTTb6LsA7r9DJVVf1i4AEoSUda U4oed7Y6I09owbk74tNT3WUdg0qkrfo= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=pfsV7Qbq; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=1Hyxt0TX1t/0uiattJSuj2ZTNM+BZCXyokp3fCNZGCs=; b=pfsV7Qbq81Z0IkI+Xvza6f76LG geyP1yqSpQ0cdv4M5vnGLyJVao4N1TuqaCz9CzumRH4FSACCzlrD9SnOzgfbEEzEvlPz+a1s+R1SD XU6IhOkvFQJ93yyWdsEupR2Mqx52rhYjF0Seid0p6zqtjuBqiFf3cZqv6TBGDQ8ShnrRoZ97CPad8 +l+HuRdyiowbD/zkPhjH4ZDmYadt9tcyGKRYa7V3/pNYx2IcgbVK+CnGZ/uUl0ZtJbAq33shpR5c0 1ZAI8rrHM/oDGodef2VoDHWu43MSu7ivZC62mvy5lcoU6nvfYdh41GMGxjhG5dmJecYvR0ryroQ/Y r3AbSCuA==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pyTDl-003HUT-CX; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 870AD302680; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id CBFAF202FCEA6; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.453785148@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:07 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 08/11] slub: Replace cmpxchg_double() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E13324000A X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: 8gc4r9f8e1achjx1n9inxxbfq48aynbg X-HE-Tag: 1684138012-893557 X-HE-Meta: U2FsdGVkX1/hBmmz8Z3C9ioHqreQ9wI6kS8dQ0S5OaSZ5M85jedwhq2RJQKqgAg3P98hceHZAWrAY8z6qJljdBQYsHUa9QecS+YXIMWPah6KJ4+CphGwzWZ9OT0t3qNCT3Fj6G+1nw/IOay/T8qw/TnbCNUJS76sbfOo5LB2GZMRnVbQst1k/Vz50RQ0xicAERJkZmSSpqmIIMGWrLX6mJ4BzTbbYpn6xJxvHdz5m4wfEutsVP4x/97Z4Dy0CEp8uQJAT4/pqiiUe63JVOEfCh5z9ounIrhrQLwI69zaZGOpsPtLqq7it18cMUIfzVmlRNo9tKh1mLybIeK+gasg63DSfsfP7IL3fwLMXYUN7YF7mztmvdgux5SHBLSUZXcN72p0a/44gWdwZH80ujLdtkdTGecU7LnzE/Ofbj/ydCxzJaAhMFxCCxkmwdMLlgvmorCC1uDKW9Qtv25eTD/omSwrgwmSdTznoKg2xYZ9+ug8zXsp5bO41MuuV3ldyaYm16/TS6OrTvPdgUMc2d11mPLtNsiAPKGMtkJB74J6uW7vE2G+jspaqxXdq6e8ycFwvPZoThmSq37+NWKanNM5k6h2Uho0SWYtH2z2bPcFHUs7fOTCfBZXTxUcWhm8fWkxbK9Imy2z1es1IDMdvJ8h4oLG7fxk1Eh1G1NVcp1jGRokIf6R7C+2A4n6kv5A+U8CSzFbEdFYG3H3sbXdonIrjz0SV4elxTdPwL/p+cnTuyEMqOgFjHg9Eixe4eGggEXp3i6jegD/VujONHI361oKwzP5PPJJcKuzTNV37HecKL4qtrEiAS6xOYaCF7W5S8y1jWHeRDHQzfA36RLeyugWnArm1ulHoXJ7waVc+rDr28mfjILElvqR9ujaEKrfq/zEDo1a7Ww1Odb4w1eXl9EYs54iqSr3of4WWC0Cfdrjx8BUCYi9bNBXqF+iDVTZfDyO+OHj5IsM+93ZT1h2u7c yAsjviJf Ri2z5tkBRbOFqZ7uyHwytiYMDU5iyIqRZTv4Y/gOtgBXXDxW3Y/5elqMoVB1E/0/1zvegN404ssMXXSoXwxroUY9R3jPWgRokxYNsDDDl9aUzX6V5zL5+FVvyq27Lb1Ngw8DISnA20Z1Tf/t89zWqS2S4VsQF5i14kC2Mu3xqOGBamw4j2vK458nu5eMeISHBMy2I6vRh4iKiYV1HNknW8cq1GwESkZ1MwickwD8GJIEQjGiC1Kayab6/AFxPHEDjnu8x3T2YAg7Iy0meea9vgQvRo1dXDxkJTdutZuDc0NieiPh5zUipDu9Ih67Px03gCtSQhsRZ4JcQMdG/YzRT128507V75Rbk/SHbA14XkgGgAmlxZXUfjU6qjPzCGaVmrOhKX0Y5S+xhVJKJWb1dgLeZ5UuPJ2PD+/Ma X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Signed-off-by: Peter Zijlstra (Intel) Acked-by: Vlastimil Babka Acked-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/slub_def.h | 12 ++- mm/slab.h | 49 ++++++++++++++-- mm/slub.c | 143 ++++++++++++++++++++++++++++------------------- 3 files changed, 138 insertions(+), 66 deletions(-) --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -39,7 +39,8 @@ enum stat_item { CPU_PARTIAL_FREE, /* Refill cpu partial on free */ CPU_PARTIAL_NODE, /* Refill cpu partial from node partial */ CPU_PARTIAL_DRAIN, /* Drain cpu partial to node partial */ - NR_SLUB_STAT_ITEMS }; + NR_SLUB_STAT_ITEMS +}; #ifndef CONFIG_SLUB_TINY /* @@ -47,8 +48,13 @@ enum stat_item { * with this_cpu_cmpxchg_double() alignment requirements. */ struct kmem_cache_cpu { - void **freelist; /* Pointer to next available object */ - unsigned long tid; /* Globally unique transaction id */ + union { + struct { + void **freelist; /* Pointer to next available object */ + unsigned long tid; /* Globally unique transaction id */ + }; + freelist_aba_t freelist_tid; + }; struct slab *slab; /* The slab from which we are allocating */ #ifdef CONFIG_SLUB_CPU_PARTIAL struct slab *partial; /* Partially allocated frozen slabs */ --- a/mm/slab.h +++ b/mm/slab.h @@ -6,6 +6,38 @@ */ void __init kmem_cache_init(void); +#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE +#ifdef CONFIG_64BIT +# ifdef system_has_cmpxchg128 +# define system_has_freelist_aba() system_has_cmpxchg128() +# define try_cmpxchg_freelist try_cmpxchg128 +# define this_cpu_cmpxchg_freelist this_cpu_cmpxchg128 +typedef u128 freelist_full_t; +# endif +#else /* CONFIG_64BIT */ +# ifdef system_has_cmpxchg64 +# define system_has_freelist_aba() system_has_cmpxchg64() +# define try_cmpxchg_freelist try_cmpxchg64 +# define this_cpu_cmpxchg_freelist this_cpu_cmpxchg64 +typedef u64 freelist_full_t; +# endif +#endif /* CONFIG_64BIT */ +#endif /* CONFIG_HAVE_ALIGNED_STRUCT_PAGE */ + +/* + * Freelist pointer and counter to cmpxchg together, avoids the typical ABA + * problems with cmpxchg of just a pointer. + */ +typedef union { +#ifdef system_has_freelist_aba + struct { + void *freelist; + unsigned long counter; + }; + freelist_full_t full; +#endif +} freelist_aba_t; + /* Reuses the bits in struct page */ struct slab { unsigned long __page_flags; @@ -38,14 +70,19 @@ struct slab { #endif }; /* Double-word boundary */ - void *freelist; /* first free object */ union { - unsigned long counters; struct { - unsigned inuse:16; - unsigned objects:15; - unsigned frozen:1; + void *freelist; /* first free object */ + union { + unsigned long counters; + struct { + unsigned inuse:16; + unsigned objects:15; + unsigned frozen:1; + }; + }; }; + freelist_aba_t freelist_counter; }; }; struct rcu_head rcu_head; @@ -72,7 +109,7 @@ SLAB_MATCH(memcg_data, memcg_data); #endif #undef SLAB_MATCH static_assert(sizeof(struct slab) <= sizeof(struct page)); -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && defined(CONFIG_SLUB) +#if defined(system_has_freelist_aba) && defined(CONFIG_SLUB) static_assert(IS_ALIGNED(offsetof(struct slab, freelist), 2*sizeof(void *))); #endif --- a/mm/slub.c +++ b/mm/slub.c @@ -292,7 +292,12 @@ static inline bool kmem_cache_has_cpu_pa /* Poison object */ #define __OBJECT_POISON ((slab_flags_t __force)0x80000000U) /* Use cmpxchg_double */ + +#ifdef system_has_freelist_aba #define __CMPXCHG_DOUBLE ((slab_flags_t __force)0x40000000U) +#else +#define __CMPXCHG_DOUBLE ((slab_flags_t __force)0U) +#endif /* * Tracking user of a slab. @@ -512,6 +517,40 @@ static __always_inline void slab_unlock( __bit_spin_unlock(PG_locked, &page->flags); } +static inline bool +__update_freelist_fast(struct slab *slab, + void *freelist_old, unsigned long counters_old, + void *freelist_new, unsigned long counters_new) +{ +#ifdef system_has_freelist_aba + freelist_aba_t old = { .freelist = freelist_old, .counter = counters_old }; + freelist_aba_t new = { .freelist = freelist_new, .counter = counters_new }; + + return try_cmpxchg_freelist(&slab->freelist_counter.full, &old.full, new.full); +#else + return false; +#endif +} + +static inline bool +__update_freelist_slow(struct slab *slab, + void *freelist_old, unsigned long counters_old, + void *freelist_new, unsigned long counters_new) +{ + bool ret = false; + + slab_lock(slab); + if (slab->freelist == freelist_old && + slab->counters == counters_old) { + slab->freelist = freelist_new; + slab->counters = counters_new; + ret = true; + } + slab_unlock(slab); + + return ret; +} + /* * Interrupts must be disabled (for the fallback code to work right), typically * by an _irqsave() lock variant. On PREEMPT_RT the preempt_disable(), which is @@ -519,33 +558,25 @@ static __always_inline void slab_unlock( * allocation/ free operation in hardirq context. Therefore nothing can * interrupt the operation. */ -static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab, +static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slab, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, const char *n) { + bool ret; + if (USE_LOCKLESS_FAST_PATH()) lockdep_assert_irqs_disabled(); -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ - defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) + if (s->flags & __CMPXCHG_DOUBLE) { - if (cmpxchg_double(&slab->freelist, &slab->counters, - freelist_old, counters_old, - freelist_new, counters_new)) - return true; - } else -#endif - { - slab_lock(slab); - if (slab->freelist == freelist_old && - slab->counters == counters_old) { - slab->freelist = freelist_new; - slab->counters = counters_new; - slab_unlock(slab); - return true; - } - slab_unlock(slab); + ret = __update_freelist_fast(slab, freelist_old, counters_old, + freelist_new, counters_new); + } else { + ret = __update_freelist_slow(slab, freelist_old, counters_old, + freelist_new, counters_new); } + if (likely(ret)) + return true; cpu_relax(); stat(s, CMPXCHG_DOUBLE_FAIL); @@ -557,36 +588,26 @@ static inline bool __cmpxchg_double_slab return false; } -static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct slab *slab, +static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, const char *n) { -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ - defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) + bool ret; + if (s->flags & __CMPXCHG_DOUBLE) { - if (cmpxchg_double(&slab->freelist, &slab->counters, - freelist_old, counters_old, - freelist_new, counters_new)) - return true; - } else -#endif - { + ret = __update_freelist_fast(slab, freelist_old, counters_old, + freelist_new, counters_new); + } else { unsigned long flags; local_irq_save(flags); - slab_lock(slab); - if (slab->freelist == freelist_old && - slab->counters == counters_old) { - slab->freelist = freelist_new; - slab->counters = counters_new; - slab_unlock(slab); - local_irq_restore(flags); - return true; - } - slab_unlock(slab); + ret = __update_freelist_slow(slab, freelist_old, counters_old, + freelist_new, counters_new); local_irq_restore(flags); } + if (likely(ret)) + return true; cpu_relax(); stat(s, CMPXCHG_DOUBLE_FAIL); @@ -2228,7 +2249,7 @@ static inline void *acquire_slab(struct VM_BUG_ON(new.frozen); new.frozen = 1; - if (!__cmpxchg_double_slab(s, slab, + if (!__slab_update_freelist(s, slab, freelist, counters, new.freelist, new.counters, "acquire_slab")) @@ -2554,7 +2575,7 @@ static void deactivate_slab(struct kmem_ } - if (!cmpxchg_double_slab(s, slab, + if (!slab_update_freelist(s, slab, old.freelist, old.counters, new.freelist, new.counters, "unfreezing slab")) { @@ -2611,7 +2632,7 @@ static void __unfreeze_partials(struct k new.frozen = 0; - } while (!__cmpxchg_double_slab(s, slab, + } while (!__slab_update_freelist(s, slab, old.freelist, old.counters, new.freelist, new.counters, "unfreezing slab")); @@ -3008,6 +3029,22 @@ static inline bool pfmemalloc_match(stru } #ifndef CONFIG_SLUB_TINY +static inline bool +__update_cpu_freelist_fast(struct kmem_cache *s, + void *freelist_old, void *freelist_new, + unsigned long tid) +{ +#ifdef system_has_freelist_aba + freelist_aba_t old = { .freelist = freelist_old, .counter = tid }; + freelist_aba_t new = { .freelist = freelist_new, .counter = next_tid(tid) }; + + return this_cpu_cmpxchg_freelist(s->cpu_slab->freelist_tid.full, + old.full, new.full) == old.full; +#else + return false; +#endif +} + /* * Check the slab->freelist and either transfer the freelist to the * per cpu freelist or deactivate the slab. @@ -3034,7 +3071,7 @@ static inline void *get_freelist(struct new.inuse = slab->objects; new.frozen = freelist != NULL; - } while (!__cmpxchg_double_slab(s, slab, + } while (!__slab_update_freelist(s, slab, freelist, counters, NULL, new.counters, "get_freelist")); @@ -3359,11 +3396,7 @@ static __always_inline void *__slab_allo * against code executing on this cpu *not* from access by * other cpus. */ - if (unlikely(!this_cpu_cmpxchg_double( - s->cpu_slab->freelist, s->cpu_slab->tid, - object, tid, - next_object, next_tid(tid)))) { - + if (unlikely(!__update_cpu_freelist_fast(s, object, next_object, tid))) { note_cmpxchg_failure("slab_alloc", s, tid); goto redo; } @@ -3631,7 +3664,7 @@ static void __slab_free(struct kmem_cach } } - } while (!cmpxchg_double_slab(s, slab, + } while (!slab_update_freelist(s, slab, prior, counters, head, new.counters, "__slab_free")); @@ -3736,11 +3769,7 @@ static __always_inline void do_slab_free set_freepointer(s, tail_obj, freelist); - if (unlikely(!this_cpu_cmpxchg_double( - s->cpu_slab->freelist, s->cpu_slab->tid, - freelist, tid, - head, next_tid(tid)))) { - + if (unlikely(!__update_cpu_freelist_fast(s, freelist, head, tid))) { note_cmpxchg_failure("slab_free", s, tid); goto redo; } @@ -4505,11 +4534,11 @@ static int kmem_cache_open(struct kmem_c } } -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ - defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) - if (system_has_cmpxchg_double() && (s->flags & SLAB_NO_CMPXCHG) == 0) +#ifdef system_has_freelist_aba + if (system_has_freelist_aba() && !(s->flags & SLAB_NO_CMPXCHG)) { /* Enable fast mode */ s->flags |= __CMPXCHG_DOUBLE; + } #endif /* From patchwork Mon May 15 07:57:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96B30C7EE24 for ; Mon, 15 May 2023 08:06:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 125A3900002; Mon, 15 May 2023 04:06:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C523900004; Mon, 15 May 2023 04:06:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A72DE900002; Mon, 15 May 2023 04:06:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 73AA7900004 for ; Mon, 15 May 2023 04:06:52 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3D87212139F for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) X-FDA: 80791758264.15.A4FE875 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf04.hostedemail.com (Postfix) with ESMTP id 79BE740002 for ; Mon, 15 May 2023 08:06:50 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=XLXPvYSK; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=4kaqfhENJznU2Mt7hBhEZrEbaoSBpAbVRLS0RsxQHdk=; b=3CJl9uudyCfi0J/z+zup3pKGzePHnN/87/NKRbVqVYNXn5Lh5uismL8pvZDPetGneJ+4pZ E4CNh0FpWfefnKksObiiMR74KPYFOQoiHX3jtCoQYc46F5681KgnlL5oy1qVhoCwiruZ4u 0qjPZE9L7zh+e2thunbbk+d/2+LqiaY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138010; a=rsa-sha256; cv=none; b=Gqf55GQ578C5GnDDjFVIaVbXfNa7BqnVEhYGCMfIG6HLowdgr/V8Pa9OzjoVpBWHuWA6+L gYtgF/IhoL4yofCL4VG3VPjZ8nem1U+VX3CxQ39dpyYMCC0XdKmE4O292D/J2BAGBLz6C2 1IWxsCUEIiuRUyA0a49jHIbij32GJgk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=XLXPvYSK; spf=none (imf04.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=4kaqfhENJznU2Mt7hBhEZrEbaoSBpAbVRLS0RsxQHdk=; b=XLXPvYSK56u1FQB+k+bhon/4pv zSD0YSZlWXiZHPUqIUl6Yyjf5NB4UWFAN3n7Oyp5hyeiOUtKZ9mBZT9y0+I9Onvas22ORYZ6Y3maD /QyoXUY4K27SRULt1VqM5bU6j/J3FLtlzDzZjEbqJpQgQUxuBK0FWXTF3USAeGyIQx0wTYpj+p7lO J/i/ZJRbdXbF8MlsCfnRwwzoTdOCq+IXrVCfsrCDR2uvOke13V5Dr4oE/McXFo/bEOt60OgXbLqEc XeXVrkZaRUSBBo2VPSQDMRDARGlBnk/FVOdYNtIj6j7ZjCFKOzNR7PzQQ9P/94djJbRTqJ9LyTCal lulzxa5w==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDl-00BQN5-1u; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 870D5302DA8; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id CF2C4202FCEAA; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.520976397@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:08 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 09/11] mm/slub: Fold slab_update_freelist() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: k67quddgfbrzwk6yfxgtjfsmoiwsmoka X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 79BE740002 X-Rspam-User: X-HE-Tag: 1684138010-652200 X-HE-Meta: U2FsdGVkX19dV9RkJKYxzYszy0411Mm52FZ+qwQCHbx1e6c8MUwikTu/mPYOY8Pi52WtB/ZiYCwSdm8rhcQoGcHyyoSMUX5CfcOwLw109+JTGDoyz2R0IgeQeM3GNlt17illscAPdvhvk4qPS/fdo/fO7zb3wE1T1se3vJlQeHGzlDK5FS94/o2AwY1MOsg1EoMDsxTSRo9MV4HxapdcdFV7HN6TzwRtCtZh+VB+gXUQQolrdKgPMgalG8QKcHEC/25vkGMF17WYGQkwcHlxmeV1ds4anvEIN7HRgpbuCV219lnCjEEvDCWDH+1vQRkDZc+sfCa1pXPcA909j2IWcvNNFstNjfEwR19Ds7K/K0DBm1d1iC8fZ4tUQJOGlkSbxS3OiGUaRIdYnZ/lT31zeCeJUUynCg4Zq7TgsRz08xFW0EWb/6CUBwCGZ/UXaNn7qOVcKdE11J7j4+6g0VO7MoG9eBPLTtNqsmF2XwoyqXWc0PeJhJ6NpJtpkMEVW4znC/IoCGvqQnmKeLdUpksnDplP8lWuxcI9Ldk+Rd3NSgS6Hl+ckyCkHpduUdlDr4hywmAgvkDxTrQ5jNOn5lRlYZ28qtVgwp6Sc1EdYCTxIn2QQn3AMl3Vn8Vpt+w/Mb8nIfWrez9Or67t8BWaDmImYxCZKEQg4CFYIl3ri588wxYTh1Jr2nUP1TzCeCGMDW0eJ6Mlqri0pkfCjw31q8CyNRQdnV2Y2qHpTBc84FiuPUbuVl3dL9gcrSsGXO1bNxYvyumQtGzBipdIhLkI8Yn3HlrAVRjgOjidR2Y5NYnSuaD5nVVF6DrXzsqsPxjDPU4JNOz+BuzW8xJjeN8IGjroSMkPNrFkPRf5g4kllTG+ct8s2K/IDHaMmKoyzXonJ5BBYBJopC1ztiBjADBZ68xtaxaUMbm3U3bni2G/r4GLvRGcFkj7Ywzwbz9w+vJJ1o1JbAGJZi+agsY7wFYZ9X7 UWfLbG70 UnbanhCJJEsI5uixqDsSssnCaJYcWwZalKOdcCtwijXp8Nklq+f2LMmzdis6DMnLF/uG/hfm+7sZv/SF8W8LdtiZ2nMJV1Iwd0NilfjdWUeW3TJgAgzjPxHhvnNOuTcAj/Y8bwyVLaxhp856Wfb4kYksO+xqORivT9p3B9FY/C99Vhfs3FO6AEjljh0CoSvt1QqMoxPZnxb+fOxLiZlFbnJSvoHccOJgYnNiBvTdSwkZ5szc76weCwh9q7vEzrR2tv0p9D977LVDYoQBp5NMQLf9RkQm8nECt+kZCUibehUPiGowT4O/6c8OQ66DHUej75iswxE5bZWnBEoyaeiwD6c7zn5G0HB0G3JhYz9gxlVhmQGBsyS03I7A7eciVYoctLpDxTZZ2khTFCyU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The two functions slab_update_freelist() and __slab_update_freelist() are nearly identical, fold and add a boolean argument and rely on constant propagation. Signed-off-by: Peter Zijlstra (Intel) --- mm/slub.c | 80 +++++++++++++++++++++----------------------------------------- 1 file changed, 28 insertions(+), 52 deletions(-) --- a/mm/slub.c +++ b/mm/slub.c @@ -559,53 +559,29 @@ __update_freelist_slow(struct slab *slab * allocation/ free operation in hardirq context. Therefore nothing can * interrupt the operation. */ -static inline bool __slab_update_freelist(struct kmem_cache *s, struct slab *slab, - void *freelist_old, unsigned long counters_old, - void *freelist_new, unsigned long counters_new, - const char *n) +static __always_inline +bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, + void *freelist_old, unsigned long counters_old, + void *freelist_new, unsigned long counters_new, + bool irq_save, const char *n) { bool ret; - if (USE_LOCKLESS_FAST_PATH()) + if (!irq_save && USE_LOCKLESS_FAST_PATH()) lockdep_assert_irqs_disabled(); if (s->flags & __CMPXCHG_DOUBLE) { ret = __update_freelist_fast(slab, freelist_old, counters_old, freelist_new, counters_new); } else { - ret = __update_freelist_slow(slab, freelist_old, counters_old, - freelist_new, counters_new); - } - if (likely(ret)) - return true; - - cpu_relax(); - stat(s, CMPXCHG_DOUBLE_FAIL); - -#ifdef SLUB_DEBUG_CMPXCHG - pr_info("%s %s: cmpxchg double redo ", n, s->name); -#endif - - return false; -} - -static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, - void *freelist_old, unsigned long counters_old, - void *freelist_new, unsigned long counters_new, - const char *n) -{ - bool ret; - - if (s->flags & __CMPXCHG_DOUBLE) { - ret = __update_freelist_fast(slab, freelist_old, counters_old, - freelist_new, counters_new); - } else { unsigned long flags; - local_irq_save(flags); + if (irq_save) + local_irq_save(flags); ret = __update_freelist_slow(slab, freelist_old, counters_old, freelist_new, counters_new); - local_irq_restore(flags); + if (irq_save) + local_irq_restore(flags); } if (likely(ret)) return true; @@ -2250,10 +2226,10 @@ static inline void *acquire_slab(struct VM_BUG_ON(new.frozen); new.frozen = 1; - if (!__slab_update_freelist(s, slab, - freelist, counters, - new.freelist, new.counters, - "acquire_slab")) + if (!slab_update_freelist(s, slab, + freelist, counters, + new.freelist, new.counters, + false, "acquire_slab")) return NULL; remove_partial(n, slab); @@ -2577,9 +2553,9 @@ static void deactivate_slab(struct kmem_ if (!slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")) { + old.freelist, old.counters, + new.freelist, new.counters, + true, "unfreezing slab")) { if (mode == M_PARTIAL) spin_unlock_irqrestore(&n->list_lock, flags); goto redo; @@ -2633,10 +2609,10 @@ static void __unfreeze_partials(struct k new.frozen = 0; - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); + } while (!slab_update_freelist(s, slab, + old.freelist, old.counters, + new.freelist, new.counters, + false, "unfreezing slab")); if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) { slab->next = slab_to_discard; @@ -3072,10 +3048,10 @@ static inline void *get_freelist(struct new.inuse = slab->objects; new.frozen = freelist != NULL; - } while (!__slab_update_freelist(s, slab, - freelist, counters, - NULL, new.counters, - "get_freelist")); + } while (!slab_update_freelist(s, slab, + freelist, counters, + NULL, new.counters, + false, "get_freelist")); return freelist; } @@ -3666,9 +3642,9 @@ static void __slab_free(struct kmem_cach } } while (!slab_update_freelist(s, slab, - prior, counters, - head, new.counters, - "__slab_free")); + prior, counters, + head, new.counters, + true, "__slab_free")); if (likely(!n)) { From patchwork Mon May 15 07:57:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13240937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51AA0C7EE26 for ; Mon, 15 May 2023 08:07:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 09445280004; Mon, 15 May 2023 04:06:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F060F280008; Mon, 15 May 2023 04:06:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 990AB280004; Mon, 15 May 2023 04:06:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 555B8280005 for ; Mon, 15 May 2023 04:06:55 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2EC90AF366 for ; Mon, 15 May 2023 08:06:55 +0000 (UTC) X-FDA: 80791758390.23.01088AD Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id E5A82100002 for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oExTQSqY; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=A512lmdZ5mu+BsxXuNhkbi41aEPQspSS2JnnTxvGyYs=; b=d5gYhmfa60HjADMmc5kSvOyFVPlkWsxh3iFWfI7ZDMQEzt1WLo080/wTfewfKkqWsu/BTQ otj6NsYj2v4xSwGfGGVSgS2UaPDf4BtyzgIjhHoy62X2y2JyeZirI5WEo3DPtMMHAGSFSa wRTQ0e2hvbMc2+brmQJd7ZwJR+JyCFI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oExTQSqY; spf=none (imf05.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138013; a=rsa-sha256; cv=none; b=iIXe5CPLyhYYzJCOmzj8WbRjz6zZRzPBH7AS5iHcMrhsxzhxfuB77WJ1ebecSfn5djJfdi 9NvWgbS4ug9jdMGStd455d7g6g+ucLIsT4tTNS+OmYgEXZULOs9qWycwdvgaXW97dzKm2f By6ruxiBcnMpIg5dt2iJNjkDy2TaLHY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=A512lmdZ5mu+BsxXuNhkbi41aEPQspSS2JnnTxvGyYs=; b=oExTQSqYYsszhfdd1ylxeG6Q6a tG+d+I5HsF7cIpvCdFkNzi543XG1LNV7upSjhuLCTeHoQtkIqE1s4RRIOaAckIGvkm5Oryhf142XL LhN3/umz0XmDOUxaMFls5DCRmjf38fycbdxzwI48qvCv8QSvCS1reHFBKHmNnsXXZygD3Ygpe/x28 QXWWWfWquxFT5aJynme5K7e9jcFTyZlADDy4RDwJSIxMdcShS+XcLkh0Ppy0gFczCWd9MIhu3RcHN jQT1+d8loKxnQaJeBZXMi87o9/Mn2j+Qur//DsCEwMzzg/2PQt/VmqMZrBOEjnIeaxQZnsZqk9gr9 xMzDSeiQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pyTDl-003HUa-Jl; Mon, 15 May 2023 08:06:17 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8DFA5303434; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D522D202FCEA9; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.589824283@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:09 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 10/11] arch: Remove cmpxchg_double References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: kxkntuxkk5877g4xt1scwxc33kwnc8cj X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: E5A82100002 X-Rspam-User: X-HE-Tag: 1684138012-486853 X-HE-Meta: U2FsdGVkX1+fjZ4ld7IC/omZWQU1mQJsQsPVhIyr+5tg8x38XYg3LhBiJ0sruENqpTQTo5keXifxVVGGOp/s3Rm3/ILXXfLdHzUvHt5EO5HWi18vZ71bUxjQGrBj5qWyMj/z4kVLx2Xpu5lfYLTAnSUWoVFVZRLqLKmfGjyGom6QEDr3rpCTAPlp2EUqCJ8FIuzxojU4xHZuqMXRg12W+RKocw8+IsuowsXZk4mtgqU8iahsoV73Dj5aXRUgxlSpxW6iK64iXbObE56XE/cs9/A7XorHqRIzWCuuC/yM6ZFmGnk8OFpkwNB0QW/VkdSfdTAEO8RlXJfnf7Nd8ktVtrF70T81nwQGGUwC5lHuqxidqVXsTrT25yffiN5ITmd6nSVz7ZojDPfJ1dZRSqFtK3/x7ioChAzXCIdBnPZYBmNLrm48dyrWzBjQiocg9QdHqp2RexZLTlEKVha9c8KvdtSqmcJS66pgBOjSAABA+7LnBVDHPQfr/LHQqlz5uTPyLW9/wFluYEDDUpmBTizt5WQn8RXtM9WJemfGmACxj2ri3Xjizup2ImCZKRyO9qMKQo35Jo2gFnvPbCm4xsJL6A8m1Ad+ixf6rgzHsHJMEoQlchi6hoRC6yNWJuuE9Z4TIl7yVTRjkvAKxonBFamtiNvw80DYliBW1t6qQ7VFAcd1dAGVCQNrhyM3aVsQf+Kes/eif1d1OB6fakJnkbzIagg6wLGS5nJALPqeaYVe6nuekxyqIQzC+8mC/5vF8MoogqT2aaDjtqqCG7+Xvbi09jm4FfM92GdZI64TsdL5hzPU+EFK7qUChcT4biGXE/5vniHUb5gvA/OAQ9dERJO6Q4fxn6g6ctYy3sCuRTS/HBx16OTQVLQxTqjixfgd6IR+CbCp/gv8IXef/gQzkWrpTKeL+AWoavpTq5V7CFEd/aQ1sIxVni/xegNNdP+UKDON2bViAiA/g8tevI9i1yf BxBLUwcf QwwG1lv+xL0jbawZwqyAuaQE3Cb/Eo0ycZL0F6KUuRX3spl0GFa3bwYzVhQiG+aUXXei4Vls7Acvaj6mHNA79ZOMXFCBQBrFupilA268Mt3fPcwZ7b8qfTjitYFJxSpuzrQxdILohWxk38hCWmqMZjVkE1E1HwlxoxngTA/2qNvrWcBxEWoDHIUo4zD3KCG4FVsyz2kuBw2v37x0cKfOPMT0tG3SFBVgBGN1QrHoXD0rZe1di0UCIvv2SepWmMAhEuqOEN613z9221mIn3JryuvG0hQMvt290v2H2ODAiFcbCfK0AKeIOrZhtBvLDXqVhtATQ1zZngg1MfIHHpHJqtLl3mMG5x4MRa+Gu2bnj9AaGov3hxJNd2a66sgDlZwUc2T2OVm3ToDrvyrA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No moar users, remove the monster. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens --- Documentation/core-api/this_cpu_ops.rst | 2 - arch/arm64/include/asm/atomic_ll_sc.h | 33 ---------------- arch/arm64/include/asm/atomic_lse.h | 36 ------------------ arch/arm64/include/asm/cmpxchg.h | 46 ----------------------- arch/arm64/include/asm/percpu.h | 10 ----- arch/s390/include/asm/cmpxchg.h | 34 ----------------- arch/s390/include/asm/percpu.h | 18 --------- arch/x86/include/asm/cmpxchg.h | 25 ------------ arch/x86/include/asm/cmpxchg_32.h | 1 arch/x86/include/asm/cmpxchg_64.h | 1 arch/x86/include/asm/percpu.h | 41 -------------------- include/asm-generic/percpu.h | 58 ----------------------------- include/linux/atomic/atomic-instrumented.h | 17 -------- include/linux/percpu-defs.h | 38 ------------------- scripts/atomic/gen-atomic-instrumented.sh | 15 ++----- 15 files changed, 5 insertions(+), 370 deletions(-) --- a/Documentation/core-api/this_cpu_ops.rst +++ b/Documentation/core-api/this_cpu_ops.rst @@ -53,7 +53,6 @@ are defined. These operations can be use this_cpu_add_return(pcp, val) this_cpu_xchg(pcp, nval) this_cpu_cmpxchg(pcp, oval, nval) - this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) this_cpu_sub(pcp, val) this_cpu_inc(pcp) this_cpu_dec(pcp) @@ -242,7 +241,6 @@ modifies the variable, then RMW actions __this_cpu_add_return(pcp, val) __this_cpu_xchg(pcp, nval) __this_cpu_cmpxchg(pcp, oval, nval) - __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) __this_cpu_sub(pcp, val) __this_cpu_inc(pcp) __this_cpu_dec(pcp) --- a/arch/arm64/include/asm/atomic_ll_sc.h +++ b/arch/arm64/include/asm/atomic_ll_sc.h @@ -294,39 +294,6 @@ __CMPXCHG_CASE( , , mb_, 64, dmb ish, #undef __CMPXCHG_CASE -#define __CMPXCHG_DBL(name, mb, rel, cl) \ -static __always_inline long \ -__ll_sc__cmpxchg_double##name(unsigned long old1, \ - unsigned long old2, \ - unsigned long new1, \ - unsigned long new2, \ - volatile void *ptr) \ -{ \ - unsigned long tmp, ret; \ - \ - asm volatile("// __cmpxchg_double" #name "\n" \ - " prfm pstl1strm, %2\n" \ - "1: ldxp %0, %1, %2\n" \ - " eor %0, %0, %3\n" \ - " eor %1, %1, %4\n" \ - " orr %1, %0, %1\n" \ - " cbnz %1, 2f\n" \ - " st" #rel "xp %w0, %5, %6, %2\n" \ - " cbnz %w0, 1b\n" \ - " " #mb "\n" \ - "2:" \ - : "=&r" (tmp), "=&r" (ret), "+Q" (*(__uint128_t *)ptr) \ - : "r" (old1), "r" (old2), "r" (new1), "r" (new2) \ - : cl); \ - \ - return ret; \ -} - -__CMPXCHG_DBL( , , , ) -__CMPXCHG_DBL(_mb, dmb ish, l, "memory") - -#undef __CMPXCHG_DBL - union __u128_halves { u128 full; struct { --- a/arch/arm64/include/asm/atomic_lse.h +++ b/arch/arm64/include/asm/atomic_lse.h @@ -281,42 +281,6 @@ __CMPXCHG_CASE(x, , mb_, 64, al, "memo #undef __CMPXCHG_CASE -#define __CMPXCHG_DBL(name, mb, cl...) \ -static __always_inline long \ -__lse__cmpxchg_double##name(unsigned long old1, \ - unsigned long old2, \ - unsigned long new1, \ - unsigned long new2, \ - volatile void *ptr) \ -{ \ - unsigned long oldval1 = old1; \ - unsigned long oldval2 = old2; \ - register unsigned long x0 asm ("x0") = old1; \ - register unsigned long x1 asm ("x1") = old2; \ - register unsigned long x2 asm ("x2") = new1; \ - register unsigned long x3 asm ("x3") = new2; \ - register unsigned long x4 asm ("x4") = (unsigned long)ptr; \ - \ - asm volatile( \ - __LSE_PREAMBLE \ - " casp" #mb "\t%[old1], %[old2], %[new1], %[new2], %[v]\n"\ - " eor %[old1], %[old1], %[oldval1]\n" \ - " eor %[old2], %[old2], %[oldval2]\n" \ - " orr %[old1], %[old1], %[old2]" \ - : [old1] "+&r" (x0), [old2] "+&r" (x1), \ - [v] "+Q" (*(__uint128_t *)ptr) \ - : [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \ - [oldval1] "r" (oldval1), [oldval2] "r" (oldval2) \ - : cl); \ - \ - return x0; \ -} - -__CMPXCHG_DBL( , ) -__CMPXCHG_DBL(_mb, al, "memory") - -#undef __CMPXCHG_DBL - #define __CMPXCHG128(name, mb, cl...) \ static __always_inline u128 \ __lse__cmpxchg128##name(volatile u128 *ptr, u128 old, u128 new) \ --- a/arch/arm64/include/asm/cmpxchg.h +++ b/arch/arm64/include/asm/cmpxchg.h @@ -130,22 +130,6 @@ __CMPXCHG_CASE(mb_, 64) #undef __CMPXCHG_CASE -#define __CMPXCHG_DBL(name) \ -static inline long __cmpxchg_double##name(unsigned long old1, \ - unsigned long old2, \ - unsigned long new1, \ - unsigned long new2, \ - volatile void *ptr) \ -{ \ - return __lse_ll_sc_body(_cmpxchg_double##name, \ - old1, old2, new1, new2, ptr); \ -} - -__CMPXCHG_DBL( ) -__CMPXCHG_DBL(_mb) - -#undef __CMPXCHG_DBL - #define __CMPXCHG128(name) \ static inline u128 __cmpxchg128##name(volatile u128 *ptr, \ u128 old, u128 new) \ @@ -211,36 +195,6 @@ __CMPXCHG_GEN(_mb) #define arch_cmpxchg64 arch_cmpxchg #define arch_cmpxchg64_local arch_cmpxchg_local -/* cmpxchg_double */ -#define system_has_cmpxchg_double() 1 - -#define __cmpxchg_double_check(ptr1, ptr2) \ -({ \ - if (sizeof(*(ptr1)) != 8) \ - BUILD_BUG(); \ - VM_BUG_ON((unsigned long *)(ptr2) - (unsigned long *)(ptr1) != 1); \ -}) - -#define arch_cmpxchg_double(ptr1, ptr2, o1, o2, n1, n2) \ -({ \ - int __ret; \ - __cmpxchg_double_check(ptr1, ptr2); \ - __ret = !__cmpxchg_double_mb((unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2), \ - ptr1); \ - __ret; \ -}) - -#define arch_cmpxchg_double_local(ptr1, ptr2, o1, o2, n1, n2) \ -({ \ - int __ret; \ - __cmpxchg_double_check(ptr1, ptr2); \ - __ret = !__cmpxchg_double((unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2), \ - ptr1); \ - __ret; \ -}) - /* cmpxchg128 */ #define system_has_cmpxchg128() 1 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -145,16 +145,6 @@ PERCPU_RET_OP(add, add, ldadd) * preemption point when TIF_NEED_RESCHED gets set while preemption is * disabled. */ -#define this_cpu_cmpxchg_double_8(ptr1, ptr2, o1, o2, n1, n2) \ -({ \ - int __ret; \ - preempt_disable_notrace(); \ - __ret = cmpxchg_double_local( raw_cpu_ptr(&(ptr1)), \ - raw_cpu_ptr(&(ptr2)), \ - o1, o2, n1, n2); \ - preempt_enable_notrace(); \ - __ret; \ -}) #define _pcp_protect(op, pcp, ...) \ ({ \ --- a/arch/s390/include/asm/cmpxchg.h +++ b/arch/s390/include/asm/cmpxchg.h @@ -190,40 +190,6 @@ static __always_inline unsigned long __c #define arch_cmpxchg_local arch_cmpxchg #define arch_cmpxchg64_local arch_cmpxchg -#define system_has_cmpxchg_double() 1 - -static __always_inline int __cmpxchg_double(unsigned long p1, unsigned long p2, - unsigned long o1, unsigned long o2, - unsigned long n1, unsigned long n2) -{ - union register_pair old = { .even = o1, .odd = o2, }; - union register_pair new = { .even = n1, .odd = n2, }; - int cc; - - asm volatile( - " cdsg %[old],%[new],%[ptr]\n" - " ipm %[cc]\n" - " srl %[cc],28\n" - : [cc] "=&d" (cc), [old] "+&d" (old.pair) - : [new] "d" (new.pair), - [ptr] "QS" (*(unsigned long *)p1), "Q" (*(unsigned long *)p2) - : "memory", "cc"); - return !cc; -} - -#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ -({ \ - typeof(p1) __p1 = (p1); \ - typeof(p2) __p2 = (p2); \ - \ - BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \ - BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \ - VM_BUG_ON((unsigned long)((__p1) + 1) != (unsigned long)(__p2));\ - __cmpxchg_double((unsigned long)__p1, (unsigned long)__p2, \ - (unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2)); \ -}) - #define system_has_cmpxchg128() 1 static __always_inline u128 arch_cmpxchg128(volatile u128 *ptr, u128 old, u128 new) --- a/arch/s390/include/asm/percpu.h +++ b/arch/s390/include/asm/percpu.h @@ -180,24 +180,6 @@ #define this_cpu_xchg_4(pcp, nval) arch_this_cpu_xchg(pcp, nval) #define this_cpu_xchg_8(pcp, nval) arch_this_cpu_xchg(pcp, nval) -#define arch_this_cpu_cmpxchg_double(pcp1, pcp2, o1, o2, n1, n2) \ -({ \ - typeof(pcp1) *p1__; \ - typeof(pcp2) *p2__; \ - int ret__; \ - \ - preempt_disable_notrace(); \ - p1__ = raw_cpu_ptr(&(pcp1)); \ - p2__ = raw_cpu_ptr(&(pcp2)); \ - ret__ = __cmpxchg_double((unsigned long)p1__, (unsigned long)p2__, \ - (unsigned long)(o1), (unsigned long)(o2), \ - (unsigned long)(n1), (unsigned long)(n2)); \ - preempt_enable_notrace(); \ - ret__; \ -}) - -#define this_cpu_cmpxchg_double_8 arch_this_cpu_cmpxchg_double - #include #endif /* __ARCH_S390_PERCPU__ */ --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -239,29 +239,4 @@ extern void __add_wrong_size(void) #define __xadd(ptr, inc, lock) __xchg_op((ptr), (inc), xadd, lock) #define xadd(ptr, inc) __xadd((ptr), (inc), LOCK_PREFIX) -#define __cmpxchg_double(pfx, p1, p2, o1, o2, n1, n2) \ -({ \ - bool __ret; \ - __typeof__(*(p1)) __old1 = (o1), __new1 = (n1); \ - __typeof__(*(p2)) __old2 = (o2), __new2 = (n2); \ - BUILD_BUG_ON(sizeof(*(p1)) != sizeof(long)); \ - BUILD_BUG_ON(sizeof(*(p2)) != sizeof(long)); \ - VM_BUG_ON((unsigned long)(p1) % (2 * sizeof(long))); \ - VM_BUG_ON((unsigned long)((p1) + 1) != (unsigned long)(p2)); \ - asm volatile(pfx "cmpxchg%c5b %1" \ - CC_SET(e) \ - : CC_OUT(e) (__ret), \ - "+m" (*(p1)), "+m" (*(p2)), \ - "+a" (__old1), "+d" (__old2) \ - : "i" (2 * sizeof(long)), \ - "b" (__new1), "c" (__new2)); \ - __ret; \ -}) - -#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ - __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) - -#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ - __cmpxchg_double(, p1, p2, o1, o2, n1, n2) - #endif /* ASM_X86_CMPXCHG_H */ --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -103,7 +103,6 @@ static inline bool __try_cmpxchg64(volat #endif -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX8) #define system_has_cmpxchg64() boot_cpu_has(X86_FEATURE_CX8) #endif /* _ASM_X86_CMPXCHG_32_H */ --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -81,7 +81,6 @@ static __always_inline bool arch_try_cmp return __arch_try_cmpxchg128(ptr, oldp, new,); } -#define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) #define system_has_cmpxchg128() boot_cpu_has(X86_FEATURE_CX16) #endif /* _ASM_X86_CMPXCHG_64_H */ --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -349,23 +349,6 @@ do { \ #define this_cpu_cmpxchg_2(pcp, oval, nval) percpu_cmpxchg_op(2, volatile, pcp, oval, nval) #define this_cpu_cmpxchg_4(pcp, oval, nval) percpu_cmpxchg_op(4, volatile, pcp, oval, nval) -#ifdef CONFIG_X86_CMPXCHG64 -#define percpu_cmpxchg8b_double(pcp1, pcp2, o1, o2, n1, n2) \ -({ \ - bool __ret; \ - typeof(pcp1) __o1 = (o1), __n1 = (n1); \ - typeof(pcp2) __o2 = (o2), __n2 = (n2); \ - asm volatile("cmpxchg8b "__percpu_arg(1) \ - CC_SET(z) \ - : CC_OUT(z) (__ret), "+m" (pcp1), "+m" (pcp2), "+a" (__o1), "+d" (__o2) \ - : "b" (__n1), "c" (__n2)); \ - __ret; \ -}) - -#define raw_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double -#define this_cpu_cmpxchg_double_4 percpu_cmpxchg8b_double -#endif /* CONFIG_X86_CMPXCHG64 */ - /* * Per cpu atomic 64 bit operations are only available under 64 bit. * 32 bit must fall back to generic operations. @@ -388,30 +371,6 @@ do { \ #define this_cpu_add_return_8(pcp, val) percpu_add_return_op(8, volatile, pcp, val) #define this_cpu_xchg_8(pcp, nval) percpu_xchg_op(8, volatile, pcp, nval) #define this_cpu_cmpxchg_8(pcp, oval, nval) percpu_cmpxchg_op(8, volatile, pcp, oval, nval) - -/* - * Pretty complex macro to generate cmpxchg16 instruction. The instruction - * is not supported on early AMD64 processors so we must be able to emulate - * it in software. The address used in the cmpxchg16 instruction must be - * aligned to a 16 byte boundary. - */ -#define percpu_cmpxchg16b_double(pcp1, pcp2, o1, o2, n1, n2) \ -({ \ - bool __ret; \ - typeof(pcp1) __o1 = (o1), __n1 = (n1); \ - typeof(pcp2) __o2 = (o2), __n2 = (n2); \ - alternative_io("leaq %P1,%%rsi\n\tcall this_cpu_cmpxchg16b_emu\n\t", \ - "cmpxchg16b " __percpu_arg(1) "\n\tsetz %0\n\t", \ - X86_FEATURE_CX16, \ - ASM_OUTPUT2("=a" (__ret), "+m" (pcp1), \ - "+m" (pcp2), "+d" (__o2)), \ - "b" (__n1), "c" (__n2), "a" (__o1) : "rsi"); \ - __ret; \ -}) - -#define raw_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double -#define this_cpu_cmpxchg_double_8 percpu_cmpxchg16b_double - #endif static __always_inline bool x86_this_cpu_constant_test_bit(unsigned int nr, --- a/include/asm-generic/percpu.h +++ b/include/asm-generic/percpu.h @@ -99,19 +99,6 @@ do { \ __ret; \ }) -#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ -({ \ - typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \ - typeof(pcp2) *__p2 = raw_cpu_ptr(&(pcp2)); \ - int __ret = 0; \ - if (*__p1 == (oval1) && *__p2 == (oval2)) { \ - *__p1 = nval1; \ - *__p2 = nval2; \ - __ret = 1; \ - } \ - (__ret); \ -}) - #define __this_cpu_generic_read_nopreempt(pcp) \ ({ \ typeof(pcp) ___ret; \ @@ -180,17 +167,6 @@ do { \ __ret; \ }) -#define this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ -({ \ - int __ret; \ - unsigned long __flags; \ - raw_local_irq_save(__flags); \ - __ret = raw_cpu_generic_cmpxchg_double(pcp1, pcp2, \ - oval1, oval2, nval1, nval2); \ - raw_local_irq_restore(__flags); \ - __ret; \ -}) - #ifndef raw_cpu_read_1 #define raw_cpu_read_1(pcp) raw_cpu_generic_read(pcp) #endif @@ -307,23 +283,6 @@ do { \ raw_cpu_generic_cmpxchg(pcp, oval, nval) #endif -#ifndef raw_cpu_cmpxchg_double_1 -#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef raw_cpu_cmpxchg_double_2 -#define raw_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef raw_cpu_cmpxchg_double_4 -#define raw_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef raw_cpu_cmpxchg_double_8 -#define raw_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif - #ifndef this_cpu_read_1 #define this_cpu_read_1(pcp) this_cpu_generic_read(pcp) #endif @@ -440,21 +399,4 @@ do { \ this_cpu_generic_cmpxchg(pcp, oval, nval) #endif -#ifndef this_cpu_cmpxchg_double_1 -#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef this_cpu_cmpxchg_double_2 -#define this_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef this_cpu_cmpxchg_double_4 -#define this_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif -#ifndef this_cpu_cmpxchg_double_8 -#define this_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) -#endif - #endif /* _ASM_GENERIC_PERCPU_H_ */ --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -2234,21 +2234,6 @@ atomic_long_dec_if_positive(atomic_long_ arch_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ }) -#define cmpxchg_double(ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - kcsan_mb(); \ - instrument_atomic_read_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ - arch_cmpxchg_double(__ai_ptr, __VA_ARGS__); \ -}) - - -#define cmpxchg_double_local(ptr, ...) \ -({ \ - typeof(ptr) __ai_ptr = (ptr); \ - instrument_atomic_read_write(__ai_ptr, 2 * sizeof(*__ai_ptr)); \ - arch_cmpxchg_double_local(__ai_ptr, __VA_ARGS__); \ -}) #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 82d1be694fab30414527d0877c29fa75ed5a0b74 +// 3611991b015450e119bcd7417a9431af7f3ba13c --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -343,33 +343,6 @@ static __always_inline void __this_cpu_p pscr2_ret__; \ }) -/* - * Special handling for cmpxchg_double. cmpxchg_double is passed two - * percpu variables. The first has to be aligned to a double word - * boundary and the second has to follow directly thereafter. - * We enforce this on all architectures even if they don't support - * a double cmpxchg instruction, since it's a cheap requirement, and it - * avoids breaking the requirement for architectures with the instruction. - */ -#define __pcpu_double_call_return_bool(stem, pcp1, pcp2, ...) \ -({ \ - bool pdcrb_ret__; \ - __verify_pcpu_ptr(&(pcp1)); \ - BUILD_BUG_ON(sizeof(pcp1) != sizeof(pcp2)); \ - VM_BUG_ON((unsigned long)(&(pcp1)) % (2 * sizeof(pcp1))); \ - VM_BUG_ON((unsigned long)(&(pcp2)) != \ - (unsigned long)(&(pcp1)) + sizeof(pcp1)); \ - switch(sizeof(pcp1)) { \ - case 1: pdcrb_ret__ = stem##1(pcp1, pcp2, __VA_ARGS__); break; \ - case 2: pdcrb_ret__ = stem##2(pcp1, pcp2, __VA_ARGS__); break; \ - case 4: pdcrb_ret__ = stem##4(pcp1, pcp2, __VA_ARGS__); break; \ - case 8: pdcrb_ret__ = stem##8(pcp1, pcp2, __VA_ARGS__); break; \ - default: \ - __bad_size_call_parameter(); break; \ - } \ - pdcrb_ret__; \ -}) - #define __pcpu_size_call(stem, variable, ...) \ do { \ __verify_pcpu_ptr(&(variable)); \ @@ -426,9 +399,6 @@ do { \ #define raw_cpu_xchg(pcp, nval) __pcpu_size_call_return2(raw_cpu_xchg_, pcp, nval) #define raw_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(raw_cpu_cmpxchg_, pcp, oval, nval) -#define raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - __pcpu_double_call_return_bool(raw_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) - #define raw_cpu_sub(pcp, val) raw_cpu_add(pcp, -(val)) #define raw_cpu_inc(pcp) raw_cpu_add(pcp, 1) #define raw_cpu_dec(pcp) raw_cpu_sub(pcp, 1) @@ -488,11 +458,6 @@ do { \ raw_cpu_cmpxchg(pcp, oval, nval); \ }) -#define __this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ -({ __this_cpu_preempt_check("cmpxchg_double"); \ - raw_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2); \ -}) - #define __this_cpu_sub(pcp, val) __this_cpu_add(pcp, -(typeof(pcp))(val)) #define __this_cpu_inc(pcp) __this_cpu_add(pcp, 1) #define __this_cpu_dec(pcp) __this_cpu_sub(pcp, 1) @@ -513,9 +478,6 @@ do { \ #define this_cpu_xchg(pcp, nval) __pcpu_size_call_return2(this_cpu_xchg_, pcp, nval) #define this_cpu_cmpxchg(pcp, oval, nval) \ __pcpu_size_call_return2(this_cpu_cmpxchg_, pcp, oval, nval) -#define this_cpu_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \ - __pcpu_double_call_return_bool(this_cpu_cmpxchg_double_, pcp1, pcp2, oval1, oval2, nval1, nval2) - #define this_cpu_sub(pcp, val) this_cpu_add(pcp, -(typeof(pcp))(val)) #define this_cpu_inc(pcp) this_cpu_add(pcp, 1) #define this_cpu_dec(pcp) this_cpu_sub(pcp, 1) --- a/scripts/atomic/gen-atomic-instrumented.sh +++ b/scripts/atomic/gen-atomic-instrumented.sh @@ -84,7 +84,6 @@ gen_xchg() { local xchg="$1"; shift local order="$1"; shift - local mult="$1"; shift kcsan_barrier="" if [ "${xchg%_local}" = "${xchg}" ]; then @@ -104,8 +103,8 @@ cat < X-Patchwork-Id: 13240934 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32D51C77B7D for ; Mon, 15 May 2023 08:07:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6577B280002; Mon, 15 May 2023 04:06:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DFF4280001; Mon, 15 May 2023 04:06:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3BCB1280002; Mon, 15 May 2023 04:06:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 27111280001 for ; Mon, 15 May 2023 04:06:54 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0005B81331 for ; Mon, 15 May 2023 08:06:53 +0000 (UTC) X-FDA: 80791758306.09.8A3D577 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) by imf22.hostedemail.com (Postfix) with ESMTP id 4C48BC000C for ; Mon, 15 May 2023 08:06:52 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=jDA5YrBA; spf=none (imf22.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684138012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=qQ97W8DjtNWYNvTOAleOKLKZr+6FI4styhSio5/BDPE=; b=RlyJiyTX86KW4Zp4U19yajjahULv1f6DPcvTL2r2huKWhQIi8jQOo6YK20zLotCEWjFsJ5 4CIfSXRugJ7ucAsHu0/0JyXE9AIONfAOFseRjziZJ2VC7zol0QaJKx17luzXks1IAJN7Ak j7Jmnio70u0TAXz9BiwCiNiXd2HhFoc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=infradead.org header.s=desiato.20200630 header.b=jDA5YrBA; spf=none (imf22.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.92.199) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684138012; a=rsa-sha256; cv=none; b=5JL1p+sAxvTZZkkE+U5Lz5RpFeG6UuK8rOJZkS8NO2bvE6P0kXYh8nKMEHjIWkwJzisl1r WfnCumFANi/4JFhRg781cCWxXQtGmD+YgfuOCAMwbSiv/8LuTEpfZwv5KuAYxsT1nDK+21 siPwISRcePTPGHKsMLImCH+2u3nUNgU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=qQ97W8DjtNWYNvTOAleOKLKZr+6FI4styhSio5/BDPE=; b=jDA5YrBAqv2XXwX2hyoap3XhWh FUFqXWI9xbofaUhrnD/ux504IzigT+5UAWA2nMY5JWOA6Oqia5Te8jXiDxFNK8pNpovPpq5Ipx4Tp BoP1Xn/k2yhszEl4J3D1i1ULvw2aVOT2NmbFxiAt/QCeyGGnPAoLUpB5+TENS4VG9T0BTK9qeOYph j0IJUDpNmFgbT4b5PUux6ZW+r1nWua3QIgMmTlB8o48//Is03yKfVTJtReG1SFLB/GbnnyxmvNPD4 42RHsxPs8LP1AfUAk0r3kC5W2n+QFqPC+el6SkJJV5JBjO1M1OFIEMoIuucEObRX9HHJj+weRarpG WQ5sFIOg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1pyTDl-00BQN6-1z; Mon, 15 May 2023 08:06:18 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 934F2305ECF; Mon, 15 May 2023 10:06:15 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id D82FB202FCEA8; Mon, 15 May 2023 10:06:10 +0200 (CEST) Message-ID: <20230515080554.657068280@infradead.org> User-Agent: quilt/0.66 Date: Mon, 15 May 2023 09:57:10 +0200 From: Peter Zijlstra To: torvalds@linux-foundation.org Cc: corbet@lwn.net, will@kernel.org, peterz@infradead.org, boqun.feng@gmail.com, mark.rutland@arm.com, catalin.marinas@arm.com, dennis@kernel.org, tj@kernel.org, cl@linux.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, Arnd Bergmann , Herbert Xu , davem@davemloft.net, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, Andrew Morton , vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-crypto@vger.kernel.org Subject: [PATCH v3 11/11] s390/cpum_sf: Convert to cmpxchg128() References: <20230515075659.118447996@infradead.org> MIME-Version: 1.0 X-Stat-Signature: ayw3bwwart4zcxms6usqrs819i5ah8ky X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4C48BC000C X-Rspam-User: X-HE-Tag: 1684138012-819472 X-HE-Meta: U2FsdGVkX19mGaF0+D9yAnoczljgez2aT+QAihUFnpBDpEfHDxJqQ+a40YSAMfg4MriFm1mxymYqpoiHGAqf0E3MUdL6lpvxKYlckHeuX+iZV4iyPk+r0t9G/JJvWlpe+N+Uq+WkGG4uZwnGHb2wDf+A6tdwdbwgsb0aoWRmwXQzoseJMC0ok7LLSj96PqEiErGffJInkx0HzxZC7HkF35XqKbeAqP71Ns503QGxrpkKdqLQyr7JIrph2B89eTgcDmR1jdJ04SVQs3iMBtAsHsBS6Gl/P1640bIBa4Tbuo0/Hyr86l1gtTOF0py39xp668iZzWW05g1QM2EvWumKIsgETgNGzfj/5xVbO8BHJrDTUVLhcRhCok0ZSBeZN49AorJ6WnEa8AXAt7D36wLdT3nzkcKtqSGmECBkSiOfnE8YrL9kdZZm0o5KpeP+X81BtjMqipW2X3u9dy6xuQrb0WfLs2E+uYuWMBgrKkdlV21OZC47MadsrD66zaHjxW45T3ZIOnVqRS+w4RBVYYG7Py/+Foeku8m80B660eJQ/UzCs4MWOidZrJK2zJ7E1lAQ4Uf6hKyA841zo2545USCPC4MvdHUWApWY8Jcx5wyYTCM3WbrJ0QtLmPg0wEX3ZrLYLISX58vzx0OxJZoLAccb1pwWu/A6IZAZYzVtZ78iJr+UeKqW/tcXkdGaWnlVwvCYei/S5Jye3PB2lhAVDncRdfm/BHpp2xjxY/fHEaVviQmEmi+PS0AHHk74+DP/sglO5dSy4DdFgFFf0g8FL8sFp9D7LAKb4a0apXI+thYLFXOiBZMg+7V06mBw2OlgSRRjRAzQeMWGvbI7njcNSN9vWLnIRKxJBPA/EhABTml2KqRzHFWOQaDr4fKDOXQ8udKukeFadaYlH9LIOB8q/9iaDxv5OKv1JX+/8sfDohccRsLtmjkuY2G5Gx1YdLFhY3cT//+XxLWIyrQa5LyPd+ Hp1ENGtM iCqWwutjqob6Tuq6NS0dmKErsvwWqM8m+nzhMwx2IFXSNpsZ4PJumriCyPPqLJELGMrymOKJKbYMVZp5dg1V1xNeJJiFPeSWKgMjorsoMgha0vcuZYEdjU1D06nOlLzE+zFsUT5EEqQOb5R6CRF4g8UGEzMNlugUJqOX/J5FaUyxcXNhXKEw5uayaQJBo7fBmXTTRXPHk5JriikVmBHOeBpZpCmDHu1Du+PxWGf0TdZfwcsq/oJtD+2g8Pz4q2tJWzUn3n/uD5CbC4bnR+O6lnXL1tF0wQSF/KcH4h6YREBcE6XgYYBqc+xQI55qpQtUKHSr10rPrq3sZWkyQHM//bF01/Ke6TNTpVgj0oN6UYyLOuldnfjZwwgvp1lcimSAQvnUJMjKUtGWy6lIWKrUn1FIsDjmj4uJuh89w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now that there is a cross arch u128 and cmpxchg128(), use those instead of the custom CDSG helper. Signed-off-by: Peter Zijlstra (Intel) Acked-by: Heiko Carstens --- arch/s390/include/asm/cpu_mf.h | 2 +- arch/s390/kernel/perf_cpum_sf.c | 16 +++------------- 2 files changed, 4 insertions(+), 14 deletions(-) --- a/arch/s390/include/asm/cpu_mf.h +++ b/arch/s390/include/asm/cpu_mf.h @@ -140,7 +140,7 @@ union hws_trailer_header { unsigned int dsdes:16; /* 48-63: size of diagnostic SDE */ unsigned long long overflow; /* 64 - Overflow Count */ }; - __uint128_t val; + u128 val; }; struct hws_trailer_entry { --- a/arch/s390/kernel/perf_cpum_sf.c +++ b/arch/s390/kernel/perf_cpum_sf.c @@ -1275,16 +1275,6 @@ static void hw_collect_samples(struct pe } } -static inline __uint128_t __cdsg(__uint128_t *ptr, __uint128_t old, __uint128_t new) -{ - asm volatile( - " cdsg %[old],%[new],%[ptr]\n" - : [old] "+d" (old), [ptr] "+QS" (*ptr) - : [new] "d" (new) - : "memory", "cc"); - return old; -} - /* hw_perf_event_update() - Process sampling buffer * @event: The perf event * @flush_all: Flag to also flush partially filled sample-data-blocks @@ -1362,7 +1352,7 @@ static void hw_perf_event_update(struct new.f = 0; new.a = 1; new.overflow = 0; - prev.val = __cdsg(&te->header.val, old.val, new.val); + prev.val = cmpxchg128(&te->header.val, old.val, new.val); } while (prev.val != old.val); /* Advance to next sample-data-block */ @@ -1572,7 +1562,7 @@ static bool aux_set_alert(struct aux_buf } new.a = 1; new.overflow = 0; - prev.val = __cdsg(&te->header.val, old.val, new.val); + prev.val = cmpxchg128(&te->header.val, old.val, new.val); } while (prev.val != old.val); return true; } @@ -1646,7 +1636,7 @@ static bool aux_reset_buffer(struct aux_ new.a = 1; else new.a = 0; - prev.val = __cdsg(&te->header.val, old.val, new.val); + prev.val = cmpxchg128(&te->header.val, old.val, new.val); } while (prev.val != old.val); *overflow += orig_overflow; }