From patchwork Fri Feb 14 15:07:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13975061 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C740C02198 for ; Fri, 14 Feb 2025 15:09:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 742A36B007B; Fri, 14 Feb 2025 10:09:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CB926B0082; Fri, 14 Feb 2025 10:09:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 545316B0083; Fri, 14 Feb 2025 10:09:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3152F6B007B for ; Fri, 14 Feb 2025 10:09:43 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D788EA04FC for ; Fri, 14 Feb 2025 15:09:42 +0000 (UTC) X-FDA: 83118884604.30.193013D Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) by imf13.hostedemail.com (Postfix) with ESMTP id E147320002 for ; Fri, 14 Feb 2025 15:09:40 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="faBLHX/C"; spf=pass (imf13.hostedemail.com: domain of ubizjak@gmail.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=ubizjak@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739545781; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=UVqMZ+3Mpmpk9SfLsBXSOCQc7oNYAD8Y16TaUhOCH8Y=; b=KJgQ3lkzMadCBwA8OeVhrAcS6R5LPra+jsSm/ZD6DKLZJG/INufbkkWC+9LnnnnVuPrX3w 7eaZG7vMg4tzBcni0daYD/fBLp6nYS87cmDEElHE8DWkomNGupNwcZAIPujavPnr3IOPGn Fq1/qVJ+90EiFJtNueQSlif6rSKcviI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="faBLHX/C"; spf=pass (imf13.hostedemail.com: domain of ubizjak@gmail.com designates 209.85.221.53 as permitted sender) smtp.mailfrom=ubizjak@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739545781; a=rsa-sha256; cv=none; b=eAt33cgyFuWcw1vhH69wIZgBRCayeFIZGHrhUbQA4V5E75OJgbss0y/ShREzNXt6XgEyWf MmrIftRZv+D+rWzhZOMtJlCewjcTZYV88C7ErZGM4bf2wp7XMF1tBfFrntCTedF+dR5Xas Bx0Biw6bV4GfQhZjM9zaZURkEbW7Dq0= Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-38f22fe889aso1645192f8f.3 for ; Fri, 14 Feb 2025 07:09:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739545779; x=1740150579; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=UVqMZ+3Mpmpk9SfLsBXSOCQc7oNYAD8Y16TaUhOCH8Y=; b=faBLHX/Ca1XL72EPWkNrBDmPOmOSHcrmHVGrKi308sgumBUHtC1ilET8etD9TOm8Uo ydlT+3+ezuPmdN7KtpQhurfboxUDyENkJ7XcvQAbpwiJCUSItjmavcWQ1xuOHaWC8lfc jrDHhsx4aOeL52j0UpHynHSjBhFUJ7/AMSBoQex5kOeBsreQTNcUi9StgK4BnpgY9q9p FdNIZbqogvXsE1rgkTr3ef3k1dyJ88DvaH7leJV6GYKo3/LCrT9zwEmZg+d6DgmH0Typ 7egGpbl5c4+lJfzhDq/4Se9LxeBJvFp+6WfbLEcUIR1+6MVJCWoDADotXnbCJ3X9r6LB zRUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739545779; x=1740150579; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=UVqMZ+3Mpmpk9SfLsBXSOCQc7oNYAD8Y16TaUhOCH8Y=; b=sWTkSkL4yXGOzUtFn/l0u95RLOtVOsubGKFPJjRKUIvds6L5HYqF7kj3sEFUfyUu24 nxJxloOiB3ZkEKpwDccVE84b+f5fqkeAlGD9Cu2j1Q85NnDxN/hY8MBy88f46Q0j7uD3 MTzNx9rrmwjJWLjrRRdknXjw4XaNIpeg9U7gx8yxY3ipcoVs9Lkr8q1dfkmjE8fyE/kL Q/U+FyLii3SjIvlAc9p9QIvi+x+EXKE8HKXMYjsK10GRpxR/f9sCQNpShg196+1Fv0A4 mbDvez8TGqLpDYNZ+i6ppu/izfGL7/bvn3frHLnBRDOsAQf8xRbH3ffhrRAJ56yTMswm 3I0A== X-Forwarded-Encrypted: i=1; AJvYcCXX5KKVa5QxM4Y8N61xxSvb2tOYMaqWyClOEZLtYOnSHqV+H/7DABDy7njh0gzNsZXYyKhvSYEyxQ==@kvack.org X-Gm-Message-State: AOJu0Yweg2SyVzMdMt4iDBZAxxlmW4mf3ubmGBxED/6GJvA4+Lq8RiEj e/dGEdN9JTC5+Wm5TFeWrpbX9MP+O5FM0vYl2J4usd1CAKe/fH4R X-Gm-Gg: ASbGncsTXQ/X7nBpk6SzghOEGO6oNariowvt0ugzvFJazDzyeps2DphJNiLizLWiMk6 IVVsBpwqEg/5a1JMCz4jLGd9BrKyUSqIlE+bw9ghEUzZSD9aIe9zQ1xclvJ+CYPlSgAwBQPrd9g bmlnBnzXh3uawPODEOEf6vyke7mn9sYtksCB3qrxBgXoeMtzcB8yMCG/UspMp1rxTqTHhNgXoxp 0RBUHLHjFN0CDyIY1ESA83UwKHk7XAzpITXWJeok+pSxGA8kK6OC0xPXVfoe6PXeCe8mEiG66gK +n7RodKEbnyWrr7pvxaKQG7p X-Google-Smtp-Source: AGHT+IEXQWbeVAMShMrrzZEIZ97y+Aoj4EgRTTgtulGM8VILACGiWcO0VRckkcgDb+WqEeYDbjg47w== X-Received: by 2002:a05:6000:1568:b0:38f:2095:2fe7 with SMTP id ffacd0b85a97d-38f244e502fmr9519949f8f.12.1739545778732; Fri, 14 Feb 2025 07:09:38 -0800 (PST) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aba533bee9dsm357447266b.173.2025.02.14.07.09.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 07:09:38 -0800 (PST) From: Uros Bizjak To: x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Dennis Zhou , Tejun Heo , Christoph Lameter , "Peter Zijlstra (Intel)" Subject: [PATCH v2 1/2] x86/locking: Use ALT_OUTPUT_SP() for percpu_{,try_}cmpxchg{64,128}_op() Date: Fri, 14 Feb 2025 16:07:46 +0100 Message-ID: <20250214150929.5780-1-ubizjak@gmail.com> X-Mailer: git-send-email 2.42.0 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E147320002 X-Stat-Signature: 6du458om1r9odankxcz8y4ftbetejfqc X-HE-Tag: 1739545780-496295 X-HE-Meta: U2FsdGVkX1/N0a0H4Zj+xMZkoLharnzB/h6XkO4iTxARBwO0ymXeEbvDey6rEaQFE2n1gHQvsxMDlrCz83ZeiEMaiz1zA+/EWKSGvieC7V9bawFghEVB5jprTdcd2+hoq9O84OqdGIyctz92HlTdBI82JxTQtOaBFX/8e/tPRhc8vAUgDHtCyFWNFmwyPyIHXys8AB1vECqGWPzBGuoNZJt4QK1JilFVA/JoyKqi1D1mE+UX3coF/eaLmisf5G9AlN15NELs1SBhTFcB2Obbm0p06SBCq74KkwhSHZpFn6RkRDquR9BIHnPBsBVM445C98pNTGVPBJlqbjazB7COi9Q5d7nBJRXvsAS3VjcAZEtFhRr0VFQToOjCB7jPJcOiyztIHhJK42AQSz18mmHOwM3nz9PQTBLgWjRZI63pX1J/r/JXohlb/lgqh/uiStEUBdCb/oJTZPqMA5u43ADg3gjhPu4DJ6Q8ZYtmD91tAJcb0NlxSdQMOH+C61jQqMu31NAn8OGLuEhcx9bGe/CUh/eGScgvGOvRwI7J6LFo9k0MRtfFl1+KftzMnc8MQeaimoyV68P7golcrNAZAmsPVaPo1g72antqKehGx8GgJJesdvQL1Mgv4Vc/iHYf6T2y26dCiYyxu/9u+dEcEZxk4xtArjFeZaEtJLhw+Jz1QBVKrXUIfbdJHiKkMp4HLraz2rpcfP4x9Xgq95+j0zLsx8nxJbHjt1EA6VSkOuWjM3PAOn/AOzut7UvdbauX+yysw0vXGxIZ+oSMuvJpsTN0/Nm4KqFktZLWHF85FsOYBon4AARUO1oLi85FIoJerqsSwl+GCGNdKWfkJMtTamn8BzlzTCOwmXJvZHmshU1tGhs+SKX6PRkK/IvjHo3wYqZ7g2MXUu/53ZEOH92EwULMuWXcv6rDYEfYhYZQKNvDSO8QBlJJQSxD3q2GK3mPadOrtZv6i46ledxOdig3zKU sV4cpNdY QdCC30ot5S7sIlutn7d0xhx1wDY16WWecRlP6nOmFEvCF36AEv3d2SVzEBNeZ1J1yrZ1SyKEv/JO/No8eKyE5ZX0HQT8QTUJETL5zRolyXAXpEKEYFKKN+Nvw1CkjGvGb1uwE4zojn+jmFAQ31J1f136VZobtnrWd3F/UpShuJ8vz+UGCFijiBoSkqJQO7fkDYFGrV8FM6rpbsj5Z1buXUnmBLJpXe4NiETcKkfJzx3+ui3fLvhpqb+QqUfqEeojPydYDChR1YiDH5syLReScb3Zos3bfN8ojUN1BZyOlZ1JmdSUwNhKN/PXCCUTCiD+wiAAFznPQYJd9TPhyzIkAzk4o5v23iHLk974eT5dqm8dHpxtTO+nuHtZCHNa1TG5rWvMomBF2JbtrpnkK+s7ayJzojRdv001rNwA8yV9PXRa3QVJbWWytWeo/rfvPYrdGzzGuGWiQqIYNNeykPIGbexhoLNvHTtyG88fMgJ7Ee8tVpYFrjqHbMxRWqJ8pEH/tD3lh97OZsI994cb3ejpVjuEYXga4SauPr27jvhz1I1LLzJEsfudhooI/Jiuj07nKeIihxKgHmMMjMxxQXG75cqULX2ED4z+2YPmqDoFb3m3NQ9Hai5TanCRCqVWf3PmCCQ89CUrZzFeSkac= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: percpu_{,try_}cmpxchg{64,128}() macros use CALL instruction inside asm statement in one of their alternatives. Use ALT_OUTPUT_SP() macro to add required dependence on %esp register. ALT_OUTPUT_SP() implements the above dependence by adding ASM_CALL_CONSTRAINT to its arguments. This constraint should be used for any inline asm which has a CALL instruction, otherwise the compiler may schedule the asm before the frame pointer gets set up by the containing function, causing objtool to print a "call without frame pointer save/setup" warning. Signed-off-by: Uros Bizjak Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: "Peter Zijlstra (Intel)" --- v2: Expand commit message with the explanation of ASM_CALL_CONSTRAINT. --- arch/x86/include/asm/percpu.h | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index e525cd85f999..0ab991fba7de 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -350,9 +350,9 @@ do { \ \ asm qual (ALTERNATIVE("call this_cpu_cmpxchg8b_emu", \ "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \ - : [var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high) \ + : ALT_OUTPUT_SP([var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), \ + "+d" (old__.high)) \ : "b" (new__.low), \ "c" (new__.high), \ "S" (&(_var)) \ @@ -381,10 +381,10 @@ do { \ asm qual (ALTERNATIVE("call this_cpu_cmpxchg8b_emu", \ "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \ CC_SET(z) \ - : CC_OUT(z) (success), \ - [var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high) \ + : ALT_OUTPUT_SP(CC_OUT(z) (success), \ + [var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), \ + "+d" (old__.high)) \ : "b" (new__.low), \ "c" (new__.high), \ "S" (&(_var)) \ @@ -421,9 +421,9 @@ do { \ \ asm qual (ALTERNATIVE("call this_cpu_cmpxchg16b_emu", \ "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \ - : [var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high) \ + : ALT_OUTPUT_SP([var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), \ + "+d" (old__.high)) \ : "b" (new__.low), \ "c" (new__.high), \ "S" (&(_var)) \ @@ -452,10 +452,10 @@ do { \ asm qual (ALTERNATIVE("call this_cpu_cmpxchg16b_emu", \ "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \ CC_SET(z) \ - : CC_OUT(z) (success), \ - [var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high) \ + : ALT_OUTPUT_SP(CC_OUT(z) (success), \ + [var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), \ + "+d" (old__.high)) \ : "b" (new__.low), \ "c" (new__.high), \ "S" (&(_var)) \ From patchwork Fri Feb 14 15:07:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13975062 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F9E1C021A6 for ; Fri, 14 Feb 2025 15:09:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFE636B0082; Fri, 14 Feb 2025 10:09:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D5C546B0083; Fri, 14 Feb 2025 10:09:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B389E6B0088; Fri, 14 Feb 2025 10:09:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 950286B0082 for ; Fri, 14 Feb 2025 10:09:44 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 45DF7C0512 for ; Fri, 14 Feb 2025 15:09:44 +0000 (UTC) X-FDA: 83118884688.18.EB6D0AC Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by imf02.hostedemail.com (Postfix) with ESMTP id 411F68000E for ; Fri, 14 Feb 2025 15:09:42 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bGplA+ne; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of ubizjak@gmail.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=ubizjak@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739545782; a=rsa-sha256; cv=none; b=lBQ80NBJX2qEXYirmKzfIlnhb8iVpvihqRtbyMlHo4m96yyFjsJsFjAPoCxod0bQUGPMUk rAqYpApCIHqKUcIIgRIt8tXofeswPtZnLFBNYCNq3HU+zpGD6kFuYuK41rrLCL13RM+09c n61aEOa357/IN4DGPluFESEO5N1FdUI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bGplA+ne; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of ubizjak@gmail.com designates 209.85.221.42 as permitted sender) smtp.mailfrom=ubizjak@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739545782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2zJBQ4YpMD9soorm1behI7kialp0O7eXmIoL4a6+Dak=; b=Oo0TI47fzRBRPLy/u3kGY1p9lZ3Z6MvFJ8anb4RR92piEScnwl8e6uFENAE+dYUTgq891J HdOwTx6O6Ce7FcjTRwMTjlHA6jl1HyoRzozo3ph1E5h0kkaLbbEGtXBX1N5RGKFyIhRnc5 rF6pxC3yj4C6JDEs1M2e/TnRa7HWc9o= Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-38dd93a4e8eso1874663f8f.1 for ; Fri, 14 Feb 2025 07:09:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739545781; x=1740150581; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2zJBQ4YpMD9soorm1behI7kialp0O7eXmIoL4a6+Dak=; b=bGplA+neLkxkxCErSvJzbINeB8oPqLuRoXXzL56Rwv/WwfOAssFx7z8BjRH+fsIJy1 2uc1RKxLRJSadhODNUv6otYEtiRlq1m8Fqm4831WeBEYfWv1jtdCgVkRbLm7PjDW9hMB V1eQymPTlk4H1C6aCv74oBt7zQaqY3qLClGYl5wyfkLjn4uq1h7zM9R/kfRWAFZTzD28 OqG8ghwArr0Dm1TieEHnBY5/YMZlDWjL2JzvQpxksGGC706HpjPJO8rjoOAewyxVxVeM Q7i8MeW9iikP+kwvNEKUdkpRLNRqTItz55TjMi05NglaHsgx0ffh0MBjuByQ4D6HvzgU MT+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739545781; x=1740150581; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2zJBQ4YpMD9soorm1behI7kialp0O7eXmIoL4a6+Dak=; b=d8wrQ7O5PfaB7GKShTsOq0miIB4eNaqzL2A7s7XpmpqiqF64ebx4QZy9VpFCyp4aPl lKaZ18LVfRRvJ8TVshr/Mq3LO4gB++jXnv5iJPexCc96i9cPUyxREVnpfXfwlPBj9aV9 mjv58KylkFvdbITu8CmhQgygN/G5doHCXf4wBOK5bV8wgNtUFmGcqX1bW7e/uM0lHvBo Arg5HhYEAuyb/31GIOaw38rsSZY0qG3cG4e/hZ8srjbNU/O6lvmM3rrbRJWsb5HPpn0x u6ow+dfPezKhIFrSGi06y92Pt6rQST3xydgCQq0RTvRwCNfl0PfXm7strOgtMKIl3RvF gb4w== X-Forwarded-Encrypted: i=1; AJvYcCXY9QqDib2LXaGRPfy7le5zNUStKByI+NQayry1Qmli3keKyhJbykMfKOCUmx2A5k5ZYO1+wvioiA==@kvack.org X-Gm-Message-State: AOJu0YwEZlStkJQoB+8/xzDalhCY5IaK9RJ6Yj9QYQgcFZDze8haT0vq LOVG0BqXnDEs6lPmNvgf3P7vcEKKnNKGjdq6HpJ9erd35jGG04ea X-Gm-Gg: ASbGncvpb6hnNNTOcsphjHAHOB2pKjd0RSCkL77ekaE6g9J2WgRqBfcUj+eTYsXq9iI 0W56j4gG11k7nSGKQumlVd+gC1x9na5BrjdfXTpzKh6fV6A1OIhGKUjX263eQgzMGHTD88V3hdw 3lGKi3LAjaKJT31eMeI2mypAszsGQhDO9e2VtNWQAbMpJ+JOPnXO0JreWa/xKqDLrXvcNftYeuY CJZabt9Q2rpNA3RyVGWIx2/vQLjokAbDsDpkHlZnP5c05N0VId1B7VSejPpK+bGfKQiiXsEkuvT IyG/JF1nPqE7tTxvZXZR1JCt X-Google-Smtp-Source: AGHT+IFum9XwSDM3eHz0B6ljntzlQn+pA3NOKxdBCEjZBiROAFX1QudujE8vXpbeAXil4HYQjBINIg== X-Received: by 2002:a5d:47c9:0:b0:38d:d5af:29b4 with SMTP id ffacd0b85a97d-38f24526116mr10583092f8f.45.1739545780313; Fri, 14 Feb 2025 07:09:40 -0800 (PST) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aba533bee9dsm357447266b.173.2025.02.14.07.09.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2025 07:09:39 -0800 (PST) From: Uros Bizjak To: x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Dennis Zhou , Tejun Heo , Christoph Lameter , "Peter Zijlstra (Intel)" Subject: [PATCH v2 2/2] x86/locking: Use asm_inline for {,try_}cmpxchg{64,128} emulations Date: Fri, 14 Feb 2025 16:07:47 +0100 Message-ID: <20250214150929.5780-2-ubizjak@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20250214150929.5780-1-ubizjak@gmail.com> References: <20250214150929.5780-1-ubizjak@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 411F68000E X-Rspamd-Server: rspam12 X-Stat-Signature: pgjwq7wgwr9fbnxr85m4rbn43kptjzco X-HE-Tag: 1739545782-410736 X-HE-Meta: U2FsdGVkX1/NFIxjBw+f/DvcTRo9MRYAUrphiPkYhNkYb1mqeZYqpKCvTOjCvuUNbVrZrhPC72DiYscF/8RtFZDR3bWj8dZ8oi7SpgPmpqveanZQUyjTlLSlCBTlbWf1HB5m0sQemEhNOtId5H6rzRqKfReHkQQQMnJY9IJlC/AhlRUkpBqZuyD05XaH5CEjvTNUEs923XiJtBLvBRvfgSIakcjG8/P+f2mGr1ZkrCHpHZNvV/hI8eAWiw0rjnBJvdCgSf9r4uA7r9tu9H/BwZZpeELeV67WkPy1MVZH/X4aI5Zb2xXc8D5uYpFRlJmGDJVE+s0qdWszxyWkYszXPKB7s+bkwZKHu6rAGVl4UQtDRftEEO8dtg1lUxBXvTkMVnJz4r3Fuu0xp0CohKPjQpj5sY9LNdJTMXt+uAuzGxt1mbbezwDhOhdA5NqZHtGcjK1SufJ2jWR6+K2tAKB3yeiyMTpQqvEaDcSvXncmZK7aAumNbJbLUwsB4mP/dkCGjFfQm7wa8G60s32+j3Ssc4Mep3l3xEFQOtAyCKybN563Y0tnYxlr6k+XFaNN2MyavX58JzpDOp3R2JGyRn5v7oVcWzp9IS+XGnjyCvHjGVkUbDR22PWZ8chaWS/Et11qJEdSmj9aG7LQ2GLJ1bMt9iC0NbV3mVkCxyxft49ALl0K4a/WpVwr/Tme8901B334znkHhjqx3mvldI97qklmM5caQgOg23B/L9K7EoytjXoDifwNvisb0XNS04Fa1UkZxTck8sF54oqNuMqXMQOsCapjq3+aTq69rgKm6EPt7kgfvKIovuTjcPV5k2bPZbyeLZe+xe2tfjx3RgxkcLsN8o5JZyz3FCBz2VsqnZ2KTXDyQ9A3QfdSy0StbFpkE5MVvbDHwpMG25IIbp6UuHMqp0n4QQsTmIappBxCM0uFYw+0byhsjTjz+aCgHo0EpuKMpgU1Qua48YLMPMAY0fY 2Eqogssu zhhAL6wn1zOYic/2s5/MJONpSMlhMWrJCRRpfFFN2a1l6B6xbpKMF8tJctqCNv1lG/ShPO657x3fEmEj8AInv+FaCmNnRsQlaE9byvSmFVeDxSiwQWJ2KZ2RtIXStV085QYya4/uVnH4ciiUso+3RNi2R9uYoau+TcLl8m5+3iSJbSGCbWhUFnQwOvkV4M264aIChYcWO3Px7V1QLYfUiif/io/Maft1TIvG3sfB00Sw3TCRgJL3lEEw9hTC5RIwXSHYppGcBBHb4HBvAA1wvWjr2B5ADKUR3c9e6ce+/EbVTuZoPQhTiOe+TPUtLudHCW1tyqzWP7oNwNZrZuPJHc7E7gJAwtVVZhk/NQG4tRmFjdiKWbImBwx9fsN+W7p3Vrih4GniTrSCueT+LiMZGO1seFzuuUEKzXtDiIVBoZIDnqvv97ZDTTql4rY5DHC1kwUVDNbpjjq7M9TAjF7ArUA2nOcCUsRWFYhSYY0Oc0boLhGdJIvVRljmJPpVomV87SR1sotiyDCjn2IVOlb3yk8WhNa+dXFURjpTQAYris18laPJG8yj6E9L6VOUYnk22rOXdhgpoWtthWsZOet3X+uLrTfn7byjCvtfQ98Qd5Q9yBFR4UKO2JrGERJVv/XbjAYdSVVcOdyQEVFRGvDEdZi39SU07c8IkFOCL9jI9q8771ZJ7uydsA4q/Rw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: According to [1], the usage of asm pseudo directives in the asm template can confuse the compiler to wrongly estimate the size of the generated code. ALTERNATIVE macro expands to several asm pseudo directives, so its usage in {,try_}cmpxchg{64,128} causes instruction length estimate to fail by an order of magnitude (the specially instrumented compiler reports the estimated length of these asm templates to be more than 20 instructions long). This wrong estimate further causes unoptimal inlining decisions, unoptimal instruction scheduling and unoptimal code block alignments for functions that use these locking primitives. Use asm_inline [2], a feature that makes GCC pretend some inline assembler code is tiny (while it would think it is huge), instead of just asm. For code size estimation, the size of the asm is then taken as the minimum size of one instruction, ignoring how many instructions compiler thinks it is. The effect of this patch on x86_64 target is minor, since 128-bit functions are rarely used on this target. The code size of the resulting defconfig object file stays the same: text data bss dec hex filename 27456612 4638523 814148 32909283 1f627e3 vmlinux-old.o 27456612 4638523 814148 32909283 1f627e3 vmlinux-new.o but the patch has minor effect on code layout due to the different scheduling decisions in functions containing changed macros. There is no effect on x64_32 target, the code size of the resulting defconfig object file and the code layout stays the same: text data bss dec hex filename 18883870 2679275 1707916 23271061 1631695 vmlinux-old.o 18883870 2679275 1707916 23271061 1631695 vmlinux-new.o [1] https://gcc.gnu.org/onlinedocs/gcc/Size-of-an-asm.html [2] https://gcc.gnu.org/pipermail/gcc-patches/2018-December/512349.html Signed-off-by: Uros Bizjak Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: "Peter Zijlstra (Intel)" --- v2: Expand commit message with the explanation of asm_inline the explanation of benefits of the patch and with some code size measurements. --- arch/x86/include/asm/cmpxchg_32.h | 32 +++++++------ arch/x86/include/asm/percpu.h | 77 +++++++++++++++---------------- 2 files changed, 55 insertions(+), 54 deletions(-) diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h index fd1282a783dd..95b5f990ca88 100644 --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -91,12 +91,14 @@ static __always_inline bool __try_cmpxchg64_local(volatile u64 *ptr, u64 *oldp, union __u64_halves o = { .full = (_old), }, \ n = { .full = (_new), }; \ \ - asm volatile(ALTERNATIVE(_lock_loc \ - "call cmpxchg8b_emu", \ - _lock "cmpxchg8b %a[ptr]", X86_FEATURE_CX8) \ - : ALT_OUTPUT_SP("+a" (o.low), "+d" (o.high)) \ - : "b" (n.low), "c" (n.high), [ptr] "S" (_ptr) \ - : "memory"); \ + asm_inline volatile( \ + ALTERNATIVE(_lock_loc \ + "call cmpxchg8b_emu", \ + _lock "cmpxchg8b %a[ptr]", X86_FEATURE_CX8) \ + : ALT_OUTPUT_SP("+a" (o.low), "+d" (o.high)) \ + : "b" (n.low), "c" (n.high), \ + [ptr] "S" (_ptr) \ + : "memory"); \ \ o.full; \ }) @@ -119,14 +121,16 @@ static __always_inline u64 arch_cmpxchg64_local(volatile u64 *ptr, u64 old, u64 n = { .full = (_new), }; \ bool ret; \ \ - asm volatile(ALTERNATIVE(_lock_loc \ - "call cmpxchg8b_emu", \ - _lock "cmpxchg8b %a[ptr]", X86_FEATURE_CX8) \ - CC_SET(e) \ - : ALT_OUTPUT_SP(CC_OUT(e) (ret), \ - "+a" (o.low), "+d" (o.high)) \ - : "b" (n.low), "c" (n.high), [ptr] "S" (_ptr) \ - : "memory"); \ + asm_inline volatile( \ + ALTERNATIVE(_lock_loc \ + "call cmpxchg8b_emu", \ + _lock "cmpxchg8b %a[ptr]", X86_FEATURE_CX8) \ + CC_SET(e) \ + : ALT_OUTPUT_SP(CC_OUT(e) (ret), \ + "+a" (o.low), "+d" (o.high)) \ + : "b" (n.low), "c" (n.high), \ + [ptr] "S" (_ptr) \ + : "memory"); \ \ if (unlikely(!ret)) \ *(_oldp) = o.full; \ diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 0ab991fba7de..08f5f61690b7 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -348,15 +348,14 @@ do { \ old__.var = _oval; \ new__.var = _nval; \ \ - asm qual (ALTERNATIVE("call this_cpu_cmpxchg8b_emu", \ - "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \ - : ALT_OUTPUT_SP([var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high)) \ - : "b" (new__.low), \ - "c" (new__.high), \ - "S" (&(_var)) \ - : "memory"); \ + asm_inline qual ( \ + ALTERNATIVE("call this_cpu_cmpxchg8b_emu", \ + "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \ + : ALT_OUTPUT_SP([var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), "+d" (old__.high)) \ + : "b" (new__.low), "c" (new__.high), \ + "S" (&(_var)) \ + : "memory"); \ \ old__.var; \ }) @@ -378,17 +377,16 @@ do { \ old__.var = *_oval; \ new__.var = _nval; \ \ - asm qual (ALTERNATIVE("call this_cpu_cmpxchg8b_emu", \ - "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \ - CC_SET(z) \ - : ALT_OUTPUT_SP(CC_OUT(z) (success), \ - [var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high)) \ - : "b" (new__.low), \ - "c" (new__.high), \ - "S" (&(_var)) \ - : "memory"); \ + asm_inline qual ( \ + ALTERNATIVE("call this_cpu_cmpxchg8b_emu", \ + "cmpxchg8b " __percpu_arg([var]), X86_FEATURE_CX8) \ + CC_SET(z) \ + : ALT_OUTPUT_SP(CC_OUT(z) (success), \ + [var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), "+d" (old__.high)) \ + : "b" (new__.low), "c" (new__.high), \ + "S" (&(_var)) \ + : "memory"); \ if (unlikely(!success)) \ *_oval = old__.var; \ \ @@ -419,15 +417,14 @@ do { \ old__.var = _oval; \ new__.var = _nval; \ \ - asm qual (ALTERNATIVE("call this_cpu_cmpxchg16b_emu", \ - "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \ - : ALT_OUTPUT_SP([var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high)) \ - : "b" (new__.low), \ - "c" (new__.high), \ - "S" (&(_var)) \ - : "memory"); \ + asm_inline qual ( \ + ALTERNATIVE("call this_cpu_cmpxchg16b_emu", \ + "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \ + : ALT_OUTPUT_SP([var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), "+d" (old__.high)) \ + : "b" (new__.low), "c" (new__.high), \ + "S" (&(_var)) \ + : "memory"); \ \ old__.var; \ }) @@ -449,19 +446,19 @@ do { \ old__.var = *_oval; \ new__.var = _nval; \ \ - asm qual (ALTERNATIVE("call this_cpu_cmpxchg16b_emu", \ - "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \ - CC_SET(z) \ - : ALT_OUTPUT_SP(CC_OUT(z) (success), \ - [var] "+m" (__my_cpu_var(_var)), \ - "+a" (old__.low), \ - "+d" (old__.high)) \ - : "b" (new__.low), \ - "c" (new__.high), \ - "S" (&(_var)) \ - : "memory"); \ + asm_inline qual ( \ + ALTERNATIVE("call this_cpu_cmpxchg16b_emu", \ + "cmpxchg16b " __percpu_arg([var]), X86_FEATURE_CX16) \ + CC_SET(z) \ + : ALT_OUTPUT_SP(CC_OUT(z) (success), \ + [var] "+m" (__my_cpu_var(_var)), \ + "+a" (old__.low), "+d" (old__.high)) \ + : "b" (new__.low), "c" (new__.high), \ + "S" (&(_var)) \ + : "memory"); \ if (unlikely(!success)) \ *_oval = old__.var; \ + \ likely(success); \ })