From patchwork Sun Oct 1 10:39:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13405353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E67BE748F6 for ; Sun, 1 Oct 2023 10:41:35 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.610998.950432 (Exim 4.92) (envelope-from ) id 1qmtsp-0000CY-RJ; Sun, 01 Oct 2023 10:41:07 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 610998.950432; Sun, 01 Oct 2023 10:41:07 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qmtsp-0000CR-Od; Sun, 01 Oct 2023 10:41:07 +0000 Received: by outflank-mailman (input) for mailman id 610998; Sun, 01 Oct 2023 10:41:06 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qmtso-0000CG-Ra for xen-devel@lists.xenproject.org; Sun, 01 Oct 2023 10:41:06 +0000 Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [2a00:1450:4864:20::633]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 0581e699-6047-11ee-878e-cb3800f73035; Sun, 01 Oct 2023 12:41:04 +0200 (CEST) Received: by mail-ej1-x633.google.com with SMTP id a640c23a62f3a-99c3c8adb27so2174763966b.1 for ; Sun, 01 Oct 2023 03:41:04 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id jx14-20020a170906ca4e00b0099caf5bed64sm15333657ejb.57.2023.10.01.03.41.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Oct 2023 03:41:03 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 0581e699-6047-11ee-878e-cb3800f73035 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696156864; x=1696761664; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=hd3IQ5VOgFAiZIrz9J6WO1XKlWIuJJ+vlxIzopklsp0=; b=LgEv3X/KiAiQ2EP9sTRdG6crDMhtnFEh+mRN1JVtnOukCD1JtJWZ9sr6dT/FZGAtI5 /uQ3k9dv17atpZ7uaf1RpXzWbLGPtG1SQa99IsRnKxWMTqCauh4Wh2P4RJ1+vJ16URGe uRbxFuLJoXPNtQq7h7BouSf+1ffGN4OJYSEeDJncfOyRcHWh+MeG6CZUCAThBj4xkEun ctDYY5P6xer+Fg1npDrGdpZCTiUQInbWAlkBXJDluhXgdHuzZ84AgRMhSQ6qduUTL8BR B/TvzP1/D53Nd2HhVNC+mc4Ql5FDnKBG2DJ+/pB1U7mIp3LvAeD07/o8Ms0OgP7J5OrT WFpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696156864; x=1696761664; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hd3IQ5VOgFAiZIrz9J6WO1XKlWIuJJ+vlxIzopklsp0=; b=NuhsdhmXowd1TuVY2GAKLw7z3GIHoStc8iwNHlhroeJe8amJoB87CZJfmmNh9tJSHV TDSV2rPvMVNmvvjQ6PivKkJxJltTRmPSF6hgWzESbv1Kg1Py45v/kCfz9ZeAP3Fic8+S ljsEIb5/8kKEDSHoD0IryC4CPaRWcLwaRKdrm+K5qOBZHTKnurTUbkJGgQmOnlgwHOLb AiiT8iEBBgCvnyd8lRmNV5AO3qXaU4C5BYE5q1TXJ7zBqIOi3Vh4/arUbOFQq2v3pSz9 c8SxevBHZJJok9S2Xq/mDIrJnP0P9I4XSGMSgE5BNQ9oj1JyVNro/R0nw05IiOT9ekmb HqYw== X-Gm-Message-State: AOJu0YzHZukMCnGVujhOBiPxLL1vOZw+WEW8Y0Jzk2aKVeBdPYs6qe5q gXANvrYpq3c4VSIgbahn4cs= X-Google-Smtp-Source: AGHT+IG9JZc6TmvZHbVpgjX2iUva5UEpfEXoJsk9uCuk4m2g+BHUY+ApqFLeLJsTAyFlG09evitVDA== X-Received: by 2002:a17:907:774f:b0:9ae:673a:88b9 with SMTP id kx15-20020a170907774f00b009ae673a88b9mr7550136ejc.22.1696156863846; Sun, 01 Oct 2023 03:41:03 -0700 (PDT) From: Uros Bizjak To: linux-kernel@vger.kernel.org, x86@kernel.org, xen-devel@lists.xenproject.org Cc: Uros Bizjak , Will Deacon , Peter Zijlstra , Ingo Molnar , Boqun Feng , Mark Rutland Subject: [PATCH v2 1/3] locking/generic: Add generic support for sync_try_cmpxchg and its fallback Date: Sun, 1 Oct 2023 12:39:09 +0200 Message-ID: <20231001104053.9644-1-ubizjak@gmail.com> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Provide the generic sync_try_cmpxchg function from the raw_ prefixed version, also adding explicit instrumentation. The patch amends existing scripts to generate sync_try_cmpxchg locking primitive and its raw_sync_try_cmpxchg fallback, while leaving existing macros from try_cmpxchg family unchanged. The target can define its own arch_sync_try_cmpxchg to override the generic version of raw_sync_try_cmpxchg. This allows the target to generate more optimal assembly than the generic version. Additionally, the patch renames two scripts to better reflect whet they really do. Cc: Will Deacon Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Boqun Feng Cc: Mark Rutland Signed-off-by: Uros Bizjak --- v2: Improve commit description. --- include/linux/atomic/atomic-arch-fallback.h | 15 +++++++++- include/linux/atomic/atomic-instrumented.h | 10 ++++++- scripts/atomic/gen-atomic-fallback.sh | 33 +++++++++++---------- scripts/atomic/gen-atomic-instrumented.sh | 3 +- 4 files changed, 43 insertions(+), 18 deletions(-) diff --git a/include/linux/atomic/atomic-arch-fallback.h b/include/linux/atomic/atomic-arch-fallback.h index b83ef19da13d..5e95faa959c4 100644 --- a/include/linux/atomic/atomic-arch-fallback.h +++ b/include/linux/atomic/atomic-arch-fallback.h @@ -428,6 +428,19 @@ extern void raw_cmpxchg128_relaxed_not_implemented(void); #define raw_sync_cmpxchg arch_sync_cmpxchg +#ifdef arch_sync_try_cmpxchg +#define raw_sync_try_cmpxchg arch_sync_try_cmpxchg +#else +#define raw_sync_try_cmpxchg(_ptr, _oldp, _new) \ +({ \ + typeof(*(_ptr)) *___op = (_oldp), ___o = *___op, ___r; \ + ___r = raw_sync_cmpxchg((_ptr), ___o, (_new)); \ + if (unlikely(___r != ___o)) \ + *___op = ___r; \ + likely(___r == ___o); \ +}) +#endif + /** * raw_atomic_read() - atomic load with relaxed ordering * @v: pointer to atomic_t @@ -4649,4 +4662,4 @@ raw_atomic64_dec_if_positive(atomic64_t *v) } #endif /* _LINUX_ATOMIC_FALLBACK_H */ -// 2fdd6702823fa842f9cea57a002e6e4476ae780c +// eec048affea735b8464f58e6d96992101f8f85f1 diff --git a/include/linux/atomic/atomic-instrumented.h b/include/linux/atomic/atomic-instrumented.h index d401b406ef7c..54d7bbe0aeaa 100644 --- a/include/linux/atomic/atomic-instrumented.h +++ b/include/linux/atomic/atomic-instrumented.h @@ -4998,6 +4998,14 @@ atomic_long_dec_if_positive(atomic_long_t *v) raw_try_cmpxchg128_local(__ai_ptr, __ai_oldp, __VA_ARGS__); \ }) +#define sync_try_cmpxchg(ptr, ...) \ +({ \ + typeof(ptr) __ai_ptr = (ptr); \ + kcsan_mb(); \ + instrument_atomic_read_write(__ai_ptr, sizeof(*__ai_ptr)); \ + raw_sync_try_cmpxchg(__ai_ptr, __VA_ARGS__); \ +}) + #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */ -// 1568f875fef72097413caab8339120c065a39aa4 +// 2cc4bc990fef44d3836ec108f11b610f3f438184 diff --git a/scripts/atomic/gen-atomic-fallback.sh b/scripts/atomic/gen-atomic-fallback.sh index a45154cefa48..f80d69cfeb1f 100755 --- a/scripts/atomic/gen-atomic-fallback.sh +++ b/scripts/atomic/gen-atomic-fallback.sh @@ -223,14 +223,15 @@ gen_xchg_fallbacks() gen_try_cmpxchg_fallback() { + local prefix="$1"; shift local cmpxchg="$1"; shift; - local order="$1"; shift; + local suffix="$1"; shift; cat < X-Patchwork-Id: 13405352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DC1B8E748ED for ; Sun, 1 Oct 2023 10:41:31 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.610999.950439 (Exim 4.92) (envelope-from ) id 1qmtsq-0000HE-4p; Sun, 01 Oct 2023 10:41:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 610999.950439; Sun, 01 Oct 2023 10:41:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qmtsp-0000FS-V5; Sun, 01 Oct 2023 10:41:07 +0000 Received: by outflank-mailman (input) for mailman id 610999; Sun, 01 Oct 2023 10:41:07 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qmtsp-0000CG-3Z for xen-devel@lists.xenproject.org; Sun, 01 Oct 2023 10:41:07 +0000 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [2a00:1450:4864:20::62e]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 05e223a0-6047-11ee-878e-cb3800f73035; Sun, 01 Oct 2023 12:41:05 +0200 (CEST) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-9b281a2aa94so1532197566b.2 for ; Sun, 01 Oct 2023 03:41:05 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id jx14-20020a170906ca4e00b0099caf5bed64sm15333657ejb.57.2023.10.01.03.41.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Oct 2023 03:41:04 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 05e223a0-6047-11ee-878e-cb3800f73035 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696156865; x=1696761665; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w6ur2qb0x8q/rSGOpgVJ8elvvt/Yb95eKWrvLo4QZJM=; b=fLCBdRga6In0ISY52389OVu1Sef8QQrL/3vlC5+YvKN0aRwu9of1czxC2lc/pozA4u NVZfqdIXHXSAivZYIXfcZv3RfAuZgoDziY4VqAVxQk7nJjIzzqdCZnpIHkL+ctLJbVxW Zi4lis9Jbk/HKUamEc/gAc0DPHJQd3+j0SGFa0kmKsxLHjE+RChkizMPh2xRSo1g+pAM x69xfq+cQaBNzvQgZ9p0PEBTppCYUfCFD2DcdpLcWskLiAwxP/eQNZG+78/davFGCpan w4ZgM262hZqACC49H89RxBOvzwwxFVHyuCbRFPFTRgJQ4x+s5sSVV026LVt6uiCx2Vcs 33pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696156865; x=1696761665; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w6ur2qb0x8q/rSGOpgVJ8elvvt/Yb95eKWrvLo4QZJM=; b=wcoOekZpzrpQ5zwScuu4ji6C1JGwx8a/EmatnhlCGCKq7F9x4lzGliNDfLMxwdecPd 3Zv0FNptiAQThrx+tOBk1AZMcs6diIeu3PHad2ltNOs3PtYUypFGRGnTLe4rUHZh2wKu ay63fGMNnvJTqbyLYFCt0+Z1XtBm2+aBMutuAQmWGpg+G0bIclRyaTQfJE/kgmiGpPSO yOqKj4oGT7dIC3CeoBcwnVdf0LijJ2pE29GzlHsZ8Cbirw+WLI5o8TTJH1BMM1RzTi7V xJHvh7//VaoEvl2+YL8gCSKXsMknX5id9R9EmpA7Wyz/lqnwZBIQYNOatZrhHzhJMSsr lO2g== X-Gm-Message-State: AOJu0YyQB5W9G/VLwalE9OmrNecChjoHORquNkqMdOAC0P2lCPnsKvDT 9S2XvqNaglLhk14kaqD6SIo= X-Google-Smtp-Source: AGHT+IHiSe0AHyaEO3R17ZhQDfsCes5+q/9A9cLMJMLgBfw1ghBgw9I+0cdDupifN0wboisrAnr7BQ== X-Received: by 2002:a17:906:51d5:b0:9a2:26d8:f184 with SMTP id v21-20020a17090651d500b009a226d8f184mr7068105ejk.51.1696156864627; Sun, 01 Oct 2023 03:41:04 -0700 (PDT) From: Uros Bizjak To: linux-kernel@vger.kernel.org, x86@kernel.org, xen-devel@lists.xenproject.org Cc: Uros Bizjak , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v2 2/3] locking/x86: Introduce arch_sync_try_cmpxchg Date: Sun, 1 Oct 2023 12:39:10 +0200 Message-ID: <20231001104053.9644-2-ubizjak@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231001104053.9644-1-ubizjak@gmail.com> References: <20231001104053.9644-1-ubizjak@gmail.com> MIME-Version: 1.0 Introduce arch_sync_try_cmpxchg macro to improve code using sync_try_cmpxchg locking primitive. The new definitions use existing __raw_try_cmpxchg macros, but use its own "lock; " prefix. The new macros improve assembly of the cmpxchg loop in evtchn_fifo_unmask() from drivers/xen/events/events_fifo.c from: 57a: 85 c0 test %eax,%eax 57c: 78 52 js 5d0 <...> 57e: 89 c1 mov %eax,%ecx 580: 25 ff ff ff af and $0xafffffff,%eax 585: c7 04 24 00 00 00 00 movl $0x0,(%rsp) 58c: 81 e1 ff ff ff ef and $0xefffffff,%ecx 592: 89 4c 24 04 mov %ecx,0x4(%rsp) 596: 89 44 24 08 mov %eax,0x8(%rsp) 59a: 8b 74 24 08 mov 0x8(%rsp),%esi 59e: 8b 44 24 04 mov 0x4(%rsp),%eax 5a2: f0 0f b1 32 lock cmpxchg %esi,(%rdx) 5a6: 89 04 24 mov %eax,(%rsp) 5a9: 8b 04 24 mov (%rsp),%eax 5ac: 39 c1 cmp %eax,%ecx 5ae: 74 07 je 5b7 <...> 5b0: a9 00 00 00 40 test $0x40000000,%eax 5b5: 75 c3 jne 57a <...> <...> to: 578: a9 00 00 00 40 test $0x40000000,%eax 57d: 74 2b je 5aa <...> 57f: 85 c0 test %eax,%eax 581: 78 40 js 5c3 <...> 583: 89 c1 mov %eax,%ecx 585: 25 ff ff ff af and $0xafffffff,%eax 58a: 81 e1 ff ff ff ef and $0xefffffff,%ecx 590: 89 4c 24 04 mov %ecx,0x4(%rsp) 594: 89 44 24 08 mov %eax,0x8(%rsp) 598: 8b 4c 24 08 mov 0x8(%rsp),%ecx 59c: 8b 44 24 04 mov 0x4(%rsp),%eax 5a0: f0 0f b1 0a lock cmpxchg %ecx,(%rdx) 5a4: 89 44 24 04 mov %eax,0x4(%rsp) 5a8: 75 30 jne 5da <...> <...> 5da: 8b 44 24 04 mov 0x4(%rsp),%eax 5de: eb 98 jmp 578 <...> The new code removes move instructions from 585: 5a6: and 5a9: and the compare from 5ac:. Additionally, the compiler assumes that cmpxchg success is more probable and optimizes code flow accordingly. Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Signed-off-by: Uros Bizjak --- v2: Improve commit description. --- arch/x86/include/asm/cmpxchg.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index d53636506134..5612648b0202 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -221,12 +221,18 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) +#define __sync_try_cmpxchg(ptr, pold, new, size) \ + __raw_try_cmpxchg((ptr), (pold), (new), (size), "lock; ") + #define __try_cmpxchg_local(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), "") #define arch_try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) +#define arch_sync_try_cmpxchg(ptr, pold, new) \ + __sync_try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) + #define arch_try_cmpxchg_local(ptr, pold, new) \ __try_cmpxchg_local((ptr), (pold), (new), sizeof(*(ptr))) From patchwork Sun Oct 1 10:39:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uros Bizjak X-Patchwork-Id: 13405354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 13E7DE748F7 for ; Sun, 1 Oct 2023 10:41:36 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.611000.950453 (Exim 4.92) (envelope-from ) id 1qmtsu-0000kF-94; Sun, 01 Oct 2023 10:41:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 611000.950453; Sun, 01 Oct 2023 10:41:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qmtsu-0000k8-60; Sun, 01 Oct 2023 10:41:12 +0000 Received: by outflank-mailman (input) for mailman id 611000; Sun, 01 Oct 2023 10:41:10 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qmtss-0000j6-Rd for xen-devel@lists.xenproject.org; Sun, 01 Oct 2023 10:41:10 +0000 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [2a00:1450:4864:20::62e]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 0698bd3e-6047-11ee-9b0d-b553b5be7939; Sun, 01 Oct 2023 12:41:06 +0200 (CEST) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-99357737980so2133592366b.2 for ; Sun, 01 Oct 2023 03:41:06 -0700 (PDT) Received: from localhost.localdomain ([46.248.82.114]) by smtp.gmail.com with ESMTPSA id jx14-20020a170906ca4e00b0099caf5bed64sm15333657ejb.57.2023.10.01.03.41.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Oct 2023 03:41:05 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 0698bd3e-6047-11ee-9b0d-b553b5be7939 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696156866; x=1696761666; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Av1YVGUYjKuKpOxhCUYf9bVAmtK6lNc0JKDBqDmChEQ=; b=QRxPxvhT7dgDeFxDXSqCVh7uF7cc6Mhck78m3K3lj/XQZsnfvYmOpvYgH2ua5wX3yB L4bTiWTmg1TsSO9Bdx9dc4ya8XIBID96M+evuLRc5W3THhFIJgrUYObhOigzh63N3Kc8 0IDd8j0em2MS8tHdPJxHmiXQXYb8pz0UUpnC8dvNHRMSe8Mn3YQLYRuE2goLye4OFGFy vKCmcC7+3Hlhi2XAHerHWGozQl6BhsUScciukFfW/Bb081tkpc84215SdkfHl+AoX3Tw IQzJyuVnOrxiWkxqlQm+HmCwG/z6Tpw1oCsIE9S34G4GbGCvS1Hn+AdrT0IXZWuDdMFa NNBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696156866; x=1696761666; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Av1YVGUYjKuKpOxhCUYf9bVAmtK6lNc0JKDBqDmChEQ=; b=QvqGszteoMESD/WWcSBx4x9kOrvDg6EQGCY8RtwvAcPM2d1NxQNvLSIq7iPKwmhTDU mMYgrw5diaYc1LmgAu/ySURtD5QAjA++Rr1hcZnJTW2SvD07uoOgBoqljcf9JyMr66JX 2eShB/d3A+1nqbmzJ8w6u4FiEjbt4cHImrllCWTtRW8V9iH779UmNN46/sTV0oTRC5Hc 4vIkuA0ZcC+XZQLSnULHYlrC92d5cstY9fJ8N8Mdh0n7QU/zotcjr0ABddy6lPSUx4pA tdRIQVPSqZFlKGrUlzkveq9RPJ7x5JwLz3NW3sXL5oPb5XluhREReIv38tYcrmgWBo8u bVuQ== X-Gm-Message-State: AOJu0YzY+nTE+ldvtlA+s02wmw88niaalTH1uv+YBhhVdjZfrPy7pB52 5yg/QKoYsWemHZXKEpj6FdzLbW1ixLW10w== X-Google-Smtp-Source: AGHT+IEaRBVRFHBq8PsqtS7Xotm/O/WCU7PUdbnpb1YCwTqlB9U799c5jGmIPkQv2u5NF0F2mavHLA== X-Received: by 2002:a17:906:7695:b0:9ad:e17c:464e with SMTP id o21-20020a170906769500b009ade17c464emr8738266ejm.68.1696156865773; Sun, 01 Oct 2023 03:41:05 -0700 (PDT) From: Uros Bizjak To: linux-kernel@vger.kernel.org, x86@kernel.org, xen-devel@lists.xenproject.org Cc: Uros Bizjak , Peter Zijlstra , Ingo Molnar , Stefano Stabellini , Oleksandr Tyshchenko , Juergen Gross Subject: [PATCH v2 3/3] xen: Use sync_try_cmpxchg instead of sync_cmpxchg Date: Sun, 1 Oct 2023 12:39:11 +0200 Message-ID: <20231001104053.9644-3-ubizjak@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231001104053.9644-1-ubizjak@gmail.com> References: <20231001104053.9644-1-ubizjak@gmail.com> MIME-Version: 1.0 Use sync_try_cmpxchg instead of sync_cmpxchg (*ptr, old, new) == old in clear_masked_cond(), clear_linked() and gnttab_end_foreign_access_ref_v1(). x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg), improving the cmpxchg loop in gnttab_end_foreign_access_ref_v1 from: 174: eb 0e jmp 184 <...> 176: 89 d0 mov %edx,%eax 178: f0 66 0f b1 31 lock cmpxchg %si,(%rcx) 17d: 66 39 c2 cmp %ax,%dx 180: 74 11 je 193 <...> 182: 89 c2 mov %eax,%edx 184: 89 d6 mov %edx,%esi 186: 66 83 e6 18 and $0x18,%si 18a: 74 ea je 176 <...> to: 614: 89 c1 mov %eax,%ecx 616: 66 83 e1 18 and $0x18,%cx 61a: 75 11 jne 62d <...> 61c: f0 66 0f b1 0a lock cmpxchg %cx,(%rdx) 621: 75 f1 jne 614 <...> No functional change intended. Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Stefano Stabellini Cc: Oleksandr Tyshchenko Acked-by: Juergen Gross Signed-off-by: Uros Bizjak --- v2: Improve commit description. --- drivers/xen/events/events_fifo.c | 26 ++++++++++++-------------- drivers/xen/grant-table.c | 10 ++++------ 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/drivers/xen/events/events_fifo.c b/drivers/xen/events/events_fifo.c index ad9fe51d3fb3..655775db7caf 100644 --- a/drivers/xen/events/events_fifo.c +++ b/drivers/xen/events/events_fifo.c @@ -226,21 +226,20 @@ static bool evtchn_fifo_is_masked(evtchn_port_t port) */ static bool clear_masked_cond(volatile event_word_t *word) { - event_word_t new, old, w; + event_word_t new, old; - w = *word; + old = *word; do { - if (!(w & (1 << EVTCHN_FIFO_MASKED))) + if (!(old & (1 << EVTCHN_FIFO_MASKED))) return true; - if (w & (1 << EVTCHN_FIFO_PENDING)) + if (old & (1 << EVTCHN_FIFO_PENDING)) return false; - old = w & ~(1 << EVTCHN_FIFO_BUSY); + old = old & ~(1 << EVTCHN_FIFO_BUSY); new = old & ~(1 << EVTCHN_FIFO_MASKED); - w = sync_cmpxchg(word, old, new); - } while (w != old); + } while (!sync_try_cmpxchg(word, &old, new)); return true; } @@ -259,17 +258,16 @@ static void evtchn_fifo_unmask(evtchn_port_t port) static uint32_t clear_linked(volatile event_word_t *word) { - event_word_t new, old, w; + event_word_t new, old; - w = *word; + old = *word; do { - old = w; - new = (w & ~((1 << EVTCHN_FIFO_LINKED) - | EVTCHN_FIFO_LINK_MASK)); - } while ((w = sync_cmpxchg(word, old, new)) != old); + new = (old & ~((1 << EVTCHN_FIFO_LINKED) + | EVTCHN_FIFO_LINK_MASK)); + } while (!sync_try_cmpxchg(word, &old, new)); - return w & EVTCHN_FIFO_LINK_MASK; + return old & EVTCHN_FIFO_LINK_MASK; } static void consume_one_event(unsigned cpu, struct evtchn_loop_ctrl *ctrl, diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c index 35659bf70746..04a6b470b15d 100644 --- a/drivers/xen/grant-table.c +++ b/drivers/xen/grant-table.c @@ -427,16 +427,14 @@ EXPORT_SYMBOL_GPL(gnttab_grant_foreign_access); static int gnttab_end_foreign_access_ref_v1(grant_ref_t ref) { - u16 flags, nflags; - u16 *pflags; + u16 *pflags = &gnttab_shared.v1[ref].flags; + u16 flags; - pflags = &gnttab_shared.v1[ref].flags; - nflags = *pflags; + flags = *pflags; do { - flags = nflags; if (flags & (GTF_reading|GTF_writing)) return 0; - } while ((nflags = sync_cmpxchg(pflags, flags, 0)) != flags); + } while (!sync_try_cmpxchg(pflags, &flags, 0)); return 1; }