From patchwork Tue Nov 5 18:30:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Okanovic, Haris" X-Patchwork-Id: 13863360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F1A3D3177E for ; Tue, 5 Nov 2024 18:35:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bT6rRx7a7sWccvBSdvoPKKTuXoItJoU+EpMvF26uVsc=; b=ih2vggD4uYdWQ3YwwrP4O3HUMB UAHMd6BXrvOINrAQk92xNh1f59mxiTe6CCNF+lxYeLeSNozLLlCzm6SCpn5mRZCOwZWuBUVqHpvvw HoP3VwTT/O/Yvl6th8nlO3Pwl97i3Qog9RBYEDOMF8hopQgoKcuuE778mQl22LZSgRrtqYzmUnbYg BdkJonlvhMXu8frVCddbY3Y4foinBoNHCBO9aQF3HEkx5XiPVP3vCyXe7T96sgSRPsrW8YI6uS/qP eCQ9dPHeb9lVUgbCJDnR0RV/+RU59Qq7uDS3XrD4MvCDOxtMUASsL+P+m8G4c7mD5RkHO0IqmE4to dAPrWKtQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t8OOE-00000000O0O-0amd; Tue, 05 Nov 2024 18:34:54 +0000 Received: from smtp-fw-52004.amazon.com ([52.119.213.154]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t8OLB-00000000NP3-0Zpt for linux-arm-kernel@lists.infradead.org; Tue, 05 Nov 2024 18:31:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1730831505; x=1762367505; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bT6rRx7a7sWccvBSdvoPKKTuXoItJoU+EpMvF26uVsc=; b=Gf6Y1sCTSF+raYdgerjfLmYouQlnqi4wur4jY1778cWqoApI95HGDdWX 7hQH5ydEmdrT7PqnNGyMJGmF9wIErUS3/CzY+CPCegybMH4epu3oElsPG WIUPIOUVDLydGTuz27dhKjm4oQGbjyTjf3K38+706UkJc+Pb1qoO46VVf M=; X-IronPort-AV: E=Sophos;i="6.11,260,1725321600"; d="scan'208";a="245128567" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.43.8.2]) by smtp-border-fw-52004.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Nov 2024 18:31:40 +0000 Received: from EX19MTAUWA001.ant.amazon.com [10.0.21.151:50603] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.28.125:2525] with esmtp (Farcaster) id 39bb5814-4f48-4caf-ba41-fae750bfbae2; Tue, 5 Nov 2024 18:31:40 +0000 (UTC) X-Farcaster-Flow-ID: 39bb5814-4f48-4caf-ba41-fae750bfbae2 Received: from EX19D001UWA003.ant.amazon.com (10.13.138.211) by EX19MTAUWA001.ant.amazon.com (10.250.64.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 5 Nov 2024 18:31:39 +0000 Received: from u34cccd802f2d52.amazon.com (10.106.239.17) by EX19D001UWA003.ant.amazon.com (10.13.138.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Tue, 5 Nov 2024 18:31:34 +0000 From: Haris Okanovic To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 1/5] asm-generic: add smp_vcond_load_relaxed() Date: Tue, 5 Nov 2024 12:30:37 -0600 Message-ID: <20241105183041.1531976-2-harisokn@amazon.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241105183041.1531976-1-harisokn@amazon.com> References: <20240925232425.2763385-1-ankur.a.arora@oracle.com> <20241105183041.1531976-1-harisokn@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.106.239.17] X-ClientProxiedBy: EX19D033UWA004.ant.amazon.com (10.13.139.85) To EX19D001UWA003.ant.amazon.com (10.13.138.211) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241105_103145_386828_C9410833 X-CRM114-Status: GOOD ( 10.68 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Relaxed poll until desired mask/value is observed at the specified address or timeout. This macro is a specialization of the generic smp_cond_load_relaxed(), which takes a simple mask/value condition (vcond) instead of an arbitrary expression. It allows architectures to better specialize the implementation, e.g. to enable wfe() polling of the address on arm. Signed-off-by: Haris Okanovic --- include/asm-generic/barrier.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h index d4f581c1e21d..112027eabbfc 100644 --- a/include/asm-generic/barrier.h +++ b/include/asm-generic/barrier.h @@ -256,6 +256,31 @@ do { \ }) #endif +/** + * smp_vcond_load_relaxed() - (Spin) wait until an expected value at address + * with no ordering guarantees. Spins until `(*addr & mask) == val` or + * `nsecs` elapse, and returns the last observed `*addr` value. + * + * @nsecs: timeout in nanoseconds + * @addr: pointer to an integer + * @mask: a bit mask applied to read values + * @val: Expected value with mask + */ +#ifndef smp_vcond_load_relaxed +#define smp_vcond_load_relaxed(nsecs, addr, mask, val) ({ \ + const u64 __start = local_clock_noinstr(); \ + u64 __nsecs = (nsecs); \ + typeof(addr) __addr = (addr); \ + typeof(*__addr) __mask = (mask); \ + typeof(*__addr) __val = (val); \ + typeof(*__addr) __cur; \ + smp_cond_load_relaxed(__addr, ( \ + (VAL & __mask) == __val || \ + local_clock_noinstr() - __start > __nsecs \ + )); \ +}) +#endif + /** * smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering * @ptr: pointer to the variable to wait on From patchwork Tue Nov 5 18:30:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Okanovic, Haris" X-Patchwork-Id: 13863361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA152D3177E for ; Tue, 5 Nov 2024 18:36:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aZOVkbd/aqSS6wL2tHU2ZYavLfm4wZ7pJZkt2CDBd7U=; b=WzPR9FJIK6YMyS2jbv+cNWHrrN Ni3wdwKxSlx3eFYkSqE/8Sv7e2pVNQmKgI9g/SBcBtzFGiwG6EAtcepx7bmKHyyPRWfYq0agNsrFq AaqYE1sIjnAxcs7SV/pqhTeQcfoGo32QDIG6g99cGLOL1YmVvp9yEMUm8Jgpg4Vbz4dVVH0rnoE0Q QG8o6R1f00/1WeXE68m3AAuy08xzJdDLZTceUJ/iq5NpUvXgrjuIehrMB9DRIWF7onpYZU+BOtl/C WDNuNaiDIAzfvx/xrVzteFgakBIuZ6+EtV43lzbcikVMgRGFuNfu7INH23YAGBy0F2uObFgtVQz8V l6nZxzYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t8OPu-00000000OFl-0JC8; Tue, 05 Nov 2024 18:36:38 +0000 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t8OLM-00000000NSQ-34xX for linux-arm-kernel@lists.infradead.org; Tue, 05 Nov 2024 18:31:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1730831517; x=1762367517; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aZOVkbd/aqSS6wL2tHU2ZYavLfm4wZ7pJZkt2CDBd7U=; b=lgahfxVR8Rr5NjZ0abKtetzZwqeJURjgg3Hv4o01q7EFbRgrC7j5nFfC R3DLEpPdkRzEg45t9i70KDfuaxB9L6zgLPTpk0VA/cwNURlwxNp+eXAPq HNDVRVsASJ6xIvcOAiQFr95WTXZnTCmnIIDoFS6oGOqaa7F3xVP9cTmUk A=; X-IronPort-AV: E=Sophos;i="6.11,260,1725321600"; d="scan'208";a="382672781" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-33001.sea14.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Nov 2024 18:31:56 +0000 Received: from EX19MTAUWB001.ant.amazon.com [10.0.38.20:51046] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.10.37:2525] with esmtp (Farcaster) id 3cfefb00-6800-4172-8693-375f47215f5a; Tue, 5 Nov 2024 18:31:55 +0000 (UTC) X-Farcaster-Flow-ID: 3cfefb00-6800-4172-8693-375f47215f5a Received: from EX19D001UWA003.ant.amazon.com (10.13.138.211) by EX19MTAUWB001.ant.amazon.com (10.250.64.248) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 5 Nov 2024 18:31:55 +0000 Received: from u34cccd802f2d52.amazon.com (10.106.239.17) by EX19D001UWA003.ant.amazon.com (10.13.138.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Tue, 5 Nov 2024 18:31:50 +0000 From: Haris Okanovic To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 2/5] arm64: add __READ_ONCE_EX() Date: Tue, 5 Nov 2024 12:30:38 -0600 Message-ID: <20241105183041.1531976-3-harisokn@amazon.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241105183041.1531976-1-harisokn@amazon.com> References: <20240925232425.2763385-1-ankur.a.arora@oracle.com> <20241105183041.1531976-1-harisokn@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.106.239.17] X-ClientProxiedBy: EX19D033UWA004.ant.amazon.com (10.13.139.85) To EX19D001UWA003.ant.amazon.com (10.13.138.211) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241105_103156_873191_3963411F X-CRM114-Status: GOOD ( 11.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Perform an exclusive load, which atomically loads a word and arms the exclusive monitor to enable wfet()/wfe() accelerated polling. https://developer.arm.com/documentation/dht0008/a/arm-synchronization-primitives/exclusive-accesses/exclusive-monitors Signed-off-by: Haris Okanovic --- arch/arm64/include/asm/readex.h | 46 +++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 arch/arm64/include/asm/readex.h diff --git a/arch/arm64/include/asm/readex.h b/arch/arm64/include/asm/readex.h new file mode 100644 index 000000000000..51963c3107e1 --- /dev/null +++ b/arch/arm64/include/asm/readex.h @@ -0,0 +1,46 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Based on arch/arm64/include/asm/rwonce.h + * + * Copyright (C) 2020 Google LLC. + * Copyright (C) 2024 Amazon.com, Inc. or its affiliates. + */ + +#ifndef __ASM_READEX_H +#define __ASM_READEX_H + +#define __LOAD_EX(sfx, regs...) "ldaxr" #sfx "\t" #regs + +#define __READ_ONCE_EX(x) \ +({ \ + typeof(&(x)) __x = &(x); \ + int atomic = 1; \ + union { __unqual_scalar_typeof(*__x) __val; char __c[1]; } __u; \ + switch (sizeof(x)) { \ + case 1: \ + asm volatile(__LOAD_EX(b, %w0, %1) \ + : "=r" (*(__u8 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 2: \ + asm volatile(__LOAD_EX(h, %w0, %1) \ + : "=r" (*(__u16 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 4: \ + asm volatile(__LOAD_EX(, %w0, %1) \ + : "=r" (*(__u32 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + case 8: \ + asm volatile(__LOAD_EX(, %0, %1) \ + : "=r" (*(__u64 *)__u.__c) \ + : "Q" (*__x) : "memory"); \ + break; \ + default: \ + atomic = 0; \ + } \ + atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(__x))__x);\ +}) + +#endif From patchwork Tue Nov 5 18:30:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Okanovic, Haris" X-Patchwork-Id: 13863436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 439CCD328A4 for ; Tue, 5 Nov 2024 18:38:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=jU1idzmOqOXlXb1t2IMpZrPTuLA9eFztDBmS3LLVVOw=; b=kbkanbIwqKRScOgI8A8zy8piKB /VaRBmEzUURDf5kR7wEj3KmQXfbMsVhPtJ3N0dI7EKMz1+LxBq87p2f1zq0VFWJeLGBTu0ZQ7s9Fv 78TFHfFXRa3+K/qrIoc5uDLmlz/Nz5RzUCc+NSDXqGBObFAcPV29gUnsVSUgktlkQ+23GB0vgdUDf EiaB56a5RJ5hiqqCmnnJ7KcrEBZKhbLFukhlc3KByYBHZjPBqtAz66pfSyBiC2VZ/IgtaWBR+iAzM oyFAAwHGGj5emJ/HCYNOyWcbKScfryH1panV44peyO0BGxqyIWBil24WRgpgmHc/lcC6yB6hFf9v+ ZnDKsP6A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t8ORd-00000000Oic-1A3e; Tue, 05 Nov 2024 18:38:25 +0000 Received: from smtp-fw-9106.amazon.com ([207.171.188.206]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t8OLg-00000000NYm-1DDP for linux-arm-kernel@lists.infradead.org; Tue, 05 Nov 2024 18:32:18 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1730831536; x=1762367536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jU1idzmOqOXlXb1t2IMpZrPTuLA9eFztDBmS3LLVVOw=; b=W2WNybKHNiaqe0fI03wsDQSkjpM3gn76FGWkycZNpaJA5eLt1UGteJvp R2t9t6rbgZFX+PkaKSvXOIIuGdI/D3A0BdS4DhwExTS7JSfUBNBwXjTL6 ntNwKxS0DzzSG/AuaQVuIQlvlvYkXZk/bwuRxww1a+pMKzUzyLcVO0qLB Y=; X-IronPort-AV: E=Sophos;i="6.11,260,1725321600"; d="scan'208";a="773030717" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-9106.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Nov 2024 18:32:13 +0000 Received: from EX19MTAUWA001.ant.amazon.com [10.0.21.151:34625] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.28.125:2525] with esmtp (Farcaster) id f9c0b0d1-8be1-4e25-ab5f-c68efeaab89c; Tue, 5 Nov 2024 18:32:12 +0000 (UTC) X-Farcaster-Flow-ID: f9c0b0d1-8be1-4e25-ab5f-c68efeaab89c Received: from EX19D001UWA003.ant.amazon.com (10.13.138.211) by EX19MTAUWA001.ant.amazon.com (10.250.64.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 5 Nov 2024 18:32:12 +0000 Received: from u34cccd802f2d52.amazon.com (10.106.239.17) by EX19D001UWA003.ant.amazon.com (10.13.138.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Tue, 5 Nov 2024 18:32:06 +0000 From: Haris Okanovic To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 3/5] arm64: refactor delay() to enable polling for value Date: Tue, 5 Nov 2024 12:30:39 -0600 Message-ID: <20241105183041.1531976-4-harisokn@amazon.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241105183041.1531976-1-harisokn@amazon.com> References: <20240925232425.2763385-1-ankur.a.arora@oracle.com> <20241105183041.1531976-1-harisokn@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.106.239.17] X-ClientProxiedBy: EX19D033UWA004.ant.amazon.com (10.13.139.85) To EX19D001UWA003.ant.amazon.com (10.13.138.211) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241105_103216_486432_B6602B4A X-CRM114-Status: GOOD ( 13.28 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Refactor arm64's delay() to poll for a mask/value condition (vcond) in it's wfet(), wfe(), and relaxed polling loops. Signed-off-by: Haris Okanovic --- arch/arm64/lib/delay.c | 70 ++++++++++++++++++++++++++++++------------ 1 file changed, 50 insertions(+), 20 deletions(-) diff --git a/arch/arm64/lib/delay.c b/arch/arm64/lib/delay.c index cb2062e7e234..a7c3040af316 100644 --- a/arch/arm64/lib/delay.c +++ b/arch/arm64/lib/delay.c @@ -14,43 +14,73 @@ #include #include +#include -#define USECS_TO_CYCLES(time_usecs) \ - xloops_to_cycles((time_usecs) * 0x10C7UL) - -static inline unsigned long xloops_to_cycles(unsigned long xloops) +static inline u64 xloops_to_cycles(u64 xloops) { return (xloops * loops_per_jiffy * HZ) >> 32; } -void __delay(unsigned long cycles) +#define USECS_TO_XLOOPS(time_usecs) \ + ((time_usecs) * 0x10C7UL) + +#define USECS_TO_CYCLES(time_usecs) \ + xloops_to_cycles(USECS_TO_XLOOPS(time_usecs)) + +#define NSECS_TO_XLOOPS(time_nsecs) \ + ((time_nsecs) * 0x10C7UL) + +#define NSECS_TO_CYCLES(time_nsecs) \ + xloops_to_cycles(NSECS_TO_XLOOPS(time_nsecs)) + +static unsigned long __delay_until_ul(u64 cycles, unsigned long* addr, unsigned long mask, unsigned long val) { - cycles_t start = get_cycles(); + u64 start = get_cycles(); + unsigned long cur; if (alternative_has_cap_unlikely(ARM64_HAS_WFXT)) { u64 end = start + cycles; - /* - * Start with WFIT. If an interrupt makes us resume - * early, use a WFET loop to complete the delay. - */ - wfit(end); - while ((get_cycles() - start) < cycles) + do { + cur = __READ_ONCE_EX(*addr); + if ((cur & mask) == val) { + break; + } wfet(end); - } else if (arch_timer_evtstrm_available()) { - const cycles_t timer_evt_period = + } while ((get_cycles() - start) < cycles); + } else if (arch_timer_evtstrm_available()) { + const u64 timer_evt_period = USECS_TO_CYCLES(ARCH_TIMER_EVT_STREAM_PERIOD_US); - while ((get_cycles() - start + timer_evt_period) < cycles) + do { + cur = __READ_ONCE_EX(*addr); + if ((cur & mask) == val) { + break; + } wfe(); + } while ((get_cycles() - start + timer_evt_period) < cycles); + } else { + do { + cur = __READ_ONCE_EX(*addr); + if ((cur & mask) == val) { + break; + } + cpu_relax(); + } while ((get_cycles() - start) < cycles); } - while ((get_cycles() - start) < cycles) - cpu_relax(); + return cur; +} + +void __delay(unsigned long cycles) +{ + /* constant word for wfet()/wfe() to poll */ + unsigned long dummy ____cacheline_aligned = 0; + __delay_until_ul(cycles, &dummy, 0, 1); } EXPORT_SYMBOL(__delay); -inline void __const_udelay(unsigned long xloops) +void __const_udelay(unsigned long xloops) { __delay(xloops_to_cycles(xloops)); } @@ -58,12 +88,12 @@ EXPORT_SYMBOL(__const_udelay); void __udelay(unsigned long usecs) { - __const_udelay(usecs * 0x10C7UL); /* 2**32 / 1000000 (rounded up) */ + __delay(USECS_TO_CYCLES(usecs)); } EXPORT_SYMBOL(__udelay); void __ndelay(unsigned long nsecs) { - __const_udelay(nsecs * 0x5UL); /* 2**32 / 1000000000 (rounded up) */ + __delay(NSECS_TO_CYCLES(nsecs)); } EXPORT_SYMBOL(__ndelay); From patchwork Tue Nov 5 18:30:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Okanovic, Haris" X-Patchwork-Id: 13863437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 01F72D33A20 for ; Tue, 5 Nov 2024 18:40:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=XT81pCaCySeLSnjYgvTXLKqn2CWQcbY+GGEiD2145i8=; b=WYpYSmRErQj3WmSjnU+EzZxABy oBS+RfplcVrGUn0iVgZPEGaakER7cVU+5RztmXJniX8B1MLfr8bQeeFGFrWU67gz+XeRm4zdV4z7m PWy3i/WD81KdDVu53wCdsNGLeKqzSB/PBeyEZgLemenNU33YlYS8fwUCYVFHe7JiId4fLtV83yA2Z gDGW/fSWcarTxiaKhBnzqT5/kgF6eqn2UlA94ILFLdvAxTTRQhbhbq9NjaB7QJV1wUETbzupzAzFz 1a6ALl9etWDasQOhBmZN/9KKvIc+q9oxJejUflwU2+YEAL90msyXQR1EktS44yiLYodq0hD+KPyCC jTNWhdJQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t8OTM-00000000PFI-2EhZ; Tue, 05 Nov 2024 18:40:12 +0000 Received: from smtp-fw-52003.amazon.com ([52.119.213.152]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t8OLp-00000000NcS-2pqo for linux-arm-kernel@lists.infradead.org; Tue, 05 Nov 2024 18:32:27 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1730831546; x=1762367546; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XT81pCaCySeLSnjYgvTXLKqn2CWQcbY+GGEiD2145i8=; b=qm8lqGbr00hwgViKL07yykqvVvS7Is6gVgyheDXiVaQ+t/8Pu397Dfq8 UxuZ3kOQDrQGthtJEFFdPh/GTkTBmZQFzUHyrWbJnFDqNsFuoHagn9RmX 7gNrl63jpZJjeQzM0rv0AsRHHqX+RN/a84mzobuRhGz+/s/FuHaU+Pg1d M=; X-IronPort-AV: E=Sophos;i="6.11,260,1725321600"; d="scan'208";a="39203526" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-52003.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Nov 2024 18:32:22 +0000 Received: from EX19MTAUWC001.ant.amazon.com [10.0.7.35:25778] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.63.130:2525] with esmtp (Farcaster) id fb722a64-c7d9-4601-a9fb-a5752e9890eb; Tue, 5 Nov 2024 18:32:20 +0000 (UTC) X-Farcaster-Flow-ID: fb722a64-c7d9-4601-a9fb-a5752e9890eb Received: from EX19D001UWA003.ant.amazon.com (10.13.138.211) by EX19MTAUWC001.ant.amazon.com (10.250.64.174) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 5 Nov 2024 18:32:20 +0000 Received: from u34cccd802f2d52.amazon.com (10.106.239.17) by EX19D001UWA003.ant.amazon.com (10.13.138.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Tue, 5 Nov 2024 18:32:15 +0000 From: Haris Okanovic To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 4/5] arm64: add smp_vcond_load_relaxed() Date: Tue, 5 Nov 2024 12:30:40 -0600 Message-ID: <20241105183041.1531976-5-harisokn@amazon.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241105183041.1531976-1-harisokn@amazon.com> References: <20240925232425.2763385-1-ankur.a.arora@oracle.com> <20241105183041.1531976-1-harisokn@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.106.239.17] X-ClientProxiedBy: EX19D033UWA004.ant.amazon.com (10.13.139.85) To EX19D001UWA003.ant.amazon.com (10.13.138.211) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241105_103225_904291_E56389D2 X-CRM114-Status: GOOD ( 10.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement smp_vcond_load_relaxed() atop __delay_until_ul() on arm64, to reduce number of busy loops while waiting for a value condition. This implementation only support unsigned long words. It can be extended via the enclosed case structure in barrier.h as needed. Signed-off-by: Haris Okanovic --- arch/arm64/include/asm/barrier.h | 18 ++++++++++++++++++ arch/arm64/lib/delay.c | 16 ++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h index 1ca947d5c939..188327e3ce72 100644 --- a/arch/arm64/include/asm/barrier.h +++ b/arch/arm64/include/asm/barrier.h @@ -203,6 +203,24 @@ do { \ (typeof(*ptr))VAL; \ }) +extern unsigned long __smp_vcond_load_relaxed_ul( + u64 nsecs, unsigned long* addr, unsigned long mask, unsigned long val); + +#define smp_vcond_load_relaxed(nsecs, addr, mask, val) ({ \ + u64 __nsecs = (nsecs); \ + typeof(addr) __addr = (addr); \ + typeof(*__addr) __mask = (mask); \ + typeof(*__addr) __val = (val); \ + typeof(*__addr) __cur; \ + switch (sizeof(*__addr)) { \ + case sizeof(unsigned long): \ + __cur = __smp_vcond_load_relaxed_ul( \ + __nsecs, __addr, __mask, __val); \ + break; \ + } \ + (__cur); \ +}) + #define smp_cond_load_acquire(ptr, cond_expr) \ ({ \ typeof(ptr) __PTR = (ptr); \ diff --git a/arch/arm64/lib/delay.c b/arch/arm64/lib/delay.c index a7c3040af316..a61a13b04439 100644 --- a/arch/arm64/lib/delay.c +++ b/arch/arm64/lib/delay.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -97,3 +98,18 @@ void __ndelay(unsigned long nsecs) __delay(NSECS_TO_CYCLES(nsecs)); } EXPORT_SYMBOL(__ndelay); + +unsigned long __smp_vcond_load_relaxed_ul( + u64 nsecs, unsigned long* addr, unsigned long mask, unsigned long val) +{ + const u64 start = local_clock_noinstr(); + const u64 cycles = NSECS_TO_CYCLES(nsecs); + unsigned long cur; + + do { + cur = __delay_until_ul(cycles, addr, mask, val); + } while((cur & mask) != val && local_clock_noinstr() - start < nsecs); + + return cur; +} +EXPORT_SYMBOL(__smp_vcond_load_relaxed_ul); From patchwork Tue Nov 5 18:30:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Okanovic, Haris" X-Patchwork-Id: 13863438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3ED3D33A23 for ; Tue, 5 Nov 2024 18:42:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type: Content-Transfer-Encoding:MIME-Version:References:In-Reply-To:Message-ID:Date :Subject:CC:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TrNQ3LJeAMesAmpKUJ7N1Al9kS1bwIA8hsz/KHf4MgU=; b=cBaUdPA7SuvRWE2V8iT/vwPYGj 9KPDnf4Bhb5A33LuDdBEe61aDXeVlq1pA6mf+7pDFzmd020fZjFIVc2CGFvQgVCI82cLu6WJIZJQl 0xvOs2u9WTWNn9c4jvpjQS6aSpb/oTAoZgTmcebgzD7m8UV6ugBZ1WG/kmJAabUjZSmz5iSgDtq5K h2h+Gzm6dWLFZslHSOVVpRvh5gyH9lstvNEoW0knmTNt/o7u3IYCku1u7pSbwWEImqa+DUFDLUQD/ kgf23GCf9dNkDMn7bBdswwsZuyR6JLLE+hLUlZ30VoiXoAZB35HAL+ck3ks8AYCh7OS6RHXfsjfkQ +ty5w3Mw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t8OV7-00000000Plb-08JZ; Tue, 05 Nov 2024 18:42:01 +0000 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t8OM7-00000000Ngw-1ONS for linux-arm-kernel@lists.infradead.org; Tue, 05 Nov 2024 18:32:45 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1730831563; x=1762367563; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TrNQ3LJeAMesAmpKUJ7N1Al9kS1bwIA8hsz/KHf4MgU=; b=ddfYjFa7MUuOxQ8QeYde8QDpUhi4qbsRHpdPsao8CgU5mYvSvyDMm0Z5 yG58tJ8aABdkg+Ez6Jb6TQdK97Ujygc+JumhmqtN//DUedfULm0vIiP6T bYIf86WQ2GeRTdBAiCZKpJ5RsgwA4IdR5xE26U5NqDEyvCOsKyQIksc7P E=; X-IronPort-AV: E=Sophos;i="6.11,260,1725321600"; d="scan'208";a="440503091" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.43.8.6]) by smtp-border-fw-2101.iad2.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Nov 2024 18:32:39 +0000 Received: from EX19MTAUWA002.ant.amazon.com [10.0.7.35:4506] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.17.170:2525] with esmtp (Farcaster) id 3e5bd2bb-9e63-4844-b995-362add075696; Tue, 5 Nov 2024 18:32:39 +0000 (UTC) X-Farcaster-Flow-ID: 3e5bd2bb-9e63-4844-b995-362add075696 Received: from EX19D001UWA003.ant.amazon.com (10.13.138.211) by EX19MTAUWA002.ant.amazon.com (10.250.64.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.34; Tue, 5 Nov 2024 18:32:37 +0000 Received: from u34cccd802f2d52.amazon.com (10.106.239.17) by EX19D001UWA003.ant.amazon.com (10.13.138.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1258.35; Tue, 5 Nov 2024 18:32:32 +0000 From: Haris Okanovic To: , CC: , , , , , , , , , , , , , , , , , , , , , , , , , , , , Subject: [PATCH 5/5] cpuidle: implement poll_idle() using smp_vcond_load_relaxed() Date: Tue, 5 Nov 2024 12:30:41 -0600 Message-ID: <20241105183041.1531976-6-harisokn@amazon.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241105183041.1531976-1-harisokn@amazon.com> References: <20240925232425.2763385-1-ankur.a.arora@oracle.com> <20241105183041.1531976-1-harisokn@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.106.239.17] X-ClientProxiedBy: EX19D033UWA004.ant.amazon.com (10.13.139.85) To EX19D001UWA003.ant.amazon.com (10.13.138.211) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241105_103243_561890_08A29204 X-CRM114-Status: GOOD ( 10.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement poll_idle() using smp_vcond_load_relaxed() function. Signed-off-by: Haris Okanovic --- drivers/cpuidle/poll_state.c | 36 +++++------------------------------- 1 file changed, 5 insertions(+), 31 deletions(-) diff --git a/drivers/cpuidle/poll_state.c b/drivers/cpuidle/poll_state.c index 61df2395585e..5553e6f31702 100644 --- a/drivers/cpuidle/poll_state.c +++ b/drivers/cpuidle/poll_state.c @@ -7,46 +7,20 @@ #include #include #include - -#ifdef CONFIG_ARM64 -/* - * POLL_IDLE_RELAX_COUNT determines how often we check for timeout - * while polling for TIF_NEED_RESCHED in thread_info->flags. - * - * Set this to a low value since arm64, instead of polling, uses a - * event based mechanism. - */ -#define POLL_IDLE_RELAX_COUNT 1 -#else -#define POLL_IDLE_RELAX_COUNT 200 -#endif +#include static int __cpuidle poll_idle(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { - u64 time_start; - - time_start = local_clock_noinstr(); + unsigned long flags; dev->poll_time_limit = false; raw_local_irq_enable(); if (!current_set_polling_and_test()) { - u64 limit; - - limit = cpuidle_poll_time(drv, dev); - - while (!need_resched()) { - unsigned int loop_count = 0; - if (local_clock_noinstr() - time_start > limit) { - dev->poll_time_limit = true; - break; - } - - smp_cond_load_relaxed(¤t_thread_info()->flags, - VAL & _TIF_NEED_RESCHED || - loop_count++ >= POLL_IDLE_RELAX_COUNT); - } + u64 limit = cpuidle_poll_time(drv, dev); + flags = smp_vcond_load_relaxed(limit, ¤t_thread_info()->flags, _TIF_NEED_RESCHED, _TIF_NEED_RESCHED); + dev->poll_time_limit = !(flags & _TIF_NEED_RESCHED); } raw_local_irq_disable();