From patchwork Sat Sep 23 00:41:50 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christoffer Dall X-Patchwork-Id: 9967337 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C3FF16035E for ; Sat, 23 Sep 2017 00:46:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B602D299D2 for ; Sat, 23 Sep 2017 00:46:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA59E299CF; Sat, 23 Sep 2017 00:46:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 337EC299CD for ; Sat, 23 Sep 2017 00:46:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=h44PU7+waxLysG2EPkO/4VtFZOUCBB1f7aUmfepGo/0=; b=TmmAj6kLwEFimD2AXDzILQVudL XI6tsMrzNguWNKlbSTAaTJr3Kkkbs6Vobj5FPxMns6iOGgdJzZfRMHCSRtIAiu0A0ZiQzL4wA7fmv f1VfgIg0MgYQuxcdwRbaTjWCi0htoN3KLE2HYVQXnYGnqcqj1jZt40IjNvXvqY0FHdNgrA+QASIVK +wJ9kLh1pywUg5OEsk7PscrX85+Ao0qKOeR+5QQK3RTToF5+QgMx22nPHfpqjMwxptlXoc5cRGV9q raBf84z+wd+T9rClxAac7rXDsnES7E84v+Rk9Y3NH7kT3kmlPhzQdJOMr/ZRGKtFvGlGEQe1FnjiC cQuDaVOQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dvYZI-00065d-1r; Sat, 23 Sep 2017 00:45:16 +0000 Received: from mail-wm0-x235.google.com ([2a00:1450:400c:c09::235]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dvYWk-0003kQ-4P for linux-arm-kernel@lists.infradead.org; Sat, 23 Sep 2017 00:42:49 +0000 Received: by mail-wm0-x235.google.com with SMTP id q124so7622851wmb.0 for ; Fri, 22 Sep 2017 17:42:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=UrUaGbTwx9w7NzOS3mEd7mizyCePiIijpMntJTgJXZg=; b=jLEOyULWHcVYs8wpXMvWY0sTpchYSTF7bFbcPjLC7tSMy8tpm53DcZ251DLbph9JNx PPNAQrmioVyMObp+eRtPT2hJGRz2XV7njv+ZnAIHPjh8yG4FDt3OTOlphoezIn6C4x+H dKgepc7Wxg/fFE/PiwQ9iKAWAKcKLoPCalp9o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=UrUaGbTwx9w7NzOS3mEd7mizyCePiIijpMntJTgJXZg=; b=ag3tkT+3A5P966pDCyiz8hmnvtLF+pFiYMLq3X+svJRW2O8pgMugg85TnFQSaZRMdQ AKzb/hx6faNJ4vppQ3XjXLXxjIqdC7ftX8PEwioLohYJSHdnJyhtMsn9bgvKmyilt8Lk 2fE6BR3CYI2qh1YSlbGkHk/MQHYXYCBgK5U9xELDnJMIVVSUeNDJII82+jZ4HOMfD1r8 TbrajWCrr/1UNoHbAuw9EWBZuIhKE9seVE8KA6EFWN9RC5W+IiAPA3fUGvdLWP0uGKmo g04HSakG7ryQMoFo/qUjE/noCegC8mOH2cjwLjd0UUMCkSw7wfUfxacArWzFdFevCvcD w3lQ== X-Gm-Message-State: AHPjjUhCTzDYCwNoEdAIyMcXGov/frHgNXD6loDPbpcTHhm8qYt8zLxE AhdxWxwgLjTQcZ0ngk99eyxPHg== X-Google-Smtp-Source: AOwi7QADJKUID+Ki7zNIwkJNYDub/5tfo0VxlJLCI3BgWPQz54hbNdAPq+e6T4lbRHeBRtOnUyDTNg== X-Received: by 10.80.135.65 with SMTP id 1mr6287019edv.266.1506127327802; Fri, 22 Sep 2017 17:42:07 -0700 (PDT) Received: from localhost.localdomain (xd93dd96b.cust.hiper.dk. [217.61.217.107]) by smtp.gmail.com with ESMTPSA id c25sm646085edb.57.2017.09.22.17.42.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 22 Sep 2017 17:42:06 -0700 (PDT) From: Christoffer Dall To: kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: [PATCH v3 03/20] arm64: Use the physical counter when available for read_cycles Date: Sat, 23 Sep 2017 02:41:50 +0200 Message-Id: <20170923004207.22356-4-cdall@linaro.org> X-Mailer: git-send-email 2.9.0 In-Reply-To: <20170923004207.22356-1-cdall@linaro.org> References: <20170923004207.22356-1-cdall@linaro.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170922_174238_926741_15D19BC7 X-CRM114-Status: GOOD ( 19.26 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Christoffer Dall , kvm@vger.kernel.org, Marc Zyngier , Catalin Marinas , Will Deacon MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently get_cycles() is hardwired to arch_counter_get_cntvct() on arm64, but as we move to using the physical timer for the in-kernel time-keeping, we need to make that more flexible. First, we need to make sure the physical counter can be read on equal terms to the virtual counter, which includes adding physical counter read functions for timers that require errata. Second, we need to make a choice between reading the physical vs virtual counter, depending on which timer is used for time keeping in the kernel otherwise. We can do this using a static key to avoid a performance penalty during runtime when reading the counter. Cc: Catalin Marinas Cc: Will Deacon Cc: Mark Rutland Cc: Marc Zyngier Signed-off-by: Christoffer Dall --- arch/arm64/include/asm/arch_timer.h | 15 ++++++++++++--- arch/arm64/include/asm/timex.h | 2 +- drivers/clocksource/arm_arch_timer.c | 32 ++++++++++++++++++++++++++++++-- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h index 1859a1c..c56d8cd 100644 --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -30,6 +30,8 @@ #include +extern struct static_key_false arch_timer_phys_counter_available; + #if IS_ENABLED(CONFIG_ARM_ARCH_TIMER_OOL_WORKAROUND) extern struct static_key_false arch_timer_read_ool_enabled; #define needs_unstable_timer_counter_workaround() \ @@ -52,6 +54,7 @@ struct arch_timer_erratum_workaround { const char *desc; u32 (*read_cntp_tval_el0)(void); u32 (*read_cntv_tval_el0)(void); + u64 (*read_cntpct_el0)(void); u64 (*read_cntvct_el0)(void); int (*set_next_event_phys)(unsigned long, struct clock_event_device *); int (*set_next_event_virt)(unsigned long, struct clock_event_device *); @@ -148,10 +151,8 @@ static inline void arch_timer_set_cntkctl(u32 cntkctl) static inline u64 arch_counter_get_cntpct(void) { - u64 cval; isb(); - asm volatile("mrs %0, cntpct_el0" : "=r" (cval)); - return cval; + return arch_timer_reg_read_stable(cntpct_el0); } static inline u64 arch_counter_get_cntvct(void) @@ -160,6 +161,14 @@ static inline u64 arch_counter_get_cntvct(void) return arch_timer_reg_read_stable(cntvct_el0); } +static inline u64 arch_counter_get_cycles(void) +{ + if (static_branch_unlikely(&arch_timer_phys_counter_available)) + return arch_counter_get_cntpct(); + else + return arch_counter_get_cntvct(); +} + static inline int arch_timer_arch_init(void) { return 0; diff --git a/arch/arm64/include/asm/timex.h b/arch/arm64/include/asm/timex.h index 81a076e..c0d214c 100644 --- a/arch/arm64/include/asm/timex.h +++ b/arch/arm64/include/asm/timex.h @@ -22,7 +22,7 @@ * Use the current timer as a cycle counter since this is what we use for * the delay loop. */ -#define get_cycles() arch_counter_get_cntvct() +#define get_cycles() arch_counter_get_cycles() #include diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c index 9b3322a..f35da20 100644 --- a/drivers/clocksource/arm_arch_timer.c +++ b/drivers/clocksource/arm_arch_timer.c @@ -77,6 +77,9 @@ static bool arch_timer_mem_use_virtual; static bool arch_counter_suspend_stop; static bool vdso_default = true; +DEFINE_STATIC_KEY_FALSE(arch_timer_phys_counter_available); +EXPORT_SYMBOL_GPL(arch_timer_phys_counter_available); + static bool evtstrm_enable = IS_ENABLED(CONFIG_ARM_ARCH_TIMER_EVTSTREAM); static int __init early_evtstrm_cfg(char *buf) @@ -217,6 +220,11 @@ static u32 notrace fsl_a008585_read_cntv_tval_el0(void) return __fsl_a008585_read_reg(cntv_tval_el0); } +static u64 notrace fsl_a008585_read_cntpct_el0(void) +{ + return __fsl_a008585_read_reg(cntpct_el0); +} + static u64 notrace fsl_a008585_read_cntvct_el0(void) { return __fsl_a008585_read_reg(cntvct_el0); @@ -258,6 +266,11 @@ static u32 notrace hisi_161010101_read_cntv_tval_el0(void) return __hisi_161010101_read_reg(cntv_tval_el0); } +static u64 notrace hisi_161010101_read_cntpct_el0(void) +{ + return __hisi_161010101_read_reg(cntpct_el0); +} + static u64 notrace hisi_161010101_read_cntvct_el0(void) { return __hisi_161010101_read_reg(cntvct_el0); @@ -288,6 +301,15 @@ static struct ate_acpi_oem_info hisi_161010101_oem_info[] = { #endif #ifdef CONFIG_ARM64_ERRATUM_858921 +static u64 notrace arm64_858921_read_cntpct_el0(void) +{ + u64 old, new; + + old = read_sysreg(cntpct_el0); + new = read_sysreg(cntpct_el0); + return (((old ^ new) >> 32) & 1) ? old : new; +} + static u64 notrace arm64_858921_read_cntvct_el0(void) { u64 old, new; @@ -346,6 +368,7 @@ static const struct arch_timer_erratum_workaround ool_workarounds[] = { .desc = "Freescale erratum a005858", .read_cntp_tval_el0 = fsl_a008585_read_cntp_tval_el0, .read_cntv_tval_el0 = fsl_a008585_read_cntv_tval_el0, + .read_cntpct_el0 = fsl_a008585_read_cntpct_el0, .read_cntvct_el0 = fsl_a008585_read_cntvct_el0, .set_next_event_phys = erratum_set_next_event_tval_phys, .set_next_event_virt = erratum_set_next_event_tval_virt, @@ -358,6 +381,7 @@ static const struct arch_timer_erratum_workaround ool_workarounds[] = { .desc = "HiSilicon erratum 161010101", .read_cntp_tval_el0 = hisi_161010101_read_cntp_tval_el0, .read_cntv_tval_el0 = hisi_161010101_read_cntv_tval_el0, + .read_cntpct_el0 = hisi_161010101_read_cntpct_el0, .read_cntvct_el0 = hisi_161010101_read_cntvct_el0, .set_next_event_phys = erratum_set_next_event_tval_phys, .set_next_event_virt = erratum_set_next_event_tval_virt, @@ -368,6 +392,7 @@ static const struct arch_timer_erratum_workaround ool_workarounds[] = { .desc = "HiSilicon erratum 161010101", .read_cntp_tval_el0 = hisi_161010101_read_cntp_tval_el0, .read_cntv_tval_el0 = hisi_161010101_read_cntv_tval_el0, + .read_cntpct_el0 = hisi_161010101_read_cntpct_el0, .read_cntvct_el0 = hisi_161010101_read_cntvct_el0, .set_next_event_phys = erratum_set_next_event_tval_phys, .set_next_event_virt = erratum_set_next_event_tval_virt, @@ -378,6 +403,7 @@ static const struct arch_timer_erratum_workaround ool_workarounds[] = { .match_type = ate_match_local_cap_id, .id = (void *)ARM64_WORKAROUND_858921, .desc = "ARM erratum 858921", + .read_cntpct_el0 = arm64_858921_read_cntpct_el0, .read_cntvct_el0 = arm64_858921_read_cntvct_el0, }, #endif @@ -890,10 +916,12 @@ static void __init arch_counter_register(unsigned type) /* Register the CP15 based counter if we have one */ if (type & ARCH_TIMER_TYPE_CP15) { - if (arch_timer_uses_ppi == ARCH_TIMER_VIRT_PPI) + if (arch_timer_uses_ppi == ARCH_TIMER_VIRT_PPI) { arch_timer_read_counter = arch_counter_get_cntvct; - else + } else { arch_timer_read_counter = arch_counter_get_cntpct; + static_branch_enable(&arch_timer_phys_counter_available); + } clocksource_counter.archdata.vdso_direct = vdso_default; } else {