From patchwork Mon Dec 23 14:31:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308407 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B8A0B138D for ; Mon, 23 Dec 2019 14:31:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8C7FF2075B for ; Mon, 23 Dec 2019 14:31:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="C9eeZNaE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726933AbfLWObF (ORCPT ); Mon, 23 Dec 2019 09:31:05 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:6548 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726884AbfLWObE (ORCPT ); Mon, 23 Dec 2019 09:31:04 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGT3tYfz9txhf; Mon, 23 Dec 2019 15:30:57 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=C9eeZNaE; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id hIex8qJqEKAO; Mon, 23 Dec 2019 15:30:57 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGT2gPfz9txjv; Mon, 23 Dec 2019 15:30:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111457; bh=pvdta7Gn2k+DdQszh3qC/bVMHcqn/+lDf822F1fVxgo=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=C9eeZNaEwLksCAOPtOK2YA91yeobYcANmXFdPWJd7jgXKvdlEJgzrOAuz098CPWj/ g5ke7y9h+SpSrlT7P7BkHLDP/vtYx20JVclWX5F0r/PuQePAftK/FAkQoEKDytP8zJ NWaC994zfdlrVAWsO1lrJBHI8e8kmzW5QmJ9MQEI= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 791168B7AB; Mon, 23 Dec 2019 15:31:02 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 2XlQ65p-nunC; Mon, 23 Dec 2019 15:31:02 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 3A9B08B7A1; Mon, 23 Dec 2019 15:31:02 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 40C4A637D4; Mon, 23 Dec 2019 14:31:02 +0000 (UTC) Message-Id: <47701b5fb73cf536db074031db8e6e3fa3695168.1577111365.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 01/10] lib: vdso: ensure all arches have 32bit fallback To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:02 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org In order to simplify next step which moves fallback call at arch level, ensure all arches have a 32bit fallback instead of handling the lack of 32bit fallback in the common code based on VDSO_HAS_32BIT_FALLBACK Signed-off-by: Christophe Leroy --- arch/arm/include/asm/vdso/gettimeofday.h | 26 +++++++++++++++++++++ arch/arm64/include/asm/vdso/compat_gettimeofday.h | 2 -- arch/arm64/include/asm/vdso/gettimeofday.h | 26 +++++++++++++++++++++ arch/mips/include/asm/vdso/gettimeofday.h | 28 +++++++++++++++++++++-- arch/x86/include/asm/vdso/gettimeofday.h | 28 +++++++++++++++++++++-- lib/vdso/gettimeofday.c | 10 -------- 6 files changed, 104 insertions(+), 16 deletions(-) diff --git a/arch/arm/include/asm/vdso/gettimeofday.h b/arch/arm/include/asm/vdso/gettimeofday.h index 0ad2429c324f..55f8ad6e7777 100644 --- a/arch/arm/include/asm/vdso/gettimeofday.h +++ b/arch/arm/include/asm/vdso/gettimeofday.h @@ -70,6 +70,32 @@ static __always_inline int clock_getres_fallback( return ret; } +static __always_inline +long clock_gettime32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_gettime_fallback(clock, &ts); + + if (likely(!ret)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + +static __always_inline +long clock_getres32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_getres_fallback(clock, &ts); + + if (likely(!ret && _ts)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + static __always_inline u64 __arch_get_hw_counter(int clock_mode) { #ifdef CONFIG_ARM_ARCH_TIMER diff --git a/arch/arm64/include/asm/vdso/compat_gettimeofday.h b/arch/arm64/include/asm/vdso/compat_gettimeofday.h index c50ee1b7d5cd..bab700e37a03 100644 --- a/arch/arm64/include/asm/vdso/compat_gettimeofday.h +++ b/arch/arm64/include/asm/vdso/compat_gettimeofday.h @@ -16,8 +16,6 @@ #define VDSO_HAS_CLOCK_GETRES 1 -#define VDSO_HAS_32BIT_FALLBACK 1 - static __always_inline int gettimeofday_fallback(struct __kernel_old_timeval *_tv, struct timezone *_tz) diff --git a/arch/arm64/include/asm/vdso/gettimeofday.h b/arch/arm64/include/asm/vdso/gettimeofday.h index b08f476b72b4..c41c86a07423 100644 --- a/arch/arm64/include/asm/vdso/gettimeofday.h +++ b/arch/arm64/include/asm/vdso/gettimeofday.h @@ -66,6 +66,32 @@ int clock_getres_fallback(clockid_t _clkid, struct __kernel_timespec *_ts) return ret; } +static __always_inline +long clock_gettime32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_gettime_fallback(clock, &ts); + + if (likely(!ret)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + +static __always_inline +long clock_getres32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_getres_fallback(clock, &ts); + + if (likely(!ret && _ts)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) { u64 res; diff --git a/arch/mips/include/asm/vdso/gettimeofday.h b/arch/mips/include/asm/vdso/gettimeofday.h index b08825531e9f..60608e930a5c 100644 --- a/arch/mips/include/asm/vdso/gettimeofday.h +++ b/arch/mips/include/asm/vdso/gettimeofday.h @@ -109,8 +109,6 @@ static __always_inline int clock_getres_fallback( #if _MIPS_SIM != _MIPS_SIM_ABI64 -#define VDSO_HAS_32BIT_FALLBACK 1 - static __always_inline long clock_gettime32_fallback( clockid_t _clkid, struct old_timespec32 *_ts) @@ -150,6 +148,32 @@ static __always_inline int clock_getres32_fallback( return error ? -ret : ret; } +#else +static __always_inline +long clock_gettime32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_gettime_fallback(clock, &ts); + + if (likely(!ret)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + +static __always_inline +long clock_getres32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_getres_fallback(clock, &ts); + + if (likely(!ret && _ts)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} #endif #ifdef CONFIG_CSRC_R4K diff --git a/arch/x86/include/asm/vdso/gettimeofday.h b/arch/x86/include/asm/vdso/gettimeofday.h index e9ee139cf29e..e1e16c2fdba0 100644 --- a/arch/x86/include/asm/vdso/gettimeofday.h +++ b/arch/x86/include/asm/vdso/gettimeofday.h @@ -94,9 +94,33 @@ long clock_getres_fallback(clockid_t _clkid, struct __kernel_timespec *_ts) return ret; } -#else +static __always_inline +long clock_gettime32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_gettime_fallback(clock, &ts); -#define VDSO_HAS_32BIT_FALLBACK 1 + if (likely(!ret)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + +static __always_inline +long clock_getres32_fallback(clockid_t _clkid, struct old_timespec32 *_ts) +{ + struct __kernel_timespec ts; + int ret = clock_getres_fallback(clock, &ts); + + if (likely(!ret && _ts)) { + _ts->tv_sec = ts.tv_sec; + _ts->tv_nsec = ts.tv_nsec; + } + return ret; +} + +#else static __always_inline long clock_gettime_fallback(clockid_t _clkid, struct __kernel_timespec *_ts) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 9ecfd3b547ba..59189ed49352 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -125,13 +125,8 @@ __cvdso_clock_gettime32(clockid_t clock, struct old_timespec32 *res) ret = __cvdso_clock_gettime_common(clock, &ts); -#ifdef VDSO_HAS_32BIT_FALLBACK if (unlikely(ret)) return clock_gettime32_fallback(clock, res); -#else - if (unlikely(ret)) - ret = clock_gettime_fallback(clock, &ts); -#endif if (likely(!ret)) { res->tv_sec = ts.tv_sec; @@ -238,13 +233,8 @@ __cvdso_clock_getres_time32(clockid_t clock, struct old_timespec32 *res) ret = __cvdso_clock_getres_common(clock, &ts); -#ifdef VDSO_HAS_32BIT_FALLBACK if (unlikely(ret)) return clock_getres32_fallback(clock, res); -#else - if (unlikely(ret)) - ret = clock_getres_fallback(clock, &ts); -#endif if (likely(!ret && res)) { res->tv_sec = ts.tv_sec; From patchwork Mon Dec 23 14:31:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308403 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2502921 for ; Mon, 23 Dec 2019 14:31:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7D93F20CC7 for ; Mon, 23 Dec 2019 14:31:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="vogtJFcg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727392AbfLWObs (ORCPT ); Mon, 23 Dec 2019 09:31:48 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:26814 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726840AbfLWObG (ORCPT ); Mon, 23 Dec 2019 09:31:06 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGV4xxlz9txhh; Mon, 23 Dec 2019 15:30:58 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=vogtJFcg; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id RmEcF54nG6aG; Mon, 23 Dec 2019 15:30:58 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGV2XJ3z9txhg; Mon, 23 Dec 2019 15:30:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111458; bh=B8Yu9pIP98J6v2ZJy2Ayu+97WIEYkB26j/iWHxqbNSc=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=vogtJFcgdzO5SlcK8lUrocAWAD0fW79yjfeeJL3JhePqFA65XxnHiKjuN7b1f/sRP B2R+msNiaHffrQRkMd42gAPAQ750iuE3zzcD/BmKrHFX82vV+jtH6f9p3+RZsBG7uy tNI4wng0WUaJ8a1btnKEAq4NbmvbFRlPCxZnVUUA= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 745088B7AB; Mon, 23 Dec 2019 15:31:03 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id iO5cqfr7b5RU; Mon, 23 Dec 2019 15:31:03 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 3DAFA8B7A1; Mon, 23 Dec 2019 15:31:03 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 484EC637D4; Mon, 23 Dec 2019 14:31:03 +0000 (UTC) Message-Id: In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 02/10] lib: vdso: move call to fallback out of common code. To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:03 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org On powerpc, VDSO functions and syscalls cannot be implemented in C because the Linux kernel ABI requires that CR[SO] bit is set in case of error and cleared when no error. As this cannot be done in C, C VDSO functions and syscall'based fallback need a trampoline in ASM. By moving the fallback calls out of the common code, arches like powerpc can implement both the call to C VDSO and the fallback call in a single trampoline function. The two advantages are: - No need play back and forth with CR[SO] and negative return value. - No stack frame is required in VDSO C functions for the fallbacks. The performance improvement is significant on powerpc. Signed-off-by: Christophe Leroy --- arch/arm/vdso/vgettimeofday.c | 28 +++++++++++++++--- arch/arm64/kernel/vdso/vgettimeofday.c | 21 ++++++++++++-- arch/arm64/kernel/vdso32/vgettimeofday.c | 35 ++++++++++++++++++++--- arch/mips/vdso/vgettimeofday.c | 49 +++++++++++++++++++++++++++----- arch/x86/entry/vdso/vclock_gettime.c | 42 +++++++++++++++++++++++---- lib/vdso/gettimeofday.c | 31 ++++---------------- 6 files changed, 156 insertions(+), 50 deletions(-) diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 1976c6f325a4..5451afb715e6 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -10,25 +10,45 @@ int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - return __cvdso_clock_gettime32(clock, ts); + int ret = __cvdso_clock_gettime32(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime32_fallback(clock, &ts); } int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - return __cvdso_clock_gettime(clock, ts); + int ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - return __cvdso_gettimeofday(tv, tz); + int ret = __cvdso_gettimeofday(tv, tz); + + if (likely(!ret)) + return ret; + + return gettimeofday_fallback(tv, tz); } int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { - return __cvdso_clock_getres_time32(clock_id, res); + int ret = __cvdso_clock_getres_time32(clock_id, res); + + if (likely(!ret)) + return ret; + + return clock_getres32_fallback(clock, res); } /* Avoid unresolved references emitted by GCC */ diff --git a/arch/arm64/kernel/vdso/vgettimeofday.c b/arch/arm64/kernel/vdso/vgettimeofday.c index 747635501a14..62694876b216 100644 --- a/arch/arm64/kernel/vdso/vgettimeofday.c +++ b/arch/arm64/kernel/vdso/vgettimeofday.c @@ -11,17 +11,32 @@ int __kernel_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - return __cvdso_clock_gettime(clock, ts); + int ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } int __kernel_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - return __cvdso_gettimeofday(tv, tz); + int ret = __cvdso_gettimeofday(tv, tz); + + if (likely(!ret)) + return ret; + + return gettimeofday_fallback(tv, tz); } int __kernel_clock_getres(clockid_t clock_id, struct __kernel_timespec *res) { - return __cvdso_clock_getres(clock_id, res); + int ret = __cvdso_clock_getres(clock_id, res); + + if (likely(!ret)) + return ret; + + return clock_getres_fallback(clock, res); } diff --git a/arch/arm64/kernel/vdso32/vgettimeofday.c b/arch/arm64/kernel/vdso32/vgettimeofday.c index 54fc1c2ce93f..6770d2bedd1f 100644 --- a/arch/arm64/kernel/vdso32/vgettimeofday.c +++ b/arch/arm64/kernel/vdso32/vgettimeofday.c @@ -11,37 +11,64 @@ int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { + int ret; + /* The checks below are required for ABI consistency with arm */ if ((u32)ts >= TASK_SIZE_32) return -EFAULT; - return __cvdso_clock_gettime32(clock, ts); + ret = __cvdso_clock_gettime32(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime32_fallback(clock, &ts); } int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { + int ret; + /* The checks below are required for ABI consistency with arm */ if ((u32)ts >= TASK_SIZE_32) return -EFAULT; - return __cvdso_clock_gettime(clock, ts); + ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - return __cvdso_gettimeofday(tv, tz); + int ret = __cvdso_gettimeofday(tv, tz); + + if (likely(!ret)) + return ret; + + return gettimeofday_fallback(tv, tz); } int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { + int ret; + struct __kernel_timespec ts; + /* The checks below are required for ABI consistency with arm */ if ((u32)res >= TASK_SIZE_32) return -EFAULT; - return __cvdso_clock_getres_time32(clock_id, res); + ret = __cvdso_clock_getres_time32(clock_id, res); + + if (likely(!ret)) + return ret; + + return clock_getres32_fallback(clock, res); } /* Avoid unresolved references emitted by GCC */ diff --git a/arch/mips/vdso/vgettimeofday.c b/arch/mips/vdso/vgettimeofday.c index 6ebdc37c89fc..02758c58454c 100644 --- a/arch/mips/vdso/vgettimeofday.c +++ b/arch/mips/vdso/vgettimeofday.c @@ -14,25 +14,45 @@ int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - return __cvdso_clock_gettime32(clock, ts); + int ret = __cvdso_clock_gettime32(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime32_fallback(clock, ts); } int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - return __cvdso_gettimeofday(tv, tz); + int ret = __cvdso_gettimeofday(tv, tz); + + if (likely(!ret)) + return ret; + + return gettimeofday_fallback(tv, tz); } int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { - return __cvdso_clock_getres_time32(clock_id, res); + int ret = __cvdso_clock_getres_time32(clock_id, res); + + if (likely(!ret)) + return ret; + + return clock_getres32_fallback(clock, res); } int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - return __cvdso_clock_gettime(clock, ts); + int ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } #else @@ -40,19 +60,34 @@ int __vdso_clock_gettime64(clockid_t clock, int __vdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - return __cvdso_clock_gettime(clock, ts); + int ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - return __cvdso_gettimeofday(tv, tz); + int ret = __cvdso_gettimeofday(tv, tz); + + if (likely(!ret)) + return ret; + + return gettimeofday_fallback(tv, tz); } int __vdso_clock_getres(clockid_t clock_id, struct __kernel_timespec *res) { - return __cvdso_clock_getres(clock_id, res); + int ret = __cvdso_clock_getres(clock_id, res); + + if (likely(!ret)) + return ret; + + return clock_getres_fallback(clock, res); } #endif diff --git a/arch/x86/entry/vdso/vclock_gettime.c b/arch/x86/entry/vdso/vclock_gettime.c index 7d70935b6758..fde511cfe284 100644 --- a/arch/x86/entry/vdso/vclock_gettime.c +++ b/arch/x86/entry/vdso/vclock_gettime.c @@ -19,7 +19,12 @@ extern __kernel_old_time_t __vdso_time(__kernel_old_time_t *t); int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - return __cvdso_gettimeofday(tv, tz); + int ret = __cvdso_gettimeofday(tv, tz); + + if (likely(!ret)) + return ret; + + return gettimeofday_fallback(tv, tz); } int gettimeofday(struct __kernel_old_timeval *, struct timezone *) @@ -40,7 +45,12 @@ extern int __vdso_clock_getres(clockid_t clock, struct __kernel_timespec *res); int __vdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - return __cvdso_clock_gettime(clock, ts); + int ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } int clock_gettime(clockid_t, struct __kernel_timespec *) @@ -49,7 +59,12 @@ int clock_gettime(clockid_t, struct __kernel_timespec *) int __vdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) { - return __cvdso_clock_getres(clock, res); + int ret = __cvdso_clock_getres(clock_id, res); + + if (likely(!ret)) + return ret; + + return clock_getres_fallback(clock, res); } int clock_getres(clockid_t, struct __kernel_timespec *) __attribute__((weak, alias("__vdso_clock_getres"))); @@ -61,7 +76,12 @@ extern int __vdso_clock_getres(clockid_t clock, struct old_timespec32 *res); int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - return __cvdso_clock_gettime32(clock, ts); + int ret = __cvdso_clock_gettime32(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime32_fallback(clock, ts); } int clock_gettime(clockid_t, struct old_timespec32 *) @@ -69,7 +89,12 @@ int clock_gettime(clockid_t, struct old_timespec32 *) int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - return __cvdso_clock_gettime(clock, ts); + int ret = __cvdso_clock_gettime(clock, ts); + + if (likely(!ret)) + return ret; + + return clock_gettime_fallback(clock, ts); } int clock_gettime64(clockid_t, struct __kernel_timespec *) @@ -77,7 +102,12 @@ int clock_gettime64(clockid_t, struct __kernel_timespec *) int __vdso_clock_getres(clockid_t clock, struct old_timespec32 *res) { - return __cvdso_clock_getres_time32(clock, res); + int ret = __cvdso_clock_getres_time32(clock, res); + + if (likely(!ret)) + return ret; + + return clock_getres32_fallback(clock, res); } int clock_getres(clockid_t, struct old_timespec32 *) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 59189ed49352..4618e274f1d5 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -16,9 +16,6 @@ * - __arch_get_vdso_data(): to get the vdso datapage. * - __arch_get_hw_counter(): to get the hw counter based on the * clock_mode. - * - gettimeofday_fallback(): fallback for gettimeofday. - * - clock_gettime_fallback(): fallback for clock_gettime. - * - clock_getres_fallback(): fallback for clock_getres. */ #ifdef ENABLE_COMPAT_VDSO #include @@ -110,23 +107,14 @@ __cvdso_clock_gettime_common(clockid_t clock, struct __kernel_timespec *ts) static __maybe_unused int __cvdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime_common(clock, ts); - - if (unlikely(ret)) - return clock_gettime_fallback(clock, ts); - return 0; + return __cvdso_clock_gettime_common(clock, ts); } static __maybe_unused int __cvdso_clock_gettime32(clockid_t clock, struct old_timespec32 *res) { struct __kernel_timespec ts; - int ret; - - ret = __cvdso_clock_gettime_common(clock, &ts); - - if (unlikely(ret)) - return clock_gettime32_fallback(clock, res); + int ret = __cvdso_clock_gettime_common(clock, &ts); if (likely(!ret)) { res->tv_sec = ts.tv_sec; @@ -144,7 +132,7 @@ __cvdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) struct __kernel_timespec ts; if (do_hres(&vd[CS_HRES_COARSE], CLOCK_REALTIME, &ts)) - return gettimeofday_fallback(tv, tz); + return -1; tv->tv_sec = ts.tv_sec; tv->tv_usec = (u32)ts.tv_nsec / NSEC_PER_USEC; @@ -218,23 +206,14 @@ int __cvdso_clock_getres_common(clockid_t clock, struct __kernel_timespec *res) int __cvdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) { - int ret = __cvdso_clock_getres_common(clock, res); - - if (unlikely(ret)) - return clock_getres_fallback(clock, res); - return 0; + return __cvdso_clock_getres_common(clock, res); } static __maybe_unused int __cvdso_clock_getres_time32(clockid_t clock, struct old_timespec32 *res) { struct __kernel_timespec ts; - int ret; - - ret = __cvdso_clock_getres_common(clock, &ts); - - if (unlikely(ret)) - return clock_getres32_fallback(clock, res); + int ret = __cvdso_clock_getres_common(clock, &ts); if (likely(!ret && res)) { res->tv_sec = ts.tv_sec; From patchwork Mon Dec 23 14:31:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308401 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5A8B138D for ; Mon, 23 Dec 2019 14:31:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B4F7821569 for ; Mon, 23 Dec 2019 14:31:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="Qj9FKBHM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726994AbfLWObH (ORCPT ); Mon, 23 Dec 2019 09:31:07 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:38665 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726943AbfLWObG (ORCPT ); Mon, 23 Dec 2019 09:31:06 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGW3VmLz9txhj; Mon, 23 Dec 2019 15:30:59 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=Qj9FKBHM; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id LIL9pRTQWaO2; Mon, 23 Dec 2019 15:30:59 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGW2Hnkz9txhg; Mon, 23 Dec 2019 15:30:59 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111459; bh=o7emI0agsMOH48CbNwosCV8op5LYe8FM1Ks4MgVPYns=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=Qj9FKBHMuY0RRfNbOXsqQ8E+KL1fR3ZgxUrpNf3HarY3lPxPrTAfqokClU23/DkIj U5B85uRpLkPs/m0bEq1zPcTwgymtrDiClZV+z532uMrnb7SHTgTKxeG1X83M5w47Aw IJvQYAXDAzMKevrTvvGquV1w/aN9xnyKpSIRUU7Q= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6BB178B7AB; Mon, 23 Dec 2019 15:31:04 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id cwh0CNl0fPeX; Mon, 23 Dec 2019 15:31:04 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 441C48B7A1; Mon, 23 Dec 2019 15:31:04 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 4E61A637D4; Mon, 23 Dec 2019 14:31:04 +0000 (UTC) Message-Id: <481907f7211425380ba1637a978c8a12143d2705.1577111366.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 03/10] lib: vdso: Change __cvdso_clock_gettime/getres_common() to __cvdso_clock_gettime/getres() To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:04 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org __cvdso_clock_getres() just calls __cvdso_clock_getres_common(). __cvdso_clock_gettime() just calls __cvdso_clock_getres_common(). Drop __cvdso_clock_getres() and __cvdso_clock_gettime() Rename __cvdso_clock_gettime_common() into __cvdso_clock_gettime() Rename __cvdso_clock_getres_common() into __cvdso_clock_getres() Signed-off-by: Christophe Leroy --- lib/vdso/gettimeofday.c | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 4618e274f1d5..c6eeeb47f446 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -79,7 +79,7 @@ static void do_coarse(const struct vdso_data *vd, clockid_t clk, } static __maybe_unused int -__cvdso_clock_gettime_common(clockid_t clock, struct __kernel_timespec *ts) +__cvdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { const struct vdso_data *vd = __arch_get_vdso_data(); u32 msk; @@ -105,16 +105,10 @@ __cvdso_clock_gettime_common(clockid_t clock, struct __kernel_timespec *ts) } static __maybe_unused int -__cvdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) -{ - return __cvdso_clock_gettime_common(clock, ts); -} - -static __maybe_unused int __cvdso_clock_gettime32(clockid_t clock, struct old_timespec32 *res) { struct __kernel_timespec ts; - int ret = __cvdso_clock_gettime_common(clock, &ts); + int ret = __cvdso_clock_gettime(clock, &ts); if (likely(!ret)) { res->tv_sec = ts.tv_sec; @@ -161,7 +155,7 @@ static __maybe_unused __kernel_old_time_t __cvdso_time(__kernel_old_time_t *time #ifdef VDSO_HAS_CLOCK_GETRES static __maybe_unused -int __cvdso_clock_getres_common(clockid_t clock, struct __kernel_timespec *res) +int __cvdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) { const struct vdso_data *vd = __arch_get_vdso_data(); u64 hrtimer_res; @@ -204,16 +198,11 @@ int __cvdso_clock_getres_common(clockid_t clock, struct __kernel_timespec *res) return 0; } -int __cvdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) -{ - return __cvdso_clock_getres_common(clock, res); -} - static __maybe_unused int __cvdso_clock_getres_time32(clockid_t clock, struct old_timespec32 *res) { struct __kernel_timespec ts; - int ret = __cvdso_clock_getres_common(clock, &ts); + int ret = __cvdso_clock_getres(clock, &ts); if (likely(!ret && res)) { res->tv_sec = ts.tv_sec; From patchwork Mon Dec 23 14:31:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308399 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6670138D for ; Mon, 23 Dec 2019 14:31:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A623521569 for ; Mon, 23 Dec 2019 14:31:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="tKloNmQ7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726943AbfLWObn (ORCPT ); Mon, 23 Dec 2019 09:31:43 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:20713 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726961AbfLWObI (ORCPT ); Mon, 23 Dec 2019 09:31:08 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGX4NCpz9txhk; Mon, 23 Dec 2019 15:31:00 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=tKloNmQ7; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id wg-FKRBGZb_D; Mon, 23 Dec 2019 15:31:00 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGX3CQYz9txhg; Mon, 23 Dec 2019 15:31:00 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111460; bh=1sCTMh9eLrKbXMBzZ6ByOaUxXu4FVEq0WFbzGJwzJ1U=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=tKloNmQ7zRuNJRg6o0sQZEisTztwPgiRt98F5HOR0wix3oiGQX5WQfx+XiUl1vYfC QDhjfXmwPgV2gfPyzdif2vpw7ca12gkm7LJtcVpyuyBRvLyW/aFwyqcH4tdNfaFqR+ yoblfORQR58ZCkx++FGC+m86d0l0jPdK5AmcT+YY= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 8CA598B7AB; Mon, 23 Dec 2019 15:31:05 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id di2S7ACdtTLy; Mon, 23 Dec 2019 15:31:05 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 56DE88B7A1; Mon, 23 Dec 2019 15:31:05 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 5BCE8637D4; Mon, 23 Dec 2019 14:31:05 +0000 (UTC) Message-Id: <02861d0a05c2d48db4e9ee9093e2e2598093c372.1577111366.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 04/10] lib: vdso: get pointer to vdso data from the arch To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:05 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org On powerpc, __arch_get_vdso_data() clobbers the link register, requiring the caller to set a stack frame in order to save it. As the parent function already has to set a stack frame and save the link register to call the C vdso function, retriving the vdso data pointer there is lighter. Give arches the opportunity to hand the vdso data pointer to C vdso functions. Signed-off-by: Christophe Leroy --- arch/arm/vdso/vgettimeofday.c | 12 ++++++++---- arch/arm64/kernel/vdso/vgettimeofday.c | 9 ++++++--- arch/arm64/kernel/vdso32/vgettimeofday.c | 12 ++++++++---- arch/mips/vdso/vgettimeofday.c | 21 ++++++++++++++------- arch/x86/entry/vdso/vclock_gettime.c | 22 +++++++++++++++------- lib/vdso/gettimeofday.c | 28 ++++++++++++++-------------- 6 files changed, 65 insertions(+), 39 deletions(-) diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 5451afb715e6..efad7d508d06 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -10,7 +10,8 @@ int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - int ret = __cvdso_clock_gettime32(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime32(vd, clock, ts); if (likely(!ret)) return ret; @@ -21,7 +22,8 @@ int __vdso_clock_gettime(clockid_t clock, int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -32,7 +34,8 @@ int __vdso_clock_gettime64(clockid_t clock, int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - int ret = __cvdso_gettimeofday(tv, tz); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_gettimeofday(vd, tv, tz); if (likely(!ret)) return ret; @@ -43,7 +46,8 @@ int __vdso_gettimeofday(struct __kernel_old_timeval *tv, int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { - int ret = __cvdso_clock_getres_time32(clock_id, res); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_getres_time32(vd, clock_id, res); if (likely(!ret)) return ret; diff --git a/arch/arm64/kernel/vdso/vgettimeofday.c b/arch/arm64/kernel/vdso/vgettimeofday.c index 62694876b216..9a7122ec6d17 100644 --- a/arch/arm64/kernel/vdso/vgettimeofday.c +++ b/arch/arm64/kernel/vdso/vgettimeofday.c @@ -11,7 +11,8 @@ int __kernel_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -22,7 +23,8 @@ int __kernel_clock_gettime(clockid_t clock, int __kernel_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - int ret = __cvdso_gettimeofday(tv, tz); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_gettimeofday(vd, tv, tz); if (likely(!ret)) return ret; @@ -33,7 +35,8 @@ int __kernel_gettimeofday(struct __kernel_old_timeval *tv, int __kernel_clock_getres(clockid_t clock_id, struct __kernel_timespec *res) { - int ret = __cvdso_clock_getres(clock_id, res); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_getres(vd, clock_id, res); if (likely(!ret)) return ret; diff --git a/arch/arm64/kernel/vdso32/vgettimeofday.c b/arch/arm64/kernel/vdso32/vgettimeofday.c index 6770d2bedd1f..3eb6a82c1c25 100644 --- a/arch/arm64/kernel/vdso32/vgettimeofday.c +++ b/arch/arm64/kernel/vdso32/vgettimeofday.c @@ -11,13 +11,14 @@ int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { + const struct vdso_data *vd = __arch_get_vdso_data(); int ret; /* The checks below are required for ABI consistency with arm */ if ((u32)ts >= TASK_SIZE_32) return -EFAULT; - ret = __cvdso_clock_gettime32(clock, ts); + ret = __cvdso_clock_gettime32(vd, clock, ts); if (likely(!ret)) return ret; @@ -28,13 +29,14 @@ int __vdso_clock_gettime(clockid_t clock, int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { + const struct vdso_data *vd = __arch_get_vdso_data(); int ret; /* The checks below are required for ABI consistency with arm */ if ((u32)ts >= TASK_SIZE_32) return -EFAULT; - ret = __cvdso_clock_gettime(clock, ts); + ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -45,7 +47,8 @@ int __vdso_clock_gettime64(clockid_t clock, int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - int ret = __cvdso_gettimeofday(tv, tz); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_gettimeofday(vd, tv, tz); if (likely(!ret)) return ret; @@ -56,6 +59,7 @@ int __vdso_gettimeofday(struct __kernel_old_timeval *tv, int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { + const struct vdso_data *vd = __arch_get_vdso_data(); int ret; struct __kernel_timespec ts; @@ -63,7 +67,7 @@ int __vdso_clock_getres(clockid_t clock_id, if ((u32)res >= TASK_SIZE_32) return -EFAULT; - ret = __cvdso_clock_getres_time32(clock_id, res); + ret = __cvdso_clock_getres_time32(vd, clock_id, res); if (likely(!ret)) return ret; diff --git a/arch/mips/vdso/vgettimeofday.c b/arch/mips/vdso/vgettimeofday.c index 02758c58454c..65d79d4695da 100644 --- a/arch/mips/vdso/vgettimeofday.c +++ b/arch/mips/vdso/vgettimeofday.c @@ -14,7 +14,8 @@ int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - int ret = __cvdso_clock_gettime32(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime32(vd, clock, ts); if (likely(!ret)) return ret; @@ -25,7 +26,8 @@ int __vdso_clock_gettime(clockid_t clock, int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - int ret = __cvdso_gettimeofday(tv, tz); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_gettimeofday(vd, tv, tz); if (likely(!ret)) return ret; @@ -36,7 +38,8 @@ int __vdso_gettimeofday(struct __kernel_old_timeval *tv, int __vdso_clock_getres(clockid_t clock_id, struct old_timespec32 *res) { - int ret = __cvdso_clock_getres_time32(clock_id, res); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_getres_time32(vd, clock_id, res); if (likely(!ret)) return ret; @@ -47,7 +50,8 @@ int __vdso_clock_getres(clockid_t clock_id, int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -60,7 +64,8 @@ int __vdso_clock_gettime64(clockid_t clock, int __vdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -71,7 +76,8 @@ int __vdso_clock_gettime(clockid_t clock, int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - int ret = __cvdso_gettimeofday(tv, tz); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_gettimeofday(vd, tv, tz); if (likely(!ret)) return ret; @@ -82,7 +88,8 @@ int __vdso_gettimeofday(struct __kernel_old_timeval *tv, int __vdso_clock_getres(clockid_t clock_id, struct __kernel_timespec *res) { - int ret = __cvdso_clock_getres(clock_id, res); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_getres(vd, clock_id, res); if (likely(!ret)) return ret; diff --git a/arch/x86/entry/vdso/vclock_gettime.c b/arch/x86/entry/vdso/vclock_gettime.c index fde511cfe284..b87847b660dd 100644 --- a/arch/x86/entry/vdso/vclock_gettime.c +++ b/arch/x86/entry/vdso/vclock_gettime.c @@ -19,7 +19,8 @@ extern __kernel_old_time_t __vdso_time(__kernel_old_time_t *t); int __vdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) { - int ret = __cvdso_gettimeofday(tv, tz); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_gettimeofday(vd, tv, tz); if (likely(!ret)) return ret; @@ -32,7 +33,9 @@ int gettimeofday(struct __kernel_old_timeval *, struct timezone *) __kernel_old_time_t __vdso_time(__kernel_old_time_t *t) { - return __cvdso_time(t); + const struct vdso_data *vd = __arch_get_vdso_data(); + + return __cvdso_time(vd, t); } __kernel_old_time_t time(__kernel_old_time_t *t) __attribute__((weak, alias("__vdso_time"))); @@ -45,7 +48,8 @@ extern int __vdso_clock_getres(clockid_t clock, struct __kernel_timespec *res); int __vdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -59,7 +63,8 @@ int clock_gettime(clockid_t, struct __kernel_timespec *) int __vdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) { - int ret = __cvdso_clock_getres(clock_id, res); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_getres(vd, clock_id, res); if (likely(!ret)) return ret; @@ -76,7 +81,8 @@ extern int __vdso_clock_getres(clockid_t clock, struct old_timespec32 *res); int __vdso_clock_gettime(clockid_t clock, struct old_timespec32 *ts) { - int ret = __cvdso_clock_gettime32(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime32(vd, clock, ts); if (likely(!ret)) return ret; @@ -89,7 +95,8 @@ int clock_gettime(clockid_t, struct old_timespec32 *) int __vdso_clock_gettime64(clockid_t clock, struct __kernel_timespec *ts) { - int ret = __cvdso_clock_gettime(clock, ts); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_gettime(vd, clock, ts); if (likely(!ret)) return ret; @@ -102,7 +109,8 @@ int clock_gettime64(clockid_t, struct __kernel_timespec *) int __vdso_clock_getres(clockid_t clock, struct old_timespec32 *res) { - int ret = __cvdso_clock_getres_time32(clock, res); + const struct vdso_data *vd = __arch_get_vdso_data(); + int ret = __cvdso_clock_getres_time32(vd, clock, res); if (likely(!ret)) return ret; diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index c6eeeb47f446..24e1ba838260 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -13,7 +13,6 @@ /* * The generic vDSO implementation requires that gettimeofday.h * provides: - * - __arch_get_vdso_data(): to get the vdso datapage. * - __arch_get_hw_counter(): to get the hw counter based on the * clock_mode. */ @@ -79,9 +78,9 @@ static void do_coarse(const struct vdso_data *vd, clockid_t clk, } static __maybe_unused int -__cvdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) +__cvdso_clock_gettime(const struct vdso_data *vd, clockid_t clock, + struct __kernel_timespec *ts) { - const struct vdso_data *vd = __arch_get_vdso_data(); u32 msk; /* Check for negative values or invalid clocks */ @@ -105,10 +104,11 @@ __cvdso_clock_gettime(clockid_t clock, struct __kernel_timespec *ts) } static __maybe_unused int -__cvdso_clock_gettime32(clockid_t clock, struct old_timespec32 *res) +__cvdso_clock_gettime32(const struct vdso_data *vd, clockid_t clock, + struct old_timespec32 *res) { struct __kernel_timespec ts; - int ret = __cvdso_clock_gettime(clock, &ts); + int ret = __cvdso_clock_gettime(vd, clock, &ts); if (likely(!ret)) { res->tv_sec = ts.tv_sec; @@ -118,10 +118,9 @@ __cvdso_clock_gettime32(clockid_t clock, struct old_timespec32 *res) } static __maybe_unused int -__cvdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) +__cvdso_gettimeofday(const struct vdso_data *vd, struct __kernel_old_timeval *tv, + struct timezone *tz) { - const struct vdso_data *vd = __arch_get_vdso_data(); - if (likely(tv != NULL)) { struct __kernel_timespec ts; @@ -141,9 +140,9 @@ __cvdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz) } #ifdef VDSO_HAS_TIME -static __maybe_unused __kernel_old_time_t __cvdso_time(__kernel_old_time_t *time) +static __maybe_unused __kernel_old_time_t +__cvdso_time(const struct vdso_data *vd, __kernel_old_time_t *time) { - const struct vdso_data *vd = __arch_get_vdso_data(); __kernel_old_time_t t = READ_ONCE(vd[CS_HRES_COARSE].basetime[CLOCK_REALTIME].sec); if (time) @@ -155,9 +154,9 @@ static __maybe_unused __kernel_old_time_t __cvdso_time(__kernel_old_time_t *time #ifdef VDSO_HAS_CLOCK_GETRES static __maybe_unused -int __cvdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) +int __cvdso_clock_getres(const struct vdso_data *vd, clockid_t clock, + struct __kernel_timespec *res) { - const struct vdso_data *vd = __arch_get_vdso_data(); u64 hrtimer_res; u32 msk; u64 ns; @@ -199,10 +198,11 @@ int __cvdso_clock_getres(clockid_t clock, struct __kernel_timespec *res) } static __maybe_unused int -__cvdso_clock_getres_time32(clockid_t clock, struct old_timespec32 *res) +__cvdso_clock_getres_time32(const struct vdso_data *vd, clockid_t clock, + struct old_timespec32 *res) { struct __kernel_timespec ts; - int ret = __cvdso_clock_getres(clock, &ts); + int ret = __cvdso_clock_getres(vd, clock, &ts); if (likely(!ret && res)) { res->tv_sec = ts.tv_sec; From patchwork Mon Dec 23 14:31:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308379 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8006D17EE for ; Mon, 23 Dec 2019 14:31:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5EBAF22522 for ; Mon, 23 Dec 2019 14:31:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="WqRlriy0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727083AbfLWObK (ORCPT ); Mon, 23 Dec 2019 09:31:10 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:14019 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726884AbfLWObI (ORCPT ); Mon, 23 Dec 2019 09:31:08 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGY4PS8z9txhl; Mon, 23 Dec 2019 15:31:01 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=WqRlriy0; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id lyyav2mt8lCR; Mon, 23 Dec 2019 15:31:01 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGY3DMLz9txhg; Mon, 23 Dec 2019 15:31:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111461; bh=4HlVdfb/m2bnebYsqTVj7kFlC+Ng414ig/AZSZp6k7s=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=WqRlriy0DqcZh60SDjoGMQijbqFO68IO3uompTe3PTUn5JPopZMKehw9+2nPSPIZP hwfYncZjR/MLSaTvHfyl3A0w9elV2k2/xL/eg8+TK6KKCyFx1bzcUuK7n2o2YsIyfr HAsfky9f/aUORYv0A9P41kc9Lg3ATeICFVaYId54= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 8959D8B7AB; Mon, 23 Dec 2019 15:31:06 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id pPU4VC3HWwT0; Mon, 23 Dec 2019 15:31:06 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 5CBAE8B7A1; Mon, 23 Dec 2019 15:31:06 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 66892637D4; Mon, 23 Dec 2019 14:31:06 +0000 (UTC) Message-Id: In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 05/10] lib: vdso: inline do_hres() To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:06 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org do_hres() is called from several places, so GCC doesn't inline it at first. do_hres() takes a struct __kernel_timespec * parameter for passing the result. In the 32 bits case, this parameter corresponds to a local var in the caller. In order to provide a pointer to this structure, the caller has to put it in its stack and do_hres() has to write the result in the stack. This is suboptimal, especially on RISC processor like powerpc. By making GCC inline the function, the struct __kernel_timespec remains a local var using registers, avoiding the need to write and read stack. The improvement is significant on powerpc. Signed-off-by: Christophe Leroy Acked-by: Arnd Bergmann --- lib/vdso/gettimeofday.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 24e1ba838260..86d5b1c8796b 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -34,8 +34,8 @@ u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) } #endif -static int do_hres(const struct vdso_data *vd, clockid_t clk, - struct __kernel_timespec *ts) +static inline int do_hres(const struct vdso_data *vd, clockid_t clk, + struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts = &vd->basetime[clk]; u64 cycles, last, sec, ns; From patchwork Mon Dec 23 14:31:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308393 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3207E138D for ; Mon, 23 Dec 2019 14:31:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0F65320CC7 for ; Mon, 23 Dec 2019 14:31:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="KCu8JdhV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727070AbfLWObK (ORCPT ); Mon, 23 Dec 2019 09:31:10 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:38665 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727008AbfLWObJ (ORCPT ); Mon, 23 Dec 2019 09:31:09 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGZ4Tyvz9txjv; Mon, 23 Dec 2019 15:31:02 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=KCu8JdhV; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id mFPGrRR_Px4H; Mon, 23 Dec 2019 15:31:02 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGZ3S6Mz9txhg; Mon, 23 Dec 2019 15:31:02 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111462; bh=gGQQWU2/OzDgivXq6POKzHHoU8nAP0pffpFK4Hw4f5o=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=KCu8JdhV8ril/SLZ2cIjpvSR6KUhaaxUBf12AAawmc6m9awzyxj0erkSut9SwKMy4 lIa9o/tvu2IQe7H4F3ik2pCVBURsKq/6FA5TZsotz4+/RJuPrSovLLYC7Qh7aSnqoZ 0urZbdFBzVdeOA77s9XkFrxcctol1LftVrxIWgMg= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 908908B7AB; Mon, 23 Dec 2019 15:31:07 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 0Ts-AinkXcCE; Mon, 23 Dec 2019 15:31:07 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6226C8B7A1; Mon, 23 Dec 2019 15:31:07 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 6C7BC637D4; Mon, 23 Dec 2019 14:31:07 +0000 (UTC) Message-Id: <21e8afa38c02ca8672c2690307383507fe63b454.1577111367.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 06/10] lib: vdso: make do_coarse() return 0 To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:07 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org do_coarse() is similare to do_hres() except that it never fails. Change its type to int instead of void and get it return 0 at all time. This cleans the code a bit. Signed-off-by: Christophe Leroy --- lib/vdso/gettimeofday.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 86d5b1c8796b..17b4cff6e5f0 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -64,7 +64,7 @@ static inline int do_hres(const struct vdso_data *vd, clockid_t clk, return 0; } -static void do_coarse(const struct vdso_data *vd, clockid_t clk, +static int do_coarse(const struct vdso_data *vd, clockid_t clk, struct __kernel_timespec *ts) { const struct vdso_timestamp *vdso_ts = &vd->basetime[clk]; @@ -75,6 +75,8 @@ static void do_coarse(const struct vdso_data *vd, clockid_t clk, ts->tv_sec = vdso_ts->sec; ts->tv_nsec = vdso_ts->nsec; } while (unlikely(vdso_read_retry(vd, seq))); + + return 0; } static __maybe_unused int @@ -92,14 +94,13 @@ __cvdso_clock_gettime(const struct vdso_data *vd, clockid_t clock, * clocks are handled in the VDSO directly. */ msk = 1U << clock; - if (likely(msk & VDSO_HRES)) { + if (likely(msk & VDSO_HRES)) return do_hres(&vd[CS_HRES_COARSE], clock, ts); - } else if (msk & VDSO_COARSE) { - do_coarse(&vd[CS_HRES_COARSE], clock, ts); - return 0; - } else if (msk & VDSO_RAW) { + else if (msk & VDSO_COARSE) + return do_coarse(&vd[CS_HRES_COARSE], clock, ts); + else if (msk & VDSO_RAW) return do_hres(&vd[CS_RAW], clock, ts); - } + return -1; } From patchwork Mon Dec 23 14:31:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308395 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5C000921 for ; Mon, 23 Dec 2019 14:31:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3BA9621775 for ; Mon, 23 Dec 2019 14:31:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="HzWceXyE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727322AbfLWObf (ORCPT ); Mon, 23 Dec 2019 09:31:35 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:18097 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726943AbfLWObK (ORCPT ); Mon, 23 Dec 2019 09:31:10 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGb4QXVz9txjy; Mon, 23 Dec 2019 15:31:03 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=HzWceXyE; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id bkTv1TgFVniR; Mon, 23 Dec 2019 15:31:03 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGb3Gynz9txjx; Mon, 23 Dec 2019 15:31:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111463; bh=N2ZgUlBGtsvYJmL4aNIvn4kelyr3xUIlcSwbXOh8dAY=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=HzWceXyEVD1Emm0wwEOE1bCdfm4qtJn+RLfuJ+8Hl/RwcdV35a4DNiyknWPUyNLZq wVAMMpO7O4TDPh8O30kOtwmkeCpiX9AOkoMz5d8KvvzSAP/UyJu/lPQ2RvdFwZHmwj 97Amm7bJkkQbo4p6/X7Ca0N8qAiBNpv9MLgC0LWY= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 8E2F98B7AB; Mon, 23 Dec 2019 15:31:08 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id LhG5tUkjJ7iO; Mon, 23 Dec 2019 15:31:08 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 67BCF8B7A1; Mon, 23 Dec 2019 15:31:08 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 71E13637D4; Mon, 23 Dec 2019 14:31:08 +0000 (UTC) Message-Id: In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 07/10] lib: vdso: don't use READ_ONCE() in __c_kernel_time() To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:08 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org READ_ONCE() forces the read of the 64 bit value of vd[CS_HRES_COARSE].basetime[CLOCK_REALTIME].sec allthough only the lower part is needed. This results in a suboptimal code: 00000af4 <__c_kernel_time>: af4: 2c 03 00 00 cmpwi r3,0 af8: 81 44 00 20 lwz r10,32(r4) afc: 81 64 00 24 lwz r11,36(r4) b00: 41 82 00 08 beq b08 <__c_kernel_time+0x14> b04: 91 63 00 00 stw r11,0(r3) b08: 7d 63 5b 78 mr r3,r11 b0c: 4e 80 00 20 blr By removing the READ_ONCE(), only the lower part is read from memory, and the code is cleaner: 00000af4 <__c_kernel_time>: af4: 7c 69 1b 79 mr. r9,r3 af8: 80 64 00 24 lwz r3,36(r4) afc: 4d 82 00 20 beqlr b00: 90 69 00 00 stw r3,0(r9) b04: 4e 80 00 20 blr Signed-off-by: Christophe Leroy Reviewed-by: Andy Lutomirski --- lib/vdso/gettimeofday.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 17b4cff6e5f0..5a17a9d2e6cd 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -144,7 +144,7 @@ __cvdso_gettimeofday(const struct vdso_data *vd, struct __kernel_old_timeval *tv static __maybe_unused __kernel_old_time_t __cvdso_time(const struct vdso_data *vd, __kernel_old_time_t *time) { - __kernel_old_time_t t = READ_ONCE(vd[CS_HRES_COARSE].basetime[CLOCK_REALTIME].sec); + __kernel_old_time_t t = vd[CS_HRES_COARSE].basetime[CLOCK_REALTIME].sec; if (time) *time = t; From patchwork Mon Dec 23 14:31:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308397 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 39440921 for ; Mon, 23 Dec 2019 14:31:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 17CB421927 for ; Mon, 23 Dec 2019 14:31:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="bC4sfPet" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727109AbfLWObf (ORCPT ); Mon, 23 Dec 2019 09:31:35 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:20713 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727050AbfLWObK (ORCPT ); Mon, 23 Dec 2019 09:31:10 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGc4X9kz9txk0; Mon, 23 Dec 2019 15:31:04 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=bC4sfPet; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id pExXBshqEC32; Mon, 23 Dec 2019 15:31:04 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGc3Ssxz9txjx; Mon, 23 Dec 2019 15:31:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111464; bh=JG69RWeha6scpE4l/gVcRXcizrOVP3foZOy4iqNP618=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=bC4sfPetFvQitz/cwbqgoP0D7XeTFqeBALfhCtuKUQ5tMvQT7pHJQS1+b0l9JrPcz eexkkMeeDT2T/XBLr2WnAlkrSdW2CUB8+pi/JPWE4HSu5wiK5QvoZWzR/uDslt9/8p wZELsHr6VHtkcMUSktuN1Njvgb6csIiPiSV/VV5s= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 932568B7AB; Mon, 23 Dec 2019 15:31:09 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id ZDsnIZyOiV_3; Mon, 23 Dec 2019 15:31:09 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 6EFD88B7A1; Mon, 23 Dec 2019 15:31:09 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 79148637D4; Mon, 23 Dec 2019 14:31:09 +0000 (UTC) Message-Id: In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 08/10] lib: vdso: Avoid duplication in __cvdso_clock_getres() To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:09 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org VDSO_HRES and VDSO_RAW clocks are handled the same way. Don't duplicate code. Signed-off-by: Christophe Leroy Reviewed-by: Andy Lutomirski --- lib/vdso/gettimeofday.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/lib/vdso/gettimeofday.c b/lib/vdso/gettimeofday.c index 5a17a9d2e6cd..aa4a167bf1e0 100644 --- a/lib/vdso/gettimeofday.c +++ b/lib/vdso/gettimeofday.c @@ -172,7 +172,7 @@ int __cvdso_clock_getres(const struct vdso_data *vd, clockid_t clock, * clocks are handled in the VDSO directly. */ msk = 1U << clock; - if (msk & VDSO_HRES) { + if (msk & (VDSO_HRES | VDSO_RAW)) { /* * Preserves the behaviour of posix_get_hrtimer_res(). */ @@ -182,11 +182,6 @@ int __cvdso_clock_getres(const struct vdso_data *vd, clockid_t clock, * Preserves the behaviour of posix_get_coarse_res(). */ ns = LOW_RES_NSEC; - } else if (msk & VDSO_RAW) { - /* - * Preserves the behaviour of posix_get_hrtimer_res(). - */ - ns = hrtimer_res; } else { return -1; } From patchwork Mon Dec 23 14:31:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308387 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F1B9F138D for ; Mon, 23 Dec 2019 14:31:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C718520709 for ; Mon, 23 Dec 2019 14:31:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="CYa8vo7J" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727132AbfLWOb0 (ORCPT ); Mon, 23 Dec 2019 09:31:26 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:9418 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727100AbfLWObM (ORCPT ); Mon, 23 Dec 2019 09:31:12 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGd5Q88z9txk1; Mon, 23 Dec 2019 15:31:05 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=CYa8vo7J; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id sgBO7Sf4a4HI; Mon, 23 Dec 2019 15:31:05 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGd4MbJz9txjx; Mon, 23 Dec 2019 15:31:05 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111465; bh=t9hQX4/SzVKeiT0tcSoxOBftxg2wIkN3Ts7FVCdCdLs=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=CYa8vo7Jpg7BwLn28d4u3EMnxVYWj7Q5ev7rMu3Fdz+VQMKfALP1kDWTibo5y+IXM Ba5KGc4ye0BfgBAvDGRxgtNb+yLundPZIg32c/Sk5Rh04JSQO5i6IXSZ2vewiqB4S4 MtbJX3cOY2pqrrMvHlgXwSWKilhAW+G96N8QCVCQ= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id B06838B7AB; Mon, 23 Dec 2019 15:31:10 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id XL0AoMbhhIk0; Mon, 23 Dec 2019 15:31:10 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 767B58B7A1; Mon, 23 Dec 2019 15:31:10 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 80B72637D4; Mon, 23 Dec 2019 14:31:10 +0000 (UTC) Message-Id: <4e44aca7d8c3b5ebb1453d499354ca5233d6defb.1577111367.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 09/10] powerpc/vdso32: inline __get_datapage() To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:10 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org __get_datapage() is only a few instructions to retrieve the address of the page where the kernel stores data to the VDSO. By inlining this function into its users, a bl/blr pair and a mflr/mtlr pair is avoided, plus a few reg moves. The improvement is noticeable (about 55 nsec/call on an 8xx) vdsotest before the patch: gettimeofday: vdso: 731 nsec/call clock-gettime-realtime-coarse: vdso: 668 nsec/call clock-gettime-monotonic-coarse: vdso: 745 nsec/call vdsotest after the patch: gettimeofday: vdso: 677 nsec/call clock-gettime-realtime-coarse: vdso: 613 nsec/call clock-gettime-monotonic-coarse: vdso: 690 nsec/call Signed-off-by: Christophe Leroy --- v3: define get_datapage macro in asm/vdso_datapage.h v4: fixed build failure with old binutils --- arch/powerpc/include/asm/vdso_datapage.h | 10 ++++++++++ arch/powerpc/kernel/vdso32/cacheflush.S | 9 ++++----- arch/powerpc/kernel/vdso32/datapage.S | 28 +++------------------------- arch/powerpc/kernel/vdso32/gettimeofday.S | 12 +++++------- 4 files changed, 22 insertions(+), 37 deletions(-) diff --git a/arch/powerpc/include/asm/vdso_datapage.h b/arch/powerpc/include/asm/vdso_datapage.h index 40f13f3626d3..ee5319a6f4e3 100644 --- a/arch/powerpc/include/asm/vdso_datapage.h +++ b/arch/powerpc/include/asm/vdso_datapage.h @@ -118,6 +118,16 @@ struct vdso_data { extern struct vdso_data *vdso_data; +#else /* __ASSEMBLY__ */ + +.macro get_datapage ptr, tmp + bcl 20, 31, .+4 + mflr \ptr + addi \ptr, \ptr, (__kernel_datapage_offset - (.-4))@l + lwz \tmp, 0(\ptr) + add \ptr, \tmp, \ptr +.endm + #endif /* __ASSEMBLY__ */ #endif /* __KERNEL__ */ diff --git a/arch/powerpc/kernel/vdso32/cacheflush.S b/arch/powerpc/kernel/vdso32/cacheflush.S index 7f882e7b9f43..d178ec8c279d 100644 --- a/arch/powerpc/kernel/vdso32/cacheflush.S +++ b/arch/powerpc/kernel/vdso32/cacheflush.S @@ -8,6 +8,7 @@ #include #include #include +#include #include .text @@ -24,14 +25,12 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache) .cfi_startproc mflr r12 .cfi_register lr,r12 - mr r11,r3 - bl __get_datapage@local + get_datapage r10, r0 mtlr r12 - mr r10,r3 lwz r7,CFG_DCACHE_BLOCKSZ(r10) addi r5,r7,-1 - andc r6,r11,r5 /* round low to line bdy */ + andc r6,r3,r5 /* round low to line bdy */ subf r8,r6,r4 /* compute length */ add r8,r8,r5 /* ensure we get enough */ lwz r9,CFG_DCACHE_LOGBLOCKSZ(r10) @@ -48,7 +47,7 @@ V_FUNCTION_BEGIN(__kernel_sync_dicache) lwz r7,CFG_ICACHE_BLOCKSZ(r10) addi r5,r7,-1 - andc r6,r11,r5 /* round low to line bdy */ + andc r6,r3,r5 /* round low to line bdy */ subf r8,r6,r4 /* compute length */ add r8,r8,r5 lwz r9,CFG_ICACHE_LOGBLOCKSZ(r10) diff --git a/arch/powerpc/kernel/vdso32/datapage.S b/arch/powerpc/kernel/vdso32/datapage.S index 6c7401bd284e..1095d818f94a 100644 --- a/arch/powerpc/kernel/vdso32/datapage.S +++ b/arch/powerpc/kernel/vdso32/datapage.S @@ -10,35 +10,13 @@ #include #include #include +#include .text .global __kernel_datapage_offset; __kernel_datapage_offset: .long 0 -V_FUNCTION_BEGIN(__get_datapage) - .cfi_startproc - /* We don't want that exposed or overridable as we want other objects - * to be able to bl directly to here - */ - .protected __get_datapage - .hidden __get_datapage - - mflr r0 - .cfi_register lr,r0 - - bcl 20,31,data_page_branch -data_page_branch: - mflr r3 - mtlr r0 - addi r3, r3, __kernel_datapage_offset-data_page_branch - lwz r0,0(r3) - .cfi_restore lr - add r3,r0,r3 - blr - .cfi_endproc -V_FUNCTION_END(__get_datapage) - /* * void *__kernel_get_syscall_map(unsigned int *syscall_count) ; * @@ -53,7 +31,7 @@ V_FUNCTION_BEGIN(__kernel_get_syscall_map) mflr r12 .cfi_register lr,r12 mr r4,r3 - bl __get_datapage@local + get_datapage r3, r0 mtlr r12 addi r3,r3,CFG_SYSCALL_MAP32 cmpli cr0,r4,0 @@ -75,7 +53,7 @@ V_FUNCTION_BEGIN(__kernel_get_tbfreq) .cfi_startproc mflr r12 .cfi_register lr,r12 - bl __get_datapage@local + get_datapage r3, r0 lwz r4,(CFG_TB_TICKS_PER_SEC + 4)(r3) lwz r3,CFG_TB_TICKS_PER_SEC(r3) mtlr r12 diff --git a/arch/powerpc/kernel/vdso32/gettimeofday.S b/arch/powerpc/kernel/vdso32/gettimeofday.S index 3306672f57a9..d6c1d331e8cb 100644 --- a/arch/powerpc/kernel/vdso32/gettimeofday.S +++ b/arch/powerpc/kernel/vdso32/gettimeofday.S @@ -9,6 +9,7 @@ #include #include #include +#include #include #include @@ -33,8 +34,7 @@ V_FUNCTION_BEGIN(__kernel_gettimeofday) mr r10,r3 /* r10 saves tv */ mr r11,r4 /* r11 saves tz */ - bl __get_datapage@local /* get data page */ - mr r9, r3 /* datapage ptr in r9 */ + get_datapage r9, r0 cmplwi r10,0 /* check if tv is NULL */ beq 3f lis r7,1000000@ha /* load up USEC_PER_SEC */ @@ -74,8 +74,7 @@ V_FUNCTION_BEGIN(__kernel_clock_gettime) mflr r12 /* r12 saves lr */ .cfi_register lr,r12 mr r11,r4 /* r11 saves tp */ - bl __get_datapage@local /* get data page */ - mr r9,r3 /* datapage ptr in r9 */ + get_datapage r9, r0 lis r7,NSEC_PER_SEC@h /* want nanoseconds */ ori r7,r7,NSEC_PER_SEC@l 50: bl __do_get_tspec@local /* get sec/nsec from tb & kernel */ @@ -156,7 +155,7 @@ V_FUNCTION_BEGIN(__kernel_clock_getres) mflr r12 .cfi_register lr,r12 - bl __get_datapage@local /* get data page */ + get_datapage r3, r0 lwz r5, CLOCK_HRTIMER_RES(r3) mtlr r12 li r3,0 @@ -190,8 +189,7 @@ V_FUNCTION_BEGIN(__kernel_time) .cfi_register lr,r12 mr r11,r3 /* r11 holds t */ - bl __get_datapage@local - mr r9, r3 /* datapage ptr in r9 */ + get_datapage r9, r0 lwz r3,STAMP_XTIME_SEC+LOPART(r9) From patchwork Mon Dec 23 14:31:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christophe Leroy X-Patchwork-Id: 11308385 X-Patchwork-Delegate: paulburton@kernel.org Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 440ED138D for ; Mon, 23 Dec 2019 14:31:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EF46120709 for ; Mon, 23 Dec 2019 14:31:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=c-s.fr header.i=@c-s.fr header.b="IbJAAfab" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727183AbfLWObW (ORCPT ); Mon, 23 Dec 2019 09:31:22 -0500 Received: from pegase1.c-s.fr ([93.17.236.30]:51910 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727132AbfLWObO (ORCPT ); Mon, 23 Dec 2019 09:31:14 -0500 Received: from localhost (mailhub1-ext [192.168.12.233]) by localhost (Postfix) with ESMTP id 47hMGf5lZPz9txhg; Mon, 23 Dec 2019 15:31:06 +0100 (CET) Authentication-Results: localhost; dkim=pass reason="1024-bit key; insecure key" header.d=c-s.fr header.i=@c-s.fr header.b=IbJAAfab; dkim-adsp=pass; dkim-atps=neutral X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id bxkl72vvCPnV; Mon, 23 Dec 2019 15:31:06 +0100 (CET) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 47hMGf4XW1z9txjx; Mon, 23 Dec 2019 15:31:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=c-s.fr; s=mail; t=1577111466; bh=+8OmdU+o5PIgDQxQpV7oA0SQW5Tg66AaRlGpPEcqetM=; h=In-Reply-To:References:From:Subject:To:Cc:Date:From; b=IbJAAfabOg9QHdZ9kXWdFicAtxmqDF6clK8TxJTlSDe08yrR0k3j+b9GYd3y+JFfE 3DQbG+KaerxSQVsxOcVZq2NJGy0P28o1aiJMrDCJIqS1U8MtkhP0ZchrI9m+zg9xTX mOsctj61JNOIKTIwVwzYkfm/l0Yp58A+DvGGT/FI= Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id B1A168B7AB; Mon, 23 Dec 2019 15:31:11 +0100 (CET) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id PkSB3NvEJQPO; Mon, 23 Dec 2019 15:31:11 +0100 (CET) Received: from po16098vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.230.100]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 7E1B08B7A1; Mon, 23 Dec 2019 15:31:11 +0100 (CET) Received: by po16098vm.idsi0.si.c-s.fr (Postfix, from userid 0) id 8754A637D4; Mon, 23 Dec 2019 14:31:11 +0000 (UTC) Message-Id: <24923718d2a6ab0c79aebec13eb6acebdc3cf0e9.1577111367.git.christophe.leroy@c-s.fr> In-Reply-To: References: From: Christophe Leroy Subject: [RFC PATCH v2 10/10] powerpc/32: Switch VDSO to C implementation. To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , arnd@arndb.de, tglx@linutronix.de, vincenzo.frascino@arm.com, luto@kernel.org Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, linux-mips@vger.kernel.org, x86@kernel.org Date: Mon, 23 Dec 2019 14:31:11 +0000 (UTC) Sender: linux-mips-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org This is a tentative to switch powerpc/32 vdso to generic C implementation. It will likely not work on 64 bits or even build properly at the moment, hence the RFC status. powerpc is a bit special for VDSO as well as system calls in the way that it requires setting CR SO bit which cannot be done in C. Therefore, entry/exit and fallback needs to be performed in ASM. On powerpc 8xx, performance is degraded by 30-40% for gettime and by 15-20% for getres On a powerpc885 at 132MHz: With current powerpc/32 ASM VDSO: gettimeofday: vdso: 737 nsec/call clock-getres-realtime-coarse: vdso: 3081 nsec/call clock-gettime-realtime-coarse: vdso: 2861 nsec/call clock-getres-realtime: vdso: 475 nsec/call clock-gettime-realtime: vdso: 892 nsec/call clock-getres-boottime: vdso: 2621 nsec/call clock-gettime-boottime: vdso: 3857 nsec/call clock-getres-tai: vdso: 2620 nsec/call clock-gettime-tai: vdso: 3854 nsec/call clock-getres-monotonic-raw: vdso: 2621 nsec/call clock-gettime-monotonic-raw: vdso: 3499 nsec/call clock-getres-monotonic-coarse: vdso: 3083 nsec/call clock-gettime-monotonic-coarse: vdso: 3082 nsec/call clock-getres-monotonic: vdso: 475 nsec/call clock-gettime-monotonic: vdso: 1014 nsec/call Once switched to C implementation: gettimeofday: vdso: 1016 nsec/call clock-getres-realtime-coarse: vdso: 614 nsec/call clock-gettime-realtime-coarse: vdso: 760 nsec/call clock-getres-realtime: vdso: 560 nsec/call clock-gettime-realtime: vdso: 1192 nsec/call clock-getres-boottime: vdso: 560 nsec/call clock-gettime-boottime: vdso: 1194 nsec/call clock-getres-tai: vdso: 560 nsec/call clock-gettime-tai: vdso: 1192 nsec/call clock-getres-monotonic-raw: vdso: 560 nsec/call clock-gettime-monotonic-raw: vdso: 1248 nsec/call clock-getres-monotonic-coarse: vdso: 614 nsec/call clock-gettime-monotonic-coarse: vdso: 760 nsec/call clock-getres-monotonic: vdso: 560 nsec/call clock-gettime-monotonic: vdso: 1192 nsec/call On a powerpc 8321 running at 333MHz With current powerpc/32 ASM VDSO: gettimeofday: vdso: 190 nsec/call clock-getres-realtime-coarse: vdso: 1449 nsec/call clock-gettime-realtime-coarse: vdso: 1352 nsec/call clock-getres-realtime: vdso: 135 nsec/call clock-gettime-realtime: vdso: 244 nsec/call clock-getres-boottime: vdso: 1313 nsec/call clock-gettime-boottime: vdso: 1701 nsec/call clock-getres-tai: vdso: 1268 nsec/call clock-gettime-tai: vdso: 1742 nsec/call clock-getres-monotonic-raw: vdso: 1310 nsec/call clock-gettime-monotonic-raw: vdso: 1584 nsec/call clock-getres-monotonic-coarse: vdso: 1488 nsec/call clock-gettime-monotonic-coarse: vdso: 1503 nsec/call clock-getres-monotonic: vdso: 135 nsec/call clock-gettime-monotonic: vdso: 283 nsec/call Once switched to C implementation: gettimeofday: vdso: 347 nsec/call clock-getres-realtime-coarse: vdso: 169 nsec/call clock-gettime-realtime-coarse: vdso: 271 nsec/call clock-getres-realtime: vdso: 150 nsec/call clock-gettime-realtime: vdso: 383 nsec/call clock-getres-boottime: vdso: 157 nsec/call clock-gettime-boottime: vdso: 377 nsec/call clock-getres-tai: vdso: 150 nsec/call clock-gettime-tai: vdso: 380 nsec/call clock-getres-monotonic-raw: vdso: 153 nsec/call clock-gettime-monotonic-raw: vdso: 407 nsec/call clock-getres-monotonic-coarse: vdso: 169 nsec/call clock-gettime-monotonic-coarse: vdso: 271 nsec/call clock-getres-monotonic: vdso: 153 nsec/call clock-gettime-monotonic: vdso: 377 nsec/call Signed-off-by: Christophe Leroy --- arch/powerpc/Kconfig | 2 + arch/powerpc/include/asm/vdso/gettimeofday.h | 45 +++++ arch/powerpc/include/asm/vdso/vsyscall.h | 27 +++ arch/powerpc/include/asm/vdso_datapage.h | 18 +- arch/powerpc/kernel/asm-offsets.c | 23 +-- arch/powerpc/kernel/time.c | 92 +--------- arch/powerpc/kernel/vdso.c | 19 +- arch/powerpc/kernel/vdso32/Makefile | 19 +- arch/powerpc/kernel/vdso32/gettimeofday.S | 261 ++++----------------------- arch/powerpc/kernel/vdso32/vgettimeofday.c | 32 ++++ 10 files changed, 178 insertions(+), 360 deletions(-) create mode 100644 arch/powerpc/include/asm/vdso/gettimeofday.h create mode 100644 arch/powerpc/include/asm/vdso/vsyscall.h create mode 100644 arch/powerpc/kernel/vdso32/vgettimeofday.c diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1ec34e16ed65..bd04c68baf91 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -169,6 +169,7 @@ config PPC select GENERIC_STRNCPY_FROM_USER select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL + select GENERIC_GETTIMEOFDAY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU select HAVE_ARCH_JUMP_LABEL @@ -198,6 +199,7 @@ config PPC select HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER select HAVE_GCC_PLUGINS if GCC_VERSION >= 50200 # plugin support on gcc <= 5.1 is buggy on PPC + select HAVE_GENERIC_VDSO select HAVE_HW_BREAKPOINT if PERF_EVENTS && (PPC_BOOK3S || PPC_8xx) select HAVE_IDE select HAVE_IOREMAP_PROT diff --git a/arch/powerpc/include/asm/vdso/gettimeofday.h b/arch/powerpc/include/asm/vdso/gettimeofday.h new file mode 100644 index 000000000000..e170e12a78bb --- /dev/null +++ b/arch/powerpc/include/asm/vdso/gettimeofday.h @@ -0,0 +1,45 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_VDSO_GETTIMEOFDAY_H +#define __ASM_VDSO_GETTIMEOFDAY_H + +#ifndef __ASSEMBLY__ + +#include +#include +#include + +#define VDSO_HAS_CLOCK_GETRES 1 + +#define VDSO_HAS_TIME 1 + +static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) +{ + /* + * clock_mode == 0 implies that vDSO are enabled otherwise + * fallback on syscall. + */ + if (clock_mode) + return ULLONG_MAX; + + return get_tb() & LLONG_MAX; +} + +/* + * powerpc specific delta calculation. + * + * This variant removes the masking of the subtraction because the + * clocksource mask of all VDSO capable clocksources on powerpc is U64_MAX + * which would result in a pointless operation. The compiler cannot + * optimize it away as the mask comes from the vdso data and is not compile + * time constant. + */ +static __always_inline +u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) +{ + return (cycles - last) * mult; +} +#define vdso_calc_delta vdso_calc_delta + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ASM_VDSO_GETTIMEOFDAY_H */ diff --git a/arch/powerpc/include/asm/vdso/vsyscall.h b/arch/powerpc/include/asm/vdso/vsyscall.h new file mode 100644 index 000000000000..d12c2298cbb8 --- /dev/null +++ b/arch/powerpc/include/asm/vdso/vsyscall.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_VDSO_VSYSCALL_H +#define __ASM_VDSO_VSYSCALL_H + +#ifndef __ASSEMBLY__ + +#include +#include + +extern struct vdso_arch_data *vdso_arch_data; + +/* + * Update the vDSO data page to keep in sync with kernel timekeeping. + */ +static __always_inline +struct vdso_data *__powerpc_get_k_vdso_data(void) +{ + return vdso_arch_data->data; +} +#define __arch_get_k_vdso_data __powerpc_get_k_vdso_data + +/* The asm-generic header needs to be included after the definitions above */ +#include + +#endif /* !__ASSEMBLY__ */ + +#endif /* __ASM_VDSO_VSYSCALL_H */ diff --git a/arch/powerpc/include/asm/vdso_datapage.h b/arch/powerpc/include/asm/vdso_datapage.h index ee5319a6f4e3..6b9424ca91c4 100644 --- a/arch/powerpc/include/asm/vdso_datapage.h +++ b/arch/powerpc/include/asm/vdso_datapage.h @@ -36,6 +36,7 @@ #include #include +#include #define SYSCALL_MAP_SIZE ((NR_syscalls + 31) / 32) @@ -93,20 +94,9 @@ struct vdso_data { /* * And here is the simpler 32 bits version */ -struct vdso_data { - __u64 tb_orig_stamp; /* Timebase at boot 0x30 */ +struct vdso_arch_data { + struct vdso_data data[CS_BASES]; __u64 tb_ticks_per_sec; /* Timebase tics / sec 0x38 */ - __u64 tb_to_xs; /* Inverse of TB to 2^20 0x40 */ - __u64 stamp_xsec; /* 0x48 */ - __u32 tb_update_count; /* Timebase atomicity ctr 0x50 */ - __u32 tz_minuteswest; /* Minutes west of Greenwich 0x58 */ - __u32 tz_dsttime; /* Type of dst correction 0x5C */ - __s32 wtom_clock_sec; /* Wall to monotonic clock */ - __s32 wtom_clock_nsec; - __s32 stamp_xtime_sec; /* xtime seconds as at tb_orig_stamp */ - __s32 stamp_xtime_nsec; /* xtime nsecs as at tb_orig_stamp */ - __u32 stamp_sec_fraction; /* fractional seconds of stamp_xtime */ - __u32 hrtimer_res; /* hrtimer resolution */ __u32 syscall_map_32[SYSCALL_MAP_SIZE]; /* map of syscalls */ __u32 dcache_block_size; /* L1 d-cache block size */ __u32 icache_block_size; /* L1 i-cache block size */ @@ -116,7 +106,7 @@ struct vdso_data { #endif /* CONFIG_PPC64 */ -extern struct vdso_data *vdso_data; +extern struct vdso_arch_data *vdso_arch_data; #else /* __ASSEMBLY__ */ diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c index 3d47aec7becf..5927c9cf9cc7 100644 --- a/arch/powerpc/kernel/asm-offsets.c +++ b/arch/powerpc/kernel/asm-offsets.c @@ -376,23 +376,12 @@ int main(void) #endif /* ! CONFIG_PPC64 */ /* datapage offsets for use by vdso */ - OFFSET(CFG_TB_ORIG_STAMP, vdso_data, tb_orig_stamp); - OFFSET(CFG_TB_TICKS_PER_SEC, vdso_data, tb_ticks_per_sec); - OFFSET(CFG_TB_TO_XS, vdso_data, tb_to_xs); - OFFSET(CFG_TB_UPDATE_COUNT, vdso_data, tb_update_count); - OFFSET(CFG_TZ_MINUTEWEST, vdso_data, tz_minuteswest); - OFFSET(CFG_TZ_DSTTIME, vdso_data, tz_dsttime); - OFFSET(CFG_SYSCALL_MAP32, vdso_data, syscall_map_32); - OFFSET(WTOM_CLOCK_SEC, vdso_data, wtom_clock_sec); - OFFSET(WTOM_CLOCK_NSEC, vdso_data, wtom_clock_nsec); - OFFSET(STAMP_XTIME_SEC, vdso_data, stamp_xtime_sec); - OFFSET(STAMP_XTIME_NSEC, vdso_data, stamp_xtime_nsec); - OFFSET(STAMP_SEC_FRAC, vdso_data, stamp_sec_fraction); - OFFSET(CLOCK_HRTIMER_RES, vdso_data, hrtimer_res); - OFFSET(CFG_ICACHE_BLOCKSZ, vdso_data, icache_block_size); - OFFSET(CFG_DCACHE_BLOCKSZ, vdso_data, dcache_block_size); - OFFSET(CFG_ICACHE_LOGBLOCKSZ, vdso_data, icache_log_block_size); - OFFSET(CFG_DCACHE_LOGBLOCKSZ, vdso_data, dcache_log_block_size); + OFFSET(CFG_TB_TICKS_PER_SEC, vdso_arch_data, tb_ticks_per_sec); + OFFSET(CFG_SYSCALL_MAP32, vdso_arch_data, syscall_map_32); + OFFSET(CFG_ICACHE_BLOCKSZ, vdso_arch_data, icache_block_size); + OFFSET(CFG_DCACHE_BLOCKSZ, vdso_arch_data, dcache_block_size); + OFFSET(CFG_ICACHE_LOGBLOCKSZ, vdso_arch_data, icache_log_block_size); + OFFSET(CFG_DCACHE_LOGBLOCKSZ, vdso_arch_data, dcache_log_block_size); #ifdef CONFIG_PPC64 OFFSET(CFG_SYSCALL_MAP64, vdso_data, syscall_map_64); OFFSET(TVAL64_TV_SEC, __kernel_old_timeval, tv_sec); diff --git a/arch/powerpc/kernel/time.c b/arch/powerpc/kernel/time.c index 1168e8b37e30..e0c8023db42c 100644 --- a/arch/powerpc/kernel/time.c +++ b/arch/powerpc/kernel/time.c @@ -882,95 +882,6 @@ static notrace u64 timebase_read(struct clocksource *cs) return (u64)get_tb(); } - -void update_vsyscall(struct timekeeper *tk) -{ - struct timespec64 xt; - struct clocksource *clock = tk->tkr_mono.clock; - u32 mult = tk->tkr_mono.mult; - u32 shift = tk->tkr_mono.shift; - u64 cycle_last = tk->tkr_mono.cycle_last; - u64 new_tb_to_xs, new_stamp_xsec; - u64 frac_sec; - - if (clock != &clocksource_timebase) - return; - - xt.tv_sec = tk->xtime_sec; - xt.tv_nsec = (long)(tk->tkr_mono.xtime_nsec >> tk->tkr_mono.shift); - - /* Make userspace gettimeofday spin until we're done. */ - ++vdso_data->tb_update_count; - smp_mb(); - - /* - * This computes ((2^20 / 1e9) * mult) >> shift as a - * 0.64 fixed-point fraction. - * The computation in the else clause below won't overflow - * (as long as the timebase frequency is >= 1.049 MHz) - * but loses precision because we lose the low bits of the constant - * in the shift. Note that 19342813113834067 ~= 2^(20+64) / 1e9. - * For a shift of 24 the error is about 0.5e-9, or about 0.5ns - * over a second. (Shift values are usually 22, 23 or 24.) - * For high frequency clocks such as the 512MHz timebase clock - * on POWER[6789], the mult value is small (e.g. 32768000) - * and so we can shift the constant by 16 initially - * (295147905179 ~= 2^(20+64-16) / 1e9) and then do the - * remaining shifts after the multiplication, which gives a - * more accurate result (e.g. with mult = 32768000, shift = 24, - * the error is only about 1.2e-12, or 0.7ns over 10 minutes). - */ - if (mult <= 62500000 && clock->shift >= 16) - new_tb_to_xs = ((u64) mult * 295147905179ULL) >> (clock->shift - 16); - else - new_tb_to_xs = (u64) mult * (19342813113834067ULL >> clock->shift); - - /* - * Compute the fractional second in units of 2^-32 seconds. - * The fractional second is tk->tkr_mono.xtime_nsec >> tk->tkr_mono.shift - * in nanoseconds, so multiplying that by 2^32 / 1e9 gives - * it in units of 2^-32 seconds. - * We assume shift <= 32 because clocks_calc_mult_shift() - * generates shift values in the range 0 - 32. - */ - frac_sec = tk->tkr_mono.xtime_nsec << (32 - shift); - do_div(frac_sec, NSEC_PER_SEC); - - /* - * Work out new stamp_xsec value for any legacy users of systemcfg. - * stamp_xsec is in units of 2^-20 seconds. - */ - new_stamp_xsec = frac_sec >> 12; - new_stamp_xsec += tk->xtime_sec * XSEC_PER_SEC; - - /* - * tb_update_count is used to allow the userspace gettimeofday code - * to assure itself that it sees a consistent view of the tb_to_xs and - * stamp_xsec variables. It reads the tb_update_count, then reads - * tb_to_xs and stamp_xsec and then reads tb_update_count again. If - * the two values of tb_update_count match and are even then the - * tb_to_xs and stamp_xsec values are consistent. If not, then it - * loops back and reads them again until this criteria is met. - */ - vdso_data->tb_orig_stamp = cycle_last; - vdso_data->stamp_xsec = new_stamp_xsec; - vdso_data->tb_to_xs = new_tb_to_xs; - vdso_data->wtom_clock_sec = tk->wall_to_monotonic.tv_sec; - vdso_data->wtom_clock_nsec = tk->wall_to_monotonic.tv_nsec; - vdso_data->stamp_xtime_sec = xt.tv_sec; - vdso_data->stamp_xtime_nsec = xt.tv_nsec; - vdso_data->stamp_sec_fraction = frac_sec; - vdso_data->hrtimer_res = hrtimer_resolution; - smp_wmb(); - ++(vdso_data->tb_update_count); -} - -void update_vsyscall_tz(void) -{ - vdso_data->tz_minuteswest = sys_tz.tz_minuteswest; - vdso_data->tz_dsttime = sys_tz.tz_dsttime; -} - static void __init clocksource_init(void) { struct clocksource *clock; @@ -1140,8 +1051,7 @@ void __init time_init(void) sys_tz.tz_dsttime = 0; } - vdso_data->tb_update_count = 0; - vdso_data->tb_ticks_per_sec = tb_ticks_per_sec; + vdso_arch_data->tb_ticks_per_sec = tb_ticks_per_sec; /* initialise and enable the large decrementer (if we have one) */ set_decrementer_max(); diff --git a/arch/powerpc/kernel/vdso.c b/arch/powerpc/kernel/vdso.c index eae9ddaecbcf..d1e4f3a3a781 100644 --- a/arch/powerpc/kernel/vdso.c +++ b/arch/powerpc/kernel/vdso.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -71,10 +72,10 @@ static int vdso_ready; * with it, it will become dynamically allocated */ static union { - struct vdso_data data; + struct vdso_arch_data arch_data; u8 page[PAGE_SIZE]; } vdso_data_store __page_aligned_data; -struct vdso_data *vdso_data = &vdso_data_store.data; +struct vdso_arch_data *vdso_arch_data = &vdso_data_store.arch_data; /* Format of the patch table */ struct vdso_patch_def @@ -661,7 +662,7 @@ static void __init vdso_setup_syscall_map(void) 0x80000000UL >> (i & 0x1f); #else /* CONFIG_PPC64 */ if (sys_call_table[i] != sys_ni_syscall) - vdso_data->syscall_map_32[i >> 5] |= + vdso_arch_data->syscall_map_32[i >> 5] |= 0x80000000UL >> (i & 0x1f); #endif /* CONFIG_PPC64 */ } @@ -729,10 +730,10 @@ static int __init vdso_init(void) vdso64_pages = (&vdso64_end - &vdso64_start) >> PAGE_SHIFT; DBG("vdso64_kbase: %p, 0x%x pages\n", vdso64_kbase, vdso64_pages); #else - vdso_data->dcache_block_size = L1_CACHE_BYTES; - vdso_data->dcache_log_block_size = L1_CACHE_SHIFT; - vdso_data->icache_block_size = L1_CACHE_BYTES; - vdso_data->icache_log_block_size = L1_CACHE_SHIFT; + vdso_arch_data->dcache_block_size = L1_CACHE_BYTES; + vdso_arch_data->dcache_log_block_size = L1_CACHE_SHIFT; + vdso_arch_data->icache_block_size = L1_CACHE_BYTES; + vdso_arch_data->icache_log_block_size = L1_CACHE_SHIFT; #endif /* CONFIG_PPC64 */ @@ -775,7 +776,7 @@ static int __init vdso_init(void) get_page(pg); vdso32_pagelist[i] = pg; } - vdso32_pagelist[i++] = virt_to_page(vdso_data); + vdso32_pagelist[i++] = virt_to_page(vdso_arch_data); vdso32_pagelist[i] = NULL; #endif @@ -792,7 +793,7 @@ static int __init vdso_init(void) vdso64_pagelist[i] = NULL; #endif /* CONFIG_PPC64 */ - get_page(virt_to_page(vdso_data)); + get_page(virt_to_page(vdso_arch_data)); smp_wmb(); vdso_ready = 1; diff --git a/arch/powerpc/kernel/vdso32/Makefile b/arch/powerpc/kernel/vdso32/Makefile index 06f54d947057..09edcd1a2dc7 100644 --- a/arch/powerpc/kernel/vdso32/Makefile +++ b/arch/powerpc/kernel/vdso32/Makefile @@ -2,10 +2,17 @@ # List of files in the vdso, has to be asm only for now +ARCH_REL_TYPE_ABS := R_PPC_JUMP_SLOT|R_PPC_GLOB_DAT|R_PPC_ADDR32|R_PPC_ADDR24|R_PPC_ADDR16|R_PPC_ADDR16_LO|R_PPC_ADDR16_HI|R_PPC_ADDR16_HA|R_PPC_ADDR14|R_PPC_ADDR14_BRTAKEN|R_PPC_ADDR14_BRNTAKEN +include $(srctree)/lib/vdso/Makefile + obj-vdso32-$(CONFIG_PPC64) = getcpu.o obj-vdso32 = sigtramp.o gettimeofday.o datapage.o cacheflush.o note.o \ $(obj-vdso32-y) +ifneq ($(c-gettimeofday-y),) + CFLAGS_vgettimeofday.o += -include $(c-gettimeofday-y) +endif + # Build rules ifdef CROSS32_COMPILE @@ -38,8 +45,8 @@ CPPFLAGS_vdso32.lds += -P -C -Upowerpc $(obj)/vdso32_wrapper.o : $(obj)/vdso32.so # link rule for the .so file, .lds has to be first -$(obj)/vdso32.so.dbg: $(src)/vdso32.lds $(obj-vdso32) FORCE - $(call if_changed,vdso32ld) +$(obj)/vdso32.so.dbg: $(src)/vdso32.lds $(obj-vdso32) $(obj)/vgettimeofday.o FORCE + $(call if_changed,vdso32ld_and_check) # strip rule for the .so file $(obj)/%.so: OBJCOPYFLAGS := -S @@ -49,12 +56,16 @@ $(obj)/%.so: $(obj)/%.so.dbg FORCE # assembly rules for the .S files $(obj-vdso32): %.o: %.S FORCE $(call if_changed_dep,vdso32as) +$(obj)/vgettimeofday.o: %.o: %.c FORCE + $(call if_changed_dep,vdso32cc) # actual build commands -quiet_cmd_vdso32ld = VDSO32L $@ - cmd_vdso32ld = $(VDSOCC) $(c_flags) $(CC32FLAGS) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) +quiet_cmd_vdso32ld_and_check = VDSO32L $@ + cmd_vdso32ld_and_check = $(VDSOCC) $(c_flags) $(CC32FLAGS) -o $@ -Wl,-T$(filter %.lds,$^) $(filter %.o,$^) ; $(cmd_vdso_check) quiet_cmd_vdso32as = VDSO32A $@ cmd_vdso32as = $(VDSOCC) $(a_flags) $(CC32FLAGS) -c -o $@ $< +quiet_cmd_vdso32cc = VDSO32A $@ + cmd_vdso32cc = $(VDSOCC) $(c_flags) $(CC32FLAGS) -c -o $@ $< # install commands for the unstripped file quiet_cmd_vdso_install = INSTALL $@ diff --git a/arch/powerpc/kernel/vdso32/gettimeofday.S b/arch/powerpc/kernel/vdso32/gettimeofday.S index d6c1d331e8cb..d767d0089ad0 100644 --- a/arch/powerpc/kernel/vdso32/gettimeofday.S +++ b/arch/powerpc/kernel/vdso32/gettimeofday.S @@ -13,14 +13,32 @@ #include #include -/* Offset for the low 32-bit part of a field of long type */ -#ifdef CONFIG_PPC64 -#define LOPART 4 -#else -#define LOPART 0 -#endif - .text + +.macro cvdso_call_with_fallback funct, syscall + stwu r1, -16(r1) + mflr r0 + stw r0, 20(r1) + stw r3, 8(r1) + stw r4, 12(r1) + get_datapage r5, r0 + bl \funct + lwz r0, 20(r1) + cmpwi r3, 0 + mtlr r0 + bne- 99f + addi r1, r1, 16 + crclr so + blr +99: + lwz r3, 8(r1) + lwz r4, 12(r1) + li r0, \syscall + addi r1, r1, 16 + sc + blr +.endm + /* * Exact prototype of gettimeofday * @@ -29,31 +47,7 @@ */ V_FUNCTION_BEGIN(__kernel_gettimeofday) .cfi_startproc - mflr r12 - .cfi_register lr,r12 - - mr r10,r3 /* r10 saves tv */ - mr r11,r4 /* r11 saves tz */ - get_datapage r9, r0 - cmplwi r10,0 /* check if tv is NULL */ - beq 3f - lis r7,1000000@ha /* load up USEC_PER_SEC */ - addi r7,r7,1000000@l /* so we get microseconds in r4 */ - bl __do_get_tspec@local /* get sec/usec from tb & kernel */ - stw r3,TVAL32_TV_SEC(r10) - stw r4,TVAL32_TV_USEC(r10) - -3: cmplwi r11,0 /* check if tz is NULL */ - beq 1f - lwz r4,CFG_TZ_MINUTEWEST(r9)/* fill tz */ - lwz r5,CFG_TZ_DSTTIME(r9) - stw r4,TZONE_TZ_MINWEST(r11) - stw r5,TZONE_TZ_DSTTIME(r11) - -1: mtlr r12 - crclr cr0*4+so - li r3,0 - blr + cvdso_call_with_fallback __c_kernel_gettimeofday, __NR_gettimeofday .cfi_endproc V_FUNCTION_END(__kernel_gettimeofday) @@ -65,76 +59,7 @@ V_FUNCTION_END(__kernel_gettimeofday) */ V_FUNCTION_BEGIN(__kernel_clock_gettime) .cfi_startproc - /* Check for supported clock IDs */ - cmpli cr0,r3,CLOCK_REALTIME - cmpli cr1,r3,CLOCK_MONOTONIC - cror cr0*4+eq,cr0*4+eq,cr1*4+eq - bne cr0,99f - - mflr r12 /* r12 saves lr */ - .cfi_register lr,r12 - mr r11,r4 /* r11 saves tp */ - get_datapage r9, r0 - lis r7,NSEC_PER_SEC@h /* want nanoseconds */ - ori r7,r7,NSEC_PER_SEC@l -50: bl __do_get_tspec@local /* get sec/nsec from tb & kernel */ - bne cr1,80f /* not monotonic -> all done */ - - /* - * CLOCK_MONOTONIC - */ - - /* now we must fixup using wall to monotonic. We need to snapshot - * that value and do the counter trick again. Fortunately, we still - * have the counter value in r8 that was returned by __do_get_xsec. - * At this point, r3,r4 contain our sec/nsec values, r5 and r6 - * can be used, r7 contains NSEC_PER_SEC. - */ - - lwz r5,(WTOM_CLOCK_SEC+LOPART)(r9) - lwz r6,WTOM_CLOCK_NSEC(r9) - - /* We now have our offset in r5,r6. We create a fake dependency - * on that value and re-check the counter - */ - or r0,r6,r5 - xor r0,r0,r0 - add r9,r9,r0 - lwz r0,(CFG_TB_UPDATE_COUNT+LOPART)(r9) - cmpl cr0,r8,r0 /* check if updated */ - bne- 50b - - /* Calculate and store result. Note that this mimics the C code, - * which may cause funny results if nsec goes negative... is that - * possible at all ? - */ - add r3,r3,r5 - add r4,r4,r6 - cmpw cr0,r4,r7 - cmpwi cr1,r4,0 - blt 1f - subf r4,r7,r4 - addi r3,r3,1 -1: bge cr1,80f - addi r3,r3,-1 - add r4,r4,r7 - -80: stw r3,TSPC32_TV_SEC(r11) - stw r4,TSPC32_TV_NSEC(r11) - - mtlr r12 - crclr cr0*4+so - li r3,0 - blr - - /* - * syscall fallback - */ -99: - li r0,__NR_clock_gettime - .cfi_restore lr - sc - blr + cvdso_call_with_fallback __c_kernel_clock_gettime, __NR_clock_gettime .cfi_endproc V_FUNCTION_END(__kernel_clock_gettime) @@ -147,32 +72,7 @@ V_FUNCTION_END(__kernel_clock_gettime) */ V_FUNCTION_BEGIN(__kernel_clock_getres) .cfi_startproc - /* Check for supported clock IDs */ - cmpwi cr0,r3,CLOCK_REALTIME - cmpwi cr1,r3,CLOCK_MONOTONIC - cror cr0*4+eq,cr0*4+eq,cr1*4+eq - bne cr0,99f - - mflr r12 - .cfi_register lr,r12 - get_datapage r3, r0 - lwz r5, CLOCK_HRTIMER_RES(r3) - mtlr r12 - li r3,0 - cmpli cr0,r4,0 - crclr cr0*4+so - beqlr - stw r3,TSPC32_TV_SEC(r4) - stw r5,TSPC32_TV_NSEC(r4) - blr - - /* - * syscall fallback - */ -99: - li r0,__NR_clock_getres - sc - blr + cvdso_call_with_fallback __c_kernel_clock_getres, __NR_clock_getres .cfi_endproc V_FUNCTION_END(__kernel_clock_getres) @@ -185,104 +85,15 @@ V_FUNCTION_END(__kernel_clock_getres) */ V_FUNCTION_BEGIN(__kernel_time) .cfi_startproc - mflr r12 - .cfi_register lr,r12 - - mr r11,r3 /* r11 holds t */ - get_datapage r9, r0 - - lwz r3,STAMP_XTIME_SEC+LOPART(r9) - - cmplwi r11,0 /* check if t is NULL */ - beq 2f - stw r3,0(r11) /* store result at *t */ -2: mtlr r12 + stwu r1, -16(r1) + mflr r0 + stw r0, 20(r1) + get_datapage r4, r0 + bl __c_kernel_time + lwz r0, 20(r1) crclr cr0*4+so + mtlr r0 + addi r1, r1, 16 blr .cfi_endproc V_FUNCTION_END(__kernel_time) - -/* - * This is the core of clock_gettime() and gettimeofday(), - * it returns the current time in r3 (seconds) and r4. - * On entry, r7 gives the resolution of r4, either USEC_PER_SEC - * or NSEC_PER_SEC, giving r4 in microseconds or nanoseconds. - * It expects the datapage ptr in r9 and doesn't clobber it. - * It clobbers r0, r5 and r6. - * On return, r8 contains the counter value that can be reused. - * This clobbers cr0 but not any other cr field. - */ -__do_get_tspec: - .cfi_startproc - /* Check for update count & load values. We use the low - * order 32 bits of the update count - */ -1: lwz r8,(CFG_TB_UPDATE_COUNT+LOPART)(r9) - andi. r0,r8,1 /* pending update ? loop */ - bne- 1b - xor r0,r8,r8 /* create dependency */ - add r9,r9,r0 - - /* Load orig stamp (offset to TB) */ - lwz r5,CFG_TB_ORIG_STAMP(r9) - lwz r6,(CFG_TB_ORIG_STAMP+4)(r9) - - /* Get a stable TB value */ -2: MFTBU(r3) - MFTBL(r4) - MFTBU(r0) - cmplw cr0,r3,r0 - bne- 2b - - /* Subtract tb orig stamp and shift left 12 bits. - */ - subfc r4,r6,r4 - subfe r0,r5,r3 - slwi r0,r0,12 - rlwimi. r0,r4,12,20,31 - slwi r4,r4,12 - - /* - * Load scale factor & do multiplication. - * We only use the high 32 bits of the tb_to_xs value. - * Even with a 1GHz timebase clock, the high 32 bits of - * tb_to_xs will be at least 4 million, so the error from - * ignoring the low 32 bits will be no more than 0.25ppm. - * The error will just make the clock run very very slightly - * slow until the next time the kernel updates the VDSO data, - * at which point the clock will catch up to the kernel's value, - * so there is no long-term error accumulation. - */ - lwz r5,CFG_TB_TO_XS(r9) /* load values */ - mulhwu r4,r4,r5 - li r3,0 - - beq+ 4f /* skip high part computation if 0 */ - mulhwu r3,r0,r5 - mullw r5,r0,r5 - addc r4,r4,r5 - addze r3,r3 -4: - /* At this point, we have seconds since the xtime stamp - * as a 32.32 fixed-point number in r3 and r4. - * Load & add the xtime stamp. - */ - lwz r5,STAMP_XTIME_SEC+LOPART(r9) - lwz r6,STAMP_SEC_FRAC(r9) - addc r4,r4,r6 - adde r3,r3,r5 - - /* We create a fake dependency on the result in r3/r4 - * and re-check the counter - */ - or r6,r4,r3 - xor r0,r6,r6 - add r9,r9,r0 - lwz r0,(CFG_TB_UPDATE_COUNT+LOPART)(r9) - cmplw cr0,r8,r0 /* check if updated */ - bne- 1b - - mulhwu r4,r4,r7 /* convert to micro or nanoseconds */ - - blr - .cfi_endproc diff --git a/arch/powerpc/kernel/vdso32/vgettimeofday.c b/arch/powerpc/kernel/vdso32/vgettimeofday.c new file mode 100644 index 000000000000..290b4415bf85 --- /dev/null +++ b/arch/powerpc/kernel/vdso32/vgettimeofday.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * ARM64 userspace implementations of gettimeofday() and similar. + * + * Copyright (C) 2018 ARM Limited + * + */ +#include +#include + +int __c_kernel_clock_gettime(clockid_t clock, struct old_timespec32 *ts, + const struct vdso_data *vd) +{ + return __cvdso_clock_gettime32(vd, clock, ts); +} + +int __c_kernel_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz, + const struct vdso_data *vd) +{ + return __cvdso_gettimeofday(vd, tv, tz); +} + +int __c_kernel_clock_getres(clockid_t clock_id, struct old_timespec32 *res, + const struct vdso_data *vd) +{ + return __cvdso_clock_getres_time32(vd, clock_id, res); +} + +time_t __c_kernel_time(time_t *time, const struct vdso_data *vd) +{ + return __cvdso_time(vd, time); +}