From patchwork Fri Oct 27 22:26:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Salyzyn X-Patchwork-Id: 10030809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 61C0C602D6 for ; Fri, 27 Oct 2017 22:49:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68D5828FF5 for ; Fri, 27 Oct 2017 22:49:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4F06528FF6; Fri, 27 Oct 2017 22:49:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0F90228FF2 for ; Fri, 27 Oct 2017 22:49:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=tFhIwsxJKgimr5tojXoa23iyMmUPbhzG50RE/veH33c=; b=CrL iqy2f8qAMprmJOR57gEPVOj1nmdMLfaTyhbqwBxnBz0umvGlyiGYDoqVgAJawn66Zz7Z7uWnPZ3SC r5RqBitOTwkoRkoLiUrfsKRX2sb/ZALX6lHcszWLFGBhUeI8cEKufQZD1Kl66mtdsR3j4UMpMdSQ/ L10OGhm6ft76mzogUh4Wvw4vtToCmkqMa9W19Pl2P4PS1BTFtB4ZE1pR1660Oh3HRAaFnxrwusnfr Y14DXODHb9V+oXKQ1n6Kw48QFStl/CpAB0GZlcWI6Y62kh+O/WQA9MlnaDoXsvyJ7XM+2FbXJycsq 2AoKM5SasgEo7u0xKrXVn2vJX8XuT2A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e8DRD-0005aD-EL; Fri, 27 Oct 2017 22:49:15 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e8DRB-0005a2-0H for linux-arm-kernel@bombadil.infradead.org; Fri, 27 Oct 2017 22:49:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Message-Id:Date:Subject:Cc:To:From: Sender:Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=JDdhhKBEZ0JLqstsG0kUsUevywunnlTFRkkL/TktHiU=; b=YSnp76UHeox0zfgsfO6zqEXRu wgLjwG1CdEhNvy85Yu2Gg5uTN0sOe7e8GkTzopi/uGbqHdhPtMZmwEvRqhbrKKQbyunx214ExngiT RyOesuc7GPOkfk4Aitt93yYmANHnWOWLBO5KWsA5oLKNUNNbSMvb1mM4XxCPUxuOqJWTYS6V7GL6q 4zIupH4RQQ7fViw/D/2/8YKYA8230s5cCjZ+pdP2PpraWFMJsEfjQ7PbF8uitmr8KqSbAsVJbWwdY 3+F74Xyi5UGlp3w54s+THoq4814YzMHDqV22H4BQR9Hdzcr5/onJDDywqQssInGrMErpj8RrQTkCN Nshjgu7xg==; Received: from mail-io0-x243.google.com ([2607:f8b0:4001:c06::243]) by merlin.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e8D64-0000ME-W0 for linux-arm-kernel@lists.infradead.org; Fri, 27 Oct 2017 22:27:26 +0000 Received: by mail-io0-x243.google.com with SMTP id b186so15752735iof.8 for ; Fri, 27 Oct 2017 15:27:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=JDdhhKBEZ0JLqstsG0kUsUevywunnlTFRkkL/TktHiU=; b=Bo1dB3PMTZwpm7JJZ/3ak/iTenm5/dzPatjUNzdNnmmKTMInrrZh1syuJklSw/Wl9V lv5vE5XM3rk3K/bfumn0KToBnjByQOWimDWoW6cOmklR9ExamuYrjOQHYeRpmuVoyjB2 uJ/UoOhUQD9UV3HYy6D5/+LhT92a4t3t4fpjTMrE69WlZfzPqKgPmgiMpsz7tPFKb1/m 4rCZAIWw5wFzBgxgXsBHcSRGW97O9O5YbuHPDuD9bAkyjbyGhzRszOPXCFpWR+3RRh34 sJBVz4tnQza5vd3RFQm+glCHlcpkE3MVGwIdR2WeWpFpS3CDac+TLxD3QS0pQx7lcQbg WtdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=JDdhhKBEZ0JLqstsG0kUsUevywunnlTFRkkL/TktHiU=; b=LDDDg9qs1u134uGK4Mnd+8Vlhcn9QYHvIdVTBZx3cKiMUXLtH7GjXxHF2hZrT5zM49 uly1ysYU6H+pIo34IwLui5RoYzZTLcHqUk0BWzyYXZ3FfgogrzcBfQgJCOCTEjwAXKod B4HiqAbKJKYw06a4FHDFUHTKyvsWBWc3mc+Ogh3agGhDPVZNX1HHmT6caYvZl30pfDXK +ZaQfD7jFVGChh0wBYpsfO+AmQ/sBmEH0NClcckYEwuMd+rB+fiPL3F/adBBarHnS9CY 1lSuPzPn39fVfVT0BqSJblD+5VGdA9FW2e+rRcIHcL3mVZGO+XIwGykdEy2l8biQNswT pemQ== X-Gm-Message-State: AMCzsaVNbBQ7TA3PVGINQeDslspc/Tw274FSPvLXnwqKYNQyZnigdP93 wzLGrh2/dMKlcefBHes3hzo39A== X-Google-Smtp-Source: ABhQp+Rznt28St0Am2uFFlnJnbyhVf/oIvEYznZlSCmXHdHQeOHtpWbt28nmdyjRDsdbH1hChmOYmQ== X-Received: by 10.36.208.210 with SMTP id m201mr2560455itg.152.1509143222149; Fri, 27 Oct 2017 15:27:02 -0700 (PDT) Received: from nebulus.mtv.corp.google.com ([100.98.120.17]) by smtp.gmail.com with ESMTPSA id 192sm1524988its.26.2017.10.27.15.27.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 27 Oct 2017 15:27:01 -0700 (PDT) From: Mark Salyzyn To: linux-kernel@vger.kernel.org Subject: [PATCH v3 09/12] arm: vdso: move vgettimeofday.c to lib/vdso/ Date: Fri, 27 Oct 2017 15:26:39 -0700 Message-Id: <20171027222657.58241-1-salyzyn@android.com> X-Mailer: git-send-email 2.15.0.rc2.357.g7e34df9404-goog X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Laura Abbott , Kees Cook , Ard Biesheuvel , Catalin Marinas , Kevin Brodsky , Will Deacon , Russell King , Andy Lutomirski , Mark Salyzyn , James Morse , Andrew Pinski , Dmitry Safonov , Andy Gross , John Stultz , Mark Salyzyn , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Declare arch/arm/vdso/vgettimeofday.c to be a candidate for a global implementation of the vdso timer calls. The hope is that new architectures can take advantage of the current unification of arm and arm64 implementations. We urge future efforts to merge their implementations into the global vgettimeofday.c file and thus provide functional parity. Signed-off-by: Mark Salyzyn Cc: James Morse Cc: Russell King Cc: Catalin Marinas Cc: Will Deacon Cc: Andy Lutomirski Cc: Dmitry Safonov Cc: John Stultz Cc: Mark Rutland Cc: Laura Abbott Cc: Kees Cook Cc: Ard Biesheuvel Cc: Andy Gross Cc: Kevin Brodsky Cc: Andrew Pinski Cc: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org v3: - added this change - move arch/arm/vdso/vgettimeofday.c to lib/vdso/vgettimeofday.c - adjust vgettimeofday.c to be a better global candidate, switch to using ARCH_PROVIDES_TIMER and __arch_counter_get() as more generic. --- arch/arm/vdso/compiler.h | 6 +- arch/arm/vdso/vgettimeofday.c | 344 +---------------------------------------- lib/vdso/compiler.h | 24 +++ lib/vdso/datapage.h | 24 +++ lib/vdso/vgettimeofday.c | 345 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 399 insertions(+), 344 deletions(-) create mode 100644 lib/vdso/compiler.h create mode 100644 lib/vdso/datapage.h create mode 100644 lib/vdso/vgettimeofday.c diff --git a/arch/arm/vdso/compiler.h b/arch/arm/vdso/compiler.h index ef1d664fb4fa..f55962d8a683 100644 --- a/arch/arm/vdso/compiler.h +++ b/arch/arm/vdso/compiler.h @@ -32,6 +32,10 @@ #error This code depends on AEABI system call conventions #endif +#ifdef CONFIG_ARM_ARCH_TIMER +#define ARCH_PROVIDES_TIMER +#endif + #define DEFINE_FALLBACK(name, type_arg1, name_arg1, type_arg2, name_arg2) \ static notrace long name##_fallback(type_arg1 _##name_arg1, \ type_arg2 _##name_arg2) \ @@ -50,7 +54,7 @@ static notrace long name##_fallback(type_arg1 _##name_arg1, \ return ret; \ } -#define __arch_counter_get_cntvct() arch_counter_get_cntvct() +#define __arch_counter_get() arch_counter_get_cntvct() /* Avoid unresolved references emitted by GCC */ diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 841e7fd74a1b..4b241fe60d17 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -1,345 +1,3 @@ -/* - * Userspace implementations of gettimeofday() and friends. - * - * Copyright (C) 2017 Cavium, Inc. - * Copyright (C) 2015 Mentor Graphics Corporation - * Copyright (C) 2012 ARM Limited - * - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License version 2 as - * published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program. If not, see . - * - * Author: Will Deacon - * Rewriten from arch64 version into C by: Andrew Pinski - * Reworked and rebased over arm version by: Mark Salyzyn - */ - -#include -#include -#include - #include "compiler.h" #include "datapage.h" - -DEFINE_FALLBACK(gettimeofday, struct timeval *, tv, struct timezone *, tz) -DEFINE_FALLBACK(clock_gettime, clockid_t, clock, struct timespec *, ts) -DEFINE_FALLBACK(clock_getres, clockid_t, clock, struct timespec *, ts) - -static notrace u32 vdso_read_begin(const struct vdso_data *vd) -{ - u32 seq; - - do { - seq = READ_ONCE(vd->tb_seq_count); - - if ((seq & 1) == 0) - break; - - cpu_relax(); - } while (true); - - smp_rmb(); /* Pairs with second smp_wmb in update_vsyscall */ - return seq; -} - -static notrace int vdso_read_retry(const struct vdso_data *vd, u32 start) -{ - u32 seq; - - smp_rmb(); /* Pairs with first smp_wmb in update_vsyscall */ - seq = READ_ONCE(vd->tb_seq_count); - return seq != start; -} - -static notrace int do_realtime_coarse(const struct vdso_data *vd, - struct timespec *ts) -{ - u32 seq; - - do { - seq = vdso_read_begin(vd); - - ts->tv_sec = vd->xtime_coarse_sec; - ts->tv_nsec = vd->xtime_coarse_nsec; - - } while (vdso_read_retry(vd, seq)); - - return 0; -} - -static notrace int do_monotonic_coarse(const struct vdso_data *vd, - struct timespec *ts) -{ - struct timespec tomono; - u32 seq; - - do { - seq = vdso_read_begin(vd); - - ts->tv_sec = vd->xtime_coarse_sec; - ts->tv_nsec = vd->xtime_coarse_nsec; - - tomono.tv_sec = vd->wtm_clock_sec; - tomono.tv_nsec = vd->wtm_clock_nsec; - - } while (vdso_read_retry(vd, seq)); - - ts->tv_sec += tomono.tv_sec; - timespec_add_ns(ts, tomono.tv_nsec); - - return 0; -} - -#ifdef CONFIG_ARM_ARCH_TIMER - -/* - * Returns the clock delta, in nanoseconds left-shifted by the clock - * shift. - */ -static __always_inline notrace u64 get_clock_shifted_nsec(const u64 cycle_last, - const u32 mult, - const u64 mask) -{ - u64 res; - - /* Read the virtual counter. */ - res = __arch_counter_get_cntvct(); - - res = res - cycle_last; - - res &= mask; - return res * mult; -} - -/* Code size doesn't matter (vdso is 4k/16k/64k anyway) and this is faster. */ -static __always_inline notrace int do_realtime(const struct vdso_data *vd, - struct timespec *ts) -{ - u32 seq, mult, shift; - u64 nsec, cycle_last; -#ifdef ARCH_CLOCK_FIXED_MASK - static const u64 mask = ARCH_CLOCK_FIXED_MASK; -#else - u64 mask; -#endif - - typeof(((struct vdso_data *)vd)->xtime_clock_sec) sec; - - do { - seq = vdso_read_begin(vd); - - if (vd->use_syscall) - return -1; - - cycle_last = vd->cs_cycle_last; - - mult = vd->cs_mono_mult; - shift = vd->cs_shift; -#ifndef ARCH_CLOCK_FIXED_MASK - mask = vd->cs_mask; -#endif - - sec = vd->xtime_clock_sec; - nsec = vd->xtime_clock_snsec; - - } while (unlikely(vdso_read_retry(vd, seq))); - - nsec += get_clock_shifted_nsec(cycle_last, mult, mask); - nsec >>= shift; - /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ - ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); - ts->tv_nsec = nsec; - - return 0; -} - -static __always_inline notrace int do_monotonic(const struct vdso_data *vd, - struct timespec *ts) -{ - u32 seq, mult, shift; - u64 nsec, cycle_last; -#ifdef ARCH_CLOCK_FIXED_MASK - static const u64 mask = ARCH_CLOCK_FIXED_MASK; -#else - u64 mask; -#endif - - typeof(((struct vdso_data *)vd)->wtm_clock_nsec) wtm_nsec; - typeof(ts->tv_sec) sec; - - do { - seq = vdso_read_begin(vd); - - if (vd->use_syscall) - return -1; - - cycle_last = vd->cs_cycle_last; - - mult = vd->cs_mono_mult; - shift = vd->cs_shift; -#ifndef ARCH_CLOCK_FIXED_MASK - mask = vd->cs_mask; -#endif - - sec = vd->xtime_clock_sec; - nsec = vd->xtime_clock_snsec; - - sec += vd->wtm_clock_sec; - wtm_nsec = vd->wtm_clock_nsec; - - } while (unlikely(vdso_read_retry(vd, seq))); - - nsec += get_clock_shifted_nsec(cycle_last, mult, mask); - nsec >>= shift; - nsec += wtm_nsec; - /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ - ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); - ts->tv_nsec = nsec; - - return 0; -} - -static __always_inline notrace int do_monotonic_raw(const struct vdso_data *vd, - struct timespec *ts) -{ - u32 seq, mult, shift; - u64 nsec, cycle_last; -#ifdef ARCH_CLOCK_FIXED_MASK - static const u64 mask = ARCH_CLOCK_FIXED_MASK; -#else - u64 mask; -#endif - - typeof(((struct vdso_data *)vd)->raw_time_sec) sec; - - do { - seq = vdso_read_begin(vd); - - if (vd->use_syscall) - return -1; - - cycle_last = vd->cs_cycle_last; - - mult = vd->cs_raw_mult; - shift = vd->cs_shift; -#ifndef ARCH_CLOCK_FIXED_MASK - mask = vd->cs_mask; -#endif - - sec = vd->raw_time_sec; - nsec = vd->raw_time_nsec; - - } while (unlikely(vdso_read_retry(vd, seq))); - - nsec += get_clock_shifted_nsec(cycle_last, mult, mask); - nsec >>= shift; - /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ - ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); - ts->tv_nsec = nsec; - - return 0; -} - -#else /* CONFIG_ARM_ARCH_TIMER */ - -static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) -{ - return -1; -} - -static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) -{ - return -1; -} - -static notrace int do_monotonic_raw(const struct vdso_data *vd, - struct timespec *ts) -{ - return -1; -} - -#endif /* CONFIG_ARM_ARCH_TIMER */ - -notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) -{ - const struct vdso_data *vd = __get_datapage(); - - switch (clock) { - case CLOCK_REALTIME_COARSE: - do_realtime_coarse(vd, ts); - break; - case CLOCK_MONOTONIC_COARSE: - do_monotonic_coarse(vd, ts); - break; - case CLOCK_REALTIME: - if (do_realtime(vd, ts)) - goto fallback; - break; - case CLOCK_MONOTONIC: - if (do_monotonic(vd, ts)) - goto fallback; - break; - case CLOCK_MONOTONIC_RAW: - if (do_monotonic_raw(vd, ts)) - goto fallback; - break; - default: - goto fallback; - } - - return 0; -fallback: - return clock_gettime_fallback(clock, ts); -} - -notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz) -{ - const struct vdso_data *vd = __get_datapage(); - - if (likely(tv != NULL)) { - struct timespec ts; - - if (do_realtime(vd, &ts)) - return gettimeofday_fallback(tv, tz); - - tv->tv_sec = ts.tv_sec; - tv->tv_usec = ts.tv_nsec / 1000; - } - - if (unlikely(tz != NULL)) { - tz->tz_minuteswest = vd->tz_minuteswest; - tz->tz_dsttime = vd->tz_dsttime; - } - - return 0; -} - -int __vdso_clock_getres(clockid_t clock_id, struct timespec *res) -{ - typeof(res->tv_nsec) nsec; - - if (clock_id == CLOCK_REALTIME || - clock_id == CLOCK_MONOTONIC || - clock_id == CLOCK_MONOTONIC_RAW) - nsec = MONOTONIC_RES_NSEC; - else if (clock_id == CLOCK_REALTIME_COARSE || - clock_id == CLOCK_MONOTONIC_COARSE) - nsec = LOW_RES_NSEC; - else - return clock_getres_fallback(clock_id, res); - - if (likely(res != NULL)) { - res->tv_sec = 0; - res->tv_nsec = nsec; - } - - return 0; -} +#include "../../../lib/vdso/vgettimeofday.c" diff --git a/lib/vdso/compiler.h b/lib/vdso/compiler.h new file mode 100644 index 000000000000..0e618b73e064 --- /dev/null +++ b/lib/vdso/compiler.h @@ -0,0 +1,24 @@ +/* + * Userspace implementations of fallback calls + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#ifndef __VDSO_COMPILER_H +#define __VDSO_COMPILER_H + +#error "vdso: Provide architectural overrides such as ARCH_PROVIDES_TIMER," +#error " DEFINE_FALLBACK and __arch_counter_get or any overrides. eg:" +#error " vdso entry points or compilation time helpers." + +#endif /* __VDSO_COMPILER_H */ diff --git a/lib/vdso/datapage.h b/lib/vdso/datapage.h new file mode 100644 index 000000000000..df4427e42d51 --- /dev/null +++ b/lib/vdso/datapage.h @@ -0,0 +1,24 @@ +/* + * Userspace implementations of __get_datapage + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#ifndef __VDSO_DATAPAGE_H +#define __VDSO_DATAPAGE_H + +#error "vdso: Provide a user space architecture specific definition or" +#error "prototype for struct vdso_data *__get_datapage(void). Also define" +#error "ARCH_CLOCK_FIXED_MASK if not provided by cs_mask." + +#endif /* __VDSO_DATAPAGE_H */ diff --git a/lib/vdso/vgettimeofday.c b/lib/vdso/vgettimeofday.c new file mode 100644 index 000000000000..2cafd732c65f --- /dev/null +++ b/lib/vdso/vgettimeofday.c @@ -0,0 +1,345 @@ +/* + * Userspace implementations of gettimeofday() and friends. + * + * Copyright (C) 2017 Cavium, Inc. + * Copyright (C) 2015 Mentor Graphics Corporation + * Copyright (C) 2012 ARM Limited + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + * + * Author: Will Deacon + * Rewriten from arch64 version into C by: Andrew Pinski + * Reworked and rebased over arm version by: Mark Salyzyn + */ + +#include +#include +#include + +#include "compiler.h" +#include "datapage.h" + +DEFINE_FALLBACK(gettimeofday, struct timeval *, tv, struct timezone *, tz) +DEFINE_FALLBACK(clock_gettime, clockid_t, clock, struct timespec *, ts) +DEFINE_FALLBACK(clock_getres, clockid_t, clock, struct timespec *, ts) + +static notrace u32 vdso_read_begin(const struct vdso_data *vd) +{ + u32 seq; + + do { + seq = READ_ONCE(vd->tb_seq_count); + + if ((seq & 1) == 0) + break; + + cpu_relax(); + } while (true); + + smp_rmb(); /* Pairs with second smp_wmb in update_vsyscall */ + return seq; +} + +static notrace int vdso_read_retry(const struct vdso_data *vd, u32 start) +{ + u32 seq; + + smp_rmb(); /* Pairs with first smp_wmb in update_vsyscall */ + seq = READ_ONCE(vd->tb_seq_count); + return seq != start; +} + +static notrace int do_realtime_coarse(const struct vdso_data *vd, + struct timespec *ts) +{ + u32 seq; + + do { + seq = vdso_read_begin(vd); + + ts->tv_sec = vd->xtime_coarse_sec; + ts->tv_nsec = vd->xtime_coarse_nsec; + + } while (vdso_read_retry(vd, seq)); + + return 0; +} + +static notrace int do_monotonic_coarse(const struct vdso_data *vd, + struct timespec *ts) +{ + struct timespec tomono; + u32 seq; + + do { + seq = vdso_read_begin(vd); + + ts->tv_sec = vd->xtime_coarse_sec; + ts->tv_nsec = vd->xtime_coarse_nsec; + + tomono.tv_sec = vd->wtm_clock_sec; + tomono.tv_nsec = vd->wtm_clock_nsec; + + } while (vdso_read_retry(vd, seq)); + + ts->tv_sec += tomono.tv_sec; + timespec_add_ns(ts, tomono.tv_nsec); + + return 0; +} + +#ifdef ARCH_PROVIDES_TIMER + +/* + * Returns the clock delta, in nanoseconds left-shifted by the clock + * shift. + */ +static __always_inline notrace u64 get_clock_shifted_nsec(const u64 cycle_last, + const u32 mult, + const u64 mask) +{ + u64 res; + + /* Read the virtual counter. */ + res = __arch_counter_get(); + + res = res - cycle_last; + + res &= mask; + return res * mult; +} + +/* Code size doesn't matter (vdso is 4k/16k/64k anyway) and this is faster. */ +static __always_inline notrace int do_realtime(const struct vdso_data *vd, + struct timespec *ts) +{ + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->xtime_clock_sec) sec; + + do { + seq = vdso_read_begin(vd); + + if (vd->use_syscall) + return -1; + + cycle_last = vd->cs_cycle_last; + + mult = vd->cs_mono_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif + + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; + + return 0; +} + +static __always_inline notrace int do_monotonic(const struct vdso_data *vd, + struct timespec *ts) +{ + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->wtm_clock_nsec) wtm_nsec; + typeof(ts->tv_sec) sec; + + do { + seq = vdso_read_begin(vd); + + if (vd->use_syscall) + return -1; + + cycle_last = vd->cs_cycle_last; + + mult = vd->cs_mono_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif + + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; + + sec += vd->wtm_clock_sec; + wtm_nsec = vd->wtm_clock_nsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + nsec += wtm_nsec; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; + + return 0; +} + +static __always_inline notrace int do_monotonic_raw(const struct vdso_data *vd, + struct timespec *ts) +{ + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->raw_time_sec) sec; + + do { + seq = vdso_read_begin(vd); + + if (vd->use_syscall) + return -1; + + cycle_last = vd->cs_cycle_last; + + mult = vd->cs_raw_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif + + sec = vd->raw_time_sec; + nsec = vd->raw_time_nsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; + + return 0; +} + +#else /* ARCH_PROVIDES_TIMER */ + +static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) +{ + return -1; +} + +static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) +{ + return -1; +} + +static notrace int do_monotonic_raw(const struct vdso_data *vd, + struct timespec *ts) +{ + return -1; +} + +#endif /* ARCH_PROVIDES_TIMER */ + +notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) +{ + const struct vdso_data *vd = __get_datapage(); + + switch (clock) { + case CLOCK_REALTIME_COARSE: + do_realtime_coarse(vd, ts); + break; + case CLOCK_MONOTONIC_COARSE: + do_monotonic_coarse(vd, ts); + break; + case CLOCK_REALTIME: + if (do_realtime(vd, ts)) + goto fallback; + break; + case CLOCK_MONOTONIC: + if (do_monotonic(vd, ts)) + goto fallback; + break; + case CLOCK_MONOTONIC_RAW: + if (do_monotonic_raw(vd, ts)) + goto fallback; + break; + default: + goto fallback; + } + + return 0; +fallback: + return clock_gettime_fallback(clock, ts); +} + +notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz) +{ + const struct vdso_data *vd = __get_datapage(); + + if (likely(tv != NULL)) { + struct timespec ts; + + if (do_realtime(vd, &ts)) + return gettimeofday_fallback(tv, tz); + + tv->tv_sec = ts.tv_sec; + tv->tv_usec = ts.tv_nsec / 1000; + } + + if (unlikely(tz != NULL)) { + tz->tz_minuteswest = vd->tz_minuteswest; + tz->tz_dsttime = vd->tz_dsttime; + } + + return 0; +} + +int __vdso_clock_getres(clockid_t clock_id, struct timespec *res) +{ + typeof(res->tv_nsec) nsec; + + if (clock_id == CLOCK_REALTIME || + clock_id == CLOCK_MONOTONIC || + clock_id == CLOCK_MONOTONIC_RAW) + nsec = MONOTONIC_RES_NSEC; + else if (clock_id == CLOCK_REALTIME_COARSE || + clock_id == CLOCK_MONOTONIC_COARSE) + nsec = LOW_RES_NSEC; + else + return clock_getres_fallback(clock_id, res); + + if (likely(res != NULL)) { + res->tv_sec = 0; + res->tv_nsec = nsec; + } + + return 0; +}