From patchwork Mon Jun 18 15:05:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Salyzyn X-Patchwork-Id: 10472157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3E6396053C for ; Mon, 18 Jun 2018 15:49:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D0A0289FA for ; Mon, 18 Jun 2018 15:49:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 218E628A07; Mon, 18 Jun 2018 15:49:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7F80A289FA for ; Mon, 18 Jun 2018 15:49:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=emAp9K7fxZL2z8VvSifOo161XA/VNxsYbQ7KQ/XUwi8=; b=HhAa51HBESj2/Yfay8VKoPSqQZ aCu8apWiWoPu0aZffRLC+LdOiBmh1He5aq5c9hymX+MYPY7UG1Unm2/segpnX2AqvTbnbklIlLr/Q KN3hHujEcWPHmuScGdE1kd1aXNMb38ujksdcJsDdLGZFJyBEFWqTSgc8FoZApj1J2jaZO+AAk1RWs aII9cJBDK7YMTAWg0rS5v1/zV+JXVq8NcJ18dYO9Ikftxdt2/7gkYaGm9cPayLmEKMiK8aUMFIjX2 juyQ9K94XgeragIRB5DckM3Vrau6UccYS9EbQTjPk/5hNQBq/eJ6JHLlQbABzGbyQ5Gl4zcaAg6/j ttv47IrQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fUwPH-0001yJ-U2; Mon, 18 Jun 2018 15:49:27 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fUvvx-0000OV-H4 for linux-arm-kernel@bombadil.infradead.org; Mon, 18 Jun 2018 15:19:09 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=IOCMQs9PDylpUw8XXkDzoKtAKq73cUUVEhSK1X0A1xY=; b=1izecYcGEbb62BhLPZuzyp/Kq OOgZWYPc7CTTYmURCypIljUMQdsTziCbVkMTnKGQe4STHWXRJIG9ItcFk1U8tPUlUeuWWSnXbsIP9 j9T0pC6qoXTHOCsAWITSo2icI7QIoJD6le9R+VK0DYQZy6YcRrK8Ja5Py5CC34FPJVMulFZBAYZ4b vSxRZ2QT03Nc5eaaHrtDeegNQQ0zbLoYCQLDV4NChZKBQC5rw/kQ5178I8DOgdU5fMCCyFoODUQdn EGLbrnb/hG873qwb1fwBXcZbgWa5Co83mNZCjKifYmjzwq5rgireCCKL4zHBc3hvVyBEI08ew6zpK oAsfpjf4w==; Received: from mail-pf0-x242.google.com ([2607:f8b0:400e:c00::242]) by merlin.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fUvkV-0000Bq-K9 for linux-arm-kernel@lists.infradead.org; Mon, 18 Jun 2018 15:07:20 +0000 Received: by mail-pf0-x242.google.com with SMTP id a11-v6so8312167pff.8 for ; Mon, 18 Jun 2018 08:07:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IOCMQs9PDylpUw8XXkDzoKtAKq73cUUVEhSK1X0A1xY=; b=ULkwT27GKAMqXZvDtlzfn6cRYtXUX9A8XoYxQVQngLhtHSGSrs4YMq5TOvIGNf4qMo O9iVYt3MlabauW7Rj2OlLZdnCCyGUqFqfd3uXjoJ6ZGdHdah90zHjQqKarQD/lz0DBh0 MzBER6WxpDy2q1rGoRu1QQ0kvypGXW0ckzwTEuAGNrJMKSKrztbMKDkVd8A6g99uS/ih 6SeXEZeH2ndmcEWqiW/OTgkEh09m3iZxbev5khphWr0IsWb3cBWxSulB+yrwwA82VHFB z/waoFp0m9k6jQBxMVcxYeZ7vxkVOfdvVKq5UuMHqMqRYhgYE8aljfpsJM7+LCnnlGIH Zi5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IOCMQs9PDylpUw8XXkDzoKtAKq73cUUVEhSK1X0A1xY=; b=JfQqfp/7aV1vAaF8sJ7m9JYTyQVkDTtshn3lwxHMj6ELkYGTZmAFCxE2hRPF/mhM52 l2IiCChDXj7KnB9qfJn+BrbdUGG+ajsbg/fOh2TeEzLAgpSK+Ag43svxFZjDpfvHmtXW mgMUgLSj4OhkBKMcOWaduTmOE3QXVZzQHNpD5nVqez1pf8E6OEx9S8nePmUz3wxLuEdf U7LcK2i+iENl08Ou3ulwY5MjhLeacQaUNp26SHE60dpeCmj0vNXt1BvEN+nRhf/rxgag noX1MbuiIPzGBbbEcsMknPyRkOcT8Wzp/Svv2PMZgzidIWorKH8gMfzwGSEL3GgCMuti XXjA== X-Gm-Message-State: APt69E0NuMk350i7fPNa8P18jl4NasJRT1IQT2FRRs4oIiweiPxk/9+9 cqw7J7G+krZ1YhJUZg5ZEJHWLQ== X-Google-Smtp-Source: ADUXVKIrz4lei1Edp3vjnPTPyTrySOi4+GKT5lRs2vU4y+vW8TOY1hQvIZ/SVtqcicmqKt7pgurDvw== X-Received: by 2002:a65:48c9:: with SMTP id o9-v6mr11537521pgs.262.1529334427301; Mon, 18 Jun 2018 08:07:07 -0700 (PDT) Received: from nebulus.mtv.corp.google.com ([2620:0:1000:1611:6077:8eec:bc7e:d0f4]) by smtp.gmail.com with ESMTPSA id i7-v6sm54830660pfa.34.2018.06.18.08.07.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Jun 2018 08:07:06 -0700 (PDT) From: Mark Salyzyn To: linux-kernel@vger.kernel.org Subject: RESEND [PATCH v5 04/12] arm: vdso: do calculations outside reader loops Date: Mon, 18 Jun 2018 08:05:50 -0700 Message-Id: <20180618150613.10322-5-salyzyn@android.com> X-Mailer: git-send-email 2.18.0.rc1.244.gcf134e6275-goog In-Reply-To: <20180618150613.10322-1-salyzyn@android.com> References: <20180618150613.10322-1-salyzyn@android.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180618_110719_697146_9DB47A5F X-CRM114-Status: GOOD ( 23.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Laura Abbott , Kees Cook , Ard Biesheuvel , "Paul E. McKenney" , Catalin Marinas , Kevin Brodsky , Will Deacon , Russell King , Jeremy Linton , Andy Lutomirski , James Morse , Andrew Pinski , Dmitry Safonov , Andy Gross , John Stultz , Thomas Gleixner , Mark Salyzyn , Ingo Molnar , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Take an effort to recode the arm64 vdso code from assembler to C previously submitted by Andrew Pinski , rework it for use in both arm and arm64, overlapping any optimizations for each architecture. But instead of landing it in arm64, land the result into lib/vdso and unify both implementations to simplify future maintenance. In variable timer reading loops, pick up just the values until all are synchronized, then outside of loop pick up cntvct and perform calculations to determine final offset, shifted and multiplied output value. This replaces get_ns with get_clock_shifted_nsec as cntvct reader. Signed-off-by: Mark Salyzyn Cc: James Morse Cc: Russell King Cc: Catalin Marinas Cc: Will Deacon Cc: Andy Lutomirski Cc: Dmitry Safonov Cc: John Stultz Cc: Mark Rutland Cc: Laura Abbott Cc: Kees Cook Cc: Ard Biesheuvel Cc: Andy Gross Cc: Kevin Brodsky Cc: Andrew Pinski Cc: Thomas Gleixner Cc: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Cc: Jeremy Linton v2: - split first CL into 5 of 7 pieces v4: - split into two, moving ARCH_CLOCK_FIXED_MASK to later - update commit message to reflect overall reasoning - adjust for dropping forced inline - replace typeof() with types vdso_wtm_clock_nsec_t, vdso_xtime_clock_sec and __kernel_time_t. v5: - drop including linux/time.h in favour of uapi/linux/time.h in vgettimeofday.c to limit architectural includes. - add linux/time.h to compiler.h for NSEC_PER_SEC definition. - replace last timespec_add_ns with __iter_div_u64_rem. --- arch/arm/include/asm/vdso_datapage.h | 18 +++++- arch/arm/vdso/compiler.h | 1 + arch/arm/vdso/vgettimeofday.c | 90 ++++++++++++++++++---------- 3 files changed, 76 insertions(+), 33 deletions(-) diff --git a/arch/arm/include/asm/vdso_datapage.h b/arch/arm/include/asm/vdso_datapage.h index fa3e1856244d..8dd7303db4ec 100644 --- a/arch/arm/include/asm/vdso_datapage.h +++ b/arch/arm/include/asm/vdso_datapage.h @@ -24,6 +24,16 @@ #include +#ifndef _VDSO_WTM_CLOCK_SEC_T +#define _VDSO_WTM_CLOCK_SEC_T +typedef u32 vdso_wtm_clock_nsec_t; +#endif + +#ifndef _VDSO_XTIME_CLOCK_SEC_T +#define _VDSO_XTIME_CLOCK_SEC_T +typedef u32 vdso_xtime_clock_sec_t; +#endif + /* Try to be cache-friendly on systems that don't implement the * generic timer: fit the unconditionally updated fields in the first * 32 bytes. @@ -35,9 +45,11 @@ struct vdso_data { u32 xtime_coarse_sec; /* coarse time */ u32 xtime_coarse_nsec; - u32 wtm_clock_sec; /* wall to monotonic offset */ - u32 wtm_clock_nsec; - u32 xtime_clock_sec; /* CLOCK_REALTIME - seconds */ + /* wall to monotonic offset */ + u32 wtm_clock_sec; + vdso_wtm_clock_nsec_t wtm_clock_nsec; + /* CLOCK_REALTIME - seconds */ + vdso_xtime_clock_sec_t xtime_clock_sec; u32 cs_mono_mult; /* clocksource multiplier */ u64 cs_cycle_last; /* last cycle value */ diff --git a/arch/arm/vdso/compiler.h b/arch/arm/vdso/compiler.h index af24502797e8..3edddb705a1b 100644 --- a/arch/arm/vdso/compiler.h +++ b/arch/arm/vdso/compiler.h @@ -27,6 +27,7 @@ #include /* for cpu_relax() */ #include #include +#include /* for NSEC_PER_SEC */ #ifndef CONFIG_AEABI #error This code depends on AEABI system call conventions diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 522094b147a2..59893fca03b3 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -24,7 +24,8 @@ #include #include /* for notrace */ -#include +#include /* for __iter_div_u64_rem() */ +#include /* for struct timespec */ #include "compiler.h" #include "datapage.h" @@ -79,6 +80,7 @@ static notrace int do_monotonic_coarse(const struct vdso_data *vd, { struct timespec tomono; u32 seq; + u64 nsec; do { seq = vdso_read_begin(vd); @@ -92,33 +94,41 @@ static notrace int do_monotonic_coarse(const struct vdso_data *vd, } while (vdso_read_retry(vd, seq)); ts->tv_sec += tomono.tv_sec; - timespec_add_ns(ts, tomono.tv_nsec); + /* open coding timespec_add_ns */ + ts->tv_sec += __iter_div_u64_rem(ts->tv_nsec + tomono.tv_nsec, + NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } #ifdef CONFIG_ARM_ARCH_TIMER -static notrace u64 get_ns(const struct vdso_data *vd) +/* + * Returns the clock delta, in nanoseconds left-shifted by the clock + * shift. + */ +static notrace u64 get_clock_shifted_nsec(const u64 cycle_last, + const u32 mult, + const u64 mask) { - u64 cycle_delta; - u64 cycle_now; - u64 nsec; - - cycle_now = arch_vdso_read_counter(); + u64 res; - cycle_delta = (cycle_now - vd->cs_cycle_last) & vd->cs_mask; + /* Read the virtual counter. */ + res = arch_vdso_read_counter(); - nsec = (cycle_delta * vd->cs_mono_mult) + vd->xtime_clock_snsec; - nsec >>= vd->cs_shift; + res = res - cycle_last; - return nsec; + res &= mask; + return res * mult; } static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) { - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; + u64 mask; + vdso_xtime_clock_sec_t sec; do { seq = vdso_read_begin(vd); @@ -126,22 +136,33 @@ static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - } while (vdso_read_retry(vd, seq)); + mult = vd->cs_mono_mult; + shift = vd->cs_shift; + mask = vd->cs_mask; + + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; + + } while (unlikely(vdso_read_retry(vd, seq))); - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs); + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) { - struct timespec tomono; - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; + u64 mask; + vdso_wtm_clock_nsec_t wtm_nsec; + __kernel_time_t sec; do { seq = vdso_read_begin(vd); @@ -149,17 +170,26 @@ static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - tomono.tv_sec = vd->wtm_clock_sec; - tomono.tv_nsec = vd->wtm_clock_nsec; + mult = vd->cs_mono_mult; + shift = vd->cs_shift; + mask = vd->cs_mask; - } while (vdso_read_retry(vd, seq)); + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_sec += tomono.tv_sec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs + tomono.tv_nsec); + sec += vd->wtm_clock_sec; + wtm_nsec = vd->wtm_clock_nsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + nsec += wtm_nsec; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; }