From patchwork Thu Oct 12 23:00:08 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Salyzyn X-Patchwork-Id: 10003159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 84475602BF for ; Thu, 12 Oct 2017 23:01:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 755B82876B for ; Thu, 12 Oct 2017 23:01:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6A47D28869; Thu, 12 Oct 2017 23:01:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EA29D2876B for ; Thu, 12 Oct 2017 23:01:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=p6wOKy4Lw6N9MSaZp/DZhs36XB6oQ379QJJ50RhZGC0=; b=FIl 6+4rwgtrirPSbtQda4lIpsAXQaMoFFFJVucG1QfCV1wbOI0iH1oaQLxSYUUSPjwJ4ixFhM2UXVUEn NSNq16BBoGN7ttDKjByp5S0lSVsZMfnYCPfNMkqKQP/yFiYAnFBv/lg3F6RYE0pe6zqBwsCy9cABL u/4zg9KAOWvK4jlhkE3UgFdVcf0i55KI2JwasRyuLOpq4GIRlgFgUBRVSqCralOLwReI9OV0TPRnR axHOdnUYhjI6IsDe5YfmLicBHBl1Ec+1Kxfcafmntemtvwv1izSXANdmzftODwwmJYQIVnn6XrJED Kby4xrZtxd3wEGf333kIZdEAYE+/Adw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e2mTm-00066u-Vz; Thu, 12 Oct 2017 23:01:27 +0000 Received: from mail-pf0-x232.google.com ([2607:f8b0:400e:c00::232]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e2mT1-00058v-HJ for linux-arm-kernel@lists.infradead.org; Thu, 12 Oct 2017 23:01:11 +0000 Received: by mail-pf0-x232.google.com with SMTP id m28so7021359pfi.11 for ; Thu, 12 Oct 2017 16:00:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=qg8FxpiHr+t4ubsbVb1ks28ZiMwvthlucI4DYLoT63U=; b=ZUlM8UVpNmT2Sq7AojikQaRO4/uwlr2w/oLohoH1oGPfe9AB9R2P2QIQ6vUFEdoi4f Q6MtC/6cZsJVCkT6fpSYO4erWWzpoaVkE0xYMr+Dg7hhQhKqalFzKHTEhonZh2RSMSr3 kCN7ZBOPfgba36a8E6YVDDZ6SIEKx5A47zohwVYlwFfHw2W5ZQo2dp1GeIzIngyMdZRg I1J1kIPaweCYLsAp7LcDKOg2QFCpZ+dSgr1GLpjDLBExvREsESQelngn8BH33qFZaVaJ eesMUnM3ApPXXiMtcl4HzC+WXugQvH+TgtKjTivniF09FG4f27lIxWwoZiS3kGPh/A4g AaZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=qg8FxpiHr+t4ubsbVb1ks28ZiMwvthlucI4DYLoT63U=; b=mTVazK78AOjy4q4nT4eJyOqrgPckjUEFah99+fWz9NGMRO6zmWPHt1M8DN42reRjsk hNSabYgCXMkEDyGlxM8CVD6/TkITT47rJLvB0DCSqd0L5tgk/kHRnyiFj9Wfibl9lkoZ g+fWD34RJ0ixRuIutOXs6aSX3Ne8UmoRA+6lGByBLCP1x1OjMlkQYxTRpuOEgWTzYcc4 1Y1M4J57eef4A4DrSOcREPeAMwXA6Yy3C7xJzhvzzLrCKGQoCLWW73rvtbyZkyqYym6E c8aa3NqSaLHHk6wJLIn6EwLDTwwYhCfRUE1GFczcnUDuy07GUnuOG7PN5C+PWb9w4Fzn 6v+A== X-Gm-Message-State: AMCzsaVZvI7WaObpFQI1m12/vEglxKd8QdODiRY7nUbMTo2ErEwWYn8U YcHbGNdnpm3xkpppRnReJytdEA== X-Google-Smtp-Source: AOwi7QDujX0rr5JkiO8Fc4wDYF0JSCakAhGw00EOpXiSZDyXAIvDb9nZl5T4fqC1BdMqpzqxzqBC/A== X-Received: by 10.98.53.196 with SMTP id c187mr3496704pfa.38.1507849218340; Thu, 12 Oct 2017 16:00:18 -0700 (PDT) Received: from nebulus.mtv.corp.google.com ([100.98.120.17]) by smtp.gmail.com with ESMTPSA id x1sm27161757pfk.131.2017.10.12.16.00.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 12 Oct 2017 16:00:17 -0700 (PDT) From: Mark Salyzyn To: linux-kernel@vger.kernel.org Subject: [PATCH v2 5/10] arm: vdso: do calculations outside reader loops Date: Thu, 12 Oct 2017 16:00:08 -0700 Message-Id: <20171012230012.86812-1-salyzyn@android.com> X-Mailer: git-send-email 2.15.0.rc0.271.g36b669edcc-goog X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171012_160040_034014_FBAE07A4 X-CRM114-Status: GOOD ( 13.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arm-kernel@lists.infradead.org, james.morse@arm.com, Russell King , Mark Salyzyn , Mark Salyzyn MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP In variable timer reading loops, pick up just the values until all are synchronized, then outside of loop pick up cntvct and perform calculations to determine final offset, shifted and multiplied output value. This replaces get_ns with get_clock_shifted_nsec as cntvct reader. Signed-off-by: Mark Salyzyn v2: split first CL into 5 of 7 pieces --- arch/arm/vdso/vgettimeofday.c | 95 ++++++++++++++++++++++++++++++------------- 1 file changed, 67 insertions(+), 28 deletions(-) diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index 71003a1997c4..7fcc8cfcb7df 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -99,28 +99,38 @@ static notrace int do_monotonic_coarse(const struct vdso_data *vd, #ifdef CONFIG_ARM_ARCH_TIMER -static __always_inline notrace u64 get_ns(const struct vdso_data *vd) +/* + * Returns the clock delta, in nanoseconds left-shifted by the clock + * shift. + */ +static __always_inline notrace u64 get_clock_shifted_nsec(const u64 cycle_last, + const u32 mult, + const u64 mask) { - u64 cycle_delta; - u64 cycle_now; - u64 nsec; - - cycle_now = __arch_counter_get_cntvct(); + u64 res; - cycle_delta = (cycle_now - vd->cs_cycle_last) & vd->cs_mask; + /* Read the virtual counter. */ + res = __arch_counter_get_cntvct(); - nsec = (cycle_delta * vd->cs_mono_mult) + vd->xtime_clock_snsec; - nsec >>= vd->cs_shift; + res = res - cycle_last; - return nsec; + res &= mask; + return res * mult; } /* Code size doesn't matter (vdso is 4k/16k/64k anyway) and this is faster. */ static __always_inline notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) { - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->xtime_clock_sec) sec; do { seq = vdso_read_begin(vd); @@ -128,13 +138,24 @@ static __always_inline notrace int do_realtime(const struct vdso_data *vd, if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - } while (vdso_read_retry(vd, seq)); + mult = vd->cs_mono_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif + + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs); + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } @@ -142,9 +163,16 @@ static __always_inline notrace int do_realtime(const struct vdso_data *vd, static __always_inline notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) { - struct timespec tomono; - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; +#ifdef ARCH_CLOCK_FIXED_MASK + static const u64 mask = ARCH_CLOCK_FIXED_MASK; +#else + u64 mask; +#endif + + typeof(((struct vdso_data *)vd)->wtm_clock_nsec) wtm_nsec; + typeof(ts->tv_sec) sec; do { seq = vdso_read_begin(vd); @@ -152,17 +180,28 @@ static __always_inline notrace int do_monotonic(const struct vdso_data *vd, if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - tomono.tv_sec = vd->wtm_clock_sec; - tomono.tv_nsec = vd->wtm_clock_nsec; + mult = vd->cs_mono_mult; + shift = vd->cs_shift; +#ifndef ARCH_CLOCK_FIXED_MASK + mask = vd->cs_mask; +#endif - } while (vdso_read_retry(vd, seq)); + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_sec += tomono.tv_sec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs + tomono.tv_nsec); + sec += vd->wtm_clock_sec; + wtm_nsec = vd->wtm_clock_nsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + nsec += wtm_nsec; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; }