From patchwork Tue Oct 31 18:31:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Salyzyn X-Patchwork-Id: 10035159 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 572F5602B9 for ; Tue, 31 Oct 2017 18:35:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4AF7C28A86 for ; Tue, 31 Oct 2017 18:35:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C4F228A95; Tue, 31 Oct 2017 18:35:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4C36F28A86 for ; Tue, 31 Oct 2017 18:35:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=nRNorJ+SnVssS1sMnVMkWPgrFn2ZC6O1M/PXi9VbTVg=; b=rlz OiqRv2DDcF2OA+1Qou4ocTe3lcqyRc/vPzwhHdEJfAmWQOEW8ccPkIyagxCIg3CdhJrzDtOB7F1OW Vmi8W0GIIQ1PcNKuZSpBm8wDFQ2sq1P+NqhCIH+XNuymblDQaIsq5/H7mlVLvD326TBCUnSaN0rNm klA7FBZh1EpPnamEI6QhFs7ze4Q45dZ1bLjVc0PetksZKYT50tRBKO2/p+166W1LeV9wpZURXJvd6 l9s7sA5G1K2o9ij/hYFxWL7vhnaQdChT2B842wHQRtnZ3quM87ZedLQWVDexOOYpRDO3Q1w9GlNRO nD0+Kn28P9tSDJ9ARqkWcQU3Kv8D9ew==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e9bNj-0001gz-LZ; Tue, 31 Oct 2017 18:35:23 +0000 Received: from mail-io0-x243.google.com ([2607:f8b0:4001:c06::243]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e9bKL-0005v4-OH for linux-arm-kernel@lists.infradead.org; Tue, 31 Oct 2017 18:32:00 +0000 Received: by mail-io0-x243.google.com with SMTP id 134so900367ioo.0 for ; Tue, 31 Oct 2017 11:31:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=j85hutsjoiBgdKG5nrBfkDo27fnXlYU7BkDNVxssFgY=; b=uRF2PpB7BGwtlKG2Jn7/G5WXqPsZwYOGxuGMj815ERg6ELUezubSllrgjHzF9+3X2/ 5nyTw1sgJvAws33Gan8RPnu0ovIeYXop6Y66/h4EmrrN8PdcAUhxIU1KX+P346x8uK0A EP80fDhwahs5G0Ni1cNrLrZnO73/9+OklXd/gZaOKyLaCt88ZxTYrkrUCNRASz3GLt7g QQzAe5KH9DYgxPlIju7wbdpC7bsWWEYTOilH3HIkBbaBWYcaI9Ecr3Ahjq8HzMnaE9P8 W5QzHZ5Grap620pcOOc7tXZnsh2K9WPdUBP1McErdWiYaqU7kOyD32mG+YydcZx0ebFA wStA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=j85hutsjoiBgdKG5nrBfkDo27fnXlYU7BkDNVxssFgY=; b=mJlW+pw3M/PafLqGRLi5XQi8aWKgJ6cPZ0+uZ6viiUJbq8rlkfiKNMZEmv7hTHMeII dhqin32LUja+jXruDWql5na9kkxBFZ8Q1oqL5BS/NBkwFUTNozDoKyLScvW1gDgZq4Z3 tNL90fg8/vzQnJRN1scKvsYAxhPbM3P1bPaNK4NvQSBdloWDuV6WEFAzbnBshwN3sWM3 eFoxaVQ+B9M+ExZT8zAsYw1Nr0Do/c4f8x5nenNbEG5+OvioTKEE/GftLV66yFa/zDQ+ xbsDZVKzLaHHsK0TmWro7ZR5fajvXt8B0Cx+kU8HSRu9xzo/hByq0A+83YeyuDxJI08r +zIA== X-Gm-Message-State: AMCzsaXZDFoRWQVtVgxHZ3wQI/GmbAPkQRCg7L8RmhFH9KFpS1ZrQWGi R44rEvxwfldOPLrWXLUxF50JRA== X-Google-Smtp-Source: ABhQp+Swk/l+ppO71mS8Swo8NjBKM/3iQlznT6ZVTbGNqouKm8asrTETLpjdB0lDqesGO3FE0qeRpw== X-Received: by 10.107.135.166 with SMTP id r38mr3797617ioi.171.1509474691400; Tue, 31 Oct 2017 11:31:31 -0700 (PDT) Received: from nebulus.mtv.corp.google.com ([100.98.120.17]) by smtp.gmail.com with ESMTPSA id a133sm925391ioa.60.2017.10.31.11.31.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 31 Oct 2017 11:31:30 -0700 (PDT) From: Mark Salyzyn To: linux-kernel@vger.kernel.org Subject: [PATCH v4 04/12] arm: vdso: do calculations outside reader loops Date: Tue, 31 Oct 2017 11:31:23 -0700 Message-Id: <20171031183127.81024-1-salyzyn@android.com> X-Mailer: git-send-email 2.15.0.rc2.357.g7e34df9404-goog X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171031_113153_926240_EF13A0F2 X-CRM114-Status: GOOD ( 17.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Laura Abbott , Kees Cook , Ard Biesheuvel , Catalin Marinas , Kevin Brodsky , Will Deacon , Russell King , Andy Lutomirski , James Morse , Andrew Pinski , Dmitry Safonov , Andy Gross , John Stultz , Thomas Gleixner , Mark Salyzyn , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Take an effort to recode the arm64 vdso code from assembler to C previously submitted by Andrew Pinski , rework it for use in both arm and arm64, overlapping any optimizations for each architecture. But instead of landing it in arm64, land the result into lib/vdso and unify both implementations to simplify future maintenance. In variable timer reading loops, pick up just the values until all are synchronized, then outside of loop pick up cntvct and perform calculations to determine final offset, shifted and multiplied output value. This replaces get_ns with get_clock_shifted_nsec as cntvct reader. Signed-off-by: Mark Salyzyn Cc: James Morse Cc: Russell King Cc: Catalin Marinas Cc: Will Deacon Cc: Andy Lutomirski Cc: Dmitry Safonov Cc: John Stultz Cc: Mark Rutland Cc: Laura Abbott Cc: Kees Cook Cc: Ard Biesheuvel Cc: Andy Gross Cc: Kevin Brodsky Cc: Andrew Pinski Cc: Thomas Gleixner Cc: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org v2: - split first CL into 5 of 7 pieces v4: - split into two, moving ARCH_CLOCK_FIXED_MASK to later - update commit message to reflect overall reasoning - adjust for dropping forced inline - replace typeof() with types vdso_wtm_clock_nsec_t, vdso_xtime_clock_sec and __kernel_time_t. --- arch/arm/include/asm/vdso_datapage.h | 18 ++++++-- arch/arm/vdso/vgettimeofday.c | 81 +++++++++++++++++++++++------------- 2 files changed, 68 insertions(+), 31 deletions(-) diff --git a/arch/arm/include/asm/vdso_datapage.h b/arch/arm/include/asm/vdso_datapage.h index fa3e1856244d..8dd7303db4ec 100644 --- a/arch/arm/include/asm/vdso_datapage.h +++ b/arch/arm/include/asm/vdso_datapage.h @@ -24,6 +24,16 @@ #include +#ifndef _VDSO_WTM_CLOCK_SEC_T +#define _VDSO_WTM_CLOCK_SEC_T +typedef u32 vdso_wtm_clock_nsec_t; +#endif + +#ifndef _VDSO_XTIME_CLOCK_SEC_T +#define _VDSO_XTIME_CLOCK_SEC_T +typedef u32 vdso_xtime_clock_sec_t; +#endif + /* Try to be cache-friendly on systems that don't implement the * generic timer: fit the unconditionally updated fields in the first * 32 bytes. @@ -35,9 +45,11 @@ struct vdso_data { u32 xtime_coarse_sec; /* coarse time */ u32 xtime_coarse_nsec; - u32 wtm_clock_sec; /* wall to monotonic offset */ - u32 wtm_clock_nsec; - u32 xtime_clock_sec; /* CLOCK_REALTIME - seconds */ + /* wall to monotonic offset */ + u32 wtm_clock_sec; + vdso_wtm_clock_nsec_t wtm_clock_nsec; + /* CLOCK_REALTIME - seconds */ + vdso_xtime_clock_sec_t xtime_clock_sec; u32 cs_mono_mult; /* clocksource multiplier */ u64 cs_cycle_last; /* last cycle value */ diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c index e9c99da67786..3ae96d236e48 100644 --- a/arch/arm/vdso/vgettimeofday.c +++ b/arch/arm/vdso/vgettimeofday.c @@ -99,26 +99,31 @@ static notrace int do_monotonic_coarse(const struct vdso_data *vd, #ifdef CONFIG_ARM_ARCH_TIMER -static notrace u64 get_ns(const struct vdso_data *vd) +/* + * Returns the clock delta, in nanoseconds left-shifted by the clock + * shift. + */ +static notrace u64 get_clock_shifted_nsec(const u64 cycle_last, + const u32 mult, + const u64 mask) { - u64 cycle_delta; - u64 cycle_now; - u64 nsec; + u64 res; - cycle_now = arch_vdso_read_counter(); + /* Read the virtual counter. */ + res = arch_vdso_read_counter(); - cycle_delta = (cycle_now - vd->cs_cycle_last) & vd->cs_mask; + res = res - cycle_last; - nsec = (cycle_delta * vd->cs_mono_mult) + vd->xtime_clock_snsec; - nsec >>= vd->cs_shift; - - return nsec; + res &= mask; + return res * mult; } static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) { - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; + u64 mask; + vdso_xtime_clock_sec_t sec; do { seq = vdso_read_begin(vd); @@ -126,22 +131,33 @@ static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts) if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - } while (vdso_read_retry(vd, seq)); + mult = vd->cs_mono_mult; + shift = vd->cs_shift; + mask = vd->cs_mask; + + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs); + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; } static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) { - struct timespec tomono; - u64 nsecs; - u32 seq; + u32 seq, mult, shift; + u64 nsec, cycle_last; + u64 mask; + vdso_wtm_clock_nsec_t wtm_nsec; + __kernel_time_t sec; do { seq = vdso_read_begin(vd); @@ -149,17 +165,26 @@ static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts) if (vd->use_syscall) return -1; - ts->tv_sec = vd->xtime_clock_sec; - nsecs = get_ns(vd); + cycle_last = vd->cs_cycle_last; - tomono.tv_sec = vd->wtm_clock_sec; - tomono.tv_nsec = vd->wtm_clock_nsec; + mult = vd->cs_mono_mult; + shift = vd->cs_shift; + mask = vd->cs_mask; - } while (vdso_read_retry(vd, seq)); + sec = vd->xtime_clock_sec; + nsec = vd->xtime_clock_snsec; - ts->tv_sec += tomono.tv_sec; - ts->tv_nsec = 0; - timespec_add_ns(ts, nsecs + tomono.tv_nsec); + sec += vd->wtm_clock_sec; + wtm_nsec = vd->wtm_clock_nsec; + + } while (unlikely(vdso_read_retry(vd, seq))); + + nsec += get_clock_shifted_nsec(cycle_last, mult, mask); + nsec >>= shift; + nsec += wtm_nsec; + /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */ + ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec); + ts->tv_nsec = nsec; return 0; }