From patchwork Tue May 12 12:40:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11543099 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 49E3092A for ; Tue, 12 May 2020 12:47:26 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C77320674 for ; Tue, 12 May 2020 12:47:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Vj6hqmHI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C77320674 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:Subject:To:From :Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: List-Owner; bh=Lcuhz42aXlz9WdNBVyWvK93/Mxs+eJgD5S1LhT7cyf8=; b=Vj6hqmHIC8bzXY XsJcMlbN5sOqwV8lL9HhAlt8k3nXX7H8BGhGkO4g88AspPnappcT99zW2AESNKpgwugxikTKpxl+T vipcv7pYvc3cTq0J+G8U/VdbCPSepB4t7ItNpCGBffPetm9k7bZhHRV+N6nmAwhyRmoens0fyW7fW LP+JpVpSMac0bvMm7XeWm4M6z4bUvbD7u1qTz5btQbiiYkPAI5+OdC9DgPm1AoLRVtie3cspmW/6K f4iiSlA/pg3EQG97f79Eg8ouQUVD15xXmv4hWjy7yp+vlx2vdyElV1ttvlACHqvEGIMZR80a5/s4A Dp/rqsg96M4o2aymM6bQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJY-0007NM-9W; Tue, 12 May 2020 12:47:16 +0000 Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by bombadil.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJE-0007C2-1u; Tue, 12 May 2020 12:46:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 468BD305F2E; Tue, 12 May 2020 14:46:53 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 27BD920962E28; Tue, 12 May 2020 14:46:53 +0200 (CEST) Message-ID: <20200512124450.745021268@infradead.org> User-Agent: quilt/0.66 Date: Tue, 12 May 2020 14:40:59 +0200 From: Peter Zijlstra To: Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , Leo Yan Subject: [PATCH 1/5] sched_clock: Expose struct clock_read_data References: <20200512124058.833263033@infradead.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alexander.shishkin@linux.intel.com, daniel.lezcano@linaro.org, sboyd@codeaurora.org, linux-kernel@vger.kernel.org, acme@kernel.org, "Peter Zijlstra \(Intel\)" , mingo@redhat.com, john.stultz@linaro.org, tglx@linutronix.de, jolsa@redhat.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org In order to support perf_event_mmap_page::cap_time features, an architecture needs, aside from a userspace readable counter register, to expose the exact clock data so that userspace can convert the counter register into a correct timestamp. Provide struct clock_read_data and two (seqcount) helpers so that architectures (arm64 in specific) can expose the numbers to userspace. Signed-off-by: Peter Zijlstra (Intel) --- include/linux/sched_clock.h | 28 ++++++++++++++++++++++++++++ kernel/time/sched_clock.c | 41 +++++++++++++---------------------------- 2 files changed, 41 insertions(+), 28 deletions(-) --- a/include/linux/sched_clock.h +++ b/include/linux/sched_clock.h @@ -6,6 +6,34 @@ #define LINUX_SCHED_CLOCK #ifdef CONFIG_GENERIC_SCHED_CLOCK +/** + * struct clock_read_data - data required to read from sched_clock() + * + * @epoch_ns: sched_clock() value at last update + * @epoch_cyc: Clock cycle value at last update. + * @sched_clock_mask: Bitmask for two's complement subtraction of non 64bit + * clocks. + * @read_sched_clock: Current clock source (or dummy source when suspended). + * @mult: Multipler for scaled math conversion. + * @shift: Shift value for scaled math conversion. + * + * Care must be taken when updating this structure; it is read by + * some very hot code paths. It occupies <=40 bytes and, when combined + * with the seqcount used to synchronize access, comfortably fits into + * a 64 byte cache line. + */ +struct clock_read_data { + u64 epoch_ns; + u64 epoch_cyc; + u64 sched_clock_mask; + u64 (*read_sched_clock)(void); + u32 mult; + u32 shift; +}; + +extern struct clock_read_data *sched_clock_read_begin(unsigned int *seq); +extern int sched_clock_read_retry(unsigned int seq); + extern void generic_sched_clock_init(void); extern void sched_clock_register(u64 (*read)(void), int bits, --- a/kernel/time/sched_clock.c +++ b/kernel/time/sched_clock.c @@ -20,31 +20,6 @@ #include "timekeeping.h" /** - * struct clock_read_data - data required to read from sched_clock() - * - * @epoch_ns: sched_clock() value at last update - * @epoch_cyc: Clock cycle value at last update. - * @sched_clock_mask: Bitmask for two's complement subtraction of non 64bit - * clocks. - * @read_sched_clock: Current clock source (or dummy source when suspended). - * @mult: Multipler for scaled math conversion. - * @shift: Shift value for scaled math conversion. - * - * Care must be taken when updating this structure; it is read by - * some very hot code paths. It occupies <=40 bytes and, when combined - * with the seqcount used to synchronize access, comfortably fits into - * a 64 byte cache line. - */ -struct clock_read_data { - u64 epoch_ns; - u64 epoch_cyc; - u64 sched_clock_mask; - u64 (*read_sched_clock)(void); - u32 mult; - u32 shift; -}; - -/** * struct clock_data - all data needed for sched_clock() (including * registration of a new clock source) * @@ -93,6 +68,17 @@ static inline u64 notrace cyc_to_ns(u64 return (cyc * mult) >> shift; } +struct clock_read_data *sched_clock_read_begin(unsigned int *seq) +{ + *seq = raw_read_seqcount(&cd.seq); + return cd.read_data + (*seq & 1); +} + +int sched_clock_read_retry(unsigned int seq) +{ + return read_seqcount_retry(&cd.seq, seq); +} + unsigned long long notrace sched_clock(void) { u64 cyc, res; @@ -100,13 +86,12 @@ unsigned long long notrace sched_clock(v struct clock_read_data *rd; do { - seq = raw_read_seqcount(&cd.seq); - rd = cd.read_data + (seq & 1); + rd = sched_clock_read_begin(&seq); cyc = (rd->read_sched_clock() - rd->epoch_cyc) & rd->sched_clock_mask; res = rd->epoch_ns + cyc_to_ns(cyc, rd->mult, rd->shift); - } while (read_seqcount_retry(&cd.seq, seq)); + } while (sched_clock_read_retry(seq)); return res; } From patchwork Tue May 12 12:41:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11543101 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4251D92A for ; Tue, 12 May 2020 12:47:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1DACA20674 for ; Tue, 12 May 2020 12:47:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ZYwIfu3T" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1DACA20674 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:Subject:To:From :Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: List-Owner; bh=uDvX9CetRhXWICr7ECTV7Z2AalrxdzPnrw3NeYuuLb4=; b=ZYwIfu3TGBZKf2 q/FOgdm1rY5dJEFAS9IfZoNSfecCn6uqqWMaN+uNFcoY5op5q3qc6cjZuAaa7fnfGygPeR1wNRaI5 53PBGe9QJrz6Bj+dtRbubUyw87oL5AUSUhtiffxZnwlBky+VLtzbVPdJnYLKhZ2jp18uFdeAzxQJl BAOb9HDhxCQLsTLvlXWL+R5yp8rjdai6yXjKknPh14UzLNreECWCcbOp8hw9iJmY2vVx1VB7wnaBN y037b1XFWboSN7VFPRw0feq1x/hBS8k7Y/TVgrNxwSFBg1gF08R8MOUCqix+UqFUEbgrQji5iAxUf yr9jIHMEleGaIwE4uFxg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJm-0007do-KZ; Tue, 12 May 2020 12:47:30 +0000 Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by bombadil.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJE-0007C3-26; Tue, 12 May 2020 12:46:56 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 45AFD305EEF; Tue, 12 May 2020 14:46:53 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2A06B20962E54; Tue, 12 May 2020 14:46:53 +0200 (CEST) Message-ID: <20200512124450.824507755@infradead.org> User-Agent: quilt/0.66 Date: Tue, 12 May 2020 14:41:00 +0200 From: Peter Zijlstra To: Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , Leo Yan Subject: [PATCH 2/5] arm64: perf: Implement correct cap_user_time References: <20200512124058.833263033@infradead.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alexander.shishkin@linux.intel.com, daniel.lezcano@linaro.org, sboyd@codeaurora.org, linux-kernel@vger.kernel.org, acme@kernel.org, "Peter Zijlstra \(Intel\)" , mingo@redhat.com, john.stultz@linaro.org, tglx@linutronix.de, jolsa@redhat.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org As reported by Leo; the existing implementation is broken when the clock and counter don't intersect at 0. Use the sched_clock's struct clock_read_data information to correctly implement cap_user_time and cap_user_time_zero. Note that the ARM64 counter is architecturally only guaranteed to be 56bit wide (implementations are allowed to be wider) and the existing perf ABI cannot deal with wrap-around. This implementation should also be faster than the old; seeing how we don't need to recompute mult and shift all the time. Reported-by: Leo Yan Signed-off-by: Peter Zijlstra (Intel) Reported-by: kbuild test robot --- arch/arm64/kernel/perf_event.c | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) --- a/arch/arm64/kernel/perf_event.c +++ b/arch/arm64/kernel/perf_event.c @@ -19,6 +19,7 @@ #include #include #include +#include #include /* ARMv8 Cortex-A53 specific event types. */ @@ -1165,28 +1166,45 @@ device_initcall(armv8_pmu_driver_init) void arch_perf_update_userpage(struct perf_event *event, struct perf_event_mmap_page *userpg, u64 now) { - u32 freq; - u32 shift; + struct clock_read_data *rd; + unsigned int seq; /* * Internal timekeeping for enabled/running/stopped times * is always computed with the sched_clock. */ - freq = arch_timer_get_rate(); userpg->cap_user_time = 1; + userpg->cap_user_time_zero = 1; + + do { + rd = sched_clock_read_begin(&seq); + + userpg->time_mult = rd->mult; + userpg->time_shift = rd->shift; + userpg->time_zero = rd->epoch_ns; + + /* + * This isn't strictly correct, the ARM64 counter can be + * 'short' and then we get funnies when it wraps. The correct + * thing would be to extend the perf ABI with a cycle and mask + * value, but because wrapping on ARM64 is very rare in + * practise this 'works'. + */ + userpg->time_zero -= (rd->epoch_cyc * rd->mult) >> rd->shift; + + } while (sched_clock_read_retry(seq)); + + userpg->time_offset = userpg->time_zero - now; - clocks_calc_mult_shift(&userpg->time_mult, &shift, freq, - NSEC_PER_SEC, 0); /* * time_shift is not expected to be greater than 31 due to * the original published conversion algorithm shifting a * 32-bit value (now specifies a 64-bit value) - refer * perf_event_mmap_page documentation in perf_event.h. */ - if (shift == 32) { - shift = 31; + if (userpg->shift == 32) { + userpg->shift = 31; userpg->time_mult >>= 1; } - userpg->time_shift = (u16)shift; - userpg->time_offset = -now; + } From patchwork Tue May 12 12:41:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11543103 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C34192A for ; Tue, 12 May 2020 12:47:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id ED4B120736 for ; Tue, 12 May 2020 12:47:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="rWnzToBD"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ZXKExd2Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED4B120736 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:Subject:To:From :Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: List-Owner; bh=MT4UKDvpXI0oKQN33rt6Dcj0/6hinbD6Ar4ZXXCihX8=; b=rWnzToBDv6ea0W zUUM/xuXaxqBwLNFlNFHwD+N6bZrsGNo796SSWJ+uF6mIxNj3+PZ2R8be6Mu2xyeEsx/XcinsMDs5 44at+WZRgmMVJ12CpzAo2J7/2pfradnGFLsgevlQkDrrNwhuXFVJ/gCyK70A66y03NEt7QUi9TzkK CU+vJGC679X3LETZOT+y4vpcjM2i0D0f2cbMs9VHGohHvkA4OpFP9IoOs0Yy8f5MraBursSoXSiBe jw9pKaZNKgzg2OhdtJwRBuV6sBVkTWhJ1WTpbMEqk0DtkSd8lcPtPOk3f4bZ8oj317On4PUXR1Qyj x0HEQtzOI6/F0djUhK2A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJz-0007r4-9Q; Tue, 12 May 2020 12:47:43 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJK-0007EQ-3x for linux-arm-kernel@bombadil.infradead.org; Tue, 12 May 2020 12:47:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=ngZs8drZ7QXRvmUeJGP4y1pSY454qouM1a6376s95vs=; b=ZXKExd2ZH/XmhPLQHFbAQkMSxn dtYYboA1nY3FF0+yBsWq4FobSf6GFGhLCUnrmPQKSwpiQXmIV3cV2L6rZcbI9u2qE3py6oq7zbcbs hk1tIwy76YSm8bbl/m3NhC+y3YCZNGRA/jO6FkHQ18bXcaNMQNxCkd2EDNP0UwD/tgvsXZEbcAs0B CXp1eX2wUz3BoeuMCPyAcyKmqoSNwbfC6FXLY6/zsfjiD3Y3IzSFTf2DixAOYpO/QPt8EVYqB93h8 kKe1Y0411ra2oOuYlKZR5XuqE3bLWJIkCZL7Go7oBqH84dQmlS85SxJMUpI/kgZwb1+0CdRMW+dRK IxNP2Jow==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJF-0003GZ-ID; Tue, 12 May 2020 12:46:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 44CCC304DB2; Tue, 12 May 2020 14:46:53 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 2DA2820962E55; Tue, 12 May 2020 14:46:53 +0200 (CEST) Message-ID: <20200512124450.903534446@infradead.org> User-Agent: quilt/0.66 Date: Tue, 12 May 2020 14:41:01 +0200 From: Peter Zijlstra To: Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , Leo Yan Subject: [PATCH 3/5] arm64: perf: Only advertise cap_user_time for arch_timer References: <20200512124058.833263033@infradead.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alexander.shishkin@linux.intel.com, daniel.lezcano@linaro.org, sboyd@codeaurora.org, linux-kernel@vger.kernel.org, acme@kernel.org, "Peter Zijlstra \(Intel\)" , mingo@redhat.com, john.stultz@linaro.org, tglx@linutronix.de, jolsa@redhat.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org When sched_clock is running on anything other than arch_timer, don't advertise cap_user_time*. Requested-by: Will Deacon Signed-off-by: Peter Zijlstra (Intel) --- arch/arm64/kernel/perf_event.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) --- a/arch/arm64/kernel/perf_event.c +++ b/arch/arm64/kernel/perf_event.c @@ -13,6 +13,8 @@ #include #include +#include + #include #include #include @@ -1169,16 +1171,15 @@ void arch_perf_update_userpage(struct pe struct clock_read_data *rd; unsigned int seq; - /* - * Internal timekeeping for enabled/running/stopped times - * is always computed with the sched_clock. - */ - userpg->cap_user_time = 1; - userpg->cap_user_time_zero = 1; + userpg->cap_user_time = 0; + userpg->cap_user_time_zero = 0; do { rd = sched_clock_read_begin(&seq); + if (rd->read_sched_clock != arch_timer_read_counter) + return; + userpg->time_mult = rd->mult; userpg->time_shift = rd->shift; userpg->time_zero = rd->epoch_ns; @@ -1207,4 +1208,10 @@ void arch_perf_update_userpage(struct pe userpg->time_mult >>= 1; } + /* + * Internal timekeeping for enabled/running/stopped times + * is always computed with the sched_clock. + */ + userpg->cap_user_time = 1; + userpg->cap_user_time_zero = 1; } From patchwork Tue May 12 12:41:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11543105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3BE8092A for ; Tue, 12 May 2020 12:48:07 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C53720674 for ; Tue, 12 May 2020 12:48:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="mitasZ/R"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="W6os7/Rw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C53720674 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:Subject:To:From :Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: List-Owner; bh=WKj7xIlbzvFlUZXtMSKYw8TjoOwZGNMPGLjNh1WEGAo=; b=mitasZ/RBMhmdz tB4CRGgYeIP6x5A+dkEkHjU6xKa+wq17GSdq2jaqmt7hoaia+b54xU6kBSdkjOXYSt74z5Zl1dl7Y DNrzvr3w/E7+v3jxnU/w2rYwhyWFDj4qIT3gZ5J/CZdP2h/om6BnlsAtT69bM7dH3nTsMHKJESjNo cuY7en/dDpkF3NGSP7jjXELseCAtK1gDTBqBiif/XJ7mq6r3zZJNKanyrx/Dhs/4ZerC++v4G79VK /+5DUaZH1eQv66Q9v34/060YMnAWe2o8d1sKs+zl/plaW9wNaiuqBiHzKDZCCVhjzVQ/xiJMtKPql MH8mZTDDo5hoHelAzeZw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUKG-00087G-7v; Tue, 12 May 2020 12:48:00 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJK-0007ER-3w for linux-arm-kernel@bombadil.infradead.org; Tue, 12 May 2020 12:47:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=CYYpiqrbBLgooKXOpvi8wy4NrFYsaxk2WuAfbe/cgCE=; b=W6os7/RwiqUdE+aykxc8eGzgyH fS7bISumtHbFPFrf4M7xtKFo5FeqDVH6JQa4Xq6qSDNjl2frm/JUmDi97d1mL3x0l8u4NpOaFnKW3 gx9+Pii5OQt83bS5JqZXK567UF1kC1lE2NVZudxzeS5r41/T5dV77e3KLsEfImeS0gkKs9a735SpY C5xF7Yk/YgDkA2IxcKBxDOqSYS3XQjmAAWLYr5X7OWnoNvggJxTIy+E/eZdz38Yofpf0WjbWT5Qdj SNwkmL2X79pGemtlg7hfMaUy6cW4S8JZDXl9TNeckLoQ/C+9sS1wyvLcOgzSYu85nlycGfTJPkA1j dVv7v0uQ==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJF-0003Ga-Hf; Tue, 12 May 2020 12:46:57 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 476FC3062BA; Tue, 12 May 2020 14:46:53 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3408D20962E57; Tue, 12 May 2020 14:46:53 +0200 (CEST) Message-ID: <20200512124450.981890259@infradead.org> User-Agent: quilt/0.66 Date: Tue, 12 May 2020 14:41:02 +0200 From: Peter Zijlstra To: Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , Leo Yan Subject: [PATCH 4/5] perf: Add perf_event_mmap_page::cap_user_time_short ABI References: <20200512124058.833263033@infradead.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alexander.shishkin@linux.intel.com, daniel.lezcano@linaro.org, sboyd@codeaurora.org, linux-kernel@vger.kernel.org, acme@kernel.org, "Peter Zijlstra \(Intel\)" , mingo@redhat.com, john.stultz@linaro.org, tglx@linutronix.de, jolsa@redhat.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org In order to support short clock counters, provide an ABI extention. As a whole: u64 time, delta, cyc = read_cycle_counter(); + if (cap_user_time_short) + cyc = time_cycle + ((cyc - time_cycle) & time_mask); delta = mul_u64_u32_shr(cyc, time_mult, time_shift); if (cap_user_time_zero) time = time_zero + delta; delta += time_offset; Signed-off-by: Peter Zijlstra (Intel) --- include/uapi/linux/perf_event.h | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) --- a/include/uapi/linux/perf_event.h +++ b/include/uapi/linux/perf_event.h @@ -532,9 +532,10 @@ struct perf_event_mmap_page { cap_bit0_is_deprecated : 1, /* Always 1, signals that bit 0 is zero */ cap_user_rdpmc : 1, /* The RDPMC instruction can be used to read counts */ - cap_user_time : 1, /* The time_* fields are used */ + cap_user_time : 1, /* The time_{shift,mult,offset} fields are used */ cap_user_time_zero : 1, /* The time_zero field is used */ - cap_____res : 59; + cap_user_time_short : 1, /* the time_{cycle,mask} fields are used */ + cap_____res : 58; }; }; @@ -593,13 +594,29 @@ struct perf_event_mmap_page { * ((rem * time_mult) >> time_shift); */ __u64 time_zero; + __u32 size; /* Header size up to __reserved[] fields. */ + __u32 __reserved_1; + + /* + * If cap_usr_time_short, the hardware clock is less than 64bit wide + * and we must compute the 'cyc' value, as used by cap_usr_time, as: + * + * cyc = time_cycles + ((cyc - time_cycles) & time_mask) + * + * NOTE: this form is explicitly chosen such that cap_usr_time_short + * is a correction on top of cap_usr_time, and code that doesn't + * know about cap_usr_time_short still works under the assumption + * the counter doesn't wrap. + */ + __u64 time_cycles; + __u64 time_mask; /* * Hole for extension of the self monitor capabilities */ - __u8 __reserved[118*8+4]; /* align to 1k. */ + __u8 __reserved[116*8]; /* align to 1k. */ /* * Control data for the mmap() data buffer. From patchwork Tue May 12 12:41:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11543107 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1E4292A for ; Tue, 12 May 2020 12:48:12 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 763A520674 for ; Tue, 12 May 2020 12:48:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="fElN83Vs"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="3LROw93W" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 763A520674 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:Subject:To:From :Date:Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: List-Owner; bh=66sDjYU2s2ZPDB2nhC7pdS3GGG3C09UMG3OLifb//0Q=; b=fElN83Vs6dM3DH JPofPsl2lCt0gYnABcN85GY2KQgrkU9WLCSiwZON4CK8huV49oJOQbCvvDx6NXgqBr7SgTFC4uKrO afXXHkIsewDxitL+qrgoreSNAvnGPk21P9REB5u9ydcB/zJeF+Utj2OG/G5g7QqhkSITC+IF1HXv4 n67BtTHAc8IjUxtSOBjNMVRzO3f3QVZDUk1qQb61gm7k3q7CisFTRR+82vmo7aibQwu6seD2YqpPL NG8VBIoMN2qCs6q19W0Q2eG7XmdI6p/Ck4pprEpb3s0TU3vJS6ICjHNoufMiDofm9DCd6DWIQG9aq kwByt7eAQ/vhNdjnwJog==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUKP-0008LE-SC; Tue, 12 May 2020 12:48:09 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJK-0007ES-3v for linux-arm-kernel@bombadil.infradead.org; Tue, 12 May 2020 12:47:02 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=2k0yrj8NN02Gxp4NWmpbZtnwVKPk5wX+ZXUDkjmgsRY=; b=3LROw93Wt9I9i4shqU3qi81Uvc OwOaIfDE38glFNfcDDrcRTk34OBkRFBv8ZQ1tSvBLpYNckED8YFu4IxRutb9si/HMsYSzA9WCK0mH gLahCc51jD0xDWmfDzr6MRLADdprHU1Aq3cQEG/xDUutc8rpXfYMGygPNQH8nsDHPFiJtryggSUpw HGfgFWq4zz0f5DYDIezB4eFdViBh19+qR/dWki7UBgbQ6dp2b2zlWKJBBPeMeGDQmvvnAYFz43tOp J1BVpvRNUBojMUISxLTxf5ib5Bdb+lpRnCH9ltMUTkTcnMSQhZFWg/8MkylPiesGfvqktTMlhXFGW oAGTPNFA==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYUJI-0003Gu-0s; Tue, 12 May 2020 12:47:00 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id D74F9301EFB; Tue, 12 May 2020 14:46:58 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 3616320962E56; Tue, 12 May 2020 14:46:53 +0200 (CEST) Message-ID: <20200512124451.061059334@infradead.org> User-Agent: quilt/0.66 Date: Tue, 12 May 2020 14:41:03 +0200 From: Peter Zijlstra To: Will Deacon , Marc Zyngier , Mark Rutland , Catalin Marinas , Leo Yan Subject: [PATCH 5/5] arm64: perf: Add cap_user_time_short References: <20200512124058.833263033@infradead.org> MIME-Version: 1.0 X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: alexander.shishkin@linux.intel.com, daniel.lezcano@linaro.org, sboyd@codeaurora.org, linux-kernel@vger.kernel.org, acme@kernel.org, "Peter Zijlstra \(Intel\)" , mingo@redhat.com, john.stultz@linaro.org, tglx@linutronix.de, jolsa@redhat.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This completes the ARM64 cap_user_time support. Signed-off-by: Peter Zijlstra (Intel) Reported-by: kbuild test robot --- arch/arm64/kernel/perf_event.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) --- a/arch/arm64/kernel/perf_event.c +++ b/arch/arm64/kernel/perf_event.c @@ -1173,6 +1173,7 @@ void arch_perf_update_userpage(struct pe userpg->cap_user_time = 0; userpg->cap_user_time_zero = 0; + userpg->cap_user_time_short = 0; do { rd = sched_clock_read_begin(&seq); @@ -1183,13 +1184,13 @@ void arch_perf_update_userpage(struct pe userpg->time_mult = rd->mult; userpg->time_shift = rd->shift; userpg->time_zero = rd->epoch_ns; + userpg->time_cycle = rd->epoch_cyc; + userpg->time_mask = rd->sched_clock_mask; /* - * This isn't strictly correct, the ARM64 counter can be - * 'short' and then we get funnies when it wraps. The correct - * thing would be to extend the perf ABI with a cycle and mask - * value, but because wrapping on ARM64 is very rare in - * practise this 'works'. + * Subtract the cycle base, such that software that + * doesn't know about cap_user_time_short still 'works' + * assuming no wraps. */ userpg->time_zero -= (rd->epoch_cyc * rd->mult) >> rd->shift; @@ -1214,4 +1215,5 @@ void arch_perf_update_userpage(struct pe */ userpg->cap_user_time = 1; userpg->cap_user_time_zero = 1; + userpg->cap_user_time_short = 1; }