From patchwork Tue Mar 30 10:57:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12172275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53512C433C1 for ; Tue, 30 Mar 2021 11:02:37 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5F25561928 for ; Tue, 30 Mar 2021 11:02:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F25561928 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=TfRrGKTOEwKFMGPiGh1cLuoiArDPXfw06nOdfRVyKMY=; b=CtAJIpMUCLlKCbh9n0cbgvpjm kdAzgF9rqmSyqFF3aHy/wsf8GqiyfdgbVciT6DzcxIptxhJzuL63551d4QF0ovT1Z4jgJHHQ9KED8 KNN16R6U2XNYqeIBsP5OMq/ikXLnbNqhdZpBAX6s82uhyERx92j9w4QpsYMQzMNn/2Tg5HnJrx+jx UW1fyZ9B1xGus8BGwfihvCboAUUx2DEy02jbrHzVlZgCdiIGWKHkPESARBfo/Op3EPIjEJBhbg0jQ lEubdRC9AezvDunb6uCOkRnK4jj6RJ1jbTWZuD/9WbjsNGjHUQIRdrtu/VJnhlECxkUxkaN2aKdhE bjmXQigLg==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lRC6n-003Uev-Eb; Tue, 30 Mar 2021 11:00:30 +0000 Received: from mail-pj1-x1030.google.com ([2607:f8b0:4864:20::1030]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lRC4G-003UD2-Ls for linux-arm-kernel@lists.infradead.org; Tue, 30 Mar 2021 10:58:01 +0000 Received: by mail-pj1-x1030.google.com with SMTP id bt4so7539390pjb.5 for ; Tue, 30 Mar 2021 03:57:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CYoGbwP6WEG9KfojoSzATBnTiH7wKBsolpnCs7g67Yw=; b=neXwenPWXagb8gcZbb8qmptguCq7pbLuXjSLhnqZVwIGAr8y3Sl5vsNJGENiWPuo2P mP7azCY7ATrf4/9HWA/bi3QQmyRWtyyT14wioLFzLDFTotlJupFuuiSnMuD0It93Zv8B c3CpXFhXlEbwhAd3L9fVV7dv5IFkbG0PQv6NY6+0PXzmA5N7UriYFq8MJI78+hvx3n5n RaplwcAYLyJ8KyiGxN3gLIB12kvdeD5HplVtKYmzbv32fyGPmow/UtbcCyP4hG6FTAio X5MdAyIhioeGskEzdV7ONWeFTVTlMT+e2q2M+VEtx/GOk5FnwrIIz1iqZCTOEFq78i4e VPjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CYoGbwP6WEG9KfojoSzATBnTiH7wKBsolpnCs7g67Yw=; b=cpKvioRUJWOrAI7uQo5OOgTiUo4DNveqWNZWF0NgG3EctIQtyPIyoL1ljtIf/vCXNK lHnJk79WH7W0upM8I5HLo3D/dYVx2oHypyH3oUy/Jld1PN2yoHNGdo+y0cpn9FznTJOI n0RxSCGZYPixF/JG9gFw9Mpnxw6oMR7tSyjHOiFiJOeSH6Ge/urV8tMj5gwmFMlgkt16 HkiiscL9pOE2VcB+VYDt+A4zpxD/V42hWsojexo+xkxjCbhObUlxMg+ezR5q0Busy8fV z+8LCKkIE9BzScijaNno+OTC06MROGF3T+eQoLo3mfRVDqUPplGShqvZ263w8KBTWO+q ZU1A== X-Gm-Message-State: AOAM530u/hjCXMur11rM+N4YxpkY+KxqekNSAo10T8ocA+R/x8sJoXCc DN4rSphzRnAH5IIPQA3oTDqvwunvWIb/ X-Google-Smtp-Source: ABdhPJzAol5vTvVne6h1WdHQ8hMqPV3QZH3p/jn2J+zy5iWjCGfDwZdSSGioCocxncLNM6o7a1dvqg== X-Received: by 2002:a17:90a:ab09:: with SMTP id m9mr3887720pjq.122.1617101871098; Tue, 30 Mar 2021 03:57:51 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 144sm20000439pfy.75.2021.03.30.03.57.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 03:57:50 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Vincenzo Frascino , Thomas Gleixner , Mark Rutland , Andrei Vagin , Marc Zyngier Subject: [PATCH 1/2] arm64/gettimeofday: correct the note about isb in __arch_get_hw_counter() Date: Tue, 30 Mar 2021 18:57:18 +0800 Message-Id: <20210330105719.47760-2-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210330105719.47760-1-kernelfans@gmail.com> References: <20210330105719.47760-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210330_115753_632727_9970900A X-CRM114-Status: GOOD ( 14.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The seq lock in vdso_read_retry() is behind smp_rmb(), and can not be speculated before the counter value due to the read dependency. Hence the original note is misleading. The description of getting counter value is not very clear. [1] 'mrs Xt, cntpct' may execute out of program order, either forward or backward. Hence this isb is still required to protect against backward speculation. Correct the note around the code to show the motivation. [1]: AArch64 Programmer's Guides Generic Timer: 3.1. Count and frequency Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Vincenzo Frascino Cc: Thomas Gleixner Cc: Mark Rutland Cc: Andrei Vagin Cc: Marc Zyngier To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/vdso/compat_gettimeofday.h | 9 ++++++--- arch/arm64/include/asm/vdso/gettimeofday.h | 9 ++++++--- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/vdso/compat_gettimeofday.h b/arch/arm64/include/asm/vdso/compat_gettimeofday.h index 7508b0ac1d21..b5dfda25a5dc 100644 --- a/arch/arm64/include/asm/vdso/compat_gettimeofday.h +++ b/arch/arm64/include/asm/vdso/compat_gettimeofday.h @@ -123,10 +123,13 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode, isb(); asm volatile("mrrc p15, 1, %Q0, %R0, c14" : "=r" (res)); /* - * This isb() is required to prevent that the seq lock is - * speculated. + * Getting count value may execute out of program order, either forward + * or backward. Although the caller has a read dependency on @res, but + * it can not protect backward speculation against no dependency + * instruction. Beside this purpose, this isb also severs as a + * compiler barrier for this __always_inline function. */ - isb(); + isb(); return res; } diff --git a/arch/arm64/include/asm/vdso/gettimeofday.h b/arch/arm64/include/asm/vdso/gettimeofday.h index 631ab1281633..6988a730b878 100644 --- a/arch/arm64/include/asm/vdso/gettimeofday.h +++ b/arch/arm64/include/asm/vdso/gettimeofday.h @@ -84,10 +84,13 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode, isb(); asm volatile("mrs %0, cntvct_el0" : "=r" (res) :: "memory"); /* - * This isb() is required to prevent that the seq lock is - * speculated.# + * Getting count value may execute out of program order, either forward + * or backward. Although the caller has a read dependency on @res, but + * it can not protect backward speculation against no dependency + * instruction. Beside this purpose, this isb also severs as a + * compiler barrier for this __always_inline function. */ - isb(); + isb(); return res; } From patchwork Tue Mar 30 10:57:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 12172269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_SIGNED,DKIM_VALID,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81CCFC433DB for ; Tue, 30 Mar 2021 11:00:34 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 55CEA6196C for ; Tue, 30 Mar 2021 11:00:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 55CEA6196C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=YSKSXJqrTlovf1ImRPCb0LGyzIpyAnX/AA9N1AXW04M=; b=TPP8HeeqjN+wWlSGnzGOrWeVu 8zcorXLn+ZcE/iGdICtCu4QjPm6lmX5/i7Og7dztbILxgumP2CJUmxT5XdZzBVxkKNntd/rbjjzCD gfqSbkZ4PKrkAW5qJUpWogvNUMd5D0zULQkDM0ltMSF/l/DBJ0XYDu7NPFE5V6NPbw7uwpxH3pziy y+8+LvMvTKJokkheLaHEfaVOA6KWsTXn/T2cWr9iZIHj5FnD99G3N+uDnBmldBSnR91F0ldIoxcOk hH8ZMoHwuK0UXaYWhQk+SDXgdWyr0KZ1bQIybbc+Ka/zrgvGHxXFEMztH2hMl7/2ml7bWgJZYmsLy i7nx0VmVA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lRC54-003ULR-H1; Tue, 30 Mar 2021 10:58:42 +0000 Received: from mail-pf1-x42b.google.com ([2607:f8b0:4864:20::42b]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lRC4K-003UDs-Hz for linux-arm-kernel@lists.infradead.org; Tue, 30 Mar 2021 10:58:02 +0000 Received: by mail-pf1-x42b.google.com with SMTP id j25so11911318pfe.2 for ; Tue, 30 Mar 2021 03:57:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fJSbPvojFnbEF/MBW0oxVAl1MiFmeqCZww4wOyxMW9k=; b=a/ENeiYnVcIy2JAnFWn+h4IBhPsLtxDD7Sn9KlheEGfSyOh9G8MZnBa0LDCynGJHKQ nVN0zSoSICHf6yHga9VATv5iKnTQuwLbtBvzThlNQOGCcso2fekJXXYbx0/jqOCOZw57 pLqnyXVmJcOV7rcEInvNTJ8/c2cmk4RQ9G2RGkP5p7UwhOO+gIB9i7Tu6jdX2lkdwNtR /xEhWy0mBgvXf3rkozdn2Jh6WRnaD0cBBKYcZ8JO8GSNVEaoUkABTiHjh/qCiO5acX6/ A0ztdcYg3vVURMZ+vu+I2UBe8N4ED49bfWsMbR6GnT6myb39BV2XThwS9msxQpFkRnvX kjdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fJSbPvojFnbEF/MBW0oxVAl1MiFmeqCZww4wOyxMW9k=; b=UHtzmXjjw/TpQ4CYnEYyCveKyEvKFtL4pA6cxwLFAl1waNsOwJ+lJQsjP7qwNHNe5T UqRJCd6sWLHXec1Hj2YPou/sHzO24rlrI88xUiwMZxVbiPMS4P+t3mq3sThfVIPDDl3R +NQu6Ms9jEyfPXWwQh0Xx2ycoZEH3As6QyZ7/ypq1Z24acIfAamdWpwdV0MLf3Dt+xae Kbgtn4kU6TrdltPeaZGOdakEOMvz/fPGMu8Kr85Q70GehiFSoeP/tx40FUlGXnuBvYoU kGRQ9OxuVjY6hB069Gs1uhNTYclh0AXKvTxew8j3OytAE4pglNShElqrmggLBfBaViM7 0WLw== X-Gm-Message-State: AOAM530ApnDyIzxa9Pv8qJqLEWMIyPCIIHnlT8DFYTSc0YpaKh2/Haar S3tPE+Gf9XYjYbOmXh/3tUvfK3i4MQ== X-Google-Smtp-Source: ABdhPJyETbcV75D2ImaUUc5rTmhTgLolq8KDOX/OVNaVIefsyEHcWDIMqqOv8SLOGTuv6cHHr18pZw== X-Received: by 2002:a65:5a4a:: with SMTP id z10mr27337238pgs.240.1617101874788; Tue, 30 Mar 2021 03:57:54 -0700 (PDT) Received: from x1pad.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 144sm20000439pfy.75.2021.03.30.03.57.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 03:57:54 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Catalin Marinas , Will Deacon , Vincenzo Frascino , Thomas Gleixner , Mark Rutland , Andrei Vagin , Marc Zyngier Subject: [PATCH 2/2] arm64/arch_timer: replace arch_counter_enforce_ordering() with isb Date: Tue, 30 Mar 2021 18:57:19 +0800 Message-Id: <20210330105719.47760-3-kernelfans@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210330105719.47760-1-kernelfans@gmail.com> References: <20210330105719.47760-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210330_115757_233515_DA3208FA X-CRM114-Status: GOOD ( 15.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The description of getting counter value is not very clear. [1] 'mrs Xt, cntpct' may execute out of program order, either forward or backward. Now taking a look at this group of getting counter routines. All of them are called from sched_clock(). And there is an isb to protect forward speculation. But there is no isb for the backward speculation. The current code enforces read dependency instructions anchored on getting counter. But it is not enough to protect against other no dependency instructions, and even function call can not prevent the speculation between getting counter and them. Replacing arch_counter_enforce_ordering() with isb to achieve the aim. [1]: AArch64 Programmer's Guides Generic Timer: 3.1. Count and frequency Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Vincenzo Frascino Cc: Thomas Gleixner Cc: Mark Rutland Cc: Andrei Vagin Cc: Marc Zyngier To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/arch_timer.h | 34 ++++++++--------------------- 1 file changed, 9 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/arch_timer.h b/arch/arm64/include/asm/arch_timer.h index 9f0ec21d6327..233ade46390f 100644 --- a/arch/arm64/include/asm/arch_timer.h +++ b/arch/arm64/include/asm/arch_timer.h @@ -165,32 +165,18 @@ static inline void arch_timer_set_cntkctl(u32 cntkctl) isb(); } -/* - * Ensure that reads of the counter are treated the same as memory reads - * for the purposes of ordering by subsequent memory barriers. - * - * This insanity brought to you by speculative system register reads, - * out-of-order memory accesses, sequence locks and Thomas Gleixner. - * - * http://lists.infradead.org/pipermail/linux-arm-kernel/2019-February/631195.html - */ -#define arch_counter_enforce_ordering(val) do { \ - u64 tmp, _val = (val); \ - \ - asm volatile( \ - " eor %0, %1, %1\n" \ - " add %0, sp, %0\n" \ - " ldr xzr, [%0]" \ - : "=r" (tmp) : "r" (_val)); \ -} while (0) - static __always_inline u64 __arch_counter_get_cntpct_stable(void) { u64 cnt; isb(); cnt = arch_timer_reg_read_stable(cntpct_el0); - arch_counter_enforce_ordering(cnt); + /* + * read dependency does not help against the sepculation between getting counter + * and no dependency instructions, which cause the overwritten of register value. + * So here needs isb. + */ + isb(); return cnt; } @@ -200,7 +186,7 @@ static __always_inline u64 __arch_counter_get_cntpct(void) isb(); cnt = read_sysreg(cntpct_el0); - arch_counter_enforce_ordering(cnt); + isb(); return cnt; } @@ -210,7 +196,7 @@ static __always_inline u64 __arch_counter_get_cntvct_stable(void) isb(); cnt = arch_timer_reg_read_stable(cntvct_el0); - arch_counter_enforce_ordering(cnt); + isb(); return cnt; } @@ -220,12 +206,10 @@ static __always_inline u64 __arch_counter_get_cntvct(void) isb(); cnt = read_sysreg(cntvct_el0); - arch_counter_enforce_ordering(cnt); + isb(); return cnt; } -#undef arch_counter_enforce_ordering - static inline int arch_timer_arch_init(void) { return 0;