From patchwork Fri May 17 21:25:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Collingbourne X-Patchwork-Id: 13667402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D717FC04FFE for ; Fri, 17 May 2024 21:26:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=jI0nvGwYYR1YIx6xFp3CI/6EcVhpIQW5nl3dUJ4YDuk=; b=KEH xYhg4PRIvSLosy62mny0FIoakFCfaO6UI50wimGQJ7B1vq41C2LGxYBqO4qGyanJxPvRCo24GhFG7 rbrTIrb6RLeyexGmYV9YYZtn8jssOolXEmeAeKTZzkfEANCY3X6oHRduVtbs+RX4wwyTG4Pmv26oB hKVXtby7pCX/VFeS55scCLSNe8ay4nDKJqq4hGv0FL4/9VgB+mzUN9oNEpEP1aXu8wn7j2awIQWMo 6uDrqE6vLhFU62WmkLSoY5Xfm2EaYYb2ZI89nI0woUrvjXVUKAE6SFAvnHhTQYlDRB80wYmZ42g4l FeXPde4bmLCsT6O7CcRUiWRzry5a75Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s855l-000000091zo-29eU; Fri, 17 May 2024 21:26:17 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s855j-000000091zb-2a0P for linux-arm-kernel@bombadil.infradead.org; Fri, 17 May 2024 21:26:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:Cc:To:From:Subject: Mime-Version:Message-Id:Date:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To:References; bh=YvuYeKYTpa5XmyQZgxAXy79uzj1VHSRJKNSW/enfN78=; b=oELiAGUwMj81WLjs6TvsoGQDZ8 vCKHanlcjE26K/P7I8mdVhuUS/d+2i3jBBmzvshgDgrXIlHkOKYQZ7ACJVagnYTWC7nnoBk//ADeo D+fl7uAdZmhEuoUiEiH8sKIjHJbiIWcLXBueCUXFbNpVw/EE6v7Q16fBeL8FcA3xj1LYnGi20CIm8 hq+bUYtlE4+1/dT/jl7fKYVZWWzWSinaDWRhLpQz50COo+8E1An8MFTX7hVpxvUIe+99YeZqN5ycy njsYMoz5s5wP+z0smSL5NX4ZBtUjFatVBhPbyxuBEQyJFFG3FJ7v90LxYcjEVBsSDBC8bbdHYOVVC XXBQWORw==; Received: from mail-yw1-x1149.google.com ([2607:f8b0:4864:20::1149]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s855Z-00000005nyZ-160l for linux-arm-kernel@lists.infradead.org; Fri, 17 May 2024 21:26:11 +0000 Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-61beaa137acso160266867b3.1 for ; Fri, 17 May 2024 14:26:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715981159; x=1716585959; darn=lists.infradead.org; h=cc:to:from:subject:mime-version:message-id:date:from:to:cc:subject :date:message-id:reply-to; bh=YvuYeKYTpa5XmyQZgxAXy79uzj1VHSRJKNSW/enfN78=; b=ZPGf0qPshSGuPcI5uLpFo/Z3vQK6uCVqg8x6Vji6zRO4IW/sN0Z+CoVWpq1yxQiEUO 49PlOA0w6w1OTnTeFQYAv8aRi94dHTkVZD9qqDkrgiMoKMEW0o0NvMIv0pXfnKxr3hdt KBufxECVjBs+WAc7SsZ4DWcUhKB0dFY6hUQYo6zpVaA/YIgwKseZeSIhYH3q0kp1RZBx jileurc2MGoFc6N5pEG9sDV3BOClnfMiioTtOY5NXoRhwFxZtw4yJuF4YpjW54/erlBr tm5Vofj7vHxCG6etuKP0D3wmEDid1U7chxQnSFh1i53xUszh0CRXIVrB27SXLXOlsypV k4fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715981159; x=1716585959; h=cc:to:from:subject:mime-version:message-id:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YvuYeKYTpa5XmyQZgxAXy79uzj1VHSRJKNSW/enfN78=; b=SRLQhWSA4Jd5YvHsL7/sK16UXWq7emWS1Lr+/osZmb5RncBDK1EWaSMdxpKrkcm2t/ bWH8cj0lznaQVRcVuUPuoKmtRAEbffYLKQMvpSvO2Uqptn7S0gbKrtRwjzHEflJo5qez JYR8WoItW+6U9xZYkld89RaIBS7p4BmSBmEHccWq5+6wbCOn9/HJbjUKHwS2DcMwjvJj 8mtKR8CegarIOC8QotAD2vJ91ldNsi7C+wev9558b6EGLmsHboblSvKQ0w0dk+q8DPBW sIUtd8RX8QOUEUqXC4meDKy+cMt4IIdIFu/qEKvgryyK998MEG59d4G6SI9rTDXDUetm BYVg== X-Forwarded-Encrypted: i=1; AJvYcCVdC6+8/di6EDkZ186yg4h+HCNKxrIOq3uBVuv2JryEhAqm7w6WS1qCXl3fT6P3hA1YudYCRauhqotpRrid1oco7XNj3Nqk48hHkk3KYHG+vX+P9gU= X-Gm-Message-State: AOJu0YwmI0QSHRtSlbbheFY2kFm23CtDwO9TOeTSSX9IBotXcH7fYQ9T k3xSZNugA0FZBfBr7oMENOU4RF9h5UiHKxUiYlTXxBqoRP2V0OcO+DZ1iEb1G2hJYw== X-Google-Smtp-Source: AGHT+IEhKfqD3ukKd26JhK4H1GZs8G7Njn+72ZH28znFs+mmoWPmzoSSlNtbPmJy7SGtFB79psjciTI= X-Received: from pcc-desktop.svl.corp.google.com ([2620:15c:2d3:205:34bf:58c6:2ed9:2f56]) (user=pcc job=sendgmr) by 2002:a05:690c:34c1:b0:618:8e4b:f4b0 with SMTP id 00721157ae682-622afff31bfmr66021187b3.9.1715981158899; Fri, 17 May 2024 14:25:58 -0700 (PDT) Date: Fri, 17 May 2024 14:25:51 -0700 Message-Id: <20240517212553.3597611-1-pcc@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.45.0.rc1.225.g2a3ae87e7f-goog Subject: [PATCH v2] arm64: Implement prctl(PR_{G,S}ET_TSC) From: Peter Collingbourne To: Catalin Marinas , Will Deacon , Marc Zyngier Cc: Peter Collingbourne , linux-arm-kernel@lists.infradead.org, Yichao Yu , "Robert O'Callahan" , Keno Fischer X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240517_222609_048404_85265C15 X-CRM114-Status: GOOD ( 28.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On arm64, this prctl controls access to CNTVCT_EL0, CNTVCTSS_EL0 and CNTFRQ_EL0 via CNTKCTL_EL1.EL0VCTEN. Since this bit is also used to implement various erratum workarounds, check whether the CPU needs a workaround whenever we potentially need to change it. This is needed for a correct implementation of non-instrumenting record-replay debugging on arm64 (i.e. rr; https://rr-project.org/). rr must trap and record any sources of non-determinism from the userspace program's perspective so it can be replayed later. This includes the results of syscalls as well as the results of access to architected timers exposed directly to the program. This prctl was originally added for x86 by commit 8fb402bccf20 ("generic, x86: add prctl commands PR_GET_TSC and PR_SET_TSC"), and rr uses it to trap RDTSC on x86 for the same reason. We also considered exposing this as a PTRACE_EVENT. However, prctl seems like a better choice for these reasons: 1) In general an in-process control seems more useful than an out-of-process control, since anything that you would be able to do with ptrace could also be done with prctl (tracer can inject a call to the prctl and handle signal-delivery-stops), and it avoids needing an additional process (which will complicate debugging of the ptraced process since it cannot have more than one tracer, and will be incompatible with ptrace_scope=3) in cases where that is not otherwise necessary. 2) Consistency with x86_64. Note that on x86_64, RDTSC has been there since the start, so it's the same situation as on arm64. Signed-off-by: Peter Collingbourne Link: https://linux-review.googlesource.com/id/I233a1867d1ccebe2933a347552e7eae862344421 --- v2: - added justification to commit message arch/arm64/include/asm/processor.h | 5 ++ arch/arm64/include/asm/thread_info.h | 2 + arch/arm64/kernel/process.c | 70 +++++++++++++++++++++++----- arch/arm64/kernel/traps.c | 20 +++++--- 4 files changed, 79 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index f77371232d8c6..347bd3464fcbe 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -402,5 +402,10 @@ long get_tagged_addr_ctrl(struct task_struct *task); #define GET_TAGGED_ADDR_CTRL() get_tagged_addr_ctrl(current) #endif +int get_tsc_mode(unsigned long adr); +int set_tsc_mode(unsigned int val); +#define GET_TSC_CTL(adr) get_tsc_mode((adr)) +#define SET_TSC_CTL(val) set_tsc_mode((val)) + #endif /* __ASSEMBLY__ */ #endif /* __ASM_PROCESSOR_H */ diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h index e72a3bf9e5634..1114c1c3300a1 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -81,6 +81,7 @@ void arch_setup_new_exec(void); #define TIF_SME 27 /* SME in use */ #define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */ #define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */ +#define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) @@ -97,6 +98,7 @@ void arch_setup_new_exec(void); #define _TIF_SVE (1 << TIF_SVE) #define _TIF_MTE_ASYNC_FAULT (1 << TIF_MTE_ASYNC_FAULT) #define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL) +#define _TIF_TSC_SIGSEGV (1 << TIF_TSC_SIGSEGV) #define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \ diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 4ae31b7af6c31..1a2ae7830c179 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -43,6 +43,7 @@ #include #include +#include #include #include #include @@ -472,27 +473,49 @@ static void entry_task_switch(struct task_struct *next) } /* - * ARM erratum 1418040 handling, affecting the 32bit view of CNTVCT. - * Ensure access is disabled when switching to a 32bit task, ensure - * access is enabled when switching to a 64bit task. + * Handle sysreg updates for ARM erratum 1418040 which affects the 32bit view of + * CNTVCT, various other errata which require trapping all CNTVCT{,_EL0} + * accesses and prctl(PR_SET_TSC). Ensure access is disabled iff a workaround is + * required or PR_TSC_SIGSEGV is set. */ -static void erratum_1418040_thread_switch(struct task_struct *next) +static void update_cntkctl_el1(struct task_struct *next) { - if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_1418040) || - !this_cpu_has_cap(ARM64_WORKAROUND_1418040)) - return; + struct thread_info *ti = task_thread_info(next); - if (is_compat_thread(task_thread_info(next))) + if (test_ti_thread_flag(ti, TIF_TSC_SIGSEGV) || + has_erratum_handler(read_cntvct_el0) || + (IS_ENABLED(CONFIG_ARM64_ERRATUM_1418040) && + this_cpu_has_cap(ARM64_WORKAROUND_1418040) && + is_compat_thread(ti))) sysreg_clear_set(cntkctl_el1, ARCH_TIMER_USR_VCT_ACCESS_EN, 0); else sysreg_clear_set(cntkctl_el1, 0, ARCH_TIMER_USR_VCT_ACCESS_EN); } -static void erratum_1418040_new_exec(void) +static void cntkctl_thread_switch(struct task_struct *prev, + struct task_struct *next) +{ + if ((read_ti_thread_flags(task_thread_info(prev)) & + (_TIF_32BIT | _TIF_TSC_SIGSEGV)) != + (read_ti_thread_flags(task_thread_info(next)) & + (_TIF_32BIT | _TIF_TSC_SIGSEGV))) + update_cntkctl_el1(next); +} + +static int do_set_tsc_mode(unsigned int val) { + if (val == PR_TSC_SIGSEGV) + set_thread_flag(TIF_TSC_SIGSEGV); + else if (val == PR_TSC_ENABLE) + clear_thread_flag(TIF_TSC_SIGSEGV); + else + return -EINVAL; + preempt_disable(); - erratum_1418040_thread_switch(current); + update_cntkctl_el1(current); preempt_enable(); + + return 0; } /* @@ -528,7 +551,7 @@ struct task_struct *__switch_to(struct task_struct *prev, contextidr_thread_switch(next); entry_task_switch(next); ssbs_thread_switch(next); - erratum_1418040_thread_switch(next); + cntkctl_thread_switch(prev, next); ptrauth_thread_switch_user(next); /* @@ -645,7 +668,7 @@ void arch_setup_new_exec(void) current->mm->context.flags = mmflags; ptrauth_thread_init_user(); mte_thread_init_user(); - erratum_1418040_new_exec(); + do_set_tsc_mode(PR_TSC_ENABLE); if (task_spec_ssb_noexec(current)) { arch_prctl_spec_ctrl_set(current, PR_SPEC_STORE_BYPASS, @@ -754,3 +777,26 @@ int arch_elf_adjust_prot(int prot, const struct arch_elf_state *state, return prot; } #endif + +int get_tsc_mode(unsigned long adr) +{ + unsigned int val; + + if (is_compat_task()) + return -EINVAL; + + if (test_thread_flag(TIF_TSC_SIGSEGV)) + val = PR_TSC_SIGSEGV; + else + val = PR_TSC_ENABLE; + + return put_user(val, (unsigned int __user *)adr); +} + +int set_tsc_mode(unsigned int val) +{ + if (is_compat_task()) + return -EINVAL; + + return do_set_tsc_mode(val); +} diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 215e6d7f2df8c..63fe64aafb7ad 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -601,18 +601,26 @@ static void ctr_read_handler(unsigned long esr, struct pt_regs *regs) static void cntvct_read_handler(unsigned long esr, struct pt_regs *regs) { - int rt = ESR_ELx_SYS64_ISS_RT(esr); + if (test_thread_flag(TIF_TSC_SIGSEGV)) { + force_sig(SIGSEGV); + } else { + int rt = ESR_ELx_SYS64_ISS_RT(esr); - pt_regs_write_reg(regs, rt, arch_timer_read_counter()); - arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE); + pt_regs_write_reg(regs, rt, arch_timer_read_counter()); + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE); + } } static void cntfrq_read_handler(unsigned long esr, struct pt_regs *regs) { - int rt = ESR_ELx_SYS64_ISS_RT(esr); + if (test_thread_flag(TIF_TSC_SIGSEGV)) { + force_sig(SIGSEGV); + } else { + int rt = ESR_ELx_SYS64_ISS_RT(esr); - pt_regs_write_reg(regs, rt, arch_timer_get_rate()); - arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE); + pt_regs_write_reg(regs, rt, arch_timer_get_rate()); + arm64_skip_faulting_instruction(regs, AARCH64_INSN_SIZE); + } } static void mrs_handler(unsigned long esr, struct pt_regs *regs)