From patchwork Wed Aug 19 12:13:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 11723843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E5E4E739 for ; Wed, 19 Aug 2020 12:14:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C46D12076E for ; Wed, 19 Aug 2020 12:14:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839242; bh=cU6dUF2JesHsQbTewHHBbh8QA6GTXyIQPYFFzE/8QWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=UZUDuUp62M/0iJmfL29Pg2k1DsoG8aDY5VnCNmORzZqLtp65BQ3gMaekKdP0sEiMs kMJqAscXh947T9MPJOMsSwqVSiC1cM8RMwFhFvr0r/mhw65lzyOpLOTaojnzRQ+dvB pbATN6Mjbt/rc0lwH2USHsuTgmbatrAaZppP3lIQ= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726961AbgHSMOC (ORCPT ); Wed, 19 Aug 2020 08:14:02 -0400 Received: from mail.kernel.org ([198.145.29.99]:51308 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726798AbgHSMOB (ORCPT ); Wed, 19 Aug 2020 08:14:01 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5D0C720738; Wed, 19 Aug 2020 12:14:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839240; bh=cU6dUF2JesHsQbTewHHBbh8QA6GTXyIQPYFFzE/8QWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=zFciX799O1aIbdBKyUraqxG+H21PFFm1inO08gpDk6VLww0Uaisiyic/PK7vHxChH gjH23BMddwKk9LCZ/K5onXUmlUiS1s4lXQF0vGTPPLwEu7Q+fEJpwGS/9SV9XW+Fk/ 09mO1Drjcl2rZMV3njg2HkvKZ3ZQVdTb3ggP14x0= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 1/5] arm64: vdso: Provide a define when building the vDSO Date: Wed, 19 Aug 2020 13:13:14 +0100 Message-Id: <20200819121318.52158-2-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Provide a define identifying if code is being built for the vDSO to help with writing headers that are shared between the kernel and the vDSO. Signed-off-by: Mark Brown --- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile index 45d5cfe46429..88cf0f0b91ed 100644 --- a/arch/arm64/kernel/vdso/Makefile +++ b/arch/arm64/kernel/vdso/Makefile @@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \ $(btildflags-y) -T ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18 -ccflags-y += -DDISABLE_BRANCH_PROFILING +ccflags-y += -DDISABLE_BRANCH_PROFILING -D__VDSO__ CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS) KBUILD_CFLAGS += $(DISABLE_LTO) From patchwork Wed Aug 19 12:13:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 11723845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD26A913 for ; Wed, 19 Aug 2020 12:14:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C242020738 for ; Wed, 19 Aug 2020 12:14:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839247; bh=HR11PdYBvh2kcllXkA7X72cUQrugXv/fyUKLaDrD1NM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=rgnerXxuX6e8qM2BUZlx+PMfGQ7ADBbYfaLKStObllZ2ZhIItn1P2NUpjH8nJ7ys2 k5Uudrsk4d5fi0kkSNGTMIkIfqGlF6QYTadtzA2Me1d2X3cyt5VKhRx38Nl1kEM8jn 4c3TYCx3VpUWC5z1R7JhQ9zC2ZocyByFjCzLfU8c= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727020AbgHSMOH (ORCPT ); Wed, 19 Aug 2020 08:14:07 -0400 Received: from mail.kernel.org ([198.145.29.99]:51344 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726752AbgHSMOE (ORCPT ); Wed, 19 Aug 2020 08:14:04 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E2BF0207BB; Wed, 19 Aug 2020 12:14:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839243; bh=HR11PdYBvh2kcllXkA7X72cUQrugXv/fyUKLaDrD1NM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uIiwEtfVwOI1891Y3mg15Z62mKkn2wU8U4oDMyYAsM+LxmedSKVGWbsnbN6t7FWi1 TyXxeu0E7sJrl9kF6zegPd7t5mQ3vVwjHNihK7hvU2yIVNu+uJa1EXVFioU+T2FLdX 6Ak2O8wTP4fcOkBKhu2Sd6vBL1YsRWHtqUYRMwEY= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 2/5] arm64: vdso: Add per-CPU data Date: Wed, 19 Aug 2020 13:13:15 +0100 Message-Id: <20200819121318.52158-3-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org In order to support a vDSO getcpu() implementation add per-CPU data to the vDSO data page. Do this by wrapping the generic vdso_data struct in an arm64 specific one with an array of per-CPU data. The offset of the per-CPU data applying to a CPU will be stored in TPIDRRO_EL0, this allows us to get to the per-CPU data without doing any multiplications. Since we currently only map a single data page for the vDSO but support very large numbers of CPUs TPIDRRO may be set to zero for CPUs which don't fit in the data page. This will also happen when KPTI is active since kernel_ventry uses TPIDRRO_EL0 as a scratch register in that case, add a comment to the code explaining this. Acessors for the data are provided in the header since they will be needed in multiple files and it seems neater to keep things together. Signed-off-by: Mark Brown --- arch/arm64/include/asm/processor.h | 12 +----- arch/arm64/include/asm/vdso/datapage.h | 54 ++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++++++- arch/arm64/kernel/vdso.c | 5 ++- 4 files changed, 83 insertions(+), 14 deletions(-) create mode 100644 arch/arm64/include/asm/vdso/datapage.h diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h index 240fe5e5b720..db7a804030b3 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -207,17 +207,7 @@ static inline void set_compat_ssbs_bit(struct pt_regs *regs) regs->pstate |= PSR_AA32_SSBS_BIT; } -static inline void start_thread(struct pt_regs *regs, unsigned long pc, - unsigned long sp) -{ - start_thread_common(regs, pc); - regs->pstate = PSR_MODE_EL0t; - - if (arm64_get_ssbd_state() != ARM64_SSBD_FORCE_ENABLE) - set_ssbs_bit(regs); - - regs->sp = sp; -} +void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp); static inline bool is_ttbr0_addr(unsigned long addr) { diff --git a/arch/arm64/include/asm/vdso/datapage.h b/arch/arm64/include/asm/vdso/datapage.h new file mode 100644 index 000000000000..e88d97238c52 --- /dev/null +++ b/arch/arm64/include/asm/vdso/datapage.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2020 ARM Limited + */ +#ifndef __ASM_VDSO_DATAPAGE_H +#define __ASM_VDSO_DATAPAGE_H + +#include + +struct vdso_cpu_data { + unsigned int cpu; + unsigned int node; +}; + +struct arm64_vdso_data { + /* Must be first in struct, we cast to vdso_data */ + struct vdso_data data[CS_BASES]; + struct vdso_cpu_data cpu_data[]; +}; + +#ifdef __VDSO__ +static inline struct vdso_cpu_data *__vdso_cpu_data(void) +{ + unsigned long offset; + + asm volatile( + "mrs %0, tpidrro_el0\n" + : "=r" (offset) + : + : "cc"); + + if (offset) + return (void *)(_vdso_data) + offset; + + return NULL; +} +#else +static inline size_t vdso_cpu_offset(void) +{ + size_t offset, data_end; + + offset = offsetof(struct arm64_vdso_data, cpu_data) + + smp_processor_id() * sizeof(struct vdso_cpu_data); + data_end = offset + sizeof(struct vdso_cpu_data) + 1; + + /* We only map a single page for vDSO data currently */ + if (data_end > PAGE_SIZE) + return 0; + + return offset; +} +#endif + +#endif diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c index 84ec630b8ab5..89b400f9397d 100644 --- a/arch/arm64/kernel/process.c +++ b/arch/arm64/kernel/process.c @@ -55,6 +55,7 @@ #include #include #include +#include #if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK) #include @@ -309,6 +310,28 @@ void show_regs(struct pt_regs * regs) dump_backtrace(regs, NULL, KERN_DEFAULT); } +void start_thread(struct pt_regs *regs, unsigned long pc, unsigned long sp) +{ + start_thread_common(regs, pc); + regs->pstate = PSR_MODE_EL0t; + + if (arm64_get_ssbd_state() != ARM64_SSBD_FORCE_ENABLE) + set_ssbs_bit(regs); + + regs->sp = sp; + + /* + * Store the vDSO per-CPU offset if supported. Disable + * preemption to make sure we read the CPU offset on the CPU + * we write it on. + */ + if (!arm64_kernel_unmapped_at_el0()) { + preempt_disable(); + write_sysreg(vdso_cpu_offset(), tpidrro_el0); + preempt_enable(); + } +} + static void tls_thread_flush(void) { write_sysreg(0, tpidr_el0); @@ -452,7 +475,8 @@ static void tls_thread_switch(struct task_struct *next) if (is_compat_thread(task_thread_info(next))) write_sysreg(next->thread.uw.tp_value, tpidrro_el0); else if (!arm64_kernel_unmapped_at_el0()) - write_sysreg(0, tpidrro_el0); + /* Used as scratch in KPTI trampoline so don't set here. */ + write_sysreg(vdso_cpu_offset(), tpidrro_el0); write_sysreg(*task_user_tls(next), tpidr_el0); } diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index d4202a32abc9..2a8d7ab76bee 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -28,6 +28,7 @@ #include #include #include +#include extern char vdso_start[], vdso_end[]; #ifdef CONFIG_COMPAT_VDSO @@ -77,10 +78,10 @@ static struct vdso_abi_info vdso_info[] __ro_after_init = { * The vDSO data page. */ static union { - struct vdso_data data[CS_BASES]; + struct arm64_vdso_data data; u8 page[PAGE_SIZE]; } vdso_data_store __page_aligned_data; -struct vdso_data *vdso_data = vdso_data_store.data; +struct vdso_data *vdso_data = vdso_data_store.data.data; static int __vdso_remap(enum vdso_abi abi, const struct vm_special_mapping *sm, From patchwork Wed Aug 19 12:13:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 11723847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A187913 for ; Wed, 19 Aug 2020 12:14:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 65C322076E for ; Wed, 19 Aug 2020 12:14:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839250; bh=Ftm+BgFLcJzi0bjTiz8IjEJjLINWXFqX3Rrl97pTpb8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=vaFL6RB89r8G4eTj6I5fC/X5eGxbnnv7TH/TQhxBNbuUu9t5oIl1YfkQJ9doHLoLq CVMA0ASJ0aSmM3MObc6BJ3ccwm3O1RNS30wHqD2PjNVBVcSL8KvkVHuPYtEBaoeWU2 QuKf6gtkgRaUBJ55u2LJLlcB2YCoZP3UgXAic5hU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726798AbgHSMOJ (ORCPT ); Wed, 19 Aug 2020 08:14:09 -0400 Received: from mail.kernel.org ([198.145.29.99]:51382 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727018AbgHSMOH (ORCPT ); Wed, 19 Aug 2020 08:14:07 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 874C6206DA; Wed, 19 Aug 2020 12:14:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839246; bh=Ftm+BgFLcJzi0bjTiz8IjEJjLINWXFqX3Rrl97pTpb8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hGdWwUzzLPnSon0iOKaq9SG/crBOPwsDLkl71Tmfh0jMLTfAf5jQ4WHrDimzHvqzI OLN3vtTxqe9E7ih91kchWM5gj1ra76v0Nyi+KCdk8cnu3GXosZ+xi8pK9yc/Yw0CjH GdTJGU0vhRSRF45VGT10z4h5NCYSpUPUYb7Hs2sA= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 3/5] arm64: vdso: Initialise the per-CPU vDSO data Date: Wed, 19 Aug 2020 13:13:16 +0100 Message-Id: <20200819121318.52158-4-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Register with the CPU hotplug system to initialise the per-CPU data for getcpu(). Signed-off-by: Mark Brown --- arch/arm64/kernel/vdso.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c index 2a8d7ab76bee..d9743c659341 100644 --- a/arch/arm64/kernel/vdso.c +++ b/arch/arm64/kernel/vdso.c @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -18,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -466,6 +468,26 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) } #endif /* CONFIG_COMPAT */ +static void vdso_cpu_init(void *p) +{ + struct arm64_vdso_data *data = (struct arm64_vdso_data *)vdso_data; + unsigned int cpu; + + if (vdso_cpu_offset()) { + cpu = smp_processor_id(); + + data->cpu_data[cpu].cpu = cpu; + data->cpu_data[cpu].node = cpu_to_node(cpu); + } +} + +static int vdso_cpu_online(unsigned int cpu) +{ + smp_call_function_single(cpu, vdso_cpu_init, NULL, 1); + + return 0; +} + static int vdso_mremap(const struct vm_special_mapping *sm, struct vm_area_struct *new_vma) { @@ -494,6 +516,12 @@ static int __init vdso_init(void) vdso_info[VDSO_ABI_AA64].dm = &aarch64_vdso_maps[AA64_MAP_VVAR]; vdso_info[VDSO_ABI_AA64].cm = &aarch64_vdso_maps[AA64_MAP_VDSO]; + /* + * Initialize per-CPU data, callback runs for all current and + * future CPUs. + */ + cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "vdso", vdso_cpu_online, NULL); + return __vdso_init(VDSO_ABI_AA64); } arch_initcall(vdso_init); From patchwork Wed Aug 19 12:13:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 11723849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A844E739 for ; Wed, 19 Aug 2020 12:14:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8DA86207BB for ; Wed, 19 Aug 2020 12:14:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839252; bh=KV4mnGXGCrFRun+EDN9OxgHUnFqbU7ZkJMYyr+CEL94=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=aKzuxucjyY8enfcBktoSkzglIL5SfZmtNkhgJr9nTbteLjxiRxrQhRjLzj9dZ6JD6 ZkYM1GJo4CVX6K0utXcUS9kdpXTCJJ5mBiLyrqmw12p/YhHDhaVnclroR58yEvE+rp CyBMG+aPkpq5b0YSMcgs3DQU0lE3n5ewMz4uzv/Q= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726952AbgHSMOK (ORCPT ); Wed, 19 Aug 2020 08:14:10 -0400 Received: from mail.kernel.org ([198.145.29.99]:51404 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726752AbgHSMOJ (ORCPT ); Wed, 19 Aug 2020 08:14:09 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1B60720738; Wed, 19 Aug 2020 12:14:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839248; bh=KV4mnGXGCrFRun+EDN9OxgHUnFqbU7ZkJMYyr+CEL94=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eTeToc67nVsh1gRKfuZxYg5Scyvc3Cr2zYdt+z2Q+WGLVMhzFeHmNU1s/qZSPTazs z0aAmBdhWsef0uqrpoauI/XkedNz8KOetq9wjnE7nR9P3ywxWUXJpp6t3Q8rtQ4RDW K/nrRyxQcW6z/jwcN8oXmds0QwkErAfmeo9c4gTI= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 4/5] arm64: vdso: Add getcpu() implementation Date: Wed, 19 Aug 2020 13:13:17 +0100 Message-Id: <20200819121318.52158-5-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org Some applications, especially trace ones, benefit from avoiding the syscall overhead on getcpu() calls so provide a vDSO implementation of it. Signed-off-by: Mark Brown --- arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vdso/vdso.lds.S | 1 + arch/arm64/kernel/vdso/vgetcpu.c | 48 +++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/vdso/vgetcpu.c diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile index 88cf0f0b91ed..ff350e69b8b6 100644 --- a/arch/arm64/kernel/vdso/Makefile +++ b/arch/arm64/kernel/vdso/Makefile @@ -11,7 +11,7 @@ ARCH_REL_TYPE_ABS := R_AARCH64_JUMP_SLOT|R_AARCH64_GLOB_DAT|R_AARCH64_ABS64 include $(srctree)/lib/vdso/Makefile -obj-vdso := vgettimeofday.o note.o sigreturn.o +obj-vdso := vgettimeofday.o note.o sigreturn.o vgetcpu.o # Build rules targets := $(obj-vdso) vdso.so vdso.so.dbg diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S index d808ad31e01f..ef3fb80e0349 100644 --- a/arch/arm64/kernel/vdso/vdso.lds.S +++ b/arch/arm64/kernel/vdso/vdso.lds.S @@ -80,6 +80,7 @@ VERSION __kernel_gettimeofday; __kernel_clock_gettime; __kernel_clock_getres; + __kernel_getcpu; local: *; }; } diff --git a/arch/arm64/kernel/vdso/vgetcpu.c b/arch/arm64/kernel/vdso/vgetcpu.c new file mode 100644 index 000000000000..e8972e561e08 --- /dev/null +++ b/arch/arm64/kernel/vdso/vgetcpu.c @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * ARM64 userspace implementations of getcpu() + * + * Copyright (C) 2020 ARM Limited + * + */ + +#include +#include + +struct getcpucache; + +static __always_inline +int getcpu_fallback(unsigned int *_cpu, unsigned int *_node, + struct getcpucache *_c) +{ + register unsigned int *cpu asm("x0") = _cpu; + register unsigned int *node asm("x1") = _node; + register struct getcpucache *c asm("x2") = _c; + register long ret asm ("x0"); + register long nr asm("x8") = __NR_getcpu; + + asm volatile( + " svc #0\n" + : "=r" (ret) + : "r" (cpu), "r" (node), "r" (c), "r" (nr) + : "memory"); + + return ret; +} + +int __kernel_getcpu(unsigned int *cpu, unsigned int *node, + struct getcpucache *c) +{ + struct vdso_cpu_data *cpu_data = __vdso_cpu_data(); + + if (cpu_data) { + if (cpu) + *cpu = cpu_data->cpu; + if (node) + *node = cpu_data->node; + + return 0; + } + + return getcpu_fallback(cpu, node, c); +} From patchwork Wed Aug 19 12:13:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 11723851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79A30739 for ; Wed, 19 Aug 2020 12:14:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 584592076E for ; Wed, 19 Aug 2020 12:14:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839253; bh=Qm1D4s+M/mO7mwIKm4XuWbPwM6fNbujUhQSw8qCmPVo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=ZYMvwldax3Tn2zNNE0NBMrcgGSEkq4Ue+re8kHSVc9UqJzOkKgM+HyUgSreK33LZs RKjKBunn459Yo/HmDSk+uPIMuvfTsspvO5UKlhcQlUuBTEUUwL45da+vlTM9wtZVMn DhcYkhVKb5nAQzeRTbIfHybkP5dXyokqbYQ3a+zM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726752AbgHSMON (ORCPT ); Wed, 19 Aug 2020 08:14:13 -0400 Received: from mail.kernel.org ([198.145.29.99]:51448 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727018AbgHSMOL (ORCPT ); Wed, 19 Aug 2020 08:14:11 -0400 Received: from localhost (fw-tnat.cambridge.arm.com [217.140.96.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AF5AE206DA; Wed, 19 Aug 2020 12:14:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1597839251; bh=Qm1D4s+M/mO7mwIKm4XuWbPwM6fNbujUhQSw8qCmPVo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tTpFukSQmJyyrAXX0yhMBRTggEOGVM7aeRXckhVLbgCQi3LgF58ukbdELkTN/v+/s UVIuUi9W3MAQZNMAuX/8iUqNsw1gyXwY0oqY3sCBfFFU5axWh8MshuvTA5yPuckMPp n3yXaa3UY9WcZFA8Szu0ifRmeBHC8Fmv0cRQU/Mg= From: Mark Brown To: Catalin Marinas , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, Vincenzo Frascino , linux-kselftest@vger.kernel.org, Shuah Khan , Mark Brown Subject: [PATCH v3 5/5] selftests: vdso: Support arm64 in getcpu() test Date: Wed, 19 Aug 2020 13:13:18 +0100 Message-Id: <20200819121318.52158-6-broonie@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200819121318.52158-1-broonie@kernel.org> References: <20200819121318.52158-1-broonie@kernel.org> MIME-Version: 1.0 Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org arm64 exports the vDSO ABI with a version of LINUX_2.6.39 and symbols prefixed with __kernel rather than __vdso. Update the getcpu() test to handle this. Signed-off-by: Mark Brown --- tools/testing/selftests/vDSO/vdso_test_getcpu.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tools/testing/selftests/vDSO/vdso_test_getcpu.c b/tools/testing/selftests/vDSO/vdso_test_getcpu.c index fc25ede131b8..4aeb65012b81 100644 --- a/tools/testing/selftests/vDSO/vdso_test_getcpu.c +++ b/tools/testing/selftests/vDSO/vdso_test_getcpu.c @@ -14,8 +14,18 @@ #include "../kselftest.h" #include "parse_vdso.h" +/* + * ARM64's vDSO exports its getcpu() implementation with a different + * name and version from other architectures, so we need to handle it + * as a special case. + */ +#if defined(__aarch64__) +const char *version = "LINUX_2.6.39"; +const char *name = "__kernel_getcpu"; +#else const char *version = "LINUX_2.6"; const char *name = "__vdso_getcpu"; +#endif struct getcpu_cache; typedef long (*getcpu_t)(unsigned int *, unsigned int *,