From patchwork Fri Mar 19 12:21:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150925 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B371BC43381 for ; Fri, 19 Mar 2021 12:22:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8375764F65 for ; Fri, 19 Mar 2021 12:22:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231549AbhCSMWW (ORCPT ); Fri, 19 Mar 2021 08:22:22 -0400 Received: from mail.kernel.org ([198.145.29.99]:33002 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231572AbhCSMWB (ORCPT ); Fri, 19 Mar 2021 08:22:01 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2F30064F77; Fri, 19 Mar 2021 12:21:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156520; bh=cuK8J78AbxfogMJXXKkEvrxRm/eRUIxTyi0pKFZ7yKc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qcZC5/d6k6CI6oaOf1k5SHJ0CHrLtWOwdNmADHo7pziau9yuPr1gCmkFT8shu/G1V 818EU+V8a4xshg/ZYRcFFHtoJKiYoCpy0K9k6qZ+HsXlJgMNw8YkmIswSBYP2XTYSO 6zvSu/tVfEpLWbrkBsogCPXFk7mYHKLGjFZs2b92rGEvmBt8ymIhbchr0frApuMkRU zb8AwSQj/OV7rdYRAbYAAGjbOmtYKWhQcATVvCSq3uGFF2uKcfhqvKjsyDxuj8evit 4mxMzAAE4weXwYD7mP2r9mhfdzH/oi714KZfHmb57YAQ+i4j4oiMhiEsXC9MCUQRJm 57GwdSiwEwnTQ== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 01/11] ia64: kprobes: Fix to pass correct trampoline address to the handler Date: Fri, 19 Mar 2021 21:21:55 +0900 Message-Id: <161615651484.306069.13374738532116114679.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Commit e792ff804f49 ("ia64: kprobes: Use generic kretprobe trampoline handler") missed to pass the wrong trampoline address (it passes the descriptor address instead of function entry address). This fixes it to pass correct trampoline address to __kretprobe_trampoline_handler(). This also changes to use correct symbol dereference function to get the function address from the kretprobe_trampoline. Fixes: e792ff804f49 ("ia64: kprobes: Use generic kretprobe trampoline handler") Signed-off-by: Masami Hiramatsu --- arch/ia64/kernel/kprobes.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index fc1ff8a4d7de..006fbc1d7ae9 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -398,7 +398,8 @@ static void kretprobe_trampoline(void) int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->cr_iip = __kretprobe_trampoline_handler(regs, kretprobe_trampoline, NULL); + regs->cr_iip = __kretprobe_trampoline_handler(regs, + dereference_function_descriptor(kretprobe_trampoline), NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler @@ -414,7 +415,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->b0 = ((struct fnptr *)kretprobe_trampoline)->ip; + regs->b0 = (unsigned long)dereference_function_descriptor(kretprobe_trampoline); } /* Check the instruction in the slot is break */ @@ -918,14 +919,14 @@ static struct kprobe trampoline_p = { int __init arch_init_kprobes(void) { trampoline_p.addr = - (kprobe_opcode_t *)((struct fnptr *)kretprobe_trampoline)->ip; + dereference_function_description(kretprobe_trampoline); return register_kprobe(&trampoline_p); } int __kprobes arch_trampoline_kprobe(struct kprobe *p) { if (p->addr == - (kprobe_opcode_t *)((struct fnptr *)kretprobe_trampoline)->ip) + dereference_function_descriptor(kretprobe_trampoline)) return 1; return 0; From patchwork Fri Mar 19 12:22:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150929 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7BAEC4332D for ; Fri, 19 Mar 2021 12:22:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B825964F72 for ; Fri, 19 Mar 2021 12:22:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231366AbhCSMWX (ORCPT ); Fri, 19 Mar 2021 08:22:23 -0400 Received: from mail.kernel.org ([198.145.29.99]:33078 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231704AbhCSMWM (ORCPT ); Fri, 19 Mar 2021 08:22:12 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id B38F16146D; Fri, 19 Mar 2021 12:22:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156531; bh=gA6qym6a5DphTZNgwFeBr65Ji0n4GlNkaWEMJsVYqfk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=S5UKPfCwSPvMg4G31EbkLfWnTRSWBgmOx3Y3j9UPfPdQhykg+9bWCS/H6XWfLW1/s nj3/tdyTr0mCssFsXozMdmdW8lwuQGBDUnArx/TWppOAOyKhSMT+OJjtT9MGmF4Dqt hSRmvzhcgUX9GqoCeWhBvpXIKwTmDrds6g8mAMcyhlsO9vsMnKK2+o99oa9GtU/Zfp YcA0SgnRh55L++9dTR1mSzNqBljYSwsVLZz3WyC6dkooTvilT13CXGLOktfhdgxMT3 fvtv9ACjxe0knuauQrRuXfQArHAUoQrTs7cyQnfOlPq4wtmdKzpayOtuFDSE3ZOaMB gU9mV1+E6nC0g== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 02/11] kprobes: treewide: Replace arch_deref_entry_point() with dereference_function_descriptor() Date: Fri, 19 Mar 2021 21:22:06 +0900 Message-Id: <161615652603.306069.15801129155904441412.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Replace arch_deref_entry_point() with dereference_function_descriptor() because those are doing same thing. Signed-off-by: Masami Hiramatsu --- arch/ia64/kernel/kprobes.c | 5 ----- arch/powerpc/kernel/kprobes.c | 11 ----------- include/linux/kprobes.h | 1 - kernel/kprobes.c | 7 +------ lib/error-inject.c | 3 ++- 5 files changed, 3 insertions(+), 24 deletions(-) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 006fbc1d7ae9..15871eb170c0 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -907,11 +907,6 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, return ret; } -unsigned long arch_deref_entry_point(void *entry) -{ - return ((struct fnptr *)entry)->ip; -} - static struct kprobe trampoline_p = { .pre_handler = trampoline_probe_handler }; diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index 01ab2163659e..eb0460949e1b 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -539,17 +539,6 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr) } NOKPROBE_SYMBOL(kprobe_fault_handler); -unsigned long arch_deref_entry_point(void *entry) -{ -#ifdef PPC64_ELF_ABI_v1 - if (!kernel_text_address((unsigned long)entry)) - return ppc_global_function_entry(entry); - else -#endif - return (unsigned long)entry; -} -NOKPROBE_SYMBOL(arch_deref_entry_point); - static struct kprobe trampoline_p = { .addr = (kprobe_opcode_t *) &kretprobe_trampoline, .pre_handler = trampoline_probe_handler diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 1883a4a9f16a..d65c041b5c22 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -390,7 +390,6 @@ int register_kprobe(struct kprobe *p); void unregister_kprobe(struct kprobe *p); int register_kprobes(struct kprobe **kps, int num); void unregister_kprobes(struct kprobe **kps, int num); -unsigned long arch_deref_entry_point(void *); int register_kretprobe(struct kretprobe *rp); void unregister_kretprobe(struct kretprobe *rp); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 745f08fdd7a6..2913de07f4a3 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1856,11 +1856,6 @@ static struct notifier_block kprobe_exceptions_nb = { .priority = 0x7fffffff /* we need to be notified first */ }; -unsigned long __weak arch_deref_entry_point(void *entry) -{ - return (unsigned long)entry; -} - #ifdef CONFIG_KRETPROBES unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, @@ -2324,7 +2319,7 @@ static int __init populate_kprobe_blacklist(unsigned long *start, int ret; for (iter = start; iter < end; iter++) { - entry = arch_deref_entry_point((void *)*iter); + entry = (unsigned long)dereference_function_descriptor((void *)*iter); ret = kprobe_add_ksym_blacklist(entry); if (ret == -EINVAL) continue; diff --git a/lib/error-inject.c b/lib/error-inject.c index c73651b15b76..f71875ac5f9f 100644 --- a/lib/error-inject.c +++ b/lib/error-inject.c @@ -8,6 +8,7 @@ #include #include #include +#include /* Whitelist of symbols that can be overridden for error injection. */ static LIST_HEAD(error_injection_list); @@ -64,7 +65,7 @@ static void populate_error_injection_list(struct error_injection_entry *start, mutex_lock(&ei_mutex); for (iter = start; iter < end; iter++) { - entry = arch_deref_entry_point((void *)iter->addr); + entry = (unsigned long)dereference_function_descriptor((void *)iter->addr); if (!kernel_text_address(entry) || !kallsyms_lookup_size_offset(entry, &size, &offset)) { From patchwork Fri Mar 19 12:22:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150933 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89030C433DB for ; Fri, 19 Mar 2021 12:23:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6099564F6A for ; Fri, 19 Mar 2021 12:23:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230108AbhCSMWt (ORCPT ); Fri, 19 Mar 2021 08:22:49 -0400 Received: from mail.kernel.org ([198.145.29.99]:33256 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230464AbhCSMWX (ORCPT ); Fri, 19 Mar 2021 08:22:23 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 90DF764F65; Fri, 19 Mar 2021 12:22:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156542; bh=+4EmNUqdJWP56LqWJMexLYGYsOE1auRLj9RaYGlSxKs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KcrxIsMMAtsma1zau5ULaGxxxHk9j8x3LympizOFVuKCdyJ7gjpqQtnbZ5v5YQ1sZ ly9T6j8Z55yOQgd+Z/DCMk91uv+O9gMpkfYdoVNHqUxzvMzEUOKfoy6niypOVXfWHK bmy0rd46ktoYaVJ9vSfTA7u8MFcMorZr9AGf8vV7oIdv/0XFbmPcCIZBcQbaEMhHiV ajjrZsQnzGfTXdYBOp1ay9bSSBVk4xu84/iyrXDk3Hrr1XgFO3wAKK3ya0+JxVozxt O7cpniUJfvh4ZJYrfxu9ZElCZH7hKFVzcVmCUoaZyD/aQqo4Ts0IB2ifLY2E+cgt/C nlMJQaCTY6cZg== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 03/11] kprobes: treewide: Remove trampoline_address from kretprobe_trampoline_handler() Date: Fri, 19 Mar 2021 21:22:17 +0900 Message-Id: <161615653716.306069.4625985536508261030.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Remove trampoline_address from kretprobe_trampoline_handler(). Instead of passing the address, kretprobe_trampoline_handler() can use new kretprobe_trampoline_addr(). Signed-off-by: Masami Hiramatsu --- Changes in v3: - Remove wrong kretprobe_trampoline declaration from arch/x86/include/asm/kprobes.h. Changes in v2: - Remove arch_deref_entry_point() from comment. --- arch/arc/kernel/kprobes.c | 2 +- arch/arm/probes/kprobes/core.c | 3 +-- arch/arm64/kernel/probes/kprobes.c | 3 +-- arch/csky/kernel/probes/kprobes.c | 2 +- arch/ia64/kernel/kprobes.c | 5 ++--- arch/mips/kernel/kprobes.c | 3 +-- arch/parisc/kernel/kprobes.c | 4 ++-- arch/powerpc/kernel/kprobes.c | 2 +- arch/riscv/kernel/probes/kprobes.c | 2 +- arch/s390/kernel/kprobes.c | 2 +- arch/sh/kernel/kprobes.c | 2 +- arch/sparc/kernel/kprobes.c | 2 +- arch/x86/include/asm/kprobes.h | 1 - arch/x86/kernel/kprobes/core.c | 2 +- include/linux/kprobes.h | 18 +++++++++++++----- kernel/kprobes.c | 3 +-- 16 files changed, 29 insertions(+), 27 deletions(-) diff --git a/arch/arc/kernel/kprobes.c b/arch/arc/kernel/kprobes.c index cabef45f11df..3ae01bb5820c 100644 --- a/arch/arc/kernel/kprobes.c +++ b/arch/arc/kernel/kprobes.c @@ -397,7 +397,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, static int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->ret = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->ret = __kretprobe_trampoline_handler(regs, NULL); /* By returning a non zero value, we are telling the kprobe handler * that we don't want the post_handler to run diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index a9653117ca0d..1782b41df095 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -413,8 +413,7 @@ void __naked __kprobes kretprobe_trampoline(void) /* Called from kretprobe_trampoline */ static __used __kprobes void *trampoline_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, - (void *)regs->ARM_fp); + return (void *)kretprobe_trampoline_handler(regs, (void *)regs->ARM_fp); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c index 66aac2881ba8..fce681fdfce6 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -412,8 +412,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, - (void *)kernel_stack_pointer(regs)); + return (void *)kretprobe_trampoline_handler(regs, (void *)kernel_stack_pointer(regs)); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c index 589f090f48b9..cc589bc11904 100644 --- a/arch/csky/kernel/probes/kprobes.c +++ b/arch/csky/kernel/probes/kprobes.c @@ -404,7 +404,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + return (void *)kretprobe_trampoline_handler(regs, NULL); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 15871eb170c0..a008df8e7203 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -392,14 +392,13 @@ static void __kprobes set_current_kprobe(struct kprobe *p, __this_cpu_write(current_kprobe, p); } -static void kretprobe_trampoline(void) +void kretprobe_trampoline(void) { } int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->cr_iip = __kretprobe_trampoline_handler(regs, - dereference_function_descriptor(kretprobe_trampoline), NULL); + regs->cr_iip = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/mips/kernel/kprobes.c b/arch/mips/kernel/kprobes.c index 54dfba8fa77c..001a2f07ef44 100644 --- a/arch/mips/kernel/kprobes.c +++ b/arch/mips/kernel/kprobes.c @@ -489,8 +489,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, static int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - instruction_pointer(regs) = __kretprobe_trampoline_handler(regs, - kretprobe_trampoline, NULL); + instruction_pointer(regs) = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/parisc/kernel/kprobes.c b/arch/parisc/kernel/kprobes.c index 6d21a515eea5..4a35ac6e2ca2 100644 --- a/arch/parisc/kernel/kprobes.c +++ b/arch/parisc/kernel/kprobes.c @@ -175,7 +175,7 @@ int __kprobes parisc_kprobe_ss_handler(struct pt_regs *regs) return 1; } -static inline void kretprobe_trampoline(void) +void kretprobe_trampoline(void) { asm volatile("nop"); asm volatile("nop"); @@ -193,7 +193,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, { unsigned long orig_ret_address; - orig_ret_address = __kretprobe_trampoline_handler(regs, trampoline_p.addr, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); instruction_pointer_set(regs, orig_ret_address); return 1; diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index eb0460949e1b..dfd532c43525 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -399,7 +399,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { unsigned long orig_ret_address; - orig_ret_address = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); /* * We get here through one of two paths: * 1. by taking a trap -> kprobe_handler() -> here diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index a2ec18662fee..619339f1d3ba 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -376,7 +376,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + return (void *)kretprobe_trampoline_handler(regs, NULL); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c index aae24dc75df6..b149e9169709 100644 --- a/arch/s390/kernel/kprobes.c +++ b/arch/s390/kernel/kprobes.c @@ -351,7 +351,7 @@ static void __used kretprobe_trampoline_holder(void) */ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->psw.addr = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->psw.addr = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/sh/kernel/kprobes.c b/arch/sh/kernel/kprobes.c index 756100b01e84..48356e81836a 100644 --- a/arch/sh/kernel/kprobes.c +++ b/arch/sh/kernel/kprobes.c @@ -303,7 +303,7 @@ static void __used kretprobe_trampoline_holder(void) */ int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->pc = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->pc = __kretprobe_trampoline_handler(regs, NULL); return 1; } diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c index 217c21a6986a..fa30f9dadff8 100644 --- a/arch/sparc/kernel/kprobes.c +++ b/arch/sparc/kernel/kprobes.c @@ -468,7 +468,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, { unsigned long orig_ret_address = 0; - orig_ret_address = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); regs->tpc = orig_ret_address; regs->tnpc = orig_ret_address + 4; diff --git a/arch/x86/include/asm/kprobes.h b/arch/x86/include/asm/kprobes.h index d20a3d6be36e..bec07252a0a3 100644 --- a/arch/x86/include/asm/kprobes.h +++ b/arch/x86/include/asm/kprobes.h @@ -49,7 +49,6 @@ extern __visible kprobe_opcode_t optprobe_template_end[]; extern const int kretprobe_blacklist_size; void arch_remove_kprobe(struct kprobe *p); -asmlinkage void kretprobe_trampoline(void); extern void arch_kprobe_override_function(struct pt_regs *regs); diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 71e91f16720c..427d648fffcd 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -811,7 +811,7 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) regs->ip = (unsigned long)&kretprobe_trampoline; regs->orig_ax = ~0UL; - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, ®s->sp); + return (void *)kretprobe_trampoline_handler(regs, ®s->sp); } NOKPROBE_SYMBOL(trampoline_handler); diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index d65c041b5c22..65dadd4238a2 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -205,15 +205,23 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); +void kretprobe_trampoline(void); +/* + * Since some architecture uses structured function pointer, + * use dereference_function_descriptor() to get real function address. + */ +static nokprobe_inline void *kretprobe_trampoline_addr(void) +{ + return dereference_function_descriptor(kretprobe_trampoline); +} + /* If the trampoline handler called from a kprobe, use this version */ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, - void *frame_pointer); + void *frame_pointer); static nokprobe_inline unsigned long kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, - void *frame_pointer) + void *frame_pointer) { unsigned long ret; /* @@ -222,7 +230,7 @@ unsigned long kretprobe_trampoline_handler(struct pt_regs *regs, * be running at this point. */ kprobe_busy_begin(); - ret = __kretprobe_trampoline_handler(regs, trampoline_address, frame_pointer); + ret = __kretprobe_trampoline_handler(regs, frame_pointer); kprobe_busy_end(); return ret; diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 2913de07f4a3..75c0a58c19c2 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1859,7 +1859,6 @@ static struct notifier_block kprobe_exceptions_nb = { #ifdef CONFIG_KRETPROBES unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, void *frame_pointer) { kprobe_opcode_t *correct_ret_addr = NULL; @@ -1874,7 +1873,7 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, BUG_ON(ri->fp != frame_pointer); - if (ri->ret_addr != trampoline_address) { + if (ri->ret_addr != kretprobe_trampoline_addr()) { correct_ret_addr = ri->ret_addr; /* * This is the real return address. Any other From patchwork Fri Mar 19 12:22:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150931 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDC16C433E0 for ; Fri, 19 Mar 2021 12:23:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7BE2E64F7A for ; Fri, 19 Mar 2021 12:23:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230153AbhCSMWu (ORCPT ); Fri, 19 Mar 2021 08:22:50 -0400 Received: from mail.kernel.org ([198.145.29.99]:33718 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231583AbhCSMWe (ORCPT ); Fri, 19 Mar 2021 08:22:34 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 85A8364F77; Fri, 19 Mar 2021 12:22:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156554; bh=BmKp/FA0g7ZOg1oLsp7e01PpVXipfwCQdHRKfNzoDDM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHydWS/gxndxS4SVlDM4TQ6N9SyKuRC65/D3U4fmIsWWQBqe/VzqOPgq1MFimtm70 7htJWG3A3uC8YgHjqTd4cn7PadqAVp87GWm2tyfPculEOJStvi4vDxVH/QOcYY3LAI csOO++eji+POL2QNeLlS3g40QCdGnurNN2tI4/CS6gXv1eUu2LMDEnPef+yQFzwiGA Rw4RS3rHiZ9Ke20MO+8WrRRBf6bh6Wd89ch4MOU80BlUBSxvJcua+5tP2jopeunzx0 wm0+GjC4TZFi9D8acHUccLPEgdX3qG94yb5379ih/6h1Dj/CoENipvQuUo00nceNlQ Y4ZUVY7A4DLtw== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 04/11] kprobes: Add kretprobe_find_ret_addr() for searching return address Date: Fri, 19 Mar 2021 21:22:28 +0900 Message-Id: <161615654801.306069.1595034827570242102.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add kretprobe_find_ret_addr() for searching correct return address from kretprobe instance list. Signed-off-by: Masami Hiramatsu --- Changes in v3: - Remove generic stacktrace fixup. Instead, it should be solved in each unwinder. This just provide the generic interface. Changes in v2: - Add is_kretprobe_trampoline() for checking address outside of kretprobe_find_ret_addr() - Remove unneeded addr from kretprobe_find_ret_addr() - Rename fixup_kretprobe_tramp_addr() to fixup_kretprobe_trampoline() --- include/linux/kprobes.h | 22 +++++++++++ kernel/kprobes.c | 90 +++++++++++++++++++++++++++++++++-------------- 2 files changed, 86 insertions(+), 26 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 65dadd4238a2..f530f82a046d 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -215,6 +215,14 @@ static nokprobe_inline void *kretprobe_trampoline_addr(void) return dereference_function_descriptor(kretprobe_trampoline); } +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return (void *)addr == kretprobe_trampoline_addr(); +} + +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur); + /* If the trampoline handler called from a kprobe, use this version */ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, void *frame_pointer); @@ -514,6 +522,20 @@ static inline bool is_kprobe_optinsn_slot(unsigned long addr) } #endif +#if !defined(CONFIG_KRETPROBES) +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return false; +} + +static nokprobe_inline +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + return 0; +} +#endif + /* Returns true if kprobes handled the fault */ static nokprobe_inline bool kprobe_page_fault(struct pt_regs *regs, unsigned int trap) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 75c0a58c19c2..cf19edc038e4 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1858,45 +1858,68 @@ static struct notifier_block kprobe_exceptions_nb = { #ifdef CONFIG_KRETPROBES -unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *frame_pointer) +/* This assumes the tsk is current or the task which is not running. */ +static unsigned long __kretprobe_find_ret_addr(struct task_struct *tsk, + struct llist_node **cur) { - kprobe_opcode_t *correct_ret_addr = NULL; struct kretprobe_instance *ri = NULL; - struct llist_node *first, *node; - struct kretprobe *rp; + struct llist_node *node = *cur; + + if (!node) + node = tsk->kretprobe_instances.first; + else + node = node->next; - /* Find all nodes for this frame. */ - first = node = current->kretprobe_instances.first; while (node) { ri = container_of(node, struct kretprobe_instance, llist); - - BUG_ON(ri->fp != frame_pointer); - if (ri->ret_addr != kretprobe_trampoline_addr()) { - correct_ret_addr = ri->ret_addr; - /* - * This is the real return address. Any other - * instances associated with this task are for - * other calls deeper on the call stack - */ - goto found; + *cur = node; + return (unsigned long)ri->ret_addr; } - node = node->next; } - pr_err("Oops! Kretprobe fails to find correct return address.\n"); - BUG_ON(1); + return 0; +} +NOKPROBE_SYMBOL(__kretprobe_find_ret_addr); -found: - /* Unlink all nodes for this frame. */ - current->kretprobe_instances.first = node->next; - node->next = NULL; +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + struct kretprobe_instance *ri = NULL; + unsigned long ret; + + do { + ret = __kretprobe_find_ret_addr(tsk, cur); + if (!ret) + return ret; + ri = container_of(*cur, struct kretprobe_instance, llist); + } while (ri->fp != fp); + + return ret; +} +NOKPROBE_SYMBOL(kretprobe_find_ret_addr); - /* Run them.. */ +unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, + void *frame_pointer) +{ + kprobe_opcode_t *correct_ret_addr = NULL; + struct kretprobe_instance *ri = NULL; + struct llist_node *first, *node = NULL; + struct kretprobe *rp; + + /* Find correct address and all nodes for this frame. */ + correct_ret_addr = (void *)__kretprobe_find_ret_addr(current, &node); + if (!correct_ret_addr) { + pr_err("Oops! Kretprobe fails to find correct return address.\n"); + BUG_ON(1); + } + + /* Run them. */ + first = current->kretprobe_instances.first; while (first) { ri = container_of(first, struct kretprobe_instance, llist); - first = first->next; + + BUG_ON(ri->fp != frame_pointer); rp = get_kretprobe(ri); if (rp && rp->handler) { @@ -1907,6 +1930,21 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, rp->handler(ri, regs); __this_cpu_write(current_kprobe, prev); } + if (first == node) + break; + + first = first->next; + } + + /* Unlink all nodes for this frame. */ + first = current->kretprobe_instances.first; + current->kretprobe_instances.first = node->next; + node->next = NULL; + + /* Recycle them. */ + while (first) { + ri = container_of(first, struct kretprobe_instance, llist); + first = first->next; recycle_rp_inst(ri); } From patchwork Fri Mar 19 12:22:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150935 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC499C433E6 for ; Fri, 19 Mar 2021 12:23:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A4DAF64F81 for ; Fri, 19 Mar 2021 12:23:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230092AbhCSMWu (ORCPT ); Fri, 19 Mar 2021 08:22:50 -0400 Received: from mail.kernel.org ([198.145.29.99]:33780 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230134AbhCSMWo (ORCPT ); Fri, 19 Mar 2021 08:22:44 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id AD0D56146D; Fri, 19 Mar 2021 12:22:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156564; bh=ukedJXvpQTbwD5ndIACI9Ikh1mH7iQqGENUZObVcIwU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gQXlPgdGSdAwo/qrfXPCoR+3vURgm8A00MpzX042Oe1kgE4C1KtRI9DjAOTXpVqgM Ih3jwfoF6KheL/9Yf0aOVO5R3NgekjdZXSKntZmAKXcKL09fsVboSILwYo0hfVTvzT BSgzwB1Co8Xhki8hc0e2o3uNfOu/VpqQx64P0Guzx+rU1Izm4yUBm3lsOykOUvEgOl iBNNl4q3d0n3KjgBVZFhCzfeqgnv4xG7NBMG/IyYNV2zyWV5ZyEYWUAXLzvXuc5edA gsyar+YXWUZCuNhfnBPJ+WrRdGj/sm/m9MjfMIVFyUdJpA15lvUGkFrmQErSma4v7O xEA0SbYZdiRlQ== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 05/11] x86/kprobes: Add UNWIND_HINT_FUNC on kretprobe_trampoline code Date: Fri, 19 Mar 2021 21:22:39 +0900 Message-Id: <161615655969.306069.4545805781593088526.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Josh Poimboeuf Add UNWIND_HINT_FUNC on kretporbe_trampoline code so that ORC information is generated on the kretprobe_trampoline correctly. Signed-off-by: Josh Poimboeuf --- [MH] Add patch description. --- arch/x86/include/asm/unwind_hints.h | 5 +++++ arch/x86/kernel/kprobes/core.c | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/unwind_hints.h b/arch/x86/include/asm/unwind_hints.h index 8e574c0afef8..8b33674288ea 100644 --- a/arch/x86/include/asm/unwind_hints.h +++ b/arch/x86/include/asm/unwind_hints.h @@ -52,6 +52,11 @@ UNWIND_HINT sp_reg=ORC_REG_SP sp_offset=8 type=UNWIND_HINT_TYPE_FUNC .endm +#else + +#define UNWIND_HINT_FUNC \ + UNWIND_HINT(ORC_REG_SP, 8, UNWIND_HINT_TYPE_FUNC, 0) + #endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_UNWIND_HINTS_H */ diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 427d648fffcd..b31058a152b6 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -772,6 +772,7 @@ asm( /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 " pushq %rsp\n" + UNWIND_HINT_FUNC " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" @@ -782,6 +783,7 @@ asm( " popfq\n" #else " pushl %esp\n" + UNWIND_HINT_FUNC " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" @@ -795,7 +797,6 @@ asm( ".size kretprobe_trampoline, .-kretprobe_trampoline\n" ); NOKPROBE_SYMBOL(kretprobe_trampoline); -STACK_FRAME_NON_STANDARD(kretprobe_trampoline); /* From patchwork Fri Mar 19 12:22:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150937 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AEBEC433E0 for ; Fri, 19 Mar 2021 12:23:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 286F664F65 for ; Fri, 19 Mar 2021 12:23:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230232AbhCSMXV (ORCPT ); Fri, 19 Mar 2021 08:23:21 -0400 Received: from mail.kernel.org ([198.145.29.99]:34148 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230193AbhCSMW4 (ORCPT ); Fri, 19 Mar 2021 08:22:56 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 8B1B464F6E; Fri, 19 Mar 2021 12:22:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156575; bh=KEe5WdEyJHtzuPMGzpiocUmfrX3xOnunn5dZvpKwBh8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qQwuK7XodwAna+Q6RlxmMUWFGOK5ajES/260IfJAiNcptiryhGhAC5RgbXgo6b8PK LuBWiqA9+NUAajTfPMJFNGXN5HzsfTjW4rAxgVQD842rLLxhRslaZqt9R94atO/YZd cYGiJW8GlxV3q7uKBq0oYJNWiRlUM2GDlPYo5T2Iacx6Ii8V+ELwmqsLY4oN9MS25G TB5IUlowKhxtthtTG+ry9qLmEqUv9ItL6KirBQWYxZubesNB8e+MlLwxkbyOzY51fZ /STsQKUSit4M9awIhXwrElqqo/EQaAGY8dPUA4Z+SCVpzAdF1AudGL2cxpiyBmkyel /feVumaEud33w== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 06/11] ARC: Add instruction_pointer_set() API Date: Fri, 19 Mar 2021 21:22:49 +0900 Message-Id: <161615656974.306069.13884955082773738658.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add instruction_pointer_set() API for arc. Signed-off-by: Masami Hiramatsu --- arch/arc/include/asm/ptrace.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arc/include/asm/ptrace.h b/arch/arc/include/asm/ptrace.h index 4c3c9be5bd16..cca8d6583e31 100644 --- a/arch/arc/include/asm/ptrace.h +++ b/arch/arc/include/asm/ptrace.h @@ -149,6 +149,11 @@ static inline long regs_return_value(struct pt_regs *regs) return (long)regs->r0; } +static inline void instruction_pointer_set(struct pt_regs *regs, + unsigned long val) +{ + instruction_pointer(regs) = val; +} #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PTRACE_H */ From patchwork Fri Mar 19 12:23:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150939 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5171C433DB for ; Fri, 19 Mar 2021 12:23:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7BD5F64F7A for ; Fri, 19 Mar 2021 12:23:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230008AbhCSMXW (ORCPT ); Fri, 19 Mar 2021 08:23:22 -0400 Received: from mail.kernel.org ([198.145.29.99]:34202 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229949AbhCSMXG (ORCPT ); Fri, 19 Mar 2021 08:23:06 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 1BCC664F65; Fri, 19 Mar 2021 12:23:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156586; bh=l1g8PZb9w6cDVZuCPqZ5YvMo82/g26s+RTh8ZpDtrXY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DZd1USBQeRDVGrzqTlRWFu1Q3EG5KSu9zyZMyJ2JWjh/NC2AzspiISOyseVrSjWaW WpgzfBdQluWMtWxDnEAmS4cCcZyknxiD+J+8n0euCahvgSu1N2W/lMT2l9KKP8vVdO ljlWWd0fiZgjtoBrywzLnpmL9j4kN6cHgb/Wv4KK1UJUCgDnMiuCMCF1ikJFmkVSEE zVJlzZ8B3wcPRa2FePhmrRAjsAv1eFIFgQysmdrn1hFiqPHbPUIU1cogR7UH6/sdvB yFL5pknnqeiLdT5uayOCE//c4NqmL0317ggdDhxz9mQALGYsOc33081MmxfpqfxgWf A+Jjzw+9AIiWQ== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 07/11] ia64: Add instruction_pointer_set() API Date: Fri, 19 Mar 2021 21:23:01 +0900 Message-Id: <161615658087.306069.12036720803234007510.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add instruction_pointer_set() API for ia64. Signed-off-by: Masami Hiramatsu --- arch/ia64/include/asm/ptrace.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/ia64/include/asm/ptrace.h b/arch/ia64/include/asm/ptrace.h index b3aa46090101..e382f1a6bff3 100644 --- a/arch/ia64/include/asm/ptrace.h +++ b/arch/ia64/include/asm/ptrace.h @@ -45,6 +45,7 @@ #include #include +# define ia64_psr(regs) ((struct ia64_psr *) &(regs)->cr_ipsr) /* * We use the ia64_psr(regs)->ri to determine which of the three * instructions in bundle (16 bytes) took the sample. Generate @@ -71,6 +72,12 @@ static inline long regs_return_value(struct pt_regs *regs) return -regs->r8; } +static inline void instruction_pointer_set(struct pt_regs *regs, unsigned long val) +{ + ia64_psr(regs)->ri = (val & 0xf); + regs->cr_iip = (val & ~0xfULL); +} + /* Conserve space in histogram by encoding slot bits in address * bits 2 and 3 rather than bits 0 and 1. */ @@ -87,7 +94,6 @@ static inline long regs_return_value(struct pt_regs *regs) /* given a pointer to a task_struct, return the user's pt_regs */ # define task_pt_regs(t) (((struct pt_regs *) ((char *) (t) + IA64_STK_OFFSET)) - 1) -# define ia64_psr(regs) ((struct ia64_psr *) &(regs)->cr_ipsr) # define user_mode(regs) (((struct ia64_psr *) &(regs)->cr_ipsr)->cpl != 0) # define user_stack(task,regs) ((long) regs - (long) task == IA64_STK_OFFSET - sizeof(*regs)) # define fsys_mode(task,regs) \ From patchwork Fri Mar 19 12:23:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150941 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8EBBC433E6 for ; Fri, 19 Mar 2021 12:23:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 91F5664F79 for ; Fri, 19 Mar 2021 12:23:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230173AbhCSMXW (ORCPT ); Fri, 19 Mar 2021 08:23:22 -0400 Received: from mail.kernel.org ([198.145.29.99]:34246 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230268AbhCSMXS (ORCPT ); Fri, 19 Mar 2021 08:23:18 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id EBA656146D; Fri, 19 Mar 2021 12:23:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156597; bh=LcrhRWJtXgb05ASQlIQZ+Hz25N0QWwcOZr1A52Hlftk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A2ANLwl7s4csgnpWOozqQPHMjY0Y/fv3o7vPE+Q6STc39tJMsidi39RDg7W5cOP29 8lx1xDG7U6kTc7zInqhdYhuR/+Jp9vwK9Sk0Xe9V4Ykgfz8PGXOSROLxw1cbYs95pQ keEuOc4l64aTGI9UQ1xVBPlZe79or5OOcX3HEasjaptN2SUINDXm3W3IQU0Sw8xk7R mz1kLe0+u/hnFFIaOLx89V8OlAUoR89TJ/ONaHLh2hTenZv2PPt6bW8wyxgFgMyeoO XbirmkqnXQxE86ljJ2uvJGD/YSIznajrjbkPQ809YYdtITliyslb+4NXbFDl5Gfznw v8IIE++lgMH2g== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 08/11] kprobes: Setup instruction pointer in __kretprobe_trampoline_handler Date: Fri, 19 Mar 2021 21:23:12 +0900 Message-Id: <161615659174.306069.12736134222759644948.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org To simplify the stacktrace with pt_regs from kretprobe handler, set the correct return address to the instruction pointer in the pt_regs before calling kretprobe handlers. Suggested-by: Josh Poimboeuf Signed-off-by: Masami Hiramatsu --- Changes in v3: - Cast the correct_ret_addr to unsigned long. --- kernel/kprobes.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index cf19edc038e4..4ce3e6f5d28d 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1914,6 +1914,9 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, BUG_ON(1); } + /* Set the instruction pointer to the correct address */ + instruction_pointer_set(regs, (unsigned long)correct_ret_addr); + /* Run them. */ first = current->kretprobe_instances.first; while (first) { From patchwork Fri Mar 19 12:23:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150943 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 982A6C433E0 for ; Fri, 19 Mar 2021 12:24:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6E4EF64F65 for ; Fri, 19 Mar 2021 12:24:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230368AbhCSMXy (ORCPT ); Fri, 19 Mar 2021 08:23:54 -0400 Received: from mail.kernel.org ([198.145.29.99]:34446 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230268AbhCSMX3 (ORCPT ); Fri, 19 Mar 2021 08:23:29 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 3DAEE64F6E; Fri, 19 Mar 2021 12:23:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156609; bh=nqRMt7HfWF4IDIt0k4NM1ml7U8SIpubXRyNN+jZYq1E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i3CpqxgHAR2iVT5IyvMv5xg7u+IVgga3nWpzvtzarH5j4dM1ZYLcQQ2YC9AJ5IUI5 0F43sf1CWoE373EQyCe+7nUiExUPxXN3N11nhQqy7018N9SEvLRrlLvYqZOxmzS1yZ DY5i0MLw8Y6cXztIIGd0jYwqnQNBkmcpNweZTj88LoxxeOc47BI0MmlRFD1bTTj6PF 0nl+p21SY5urIsYetpMpvA57uC7neZpTLtKI+hbojrCNj4IUrKgll4C2vY68YzEK9d i/BjQbPJW3yU1Yf+BpFlUu1BjWPYT6Jvz3xaw0A8JyJT1lCnS5SB017IRGMb3tMNmK Zx6jndGdRncJA== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 09/11] x86/kprobes: Push a fake return address at kretprobe_trampoline Date: Fri, 19 Mar 2021 21:23:23 +0900 Message-Id: <161615660310.306069.8018914961713221579.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This changes x86/kretprobe stack frame on kretprobe_trampoline a bit, which now push the kretprobe_trampoline as a fake return address at the bottom of the stack frame. With this fix, the ORC unwinder will see the kretprobe_trampoline as a return address. Signed-off-by: Masami Hiramatsu Suggested-by: Josh Poimboeuf --- arch/x86/kernel/kprobes/core.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index b31058a152b6..7e7b854e0fdf 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -769,28 +769,31 @@ asm( ".global kretprobe_trampoline\n" ".type kretprobe_trampoline, @function\n" "kretprobe_trampoline:\n" - /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 - " pushq %rsp\n" + /* Push fake return address to tell the unwinder it's a kretprobe */ + " pushq $kretprobe_trampoline\n" UNWIND_HINT_FUNC + /* Save the sp-8, this will be fixed later */ + " pushq %rsp\n" " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" " call trampoline_handler\n" - /* Replace saved sp with true return address. */ - " movq %rax, 19*8(%rsp)\n" RESTORE_REGS_STRING + " addq $8, %rsp\n" " popfq\n" #else - " pushl %esp\n" + /* Push fake return address to tell the unwinder it's a kretprobe */ + " pushl $kretprobe_trampoline\n" UNWIND_HINT_FUNC + /* Save the sp-4, this will be fixed later */ + " pushl %esp\n" " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" " call trampoline_handler\n" - /* Replace saved sp with true return address. */ - " movl %eax, 15*4(%esp)\n" RESTORE_REGS_STRING + " addl $4, %esp\n" " popfl\n" #endif " ret\n" @@ -802,8 +805,10 @@ NOKPROBE_SYMBOL(kretprobe_trampoline); /* * Called from kretprobe_trampoline */ -__used __visible void *trampoline_handler(struct pt_regs *regs) +__used __visible void trampoline_handler(struct pt_regs *regs) { + unsigned long *frame_pointer; + /* fixup registers */ regs->cs = __KERNEL_CS; #ifdef CONFIG_X86_32 @@ -811,8 +816,16 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) #endif regs->ip = (unsigned long)&kretprobe_trampoline; regs->orig_ax = ~0UL; + regs->sp += sizeof(long); + frame_pointer = ((unsigned long *)®s->sp) + 1; - return (void *)kretprobe_trampoline_handler(regs, ®s->sp); + /* Replace fake return address with real one. */ + *frame_pointer = kretprobe_trampoline_handler(regs, frame_pointer); + /* + * Move flags to sp so that kretprobe_trapmoline can return + * right after popf. + */ + regs->sp = regs->flags; } NOKPROBE_SYMBOL(trampoline_handler); From patchwork Fri Mar 19 12:23:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150947 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E44B3C433E6 for ; Fri, 19 Mar 2021 12:24:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B7F6164F65 for ; Fri, 19 Mar 2021 12:24:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230051AbhCSMXz (ORCPT ); Fri, 19 Mar 2021 08:23:55 -0400 Received: from mail.kernel.org ([198.145.29.99]:34496 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230306AbhCSMXj (ORCPT ); Fri, 19 Mar 2021 08:23:39 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id DC48A6146D; Fri, 19 Mar 2021 12:23:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156619; bh=dUWgkQqcROXVRy2P/Kgm2tfPQp3l6P6++YBCLK7lZck=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OIWhbuoYYqdWc84fs2kvqWkNChGP+jkhvZf9t2UWOvAtEyDyZck924zwIKT73oGp+ vRjbo2G0SzlnboMaLly9nlBUnt4XGOcWlyJWvBc/ZmDUFTKP8iWU2c4+X5vxF9bHY2 8rbxH8PMh8HLUGrZU/e2fZMvK/xAvOFiDJVk0NiU1q3XbH3B+ciyhZJKpBUDNRuNlr FrbNONqbss1BLkTOkWvuqyRbDoC1VBoAkRK515oBjMEHFmxxYJWIP/DtACefRdqjt7 0qsiziZF1L6ruXujFyY7U8YYddCjXAZ6wzb9Ktq5af0I0bezByE7FhHFdmfFO+NSrI qf/x7vRc/kfBQ== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 10/11] x86/unwind: Recover kretprobe trampoline entry Date: Fri, 19 Mar 2021 21:23:34 +0900 Message-Id: <161615661459.306069.7790897385197866745.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Since the kretprobe replaces the function return address with the kretprobe_trampoline on the stack, x86 unwinders can not continue the stack unwinding at that point, or record kretprobe_trampoline instead of correct return address. To fix this issue, find the correct return address from task's kretprobe_instances as like as function-graph tracer does. With this fix, the unwinder can correctly unwind the stack from kretprobe event on x86, as below. <...>-135 [003] ...1 6.722338: r_full_proxy_read_0: (vfs_read+0xab/0x1a0 <- full_proxy_read) <...>-135 [003] ...1 6.722377: => kretprobe_trace_func+0x209/0x2f0 => kretprobe_dispatcher+0x4a/0x70 => __kretprobe_trampoline_handler+0xca/0x150 => trampoline_handler+0x44/0x70 => kretprobe_trampoline+0x2a/0x50 => vfs_read+0xab/0x1a0 => ksys_read+0x5f/0xe0 => do_syscall_64+0x33/0x40 => entry_SYSCALL_64_after_hwframe+0x44/0xae Reported-by: Daniel Xu Signed-off-by: Masami Hiramatsu Suggested-by: Josh Poimboeuf --- Changes in v3: - Split out the kretprobe side patch - Fix build error when CONFIG_KRETPROBES=n. Changes in v2: - Remove kretprobe wrapper functions from unwind_orc.c - Do not fixup state->ip when unwinding with regs because kretprobe fixup instruction pointer before calling handler. --- arch/x86/include/asm/unwind.h | 17 +++++++++++++++++ arch/x86/kernel/unwind_frame.c | 4 ++-- arch/x86/kernel/unwind_guess.c | 3 +-- arch/x86/kernel/unwind_orc.c | 6 +++--- 4 files changed, 23 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/unwind.h b/arch/x86/include/asm/unwind.h index 70fc159ebe69..332aa6174b10 100644 --- a/arch/x86/include/asm/unwind.h +++ b/arch/x86/include/asm/unwind.h @@ -4,6 +4,7 @@ #include #include +#include #include #include @@ -15,6 +16,7 @@ struct unwind_state { unsigned long stack_mask; struct task_struct *task; int graph_idx; + struct llist_node *kr_cur; bool error; #if defined(CONFIG_UNWINDER_ORC) bool signal, full_regs; @@ -99,6 +101,21 @@ void unwind_module_init(struct module *mod, void *orc_ip, size_t orc_ip_size, void *orc, size_t orc_size) {} #endif +/* Recover the return address modified by instrumentation (e.g. kretprobe) */ +static inline +unsigned long unwind_recover_ret_addr(struct unwind_state *state, + unsigned long addr, unsigned long *addr_p) +{ + unsigned long ret; + + ret = ftrace_graph_ret_addr(state->task, &state->graph_idx, + addr, addr_p); + if (is_kretprobe_trampoline(ret)) + ret = kretprobe_find_ret_addr(state->task, addr_p, + &state->kr_cur); + return ret; +} + /* * This disables KASAN checking when reading a value from another task's stack, * since the other task could be running on another CPU and could have poisoned diff --git a/arch/x86/kernel/unwind_frame.c b/arch/x86/kernel/unwind_frame.c index d7c44b257f7f..24e33b44b2be 100644 --- a/arch/x86/kernel/unwind_frame.c +++ b/arch/x86/kernel/unwind_frame.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -240,8 +241,7 @@ static bool update_stack_state(struct unwind_state *state, else { addr_p = unwind_get_return_address_ptr(state); addr = READ_ONCE_TASK_STACK(state->task, *addr_p); - state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx, - addr, addr_p); + state->ip = unwind_recover_ret_addr(state, addr, addr_p); } /* Save the original stack pointer for unwind_dump(): */ diff --git a/arch/x86/kernel/unwind_guess.c b/arch/x86/kernel/unwind_guess.c index c49f10ffd8cd..884d68a6e714 100644 --- a/arch/x86/kernel/unwind_guess.c +++ b/arch/x86/kernel/unwind_guess.c @@ -15,8 +15,7 @@ unsigned long unwind_get_return_address(struct unwind_state *state) addr = READ_ONCE_NOCHECK(*state->sp); - return ftrace_graph_ret_addr(state->task, &state->graph_idx, - addr, state->sp); + return unwind_recover_ret_addr(state, addr, state->sp); } EXPORT_SYMBOL_GPL(unwind_get_return_address); diff --git a/arch/x86/kernel/unwind_orc.c b/arch/x86/kernel/unwind_orc.c index a1202536fc57..839a0698342a 100644 --- a/arch/x86/kernel/unwind_orc.c +++ b/arch/x86/kernel/unwind_orc.c @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -534,9 +535,8 @@ bool unwind_next_frame(struct unwind_state *state) if (!deref_stack_reg(state, ip_p, &state->ip)) goto err; - state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx, - state->ip, (void *)ip_p); - + state->ip = unwind_recover_ret_addr(state, state->ip, + (unsigned long *)ip_p); state->sp = sp; state->regs = NULL; state->prev_regs = NULL; From patchwork Fri Mar 19 12:23:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12150945 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC27DC433DB for ; Fri, 19 Mar 2021 12:24:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9A0BB64F6E for ; Fri, 19 Mar 2021 12:24:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230166AbhCSMX4 (ORCPT ); Fri, 19 Mar 2021 08:23:56 -0400 Received: from mail.kernel.org ([198.145.29.99]:34564 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230169AbhCSMXu (ORCPT ); Fri, 19 Mar 2021 08:23:50 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 87A7864F11; Fri, 19 Mar 2021 12:23:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1616156630; bh=oFlPu06uz5j9auNWsJ3R0Ish/3if57XYi76+/Eh3Oqs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kTY/Mk7Flul+ihiE8xxw+9ZtCqF9YcC8SqyaRpBLfFLhCkYrDSHucz2JaqpkH4eZ4 bcyyZH37nZYy9VKdq6niAnK9R2NCKBbqVY5igUa6IFcCTCCU/v+lmPtLkmw4TyaMYs f3sj4oxRqSyaVs5YCKJY/jAy0QecJbBMsXUBXIHyJb/Wx1sGuhF5zNmxO99pJ0HsoW 3tFOMlih/clCGylf6NAtA81lNA7bBJjVwhmcBfE9Jzyovr/kw9uK/7eoBBIcLyL7sx lv/9aWHqN1yF5z1eToNAuZ5yDPfpimuvYtqiOWiNwTspTkAtrxKo1/QLwIArmQrysl zS0MqdxDsrmqA== From: Masami Hiramatsu To: Steven Rostedt , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Josh Poimboeuf , linux-ia64@vger.kernel.org Subject: [PATCH -tip v3 11/11] tracing: Show kretprobe unknown indicator only for kretprobe_trampoline Date: Fri, 19 Mar 2021 21:23:45 +0900 Message-Id: <161615662521.306069.3578204343927212072.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161615650355.306069.17260992641363840330.stgit@devnote2> References: <161615650355.306069.17260992641363840330.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org ftrace shows "[unknown/kretprobe'd]" indicator all addresses in the kretprobe_trampoline, but the modified address by kretprobe should be only kretprobe_trampoline+0. Signed-off-by: Masami Hiramatsu --- kernel/trace/trace_output.c | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c index 61255bad7e01..e12437388686 100644 --- a/kernel/trace/trace_output.c +++ b/kernel/trace/trace_output.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -346,22 +347,12 @@ int trace_output_call(struct trace_iterator *iter, char *name, char *fmt, ...) } EXPORT_SYMBOL_GPL(trace_output_call); -#ifdef CONFIG_KRETPROBES -static inline const char *kretprobed(const char *name) +static inline const char *kretprobed(const char *name, unsigned long addr) { - static const char tramp_name[] = "kretprobe_trampoline"; - int size = sizeof(tramp_name); - - if (strncmp(tramp_name, name, size) == 0) + if (is_kretprobe_trampoline(addr)) return "[unknown/kretprobe'd]"; return name; } -#else -static inline const char *kretprobed(const char *name) -{ - return name; -} -#endif /* CONFIG_KRETPROBES */ void trace_seq_print_sym(struct trace_seq *s, unsigned long address, bool offset) @@ -374,7 +365,7 @@ trace_seq_print_sym(struct trace_seq *s, unsigned long address, bool offset) sprint_symbol(str, address); else kallsyms_lookup(address, NULL, NULL, NULL, str); - name = kretprobed(str); + name = kretprobed(str, address); if (name && strlen(name)) { trace_seq_puts(s, name);