From patchwork Fri Jun 18 07:05:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330363 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 397ECC48BDF for ; Fri, 18 Jun 2021 07:05:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 15673613AA for ; Fri, 18 Jun 2021 07:05:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232484AbhFRHHq (ORCPT ); Fri, 18 Jun 2021 03:07:46 -0400 Received: from mail.kernel.org ([198.145.29.99]:44818 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232486AbhFRHHn (ORCPT ); Fri, 18 Jun 2021 03:07:43 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id B91DB61351; Fri, 18 Jun 2021 07:05:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999934; bh=miHpFDmx4DL40GEYK1JdPaxrsRuu26/ovjGnumdfRuU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=d3VCCg+wmAT956pykeGY4cNGDqzOCTzUylcdGRRgOZFgQckbqF2FK7OGwQ79LaO2J BlOK70+bQkdDBcmUArgPaOehVBRrKShCdkqCxB2lx+hUwJPS+PRN8CAp9zADfKqxVO seUbawz9GzdhHiVCljrNCspyJaw5zsIAGASRy4vruChCiR+3rs75F7KI9rL6tp2tam 7E4zUtpQdUVCklBTDk2ygjmOWQzVarm6Mwlmp31Lw2ujK5qwo+Y1MvUWtvzNoQ8QzP +SFxsYreFmKXGvx/Xmpig6XlvRYylrzFaQ2L9nIybAwemMx/v8dW/YiFbAVqfcNLb3 lyCGSnXNTpESg== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 01/13] ia64: kprobes: Fix to pass correct trampoline address to the handler Date: Fri, 18 Jun 2021 16:05:31 +0900 Message-Id: <162399993125.506599.11062077324255866677.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Commit e792ff804f49 ("ia64: kprobes: Use generic kretprobe trampoline handler") missed to pass the wrong trampoline address (it passes the descriptor address instead of function entry address). This fixes it to pass correct trampoline address to __kretprobe_trampoline_handler(). This also changes to use correct symbol dereference function to get the function address from the kretprobe_trampoline. Fixes: e792ff804f49 ("ia64: kprobes: Use generic kretprobe trampoline handler") Signed-off-by: Masami Hiramatsu --- Changes in v5: - Fix a compile error typo. --- arch/ia64/kernel/kprobes.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 441ed04b1037..d4048518a1d7 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -398,7 +398,8 @@ static void kretprobe_trampoline(void) int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->cr_iip = __kretprobe_trampoline_handler(regs, kretprobe_trampoline, NULL); + regs->cr_iip = __kretprobe_trampoline_handler(regs, + dereference_function_descriptor(kretprobe_trampoline), NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler @@ -414,7 +415,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, ri->fp = NULL; /* Replace the return addr with trampoline addr */ - regs->b0 = ((struct fnptr *)kretprobe_trampoline)->ip; + regs->b0 = (unsigned long)dereference_function_descriptor(kretprobe_trampoline); } /* Check the instruction in the slot is break */ @@ -902,14 +903,14 @@ static struct kprobe trampoline_p = { int __init arch_init_kprobes(void) { trampoline_p.addr = - (kprobe_opcode_t *)((struct fnptr *)kretprobe_trampoline)->ip; + dereference_function_descriptor(kretprobe_trampoline); return register_kprobe(&trampoline_p); } int __kprobes arch_trampoline_kprobe(struct kprobe *p) { if (p->addr == - (kprobe_opcode_t *)((struct fnptr *)kretprobe_trampoline)->ip) + dereference_function_descriptor(kretprobe_trampoline)) return 1; return 0; From patchwork Fri Jun 18 07:05:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330365 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C145C48BDF for ; Fri, 18 Jun 2021 07:05:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4C022613B4 for ; Fri, 18 Jun 2021 07:05:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232442AbhFRHHx (ORCPT ); Fri, 18 Jun 2021 03:07:53 -0400 Received: from mail.kernel.org ([198.145.29.99]:44922 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232480AbhFRHHx (ORCPT ); Fri, 18 Jun 2021 03:07:53 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 1381F60BBB; Fri, 18 Jun 2021 07:05:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999944; bh=e7ZnPxxF8iIA4/Ya8Svv1yfZYvhSdNB6ApvTa6DDYxI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=P/HUwPWXAquDjBkRrujNqGg8tSGEEADN0w3t44AAhxOMtsm5cDOJOrzMQo+puzfRR Qku2pTgQGTaY8hu5yzlHrOWrCmsCygMmfbulY4WoG1lJEuFiflUmg1i0aXqMRDItd6 FEy1W52zILCVfPi+Q03RJPqyL+99tiud9LbhDp/al2POoPjzyi56d++b+fGVqfZHkZ mGIPxiEWRnWggw5RXbjLcyRsOmDpNWesLNrxn+Fe4j8X1qJetxJCt+vziqS9WmriDY /LV/ACuzY5JNP9v2pbkaI+iEE3gho6vs5Q0Ud20rRDjf2FT0akPaVeUyC+4vi41nrU oUXX9ebatdqyQ== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 02/13] kprobes: treewide: Replace arch_deref_entry_point() with dereference_symbol_descriptor() Date: Fri, 18 Jun 2021 16:05:40 +0900 Message-Id: <162399994018.506599.10332627573727646767.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Replace arch_deref_entry_point() with dereference_symbol_descriptor() because those are doing same thing. Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik --- Changes in v6: - Use dereference_symbol_descriptor() so that it can handle address in modules correctly. --- arch/ia64/kernel/kprobes.c | 5 ----- arch/powerpc/kernel/kprobes.c | 11 ----------- include/linux/kprobes.h | 1 - kernel/kprobes.c | 7 +------ lib/error-inject.c | 3 ++- 5 files changed, 3 insertions(+), 24 deletions(-) diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index d4048518a1d7..0f8573bbf520 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -891,11 +891,6 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, return ret; } -unsigned long arch_deref_entry_point(void *entry) -{ - return ((struct fnptr *)entry)->ip; -} - static struct kprobe trampoline_p = { .pre_handler = trampoline_probe_handler }; diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index c64a5feaebbe..24472f2c2cfc 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -522,17 +522,6 @@ int kprobe_fault_handler(struct pt_regs *regs, int trapnr) } NOKPROBE_SYMBOL(kprobe_fault_handler); -unsigned long arch_deref_entry_point(void *entry) -{ -#ifdef PPC64_ELF_ABI_v1 - if (!kernel_text_address((unsigned long)entry)) - return ppc_global_function_entry(entry); - else -#endif - return (unsigned long)entry; -} -NOKPROBE_SYMBOL(arch_deref_entry_point); - static struct kprobe trampoline_p = { .addr = (kprobe_opcode_t *) &kretprobe_trampoline, .pre_handler = trampoline_probe_handler diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 523ffc7bc3a8..713c3a683011 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -382,7 +382,6 @@ int register_kprobe(struct kprobe *p); void unregister_kprobe(struct kprobe *p); int register_kprobes(struct kprobe **kps, int num); void unregister_kprobes(struct kprobe **kps, int num); -unsigned long arch_deref_entry_point(void *); int register_kretprobe(struct kretprobe *rp); void unregister_kretprobe(struct kretprobe *rp); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index e41385afe79d..f8fe9d077b41 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1838,11 +1838,6 @@ static struct notifier_block kprobe_exceptions_nb = { .priority = 0x7fffffff /* we need to be notified first */ }; -unsigned long __weak arch_deref_entry_point(void *entry) -{ - return (unsigned long)entry; -} - #ifdef CONFIG_KRETPROBES unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, @@ -2305,7 +2300,7 @@ static int __init populate_kprobe_blacklist(unsigned long *start, int ret; for (iter = start; iter < end; iter++) { - entry = arch_deref_entry_point((void *)*iter); + entry = (unsigned long)dereference_symbol_descriptor((void *)*iter); ret = kprobe_add_ksym_blacklist(entry); if (ret == -EINVAL) continue; diff --git a/lib/error-inject.c b/lib/error-inject.c index c73651b15b76..2ff5ef689d72 100644 --- a/lib/error-inject.c +++ b/lib/error-inject.c @@ -8,6 +8,7 @@ #include #include #include +#include /* Whitelist of symbols that can be overridden for error injection. */ static LIST_HEAD(error_injection_list); @@ -64,7 +65,7 @@ static void populate_error_injection_list(struct error_injection_entry *start, mutex_lock(&ei_mutex); for (iter = start; iter < end; iter++) { - entry = arch_deref_entry_point((void *)iter->addr); + entry = (unsigned long)dereference_symbol_descriptor((void *)iter->addr); if (!kernel_text_address(entry) || !kallsyms_lookup_size_offset(entry, &size, &offset)) { From patchwork Fri Jun 18 07:05:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330367 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8172EC48BDF for ; Fri, 18 Jun 2021 07:05:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5ECE8613B9 for ; Fri, 18 Jun 2021 07:05:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232480AbhFRHID (ORCPT ); Fri, 18 Jun 2021 03:08:03 -0400 Received: from mail.kernel.org ([198.145.29.99]:45028 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232648AbhFRHIC (ORCPT ); Fri, 18 Jun 2021 03:08:02 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 9BB58613AA; Fri, 18 Jun 2021 07:05:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999953; bh=7xqyGxSh89hg7Vta3FWu/a/x3s+Y+h7prH87xJZMig8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qS+n6/XQlKvYFIjcYeNYYovg6mj5kI1R78dSWDv8RsbVUSWifsHDF9NXMmIrcScu7 8jIFo0tw2MnJI7j/8qtSToabBqiXDevL2V6mzwzraB+iscPNUMEbiT6ahnwwRP6bqZ Jw7m4cBz3iSVw6mmitHceyDnk632ZDABoAbUovhMaT9oH9lLJVVGgcv/eDcmHSmqfV Y6pHC63UgaP1FP8WCcNGo9xzhq9XN7jZdX2oWXj3MWdeNHQJOa/fffMUc8roayvqzN c+k1oI0iX1lNgrdnY9ATApKLcDm4ggw+9aaXQSWiz+HD5GKUQ5pwUN8//MjZUCP57b jxfASdyRMb/lg== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 03/13] kprobes: treewide: Remove trampoline_address from kretprobe_trampoline_handler() Date: Fri, 18 Jun 2021 16:05:50 +0900 Message-Id: <162399994996.506599.17672270294950096639.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Remove trampoline_address from kretprobe_trampoline_handler(). Instead of passing the address, kretprobe_trampoline_handler() can use new kretprobe_trampoline_addr(). Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik --- Changes in v8: - Use dereference_kernel_function_descriptor() to get the kretprobe_trampoline address. Changes in v3: - Remove wrong kretprobe_trampoline declaration from arch/x86/include/asm/kprobes.h. Changes in v2: - Remove arch_deref_entry_point() from comment. --- arch/arc/kernel/kprobes.c | 2 +- arch/arm/probes/kprobes/core.c | 3 +-- arch/arm64/kernel/probes/kprobes.c | 3 +-- arch/csky/kernel/probes/kprobes.c | 2 +- arch/ia64/kernel/kprobes.c | 5 ++--- arch/mips/kernel/kprobes.c | 3 +-- arch/parisc/kernel/kprobes.c | 4 ++-- arch/powerpc/kernel/kprobes.c | 2 +- arch/riscv/kernel/probes/kprobes.c | 2 +- arch/s390/kernel/kprobes.c | 2 +- arch/sh/kernel/kprobes.c | 2 +- arch/sparc/kernel/kprobes.c | 2 +- arch/x86/include/asm/kprobes.h | 1 - arch/x86/kernel/kprobes/core.c | 2 +- include/linux/kprobes.h | 18 +++++++++++++----- kernel/kprobes.c | 3 +-- 16 files changed, 29 insertions(+), 27 deletions(-) diff --git a/arch/arc/kernel/kprobes.c b/arch/arc/kernel/kprobes.c index 5f0415fc7328..3cee75c87f97 100644 --- a/arch/arc/kernel/kprobes.c +++ b/arch/arc/kernel/kprobes.c @@ -381,7 +381,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, static int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->ret = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->ret = __kretprobe_trampoline_handler(regs, NULL); /* By returning a non zero value, we are telling the kprobe handler * that we don't want the post_handler to run diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index 27e0af78e88b..583f6b1a2a6f 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -390,8 +390,7 @@ void __naked __kprobes kretprobe_trampoline(void) /* Called from kretprobe_trampoline */ static __used __kprobes void *trampoline_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, - (void *)regs->ARM_fp); + return (void *)kretprobe_trampoline_handler(regs, (void *)regs->ARM_fp); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c index 004b86eff9c2..649c970e65a2 100644 --- a/arch/arm64/kernel/probes/kprobes.c +++ b/arch/arm64/kernel/probes/kprobes.c @@ -396,8 +396,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, - (void *)kernel_stack_pointer(regs)); + return (void *)kretprobe_trampoline_handler(regs, (void *)kernel_stack_pointer(regs)); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/csky/kernel/probes/kprobes.c b/arch/csky/kernel/probes/kprobes.c index 68b22b499aeb..cc9dde2e4341 100644 --- a/arch/csky/kernel/probes/kprobes.c +++ b/arch/csky/kernel/probes/kprobes.c @@ -387,7 +387,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + return (void *)kretprobe_trampoline_handler(regs, NULL); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/ia64/kernel/kprobes.c b/arch/ia64/kernel/kprobes.c index 0f8573bbf520..44c84c20b626 100644 --- a/arch/ia64/kernel/kprobes.c +++ b/arch/ia64/kernel/kprobes.c @@ -392,14 +392,13 @@ static void __kprobes set_current_kprobe(struct kprobe *p, __this_cpu_write(current_kprobe, p); } -static void kretprobe_trampoline(void) +void kretprobe_trampoline(void) { } int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->cr_iip = __kretprobe_trampoline_handler(regs, - dereference_function_descriptor(kretprobe_trampoline), NULL); + regs->cr_iip = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/mips/kernel/kprobes.c b/arch/mips/kernel/kprobes.c index 75bff0f77319..21a4fda1e2cb 100644 --- a/arch/mips/kernel/kprobes.c +++ b/arch/mips/kernel/kprobes.c @@ -486,8 +486,7 @@ void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, static int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - instruction_pointer(regs) = __kretprobe_trampoline_handler(regs, - kretprobe_trampoline, NULL); + instruction_pointer(regs) = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/parisc/kernel/kprobes.c b/arch/parisc/kernel/kprobes.c index 6d21a515eea5..4a35ac6e2ca2 100644 --- a/arch/parisc/kernel/kprobes.c +++ b/arch/parisc/kernel/kprobes.c @@ -175,7 +175,7 @@ int __kprobes parisc_kprobe_ss_handler(struct pt_regs *regs) return 1; } -static inline void kretprobe_trampoline(void) +void kretprobe_trampoline(void) { asm volatile("nop"); asm volatile("nop"); @@ -193,7 +193,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, { unsigned long orig_ret_address; - orig_ret_address = __kretprobe_trampoline_handler(regs, trampoline_p.addr, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); instruction_pointer_set(regs, orig_ret_address); return 1; diff --git a/arch/powerpc/kernel/kprobes.c b/arch/powerpc/kernel/kprobes.c index 24472f2c2cfc..025a9f83ae88 100644 --- a/arch/powerpc/kernel/kprobes.c +++ b/arch/powerpc/kernel/kprobes.c @@ -399,7 +399,7 @@ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { unsigned long orig_ret_address; - orig_ret_address = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); /* * We get here through one of two paths: * 1. by taking a trap -> kprobe_handler() -> here diff --git a/arch/riscv/kernel/probes/kprobes.c b/arch/riscv/kernel/probes/kprobes.c index 247e33fa5bc7..07bc8804643e 100644 --- a/arch/riscv/kernel/probes/kprobes.c +++ b/arch/riscv/kernel/probes/kprobes.c @@ -370,7 +370,7 @@ int __init arch_populate_kprobe_blacklist(void) void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs) { - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + return (void *)kretprobe_trampoline_handler(regs, NULL); } void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri, diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c index 74b0bd2c24d4..1e6600765553 100644 --- a/arch/s390/kernel/kprobes.c +++ b/arch/s390/kernel/kprobes.c @@ -351,7 +351,7 @@ static void __used kretprobe_trampoline_holder(void) */ static int trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->psw.addr = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->psw.addr = __kretprobe_trampoline_handler(regs, NULL); /* * By returning a non-zero value, we are telling * kprobe_handler() that we don't want the post_handler diff --git a/arch/sh/kernel/kprobes.c b/arch/sh/kernel/kprobes.c index 1c7f358ef0be..8e76a35e6e33 100644 --- a/arch/sh/kernel/kprobes.c +++ b/arch/sh/kernel/kprobes.c @@ -303,7 +303,7 @@ static void __used kretprobe_trampoline_holder(void) */ int __kprobes trampoline_probe_handler(struct kprobe *p, struct pt_regs *regs) { - regs->pc = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + regs->pc = __kretprobe_trampoline_handler(regs, NULL); return 1; } diff --git a/arch/sparc/kernel/kprobes.c b/arch/sparc/kernel/kprobes.c index 4c05a4ee6a0e..401534236c2e 100644 --- a/arch/sparc/kernel/kprobes.c +++ b/arch/sparc/kernel/kprobes.c @@ -451,7 +451,7 @@ static int __kprobes trampoline_probe_handler(struct kprobe *p, { unsigned long orig_ret_address = 0; - orig_ret_address = __kretprobe_trampoline_handler(regs, &kretprobe_trampoline, NULL); + orig_ret_address = __kretprobe_trampoline_handler(regs, NULL); regs->tpc = orig_ret_address; regs->tnpc = orig_ret_address + 4; diff --git a/arch/x86/include/asm/kprobes.h b/arch/x86/include/asm/kprobes.h index bd7f5886a789..71ea2eab43d5 100644 --- a/arch/x86/include/asm/kprobes.h +++ b/arch/x86/include/asm/kprobes.h @@ -49,7 +49,6 @@ extern __visible kprobe_opcode_t optprobe_template_end[]; extern const int kretprobe_blacklist_size; void arch_remove_kprobe(struct kprobe *p); -asmlinkage void kretprobe_trampoline(void); extern void arch_kprobe_override_function(struct pt_regs *regs); diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index c492ad3001ca..2dccb4347453 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1070,7 +1070,7 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) regs->ip = (unsigned long)&kretprobe_trampoline; regs->orig_ax = ~0UL; - return (void *)kretprobe_trampoline_handler(regs, &kretprobe_trampoline, ®s->sp); + return (void *)kretprobe_trampoline_handler(regs, ®s->sp); } NOKPROBE_SYMBOL(trampoline_handler); diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 713c3a683011..5ce677819a25 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -197,15 +197,23 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); +void kretprobe_trampoline(void); +/* + * Since some architecture uses structured function pointer, + * use dereference_function_descriptor() to get real function address. + */ +static nokprobe_inline void *kretprobe_trampoline_addr(void) +{ + return dereference_kernel_function_descriptor(kretprobe_trampoline); +} + /* If the trampoline handler called from a kprobe, use this version */ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, - void *frame_pointer); + void *frame_pointer); static nokprobe_inline unsigned long kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, - void *frame_pointer) + void *frame_pointer) { unsigned long ret; /* @@ -214,7 +222,7 @@ unsigned long kretprobe_trampoline_handler(struct pt_regs *regs, * be running at this point. */ kprobe_busy_begin(); - ret = __kretprobe_trampoline_handler(regs, trampoline_address, frame_pointer); + ret = __kretprobe_trampoline_handler(regs, frame_pointer); kprobe_busy_end(); return ret; diff --git a/kernel/kprobes.c b/kernel/kprobes.c index f8fe9d077b41..ad7a8c81ab06 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1841,7 +1841,6 @@ static struct notifier_block kprobe_exceptions_nb = { #ifdef CONFIG_KRETPROBES unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *trampoline_address, void *frame_pointer) { kprobe_opcode_t *correct_ret_addr = NULL; @@ -1856,7 +1855,7 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, BUG_ON(ri->fp != frame_pointer); - if (ri->ret_addr != trampoline_address) { + if (ri->ret_addr != kretprobe_trampoline_addr()) { correct_ret_addr = ri->ret_addr; /* * This is the real return address. Any other From patchwork Fri Jun 18 07:05:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330369 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D93C6C48BE8 for ; Fri, 18 Jun 2021 07:06:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B9AC8613C1 for ; Fri, 18 Jun 2021 07:06:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232816AbhFRHIU (ORCPT ); Fri, 18 Jun 2021 03:08:20 -0400 Received: from mail.kernel.org ([198.145.29.99]:45136 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232648AbhFRHIM (ORCPT ); Fri, 18 Jun 2021 03:08:12 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id AF244613B4; Fri, 18 Jun 2021 07:06:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999963; bh=szqoBpM6ujoU14+uwfv95/KtSw8AhNTR0cOTG7Qviyg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uIzkq3W2A0SCkp27IX7L3fLhvOGiLc+0FI6Q2nnqwF+ldy4Umn1/RB0HUVvISQCo7 sRjtIsiZPpSC6oUAJtjU/3VA+/29rUfoe6iGmOjye/YtO/CfAhB4oEqx1TAYUdKhJc ABC/WUAJnTwVOFFJq668DHbb8nD2oVDg7Hc21M9R2/Dls0IQgUhK3u7TGg0YyHyPj/ 6mKQUk25s5q8QK8tb3qOimE+dA9svjWNKspdZ/VrH7MGRqw4LCtYI49VBLGSkC0G4s 5U7edMJf9JNxJtKa6/6Wnj/ME9gwsi9NTnyJ+qVRhOEAwl12uqJEV4BQLkRSOGEP0b PvTcxpZKZYhDQ== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 04/13] kprobes: Add kretprobe_find_ret_addr() for searching return address Date: Fri, 18 Jun 2021 16:05:59 +0900 Message-Id: <162399995900.506599.7731349506430654425.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add kretprobe_find_ret_addr() for searching correct return address from kretprobe instance list. Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik --- Changes in v6: - Replace BUG_ON() with WARN_ON_ONCE() in __kretprobe_trampoline_handler(). Changes in v3: - Remove generic stacktrace fixup. Instead, it should be solved in each unwinder. This just provide the generic interface. Changes in v2: - Add is_kretprobe_trampoline() for checking address outside of kretprobe_find_ret_addr() - Remove unneeded addr from kretprobe_find_ret_addr() - Rename fixup_kretprobe_tramp_addr() to fixup_kretprobe_trampoline() --- include/linux/kprobes.h | 22 +++++++++++ kernel/kprobes.c | 91 ++++++++++++++++++++++++++++++++++------------- 2 files changed, 87 insertions(+), 26 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 5ce677819a25..08d3415e4418 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -207,6 +207,14 @@ static nokprobe_inline void *kretprobe_trampoline_addr(void) return dereference_kernel_function_descriptor(kretprobe_trampoline); } +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return (void *)addr == kretprobe_trampoline_addr(); +} + +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur); + /* If the trampoline handler called from a kprobe, use this version */ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, void *frame_pointer); @@ -506,6 +514,20 @@ static inline bool is_kprobe_optinsn_slot(unsigned long addr) } #endif +#if !defined(CONFIG_KRETPROBES) +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return false; +} + +static nokprobe_inline +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + return 0; +} +#endif + /* Returns true if kprobes handled the fault */ static nokprobe_inline bool kprobe_page_fault(struct pt_regs *regs, unsigned int trap) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index ad7a8c81ab06..650cbe738666 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1840,45 +1840,69 @@ static struct notifier_block kprobe_exceptions_nb = { #ifdef CONFIG_KRETPROBES -unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *frame_pointer) +/* This assumes the tsk is current or the task which is not running. */ +static unsigned long __kretprobe_find_ret_addr(struct task_struct *tsk, + struct llist_node **cur) { - kprobe_opcode_t *correct_ret_addr = NULL; struct kretprobe_instance *ri = NULL; - struct llist_node *first, *node; - struct kretprobe *rp; + struct llist_node *node = *cur; + + if (!node) + node = tsk->kretprobe_instances.first; + else + node = node->next; - /* Find all nodes for this frame. */ - first = node = current->kretprobe_instances.first; while (node) { ri = container_of(node, struct kretprobe_instance, llist); - - BUG_ON(ri->fp != frame_pointer); - if (ri->ret_addr != kretprobe_trampoline_addr()) { - correct_ret_addr = ri->ret_addr; - /* - * This is the real return address. Any other - * instances associated with this task are for - * other calls deeper on the call stack - */ - goto found; + *cur = node; + return (unsigned long)ri->ret_addr; } - node = node->next; } - pr_err("Oops! Kretprobe fails to find correct return address.\n"); - BUG_ON(1); + return 0; +} +NOKPROBE_SYMBOL(__kretprobe_find_ret_addr); -found: - /* Unlink all nodes for this frame. */ - current->kretprobe_instances.first = node->next; - node->next = NULL; +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + struct kretprobe_instance *ri = NULL; + unsigned long ret; + + do { + ret = __kretprobe_find_ret_addr(tsk, cur); + if (!ret) + return ret; + ri = container_of(*cur, struct kretprobe_instance, llist); + } while (ri->fp != fp); + + return ret; +} +NOKPROBE_SYMBOL(kretprobe_find_ret_addr); - /* Run them.. */ +unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, + void *frame_pointer) +{ + kprobe_opcode_t *correct_ret_addr = NULL; + struct kretprobe_instance *ri = NULL; + struct llist_node *first, *node = NULL; + struct kretprobe *rp; + + /* Find correct address and all nodes for this frame. */ + correct_ret_addr = (void *)__kretprobe_find_ret_addr(current, &node); + if (!correct_ret_addr) { + pr_err("Oops! Kretprobe fails to find correct return address.\n"); + BUG_ON(1); + } + + /* Run them. */ + first = current->kretprobe_instances.first; while (first) { ri = container_of(first, struct kretprobe_instance, llist); - first = first->next; + + if (WARN_ON_ONCE(ri->fp != frame_pointer)) + break; rp = get_kretprobe(ri); if (rp && rp->handler) { @@ -1889,6 +1913,21 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, rp->handler(ri, regs); __this_cpu_write(current_kprobe, prev); } + if (first == node) + break; + + first = first->next; + } + + /* Unlink all nodes for this frame. */ + first = current->kretprobe_instances.first; + current->kretprobe_instances.first = node->next; + node->next = NULL; + + /* Recycle them. */ + while (first) { + ri = container_of(first, struct kretprobe_instance, llist); + first = first->next; recycle_rp_inst(ri); } From patchwork Fri Jun 18 07:06:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330371 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95A06C48BDF for ; Fri, 18 Jun 2021 07:06:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7EC1B613B4 for ; Fri, 18 Jun 2021 07:06:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232648AbhFRHI2 (ORCPT ); Fri, 18 Jun 2021 03:08:28 -0400 Received: from mail.kernel.org ([198.145.29.99]:45192 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232836AbhFRHIW (ORCPT ); Fri, 18 Jun 2021 03:08:22 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 26A3D60BBB; Fri, 18 Jun 2021 07:06:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999973; bh=kWMKhX+8lnYTT15vS+jnsaOhyRchIadTBOj6qvo8m0w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GG9itzgGfeB4cXjwhNFGh13kIK3ZpFfMfXewLAzknqgg7Jkp2NgbalBL7/xLe/FhT NksSTvg2em9F192mkuDbWK8RESBFgdZlmhplvhMaZblB9Wl1XMUZ8JBlNvHAZOkHvY qeopCemYEPeF8Qr6JmW2+Vym/CodfjM6xW+UrsSUhFcVOpEjRJ4Uluibmfd5zomNC+ 0ACKYZYWwlkcDHDCWv+lelkG0H5teALjCBaXQl+wQTWiMpC3KwDoBxUnsN16XsFYR2 k0iuMPEN+s/K/91dRfgEiGRfMg2pJAm4ub3RLQbgp8QuxB88beSJcqKsCgEWnA1Fnn A4VZUsmwa2ZqQ== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 05/13] x86/kprobes: Add UNWIND_HINT_FUNC on kretprobe_trampoline code Date: Fri, 18 Jun 2021 16:06:09 +0900 Message-Id: <162399996966.506599.810050095040575221.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Josh Poimboeuf Add UNWIND_HINT_FUNC on kretporbe_trampoline code so that ORC information is generated on the kretprobe_trampoline correctly. Note that when the CONFIG_FRAME_POINTER=y, since the kretprobe_trampoline skips updating frame pointer, the stack frame of the kretprobe_trampoline seems non-standard. So this marks it is STACK_FRAME_NON_STANDARD() and undefine UNWIND_HINT_FUNC. Anyway, with the frame pointer, FP unwinder can unwind the stack frame correctly without that hint. Signed-off-by: Josh Poimboeuf Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik --- Changes in v4: - Apply UNWIND_HINT_FUNC only if CONFIG_FRAME_POINTER=n. --- arch/x86/include/asm/unwind_hints.h | 5 +++++ arch/x86/kernel/kprobes/core.c | 17 +++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/unwind_hints.h b/arch/x86/include/asm/unwind_hints.h index 8e574c0afef8..8b33674288ea 100644 --- a/arch/x86/include/asm/unwind_hints.h +++ b/arch/x86/include/asm/unwind_hints.h @@ -52,6 +52,11 @@ UNWIND_HINT sp_reg=ORC_REG_SP sp_offset=8 type=UNWIND_HINT_TYPE_FUNC .endm +#else + +#define UNWIND_HINT_FUNC \ + UNWIND_HINT(ORC_REG_SP, 8, UNWIND_HINT_TYPE_FUNC, 0) + #endif /* __ASSEMBLY__ */ #endif /* _ASM_X86_UNWIND_HINTS_H */ diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 2dccb4347453..74f049b6e77f 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1019,6 +1019,19 @@ int kprobe_int3_handler(struct pt_regs *regs) } NOKPROBE_SYMBOL(kprobe_int3_handler); +#ifdef CONFIG_FRAME_POINTER +/* + * kretprobe_trampoline skips updating frame pointer. The frame pointer + * saved in trampoline_handler points to the real caller function's + * frame pointer. Thus the kretprobe_trampoline doesn't seems to have a + * standard stack frame with CONFIG_FRAME_POINTER=y. + * Let's mark it non-standard function. Anyway, FP unwinder can correctly + * unwind without the hint. + */ +STACK_FRAME_NON_STANDARD(kretprobe_trampoline); +#undef UNWIND_HINT_FUNC +#define UNWIND_HINT_FUNC +#endif /* * When a retprobed function returns, this code saves registers and * calls trampoline_handler() runs, which calls the kretprobe's handler. @@ -1031,6 +1044,7 @@ asm( /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 " pushq %rsp\n" + UNWIND_HINT_FUNC " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" @@ -1041,6 +1055,7 @@ asm( " popfq\n" #else " pushl %esp\n" + UNWIND_HINT_FUNC " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" @@ -1054,8 +1069,6 @@ asm( ".size kretprobe_trampoline, .-kretprobe_trampoline\n" ); NOKPROBE_SYMBOL(kretprobe_trampoline); -STACK_FRAME_NON_STANDARD(kretprobe_trampoline); - /* * Called from kretprobe_trampoline From patchwork Fri Jun 18 07:06:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330373 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C20AEC48BE8 for ; Fri, 18 Jun 2021 07:06:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AC56F613AA for ; Fri, 18 Jun 2021 07:06:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232880AbhFRHIb (ORCPT ); Fri, 18 Jun 2021 03:08:31 -0400 Received: from mail.kernel.org ([198.145.29.99]:45314 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232836AbhFRHIb (ORCPT ); Fri, 18 Jun 2021 03:08:31 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 27DEB61351; Fri, 18 Jun 2021 07:06:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999982; bh=KEe5WdEyJHtzuPMGzpiocUmfrX3xOnunn5dZvpKwBh8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KPGZ3Q8c9fRtoUEwb8fYzZZtaoq4TEJN29fsxD2jLfun2+ACEKE2Eql2SwW4PJJUc DfxZSXdTuE603PLvSkizfhCCoyXcD8FH9TIeovQH9CPFaC3dyJofbLYQaHLCLK1lU4 2z86Uaf2/i+Yv7AU3qmwOM1YBrnjJxk8PCS+/fKzuRvHcAFdikrMHLaop8M/T7/iV0 2Tv2IBCBcxDhN72l1nwDftw8LP0m7VaEm/Lqzl0jE4zkKd6HNg0U5rMsda0g0OSvWH jAlwRLQT8gdS/h4+QlHSsoVx22zuLpluaEx9R/vQe70zPL1b2RLgpk2ESv8G8Aa4A+ HTVmzveZ5tLrA== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 06/13] ARC: Add instruction_pointer_set() API Date: Fri, 18 Jun 2021 16:06:18 +0900 Message-Id: <162399997853.506599.13701157683968161733.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add instruction_pointer_set() API for arc. Signed-off-by: Masami Hiramatsu --- arch/arc/include/asm/ptrace.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/arc/include/asm/ptrace.h b/arch/arc/include/asm/ptrace.h index 4c3c9be5bd16..cca8d6583e31 100644 --- a/arch/arc/include/asm/ptrace.h +++ b/arch/arc/include/asm/ptrace.h @@ -149,6 +149,11 @@ static inline long regs_return_value(struct pt_regs *regs) return (long)regs->r0; } +static inline void instruction_pointer_set(struct pt_regs *regs, + unsigned long val) +{ + instruction_pointer(regs) = val; +} #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PTRACE_H */ From patchwork Fri Jun 18 07:06:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330375 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 170D4C48BDF for ; Fri, 18 Jun 2021 07:06:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA61E61396 for ; Fri, 18 Jun 2021 07:06:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232925AbhFRHIo (ORCPT ); Fri, 18 Jun 2021 03:08:44 -0400 Received: from mail.kernel.org ([198.145.29.99]:45458 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232903AbhFRHIk (ORCPT ); Fri, 18 Jun 2021 03:08:40 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 49A716100A; Fri, 18 Jun 2021 07:06:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623999991; bh=eXubxiP2sfbDRxWXxCS2276qipPi7ZT+e19fUSeI+18=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QlnpvzazRx6/xgsOWH3bNTfdA1pMop85FvoSqQnj+Fp/vKYT5ilFtgZAewFyEXxFM OFFznFI8IB6puBfHQiQwLWZ2ELiuV2h0+6HoE15CRc285qS7LVELIQjiB6pe4zgsBO l5thuKQKw5EvNhzTp8+61gZgeXm8fAkH9XnPsr87rDfVzKZthUmjDBCaNDs9/3z9pg cghwFbOVOHd4iR7gNBpaYTj/IrNTYJek2aIxz/3gdILhz5SLRumOnnO6JkzRtOV9eC tIJcQArwuHRKBgS51J1Gu9meChjHys9Vj6kizJhq2m6WQ7MGo/h2ABzgvqwnNJe90h 4/G531QSMcatw== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 07/13] ia64: Add instruction_pointer_set() API Date: Fri, 18 Jun 2021 16:06:27 +0900 Message-Id: <162399998747.506599.1115560529431673586.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add instruction_pointer_set() API for ia64. Signed-off-by: Masami Hiramatsu --- Changes in v4: - Make the API macro for avoiding a build error. --- arch/ia64/include/asm/ptrace.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/ia64/include/asm/ptrace.h b/arch/ia64/include/asm/ptrace.h index 08179135905c..a024afbc70e5 100644 --- a/arch/ia64/include/asm/ptrace.h +++ b/arch/ia64/include/asm/ptrace.h @@ -51,6 +51,11 @@ * the canonical representation by adding to instruction pointer. */ # define instruction_pointer(regs) ((regs)->cr_iip + ia64_psr(regs)->ri) +# define instruction_pointer_set(regs, val) \ + ({ \ + ia64_psr(regs)->ri = (val & 0xf); \ + regs->cr_iip = (val & ~0xfULL); \ + }) static inline unsigned long user_stack_pointer(struct pt_regs *regs) { From patchwork Fri Jun 18 07:06:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330377 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A19D6C49361 for ; Fri, 18 Jun 2021 07:06:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C6E461396 for ; Fri, 18 Jun 2021 07:06:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232903AbhFRHIv (ORCPT ); Fri, 18 Jun 2021 03:08:51 -0400 Received: from mail.kernel.org ([198.145.29.99]:45550 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232939AbhFRHIt (ORCPT ); Fri, 18 Jun 2021 03:08:49 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 7AA8261369; Fri, 18 Jun 2021 07:06:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000000; bh=ZVJBoicxr63FzkH5ZvJFmdAYToq8ylWnJuw9ei63N1A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IYqYUs67MX1omOtVDmo+FFuklcanudADon/9N1rO71k374ozEIyYEzjfiF5e0m+H2 QMAx60kzNwOHQIMtXkvNqDeapCcoQ9ZMkl0uCwDvyUvU6pZuF1OigcHj+OkAruAlTa xJtlQml7Kn30HuBH9ZTPzf2wp90mwO3s6IjO2cx5f8pvnXdJTkZEu/JJK7/Nbcqcdk +6JCHmLleIVl00AwBWPJU+vKifc3LNb0226gIzDLTXZth8bTRe7r6R0Jj5Y/lbSZzM H1fTBeJkWQqaTL8ejwXygKVfI1PUQDDCYQk4xwiOMfIbG3hx6UszB4rkFdcoDQK2lG 8niAVRkp9chNg== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 08/13] arm: kprobes: Make a space for regs->ARM_pc at kretprobe_trampoline Date: Fri, 18 Jun 2021 16:06:37 +0900 Message-Id: <162399999702.506599.16339931387573094059.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Change kretprobe_trampoline to make a space for regs->ARM_pc so that kretprobe_trampoline_handler can call instruction_pointer_set() safely. Signed-off-by: Masami Hiramatsu --- arch/arm/probes/kprobes/core.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm/probes/kprobes/core.c b/arch/arm/probes/kprobes/core.c index 583f6b1a2a6f..556b1560fb2c 100644 --- a/arch/arm/probes/kprobes/core.c +++ b/arch/arm/probes/kprobes/core.c @@ -374,11 +374,13 @@ int __kprobes kprobe_exceptions_notify(struct notifier_block *self, void __naked __kprobes kretprobe_trampoline(void) { __asm__ __volatile__ ( + "sub sp, sp, #16 \n\t" "stmdb sp!, {r0 - r11} \n\t" "mov r0, sp \n\t" "bl trampoline_handler \n\t" "mov lr, r0 \n\t" "ldmia sp!, {r0 - r11} \n\t" + "add sp, sp, #16 \n\t" #ifdef CONFIG_THUMB2_KERNEL "bx lr \n\t" #else From patchwork Fri Jun 18 07:06:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330379 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B485C48BDF for ; Fri, 18 Jun 2021 07:06:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0921061351 for ; Fri, 18 Jun 2021 07:06:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232953AbhFRHJA (ORCPT ); Fri, 18 Jun 2021 03:09:00 -0400 Received: from mail.kernel.org ([198.145.29.99]:45708 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232927AbhFRHJA (ORCPT ); Fri, 18 Jun 2021 03:09:00 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id A4D4460FF0; Fri, 18 Jun 2021 07:06:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000011; bh=eLCimkEr/J/o+KbLoEoS4uAKhWtQiwGcopMFmWAEMUE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YYZtQqdo99z7lWRgReN9YVZTzeyJG56BXb8WJJqzpQ9yLbAZzfQCWVmf+BxWY6wJB SmdFXOoYJa/Bxd9brzVxjgZHM5DB++C75tCeZaoyXmZhGHdzLD5xv0gpVF66DmdoFy GNR9ZU24kYOEHE4k394YcswDRN4yUvAi3jRXLhK9vS9Sc4ErLOnWqyDbDBdsVnnU7Q t+3/pEAr7EhwRgyowCFcqubeXZWvxaBpvWu3+agHopktEWst4RImq8+vycH42HKEGd BVpzMwslbGoAR/c1KCalD/HgPg48X/EQ3oaX3JqotDfrd5CvZ2RTd4MAbhuNxWaFYc Dv1MJBkDALdiQ== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 09/13] kprobes: Enable stacktrace from pt_regs in kretprobe handler Date: Fri, 18 Jun 2021 16:06:46 +0900 Message-Id: <162400000592.506599.4695807810528866713.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Since the ORC unwinder from pt_regs requires setting up regs->ip correctly, set the correct return address to the regs->ip before calling user kretprobe handler. This allows the kretrprobe handler to trace stack from the kretprobe's pt_regs by stack_trace_save_regs() (eBPF will do this), instead of stack tracing from the handler context by stack_trace_save() (ftrace will do this). Suggested-by: Josh Poimboeuf Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik Acked-by: Josh Poimboeuf --- Changes in v8: - Update comment to clarify why this is needed. Changes in v3: - Cast the correct_ret_addr to unsigned long. --- kernel/kprobes.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 650cbe738666..ba729ed05cb3 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1896,6 +1896,9 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, BUG_ON(1); } + /* Set the instruction pointer to the correct address */ + instruction_pointer_set(regs, (unsigned long)correct_ret_addr); + /* Run them. */ first = current->kretprobe_instances.first; while (first) { From patchwork Fri Jun 18 07:06:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330383 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 525DCC48BDF for ; Fri, 18 Jun 2021 07:07:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2C1E4613AA for ; Fri, 18 Jun 2021 07:07:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232103AbhFRHJV (ORCPT ); Fri, 18 Jun 2021 03:09:21 -0400 Received: from mail.kernel.org ([198.145.29.99]:45840 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232968AbhFRHJK (ORCPT ); Fri, 18 Jun 2021 03:09:10 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 419C96100A; Fri, 18 Jun 2021 07:06:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000021; bh=UKZBSR3BDB+o3rSD/O4eD6I+4zpVB9EWrJz0r6v2w9M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E6bbL3+7FVP8Ppf/ZU24U3D71avIAMR7MX5PQ9VsQtlbDj4ARCV1ukeCZnMeA9oaV SCCdLAj0hUeuzaFPJGkWcq6iHdsVQk22j7QwN60eZk7JfyrpgJyDMAJltUsvgGzygZ kvmkKqE+9bV/kYPUxIb5NwfdqSaGoOH3q9irAuIgCx41+ZBOU7FxvXUqo4KAlEt2Qu N+MVc4Y9IdWMy4zFYGR3vXTILGSsOLpb9uiSYyZN1yIc7XNWs3p5F+aY58IFbRfNqp qrYz4r+o0AZvpGWSE2Xfo/WY1NtdzUmfbXC4oCqI4qp5ECqlqo45VsmT2vAyUnjfeX YyKANwnDMaleQ== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 10/13] x86/kprobes: Push a fake return address at kretprobe_trampoline Date: Fri, 18 Jun 2021 16:06:56 +0900 Message-Id: <162400001661.506599.5153975410607447958.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This changes x86/kretprobe stack frame on kretprobe_trampoline a bit, which now push the kretprobe_trampoline as a fake return address at the bottom of the stack frame. With this fix, the ORC unwinder will see the kretprobe_trampoline as a return address. Signed-off-by: Masami Hiramatsu Suggested-by: Josh Poimboeuf Tested-by: Andrii Nakryik Acked-by: Josh Poimboeuf --- arch/x86/kernel/kprobes/core.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 74f049b6e77f..4d040aaf969b 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1041,28 +1041,31 @@ asm( ".global kretprobe_trampoline\n" ".type kretprobe_trampoline, @function\n" "kretprobe_trampoline:\n" - /* We don't bother saving the ss register */ #ifdef CONFIG_X86_64 - " pushq %rsp\n" + /* Push fake return address to tell the unwinder it's a kretprobe */ + " pushq $kretprobe_trampoline\n" UNWIND_HINT_FUNC + /* Save the sp-8, this will be fixed later */ + " pushq %rsp\n" " pushfq\n" SAVE_REGS_STRING " movq %rsp, %rdi\n" " call trampoline_handler\n" - /* Replace saved sp with true return address. */ - " movq %rax, 19*8(%rsp)\n" RESTORE_REGS_STRING + " addq $8, %rsp\n" " popfq\n" #else - " pushl %esp\n" + /* Push fake return address to tell the unwinder it's a kretprobe */ + " pushl $kretprobe_trampoline\n" UNWIND_HINT_FUNC + /* Save the sp-4, this will be fixed later */ + " pushl %esp\n" " pushfl\n" SAVE_REGS_STRING " movl %esp, %eax\n" " call trampoline_handler\n" - /* Replace saved sp with true return address. */ - " movl %eax, 15*4(%esp)\n" RESTORE_REGS_STRING + " addl $4, %esp\n" " popfl\n" #endif " ret\n" @@ -1073,8 +1076,10 @@ NOKPROBE_SYMBOL(kretprobe_trampoline); /* * Called from kretprobe_trampoline */ -__used __visible void *trampoline_handler(struct pt_regs *regs) +__used __visible void trampoline_handler(struct pt_regs *regs) { + unsigned long *frame_pointer; + /* fixup registers */ regs->cs = __KERNEL_CS; #ifdef CONFIG_X86_32 @@ -1082,8 +1087,16 @@ __used __visible void *trampoline_handler(struct pt_regs *regs) #endif regs->ip = (unsigned long)&kretprobe_trampoline; regs->orig_ax = ~0UL; + regs->sp += sizeof(long); + frame_pointer = ((unsigned long *)®s->sp) + 1; - return (void *)kretprobe_trampoline_handler(regs, ®s->sp); + /* Replace fake return address with real one. */ + *frame_pointer = kretprobe_trampoline_handler(regs, frame_pointer); + /* + * Move flags to sp so that kretprobe_trapmoline can return + * right after popf. + */ + regs->sp = regs->flags; } NOKPROBE_SYMBOL(trampoline_handler); From patchwork Fri Jun 18 07:07:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330381 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D4DFC48BE8 for ; Fri, 18 Jun 2021 07:07:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B77761369 for ; Fri, 18 Jun 2021 07:07:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232990AbhFRHJV (ORCPT ); Fri, 18 Jun 2021 03:09:21 -0400 Received: from mail.kernel.org ([198.145.29.99]:45982 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233159AbhFRHJU (ORCPT ); Fri, 18 Jun 2021 03:09:20 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 1C4D160FF0; Fri, 18 Jun 2021 07:07:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000031; bh=9e+f0llaiwYlVIJqqmbD5EIzZ4I+oSNZ+ln+j5BmB2Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LUsIm1sy9lUJn1KsEaGhA3J6JzpJ1PGhT1PkWOxYKgqACtoSadgANJaSUXcj5FCjy 3M2HEJ+nXmZAeaZVVoXhsjl5GmnQPKFO5OqlkZu7IvxjsPVQb3rvwWvzM7+snOICQG gIzkaqsHj6gt9rmsDLi3PHHjvHjCfEVgiOgKIxdDzyYcR086t7E8t+sxdaQsUZbI/A tR7v0qlBD1pza8qYe2B/p/fQB3qfKAgvd9SQKZDxJpsbYYcYzjcABZYaChCdbeoB2A o8RArD4dOGq8+KMlEio+tKtB7rVp2EkfH4tbHW3GPvdJaDAvRPs/MWLrjfa8aIWGEF ozAg50KDW7VjQ== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 11/13] x86/unwind: Recover kretprobe trampoline entry Date: Fri, 18 Jun 2021 16:07:06 +0900 Message-Id: <162400002631.506599.2413605639666466945.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Since the kretprobe replaces the function return address with the kretprobe_trampoline on the stack, x86 unwinders can not continue the stack unwinding at that point, or record kretprobe_trampoline instead of correct return address. To fix this issue, find the correct return address from task's kretprobe_instances as like as function-graph tracer does. With this fix, the unwinder can correctly unwind the stack from kretprobe event on x86, as below. <...>-135 [003] ...1 6.722338: r_full_proxy_read_0: (vfs_read+0xab/0x1a0 <- full_proxy_read) <...>-135 [003] ...1 6.722377: => kretprobe_trace_func+0x209/0x2f0 => kretprobe_dispatcher+0x4a/0x70 => __kretprobe_trampoline_handler+0xca/0x150 => trampoline_handler+0x44/0x70 => kretprobe_trampoline+0x2a/0x50 => vfs_read+0xab/0x1a0 => ksys_read+0x5f/0xe0 => do_syscall_64+0x33/0x40 => entry_SYSCALL_64_after_hwframe+0x44/0xae Reported-by: Daniel Xu Signed-off-by: Masami Hiramatsu Suggested-by: Josh Poimboeuf Tested-by: Andrii Nakryik Acked-by: Josh Poimboeuf --- Changes in v7: - Remove superfluous #include . Changes in v5: - Fix the case of interrupt happens on kretprobe_trampoline+0. Changes in v3: - Split out the kretprobe side patch - Fix build error when CONFIG_KRETPROBES=n. Changes in v2: - Remove kretprobe wrapper functions from unwind_orc.c - Do not fixup state->ip when unwinding with regs because kretprobe fixup instruction pointer before calling handler. --- arch/x86/include/asm/unwind.h | 23 +++++++++++++++++++++++ arch/x86/kernel/unwind_frame.c | 3 +-- arch/x86/kernel/unwind_guess.c | 3 +-- arch/x86/kernel/unwind_orc.c | 18 ++++++++++++++---- 4 files changed, 39 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/unwind.h b/arch/x86/include/asm/unwind.h index 70fc159ebe69..36d3971c0a2c 100644 --- a/arch/x86/include/asm/unwind.h +++ b/arch/x86/include/asm/unwind.h @@ -4,6 +4,7 @@ #include #include +#include #include #include @@ -15,6 +16,7 @@ struct unwind_state { unsigned long stack_mask; struct task_struct *task; int graph_idx; + struct llist_node *kr_cur; bool error; #if defined(CONFIG_UNWINDER_ORC) bool signal, full_regs; @@ -99,6 +101,27 @@ void unwind_module_init(struct module *mod, void *orc_ip, size_t orc_ip_size, void *orc, size_t orc_size) {} #endif +static inline +unsigned long unwind_recover_kretprobe(struct unwind_state *state, + unsigned long addr, unsigned long *addr_p) +{ + return is_kretprobe_trampoline(addr) ? + kretprobe_find_ret_addr(state->task, addr_p, &state->kr_cur) : + addr; +} + +/* Recover the return address modified by instrumentation (e.g. kretprobe) */ +static inline +unsigned long unwind_recover_ret_addr(struct unwind_state *state, + unsigned long addr, unsigned long *addr_p) +{ + unsigned long ret; + + ret = ftrace_graph_ret_addr(state->task, &state->graph_idx, + addr, addr_p); + return unwind_recover_kretprobe(state, ret, addr_p); +} + /* * This disables KASAN checking when reading a value from another task's stack, * since the other task could be running on another CPU and could have poisoned diff --git a/arch/x86/kernel/unwind_frame.c b/arch/x86/kernel/unwind_frame.c index d7c44b257f7f..8e1c50c86e5d 100644 --- a/arch/x86/kernel/unwind_frame.c +++ b/arch/x86/kernel/unwind_frame.c @@ -240,8 +240,7 @@ static bool update_stack_state(struct unwind_state *state, else { addr_p = unwind_get_return_address_ptr(state); addr = READ_ONCE_TASK_STACK(state->task, *addr_p); - state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx, - addr, addr_p); + state->ip = unwind_recover_ret_addr(state, addr, addr_p); } /* Save the original stack pointer for unwind_dump(): */ diff --git a/arch/x86/kernel/unwind_guess.c b/arch/x86/kernel/unwind_guess.c index c49f10ffd8cd..884d68a6e714 100644 --- a/arch/x86/kernel/unwind_guess.c +++ b/arch/x86/kernel/unwind_guess.c @@ -15,8 +15,7 @@ unsigned long unwind_get_return_address(struct unwind_state *state) addr = READ_ONCE_NOCHECK(*state->sp); - return ftrace_graph_ret_addr(state->task, &state->graph_idx, - addr, state->sp); + return unwind_recover_ret_addr(state, addr, state->sp); } EXPORT_SYMBOL_GPL(unwind_get_return_address); diff --git a/arch/x86/kernel/unwind_orc.c b/arch/x86/kernel/unwind_orc.c index a1202536fc57..ad6a9aece379 100644 --- a/arch/x86/kernel/unwind_orc.c +++ b/arch/x86/kernel/unwind_orc.c @@ -534,9 +534,8 @@ bool unwind_next_frame(struct unwind_state *state) if (!deref_stack_reg(state, ip_p, &state->ip)) goto err; - state->ip = ftrace_graph_ret_addr(state->task, &state->graph_idx, - state->ip, (void *)ip_p); - + state->ip = unwind_recover_ret_addr(state, state->ip, + (unsigned long *)ip_p); state->sp = sp; state->regs = NULL; state->prev_regs = NULL; @@ -549,7 +548,15 @@ bool unwind_next_frame(struct unwind_state *state) (void *)orig_ip); goto err; } - + /* + * There is a small chance to interrupt at the entry of + * kretprobe_trampoline where the ORC info doesn't exist. + * That point is right after the RET to kretprobe_trampoline + * which was modified return address. So the @addr_p must + * be right before the regs->sp. + */ + state->ip = unwind_recover_kretprobe(state, state->ip, + (unsigned long *)(state->sp - sizeof(long))); state->regs = (struct pt_regs *)sp; state->prev_regs = NULL; state->full_regs = true; @@ -562,6 +569,9 @@ bool unwind_next_frame(struct unwind_state *state) (void *)orig_ip); goto err; } + /* See UNWIND_HINT_TYPE_REGS case comment. */ + state->ip = unwind_recover_kretprobe(state, state->ip, + (unsigned long *)(state->sp - sizeof(long))); if (state->full_regs) state->prev_regs = state->regs; From patchwork Fri Jun 18 07:07:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330385 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 088EEC49361 for ; Fri, 18 Jun 2021 07:07:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF80F61396 for ; Fri, 18 Jun 2021 07:07:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233015AbhFRHJb (ORCPT ); Fri, 18 Jun 2021 03:09:31 -0400 Received: from mail.kernel.org ([198.145.29.99]:46158 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232316AbhFRHJ3 (ORCPT ); Fri, 18 Jun 2021 03:09:29 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 465FF6100A; Fri, 18 Jun 2021 07:07:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000040; bh=3tn+mKttIhl6B6aZ+Em2tVtX8FIsTpCZn3isL59O8XA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fYUuucuXybx9Cj6rRgXrnwP6OX92aZMtUezV4uFUr/lIdBX44tTYXZuyfAuxlB8bW L6l29aClqpEpL+OY/CpNTWKcNEOy0SKpysS3AUBHTkLQAS7W/WP7BVIzS8Mlt0GXO6 PFyxEyw6pLSSex65itU4+QktY8/iVuEB8RZgTfrEH+c9hih99RujGQC+BOm0us/+gI CkBKPkYlTc/0StD/2COWAda1zWGJz21+2Yt63FPlJZTpxwyEb+Kgp2O6LyNYRwG9CO yQwvGRURTZsNOkafL/cqavAsNvMpZ7eIzQ7IwoU0IYqC3l155hhMKhdSDkvX96zPJj nCo04txgC/TJA== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 12/13] tracing: Show kretprobe unknown indicator only for kretprobe_trampoline Date: Fri, 18 Jun 2021 16:07:17 +0900 Message-Id: <162400003685.506599.13956516484858326490.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org ftrace shows "[unknown/kretprobe'd]" indicator all addresses in the kretprobe_trampoline, but the modified address by kretprobe should be only kretprobe_trampoline+0. Signed-off-by: Masami Hiramatsu Acked-by: Steven Rostedt (VMware) Tested-by: Andrii Nakryik --- kernel/trace/trace_output.c | 17 ++++------------- 1 file changed, 4 insertions(+), 13 deletions(-) diff --git a/kernel/trace/trace_output.c b/kernel/trace/trace_output.c index d0368a569bfa..e46780670742 100644 --- a/kernel/trace/trace_output.c +++ b/kernel/trace/trace_output.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -346,22 +347,12 @@ int trace_output_call(struct trace_iterator *iter, char *name, char *fmt, ...) } EXPORT_SYMBOL_GPL(trace_output_call); -#ifdef CONFIG_KRETPROBES -static inline const char *kretprobed(const char *name) +static inline const char *kretprobed(const char *name, unsigned long addr) { - static const char tramp_name[] = "kretprobe_trampoline"; - int size = sizeof(tramp_name); - - if (strncmp(tramp_name, name, size) == 0) + if (is_kretprobe_trampoline(addr)) return "[unknown/kretprobe'd]"; return name; } -#else -static inline const char *kretprobed(const char *name) -{ - return name; -} -#endif /* CONFIG_KRETPROBES */ void trace_seq_print_sym(struct trace_seq *s, unsigned long address, bool offset) @@ -374,7 +365,7 @@ trace_seq_print_sym(struct trace_seq *s, unsigned long address, bool offset) sprint_symbol(str, address); else kallsyms_lookup(address, NULL, NULL, NULL, str); - name = kretprobed(str); + name = kretprobed(str, address); if (name && strlen(name)) { trace_seq_puts(s, name); From patchwork Fri Jun 18 07:07:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12330389 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0D6AC48BDF for ; Fri, 18 Jun 2021 07:07:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8284161369 for ; Fri, 18 Jun 2021 07:07:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233071AbhFRHJm (ORCPT ); Fri, 18 Jun 2021 03:09:42 -0400 Received: from mail.kernel.org ([198.145.29.99]:46378 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229782AbhFRHJi (ORCPT ); Fri, 18 Jun 2021 03:09:38 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2509661369; Fri, 18 Jun 2021 07:07:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1624000049; bh=WFXe7meLPgEvkoFkT5K9EfHGJUd1NjvkS/nTh9M7lic=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KnrLjb9Q9xngUpKdtZTyZD+zlXo3Z7DmUZvteAH6jsB1WJPznPK+PjBCKSSSfYY+q 4yxnM5om0/1YbiYTrG8eVhFhR9fJlWsvp5Mc2wY4QzaPNycycNYtwaPn+xfdVn2AdZ gSQt1WaR8uIK9AnXkGP6RrLjFhEX5JGuyzvY1FhJkjgZaVPV1tPRAabmWEj16j9aWd pqmqCQ2QktHb0YIQwc1M0te2jT+WNK/9+k0WDNqTMN08zFOUZRxJNyAvuyAX+H8Ym6 WCM6n7Q+4yHoUamSXznQeiE5yDkh92bio2NIhnh+Mx+O52BQ9iHY3XIHVPjsZrw/c7 7uI7OKQPs8Akw== From: Masami Hiramatsu To: Steven Rostedt , Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, Thomas Gleixner , Borislav Petkov , Peter Zijlstra , kernel-team@fb.com, yhs@fb.com, linux-ia64@vger.kernel.org, Abhishek Sagar , Andrii Nakryiko Subject: [PATCH -tip v8 13/13] x86/kprobes: Fixup return address in generic trampoline handler Date: Fri, 18 Jun 2021 16:07:25 +0900 Message-Id: <162400004562.506599.7549585083316952768.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <162399992186.506599.8457763707951687195.stgit@devnote2> References: <162399992186.506599.8457763707951687195.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org In x86, kretprobe trampoline address on the stack frame will be replaced with the real return address after returning from trampoline_handler. Before fixing the return address, the real return address can be found in the current->kretprobe_instances. However, since there is a window between updating the current->kretprobe_instances and fixing the address on the stack, if an interrupt caused at that timing and the interrupt handler does stacktrace, it may fail to unwind because it can not get the correct return address from current->kretprobe_instances. This will minimize that window by fixing the return address right before updating current->kretprobe_instances. Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryik --- Changes in v7: - Add a prototype for arch_kretprobe_fixup_return() --- arch/x86/kernel/kprobes/core.c | 15 +++++++++++++-- include/linux/kprobes.h | 3 +++ kernel/kprobes.c | 8 ++++++++ 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index 4d040aaf969b..53c1dcfcb145 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -1032,6 +1032,7 @@ STACK_FRAME_NON_STANDARD(kretprobe_trampoline); #undef UNWIND_HINT_FUNC #define UNWIND_HINT_FUNC #endif + /* * When a retprobed function returns, this code saves registers and * calls trampoline_handler() runs, which calls the kretprobe's handler. @@ -1073,6 +1074,17 @@ asm( ); NOKPROBE_SYMBOL(kretprobe_trampoline); +void arch_kretprobe_fixup_return(struct pt_regs *regs, + unsigned long correct_ret_addr) +{ + unsigned long *frame_pointer; + + frame_pointer = ((unsigned long *)®s->sp) + 1; + + /* Replace fake return address with real one. */ + *frame_pointer = correct_ret_addr; +} + /* * Called from kretprobe_trampoline */ @@ -1090,8 +1102,7 @@ __used __visible void trampoline_handler(struct pt_regs *regs) regs->sp += sizeof(long); frame_pointer = ((unsigned long *)®s->sp) + 1; - /* Replace fake return address with real one. */ - *frame_pointer = kretprobe_trampoline_handler(regs, frame_pointer); + kretprobe_trampoline_handler(regs, frame_pointer); /* * Move flags to sp so that kretprobe_trapmoline can return * right after popf. diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index 08d3415e4418..259bdc80e708 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -197,6 +197,9 @@ extern void arch_prepare_kretprobe(struct kretprobe_instance *ri, struct pt_regs *regs); extern int arch_trampoline_kprobe(struct kprobe *p); +void arch_kretprobe_fixup_return(struct pt_regs *regs, + unsigned long correct_ret_addr); + void kretprobe_trampoline(void); /* * Since some architecture uses structured function pointer, diff --git a/kernel/kprobes.c b/kernel/kprobes.c index ba729ed05cb3..72e8125fb0e9 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1881,6 +1881,12 @@ unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, } NOKPROBE_SYMBOL(kretprobe_find_ret_addr); +void __weak arch_kretprobe_fixup_return(struct pt_regs *regs, + unsigned long correct_ret_addr) +{ + /* Do nothing by default. */ +} + unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, void *frame_pointer) { @@ -1922,6 +1928,8 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, first = first->next; } + arch_kretprobe_fixup_return(regs, (unsigned long)correct_ret_addr); + /* Unlink all nodes for this frame. */ first = current->kretprobe_instances.first; current->kretprobe_instances.first = node->next;