From patchwork Wed Mar 31 05:44:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12174341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F4149C433C1 for ; Wed, 31 Mar 2021 05:45:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C685861584 for ; Wed, 31 Mar 2021 05:45:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233688AbhCaFow (ORCPT ); Wed, 31 Mar 2021 01:44:52 -0400 Received: from mail.kernel.org ([198.145.29.99]:43792 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233478AbhCaFok (ORCPT ); Wed, 31 Mar 2021 01:44:40 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id EF75F619AB; Wed, 31 Mar 2021 05:44:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617169480; bh=3B0ZkPXCOBAuQFYAO1fQ2vSMhOLckudtbTWf+hCfgzA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nh+wxpv3V4Ht3LbxCgN3CCbmkoIOI8I2U+muLmCtzinzhAv0iKfQLtiiTIfOLkZ5R 47qO6j97C1oensgDWklQ/bTEI+wb6sXk6qlbVJ2PIU0Nh7+8dgmd6d4Hke9g5wv1RO 0lyS9WEcFiI92hDUZQUFriqraW+VXElDUeM7dWPQ1VVFyeTOlCbb9XYaBO+qnUAoZJ DDJumwClNgN8FmKvbZJedsKss/Q7q1JDhrGQbHlZNk82XAkAKWI5IiaX4M+ltuOspD eYMPF2vJQp9t33ZpuXXUoMnbrCPqAz1JL95pb6sN9U4/Z+d+7Qbc1pGp5OVesomcJZ D+okqPzwYZRIg== From: Masami Hiramatsu To: Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Steven Rostedt Subject: [RFC PATCH -tip 1/3] x86/kprobes: Add ORC information to optprobe template Date: Wed, 31 Mar 2021 14:44:34 +0900 Message-Id: <161716947469.721514.10958896582230159703.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161716946413.721514.4057380464113663840.stgit@devnote2> References: <161716946413.721514.4057380464113663840.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-State: RFC As same as kretprobe_trampoline, move the optprobe template code in the text for making ORC information on that. Signed-off-by: Masami Hiramatsu --- arch/x86/kernel/kprobes/opt.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 4299fc865732..6d26e5cf2ba2 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -100,9 +100,13 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val) *(unsigned long *)addr = val; } +static void +optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs); + asm ( - ".pushsection .rodata\n" + ".text\n" "optprobe_template_func:\n" + ".type optprobe_template_func, @function\n" ".global optprobe_template_entry\n" "optprobe_template_entry:\n" #ifdef CONFIG_X86_64 @@ -120,7 +124,8 @@ asm ( ASM_NOP5 ".global optprobe_template_call\n" "optprobe_template_call:\n" - ASM_NOP5 + /* Dummy call for ORC */ + " callq optimized_callback\n" /* Move flags to rsp */ " movq 18*8(%rsp), %rdx\n" " movq %rdx, 19*8(%rsp)\n" @@ -141,7 +146,8 @@ asm ( ASM_NOP5 ".global optprobe_template_call\n" "optprobe_template_call:\n" - ASM_NOP5 + /* Dummy call for ORC */ + " call optimized_callback\n" /* Move flags into esp */ " movl 14*4(%esp), %edx\n" " movl %edx, 15*4(%esp)\n" @@ -152,10 +158,12 @@ asm ( #endif ".global optprobe_template_end\n" "optprobe_template_end:\n" - ".popsection\n"); + /* Dummy return for objtool */ + " ret\n" + ".size optprobe_template_func, .-optprobe_template_func\n" +); void optprobe_template_func(void); -STACK_FRAME_NON_STANDARD(optprobe_template_func); #define TMPL_CLAC_IDX \ ((long)optprobe_template_clac - (long)optprobe_template_entry) From patchwork Wed Mar 31 05:44:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12174339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA5F1C433E0 for ; Wed, 31 Mar 2021 05:45:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B1A976024A for ; Wed, 31 Mar 2021 05:45:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233690AbhCaFox (ORCPT ); Wed, 31 Mar 2021 01:44:53 -0400 Received: from mail.kernel.org ([198.145.29.99]:43846 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233595AbhCaFov (ORCPT ); Wed, 31 Mar 2021 01:44:51 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id 2D47B6024A; Wed, 31 Mar 2021 05:44:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617169491; bh=4vzFQPDT3xus/+3X6W7v5KWyPLjj8HNd8PrsBug0278=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bhwvMYVwVLISmNlfYLM8ndSdl5JUSW9i8j5Ne4EHW9xkHxBfje/uvU9I8haXXLNfq wqsey2V/otSWShRQoo6Yl0briAr537XvMaGAbsU/5MmuesLOmZkYeQTtqUBifHRmQS teERgr39hz9GfDxI/JqnfwBovVwQJQGZg/M9mp94ssIaPJ6QTHxHOKhPNoNkFTEWjP wsowFV8XEHnbkP3NAz3hViqBi3VJMuTMFrDbyPBIBW3diueaMmu1p15GCEQ3K4OGyJ DUQlALB6mFwOv50v/VjExOz6VwkIpa198jAKdPDQHqcSWW+9D01mw68VRGu0MKQ1T7 UNekaSMcaqAvA== From: Masami Hiramatsu To: Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Steven Rostedt Subject: [RFC PATCH -tip 2/3] kprobes: Add functions to find instruction buffer entry address Date: Wed, 31 Mar 2021 14:44:45 +0900 Message-Id: <161716948533.721514.17707467357877538662.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161716946413.721514.4057380464113663840.stgit@devnote2> References: <161716946413.721514.4057380464113663840.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-State: RFC Add find_kprobe_{insn,optinsn}_slot_entry() functions to find corresponding entry address of the kprobe instrurction buffer which includes given address. Signed-off-by: Masami Hiramatsu --- include/linux/kprobes.h | 8 ++++++++ kernel/kprobes.c | 25 ++++++++++++++++++------- 2 files changed, 26 insertions(+), 7 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index f530f82a046d..08adfd6cf562 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -305,6 +305,8 @@ extern void __free_insn_slot(struct kprobe_insn_cache *c, /* sleep-less address checking routine */ extern bool __is_insn_slot_addr(struct kprobe_insn_cache *c, unsigned long addr); +extern unsigned long +__find_insn_slot_entry(struct kprobe_insn_cache *c, unsigned long addr); #define DEFINE_INSN_CACHE_OPS(__name) \ extern struct kprobe_insn_cache kprobe_##__name##_slots; \ @@ -322,6 +324,12 @@ static inline void free_##__name##_slot(kprobe_opcode_t *slot, int dirty)\ static inline bool is_kprobe_##__name##_slot(unsigned long addr) \ { \ return __is_insn_slot_addr(&kprobe_##__name##_slots, addr); \ +} \ + \ +static inline unsigned long \ +find_kprobe_##__name##_slot_entry(unsigned long addr) \ +{ \ + return __find_insn_slot_entry(&kprobe_##__name##_slots, addr); \ } #define KPROBE_INSN_PAGE_SYM "kprobe_insn_page" #define KPROBE_OPTINSN_PAGE_SYM "kprobe_optinsn_page" diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 4ce3e6f5d28d..b62635969fa6 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -277,26 +277,37 @@ void __free_insn_slot(struct kprobe_insn_cache *c, } /* - * Check given address is on the page of kprobe instruction slots. - * This will be used for checking whether the address on a stack - * is on a text area or not. + * Find the entry address of the kprobe instruction slots where the + * @addr points. */ -bool __is_insn_slot_addr(struct kprobe_insn_cache *c, unsigned long addr) +unsigned long +__find_insn_slot_entry(struct kprobe_insn_cache *c, unsigned long addr) { struct kprobe_insn_page *kip; - bool ret = false; + unsigned long entry = 0, index; rcu_read_lock(); list_for_each_entry_rcu(kip, &c->pages, list) { if (addr >= (unsigned long)kip->insns && addr < (unsigned long)kip->insns + PAGE_SIZE) { - ret = true; + index = (addr - (unsigned long)kip->insns) / c->insn_size; + entry = (unsigned long)kip->insns + (index * c->insn_size); break; } } rcu_read_unlock(); - return ret; + return entry; +} + +/* + * Check given address is on the page of kprobe instruction slots. + * This will be used for checking whether the address on a stack + * is on a text area or not. + */ +bool __is_insn_slot_addr(struct kprobe_insn_cache *c, unsigned long addr) +{ + return __find_insn_slot_entry(c, addr) != 0; } int kprobe_cache_get_kallsym(struct kprobe_insn_cache *c, unsigned int *symnum, From patchwork Wed Mar 31 05:44:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Masami Hiramatsu (Google)" X-Patchwork-Id: 12174343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D19EC433DB for ; Wed, 31 Mar 2021 05:45:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 028AA619D8 for ; Wed, 31 Mar 2021 05:45:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231315AbhCaFpY (ORCPT ); Wed, 31 Mar 2021 01:45:24 -0400 Received: from mail.kernel.org ([198.145.29.99]:44010 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233716AbhCaFpC (ORCPT ); Wed, 31 Mar 2021 01:45:02 -0400 Received: by mail.kernel.org (Postfix) with ESMTPSA id DAC42619D7; Wed, 31 Mar 2021 05:44:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1617169501; bh=sSF71qlnI974wAy+ZMYm8N/K3MKLmeAecrOWanR5chM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o17jHD7Wq2FySlhs4u73r/iKdAdAnMzfjGtvz7bjSB6d3vDeTbVREJLcHynV0aiIm ExogATKA1g2qN5qvBaIR2kVSle/QjHWdFLfMxaoj9JpC+mypKWUUyFQHIk2/9YRpn1 6ZgyifbMrdGaa57DBXx9Z/BbIhnLIogLPGeY7+dw98jk1CFfqwViqZPsbyStc0S8ZP rcSLW46ehTGG9sv+zMYBQzmPfz8K+b+0gTbeXho28Vkh8zwqA5p8uIfO2zsNNt2QJU c0Eu8XtefNUCULDDpHKxkAy7k8uOhrXYJtpODxir48ZY153DXCSsteVDWokRPqsC19 Cv6t6O/sW5pPA== From: Masami Hiramatsu To: Josh Poimboeuf , Ingo Molnar Cc: X86 ML , Masami Hiramatsu , Daniel Xu , linux-kernel@vger.kernel.org, bpf@vger.kernel.org, kuba@kernel.org, mingo@redhat.com, ast@kernel.org, tglx@linutronix.de, kernel-team@fb.com, yhs@fb.com, Steven Rostedt Subject: [RFC PATCH -tip 3/3] x86/kprobes,orc: Unwind optprobe trampoline correctly Date: Wed, 31 Mar 2021 14:44:56 +0900 Message-Id: <161716949640.721514.14252504351086671126.stgit@devnote2> X-Mailer: git-send-email 2.25.1 In-Reply-To: <161716946413.721514.4057380464113663840.stgit@devnote2> References: <161716946413.721514.4057380464113663840.stgit@devnote2> User-Agent: StGit/0.19 MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-State: RFC ORC Unwinder can not unwind if an optprobe trampoline is on the stack because optprobe trampoline code has no ORC information. This uses the ORC information on the template code of the trampoline to adjust the sp register by ORC information and extract the correct probed address from the optprobe trampoline address. Signed-off-by: Masami Hiramatsu --- arch/x86/include/asm/kprobes.h | 6 ++++ arch/x86/kernel/kprobes/opt.c | 54 ++++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/unwind_orc.c | 15 +++++++++-- 3 files changed, 72 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kprobes.h b/arch/x86/include/asm/kprobes.h index 71ea2eab43d5..9bbc45fcb3f1 100644 --- a/arch/x86/include/asm/kprobes.h +++ b/arch/x86/include/asm/kprobes.h @@ -119,9 +119,15 @@ extern int kprobe_exceptions_notify(struct notifier_block *self, unsigned long val, void *data); extern int kprobe_int3_handler(struct pt_regs *regs); +unsigned long recover_optprobe_trampoline(unsigned long addr, unsigned long *sp); #else static inline int kprobe_debug_handler(struct pt_regs *regs) { return 0; } +static inline unsigned long recover_optprobe_trampoline(unsigned long addr, + unsigned long *sp) +{ + return addr; +} #endif /* CONFIG_KPROBES */ #endif /* _ASM_X86_KPROBES_H */ diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 6d26e5cf2ba2..f91922ba4844 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -30,6 +30,9 @@ #include #include #include +#include + +struct orc_entry *orc_find(unsigned long ip); #include "common.h" @@ -100,6 +103,21 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val) *(unsigned long *)addr = val; } +/* Extract mov operand */ +static unsigned long extract_set_arg1(kprobe_opcode_t *addr) +{ +#ifdef CONFIG_X86_64 + if (addr[0] != 0x48 || addr[1] != 0xbf) + return 0; + addr += 2; +#else + if (*addr != 0xb8) + return 0; + addr++; +#endif + return *(unsigned long *)addr; +} + static void optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs); @@ -483,6 +501,42 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op, goto out; } +#ifdef CONFIG_UNWINDER_ORC +unsigned long recover_optprobe_trampoline(unsigned long addr, unsigned long *sp) +{ + unsigned long offset, entry, probe_addr; + struct optimized_kprobe *op; + struct orc_entry *orc; + + entry = find_kprobe_optinsn_slot_entry(addr); + if (!entry) + return addr; + + offset = addr - entry; + + /* Decode arg1 and get the optprobe */ + op = (void *)extract_set_arg1((void *)(entry + TMPL_MOVE_IDX)); + if (!op) + return addr; + + probe_addr = (unsigned long)op->kp.addr; + + if (offset < TMPL_END_IDX) { + orc = orc_find((unsigned long)optprobe_template_func + offset); + if (!orc || orc->sp_reg != ORC_REG_SP) + return addr; + /* + * Since optprobe trampoline doesn't push caller on the stack, + * need to decrement 1 stack entry size + */ + *sp += orc->sp_offset - sizeof(long); + return probe_addr; + } else { + return probe_addr + offset - TMPL_END_IDX; + } +} +#endif + /* * Replace breakpoints (INT3) with relative jumps (JMP.d32). * Caller must call with locking kprobe_mutex and text_mutex. diff --git a/arch/x86/kernel/unwind_orc.c b/arch/x86/kernel/unwind_orc.c index c70dfeea4552..9f685f9c2358 100644 --- a/arch/x86/kernel/unwind_orc.c +++ b/arch/x86/kernel/unwind_orc.c @@ -79,7 +79,7 @@ static struct orc_entry *orc_module_find(unsigned long ip) #endif #ifdef CONFIG_DYNAMIC_FTRACE -static struct orc_entry *orc_find(unsigned long ip); +struct orc_entry *orc_find(unsigned long ip); /* * Ftrace dynamic trampolines do not have orc entries of their own. @@ -142,7 +142,7 @@ static struct orc_entry orc_fp_entry = { .end = 0, }; -static struct orc_entry *orc_find(unsigned long ip) +struct orc_entry *orc_find(unsigned long ip) { static struct orc_entry *orc; @@ -537,6 +537,7 @@ bool unwind_next_frame(struct unwind_state *state) state->ip = unwind_recover_ret_addr(state, state->ip, (unsigned long *)ip_p); + state->ip = recover_optprobe_trampoline(state->ip, &sp); state->sp = sp; state->regs = NULL; state->prev_regs = NULL; @@ -558,6 +559,14 @@ bool unwind_next_frame(struct unwind_state *state) */ state->ip = unwind_recover_kretprobe(state, state->ip, (unsigned long *)(state->sp - sizeof(long))); + + /* + * The optprobe trampoline has a unique stackframe. It has + * no caller (probed) address on the stack, Thus it has to + * decode the trampoline code and change the stack pointer + * for the next frame, but not change the pt_regs. + */ + state->ip = recover_optprobe_trampoline(state->ip, &state->sp); state->regs = (struct pt_regs *)sp; state->prev_regs = NULL; state->full_regs = true; @@ -573,7 +582,7 @@ bool unwind_next_frame(struct unwind_state *state) /* See UNWIND_HINT_TYPE_REGS case comment. */ state->ip = unwind_recover_kretprobe(state, state->ip, (unsigned long *)(state->sp - sizeof(long))); - + state->ip = recover_optprobe_trampoline(state->ip, &state->sp); if (state->full_regs) state->prev_regs = state->regs; state->regs = (void *)sp - IRET_FRAME_OFFSET;