From patchwork Fri Jan 27 13:05:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Xim X-Patchwork-Id: 13118689 X-Patchwork-Delegate: palmer@dabbelt.com Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4194EC54EAA for ; Fri, 27 Jan 2023 13:33:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JPueW8/Kn/ZszsMoQ9pp9gx1JENHQE/lAONrLbAYkwA=; b=FGX33dAV4HEZgo 6mmn1n+EtlOg6SVX3nD4NgWDyEJwFHGstOkkohuwQx/I41VH20k1MaZKqnWjKQJgJsHre6N5611Fc yc0nPM/RQNL21Ul8taFUWU+uw2hbeVP6K6OJjDkei3dEh1zwM1Ptz8AvdajGnVGXNkqIWexzFl0RB Wm+85tptjBuamY28RW+4L+vnPeQhFn/IX8Wm9r43T7gVAeptC2Btq4Eyprygs4H/VXNm5VYk1k53w lBrN9SWx6ZjaXPjUDhvt9AapoRAsAZtEH82aVaIXYpF63sU52Q0+I3sTdx3+w4Xvqof4wCtOTSEIh JffrJPL90nUBBsUpvwlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLOqd-00F1aQ-Vs; Fri, 27 Jan 2023 13:32:56 +0000 Received: from smtp80.cstnet.cn ([159.226.251.80] helo=cstnet.cn) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLOR1-00EryG-Af for linux-riscv@lists.infradead.org; Fri, 27 Jan 2023 13:06:40 +0000 Received: from cgk-Precision-3650-Tower.. (unknown [219.141.235.82]) by APP-01 (Coremail) with SMTP id qwCowADX3Eg9zNNjDyF3Ag--.52827S13; Fri, 27 Jan 2023 21:06:10 +0800 (CST) From: Chen Guokai To: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, rostedt@goodmis.org, mingo@redhat.com, sfr@canb.auug.org.au Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, liaochang1@huawei.com, Chen Guokai , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= Subject: [PATCH v6 09/13] riscv/kprobe: Search free registers from unused caller-saved ones Date: Fri, 27 Jan 2023 21:05:37 +0800 Message-Id: <20230127130541.1250865-10-chenguokai17@mails.ucas.ac.cn> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230127130541.1250865-1-chenguokai17@mails.ucas.ac.cn> References: <20230127130541.1250865-1-chenguokai17@mails.ucas.ac.cn> MIME-Version: 1.0 X-CM-TRANSID: qwCowADX3Eg9zNNjDyF3Ag--.52827S13 X-Coremail-Antispam: 1UD129KBjvJXoWxtF45Ary3JryUtrWkuryDZFb_yoWfXF15pF sxGw4FvrWUtFykCrZrAw1kGr4fKrs8KrW0vFyUtFyayr43X39IvFWvgF1aqr1DKF4jvr4x Ar45Cry8urW3AFJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUQq14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_JF0E3s1l82xGYI kIc2x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2 z4x0Y4vE2Ix0cI8IcVAFwI0_Gr0_Xr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F 4UJwA2z4x0Y4vEx4A2jsIE14v26r4UJVWxJr1l84ACjcxK6I8E87Iv6xkF7I0E14v26r4U JVWxJr1lnxkEFVAIw20F6cxK64vIFxWle2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64 xvF2IEw4CE5I8CrVC2j2WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j 6r4UMcvjeVCFs4IE7xkEbVWUJVW8JwACjcxG0xvY0x0EwIxGrwACjI8F5VA0II8E6IAqYI 8I648v4I1lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc2xSY4AK67AK6ry8MxAIw28IcxkI7VAK I48JMxC20s026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7 xvwVAFwI0_JrI_JrWlx4CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xII jxv20xvE14v26r1I6r4UMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42 xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF 7I0E14v26r4UJVWxJrUvcSsGvfC2KfnxnUUI43ZEXa7VUjTmh7UUUUU== X-Originating-IP: [219.141.235.82] X-CM-SenderInfo: xfkh0w5xrntxyrx6ztxlovh3xfdvhtffof0/1tbiBwUEE2PTsCcetQABsY X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_050627_855996_5148F23C X-CRM114-Status: GOOD ( 20.94 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org This patch further allows optprobe to use caller-saved registers that is not used across the function being optimized as free registers. Signed-off-by: Chen Guokai Co-developed-by: Liao Chang Signed-off-by: Liao Chang Reported-by: Björn Töpel Reported-by: should be used for fixes. Please change to Suggested-by:, --- arch/riscv/include/asm/kprobes.h | 1 + arch/riscv/kernel/probes/decode-insn.h | 29 +++++++ arch/riscv/kernel/probes/opt.c | 116 ++++++++++++++++++++++--- 3 files changed, 134 insertions(+), 12 deletions(-) diff --git a/arch/riscv/include/asm/kprobes.h b/arch/riscv/include/asm/kprobes.h index 75ebd02be171..f7d33f6861c6 100644 --- a/arch/riscv/include/asm/kprobes.h +++ b/arch/riscv/include/asm/kprobes.h @@ -86,6 +86,7 @@ struct arch_optimized_insn { kprobe_opcode_t *insn; unsigned long length; int rd; + u32 free_reg; }; #endif /* CONFIG_OPTPROBES */ diff --git a/arch/riscv/kernel/probes/decode-insn.h b/arch/riscv/kernel/probes/decode-insn.h index 785b023a62ea..140f5b6a9886 100644 --- a/arch/riscv/kernel/probes/decode-insn.h +++ b/arch/riscv/kernel/probes/decode-insn.h @@ -13,6 +13,35 @@ enum probe_insn { INSN_GOOD, }; +#define NRREG 32 +#define ALL_REG_OCCUPIED 0xffffffffu +/* + * Register ABI Name Saver + * x0 zero -- + * x1 ra Caller + * x2 sp Callee + * x3 gp -- + * x4 tp -- + * x5-7 t0-2 Caller + * x8 so/fp Callee + * x9 so/fp Callee + * x10-11 a0-1 Caller + * x12-17 a2-7 Caller + * x18-27 s2-11 Callee + * x28-32 t3-6 Caller + * + * If register is not caller-saved, it is potentially unsafe to used + * as a free register to form AUIPC/JALR, then use one bitmask to filter + * out these registers. Because ra is used to record return address for + * function call, so mark ra as non-caller-saved register here. + * */ +#define NON_CALLER_SAVED_MASK \ + (1 << 0) | (1 << 1) | (1 << 2) | (1 << 3) | \ + (1 << 4) | (1 << 8) | (1 << 9) | (1 << 18) | \ + (1 << 19) | (1 << 20) | (1 << 21) | (1 << 22) | \ + (1 << 23) | (1 << 24) | (1 << 25) | (1 << 26) | \ + (1 << 27) + enum probe_insn __kprobes riscv_probe_decode_insn(probe_opcode_t *addr, struct arch_probe_insn *asi); diff --git a/arch/riscv/kernel/probes/opt.c b/arch/riscv/kernel/probes/opt.c index c52d5bdc748c..e151b1c60d6d 100644 --- a/arch/riscv/kernel/probes/opt.c +++ b/arch/riscv/kernel/probes/opt.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "simulate-insn.h" #include "decode-insn.h" @@ -126,7 +127,7 @@ static void prepare_detour_buffer(kprobe_opcode_t *code, kprobe_opcode_t *slot, * as a destination register before any branch or jump instruction. */ static void find_register(unsigned long start, unsigned long end, - unsigned long *write, unsigned long *read) + unsigned long *write, unsigned long *read) { kprobe_opcode_t insn; unsigned long addr, offset = 0UL; @@ -385,18 +386,101 @@ static int search_copied_insn(unsigned long paddr, struct optimized_kprobe *op) return 0; } +static void update_free_reg(unsigned long addr, uint32_t *used_reg) +{ + kprobe_opcode_t insn = *(kprobe_opcode_t *)addr; + unsigned long offset = GET_INSN_LENGTH(insn); + +#ifdef CONFIG_RISCV_ISA_C + if (offset == RVI_INSN_LEN) + goto is_rvi; + + insn &= __COMPRESSED_INSN_MASK; + if (riscv_insn_is_c_jal(insn)) { + *used_reg |= 1 << 1; + } else if (riscv_insn_is_c_jr(insn)) { + *used_reg |= 1 << rvc_r_rs1(insn); + } else if (riscv_insn_is_c_jalr(insn)) { + *used_reg |= 1 << rvc_r_rs1(insn); + } else if (riscv_insn_is_c_beqz(insn) || riscv_insn_is_c_bnez(insn)) { + *used_reg |= 1 << rvc_b_rs(insn); + } else if (riscv_insn_is_c_sub(insn) || riscv_insn_is_c_subw(insn)) { + *used_reg |= 1 << rvc_a_rs1(insn); + *used_reg |= 1 << rvc_a_rs2(insn); + } else if (riscv_insn_is_c_sq(insn) || riscv_insn_is_c_sw(insn) || + riscv_insn_is_c_sd(insn)) { + *used_reg |= 1 << rvc_s_rs1(insn); + *used_reg |= 1 << rvc_s_rs2(insn); + } else if (riscv_insn_is_c_addi16sp(insn) || riscv_insn_is_c_addi(insn) || + riscv_insn_is_c_addiw(insn) || + riscv_insn_is_c_slli(insn)) { + *used_reg |= 1 << rvc_i_rs1(insn); + } else if (riscv_insn_is_c_sri(insn) || + riscv_insn_is_c_andi(insn)) { + *used_reg |= 1 << rvc_b_rs(insn); + } else if (riscv_insn_is_c_sqsp(insn) || riscv_insn_is_c_swsp(insn) || + riscv_insn_is_c_sdsp(insn)) { + *used_reg |= 1 << rvc_ss_rs2(insn); + *used_reg |= 1 << 2; + } else if (riscv_insn_is_c_mv(insn)) { + *used_reg |= 1 << rvc_r_rs2(insn); + } else if (riscv_insn_is_c_addi4spn(insn)) { + *used_reg |= 1 << 2; + } else if (riscv_insn_is_c_lq(insn) || riscv_insn_is_c_lw(insn) || + riscv_insn_is_c_ld(insn)) { + *used_reg |= 1 << rvc_l_rs(insn); + } else if (riscv_insn_is_c_lqsp(insn) || riscv_insn_is_c_lwsp(insn) || + riscv_insn_is_c_ldsp(insn)) { + *used_reg |= 1 << 2; + } + /* li and lui does not have source reg */ + return; +is_rvi: +#endif + if (riscv_insn_is_arith_ri(insn) || riscv_insn_is_load(insn)) { + *used_reg |= 1 << rvi_rs1(insn); + } else if (riscv_insn_is_arith_rr(insn) || riscv_insn_is_store(insn) || + riscv_insn_is_amo(insn)) { + *used_reg |= 1 << rvi_rs1(insn); + *used_reg |= 1 << rvi_rs2(insn); + } else if (riscv_insn_is_branch(insn)) { + *used_reg |= 1 << rvi_rs1(insn); + *used_reg |= 1 << rvi_rs2(insn); + } else if (riscv_insn_is_jalr(insn)) { + *used_reg |= 1 << rvi_rs1(insn); + } +} + +static bool scan_code(unsigned long *addr, unsigned long paddr, + struct optimized_kprobe *op, uint32_t *used_reg) +{ + if (insn_jump_into_range(*addr, paddr + RVC_INSN_LEN, + paddr + op->optinsn.length)) + return false; + if (search_exception_tables(*addr)) + return false; + update_free_reg(*addr, used_reg); + *addr += GET_INSN_LENGTH(*(kprobe_opcode_t *)addr); + return true; +} + /* * The kprobe based on breakpoint just requires the instrumented instruction * supports execute out-of-line or simulation, besides that, optimized kprobe * requires no near instruction jump to any instruction replaced by AUIPC/JALR. */ -static bool can_optimize(unsigned long paddr, struct optimized_kprobe *op) +static bool can_optimize(unsigned long paddr, struct optimized_kprobe *op, uint32_t *used_reg) { int ret; struct arch_probe_insn api; unsigned long addr, size = 0, offset = 0; struct kprobe *kp = get_kprobe((kprobe_opcode_t *)paddr); + /* + * All callee + */ + *used_reg = NON_CALLER_SAVED_MASK; + /* * Skip optimization if kprobe has been disarmed or instrumented * instruction doest not support XOI. @@ -426,18 +510,14 @@ static bool can_optimize(unsigned long paddr, struct optimized_kprobe *op) */ addr = paddr - offset; while (addr < paddr) { - if (insn_jump_into_range(addr, paddr + RVC_INSN_LEN, - paddr + op->optinsn.length)) + if (!scan_code(&addr, paddr, op, used_reg)) return false; - addr += GET_INSN_LENGTH(*(kprobe_opcode_t *)addr); } - - addr = paddr + op->optinsn.length; + update_free_reg((unsigned long)&kp->opcode, used_reg); + addr = paddr + GET_INSN_LENGTH(*(kprobe_opcode_t *)&kp->opcode); while (addr < paddr - offset + size) { - if (insn_jump_into_range(addr, paddr + RVC_INSN_LEN, - paddr + op->optinsn.length)) + if (!scan_code(&addr, paddr, op, used_reg)) return false; - addr += GET_INSN_LENGTH(*(kprobe_opcode_t *)addr); } return true; @@ -466,10 +546,13 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op, { long rel; int rd = 0, ra = 0, ret; + u32 used_reg; kprobe_opcode_t *code = NULL, *slot = NULL; - if (!can_optimize((unsigned long)orig->addr, op)) + if (!can_optimize((unsigned long)orig->addr, op, &used_reg)) { + op->optinsn.rd = -1; return -EILSEQ; + } code = kzalloc(MAX_OPTINSN_SIZE, GFP_KERNEL); slot = get_optinsn_slot(); @@ -490,7 +573,14 @@ int arch_prepare_optimized_kprobe(struct optimized_kprobe *op, * to detour buffer, ra is used to form JR jumping back from detour * buffer. */ - find_free_registers(orig, op, &rd, &ra); + if (used_reg == ALL_REG_OCCUPIED) { + find_free_registers(orig, op, &rd, &ra); + } else { + /* Choose one unused caller-saved register. */ + rd = ffz(used_reg); + ra = rd; + } + if (rd == 0 || ra == 0) { ret = -EILSEQ; goto on_error; @@ -534,6 +624,8 @@ void arch_optimize_kprobes(struct list_head *oplist) list_for_each_entry_safe(op, tmp, oplist, list) { WARN_ON(kprobe_disabled(&op->kp)); + if (op->optinsn.rd < 0) + continue; /* Backup instructions which will be replaced by jump address */ memcpy(op->optinsn.copied_insn, DETOUR_ADDR(op->optinsn.insn, DETOUR_INSN_OFFSET),