From patchwork Mon Sep 19 09:21:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Kuohai X-Patchwork-Id: 12979809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5338EC6FA82 for ; Mon, 19 Sep 2022 09:06:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TGvKEGbKhpmL6YcJ5f45wgU4wQSzrm13Q3/Qra6BWO4=; b=YHZ7JULxbz/qSw CJRa4VjxONGUbE9sR/5GEwixMK3UfgQcUpydrd45MtqVg8mv4ExdnQhYwNWMqQbqjULtpty92qiI2 KIDq30HjSH4AbnZINJPT6Eqb+1JiDKkocAHH+V9ZR6DzSYsaemDYb7Z7D/FS+pke+JkwZEJwCYR7R TPGFiqq8YOaOWj8HvA4pLV8u+A+R2T8QGxC7Q4jTKZBLk2qf+gniXzDnPOpPd0XPfmIJOaZADoMT5 HfTTzaqV8DNZlN48FLWTFlEIXzVF86zO2Jo1cQPhQ7mbaTFexSHKE3JJhrJuznl+XWBIjMLHxSd1X +Q1ADyorSNAQeFCJY6FA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oaCiU-00AYmd-Rs; Mon, 19 Sep 2022 09:05:27 +0000 Received: from dggsgout12.his.huawei.com ([45.249.212.56]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oaChn-00AXt8-Eh for linux-arm-kernel@lists.infradead.org; Mon, 19 Sep 2022 09:04:48 +0000 Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4MWJbZ5Jjlz6RBmp for ; Mon, 19 Sep 2022 17:02:34 +0800 (CST) Received: from k01.huawei.com (unknown [10.67.174.197]) by APP2 (Coremail) with SMTP id Syh0CgDHY22cMChjI49JBA--.48437S3; Mon, 19 Sep 2022 17:04:31 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Daniel Borkmann , Jean-Philippe Brucker , Alexei Starovoitov , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Catalin Marinas , Will Deacon Subject: [RESEND PATCH bpf-next 1/2] bpf, arm64: Jit BPF_CALL to direct call when possible Date: Mon, 19 Sep 2022 05:21:37 -0400 Message-Id: <20220919092138.1027353-2-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220919092138.1027353-1-xukuohai@huaweicloud.com> References: <20220919092138.1027353-1-xukuohai@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: Syh0CgDHY22cMChjI49JBA--.48437S3 X-Coremail-Antispam: 1UD129KBjvJXoW3JrW8tr4UuF15ury7Jw1rXrb_yoWxCFW5pa 9xGw1YkrW8XrWUGFs7JF4kAry3Kws5W347Gry8u3yFkas0gr93Gan8K34a9FZ8Ar95Zr1x XF4qyry3ua1UJrDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUB0b4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMc Ij6xIIjxv20xvE14v26r106r15McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_ Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2Ij64 vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8G jcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0xvE2I x0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26r4j6F4UMIIF0xvE42xK 8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxVWUJVW8JwCI42IY6I8E87Iv6xkF7I 0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxU7TKZDUUUU X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220919_020443_898616_7B140918 X-CRM114-Status: GOOD ( 24.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Xu Kuohai Currently BPF_CALL is always jited to indirect call, but when target is in the range of direct call, BPF_CALL can be jited to direct call. For example, the following BPF_CALL call __htab_map_lookup_elem is always jited to an indirect call: mov x10, #0xffffffffffff18f4 movk x10, #0x821, lsl #16 movk x10, #0x8000, lsl #32 blr x10 When the target is in the range of direct call, it can be jited to: bl 0xfffffffffd33bc98 This patch does such jit when possible. 1. First pass, get the maximum jited image size. Since the jited image memory is not allocated yet, the distance between jited BPF_CALL instructon and call target is unknown, so jit all BPF_CALL to indirect call to get the maximum image size. 2. Allocate image memory with the size caculated in step 1. 3. Second pass, determine the jited address and size for every bpf instruction. Since image memory is now allocated and there is only one jit method for bpf instructions other than BPF_CALL, so the jited address for the first BPF_CALL is determined, so the distance to call target is determined, so the first BPF_CALL is determined to be jited to direct or indirect call, so the jited image size after the first BPF_CALL is determined. By analogy, the jited addresses and sizes for all subsequent BPF instructions are determined. 4. Last pass, generate the final image. The jump offset of jump instruction whose target is within the jited image is determined in this pass, since the target instruction address may be changed in step 3. Tested with test_bpf.ko and some arm64 working selftests, nothing failed. Signed-off-by: Xu Kuohai --- arch/arm64/net/bpf_jit_comp.c | 71 ++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 13 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 30f76178608b..06437e34614b 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -72,6 +72,7 @@ static const int bpf2a64[] = { struct jit_ctx { const struct bpf_prog *prog; int idx; + bool write; int epilogue_offset; int *offset; int exentry_idx; @@ -91,7 +92,7 @@ struct bpf_plt { static inline void emit(const u32 insn, struct jit_ctx *ctx) { - if (ctx->image != NULL) + if (ctx->image != NULL && ctx->write) ctx->image[ctx->idx] = cpu_to_le32(insn); ctx->idx++; @@ -178,10 +179,29 @@ static inline void emit_addr_mov_i64(const int reg, const u64 val, static inline void emit_call(u64 target, struct jit_ctx *ctx) { - u8 tmp = bpf2a64[TMP_REG_1]; + u8 tmp; + long offset; + unsigned long pc; + u32 insn = AARCH64_BREAK_FAULT; + + /* if ctx->image == NULL or target == 0, the jump distance is unknown, + * emit indirect call. + */ + if (ctx->image && target) { + pc = (unsigned long)&ctx->image[ctx->idx]; + offset = (long)target - (long)pc; + if (offset >= -SZ_128M && offset < SZ_128M) + insn = aarch64_insn_gen_branch_imm(pc, target, + AARCH64_INSN_BRANCH_LINK); + } - emit_addr_mov_i64(tmp, target, ctx); - emit(A64_BLR(tmp), ctx); + if (insn == AARCH64_BREAK_FAULT) { + tmp = bpf2a64[TMP_REG_1]; + emit_addr_mov_i64(tmp, target, ctx); + emit(A64_BLR(tmp), ctx); + } else { + emit(insn, ctx); + } } static inline int bpf2a64_offset(int bpf_insn, int off, @@ -1392,13 +1412,11 @@ static int build_body(struct jit_ctx *ctx, bool extra_pass) const struct bpf_insn *insn = &prog->insnsi[i]; int ret; - if (ctx->image == NULL) - ctx->offset[i] = ctx->idx; + ctx->offset[i] = ctx->idx; ret = build_insn(insn, ctx, extra_pass); if (ret > 0) { i++; - if (ctx->image == NULL) - ctx->offset[i] = ctx->idx; + ctx->offset[i] = ctx->idx; continue; } if (ret) @@ -1409,8 +1427,7 @@ static int build_body(struct jit_ctx *ctx, bool extra_pass) * the last element with the offset after the last * instruction (end of program) */ - if (ctx->image == NULL) - ctx->offset[i] = ctx->idx; + ctx->offset[i] = ctx->idx; return 0; } @@ -1461,6 +1478,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bool extra_pass = false; struct jit_ctx ctx; u8 *image_ptr; + int body_idx; + int exentry_idx; if (!prog->jit_requested) return orig_prog; @@ -1515,6 +1534,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } + /* Get the max image size */ if (build_body(&ctx, extra_pass)) { prog = orig_prog; goto out_off; @@ -1528,7 +1548,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) extable_size = prog->aux->num_exentries * sizeof(struct exception_table_entry); - /* Now we know the actual image size. */ + /* Now we know the max image size. */ prog_size = sizeof(u32) * ctx.idx; /* also allocate space for plt target */ extable_offset = round_up(prog_size + PLT_TARGET_SIZE, extable_align); @@ -1548,15 +1568,37 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) skip_init_ctx: ctx.idx = 0; ctx.exentry_idx = 0; + ctx.write = true; build_prologue(&ctx, was_classic); + /* Record exentry_idx and ctx.idx before first build_body */ + exentry_idx = ctx.exentry_idx; + body_idx = ctx.idx; + /* Don't write instruction to memory for now */ + ctx.write = false; + + /* Determine call distance and instruction position */ if (build_body(&ctx, extra_pass)) { bpf_jit_binary_free(header); prog = orig_prog; goto out_off; } + ctx.epilogue_offset = ctx.idx; + + ctx.exentry_idx = exentry_idx; + ctx.idx = body_idx; + ctx.write = true; + + /* Determine jump offset and write result to memory */ + if (build_body(&ctx, extra_pass) || + WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) { + bpf_jit_binary_free(header); + prog = orig_prog; + goto out_off; + } + build_epilogue(&ctx); build_plt(&ctx); @@ -1567,6 +1609,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } + /* Update prog size */ + prog_size = sizeof(u32) * ctx.idx; /* And we're done. */ if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, 2, ctx.image); @@ -1574,8 +1618,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_flush_icache(header, ctx.image + ctx.idx); if (!prog->is_func || extra_pass) { - if (extra_pass && ctx.idx != jit_data->ctx.idx) { - pr_err_once("multi-func JIT bug %d != %d\n", + if (extra_pass && ctx.idx > jit_data->ctx.idx) { + pr_err_once("multi-func JIT bug %d > %d\n", ctx.idx, jit_data->ctx.idx); bpf_jit_binary_free(header); prog->bpf_func = NULL; @@ -1976,6 +2020,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, struct jit_ctx ctx = { .image = NULL, .idx = 0, + .write = true, }; /* the first 8 arguments are passed by registers */