From patchwork Fri Sep 2 15:20:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Kuohai X-Patchwork-Id: 12964322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7647BECAAD5 for ; Fri, 2 Sep 2022 15:14:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3kS1AEuav3g87X+DqdYjQeu4uLXfFIvHGdVl8shxnCA=; b=J2l1ai234qsUq0 vAxuq3O4gWB58mJ9faohApzzh1hDDnWiKsFc3C4dCd8NnW+DUFAzCkT2HbagxvUezLK6mjXRtBV++ cYM2pkZSoGvWwjqBrgjctvtgktBJpCMwOuVLOJ1cV17Q+T9TNPP+phH02SH2G2QIQ+6K0b9CTuFEn vYIei2STTzWh7gYpvmCCVYkR2xBKCfI2FVLkhBMKKRWVjI9khvGD4c7HvcuarbaW3LlH5vWXYgCFB TDP6jFHIU04rkNpKca5d59G8mRZ2oZZA8c8sKMrH/sHiXzWa7Q0B7VEOyzFVF84dAtnQH5FKiF6P3 mDij0ewYKYOIfnKCIyDg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU8Lz-0060nP-U1; Fri, 02 Sep 2022 15:13:08 +0000 Received: from dggsgout11.his.huawei.com ([45.249.212.51]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU8Lm-0060cs-4v for linux-arm-kernel@lists.infradead.org; Fri, 02 Sep 2022 15:12:56 +0000 Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4MK1Zn0D2xzlCBN for ; Fri, 2 Sep 2022 23:11:13 +0800 (CST) Received: from k01.huawei.com (unknown [10.67.174.197]) by APP3 (Coremail) with SMTP id _Ch0CgBn4UhlHRJja8YAAQ--.58277S3; Fri, 02 Sep 2022 23:12:40 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Daniel Borkmann , Jean-Philippe Brucker , Alexei Starovoitov , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Catalin Marinas , Will Deacon Subject: [PATCH bpf-next 1/2] bpf, arm64: Jit BPF_CALL to direct call when possible Date: Fri, 2 Sep 2022 11:20:41 -0400 Message-Id: <20220902152043.721806-2-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220902152043.721806-1-xukuohai@huaweicloud.com> References: <20220902152043.721806-1-xukuohai@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgBn4UhlHRJja8YAAQ--.58277S3 X-Coremail-Antispam: 1UD129KBjvJXoW3JrykurWkKFWfJr1kCF43Awb_yoWxGFyUpa 9rGw1YkrW8Xr47GFs7J3WkAry3Kws5W347KryfurWFkas0qr93Gan8K34a9FZxAr95Cr1x ZFWqyry3ua1UJrDanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBjb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Gr0_Xr1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW0oVCq3wA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ GcCE3s1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx 0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkEbVWU JVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCF04k20xvY0x0EwI xGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480 Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_GFv_WrylIxkGc2Ij64vIr41lIxAIcVC0I7 IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k2 6cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxV AFwI0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x07jn9N3UUUUU= X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220902_081254_786430_7D76CF3B X-CRM114-Status: GOOD ( 23.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Xu Kuohai Currently BPF_CALL is always jited to indirect call, but when target is in the range of direct call, a BPF_CALL can be jited to direct call. For example, the following BPF_CALL call __htab_map_lookup_elem is always jited to an indirect call: mov x10, #0xffffffffffff18f4 movk x10, #0x821, lsl #16 movk x10, #0x8000, lsl #32 blr x10 When the target is in the range of a direct call, it can be jited to: bl 0xfffffffffd33bc98 This patch does such jit when possible. 1. Before allocating jit image memory, jit all BPF_CALL to indirect call, so we could get the maximum image size at the end. 2. Allocate jit image memory with the maximum image size. 3. Since we have now allocated jit image memory, every jited instruction address is determined, so the distance to call target is determined, so whether a BPF_CALL can be jited to direct call is determined. 4. Since step 3 may adjust the position of jited instructions, update the offset of jump instructions whose target is within the jit image. Tested with test_bpf.ko and some arm64 working selftests, nothing failed. Signed-off-by: Xu Kuohai --- arch/arm64/net/bpf_jit_comp.c | 71 ++++++++++++++++++++++++++++------- 1 file changed, 58 insertions(+), 13 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 389623ae5a91..69eb29f397e5 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -72,6 +72,7 @@ static const int bpf2a64[] = { struct jit_ctx { const struct bpf_prog *prog; int idx; + bool write; int epilogue_offset; int *offset; int exentry_idx; @@ -91,7 +92,7 @@ struct bpf_plt { static inline void emit(const u32 insn, struct jit_ctx *ctx) { - if (ctx->image != NULL) + if (ctx->image != NULL && ctx->write) ctx->image[ctx->idx] = cpu_to_le32(insn); ctx->idx++; @@ -178,10 +179,29 @@ static inline void emit_addr_mov_i64(const int reg, const u64 val, static inline void emit_call(u64 target, struct jit_ctx *ctx) { - u8 tmp = bpf2a64[TMP_REG_1]; + u8 tmp; + long offset; + unsigned long pc; + u32 insn = AARCH64_BREAK_FAULT; + + /* if ctx->image == NULL or target == 0, the jump distance is unknown, + * emit indirect call. + */ + if (ctx->image && target) { + pc = (unsigned long)&ctx->image[ctx->idx]; + offset = (long)target - (long)pc; + if (offset >= -SZ_128M && offset < SZ_128M) + insn = aarch64_insn_gen_branch_imm(pc, target, + AARCH64_INSN_BRANCH_LINK); + } - emit_addr_mov_i64(tmp, target, ctx); - emit(A64_BLR(tmp), ctx); + if (insn == AARCH64_BREAK_FAULT) { + tmp = bpf2a64[TMP_REG_1]; + emit_addr_mov_i64(tmp, target, ctx); + emit(A64_BLR(tmp), ctx); + } else { + emit(insn, ctx); + } } static inline int bpf2a64_offset(int bpf_insn, int off, @@ -1392,13 +1412,11 @@ static int build_body(struct jit_ctx *ctx, bool extra_pass) const struct bpf_insn *insn = &prog->insnsi[i]; int ret; - if (ctx->image == NULL) - ctx->offset[i] = ctx->idx; + ctx->offset[i] = ctx->idx; ret = build_insn(insn, ctx, extra_pass); if (ret > 0) { i++; - if (ctx->image == NULL) - ctx->offset[i] = ctx->idx; + ctx->offset[i] = ctx->idx; continue; } if (ret) @@ -1409,8 +1427,7 @@ static int build_body(struct jit_ctx *ctx, bool extra_pass) * the last element with the offset after the last * instruction (end of program) */ - if (ctx->image == NULL) - ctx->offset[i] = ctx->idx; + ctx->offset[i] = ctx->idx; return 0; } @@ -1461,6 +1478,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bool extra_pass = false; struct jit_ctx ctx; u8 *image_ptr; + int body_offset; + int exentry_idx; if (!prog->jit_requested) return orig_prog; @@ -1515,6 +1534,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } + /* Get the max image size */ if (build_body(&ctx, extra_pass)) { prog = orig_prog; goto out_off; @@ -1528,7 +1548,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) extable_size = prog->aux->num_exentries * sizeof(struct exception_table_entry); - /* Now we know the actual image size. */ + /* Now we know the max image size. */ prog_size = sizeof(u32) * ctx.idx; /* also allocate space for plt target */ extable_offset = round_up(prog_size + PLT_TARGET_SIZE, extable_align); @@ -1548,15 +1568,37 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) skip_init_ctx: ctx.idx = 0; ctx.exentry_idx = 0; + ctx.write = true; build_prologue(&ctx, was_classic); + /* Record exentry_idx and ctx.idx before first build_body */ + exentry_idx = ctx.exentry_idx; + body_offset = ctx.idx; + /* Don't write instruction to memory for now */ + ctx.write = false; + + /* Determine call distance and instruction position */ if (build_body(&ctx, extra_pass)) { bpf_jit_binary_free(header); prog = orig_prog; goto out_off; } + ctx.epilogue_offset = ctx.idx; + + ctx.exentry_idx = exentry_idx; + ctx.idx = body_offset; + ctx.write = true; + + /* Determine jump offset and write result to memory */ + if (build_body(&ctx, extra_pass) || + WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) { + bpf_jit_binary_free(header); + prog = orig_prog; + goto out_off; + } + build_epilogue(&ctx); build_plt(&ctx); @@ -1567,6 +1609,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } + /* Update prog size */ + prog_size = sizeof(u32) * ctx.idx; /* And we're done. */ if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, prog_size, 2, ctx.image); @@ -1574,8 +1618,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_flush_icache(header, ctx.image + ctx.idx); if (!prog->is_func || extra_pass) { - if (extra_pass && ctx.idx != jit_data->ctx.idx) { - pr_err_once("multi-func JIT bug %d != %d\n", + if (extra_pass && ctx.idx > jit_data->ctx.idx) { + pr_err_once("multi-func JIT bug %d > %d\n", ctx.idx, jit_data->ctx.idx); bpf_jit_binary_free(header); prog->bpf_func = NULL; @@ -1976,6 +2020,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, struct jit_ctx ctx = { .image = NULL, .idx = 0, + .write = true, }; /* the first 8 arguments are passed by registers */ From patchwork Fri Sep 2 15:20:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xu Kuohai X-Patchwork-Id: 12964323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79D39C38145 for ; Fri, 2 Sep 2022 15:14:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9MSLxwTIKVLeEKmgpkw6pMgdlhEPfSTqDQ2pTfuTSNQ=; b=KWRg42COmTALYD Hpm3Mz1MaCIJbWena0FclTajtMxd3gV53c60WFYpYQ8S3ivHo8sYJWy9V2iJA0IS32V2nEjRy/xWm /+WhmXCHoeMEmRhTpfEghFUOzpfTkEEvv0KPskXmbNQm32diJ+WO/pgRLSrKjBWLDWWhmWxfpOAtd 6c21kb4q7kd4H99NM3o2sHnOjIFo/oHc++AVn5bfwHnj7jnhHYtHLHqPsCQLNUok5YEmC2kFgIvCF Np4ivR17JVFH9ccjhc+LiKpN+oUgUFpXeJr/bpHc3X+VCsqDeLDZOHOnpWzhm5T8niKKl51bTgDNn jrZ7e+HjSHmzXXDGz3+w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU8MB-0060vR-WB; Fri, 02 Sep 2022 15:13:20 +0000 Received: from dggsgout11.his.huawei.com ([45.249.212.51]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oU8Lm-0060cv-BK for linux-arm-kernel@lists.infradead.org; Fri, 02 Sep 2022 15:12:58 +0000 Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4MK1Zp34gSzlCGh for ; Fri, 2 Sep 2022 23:11:14 +0800 (CST) Received: from k01.huawei.com (unknown [10.67.174.197]) by APP3 (Coremail) with SMTP id _Ch0CgBn4UhlHRJja8YAAQ--.58277S4; Fri, 02 Sep 2022 23:12:42 +0800 (CST) From: Xu Kuohai To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Daniel Borkmann , Jean-Philippe Brucker , Alexei Starovoitov , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Catalin Marinas , Will Deacon Subject: [PATCH bpf-next 2/2] bpf, arm64: Eliminate false -EFBIG error in bpf trampoline Date: Fri, 2 Sep 2022 11:20:42 -0400 Message-Id: <20220902152043.721806-3-xukuohai@huaweicloud.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220902152043.721806-1-xukuohai@huaweicloud.com> References: <20220902152043.721806-1-xukuohai@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgBn4UhlHRJja8YAAQ--.58277S4 X-Coremail-Antispam: 1UD129KBjvJXoW3Jr4UGw18CrWkWrWUur4Durg_yoWxAF4rpa yDGw4YyFW8Xrs8Wa1kXFWUAF1akw1qgry7CrWUG3yS9a4YqryDGa4UGFyYvrW5urZ5Ar1x ZF4qyF98CrWxGwUanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUBYb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMc Ij6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_ Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1l42xK82IYc2Ij64 vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8G jcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r4a6rW5MIIYrxkI7VAKI48JMIIF0xvE2I x0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY6xAI w20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x 0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7IU1sa9DUUUUU== X-CM-SenderInfo: 50xn30hkdlqx5xdzvxpfor3voofrz/ X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220902_081254_882478_09267AA7 X-CRM114-Status: GOOD ( 20.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Xu Kuohai With jiting BPF_CALL to indirect call or direct call based on the jump distance, the bpf trampoline size calculated with NULL image address may be larger than the actual size needed, resulting in a fake -EFBIG error. So remove the estimate of the bpf trampoline size based on NULL image address, and check if there is enough space when emitting instructions. Signed-off-by: Xu Kuohai --- arch/arm64/net/bpf_jit_comp.c | 67 ++++++++++++++++------------------- 1 file changed, 30 insertions(+), 37 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 69eb29f397e5..34d78ca16beb 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -73,6 +73,7 @@ struct jit_ctx { const struct bpf_prog *prog; int idx; bool write; + int max_insns; int epilogue_offset; int *offset; int exentry_idx; @@ -90,11 +91,15 @@ struct bpf_plt { #define PLT_TARGET_SIZE sizeof_field(struct bpf_plt, target) #define PLT_TARGET_OFFSET offsetof(struct bpf_plt, target) -static inline void emit(const u32 insn, struct jit_ctx *ctx) +static inline void __emit(u32 insn, struct jit_ctx *ctx, int idx) { - if (ctx->image != NULL && ctx->write) - ctx->image[ctx->idx] = cpu_to_le32(insn); + if (ctx->image != NULL && ctx->write && idx < ctx->max_insns) + ctx->image[idx] = cpu_to_le32(insn); +} +static inline void emit(u32 insn, struct jit_ctx *ctx) +{ + __emit(insn, ctx, ctx->idx); ctx->idx++; } @@ -1544,6 +1549,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) build_epilogue(&ctx); build_plt(&ctx); + ctx.max_insns = ctx.idx; extable_align = __alignof__(struct exception_table_entry); extable_size = prog->aux->num_exentries * sizeof(struct exception_table_entry); @@ -1603,7 +1609,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) build_plt(&ctx); /* 3. Extra pass to validate JITed code. */ - if (validate_ctx(&ctx)) { + if (WARN_ON_ONCE(ctx.idx > ctx.max_insns) || validate_ctx(&ctx)) { bpf_jit_binary_free(header); prog = orig_prog; goto out_off; @@ -1687,7 +1693,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, int args_off, int retval_off, int run_ctx_off, bool save_ret) { - __le32 *branch; + int bridx; u64 enter_prog; u64 exit_prog; struct bpf_prog *p = l->link.prog; @@ -1725,7 +1731,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, /* if (__bpf_prog_enter(prog) == 0) * goto skip_exec_of_prog; */ - branch = ctx->image + ctx->idx; + bridx = ctx->idx; emit(A64_NOP, ctx); /* save return value to callee saved register x20 */ @@ -1740,10 +1746,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, if (save_ret) emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx); - if (ctx->image) { - int offset = &ctx->image[ctx->idx] - branch; - *branch = cpu_to_le32(A64_CBZ(1, A64_R(0), offset)); - } + __emit(A64_CBZ(1, A64_R(0), ctx->idx - bridx), ctx, bridx); /* arg1: prog */ emit(A64_MOV(1, A64_R(0), A64_R(19)), ctx); @@ -1757,7 +1760,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, int args_off, int retval_off, int run_ctx_off, - __le32 **branches) + int *bridx) { int i; @@ -1775,7 +1778,7 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, /* Save the location of branch, and generate a nop. * This nop will be replaced with a cbnz later. */ - branches[i] = ctx->image + ctx->idx; + bridx[i] = ctx->idx; emit(A64_NOP, ctx); } } @@ -1828,7 +1831,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; bool save_ret; - __le32 **branches = NULL; + int *bridx = NULL; /* trampoline stack layout: * [ parent ip ] @@ -1936,13 +1939,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, flags & BPF_TRAMP_F_RET_FENTRY_RET); if (fmod_ret->nr_links) { - branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *), - GFP_KERNEL); - if (!branches) + bridx = kcalloc(fmod_ret->nr_links, sizeof(int), GFP_KERNEL); + if (!bridx) return -ENOMEM; invoke_bpf_mod_ret(ctx, fmod_ret, args_off, retval_off, - run_ctx_off, branches); + run_ctx_off, bridx); } if (flags & BPF_TRAMP_F_CALL_ORIG) { @@ -1957,11 +1959,10 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, emit(A64_NOP, ctx); } - /* update the branches saved in invoke_bpf_mod_ret with cbnz */ - for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) { - int offset = &ctx->image[ctx->idx] - branches[i]; - *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset)); - } + /* update the bridx saved in invoke_bpf_mod_ret with cbnz */ + for (i = 0; i < fmod_ret->nr_links; i++) + __emit(A64_CBNZ(1, A64_R(10), ctx->idx - bridx[i]), ctx, + bridx[i]); for (i = 0; i < fexit->nr_links; i++) invoke_bpf_prog(ctx, fexit->links[i], args_off, retval_off, @@ -2004,7 +2005,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, if (ctx->image) bpf_flush_icache(ctx->image, ctx->image + ctx->idx); - kfree(branches); + kfree(bridx); return ctx->idx; } @@ -2018,35 +2019,27 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, int nargs = m->nr_args; int max_insns = ((long)image_end - (long)image) / AARCH64_INSN_SIZE; struct jit_ctx ctx = { - .image = NULL, + .image = image, .idx = 0, .write = true, + .max_insns = max_insns, }; /* the first 8 arguments are passed by registers */ if (nargs > 8) return -ENOTSUPP; + jit_fill_hole(image, (unsigned int)(image_end - image)); + ret = prepare_trampoline(&ctx, im, tlinks, orig_call, nargs, flags); + if (ret < 0) return ret; if (ret > max_insns) return -EFBIG; - ctx.image = image; - ctx.idx = 0; - - jit_fill_hole(image, (unsigned int)(image_end - image)); - ret = prepare_trampoline(&ctx, im, tlinks, orig_call, nargs, flags); - - if (ret > 0 && validate_code(&ctx) < 0) - ret = -EINVAL; - - if (ret > 0) - ret *= AARCH64_INSN_SIZE; - - return ret; + return validate_code(&ctx) < 0 ? -EINVAL : ret * AARCH64_INSN_SIZE; } static bool is_long_jump(void *ip, void *target)