From patchwork Sun Sep 17 00:00:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13388537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 26321C2BA2B for ; Sun, 17 Sep 2023 00:01:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=OOChRMLyByHYZf/qSaUqCi3EQpvc43eckbkW+RkDK3Q=; b=gWXZonPq8YK6e6 A3Ld6NN6nidq5Epw8+QNGRLuZUd5Yhsf6cDoAUD1qt2d2cC9U2dlQBYZUvdPvpuuEhcDE3Lz5QStD eRt673yTvGsahmW6u4iX+Q25ob1pgA/bBOFJrfGkTVBbgawcNsnq+lg8/UGvvzDkR2YT9Q0kcNJ2v CHKZ5vpmU9jE40vDvGLAORNv7lDJHfbBosMBfbv5DL8mdKtl88O0i/uia1kOdeuX/n9OJY0P/p7nY 6sHXZWHMt5ya2qinXyqA8/u14dbqerjtdQ3whRlH3+Pms/DsQwBddMU+/c6jRkdGhnVatFVOwUQMj X/kco7dbVOSV5EcyRVjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qhfE2-00CwWM-1R; Sun, 17 Sep 2023 00:01:22 +0000 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qhfDy-00CwVM-0n for linux-arm-kernel@lists.infradead.org; Sun, 17 Sep 2023 00:01:20 +0000 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-401b393ddd2so38365125e9.0 for ; Sat, 16 Sep 2023 17:01:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1694908873; x=1695513673; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WnIlyJVW2gmhP2Dn/Bd/4E/FZ5hvuh+9FvEMQcxgx7g=; b=QOnlPvipFODVY7LwrEI6Wy7u9KZjAIbZ5BWD2E1KMCCLLjdqa/gfQd/5mF/p4GjNIT yVHhov6nBYAGD8wu0lluPmG6VmnWrB67jFxnwBaeTWiKSF9oKkFil2UjNBAq/35kTQ0w PK4km2AUKzTLeVHfKysvG6ePxBpDtnC3WK8aDLNminmrl/WFFFXAc4ml3LMx9kpLKJY7 591ofUJZP3iUiN0hBH/LEB9pD6ihYGC7EcNb5CXnKvJfBG/Z3OQHK8lbEspdpI4gMoYW utPCFPs2toHzQaeA8lIkEe7teac1BKKIE238Ar+uI1ayzTV/8OHJf9EHnFIvDmvDt3Bc Wlow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694908873; x=1695513673; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WnIlyJVW2gmhP2Dn/Bd/4E/FZ5hvuh+9FvEMQcxgx7g=; b=oSfQzoWenOgJk5ajRBVy9KnsualN8VTjVLXxHGt36Xo0QG10g83mhlaKtYObDEDm1k 5rjB9CF2hhZ3qkdCFLlhJjwyl1z5j/A2zJJ+Dm3zmKUCQ6PHLvIRUclqDY3cGXBWAjH8 mwGpFrcrGVogbEo8xVCb5A0zYdTWISv4NM3o8QigoejDPhc+51kMc7XfIwgmkCXWSGWX vwKwgdv2Wun1T1LLBUvSTtVPT377q652RrdlU+BALXC0OUsEolLg3O8zXMs6BzIGsS3u WcYwjHZFC7AGjitvxFPtusnxDr7vv9TpUt4Ms/sWUvLUy7fynYR56g+umWVQxngXk1Ro RMlg== X-Gm-Message-State: AOJu0YyJjHu4y7vJiU0+jlYb3Tl9CDFLUxsw4L5o+C0QSPXwGIE/xqfn DEwVVWcvJK0J3aij0wNKTWk= X-Google-Smtp-Source: AGHT+IEAdhWjvaq6degj0KLRRnbSPHVlyis+LhgCKjvEG2cpxMXbZq2rzBpHF6yH7BHd0gUA6VyG4A== X-Received: by 2002:a7b:c857:0:b0:3ff:233f:2cfb with SMTP id c23-20020a7bc857000000b003ff233f2cfbmr4537605wml.23.1694908873308; Sat, 16 Sep 2023 17:01:13 -0700 (PDT) Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id u7-20020a7bc047000000b003fc01189b0dsm8307616wmc.42.2023.09.16.17.01.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 16 Sep 2023 17:01:12 -0700 (PDT) From: Puranjay Mohan To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Catalin Marinas , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Kumar Kartikeya Dwivedi Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v2 1/1] bpf, arm64: support exceptions Date: Sun, 17 Sep 2023 00:00:45 +0000 Message-Id: <20230917000045.56377-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230917000045.56377-1-puranjay12@gmail.com> References: <20230917000045.56377-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230916_170118_310427_099FBA9E X-CRM114-Status: GOOD ( 25.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Implement arch_bpf_stack_walk() for the ARM64 JIT. This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program. The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed. As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue. The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them. These extra instructions are only added if bpf_throw() used. Otherwise the emitted prologue/epilogue remains unchanged. [1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst Signed-off-by: Puranjay Mohan Acked-by: Kumar Kartikeya Dwivedi --- arch/arm64/net/bpf_jit_comp.c | 98 ++++++++++++++++---- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 - 2 files changed, 79 insertions(+), 20 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 7d4af64e3982..fcc55e558863 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "bpf_jit.h" @@ -285,7 +286,7 @@ static bool is_lsi_offset(int offset, int scale) /* Tail call offset to jump into */ #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8) -static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) +static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, bool is_exception_cb) { const struct bpf_prog *prog = ctx->prog; const bool is_main_prog = !bpf_is_subprog(prog); @@ -333,19 +334,28 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) emit(A64_MOV(1, A64_R(9), A64_LR), ctx); emit(A64_NOP, ctx); - /* Sign lr */ - if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) - emit(A64_PACIASP, ctx); - - /* Save FP and LR registers to stay align with ARM64 AAPCS */ - emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); - emit(A64_MOV(1, A64_FP, A64_SP), ctx); - - /* Save callee-saved registers */ - emit(A64_PUSH(r6, r7, A64_SP), ctx); - emit(A64_PUSH(r8, r9, A64_SP), ctx); - emit(A64_PUSH(fp, tcc, A64_SP), ctx); - emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx); + if (!is_exception_cb) { + /* Sign lr */ + if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) + emit(A64_PACIASP, ctx); + /* Save FP and LR registers to stay align with ARM64 AAPCS */ + emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); + emit(A64_MOV(1, A64_FP, A64_SP), ctx); + + /* Save callee-saved registers */ + emit(A64_PUSH(r6, r7, A64_SP), ctx); + emit(A64_PUSH(r8, r9, A64_SP), ctx); + emit(A64_PUSH(fp, tcc, A64_SP), ctx); + emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx); + } else { + /* Exception callback receives FP of Main Program as third parameter */ + emit(A64_MOV(1, A64_FP, A64_R(2)), ctx); + /* + * Main Program already pushed the frame record and the callee-saved registers. The + * exception callback will not push anything and re-use the main program's stack. + */ + emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); /* 10 registers are on the stack */ + } /* Set up BPF prog stack base register */ emit(A64_MOV(1, fp, A64_SP), ctx); @@ -365,6 +375,13 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) emit_bti(A64_BTI_J, ctx); } + /* + * Program acting as exception boundary should save all ARM64 Callee-saved registers as the + * exception callback needs to recover all ARM64 Callee-saved registers in its epilogue. + */ + if (prog->aux->exception_boundary) + emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx); + emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx); /* Stack must be multiples of 16B */ @@ -653,7 +670,7 @@ static void build_plt(struct jit_ctx *ctx) plt->target = (u64)&dummy_tramp; } -static void build_epilogue(struct jit_ctx *ctx) +static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb) { const u8 r0 = bpf2a64[BPF_REG_0]; const u8 r6 = bpf2a64[BPF_REG_6]; @@ -666,6 +683,14 @@ static void build_epilogue(struct jit_ctx *ctx) /* We're done with BPF stack */ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); + /* + * Program acting as exception boundary pushes R23 and R24 in addition to BPF callee-saved + * registers. Exception callback uses the boundary program's stack frame, so recover these + * extra registers in the above two cases. + */ + if (ctx->prog->aux->exception_boundary || is_exception_cb) + emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx); + /* Restore x27 and x28 */ emit(A64_POP(fpb, A64_R(28), A64_SP), ctx); /* Restore fs (x25) and x26 */ @@ -1575,7 +1600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) * BPF line info needs ctx->offset[i] to be the offset of * instruction[i] in jited image, so build prologue first. */ - if (build_prologue(&ctx, was_classic)) { + if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) { prog = orig_prog; goto out_off; } @@ -1586,7 +1611,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) } ctx.epilogue_offset = ctx.idx; - build_epilogue(&ctx); + build_epilogue(&ctx, prog->aux->exception_cb); build_plt(&ctx); extable_align = __alignof__(struct exception_table_entry); @@ -1614,7 +1639,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.idx = 0; ctx.exentry_idx = 0; - build_prologue(&ctx, was_classic); + build_prologue(&ctx, was_classic, prog->aux->exception_cb); if (build_body(&ctx, extra_pass)) { bpf_jit_binary_free(header); @@ -1622,7 +1647,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } - build_epilogue(&ctx); + build_epilogue(&ctx, prog->aux->exception_cb); build_plt(&ctx); /* 3. Extra pass to validate JITed code. */ @@ -2286,3 +2311,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, return ret; } + +bool bpf_jit_supports_exceptions(void) +{ + /* We unwind through both kernel frames (starting from within bpf_throw call) and + * BPF frames. Therefore we require FP unwinder to be enabled to walk kernel frames and + * reach BPF frames in the stack trace. + * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y + */ + return true; +} + +void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) +{ + struct stack_info stacks[] = { + stackinfo_get_task(current), + }; + + struct unwind_state state = { + .stacks = stacks, + .nr_stacks = ARRAY_SIZE(stacks), + }; + unwind_init_common(&state, current); + state.fp = (unsigned long)__builtin_frame_address(1); + state.pc = (unsigned long)__builtin_return_address(0); + + if (unwind_next_frame_record(&state)) + return; + while (1) { + /* We only use the fp in the exception callback. Pass 0 for sp as it's unavailable*/ + if (!consume_fn(cookie, (u64)state.pc, 0, (u64)state.fp)) + break; + if (unwind_next_frame_record(&state)) + break; + } +} diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index f5065576cae9..7f768d335698 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -1,6 +1,5 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 -exceptions # JIT does not support calling kfunc bpf_throw: -524 fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95 kprobe_multi_test/attach_api_addrs # bpf_program__attach_kprobe_multi_opts unexpected error: -95