From patchwork Tue Aug 13 18:49:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13762410 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-176.mta1.migadu.com (out-176.mta1.migadu.com [95.215.58.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A90F81A3BA8 for ; Tue, 13 Aug 2024 18:49:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575001; cv=none; b=gx76QmePtIbGd4pY85upO1XpPjAuMAAEpxlqFirshke+OgcYghqHoYug6jiRS5uWdBLqL5gMzx419cW1sEED8tZ2EQiZmoMc1hPo2/ujb2C9OZHtQEsTz/hwJcCYgGhP/51f4mVqSD7Iw2PtOmQAAsBkRMqCKfXNP6AwZNPL23s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575001; c=relaxed/simple; bh=TRwFfv4Z5pQwu2YpqBlGrfL0WfNS1evcbsUWQaiVUww=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JK2uZkoLgi3hjam6VIBybDBHOqdhcZDv0vGWK/7Z9Zk5esuoQu5dj4PWshY2kNgyWE4LcIal77kmQYMWTK8mTpTRzw8GkLFsWNrI+N7T44O93NMLL6DnQwYhmAb0Bz1J5QUMoASy75J9Ytui0fRsrLBEessqQz699XUkAPV+PFI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=JnTY5/Gv; arc=none smtp.client-ip=95.215.58.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="JnTY5/Gv" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1723574996; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RiSiHRm8uHhYxBfMcZIq670oTpeg0ezmWWluCd/4MuU=; b=JnTY5/Gv9kf5VhoxXX45L8hinRpzqM6G6TIFCLWmzsJu+iZ73mjQpRkTcYDtmQKQ8ztbbw fIlVG+GQswNgKRBLco6BPzWd9pF/+PK+Ru6Z+tbViZxPllmZw/Vs2XUMAboJtRpVlJpFO9 GB3e2w8ixIed+1kz25a0JqdpHIfsz5c= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [RFC PATCH bpf-next 1/6] bpf: Add gen_epilogue to bpf_verifier_ops Date: Tue, 13 Aug 2024 11:49:34 -0700 Message-ID: <20240813184943.3759630-2-martin.lau@linux.dev> In-Reply-To: <20240813184943.3759630-1-martin.lau@linux.dev> References: <20240813184943.3759630-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Martin KaFai Lau This patch adds a .gen_epilogue to the bpf_verifier_ops. It is similar to the existing .gen_prologue. Instead of allowing a subsystem to run code at the beginning of a bpf prog, it allows the subsystem to run code just before the bpf prog exit. One of the use case is to allow the upcoming bpf qdisc to ensure that the skb->dev is the same as the qdisc->dev_queue->dev. The bpf qdisc struct_ops implementation could either fix it up or drop the skb. Another use case could be in bpf_tcp_ca.c to enforce snd_cwnd has sane value (e.g. non zero). The epilogue can do the useful thing (like checking skb->dev) if it can access the bpf prog's ctx. Unlike prologue, r1 may not hold the ctx pointer. This patch saves the r1 in the stack if the .gen_epilogue has returned some instructions in the "epilogue_buf". The existing .gen_prologue is done in convert_ctx_accesses(). The new .gen_epilogue is done in the convert_ctx_accesses() also. When it sees the (BPF_JMP | BPF_EXIT) instruction, it will be patched with the earlier generated "epilogue_buf". The epilogue patching is only done for the main prog. Signed-off-by: Martin KaFai Lau Reviewed-by: Eduard Zingerman --- The .gen_epilogue will need 8 extra bytes in the stack to save the ctx pointer. It is now done after the check_max_stack_depth. The ctx pointer saving will need to be done earlier before check_max_stack_depth. include/linux/bpf.h | 2 ++ kernel/bpf/verifier.c | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b9425e410bcb..2de67bc497f4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -974,6 +974,8 @@ struct bpf_verifier_ops { struct bpf_insn_access_aux *info); int (*gen_prologue)(struct bpf_insn *insn, bool direct_write, const struct bpf_prog *prog); + int (*gen_epilogue)(struct bpf_insn *insn, const struct bpf_prog *prog, + s16 ctx_stack_off); int (*gen_ld_abs)(const struct bpf_insn *orig, struct bpf_insn *insn_buf); u32 (*convert_ctx_access)(enum bpf_access_type type, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index df3be12096cf..bbb655f0c7b5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19610,15 +19610,37 @@ static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, */ static int convert_ctx_accesses(struct bpf_verifier_env *env) { + struct bpf_subprog_info *subprogs = env->subprog_info; const struct bpf_verifier_ops *ops = env->ops; - int i, cnt, size, ctx_field_size, delta = 0; + int i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0; const int insn_cnt = env->prog->len; - struct bpf_insn insn_buf[16], *insn; + struct bpf_insn insn_buf[16], epilogue_buf[16], *insn; u32 target_size, size_default, off; struct bpf_prog *new_prog; enum bpf_access_type type; bool is_narrower_load; + if (ops->gen_epilogue) { + epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, + -(subprogs[0].stack_depth + 8)); + if (epilogue_cnt >= ARRAY_SIZE(epilogue_buf)) { + verbose(env, "bpf verifier is misconfigured\n"); + return -EINVAL; + } else if (epilogue_cnt) { + /* Save the ARG_PTR_TO_CTX for the epilogue to use */ + cnt = 0; + subprogs[0].stack_depth += 8; + insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_1, + -subprogs[0].stack_depth); + insn_buf[cnt++] = env->prog->insnsi[0]; + new_prog = bpf_patch_insn_data(env, 0, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + env->prog = new_prog; + delta += cnt - 1; + } + } + if (ops->gen_prologue || env->seen_direct_write) { if (!ops->gen_prologue) { verbose(env, "bpf verifier is misconfigured\n"); @@ -19671,6 +19693,13 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code = BPF_STX | BPF_PROBE_ATOMIC | BPF_SIZE(insn->code); env->prog->aux->num_exentries++; continue; + } else if (insn->code == (BPF_JMP | BPF_EXIT) && + epilogue_cnt && + i + delta < subprogs[1].start) { + /* Generate epilogue for the main prog */ + memcpy(insn_buf, epilogue_buf, sizeof(epilogue_buf)); + cnt = epilogue_cnt; + goto patch_insn_buf; } else { continue; } @@ -19807,6 +19836,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->dst_reg, insn->dst_reg, size * 8, 0); +patch_insn_buf: new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); if (!new_prog) return -ENOMEM; From patchwork Tue Aug 13 18:49:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13762411 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-172.mta1.migadu.com (out-172.mta1.migadu.com [95.215.58.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 200D81A4F02 for ; Tue, 13 Aug 2024 18:50:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575002; cv=none; b=kAgv2Un9vD74MrxNRcgpxrlTKEaeVGSseVceFjdGSrfwt519K7f3t/T6ACiJNpmFx2C17W+TuKr7xpiVTPi2mU1LCqXvFaWGXGJKcE5aKka9s+8TUcLg8vQo+Jb/KB6bxdL25hrJsNK2Bgv+w7/csdVXudpsfXDMXPLDs/MLg38= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575002; c=relaxed/simple; bh=Tp5JrxpYW8SID0V4QRxd2sfc0j9BTY5Hf1PEptZhhtE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gUcEnEIaRLFurVArZsBpLxfj9RNJWKNdo1PrR4J7CSD+JR6MFHt8mFnapT2zaJnZWePmp8JM5D+qX5+NkeL9rwvHfv+vNB1Dxlr85Oo7Mc5tDjh7rFzuUW1TqPNxYUEAb+J0SwqpJDK0TGSkNUmDDsQL9SJo/LNY5F4KlCz6sBk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=JWNP/AhQ; arc=none smtp.client-ip=95.215.58.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="JWNP/AhQ" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1723574999; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FIWJaA/rr7z3c0SNNY8RBM7QoiU+VLBlKjZVwreXHcc=; b=JWNP/AhQWB6T3SbpgAKKRG+G+rpueqKJu655+7r7Ya+ghnn7Znybe6xteih6h9mgLDhmnM zh8+pKaeyW7sqgKDx7wi3nag10zgbcmJFkj2yfURLpbyKog07CeBU5jI3aEdNFZFGz92M0 K/lSrrgVPXvPSDln4/CpVE96lqys+Kc= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [RFC PATCH bpf-next 2/6] bpf: Export bpf_base_func_proto Date: Tue, 13 Aug 2024 11:49:35 -0700 Message-ID: <20240813184943.3759630-3-martin.lau@linux.dev> In-Reply-To: <20240813184943.3759630-1-martin.lau@linux.dev> References: <20240813184943.3759630-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Martin KaFai Lau The bpf_testmod needs to use the bpf_tail_call helper in the next selftest patch. This patch is to EXPORT_GPL_SYMBOL the bpf_base_func_proto. Signed-off-by: Martin KaFai Lau --- kernel/bpf/helpers.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index d02ae323996b..544f7d171fe6 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2034,6 +2034,7 @@ bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return NULL; } } +EXPORT_SYMBOL_GPL(bpf_base_func_proto); void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock) From patchwork Tue Aug 13 18:49:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13762412 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-180.mta1.migadu.com (out-180.mta1.migadu.com [95.215.58.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 279DA1BF53 for ; Tue, 13 Aug 2024 18:50:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575008; cv=none; b=BWeo+A0fwgiYTODzGZCgQBAixr/onow7QvxH5aDx/o8uGVoI4t+118w0OYguP3XuIscBKedc4PD31JbxfezTC+YyVJpTP8boKwMxSoggkMjD7Ex+AEt8YacvaijUVzX46FtjoQFmeH3LGICHRoj0X+68tV+Ojop5Xer/FKEeeZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575008; c=relaxed/simple; bh=1ffyzeJazkYS5fvLnRB+//9PgQ7N9W7dyKNZOKM0nfU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fDSF0h3Os2TkO/pIbg0bAVKy1JbuUgezJZvs7wUFE2gkVxjmrla0nYFw19JsW/U7yLEFmkDk+K8cUL5l0TPTEffCkKGeGUAWr84lQnTX8HmHXHlR2D++cp0+veD/uVmQpysaXug0NatRfri8nUY6jEZa8WoFqY09PAjktabnrSs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=emiCaCqX; arc=none smtp.client-ip=95.215.58.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="emiCaCqX" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1723575002; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4vp0gvY9jG15PsEIVLmXcrBiefGfJIALcwVt/mFFKD8=; b=emiCaCqX1WfGEcOYGkDbxnhlHdc9UpgNDAv+0EZMfTQ4C12Z02xpxUdnVBq546blXU+Ulx gKb7bHd46cwO4MzNm9SJIUoTllmrOSR5MfA7OxBH1tUJzL5hyfn62hLpqWxakVEJ9TllvJ pEuRuJto5raBcvVjUeAeK+DQiHvtKq0= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [RFC PATCH bpf-next 3/6] selftests/test: test gen_prologue and gen_epilogue Date: Tue, 13 Aug 2024 11:49:36 -0700 Message-ID: <20240813184943.3759630-4-martin.lau@linux.dev> In-Reply-To: <20240813184943.3759630-1-martin.lau@linux.dev> References: <20240813184943.3759630-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Martin KaFai Lau This test adds a new struct_ops "bpf_testmod_st_ops" in bpf_testmod. The ops of the bpf_testmod_st_ops is triggered by new kfunc calls "bpf_kfunc_st_ops_test_*logue". These new kfunc calls are primarily used by the SEC("syscall") program. The test triggering sequence is like: SEC("syscall") syscall_prologue(struct st_ops_args *args) bpf_kfunc_st_op_test_prologue(args) st_ops->test_prologue(args) The .gen_prologue and .gen_epilogue of the bpf_testmod_st_ops will add PROLOGUE_A (1000) and EPILOGUE_A (10000) to args->a. .gen_prologue adds PROLOGUE_A (1000). .gen_epilogue adds EPILOGUE_A (10000). .gen_epilogue will also set the r0 to 2 * args->a. The .gen_prologue and .gen_epilogue of the bpf_testmod_st_ops will test the prog->aux->attach_func_name to decide if it needs to generate codes. The main programs of the SEC("struct_ops/..") will either call a global subprog() which does "args->a += 1" or call another new kfunc bpf_kfunc_st_ops_inc10 which does "args->a += 10". The prog_tests/struct_ops_syscall.c will test_run the SEC("syscall") programs. It checks the result by testing the args->a and the retval. For example, when triggering the ops in the 'SEC("struct_ops/test_epilogue") int BPF_PROG(test_epilogue_subprog..' the expected args->a is +1 (because of the subprog calls) + 10000 (.gen_epilogue) = 10001 The expected return value is 2 * 10001 (.gen_epilogue). Another set of tests is to have the main struct_ops program to call a subprog and call the inc10 kfunc. There is also a bpf_tail_call test for epilogue. Signed-off-by: Martin KaFai Lau --- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 190 ++++++++++++++++++ .../selftests/bpf/bpf_testmod/bpf_testmod.h | 11 + .../bpf/bpf_testmod/bpf_testmod_kfunc.h | 6 + .../bpf/prog_tests/struct_ops_syscall.c | 91 +++++++++ .../selftests/bpf/progs/struct_ops_syscall.c | 113 +++++++++++ 5 files changed, 411 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_syscall.c diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 3687a40b61c6..7194330bdefc 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include "bpf_testmod.h" @@ -920,6 +921,51 @@ __bpf_kfunc int bpf_kfunc_call_kernel_getpeername(struct addr_args *args) return err; } +static DEFINE_MUTEX(st_ops_mutex); +static struct bpf_testmod_st_ops *st_ops; + +__bpf_kfunc int bpf_kfunc_st_ops_test_prologue(struct st_ops_args *args) +{ + int ret = -1; + + mutex_lock(&st_ops_mutex); + if (st_ops && st_ops->test_prologue) + ret = st_ops->test_prologue(args); + mutex_unlock(&st_ops_mutex); + + return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_test_epilogue(struct st_ops_args *args) +{ + int ret = -1; + + mutex_lock(&st_ops_mutex); + if (st_ops && st_ops->test_epilogue) + ret = st_ops->test_epilogue(args); + mutex_unlock(&st_ops_mutex); + + return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_test_pro_epilogue(struct st_ops_args *args) +{ + int ret = -1; + + mutex_lock(&st_ops_mutex); + if (st_ops && st_ops->test_pro_epilogue) + ret = st_ops->test_pro_epilogue(args); + mutex_unlock(&st_ops_mutex); + + return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) +{ + args->a += 10; + return args->a; +} + BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) BTF_ID_FLAGS(func, bpf_kfunc_call_test1) @@ -956,6 +1002,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_sendmsg, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_sock_sendmsg, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_getsockname, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_getpeername, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_prologue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_epilogue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_pro_epilogue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_inc10, KF_TRUSTED_ARGS) BTF_KFUNCS_END(bpf_testmod_check_kfunc_ids) static int bpf_testmod_ops_init(struct btf *btf) @@ -1075,6 +1125,144 @@ struct bpf_struct_ops bpf_testmod_ops2 = { .owner = THIS_MODULE, }; +static int bpf_test_mod_st_ops__test_prologue(struct st_ops_args *args) +{ + return 0; +} + +static int bpf_test_mod_st_ops__test_epilogue(struct st_ops_args *args) +{ + return 0; +} + +static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args) +{ + return 0; +} + +static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, + const struct bpf_prog *prog) +{ + struct bpf_insn *insn = insn_buf; + + if (strcmp(prog->aux->attach_func_name, "test_prologue") && + strcmp(prog->aux->attach_func_name, "test_pro_epilogue")) + return 0; + + /* r6 = r1[0]; // r6 will be "struct st_ops *args". r1 is "u64 *ctx". + * r7 = r6->a; + * r7 += 1000; + * r6->a = r7; + */ + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0); + *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_6, offsetof(struct st_ops_args, a)); + *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_7, 1000); + *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_6, BPF_REG_7, offsetof(struct st_ops_args, a)); + *insn++ = prog->insnsi[0]; + + return insn - insn_buf; +} + +static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog, + s16 ctx_stack_off) +{ + struct bpf_insn *insn = insn_buf; + + if (strcmp(prog->aux->attach_func_name, "test_epilogue") && + strcmp(prog->aux->attach_func_name, "test_pro_epilogue")) + return 0; + + /* r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx" + * r1 = r1[0]; // r1 will be "struct st_ops *args" + * r6 = r1->a; + * r6 += 10000; + * r1->a = r6; + * r0 = r6; + * r0 *= 2; + * BPF_EXIT; + */ + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_FP, ctx_stack_off); + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0); + *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, offsetof(struct st_ops_args, a)); + *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_6, 10000); + *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_6, offsetof(struct st_ops_args, a)); + *insn++ = BPF_MOV32_REG(BPF_REG_0, BPF_REG_6); + *insn++ = BPF_ALU32_IMM(BPF_MUL, BPF_REG_0, 2); + *insn++ = BPF_EXIT_INSN(); + + return insn - insn_buf; +} + +static int st_ops_btf_struct_access(struct bpf_verifier_log *log, + const struct bpf_reg_state *reg, + int off, int size) +{ + if (off < 0 || off + size > sizeof(struct st_ops_args)) + return -EACCES; + return 0; +} + +static const struct bpf_verifier_ops st_ops_verifier_ops = { + .is_valid_access = bpf_testmod_ops_is_valid_access, + .btf_struct_access = st_ops_btf_struct_access, + .gen_prologue = st_ops_gen_prologue, + .gen_epilogue = st_ops_gen_epilogue, + .get_func_proto = bpf_base_func_proto, +}; + +static struct bpf_testmod_st_ops st_ops_cfi_stubs = { + .test_prologue = bpf_test_mod_st_ops__test_prologue, + .test_epilogue = bpf_test_mod_st_ops__test_epilogue, + .test_pro_epilogue = bpf_test_mod_st_ops__test_pro_epilogue, +}; + +static int st_ops_reg(void *kdata, struct bpf_link *link) +{ + int err = 0; + + mutex_lock(&st_ops_mutex); + if (st_ops) { + pr_err("st_ops has already been registered\n"); + err = -EEXIST; + goto unlock; + } + st_ops = kdata; + +unlock: + mutex_unlock(&st_ops_mutex); + return err; +} + +static void st_ops_unreg(void *kdata, struct bpf_link *link) +{ + mutex_lock(&st_ops_mutex); + st_ops = NULL; + mutex_unlock(&st_ops_mutex); +} + +static int st_ops_init(struct btf *btf) +{ + return 0; +} + +static int st_ops_init_member(const struct btf_type *t, + const struct btf_member *member, + void *kdata, const void *udata) +{ + return 0; +} + +static struct bpf_struct_ops testmod_st_ops = { + .verifier_ops = &st_ops_verifier_ops, + .init = st_ops_init, + .init_member = st_ops_init_member, + .reg = st_ops_reg, + .unreg = st_ops_unreg, + .cfi_stubs = &st_ops_cfi_stubs, + .name = "bpf_testmod_st_ops", + .owner = THIS_MODULE, +}; + extern int bpf_fentry_test1(int a); static int bpf_testmod_init(void) @@ -1092,8 +1280,10 @@ static int bpf_testmod_init(void) ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_testmod_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_testmod_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_testmod_kfunc_set); ret = ret ?: register_bpf_struct_ops(&bpf_bpf_testmod_ops, bpf_testmod_ops); ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops2, bpf_testmod_ops2); + ret = ret ?: register_bpf_struct_ops(&testmod_st_ops, bpf_testmod_st_ops); ret = ret ?: register_btf_id_dtor_kfuncs(bpf_testmod_dtors, ARRAY_SIZE(bpf_testmod_dtors), THIS_MODULE); diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h index fe0d402b0d65..3241a9d796ed 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h @@ -94,4 +94,15 @@ struct bpf_testmod_ops2 { int (*test_1)(void); }; +struct st_ops_args { + int a; +}; + +struct bpf_testmod_st_ops { + int (*test_prologue)(struct st_ops_args *args); + int (*test_epilogue)(struct st_ops_args *args); + int (*test_pro_epilogue)(struct st_ops_args *args); + struct module *owner; +}; + #endif /* _BPF_TESTMOD_H */ diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h index e587a79f2239..0df429a0edaa 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h @@ -144,4 +144,10 @@ void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr, struct bpf_dynptr *ptr__nulla struct bpf_testmod_ctx *bpf_testmod_ctx_create(int *err) __ksym; void bpf_testmod_ctx_release(struct bpf_testmod_ctx *ctx) __ksym; +struct st_ops_args; +int bpf_kfunc_st_ops_test_prologue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_test_epilogue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_test_pro_epilogue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) __ksym; + #endif /* _BPF_TESTMOD_KFUNC_H */ diff --git a/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c b/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c new file mode 100644 index 000000000000..a293a35b0dcc --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include "struct_ops_syscall.skel.h" + +#define EPILOGUE_A 10000 +#define PROLOGUE_A 1000 +#define KFUNC_A10 10 +#define SUBPROG_A 1 + +#define SUBPROG_TEST_MAIN SUBPROG_A +#define KFUNC_TEST_MAIN (KFUNC_A10 + SUBPROG_A) + +struct st_ops_args { + int a; +}; + +static void do_test(struct struct_ops_syscall *skel, + struct bpf_map *st_ops_map, int main_prog_a) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd, expected_a; + struct st_ops_args args; + struct bpf_link *link; + + topts.ctx_in = &args; + topts.ctx_size_in = sizeof(args); + + link = bpf_map__attach_struct_ops(st_ops_map); + if (!ASSERT_OK_PTR(link, "attach_struct_ops")) + return; + + /* gen_prologue + main prog */ + expected_a = PROLOGUE_A + main_prog_a; + memset(&args, 0, sizeof(args)); + prog_fd = bpf_program__fd(skel->progs.syscall_prologue); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(args.a, expected_a, "args.a"); + ASSERT_EQ(topts.retval, 0, "topts.retval"); + + /* main prog + gen_epilogue */ + expected_a = main_prog_a + EPILOGUE_A; + memset(&args, 0, sizeof(args)); + prog_fd = bpf_program__fd(skel->progs.syscall_epilogue); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(args.a, expected_a, "args.a"); + ASSERT_EQ(topts.retval, expected_a * 2, "topts.retval"); + + /* gen_prologue + main prog + gen_epilogue */ + expected_a = PROLOGUE_A + main_prog_a + EPILOGUE_A; + memset(&args, 0, sizeof(args)); + prog_fd = bpf_program__fd(skel->progs.syscall_pro_epilogue); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(args.a, expected_a, "args.a"); + ASSERT_EQ(topts.retval, expected_a * 2, "topts.retval"); + bpf_link__destroy(link); +} + +void test_struct_ops_syscall(void) +{ + struct struct_ops_syscall *skel; + + skel = struct_ops_syscall__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_and_load")) + return; + + if (test__start_subtest("subprog")) + do_test(skel, skel->maps.pro_epilogue_subprog_ops, + SUBPROG_TEST_MAIN); + + if (test__start_subtest("kfunc")) + do_test(skel, skel->maps.pro_epilogue_kfunc_ops, + KFUNC_TEST_MAIN); + + if (test__start_subtest("tailcall")) { + const int zero = 0; + int prog_fd = bpf_program__fd(skel->progs.test_epilogue_subprog); + int map_fd = bpf_map__fd(skel->maps.epilogue_map); + int err; + + err = bpf_map_update_elem(map_fd, &zero, &prog_fd, 0); + if (ASSERT_OK(err, "map_update(epilogue_map)")) + do_test(skel, skel->maps.pro_epilogue_tail_ops, + SUBPROG_TEST_MAIN); + } + + struct_ops_syscall__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/struct_ops_syscall.c b/tools/testing/selftests/bpf/progs/struct_ops_syscall.c new file mode 100644 index 000000000000..ee153461d9f8 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_syscall.c @@ -0,0 +1,113 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 1); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); +} epilogue_map SEC(".maps"); + +static __noinline int subprog(struct st_ops_args *args) +{ + args->a += 1; + return 0; +} + +SEC("struct_ops/test_prologue_subprog") +int BPF_PROG(test_prologue_subprog, struct st_ops_args *args) +{ + subprog(args); + return 0; +} + +SEC("struct_ops/test_epilogue_subprog") +int BPF_PROG(test_epilogue_subprog, struct st_ops_args *args) +{ + subprog(args); + return 0; +} + +SEC("struct_ops/test_pro_epilogue_subprog") +int BPF_PROG(test_pro_epilogue_subprog, struct st_ops_args *args) +{ + subprog(args); + return 0; +} + +SEC("struct_ops/test_prologue_kfunc") +int BPF_PROG(test_prologue_kfunc, struct st_ops_args *args) +{ + bpf_kfunc_st_ops_inc10(args); + subprog(args); + return 0; +} + +SEC("struct_ops/test_epilogue_kfunc") +int BPF_PROG(test_epilogue_kfunc, struct st_ops_args *args) +{ + bpf_kfunc_st_ops_inc10(args); + subprog(args); + return 0; +} + +SEC("struct_ops/test_pro_epilogue_kfunc") +int BPF_PROG(test_pro_epilogue_kfunc, struct st_ops_args *args) +{ + bpf_kfunc_st_ops_inc10(args); + subprog(args); + return 0; +} + +SEC("struct_ops/test_epilogue_tail") +int test_epilogue_tail(unsigned long long *ctx) +{ + bpf_tail_call_static(ctx, &epilogue_map, 0); + return 0; +} + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops pro_epilogue_subprog_ops = { + .test_prologue = (void *)test_prologue_subprog, + .test_epilogue = (void *)test_epilogue_subprog, + .test_pro_epilogue = (void *)test_pro_epilogue_subprog, +}; + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops pro_epilogue_kfunc_ops = { + .test_prologue = (void *)test_prologue_kfunc, + .test_epilogue = (void *)test_epilogue_kfunc, + .test_pro_epilogue = (void *)test_pro_epilogue_kfunc, +}; + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops pro_epilogue_tail_ops = { + .test_prologue = (void *)test_prologue_subprog, + .test_epilogue = (void *)test_epilogue_tail, + .test_pro_epilogue = (void *)test_pro_epilogue_subprog, +}; + +SEC("syscall") +int syscall_prologue(struct st_ops_args *args) +{ + return bpf_kfunc_st_ops_test_prologue(args); +} + +SEC("syscall") +int syscall_epilogue(struct st_ops_args *args) +{ + return bpf_kfunc_st_ops_test_epilogue(args); +} + +SEC("syscall") +int syscall_pro_epilogue(struct st_ops_args *args) +{ + return bpf_kfunc_st_ops_test_pro_epilogue(args); +} From patchwork Tue Aug 13 18:49:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13762413 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-180.mta1.migadu.com (out-180.mta1.migadu.com [95.215.58.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4A6A1A38F3 for ; Tue, 13 Aug 2024 18:50:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575009; cv=none; b=tYtswZSHPbRnfyiGf0cJWxp+F6IhwD4vGRUpRJC+CaLv/R4uGHs8R5puK0sEqXBBy9Xv0NYGKB0hhKW1m+yXRBydnv9yIL5Yk6Wh+8/xYS+kRDoWKnmXYj620hnx5btj2rOQ8/Dy54uBAur3HP8f6UlgKJWMl/etFKA9C/6B8Co= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575009; c=relaxed/simple; bh=uz8So6SUo0sr548lwQw+WAuLUgYUjcnUBFVsq33o9Do=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M8307o+3Lyxua9ydbavLF+OOBQwM75kqtOLWnfmAz0+ZDrR83yiGYtxUSR9/Bbod60XoVi68Ngd5QNq1TT3BePXK6CJdBayNiCVQASOqhvdF/9Z2kjTaXIlSe0j1I6kxJ6kRZeT81Hf/nI2tU4KrmU11nbbCyi6URV+ub8fc1C8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=SA4zHMBU; arc=none smtp.client-ip=95.215.58.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="SA4zHMBU" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1723575005; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BrNjme+CrgULDqDJJQjj2SGh+l76UDKL/LsE2MqOgcc=; b=SA4zHMBUdFeMhKJRIHKNS9gbuSkMluew2RFTj9zi5q7CTG9/QJRsbl8Egc9GxrUo86xemx YABQaTnYCjV2FxfroXx98Yuzva0yQweLw0ALO2B8BfPB+XPLIpXKHZU49yOuGlGWJuISCz YnEwB82KS85kNoQBtlSEVd4hnTPtHKs= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [RFC PATCH bpf-next 4/6] bpf: Add module parameter to gen_prologue and gen_epilogue Date: Tue, 13 Aug 2024 11:49:37 -0700 Message-ID: <20240813184943.3759630-5-martin.lau@linux.dev> In-Reply-To: <20240813184943.3759630-1-martin.lau@linux.dev> References: <20240813184943.3759630-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Martin KaFai Lau This patch adds a "struct module **module" arg to the .gen_prologue and .gen_epilogue. This will allow the .gen_pro/epilogue to make kfunc call because the verifer needs to know the kfunc's BTF. The next patch will figure the kfunc's BTF from the module. It also exposes the "btf_get_module_btf" function to help figuring out the btf from a module. Signed-off-by: Martin KaFai Lau --- include/linux/bpf.h | 4 ++-- include/linux/btf.h | 1 + kernel/bpf/btf.c | 2 +- kernel/bpf/cgroup.c | 3 ++- kernel/bpf/verifier.c | 4 ++-- net/core/filter.c | 6 +++--- tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 4 ++-- 7 files changed, 13 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2de67bc497f4..9787532813e2 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -973,9 +973,9 @@ struct bpf_verifier_ops { const struct bpf_prog *prog, struct bpf_insn_access_aux *info); int (*gen_prologue)(struct bpf_insn *insn, bool direct_write, - const struct bpf_prog *prog); + const struct bpf_prog *prog, struct module **module); int (*gen_epilogue)(struct bpf_insn *insn, const struct bpf_prog *prog, - s16 ctx_stack_off); + s16 ctx_stack_off, struct module **module); int (*gen_ld_abs)(const struct bpf_insn *orig, struct bpf_insn *insn_buf); u32 (*convert_ctx_access)(enum bpf_access_type type, diff --git a/include/linux/btf.h b/include/linux/btf.h index cffb43133c68..177187fa3819 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -580,6 +580,7 @@ bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf, int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_type); bool btf_types_are_same(const struct btf *btf1, u32 id1, const struct btf *btf2, u32 id2); +struct btf *btf_get_module_btf(const struct module *module); #else static inline const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 95426d5b634e..b230f7ff9388 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7970,7 +7970,7 @@ struct module *btf_try_get_module(const struct btf *btf) /* Returns struct btf corresponding to the struct module. * This function can return NULL or ERR_PTR. */ -static struct btf *btf_get_module_btf(const struct module *module) +struct btf *btf_get_module_btf(const struct module *module) { #ifdef CONFIG_DEBUG_INFO_BTF_MODULES struct btf_module *btf_mod, *tmp; diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index 8ba73042a239..0be053d86b56 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -2503,7 +2503,8 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, static int cg_sockopt_get_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, + struct module **module) { /* Nothing to do for sockopt argument. The data is kzalloc'ated. */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bbb655f0c7b5..5e995b7884fb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19622,7 +19622,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) if (ops->gen_epilogue) { epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, - -(subprogs[0].stack_depth + 8)); + -(subprogs[0].stack_depth + 8), NULL); if (epilogue_cnt >= ARRAY_SIZE(epilogue_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; @@ -19647,7 +19647,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) return -EINVAL; } cnt = ops->gen_prologue(insn_buf, env->seen_direct_write, - env->prog); + env->prog, NULL); if (cnt >= ARRAY_SIZE(insn_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; diff --git a/net/core/filter.c b/net/core/filter.c index 78a6f746ea0b..65d219e71ae8 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -8844,7 +8844,7 @@ static bool sock_filter_is_valid_access(int off, int size, } static int bpf_noop_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { /* Neither direct read nor direct write requires any preliminary * action. @@ -8927,7 +8927,7 @@ static int bpf_gen_ld_abs(const struct bpf_insn *orig, } static int tc_cls_act_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { return bpf_unclone_prologue(insn_buf, direct_write, prog, TC_ACT_SHOT); } @@ -9263,7 +9263,7 @@ static bool sock_ops_is_valid_access(int off, int size, } static int sk_skb_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { return bpf_unclone_prologue(insn_buf, direct_write, prog, SK_DROP); } diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 7194330bdefc..4c75346376d9 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -1141,7 +1141,7 @@ static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args) } static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { struct bpf_insn *insn = insn_buf; @@ -1164,7 +1164,7 @@ static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, } static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog, - s16 ctx_stack_off) + s16 ctx_stack_off, struct module **module) { struct bpf_insn *insn = insn_buf; From patchwork Tue Aug 13 18:49:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13762414 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-176.mta1.migadu.com (out-176.mta1.migadu.com [95.215.58.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C12F1A4F21 for ; Tue, 13 Aug 2024 18:50:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575011; cv=none; b=ca7/Q2KQQC5ZsS4YQ8EPyOvdVMESOsF6P8XC//TUvPGInCOiQoacKile0VYIsG3imKEnELNUCT5NNXhppWIKqib1ZMNuyMCbWin8GWdBmITNHs9FpKAy+kdEGJycimJPhNdHBtcIkWV9ZdYvGYG1oXEhDjFkstrOdlRByIZa3x4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575011; c=relaxed/simple; bh=Pi7cc9H2itSfv9ypiwW8gfxganbHlCd1orYSkTGO/9g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cFd5a1xUlojbhC3eg8VolIJ99/UnfjYCgX/6BNkYC7Nz+Slo5RUb+Sgs50BToDP3V3THBPBGTQ9hcDLOFJeLdNtBDB33ltQHvCjQHzqy0JLjhL7iKUUqzgdxqWj7IxwAI6N56UZLqgH4gGCxOmeSTYxJZGALLwFlbtNsI2ZF534= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=V5JJR0NX; arc=none smtp.client-ip=95.215.58.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="V5JJR0NX" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1723575007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WlN6QAkzxfASZkCrfllb9U4enBjH50DEdAQCgdgg4TA=; b=V5JJR0NXxle8F+DsuxEwdcR+rJmkFsTcEZ3PzTihGVnJM5XciePGYSx6gwQDwU/NHMSLCY Nm7EIlx82Ye0JkDXLyFbkoCNd55YF4u6VHcpNkcnAlYHPhqrrEJe62cPRQO4G5MP6WtQ7X aCk/xSu+jU13p+p2QHOskvG+W3NdvyM= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [RFC PATCH bpf-next 5/6] bpf: Allow pro/epilogue to call kfunc Date: Tue, 13 Aug 2024 11:49:38 -0700 Message-ID: <20240813184943.3759630-6-martin.lau@linux.dev> In-Reply-To: <20240813184943.3759630-1-martin.lau@linux.dev> References: <20240813184943.3759630-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Martin KaFai Lau The existing prologue has been able to call bpf helper but not a kfunc. This patch allows the prologue/epilogue to call the kfunc. The subsystem that implements the .gen_prologue and .gen_epilogue can add the BPF_PSEUDO_KFUNC_CALL instruction with insn->imm set to the btf func_id of the kfunc call. This part is the same as the bpf prog loaded from the sys_bpf. Another piece is to have a way for the subsystem to tell the btf object of the kfunc func_id. This patch uses the "struct module **module" argument added to the .gen_prologue and .gen_epilogue in the previous patch. The verifier will use btf_get_module_btf(module) to find out the btf object. The .gen_epi/prologue will usually use THIS_MODULE to initialize the "*module = THIS_MODULE". Only kfunc(s) from one module (or vmlinux) can be used in the .gen_epi/prologue now. In the future, the .gen_epi/prologue can return an array of modules and use the insn->off as an index into the array. When the returned module is NULL, the btf is btf_vmlinux. Then the insn->off stays at 0. This is the same as the sys_bpf. When the btf is from a module, the btf needs an entry in prog->aux->kfunc_btf_tab. The kfunc_btf_tab is currently sorted by insn->off which is the offset to the attr->fd_array. This module btf may or may not be in the kfunc_btf_tab. A new function "find_kfunc_desc_btf_offset" is added to search for the existing entry that has the same btf. If it is found, its offset will be used in the insn->off. If it is not found, it will find an offset value that is not used in the kfunc_btf_tab. Add a new entry to kfunc_btf_tab and set this new offset to the insn->off Once the insn->off is determined (either reuse an existing one or an unused one is found), it will call the existing add_kfunc_call() and everything else should fall through. Signed-off-by: Martin KaFai Lau Reviewed-by: Eduard Zingerman --- kernel/bpf/verifier.c | 116 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 113 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5e995b7884fb..2873e1083402 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2787,6 +2787,61 @@ static struct btf *find_kfunc_desc_btf(struct bpf_verifier_env *env, s16 offset) return btf_vmlinux ?: ERR_PTR(-ENOENT); } +static int find_kfunc_desc_btf_offset(struct bpf_verifier_env *env, struct btf *btf, + struct module *module, s16 *offset) +{ + struct bpf_kfunc_btf_tab *tab; + struct bpf_kfunc_btf *b; + s16 new_offset = S16_MAX; + u32 i; + + if (btf_is_vmlinux(btf)) { + *offset = 0; + return 0; + } + + tab = env->prog->aux->kfunc_btf_tab; + if (!tab) { + tab = kzalloc(sizeof(*tab), GFP_KERNEL); + if (!tab) + return -ENOMEM; + env->prog->aux->kfunc_btf_tab = tab; + } + + b = tab->descs; + for (i = tab->nr_descs; i > 0; i--) { + if (b[i - 1].btf == btf) { + *offset = b[i - 1].offset; + return 0; + } + /* Search new_offset from backward S16_MAX, S16_MAX-1, ... + * tab->nr_descs max out at MAX_KFUNC_BTFS which is + * smaller than S16_MAX, so it will be able to find + * a non-zero new_offset to use. + */ + if (new_offset == b[i - 1].offset) + new_offset--; + } + + if (tab->nr_descs == MAX_KFUNC_BTFS) { + verbose(env, "too many different module BTFs\n"); + return -E2BIG; + } + + if (!try_module_get(module)) + return -ENXIO; + + b = &tab->descs[tab->nr_descs++]; + btf_get(btf); + b->btf = btf; + b->module = module; + b->offset = new_offset; + *offset = new_offset; + sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), + kfunc_btf_cmp_by_off, NULL); + return 0; +} + static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) { const struct btf_type *func, *func_proto; @@ -19603,6 +19658,50 @@ static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, return 0; } +static int fixup_pro_epilogue_kfunc(struct bpf_verifier_env *env, struct bpf_insn *insns, + int cnt, struct module *module) +{ + struct btf *btf; + u32 func_id; + int i, err; + s16 offset; + + for (i = 0; i < cnt; i++) { + if (!bpf_pseudo_kfunc_call(&insns[i])) + continue; + + /* The kernel may not have BTF available, so only + * try to get a btf if the pro/epilogue calls a kfunc. + */ + btf = btf_get_module_btf(module); + if (IS_ERR_OR_NULL(btf)) { + verbose(env, "cannot find BTF from %s for kfunc used in pro/epilogue\n", + module_name(module)); + return -EINVAL; + } + + func_id = insns[i].imm; + if (btf_is_vmlinux(btf) && + btf_id_set_contains(&special_kfunc_set, func_id)) { + verbose(env, "pro/epilogue cannot use special kfunc\n"); + btf_put(btf); + return -EINVAL; + } + + err = find_kfunc_desc_btf_offset(env, btf, module, &offset); + btf_put(btf); + if (err) + return err; + + insns[i].off = offset; + err = add_kfunc_call(env, func_id, offset); + if (err) + return err; + } + + return 0; +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -19612,21 +19711,27 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprogs = env->subprog_info; const struct bpf_verifier_ops *ops = env->ops; - int i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0; + int err, i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0; const int insn_cnt = env->prog->len; struct bpf_insn insn_buf[16], epilogue_buf[16], *insn; u32 target_size, size_default, off; struct bpf_prog *new_prog; enum bpf_access_type type; bool is_narrower_load; + struct module *module; if (ops->gen_epilogue) { + module = NULL; epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, - -(subprogs[0].stack_depth + 8), NULL); + -(subprogs[0].stack_depth + 8), &module); if (epilogue_cnt >= ARRAY_SIZE(epilogue_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; } else if (epilogue_cnt) { + err = fixup_pro_epilogue_kfunc(env, epilogue_buf, epilogue_cnt, module); + if (err) + return err; + /* Save the ARG_PTR_TO_CTX for the epilogue to use */ cnt = 0; subprogs[0].stack_depth += 8; @@ -19646,12 +19751,17 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; } + module = NULL; cnt = ops->gen_prologue(insn_buf, env->seen_direct_write, - env->prog, NULL); + env->prog, &module); if (cnt >= ARRAY_SIZE(insn_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; } else if (cnt) { + err = fixup_pro_epilogue_kfunc(env, insn_buf, cnt, module); + if (err) + return err; + new_prog = bpf_patch_insn_data(env, 0, insn_buf, cnt); if (!new_prog) return -ENOMEM; From patchwork Tue Aug 13 18:49:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13762415 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-178.mta1.migadu.com (out-178.mta1.migadu.com [95.215.58.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7857C1A3BCF for ; Tue, 13 Aug 2024 18:50:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575014; cv=none; b=F0XdXRVnPtFMy+f8ThJ7ffwf97FBnBv5cFuuzLSdLEDBA6hZ9rWj4j525umsHjFv8ZmlSQXWnFwEvRHvFOnq12srfnxeu3y/k0V0rChZzyHLgIFGkxi8JLOxdunhVXiNMfiN0d23lioK7VBs1XS2hwW4qcP2Uhx9uSbLX9FkzWA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723575014; c=relaxed/simple; bh=2TLCZJDsZTCCYfFcRzoi406vm9XrhYvBt44yD8/JtXI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=F3eejMcIWOyO/q5mYT5aZovLbHZYWJxsihx5OBW5zE0eVrzLa7pFCYGBSsZ/uzJSnkN7jef5xzOBpGdkMn65P6epg9GXJQGSUDVfaF5UH41BpHCnufTvBpeIYfeA/c2c90LWrD9eL7u2KWPe8yQTxGibPQ5LDQcRZoNG5YO4dRM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=AqIkxXuC; arc=none smtp.client-ip=95.215.58.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="AqIkxXuC" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1723575010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2ztwuooFE0dM711Tb5cypKEnLdHL0COCVYbfberE/6Y=; b=AqIkxXuCY85/5E3II/gd+sGD4IFvN85cebL36CJqUQmPaqF6L4Hnq8jgBsnaW+kcsAhaeK AuN5o5esPMr9Y5X9fYEk3A0QrYmyIQSNGlTFsB7DBmviwo16LP3l6pIQaswM6K7X49Ef4J l4ynm8aSK8zQjpB1sffGO1nOGbH4xVQ= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [RFC PATCH bpf-next 6/6] selftests/bpf: Add kfunc call test in gen_prologue and gen_epilogue Date: Tue, 13 Aug 2024 11:49:39 -0700 Message-ID: <20240813184943.3759630-7-martin.lau@linux.dev> In-Reply-To: <20240813184943.3759630-1-martin.lau@linux.dev> References: <20240813184943.3759630-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC From: Martin KaFai Lau This patch changes the .gen_pro/epilogue of the bpf_testmod_st_ops to call kfunc. It will call the inc10 and inc100 kfunc. The value of the PROLOGUE_A and EPILOGUE_A macro are adjusted to reflect this change also. The inc100 kfunc is newly added in this patch which does args->a += 100. Note that it is not in the register_btf_kfunc_id_set(), so no need to declare in the bpf_testmod_kfunc.h. It is enclosed with __bpf_kfunc_{start,edn}_defs to avoid the compiler warning. Signed-off-by: Martin KaFai Lau --- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 42 ++++++++++++++++++- .../bpf/prog_tests/struct_ops_syscall.c | 5 ++- 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 4c75346376d9..6f745d29e124 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -966,6 +966,16 @@ __bpf_kfunc int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) return args->a; } +__bpf_kfunc_start_defs(); + +__bpf_kfunc int bpf_kfunc_st_ops_inc100(struct st_ops_args *args) +{ + args->a += 100; + return args->a; +} + +__bpf_kfunc_end_defs(); + BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) BTF_ID_FLAGS(func, bpf_kfunc_call_test1) @@ -1140,6 +1150,10 @@ static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args) return 0; } +BTF_ID_LIST(st_ops_epilogue_kfunc_list) +BTF_ID(func, bpf_kfunc_st_ops_inc10) +BTF_ID(func, bpf_kfunc_st_ops_inc100) + static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, const struct bpf_prog *prog, struct module **module) { @@ -1153,13 +1167,28 @@ static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, * r7 = r6->a; * r7 += 1000; * r6->a = r7; + * r7 = r1; + * r1 = r6; + * bpf_kfunc_st_ops_in10(r1) + * r1 = r6; + * bpf_kfunc_st_ops_in100(r1) + * r1 = r7; */ *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0); *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_6, offsetof(struct st_ops_args, a)); *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_7, 1000); *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_6, BPF_REG_7, offsetof(struct st_ops_args, a)); + *insn++ = BPF_MOV64_REG(BPF_REG_7, BPF_REG_1); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[0]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[1]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_7); *insn++ = prog->insnsi[0]; + *module = THIS_MODULE; return insn - insn_buf; } @@ -1177,7 +1206,10 @@ static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog * r6 = r1->a; * r6 += 10000; * r1->a = r6; - * r0 = r6; + * r6 = r1; + * bpf_kfunc_st_ops_in10(r1) + * r1 = r6; + * bpf_kfunc_st_ops_in100(r1) * r0 *= 2; * BPF_EXIT; */ @@ -1186,10 +1218,16 @@ static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, offsetof(struct st_ops_args, a)); *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_6, 10000); *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_6, offsetof(struct st_ops_args, a)); - *insn++ = BPF_MOV32_REG(BPF_REG_0, BPF_REG_6); + *insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[0]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[1]); *insn++ = BPF_ALU32_IMM(BPF_MUL, BPF_REG_0, 2); *insn++ = BPF_EXIT_INSN(); + *module = THIS_MODULE; return insn - insn_buf; } diff --git a/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c b/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c index a293a35b0dcc..2a73066adbf5 100644 --- a/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c +++ b/tools/testing/selftests/bpf/prog_tests/struct_ops_syscall.c @@ -3,10 +3,11 @@ #include #include "struct_ops_syscall.skel.h" -#define EPILOGUE_A 10000 -#define PROLOGUE_A 1000 +#define KFUNC_A100 100 #define KFUNC_A10 10 #define SUBPROG_A 1 +#define EPILOGUE_A (10000 + KFUNC_A100 + KFUNC_A10) +#define PROLOGUE_A (1000 + KFUNC_A100 + KFUNC_A10) #define SUBPROG_TEST_MAIN SUBPROG_A #define KFUNC_TEST_MAIN (KFUNC_A10 + SUBPROG_A)