From patchwork Wed Aug 21 23:34:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772228 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-185.mta0.migadu.com (out-185.mta0.migadu.com [91.218.175.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC10E17C7C8 for ; Wed, 21 Aug 2024 23:35:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.185 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283310; cv=none; b=N39xzhf71EeXoKcOsPnTcSwoJMBiHT72faGT3XX//wQIcq4d0lxRK05JAmpsacoPn6FHCTpMDsJAsWokQwzTHHJ6vkOFld02evt/tNqfdHR9TEZ3NpvrUulNOrXMbfe72zwQVnHGrU7swEFoJEtqoIlj4lTni6XGfCaw2fmKgUE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283310; c=relaxed/simple; bh=1tUz7zNqpa/RPRqNNxSuDRIigz0sbgLO5pE15f82oZU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c3GIQORVorD6n9JHsTAcFqY61adjG4O+8VHCe5QE+u0U3rniAQI4+KI8eGDLvFQcz5R4mGyIKW95efrx8MpWwwL9APPPQq/SABDzhFYIKVv01MJod5Ptfol7loL80pm5NGZMCXZ8Ad3niRwXCsopkpPZsgW7Dod4jWBf3UXx3Ro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=B10IUlIc; arc=none smtp.client-ip=91.218.175.185 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="B10IUlIc" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283306; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m4PlrW+FTIpAcA25s45OsEq9wnBOMT9A/V04nPt0HFk=; b=B10IUlIcxA+UFhpWfRvPj29AdDdRYjz47yZdaYS1qApffBpnn2kXo8Xm4ogjcq80chL0It BGtlbvGoeWALQc9qERuih6YFjBgdgyLwy7bAIUGWf7LPa5RpovEMYVqXWgyiMet+vTHPNS jjntkXyobniNdsE8vTel7AE4SHkULYU= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 1/8] bpf: Add gen_epilogue to bpf_verifier_ops Date: Wed, 21 Aug 2024 16:34:31 -0700 Message-ID: <20240821233440.1855263-2-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau This patch adds a .gen_epilogue to the bpf_verifier_ops. It is similar to the existing .gen_prologue. Instead of allowing a subsystem to run code at the beginning of a bpf prog, it allows the subsystem to run code just before the bpf prog exit. One of the use case is to allow the upcoming bpf qdisc to ensure that the skb->dev is the same as the qdisc->dev_queue->dev. The bpf qdisc struct_ops implementation could either fix it up or drop the skb. Another use case could be in bpf_tcp_ca.c to enforce snd_cwnd has sane value (e.g. non zero). The epilogue can do the useful thing (like checking skb->dev) if it can access the bpf prog's ctx. Unlike prologue, r1 may not hold the ctx pointer. This patch saves the r1 in the stack if the .gen_epilogue has returned some instructions in the "epilogue_buf". The existing .gen_prologue is done in convert_ctx_accesses(). The new .gen_epilogue is done in the convert_ctx_accesses() also. When it sees the (BPF_JMP | BPF_EXIT) instruction, it will be patched with the earlier generated "epilogue_buf". The epilogue patching is only done for the main prog. Reviewed-by: Eduard Zingerman Signed-off-by: Martin KaFai Lau --- include/linux/bpf.h | 2 ++ kernel/bpf/verifier.c | 34 ++++++++++++++++++++++++++++++++-- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f0192c173ed8..8ee9d87c332a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -974,6 +974,8 @@ struct bpf_verifier_ops { struct bpf_insn_access_aux *info); int (*gen_prologue)(struct bpf_insn *insn, bool direct_write, const struct bpf_prog *prog); + int (*gen_epilogue)(struct bpf_insn *insn, const struct bpf_prog *prog, + s16 ctx_stack_off); int (*gen_ld_abs)(const struct bpf_insn *orig, struct bpf_insn *insn_buf); u32 (*convert_ctx_access)(enum bpf_access_type type, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index df3be12096cf..bbb655f0c7b5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19610,15 +19610,37 @@ static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, */ static int convert_ctx_accesses(struct bpf_verifier_env *env) { + struct bpf_subprog_info *subprogs = env->subprog_info; const struct bpf_verifier_ops *ops = env->ops; - int i, cnt, size, ctx_field_size, delta = 0; + int i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0; const int insn_cnt = env->prog->len; - struct bpf_insn insn_buf[16], *insn; + struct bpf_insn insn_buf[16], epilogue_buf[16], *insn; u32 target_size, size_default, off; struct bpf_prog *new_prog; enum bpf_access_type type; bool is_narrower_load; + if (ops->gen_epilogue) { + epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, + -(subprogs[0].stack_depth + 8)); + if (epilogue_cnt >= ARRAY_SIZE(epilogue_buf)) { + verbose(env, "bpf verifier is misconfigured\n"); + return -EINVAL; + } else if (epilogue_cnt) { + /* Save the ARG_PTR_TO_CTX for the epilogue to use */ + cnt = 0; + subprogs[0].stack_depth += 8; + insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_1, + -subprogs[0].stack_depth); + insn_buf[cnt++] = env->prog->insnsi[0]; + new_prog = bpf_patch_insn_data(env, 0, insn_buf, cnt); + if (!new_prog) + return -ENOMEM; + env->prog = new_prog; + delta += cnt - 1; + } + } + if (ops->gen_prologue || env->seen_direct_write) { if (!ops->gen_prologue) { verbose(env, "bpf verifier is misconfigured\n"); @@ -19671,6 +19693,13 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code = BPF_STX | BPF_PROBE_ATOMIC | BPF_SIZE(insn->code); env->prog->aux->num_exentries++; continue; + } else if (insn->code == (BPF_JMP | BPF_EXIT) && + epilogue_cnt && + i + delta < subprogs[1].start) { + /* Generate epilogue for the main prog */ + memcpy(insn_buf, epilogue_buf, sizeof(epilogue_buf)); + cnt = epilogue_cnt; + goto patch_insn_buf; } else { continue; } @@ -19807,6 +19836,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->dst_reg, insn->dst_reg, size * 8, 0); +patch_insn_buf: new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); if (!new_prog) return -ENOMEM; From patchwork Wed Aug 21 23:34:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772229 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-188.mta0.migadu.com (out-188.mta0.migadu.com [91.218.175.188]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 973FB17D358 for ; Wed, 21 Aug 2024 23:35:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.188 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283312; cv=none; b=oQaDf7RC33Ki0UrLVCJbWca+2Cqn3QPfGqy6l5MIrxEGxVaUSSVyiXNxNzasuRVDRQ801/fgqjNNgncMuMArgAkDsEIcZIZ2nZfAuMLG/Rbwdsnp/07Zsl54j1dKLIXOpjkrkWvodtVlnKOG9tI9X5ENDsfX25ImozJeinTlh9s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283312; c=relaxed/simple; bh=KpvxCmkErqZkpkFlZxuOiD0H1zQQRjiBPevc5uSnYMg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C6DkivVM6ut7HmMohUdYOJqhkP9/xIVSU1vFpkGXJfmBidQiH3il4kMl7w+LgYk21c8eCCzkCXZSQ6f2UZPymgeXsfYpybPsToqoDd/5qj3RKgm996j5vBQi9M13taSWiJM1UU18tvMI31yIKd1Ou8khQnUusn3L72DDGFfxVwg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=rv/n3xTh; arc=none smtp.client-ip=91.218.175.188 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="rv/n3xTh" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283308; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a3pWwo32CSd/xdzbwlEKqQlxQioQvtTwJzQQijNHuxo=; b=rv/n3xTh+FfqfVwRgZhDC5J/Cy3JnLVmz2CXOKyp5rweKfCPAdrVW3mB7lZ4i57eRkB9QA n/pufP3a06gUyfSPMNCQH2SeIa5iT1Hi0NYJbLr6dk9vlX4IQ4ZNLcwbRkzEoe1POc3wxg l/j2pxCknOa1ZdUXRYVwehHD6rGDBHw= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 2/8] bpf: Export bpf_base_func_proto Date: Wed, 21 Aug 2024 16:34:32 -0700 Message-ID: <20240821233440.1855263-3-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau The bpf_testmod needs to use the bpf_tail_call helper in a later selftest patch. This patch is to EXPORT_GPL_SYMBOL the bpf_base_func_proto. Signed-off-by: Martin KaFai Lau --- kernel/bpf/helpers.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 12e3aa40b180..84c2034a2f53 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2034,6 +2034,7 @@ bpf_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return NULL; } } +EXPORT_SYMBOL_GPL(bpf_base_func_proto); void bpf_list_head_free(const struct btf_field *field, void *list_head, struct bpf_spin_lock *spin_lock) From patchwork Wed Aug 21 23:34:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772230 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-180.mta0.migadu.com (out-180.mta0.migadu.com [91.218.175.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94BA217C9EA for ; Wed, 21 Aug 2024 23:35:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283315; cv=none; b=JL6MmnS8P07shdRvffAQlbuQJG6r7n82lsq+JpJvpP+/1e9OU81alD++uMEaBsM7BJe6jXayt/cH7je7FIYO4LyRPgf/JOdIiAmfA0k00LZaUw5CwLIQACoVEVybuFwWOXUsRjgWtylz8NEEyC5ZAAELmnMXqLDQJOzH+38DsYY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283315; c=relaxed/simple; bh=JtoVD/Rz6445Pllrir2sUAMzNJ1c9BzjOy7KeGBQffM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PLdIqxidFZLUH+83ddQrBK/etqnr+FNtiXD/RIj9XtZWewciicuYb0gGzzI6rgnN9D3uxXOTzQu1kNTpdSK6s75Nlthfl66IkS/LsjXMeEoLObqNpn2v8fXpeUzTsSFhsY8QinYsnOB9bkYCKKP1mj6ntTIBxKmfXE2bXDCaMHs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=ZRVrjZYW; arc=none smtp.client-ip=91.218.175.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="ZRVrjZYW" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283311; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1E8zLWG9vB0lDCtRjRaMsUBDJyaY5b4HpTsKjD1g4w4=; b=ZRVrjZYWkiffPur0irx+TYEPhdDw/EYy53RBejZm25+sDfZePa2XDyVyJyGKIx6nmB9u8s hZNgSTeZUU2ojfZhE2rnK23SEUc4wAkCd41jUnOLw/M93lu4waWUMUmJLR3dWNxfOqmaRo avJkhA7K+yAGIXqAn2YsiLW/vfkN/PA= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 3/8] selftests/bpf: attach struct_ops maps before test prog runs Date: Wed, 21 Aug 2024 16:34:33 -0700 Message-ID: <20240821233440.1855263-4-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Eduard Zingerman In test_loader based tests to bpf_map__attach_struct_ops() before call to bpf_prog_test_run_opts() in order to trigger bpf_struct_ops->reg() callbacks on kernel side. This allows to use __retval macro for struct_ops tests. Signed-off-by: Eduard Zingerman Signed-off-by: Martin KaFai Lau --- tools/testing/selftests/bpf/test_loader.c | 27 +++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index 12b0c41e8d64..01feba554a41 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -729,11 +729,13 @@ void run_subtest(struct test_loader *tester, { struct test_subspec *subspec = unpriv ? &spec->unpriv : &spec->priv; struct bpf_program *tprog = NULL, *tprog_iter; + struct bpf_link *link, *links[32] = {}; struct test_spec *spec_iter; struct cap_state caps = {}; struct bpf_object *tobj; struct bpf_map *map; int retval, err, i; + int links_cnt = 0; bool should_load; if (!test__start_subtest(subspec->name)) @@ -823,6 +825,26 @@ void run_subtest(struct test_loader *tester, if (restore_capabilities(&caps)) goto tobj_cleanup; + /* Do bpf_map__attach_struct_ops() for each struct_ops map. + * This should trigger bpf_struct_ops->reg callback on kernel side. + */ + bpf_object__for_each_map(map, tobj) { + if (!bpf_map__autocreate(map) || + bpf_map__type(map) != BPF_MAP_TYPE_STRUCT_OPS) + continue; + if (links_cnt >= ARRAY_SIZE(links)) { + PRINT_FAIL("too many struct_ops maps"); + goto tobj_cleanup; + } + link = bpf_map__attach_struct_ops(map); + if (!link) { + PRINT_FAIL("bpf_map__attach_struct_ops failed for map %s: err=%d\n", + bpf_map__name(map), err); + goto tobj_cleanup; + } + links[links_cnt++] = link; + } + if (tester->pre_execution_cb) { err = tester->pre_execution_cb(tobj); if (err) { @@ -837,9 +859,14 @@ void run_subtest(struct test_loader *tester, PRINT_FAIL("Unexpected retval: %d != %d\n", retval, subspec->retval); goto tobj_cleanup; } + /* redo bpf_map__attach_struct_ops for each test */ + while (links_cnt > 0) + bpf_link__destroy(links[--links_cnt]); } tobj_cleanup: + while (links_cnt > 0) + bpf_link__destroy(links[--links_cnt]); bpf_object__close(tobj); subtest_cleanup: test__end_subtest(); From patchwork Wed Aug 21 23:34:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772231 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D02717C9EA for ; Wed, 21 Aug 2024 23:35:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.184 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283319; cv=none; b=L4bvaFIgfgvZNTi6Eks91MA3M/3zoTHJZjnox4eQrm/6t4R27qVUw10EFGHDEh73J8xfUjG4tVok1O+qedC6xJX3zEn8HFJyBnJkq9qFyUj7fp/l+6bNJJbn2Ibc+P2d5hVH4cRrGpMxsJwcMzBn9zycJMAyDmxVVfM/daO0n7U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283319; c=relaxed/simple; bh=ET1tKYWnUcfTR8hzhA/fX9HM+WomGiUcGvIdgWywe30=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=up34nAfROI1vBfmOdPtIaDBb70iKr+f7wHW2v7Ey51HQzX5jVuqDyfCytev2TeBqX4Y21h7Gd3GIyziYmF/2YntaHvyD0g9HmoOPevVV/QK3GLl3Gt+uAouvhUoeThbAKm7Wk1MUPoRd1aXGcbcrhUPCrLYjGVQx+cUWbXOJ8XY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=sFBAg9u0; arc=none smtp.client-ip=91.218.175.184 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="sFBAg9u0" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283314; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1lQfqk6iLXP3K0Usj3SE3so07MQO0KQqMTcBfQz7Ahw=; b=sFBAg9u0FOksIpmWTGt00m5IcPuGC7djvQWIsUwSTpxYXZ0kM3PTq05gzX0iHjjAq245G/ /mrOwHVl+Yc0oyv20rejElTWcd2fESZr6TIbsQHHDY4kmWD5MlIVHLO71e/mP1YdEYoSSO NcBOZzGJ39KGrWplKFin3ByWWpes6v0= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 4/8] selftests/bpf: Test gen_prologue and gen_epilogue Date: Wed, 21 Aug 2024 16:34:34 -0700 Message-ID: <20240821233440.1855263-5-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau This test adds a new struct_ops "bpf_testmod_st_ops" in bpf_testmod. The ops of the bpf_testmod_st_ops is triggered by new kfunc calls "bpf_kfunc_st_ops_test_*logue". These new kfunc calls are primarily used by the SEC("syscall") program. The test triggering sequence is like: SEC("syscall") syscall_prologue_subprog(struct st_ops_args *args) bpf_kfunc_st_op_test_prologue(args) st_ops->test_prologue(args) .gen_prologue adds 1000 to args->a .gen_epilogue adds 10000 to args->a .gen_epilogue will also set the r0 to 2 * args->a. The .gen_prologue and .gen_epilogue of the bpf_testmod_st_ops will test the prog->aux->attach_func_name to decide if it needs to generate codes. The main programs of the pro_epilogue_subprog.c will call a subprog() which does "args->a += 1". The main programs of the pro_epilogue_kfunc.c will call a new kfunc bpf_kfunc_st_ops_inc10 which does "args->a += 10". This patch uses the test_loader infra to check the __xlated instructions patched after gen_prologue and/or gen_epilogue. The __xlated check is based on Eduard's example (Thanks!) in v1. args->a is returned by the struct_ops prog (either the main prog or the epilogue). Thus, the __retval of the SEC("syscall") prog is checked. For example, when triggering the ops in the 'SEC("struct_ops/test_epilogue_subprog") int test_epilogue_subprog' The expected args->a is +1 (subprog call) + 10000 (.gen_epilogue) = 10001. The expected return value is 2 * 10001 (.gen_epilogue). Suggested-by: Eduard Zingerman Signed-off-by: Martin KaFai Lau --- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 190 ++++++++++++++++++ .../selftests/bpf/bpf_testmod/bpf_testmod.h | 11 + .../bpf/bpf_testmod/bpf_testmod_kfunc.h | 6 + .../selftests/bpf/prog_tests/pro_epilogue.c | 12 ++ .../selftests/bpf/progs/pro_epilogue_kfunc.c | 156 ++++++++++++++ .../bpf/progs/pro_epilogue_subprog.c | 125 ++++++++++++ 6 files changed, 500 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/pro_epilogue.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 3687a40b61c6..7194330bdefc 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include "bpf_testmod.h" @@ -920,6 +921,51 @@ __bpf_kfunc int bpf_kfunc_call_kernel_getpeername(struct addr_args *args) return err; } +static DEFINE_MUTEX(st_ops_mutex); +static struct bpf_testmod_st_ops *st_ops; + +__bpf_kfunc int bpf_kfunc_st_ops_test_prologue(struct st_ops_args *args) +{ + int ret = -1; + + mutex_lock(&st_ops_mutex); + if (st_ops && st_ops->test_prologue) + ret = st_ops->test_prologue(args); + mutex_unlock(&st_ops_mutex); + + return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_test_epilogue(struct st_ops_args *args) +{ + int ret = -1; + + mutex_lock(&st_ops_mutex); + if (st_ops && st_ops->test_epilogue) + ret = st_ops->test_epilogue(args); + mutex_unlock(&st_ops_mutex); + + return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_test_pro_epilogue(struct st_ops_args *args) +{ + int ret = -1; + + mutex_lock(&st_ops_mutex); + if (st_ops && st_ops->test_pro_epilogue) + ret = st_ops->test_pro_epilogue(args); + mutex_unlock(&st_ops_mutex); + + return ret; +} + +__bpf_kfunc int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) +{ + args->a += 10; + return args->a; +} + BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) BTF_ID_FLAGS(func, bpf_kfunc_call_test1) @@ -956,6 +1002,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_sendmsg, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_sock_sendmsg, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_getsockname, KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_kfunc_call_kernel_getpeername, KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_prologue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_epilogue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_test_pro_epilogue, KF_TRUSTED_ARGS | KF_SLEEPABLE) +BTF_ID_FLAGS(func, bpf_kfunc_st_ops_inc10, KF_TRUSTED_ARGS) BTF_KFUNCS_END(bpf_testmod_check_kfunc_ids) static int bpf_testmod_ops_init(struct btf *btf) @@ -1075,6 +1125,144 @@ struct bpf_struct_ops bpf_testmod_ops2 = { .owner = THIS_MODULE, }; +static int bpf_test_mod_st_ops__test_prologue(struct st_ops_args *args) +{ + return 0; +} + +static int bpf_test_mod_st_ops__test_epilogue(struct st_ops_args *args) +{ + return 0; +} + +static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args) +{ + return 0; +} + +static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, + const struct bpf_prog *prog) +{ + struct bpf_insn *insn = insn_buf; + + if (strcmp(prog->aux->attach_func_name, "test_prologue") && + strcmp(prog->aux->attach_func_name, "test_pro_epilogue")) + return 0; + + /* r6 = r1[0]; // r6 will be "struct st_ops *args". r1 is "u64 *ctx". + * r7 = r6->a; + * r7 += 1000; + * r6->a = r7; + */ + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0); + *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_6, offsetof(struct st_ops_args, a)); + *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_7, 1000); + *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_6, BPF_REG_7, offsetof(struct st_ops_args, a)); + *insn++ = prog->insnsi[0]; + + return insn - insn_buf; +} + +static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog, + s16 ctx_stack_off) +{ + struct bpf_insn *insn = insn_buf; + + if (strcmp(prog->aux->attach_func_name, "test_epilogue") && + strcmp(prog->aux->attach_func_name, "test_pro_epilogue")) + return 0; + + /* r1 = stack[ctx_stack_off]; // r1 will be "u64 *ctx" + * r1 = r1[0]; // r1 will be "struct st_ops *args" + * r6 = r1->a; + * r6 += 10000; + * r1->a = r6; + * r0 = r6; + * r0 *= 2; + * BPF_EXIT; + */ + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_FP, ctx_stack_off); + *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_1, BPF_REG_1, 0); + *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, offsetof(struct st_ops_args, a)); + *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_6, 10000); + *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_6, offsetof(struct st_ops_args, a)); + *insn++ = BPF_MOV32_REG(BPF_REG_0, BPF_REG_6); + *insn++ = BPF_ALU32_IMM(BPF_MUL, BPF_REG_0, 2); + *insn++ = BPF_EXIT_INSN(); + + return insn - insn_buf; +} + +static int st_ops_btf_struct_access(struct bpf_verifier_log *log, + const struct bpf_reg_state *reg, + int off, int size) +{ + if (off < 0 || off + size > sizeof(struct st_ops_args)) + return -EACCES; + return 0; +} + +static const struct bpf_verifier_ops st_ops_verifier_ops = { + .is_valid_access = bpf_testmod_ops_is_valid_access, + .btf_struct_access = st_ops_btf_struct_access, + .gen_prologue = st_ops_gen_prologue, + .gen_epilogue = st_ops_gen_epilogue, + .get_func_proto = bpf_base_func_proto, +}; + +static struct bpf_testmod_st_ops st_ops_cfi_stubs = { + .test_prologue = bpf_test_mod_st_ops__test_prologue, + .test_epilogue = bpf_test_mod_st_ops__test_epilogue, + .test_pro_epilogue = bpf_test_mod_st_ops__test_pro_epilogue, +}; + +static int st_ops_reg(void *kdata, struct bpf_link *link) +{ + int err = 0; + + mutex_lock(&st_ops_mutex); + if (st_ops) { + pr_err("st_ops has already been registered\n"); + err = -EEXIST; + goto unlock; + } + st_ops = kdata; + +unlock: + mutex_unlock(&st_ops_mutex); + return err; +} + +static void st_ops_unreg(void *kdata, struct bpf_link *link) +{ + mutex_lock(&st_ops_mutex); + st_ops = NULL; + mutex_unlock(&st_ops_mutex); +} + +static int st_ops_init(struct btf *btf) +{ + return 0; +} + +static int st_ops_init_member(const struct btf_type *t, + const struct btf_member *member, + void *kdata, const void *udata) +{ + return 0; +} + +static struct bpf_struct_ops testmod_st_ops = { + .verifier_ops = &st_ops_verifier_ops, + .init = st_ops_init, + .init_member = st_ops_init_member, + .reg = st_ops_reg, + .unreg = st_ops_unreg, + .cfi_stubs = &st_ops_cfi_stubs, + .name = "bpf_testmod_st_ops", + .owner = THIS_MODULE, +}; + extern int bpf_fentry_test1(int a); static int bpf_testmod_init(void) @@ -1092,8 +1280,10 @@ static int bpf_testmod_init(void) ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_testmod_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_testmod_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_testmod_kfunc_set); ret = ret ?: register_bpf_struct_ops(&bpf_bpf_testmod_ops, bpf_testmod_ops); ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops2, bpf_testmod_ops2); + ret = ret ?: register_bpf_struct_ops(&testmod_st_ops, bpf_testmod_st_ops); ret = ret ?: register_btf_id_dtor_kfuncs(bpf_testmod_dtors, ARRAY_SIZE(bpf_testmod_dtors), THIS_MODULE); diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h index fe0d402b0d65..3241a9d796ed 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h @@ -94,4 +94,15 @@ struct bpf_testmod_ops2 { int (*test_1)(void); }; +struct st_ops_args { + int a; +}; + +struct bpf_testmod_st_ops { + int (*test_prologue)(struct st_ops_args *args); + int (*test_epilogue)(struct st_ops_args *args); + int (*test_pro_epilogue)(struct st_ops_args *args); + struct module *owner; +}; + #endif /* _BPF_TESTMOD_H */ diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h index e587a79f2239..0df429a0edaa 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h @@ -144,4 +144,10 @@ void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr, struct bpf_dynptr *ptr__nulla struct bpf_testmod_ctx *bpf_testmod_ctx_create(int *err) __ksym; void bpf_testmod_ctx_release(struct bpf_testmod_ctx *ctx) __ksym; +struct st_ops_args; +int bpf_kfunc_st_ops_test_prologue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_test_epilogue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_test_pro_epilogue(struct st_ops_args *args) __ksym; +int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) __ksym; + #endif /* _BPF_TESTMOD_KFUNC_H */ diff --git a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c new file mode 100644 index 000000000000..69e4a5a1756d --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include +#include "pro_epilogue_subprog.skel.h" +#include "pro_epilogue_kfunc.skel.h" + +void test_pro_epilogue(void) +{ + RUN_TESTS(pro_epilogue_subprog); + RUN_TESTS(pro_epilogue_kfunc); +} diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c b/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c new file mode 100644 index 000000000000..7d1124cf4942 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include +#include +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +void __kfunc_btf_root(void) +{ + struct st_ops_args args = {}; + + bpf_kfunc_st_ops_inc10(&args); +} + +static __noinline __used int subprog(struct st_ops_args *args) +{ + args->a += 1; + return args->a; +} + +__success +/* prologue */ +__xlated("0: r6 = *(u64 *)(r1 +0)") +__xlated("1: r7 = *(u32 *)(r6 +0)") +__xlated("2: w7 += 1000") +__xlated("3: *(u32 *)(r6 +0) = r7") +/* main prog */ +__xlated("4: r1 = *(u64 *)(r1 +0)") +__xlated("5: r6 = r1") +__xlated("6: call kernel-function") +__xlated("7: r1 = r6") +__xlated("8: call pc+1") +__xlated("9: exit") +SEC("struct_ops/test_prologue_kfunc") +__naked int test_prologue_kfunc(void) +{ + asm volatile ( + "r1 = *(u64 *)(r1 +0);" + "r6 = r1;" + "call %[bpf_kfunc_st_ops_inc10];" + "r1 = r6;" + "call subprog;" + "exit;" + : + : __imm(bpf_kfunc_st_ops_inc10) + : __clobber_all); +} + +__success +/* save __u64 *ctx to stack */ +__xlated("0: *(u64 *)(r10 -8) = r1") +/* main prog */ +__xlated("1: r1 = *(u64 *)(r1 +0)") +__xlated("2: r6 = r1") +__xlated("3: call kernel-function") +__xlated("4: r1 = r6") +__xlated("5: call pc+") +/* epilogue */ +__xlated("6: r1 = *(u64 *)(r10 -8)") +__xlated("7: r1 = *(u64 *)(r1 +0)") +__xlated("8: r6 = *(u32 *)(r1 +0)") +__xlated("9: w6 += 10000") +__xlated("10: *(u32 *)(r1 +0) = r6") +__xlated("11: w0 = w6") +__xlated("12: w0 *= 2") +__xlated("13: exit") +SEC("struct_ops/test_epilogue_kfunc") +__naked int test_epilogue_kfunc(void) +{ + asm volatile ( + "r1 = *(u64 *)(r1 +0);" + "r6 = r1;" + "call %[bpf_kfunc_st_ops_inc10];" + "r1 = r6;" + "call subprog;" + "exit;" + : + : __imm(bpf_kfunc_st_ops_inc10) + : __clobber_all); +} + +__success +/* prologue */ +__xlated("0: r6 = *(u64 *)(r1 +0)") +__xlated("1: r7 = *(u32 *)(r6 +0)") +__xlated("2: w7 += 1000") +__xlated("3: *(u32 *)(r6 +0) = r7") +/* save __u64 *ctx to stack */ +__xlated("4: *(u64 *)(r10 -8) = r1") +/* main prog */ +__xlated("5: r1 = *(u64 *)(r1 +0)") +__xlated("6: r6 = r1") +__xlated("7: call kernel-function") +__xlated("8: r1 = r6") +__xlated("9: call pc+") +/* epilogue */ +__xlated("10: r1 = *(u64 *)(r10 -8)") +__xlated("11: r1 = *(u64 *)(r1 +0)") +__xlated("12: r6 = *(u32 *)(r1 +0)") +__xlated("13: w6 += 10000") +__xlated("14: *(u32 *)(r1 +0) = r6") +__xlated("15: w0 = w6") +__xlated("16: w0 *= 2") +__xlated("17: exit") +SEC("struct_ops/test_pro_epilogue_kfunc") +__naked int test_pro_epilogue_kfunc(void) +{ + asm volatile ( + "r1 = *(u64 *)(r1 +0);" + "r6 = r1;" + "call %[bpf_kfunc_st_ops_inc10];" + "r1 = r6;" + "call subprog;" + "exit;" + : + : __imm(bpf_kfunc_st_ops_inc10) + : __clobber_all); +} + +SEC("syscall") +__retval(1011) /* PROLOGUE_A [1000] + KFUNC_INC10 + SUBPROG_A [1] */ +int syscall_prologue_kfunc(void *ctx) +{ + struct st_ops_args args = {}; + + return bpf_kfunc_st_ops_test_prologue(&args); +} + +SEC("syscall") +__retval(20022) /* (KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +int syscall_epilogue_kfunc(void *ctx) +{ + struct st_ops_args args = {}; + + return bpf_kfunc_st_ops_test_epilogue(&args); +} + +SEC("syscall") +__retval(22022) /* (PROLOGUE_A [1000] + KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +int syscall_pro_epilogue_kfunc(void *ctx) +{ + struct st_ops_args args = {}; + + return bpf_kfunc_st_ops_test_pro_epilogue(&args); +} + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops pro_epilogue_kfunc = { + .test_prologue = (void *)test_prologue_kfunc, + .test_epilogue = (void *)test_epilogue_kfunc, + .test_pro_epilogue = (void *)test_pro_epilogue_kfunc, +}; diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c b/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c new file mode 100644 index 000000000000..c91b1bf30e37 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include +#include +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +static __noinline __used int subprog(struct st_ops_args *args) +{ + args->a += 1; + return args->a; +} + +__success +/* prologue */ +__xlated("0: r6 = *(u64 *)(r1 +0)") +__xlated("1: r7 = *(u32 *)(r6 +0)") +__xlated("2: w7 += 1000") +__xlated("3: *(u32 *)(r6 +0) = r7") +/* main prog */ +__xlated("4: r1 = *(u64 *)(r1 +0)") +__xlated("5: call pc+1") +__xlated("6: exit") +SEC("struct_ops/test_prologue_subprog") +__naked int test_prologue_subprog(void) +{ + asm volatile ( + "r1 = *(u64 *)(r1 +0);" + "call subprog;" + "exit;" + ::: __clobber_all); +} + +__success +/* save __u64 *ctx to stack */ +__xlated("0: *(u64 *)(r10 -8) = r1") +/* main prog */ +__xlated("1: r1 = *(u64 *)(r1 +0)") +__xlated("2: call pc+") +/* epilogue */ +__xlated("3: r1 = *(u64 *)(r10 -8)") +__xlated("4: r1 = *(u64 *)(r1 +0)") +__xlated("5: r6 = *(u32 *)(r1 +0)") +__xlated("6: w6 += 10000") +__xlated("7: *(u32 *)(r1 +0) = r6") +__xlated("8: w0 = w6") +__xlated("9: w0 *= 2") +__xlated("10: exit") +SEC("struct_ops/test_epilogue_subprog") +__naked int test_epilogue_subprog(void) +{ + asm volatile ( + "r1 = *(u64 *)(r1 +0);" + "call subprog;" + "exit;" + ::: __clobber_all); +} + +__success +/* prologue */ +__xlated("0: r6 = *(u64 *)(r1 +0)") +__xlated("1: r7 = *(u32 *)(r6 +0)") +__xlated("2: w7 += 1000") +__xlated("3: *(u32 *)(r6 +0) = r7") +/* save __u64 *ctx to stack */ +__xlated("4: *(u64 *)(r10 -8) = r1") +/* main prog */ +__xlated("5: r1 = *(u64 *)(r1 +0)") +__xlated("6: call pc+") +/* epilogue */ +__xlated("7: r1 = *(u64 *)(r10 -8)") +__xlated("8: r1 = *(u64 *)(r1 +0)") +__xlated("9: r6 = *(u32 *)(r1 +0)") +__xlated("10: w6 += 10000") +__xlated("11: *(u32 *)(r1 +0) = r6") +__xlated("12: w0 = w6") +__xlated("13: w0 *= 2") +__xlated("14: exit") +SEC("struct_ops/test_pro_epilogue_subprog") +__naked int test_pro_epilogue_subprog(void) +{ + asm volatile ( + "r1 = *(u64 *)(r1 +0);" + "call subprog;" + "exit;" + ::: __clobber_all); +} + +SEC("syscall") +__retval(1001) /* PROLOGUE_A [1000] + SUBPROG_A [1] */ +int syscall_prologue_subprog(void *ctx) +{ + struct st_ops_args args = {}; + + return bpf_kfunc_st_ops_test_prologue(&args); +} + +SEC("syscall") +__retval(20002) /* (SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +int syscall_epilogue_subprog(void *ctx) +{ + struct st_ops_args args = {}; + + return bpf_kfunc_st_ops_test_epilogue(&args); +} + +SEC("syscall") +__retval(22002) /* (PROLOGUE_A [1000] + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +int syscall_pro_epilogue_subprog(void *ctx) +{ + struct st_ops_args args = {}; + + return bpf_kfunc_st_ops_test_pro_epilogue(&args); +} + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops pro_epilogue_subprog = { + .test_prologue = (void *)test_prologue_subprog, + .test_epilogue = (void *)test_epilogue_subprog, + .test_pro_epilogue = (void *)test_pro_epilogue_subprog, +}; From patchwork Wed Aug 21 23:34:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772232 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-186.mta0.migadu.com (out-186.mta0.migadu.com [91.218.175.186]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CD8917BB00 for ; Wed, 21 Aug 2024 23:35:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.186 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283321; cv=none; b=eUUWSh/aFm9sf3kUJ3C+xBSrOr+7SvYbl2iWfEqaPo136EJkKhaw65OKFH9VCwYHf3nB9T6pzkKHnCzrQiJVuA1r4SaVZMPDl9o3nTggxRk/B+QCPet9apk8gzTs+QQ40WucStXPVLt0hffoY3d1+HJ3f/8IG8rY2PSx5vO3QS0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283321; c=relaxed/simple; bh=P5vL0GU2GdAXpXOJZpCtJiJ41Rl5TvhlKGALkhvRpOI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hYSWBChMpv34I6O4Abc4pEIm1WSphN6iL3EpQZ6sohk5rPT08Pe2R//XgF1zXMPSbS9E6KPIy9NJ3MlAg2yafUbp4THnrDBhZA/1kQdRRkVUFxsEAxLY1hqzswo+MoYLD64XtGwtmWSIRnrO5bhQPv5p1iWM0e7QijmlVwjXAwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=McETEN4t; arc=none smtp.client-ip=91.218.175.186 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="McETEN4t" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9rIum6Tlk68haTWumoIKSXGOj7KCH1RT3rD2OnSeY90=; b=McETEN4tjvPoRhQqhg95EnqgnDOdVxbIUUhiLfyx6vRbLQETTgBI0jvZAi3+c5RPxFQ1Js J0RMwcJHklHkz3BsX64rGS7BtI8rPFgGV2dvaoKobgrcaYFF8kCJHdMopE9mQD/+44RKZP 7p8kx1c9+ifxV8BKPMfSm5ema8LTgGA= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 5/8] selftests/bpf: Add tailcall epilogue test Date: Wed, 21 Aug 2024 16:34:35 -0700 Message-ID: <20240821233440.1855263-6-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau This patch adds a gen_epilogue test to test a main prog using a bpf_tail_call. A non test_loader test is used. The tailcall target program, "test_epilogue_subprog", needs to be used in a struct_ops map before it can be loaded. Another struct_ops map is also needed to host the actual "test_epilogue_tailcall" struct_ops program that does the bpf_tail_call. The earlier test_loader patch will attach all struct_ops maps but the bpf_testmod.c does not support >1 attached struct_ops. The earlier patch used the test_loader which has already covered checking for the patched pro/epilogue instructions. This is done by the __xlated tag. This patch goes for the regular skel load and syscall test to do the tailcall test that can also allow to directly pass the the "struct st_ops_args *args" as ctx_in to the SEC("syscall") program. Signed-off-by: Martin KaFai Lau --- .../selftests/bpf/prog_tests/pro_epilogue.c | 38 ++++++++++++ .../selftests/bpf/progs/epilogue_tailcall.c | 58 +++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/epilogue_tailcall.c diff --git a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c index 69e4a5a1756d..98de677c55a9 100644 --- a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +++ b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c @@ -4,9 +4,47 @@ #include #include "pro_epilogue_subprog.skel.h" #include "pro_epilogue_kfunc.skel.h" +#include "epilogue_tailcall.skel.h" + +struct st_ops_args { + int a; +}; + +static void test_tailcall(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct epilogue_tailcall *skel; + struct st_ops_args args; + int err, prog_fd; + + skel = epilogue_tailcall__open_and_load(); + if (!ASSERT_OK_PTR(skel, "epilogue_tailcall__open_and_load")) + return; + + topts.ctx_in = &args; + topts.ctx_size_in = sizeof(args); + + skel->links.epilogue_tailcall = + bpf_map__attach_struct_ops(skel->maps.epilogue_tailcall); + if (!ASSERT_OK_PTR(skel->links.epilogue_tailcall, "attach_struct_ops")) + goto done; + + /* tailcall prog + gen_epilogue */ + memset(&args, 0, sizeof(args)); + prog_fd = bpf_program__fd(skel->progs.syscall_epilogue_tailcall); + err = bpf_prog_test_run_opts(prog_fd, &topts); + ASSERT_OK(err, "bpf_prog_test_run_opts"); + ASSERT_EQ(args.a, 10001, "args.a"); + ASSERT_EQ(topts.retval, 10001 * 2, "topts.retval"); + +done: + epilogue_tailcall__destroy(skel); +} void test_pro_epilogue(void) { RUN_TESTS(pro_epilogue_subprog); RUN_TESTS(pro_epilogue_kfunc); + if (test__start_subtest("tailcall")) + test_tailcall(); } diff --git a/tools/testing/selftests/bpf/progs/epilogue_tailcall.c b/tools/testing/selftests/bpf/progs/epilogue_tailcall.c new file mode 100644 index 000000000000..7275dd594de0 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/epilogue_tailcall.c @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ + +#include +#include +#include "bpf_misc.h" +#include "../bpf_testmod/bpf_testmod.h" +#include "../bpf_testmod/bpf_testmod_kfunc.h" + +char _license[] SEC("license") = "GPL"; + +static __noinline __used int subprog(struct st_ops_args *args) +{ + args->a += 1; + return args->a; +} + +SEC("struct_ops/test_epilogue_subprog") +int BPF_PROG(test_epilogue_subprog, struct st_ops_args *args) +{ + subprog(args); + return args->a; +} + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 1); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); + __array(values, void (void)); +} epilogue_map SEC(".maps") = { + .values = { + [0] = (void *)&test_epilogue_subprog, + } +}; + +SEC("struct_ops/test_epilogue_tailcall") +int test_epilogue_tailcall(unsigned long long *ctx) +{ + bpf_tail_call(ctx, &epilogue_map, 0); + return 0; +} + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops epilogue_tailcall = { + .test_epilogue = (void *)test_epilogue_tailcall, +}; + +SEC(".struct_ops.link") +struct bpf_testmod_st_ops epilogue_subprog = { + .test_epilogue = (void *)test_epilogue_subprog, +}; + +SEC("syscall") +int syscall_epilogue_tailcall(struct st_ops_args *args) +{ + return bpf_kfunc_st_ops_test_epilogue(args); +} From patchwork Wed Aug 21 23:34:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772233 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8FD417BB12 for ; Wed, 21 Aug 2024 23:35:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.184 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283324; cv=none; b=peFAJ52lG3gsqdgqDIdGQ3sZUYLv7t4yYiDyQPNT4166VlWqlOqTO14uCh8T++GDjweV7mYWMUHNgpObsymAKLJt1IIM02yjONKwuqiEwE5OyPugIEqtjRYNlosC6UXxhJzIWTqauJ7JqIELfBspicfL+XlLq/CSHKYj6BpswM8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283324; c=relaxed/simple; bh=ugdQM2lsY6hQsSqLSBvupN4w+O6HzDlShx/quwjdQL8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kjNYDJ6CT4IW7PLMOi2ag1GH9gUfUx+S1lkhCx2sOIYKr7ZrxDqe0ppG2uZxaKaOwpbGORwMNHWkOQ1mGgs1U0h+qEGI99UxhDNXSNrYiUmHwgswL9XImwDr42d2aK43L5s+xg5hmAywklR2SLVL/0GSqEV+cRT4Hr3bxDwG1io= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=BKQFCQ2L; arc=none smtp.client-ip=91.218.175.184 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="BKQFCQ2L" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283321; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uppmJSrRXM7oEToWK+lo+equJrHNZ8oojVtpN0ihz0g=; b=BKQFCQ2LTIx/5hgsIHWwCDHYhbVgKr4QfGFssDcE4HxN4OeoE2hQ6tnDby3pVZnODiSv5d KucnAM5lrHql4g0XASUrDWwuoEzhoJvfGdm3AGYb01jGx1C5w6ynAIRy8P1S6U9Py9GKOm eq1uO0xO4ww8HIiec5AwH4K+jf9WRi0= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 6/8] bpf: Add module parameter to gen_prologue and gen_epilogue Date: Wed, 21 Aug 2024 16:34:36 -0700 Message-ID: <20240821233440.1855263-7-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau This patch adds a "struct module **module" arg to the .gen_prologue and .gen_epilogue. This will allow the .gen_pro/epilogue to make kfunc call because the verifer needs to know the kfunc's BTF. The next patch will figure the kfunc's BTF from the module. It also exposes the "btf_get_module_btf" function to help figuring out the btf from a module. Signed-off-by: Martin KaFai Lau --- include/linux/bpf.h | 4 ++-- include/linux/btf.h | 1 + kernel/bpf/btf.c | 2 +- kernel/bpf/cgroup.c | 3 ++- kernel/bpf/verifier.c | 4 ++-- net/core/filter.c | 6 +++--- tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c | 4 ++-- 7 files changed, 13 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 8ee9d87c332a..6d97e57c7801 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -973,9 +973,9 @@ struct bpf_verifier_ops { const struct bpf_prog *prog, struct bpf_insn_access_aux *info); int (*gen_prologue)(struct bpf_insn *insn, bool direct_write, - const struct bpf_prog *prog); + const struct bpf_prog *prog, struct module **module); int (*gen_epilogue)(struct bpf_insn *insn, const struct bpf_prog *prog, - s16 ctx_stack_off); + s16 ctx_stack_off, struct module **module); int (*gen_ld_abs)(const struct bpf_insn *orig, struct bpf_insn *insn_buf); u32 (*convert_ctx_access)(enum bpf_access_type type, diff --git a/include/linux/btf.h b/include/linux/btf.h index cffb43133c68..177187fa3819 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -580,6 +580,7 @@ bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf, int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_type); bool btf_types_are_same(const struct btf *btf1, u32 id1, const struct btf *btf2, u32 id2); +struct btf *btf_get_module_btf(const struct module *module); #else static inline const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index b12db397303e..6911f8cdb736 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7965,7 +7965,7 @@ struct module *btf_try_get_module(const struct btf *btf) /* Returns struct btf corresponding to the struct module. * This function can return NULL or ERR_PTR. */ -static struct btf *btf_get_module_btf(const struct module *module) +struct btf *btf_get_module_btf(const struct module *module) { #ifdef CONFIG_DEBUG_INFO_BTF_MODULES struct btf_module *btf_mod, *tmp; diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index e7113d700b87..cbc7df8dfbad 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -2503,7 +2503,8 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, static int cg_sockopt_get_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, + struct module **module) { /* Nothing to do for sockopt argument. The data is kzalloc'ated. */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bbb655f0c7b5..5e995b7884fb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19622,7 +19622,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) if (ops->gen_epilogue) { epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, - -(subprogs[0].stack_depth + 8)); + -(subprogs[0].stack_depth + 8), NULL); if (epilogue_cnt >= ARRAY_SIZE(epilogue_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; @@ -19647,7 +19647,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) return -EINVAL; } cnt = ops->gen_prologue(insn_buf, env->seen_direct_write, - env->prog); + env->prog, NULL); if (cnt >= ARRAY_SIZE(insn_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; diff --git a/net/core/filter.c b/net/core/filter.c index 78a6f746ea0b..65d219e71ae8 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -8844,7 +8844,7 @@ static bool sock_filter_is_valid_access(int off, int size, } static int bpf_noop_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { /* Neither direct read nor direct write requires any preliminary * action. @@ -8927,7 +8927,7 @@ static int bpf_gen_ld_abs(const struct bpf_insn *orig, } static int tc_cls_act_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { return bpf_unclone_prologue(insn_buf, direct_write, prog, TC_ACT_SHOT); } @@ -9263,7 +9263,7 @@ static bool sock_ops_is_valid_access(int off, int size, } static int sk_skb_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { return bpf_unclone_prologue(insn_buf, direct_write, prog, SK_DROP); } diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 7194330bdefc..4c75346376d9 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -1141,7 +1141,7 @@ static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args) } static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, - const struct bpf_prog *prog) + const struct bpf_prog *prog, struct module **module) { struct bpf_insn *insn = insn_buf; @@ -1164,7 +1164,7 @@ static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, } static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *prog, - s16 ctx_stack_off) + s16 ctx_stack_off, struct module **module) { struct bpf_insn *insn = insn_buf; From patchwork Wed Aug 21 23:34:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772234 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-183.mta0.migadu.com (out-183.mta0.migadu.com [91.218.175.183]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C85DB17BB12 for ; Wed, 21 Aug 2024 23:35:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.183 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283327; cv=none; b=EnaFIKcOba0+FW7E0A6DJYMEemfqBxWmGE4fxmpZy8QsBacjDG3+sJ//URVghfbbZW4SS4jL1jJhr/lfLzMm4Es0m+n/Eu6bc2M6pqEWr5uy0VmPfag8zWL1U4xB/ZBHr7Vokrc//eQEgLsFA3Yqr/qeftQQbV3dTT66OwP4o84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283327; c=relaxed/simple; bh=hpkwCapS0+PgvtZqDsWqvhK6VfSfWT0hlFAqCMT11Gw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nZWs/4TK6c6Dxs28N2+/+zysOwHrFujTbATSXUXKp/5biwyGzumhUMLnmTuSn6ux2C84lP+RhQfoYnIQDGOkyq9wey7xK6gMxPI8XRpHTv6LY92sh15nAKo6+qGizGhIfGcFD6G3tu9M5o150vuutHYn/Od9lOefFazB19Vy9jI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=w0b5RFI6; arc=none smtp.client-ip=91.218.175.183 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="w0b5RFI6" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283324; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZqNDS4pWUNs5ErrGOt/CkchEH/gP5aIHQycT+Mf0CJw=; b=w0b5RFI6WovoyMdVoMz3I4DVG6pc3r5/yLtCrJ4iVqTpi6NTbjy3DiPmHVA7pEqYQyZVj3 r0dnpKlc7WoVLGA8ooqfc2r5z76p++Ol57Sb9i5MPerDV/MDQc1ezZCsWp8lsSae+s2SBe +mBgdSHivwea8P7y8oes/Un4b2cxzos= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 7/8] bpf: Allow pro/epilogue to call kfunc Date: Wed, 21 Aug 2024 16:34:37 -0700 Message-ID: <20240821233440.1855263-8-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau The existing prologue has been able to call bpf helper but not a kfunc. This patch allows the prologue/epilogue to call the kfunc. The subsystem that implements the .gen_prologue and .gen_epilogue can add the BPF_PSEUDO_KFUNC_CALL instruction with insn->imm set to the btf func_id of the kfunc call. This part is the same as the bpf prog loaded from the sys_bpf. Another piece is to have a way for the subsystem to tell the btf object of the kfunc func_id. This patch uses the "struct module **module" argument added to the .gen_prologue and .gen_epilogue in the previous patch. The verifier will use btf_get_module_btf(module) to find out the btf object. The .gen_epi/prologue will usually use THIS_MODULE to initialize the "*module = THIS_MODULE". Only kfunc(s) from one module (or vmlinux) can be used in the .gen_epi/prologue now. In the future, the .gen_epi/prologue can return an array of modules and use the insn->off as an index into the array. When the returned module is NULL, the btf is btf_vmlinux. Then the insn->off stays at 0. This is the same as the sys_bpf. When the btf is from a module, the btf needs an entry in prog->aux->kfunc_btf_tab. The kfunc_btf_tab is currently sorted by insn->off which is the offset to the attr->fd_array. This module btf may or may not be in the kfunc_btf_tab. A new function "find_kfunc_desc_btf_offset" is added to search for the existing entry that has the same btf. If it is found, its offset will be used in the insn->off. If it is not found, it will find an offset value that is not used in the kfunc_btf_tab. Add a new entry to kfunc_btf_tab and set this new offset to the insn->off Once the insn->off is determined (either reuse an existing one or an unused one is found), it will call the existing add_kfunc_call() and everything else should fall through. Reviewed-by: Eduard Zingerman Signed-off-by: Martin KaFai Lau --- kernel/bpf/verifier.c | 115 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 112 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5e995b7884fb..f4ac254a7661 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2787,6 +2787,60 @@ static struct btf *find_kfunc_desc_btf(struct bpf_verifier_env *env, s16 offset) return btf_vmlinux ?: ERR_PTR(-ENOENT); } +static int find_kfunc_desc_btf_offset(struct bpf_verifier_env *env, struct btf *btf, + struct module *module, s16 *offset) +{ + struct bpf_kfunc_btf_tab *tab; + struct bpf_kfunc_btf *b; + s16 new_offset = 1; /* 0 is reserved for btf_vmlinux */ + u32 i; + + if (btf_is_vmlinux(btf)) { + *offset = 0; + return 0; + } + + tab = env->prog->aux->kfunc_btf_tab; + if (!tab) { + tab = kzalloc(sizeof(*tab), GFP_KERNEL); + if (!tab) + return -ENOMEM; + env->prog->aux->kfunc_btf_tab = tab; + } + + b = tab->descs; + for (i = 0; i < tab->nr_descs; i++) { + if (b[i].btf == btf) { + *offset = b[i].offset; + return 0; + } + /* tab->nr_descs (from the sys_bpf) max out at MAX_KFUNC_BTFS + * which is smaller than S16_MAX, so it will be able to find + * a new_offset to use. + */ + if (new_offset == b[i].offset) + new_offset++; + } + + if (tab->nr_descs == MAX_KFUNC_BTFS) { + verbose(env, "too many different module BTFs\n"); + return -E2BIG; + } + + if (!try_module_get(module)) + return -ENXIO; + + b = &tab->descs[tab->nr_descs++]; + btf_get(btf); + b->btf = btf; + b->module = module; + b->offset = new_offset; + *offset = new_offset; + sort(tab->descs, tab->nr_descs, sizeof(tab->descs[0]), + kfunc_btf_cmp_by_off, NULL); + return 0; +} + static int add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, s16 offset) { const struct btf_type *func, *func_proto; @@ -19603,6 +19657,50 @@ static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, return 0; } +static int fixup_pro_epilogue_kfunc(struct bpf_verifier_env *env, struct bpf_insn *insns, + int cnt, struct module *module) +{ + struct btf *btf; + u32 func_id; + int i, err; + s16 offset; + + for (i = 0; i < cnt; i++) { + if (!bpf_pseudo_kfunc_call(&insns[i])) + continue; + + /* The kernel may not have BTF available, so only + * try to get a btf if the pro/epilogue calls a kfunc. + */ + btf = btf_get_module_btf(module); + if (IS_ERR_OR_NULL(btf)) { + verbose(env, "cannot find BTF from %s for kfunc used in pro/epilogue\n", + module_name(module)); + return -EINVAL; + } + + func_id = insns[i].imm; + if (btf_is_vmlinux(btf) && + btf_id_set_contains(&special_kfunc_set, func_id)) { + verbose(env, "pro/epilogue cannot use special kfunc\n"); + btf_put(btf); + return -EINVAL; + } + + err = find_kfunc_desc_btf_offset(env, btf, module, &offset); + btf_put(btf); + if (err) + return err; + + insns[i].off = offset; + err = add_kfunc_call(env, func_id, offset); + if (err) + return err; + } + + return 0; +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -19612,21 +19710,27 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprogs = env->subprog_info; const struct bpf_verifier_ops *ops = env->ops; - int i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0; + int err, i, cnt, size, ctx_field_size, delta = 0, epilogue_cnt = 0; const int insn_cnt = env->prog->len; struct bpf_insn insn_buf[16], epilogue_buf[16], *insn; u32 target_size, size_default, off; struct bpf_prog *new_prog; enum bpf_access_type type; bool is_narrower_load; + struct module *module; if (ops->gen_epilogue) { + module = NULL; epilogue_cnt = ops->gen_epilogue(epilogue_buf, env->prog, - -(subprogs[0].stack_depth + 8), NULL); + -(subprogs[0].stack_depth + 8), &module); if (epilogue_cnt >= ARRAY_SIZE(epilogue_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; } else if (epilogue_cnt) { + err = fixup_pro_epilogue_kfunc(env, epilogue_buf, epilogue_cnt, module); + if (err) + return err; + /* Save the ARG_PTR_TO_CTX for the epilogue to use */ cnt = 0; subprogs[0].stack_depth += 8; @@ -19646,12 +19750,17 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; } + module = NULL; cnt = ops->gen_prologue(insn_buf, env->seen_direct_write, - env->prog, NULL); + env->prog, &module); if (cnt >= ARRAY_SIZE(insn_buf)) { verbose(env, "bpf verifier is misconfigured\n"); return -EINVAL; } else if (cnt) { + err = fixup_pro_epilogue_kfunc(env, insn_buf, cnt, module); + if (err) + return err; + new_prog = bpf_patch_insn_data(env, 0, insn_buf, cnt); if (!new_prog) return -ENOMEM; From patchwork Wed Aug 21 23:34:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Martin KaFai Lau X-Patchwork-Id: 13772235 X-Patchwork-Delegate: bpf@iogearbox.net Received: from out-185.mta0.migadu.com (out-185.mta0.migadu.com [91.218.175.185]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B90C417CA19 for ; Wed, 21 Aug 2024 23:35:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.185 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283330; cv=none; b=FngKqTYy9sBp8a3PdYN1udZed3xmpJNpVBkSnjfGIhjLbzYWOKyqUGHCsdBmC0sIt6ERZhZFFNgRzCP41VDeiQeVzIMKHp6FIbERo9wJLG9YFow7a8HhVQVsWhjdBcuINIzHRz/Vhr0HPYCWYJekloTLJeIg5Jx4eP/fQYMHOiI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724283330; c=relaxed/simple; bh=3WhTgvQytk56V9rStk9xNUUiRI7Pq7dr6L7nhi9ZeFw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HkgdhFgxCNN/qDkYY/mPca/g730wql025jWMvrRVLyUZr9hCfDu5AebpAxx+SpcQNUkaKqIbKuSoLCLRRt1wmd7723E1koAdUZO34l8zb/U9ybBpkJgw5PR4I3CvAnBkzUB4uQ3JRkdK/pP2fdUhd4WyeQYcuj1QQq+PusLWYDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=Q8G2cIj2; arc=none smtp.client-ip=91.218.175.185 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Q8G2cIj2" X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1724283327; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wAudpGr4s0jGpyx5HbEs+P/xvf8PX7y8TNaKM8qptck=; b=Q8G2cIj2HL83VsPKNfvXsvAtJvT863cV1MxjtCFjuDBWw36FtaQmrwqrvYBbBeV2R5DbD1 udM7zjEuw2OyWgnGw6uNpMjDu5X9yMS4fTklUqgJkhj/ofBVoUz9aJ5vIX4ut0cSXw95sq 1g92+9nQYKFJfEOFUFzj8ig8otyIOw8= From: Martin KaFai Lau To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Eduard Zingerman , Yonghong Song , Amery Hung , kernel-team@meta.com Subject: [PATCH v2 bpf-next 8/8] selftests/bpf: Add kfunc call test in gen_prologue and gen_epilogue Date: Wed, 21 Aug 2024 16:34:38 -0700 Message-ID: <20240821233440.1855263-9-martin.lau@linux.dev> In-Reply-To: <20240821233440.1855263-1-martin.lau@linux.dev> References: <20240821233440.1855263-1-martin.lau@linux.dev> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Patchwork-Delegate: bpf@iogearbox.net From: Martin KaFai Lau This patch changes the .gen_pro/epilogue of the bpf_testmod_st_ops to call kfunc. It will call the inc10 and inc100 kfunc. The __xlated instructions have been adjusted accordinly. The same goes for the __retval. The inc100 kfunc is newly added in this patch which does args->a += 100. Note that it is not in the register_btf_kfunc_id_set(), so no need to declare in the bpf_testmod_kfunc.h. It is enclosed with __bpf_kfunc_{start,end}_defs to avoid the compiler warning. Signed-off-by: Martin KaFai Lau --- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 42 +++++++++++- .../selftests/bpf/prog_tests/pro_epilogue.c | 4 +- .../selftests/bpf/progs/pro_epilogue_kfunc.c | 68 ++++++++++++------- .../bpf/progs/pro_epilogue_subprog.c | 58 ++++++++++------ 4 files changed, 123 insertions(+), 49 deletions(-) diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 4c75346376d9..6f745d29e124 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -966,6 +966,16 @@ __bpf_kfunc int bpf_kfunc_st_ops_inc10(struct st_ops_args *args) return args->a; } +__bpf_kfunc_start_defs(); + +__bpf_kfunc int bpf_kfunc_st_ops_inc100(struct st_ops_args *args) +{ + args->a += 100; + return args->a; +} + +__bpf_kfunc_end_defs(); + BTF_KFUNCS_START(bpf_testmod_check_kfunc_ids) BTF_ID_FLAGS(func, bpf_testmod_test_mod_kfunc) BTF_ID_FLAGS(func, bpf_kfunc_call_test1) @@ -1140,6 +1150,10 @@ static int bpf_test_mod_st_ops__test_pro_epilogue(struct st_ops_args *args) return 0; } +BTF_ID_LIST(st_ops_epilogue_kfunc_list) +BTF_ID(func, bpf_kfunc_st_ops_inc10) +BTF_ID(func, bpf_kfunc_st_ops_inc100) + static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, const struct bpf_prog *prog, struct module **module) { @@ -1153,13 +1167,28 @@ static int st_ops_gen_prologue(struct bpf_insn *insn_buf, bool direct_write, * r7 = r6->a; * r7 += 1000; * r6->a = r7; + * r7 = r1; + * r1 = r6; + * bpf_kfunc_st_ops_in10(r1) + * r1 = r6; + * bpf_kfunc_st_ops_in100(r1) + * r1 = r7; */ *insn++ = BPF_LDX_MEM(BPF_DW, BPF_REG_6, BPF_REG_1, 0); *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_7, BPF_REG_6, offsetof(struct st_ops_args, a)); *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_7, 1000); *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_6, BPF_REG_7, offsetof(struct st_ops_args, a)); + *insn++ = BPF_MOV64_REG(BPF_REG_7, BPF_REG_1); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[0]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[1]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_7); *insn++ = prog->insnsi[0]; + *module = THIS_MODULE; return insn - insn_buf; } @@ -1177,7 +1206,10 @@ static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog * r6 = r1->a; * r6 += 10000; * r1->a = r6; - * r0 = r6; + * r6 = r1; + * bpf_kfunc_st_ops_in10(r1) + * r1 = r6; + * bpf_kfunc_st_ops_in100(r1) * r0 *= 2; * BPF_EXIT; */ @@ -1186,10 +1218,16 @@ static int st_ops_gen_epilogue(struct bpf_insn *insn_buf, const struct bpf_prog *insn++ = BPF_LDX_MEM(BPF_W, BPF_REG_6, BPF_REG_1, offsetof(struct st_ops_args, a)); *insn++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_6, 10000); *insn++ = BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_6, offsetof(struct st_ops_args, a)); - *insn++ = BPF_MOV32_REG(BPF_REG_0, BPF_REG_6); + *insn++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[0]); + *insn++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6); + *insn++ = BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, + st_ops_epilogue_kfunc_list[1]); *insn++ = BPF_ALU32_IMM(BPF_MUL, BPF_REG_0, 2); *insn++ = BPF_EXIT_INSN(); + *module = THIS_MODULE; return insn - insn_buf; } diff --git a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c index 98de677c55a9..db6e98096335 100644 --- a/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c +++ b/tools/testing/selftests/bpf/prog_tests/pro_epilogue.c @@ -34,8 +34,8 @@ static void test_tailcall(void) prog_fd = bpf_program__fd(skel->progs.syscall_epilogue_tailcall); err = bpf_prog_test_run_opts(prog_fd, &topts); ASSERT_OK(err, "bpf_prog_test_run_opts"); - ASSERT_EQ(args.a, 10001, "args.a"); - ASSERT_EQ(topts.retval, 10001 * 2, "topts.retval"); + ASSERT_EQ(args.a, 10111, "args.a"); + ASSERT_EQ(topts.retval, 10111 * 2, "topts.retval"); done: epilogue_tailcall__destroy(skel); diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c b/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c index 7d1124cf4942..2bd306f16610 100644 --- a/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c +++ b/tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c @@ -28,13 +28,19 @@ __xlated("0: r6 = *(u64 *)(r1 +0)") __xlated("1: r7 = *(u32 *)(r6 +0)") __xlated("2: w7 += 1000") __xlated("3: *(u32 *)(r6 +0) = r7") -/* main prog */ -__xlated("4: r1 = *(u64 *)(r1 +0)") -__xlated("5: r6 = r1") +__xlated("4: r7 = r1") +__xlated("5: r1 = r6") __xlated("6: call kernel-function") __xlated("7: r1 = r6") -__xlated("8: call pc+1") -__xlated("9: exit") +__xlated("8: call kernel-function") +__xlated("9: r1 = r7") +/* main prog */ +__xlated("10: r1 = *(u64 *)(r1 +0)") +__xlated("11: r6 = r1") +__xlated("12: call kernel-function") +__xlated("13: r1 = r6") +__xlated("14: call pc+1") +__xlated("15: exit") SEC("struct_ops/test_prologue_kfunc") __naked int test_prologue_kfunc(void) { @@ -65,9 +71,12 @@ __xlated("7: r1 = *(u64 *)(r1 +0)") __xlated("8: r6 = *(u32 *)(r1 +0)") __xlated("9: w6 += 10000") __xlated("10: *(u32 *)(r1 +0) = r6") -__xlated("11: w0 = w6") -__xlated("12: w0 *= 2") -__xlated("13: exit") +__xlated("11: r6 = r1") +__xlated("12: call kernel-function") +__xlated("13: r1 = r6") +__xlated("14: call kernel-function") +__xlated("15: w0 *= 2") +__xlated("16: exit") SEC("struct_ops/test_epilogue_kfunc") __naked int test_epilogue_kfunc(void) { @@ -89,23 +98,32 @@ __xlated("0: r6 = *(u64 *)(r1 +0)") __xlated("1: r7 = *(u32 *)(r6 +0)") __xlated("2: w7 += 1000") __xlated("3: *(u32 *)(r6 +0) = r7") +__xlated("4: r7 = r1") +__xlated("5: r1 = r6") +__xlated("6: call kernel-function") +__xlated("7: r1 = r6") +__xlated("8: call kernel-function") +__xlated("9: r1 = r7") /* save __u64 *ctx to stack */ -__xlated("4: *(u64 *)(r10 -8) = r1") +__xlated("10: *(u64 *)(r10 -8) = r1") /* main prog */ -__xlated("5: r1 = *(u64 *)(r1 +0)") -__xlated("6: r6 = r1") -__xlated("7: call kernel-function") -__xlated("8: r1 = r6") -__xlated("9: call pc+") -/* epilogue */ -__xlated("10: r1 = *(u64 *)(r10 -8)") __xlated("11: r1 = *(u64 *)(r1 +0)") -__xlated("12: r6 = *(u32 *)(r1 +0)") -__xlated("13: w6 += 10000") -__xlated("14: *(u32 *)(r1 +0) = r6") -__xlated("15: w0 = w6") -__xlated("16: w0 *= 2") -__xlated("17: exit") +__xlated("12: r6 = r1") +__xlated("13: call kernel-function") +__xlated("14: r1 = r6") +__xlated("15: call pc+") +/* epilogue */ +__xlated("16: r1 = *(u64 *)(r10 -8)") +__xlated("17: r1 = *(u64 *)(r1 +0)") +__xlated("18: r6 = *(u32 *)(r1 +0)") +__xlated("19: w6 += 10000") +__xlated("20: *(u32 *)(r1 +0) = r6") +__xlated("21: r6 = r1") +__xlated("22: call kernel-function") +__xlated("23: r1 = r6") +__xlated("24: call kernel-function") +__xlated("25: w0 *= 2") +__xlated("26: exit") SEC("struct_ops/test_pro_epilogue_kfunc") __naked int test_pro_epilogue_kfunc(void) { @@ -122,7 +140,7 @@ __naked int test_pro_epilogue_kfunc(void) } SEC("syscall") -__retval(1011) /* PROLOGUE_A [1000] + KFUNC_INC10 + SUBPROG_A [1] */ +__retval(1121) /* PROLOGUE_A [1110] + KFUNC_INC10 + SUBPROG_A [1] */ int syscall_prologue_kfunc(void *ctx) { struct st_ops_args args = {}; @@ -131,7 +149,7 @@ int syscall_prologue_kfunc(void *ctx) } SEC("syscall") -__retval(20022) /* (KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +__retval(20242) /* (KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10110]) * 2 */ int syscall_epilogue_kfunc(void *ctx) { struct st_ops_args args = {}; @@ -140,7 +158,7 @@ int syscall_epilogue_kfunc(void *ctx) } SEC("syscall") -__retval(22022) /* (PROLOGUE_A [1000] + KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +__retval(22462) /* (PROLOGUE_A [1110] + KFUNC_INC10 + SUBPROG_A [1] + EPILOGUE_A [10110]) * 2 */ int syscall_pro_epilogue_kfunc(void *ctx) { struct st_ops_args args = {}; diff --git a/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c b/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c index c91b1bf30e37..3d9cc25c024b 100644 --- a/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c +++ b/tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c @@ -21,10 +21,16 @@ __xlated("0: r6 = *(u64 *)(r1 +0)") __xlated("1: r7 = *(u32 *)(r6 +0)") __xlated("2: w7 += 1000") __xlated("3: *(u32 *)(r6 +0) = r7") +__xlated("4: r7 = r1") +__xlated("5: r1 = r6") +__xlated("6: call kernel-function") +__xlated("7: r1 = r6") +__xlated("8: call kernel-function") +__xlated("9: r1 = r7") /* main prog */ -__xlated("4: r1 = *(u64 *)(r1 +0)") -__xlated("5: call pc+1") -__xlated("6: exit") +__xlated("10: r1 = *(u64 *)(r1 +0)") +__xlated("11: call pc+1") +__xlated("12: exit") SEC("struct_ops/test_prologue_subprog") __naked int test_prologue_subprog(void) { @@ -47,9 +53,12 @@ __xlated("4: r1 = *(u64 *)(r1 +0)") __xlated("5: r6 = *(u32 *)(r1 +0)") __xlated("6: w6 += 10000") __xlated("7: *(u32 *)(r1 +0) = r6") -__xlated("8: w0 = w6") -__xlated("9: w0 *= 2") -__xlated("10: exit") +__xlated("8: r6 = r1") +__xlated("9: call kernel-function") +__xlated("10: r1 = r6") +__xlated("11: call kernel-function") +__xlated("12: w0 *= 2") +__xlated("13: exit") SEC("struct_ops/test_epilogue_subprog") __naked int test_epilogue_subprog(void) { @@ -66,20 +75,29 @@ __xlated("0: r6 = *(u64 *)(r1 +0)") __xlated("1: r7 = *(u32 *)(r6 +0)") __xlated("2: w7 += 1000") __xlated("3: *(u32 *)(r6 +0) = r7") +__xlated("4: r7 = r1") +__xlated("5: r1 = r6") +__xlated("6: call kernel-function") +__xlated("7: r1 = r6") +__xlated("8: call kernel-function") +__xlated("9: r1 = r7") /* save __u64 *ctx to stack */ -__xlated("4: *(u64 *)(r10 -8) = r1") +__xlated("10: *(u64 *)(r10 -8) = r1") /* main prog */ -__xlated("5: r1 = *(u64 *)(r1 +0)") -__xlated("6: call pc+") +__xlated("11: r1 = *(u64 *)(r1 +0)") +__xlated("12: call pc+") /* epilogue */ -__xlated("7: r1 = *(u64 *)(r10 -8)") -__xlated("8: r1 = *(u64 *)(r1 +0)") -__xlated("9: r6 = *(u32 *)(r1 +0)") -__xlated("10: w6 += 10000") -__xlated("11: *(u32 *)(r1 +0) = r6") -__xlated("12: w0 = w6") -__xlated("13: w0 *= 2") -__xlated("14: exit") +__xlated("13: r1 = *(u64 *)(r10 -8)") +__xlated("14: r1 = *(u64 *)(r1 +0)") +__xlated("15: r6 = *(u32 *)(r1 +0)") +__xlated("16: w6 += 10000") +__xlated("17: *(u32 *)(r1 +0) = r6") +__xlated("18: r6 = r1") +__xlated("19: call kernel-function") +__xlated("20: r1 = r6") +__xlated("21: call kernel-function") +__xlated("22: w0 *= 2") +__xlated("23: exit") SEC("struct_ops/test_pro_epilogue_subprog") __naked int test_pro_epilogue_subprog(void) { @@ -91,7 +109,7 @@ __naked int test_pro_epilogue_subprog(void) } SEC("syscall") -__retval(1001) /* PROLOGUE_A [1000] + SUBPROG_A [1] */ +__retval(1111) /* PROLOGUE_A [1110] + SUBPROG_A [1] */ int syscall_prologue_subprog(void *ctx) { struct st_ops_args args = {}; @@ -100,7 +118,7 @@ int syscall_prologue_subprog(void *ctx) } SEC("syscall") -__retval(20002) /* (SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +__retval(20222) /* (SUBPROG_A [1] + EPILOGUE_A [10110]) * 2 */ int syscall_epilogue_subprog(void *ctx) { struct st_ops_args args = {}; @@ -109,7 +127,7 @@ int syscall_epilogue_subprog(void *ctx) } SEC("syscall") -__retval(22002) /* (PROLOGUE_A [1000] + SUBPROG_A [1] + EPILOGUE_A [10000]) * 2 */ +__retval(22442) /* (PROLOGUE_A [1110] + SUBPROG_A [1] + EPILOGUE_A [10110]) * 2 */ int syscall_pro_epilogue_subprog(void *ctx) { struct st_ops_args args = {};