From patchwork Tue Oct 17 16:22:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425606 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 872DB45F7C for ; Tue, 17 Oct 2023 16:23:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="loqqkKid" Received: from mail-yb1-xb30.google.com (mail-yb1-xb30.google.com [IPv6:2607:f8b0:4864:20::b30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D2C4FA for ; Tue, 17 Oct 2023 09:23:12 -0700 (PDT) Received: by mail-yb1-xb30.google.com with SMTP id 3f1490d57ef6-d9a4c0d89f7so6887074276.1 for ; Tue, 17 Oct 2023 09:23:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559791; x=1698164591; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=P89SVJtq5j8zsLeBIQYzZ3INapX8tpzdjvC7+Jnea5w=; b=loqqkKidr/Fn1YM1Ah78saqWBpnd1DhbI6gG3LB9ORExltUZFyqMPL3SvzA44mnnoD rDOm3d1DZlxgyfIeyrrxhGKFKWPyiEhNd/d/3dcWay/trRvrXoT7RupR2UgJCSJL1Nie efVHeLP4o7LeYoVWnMXgsWYEv2hJkc6WjIGWanxDZor+EJV1QTxUogXc6yDjI/8eKyfG JUPWc3uVoEfNVnnWn/hrIJEnSU79Hgp6g+j8kG51g2qwusF/5YS/5sti+kuquIAR4sKt IE4kJ8Nh1RTv6gtEh1Kk1GFnuoH4gaFkwgGa+fCBgJc4zOHRSxONjDKlg+YZouKmx0Ze VePw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559791; x=1698164591; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P89SVJtq5j8zsLeBIQYzZ3INapX8tpzdjvC7+Jnea5w=; b=uYt1deQdgGb9YJ1sYwgYWjRquH+aYrGRX+T6Ei3QfBMjzVVYfrZp59yhBuQU31hDQs WdhEvjgFHjvMPYyqeTwIhX2FDn8tuf0KkItDSHxx/S70BVIMBxCf1BG4b7gWQTUfwDbu fe3fVtdy+5y5yiy9JRQxjsPTGGkpmFwPOu30yxkls2il3H+khg/fPlJeH8Qj+ol0VVRW PxniYi/uM8s8Fyi4k5XEZLuwFzVE6bbnYCxdAZojsPgBNjNECzK/Y/0OOPjZav+LGfy2 qb6k21qljN5iUcNbVpSSGNdOIExMp0YIeWLLxis78K9d4mXPM1WKqL693MEhOMj/+BZJ GB5w== X-Gm-Message-State: AOJu0YzM885C1YoIKCJUnddGmkMVWbaUVBACuNxboJGCy5Rq+ST/UbWi b394D8ajDTcmWCzKl47MshIaNVdcCxg= X-Google-Smtp-Source: AGHT+IE4cW/FMMTrVLg4DTwpa+IedrdfF5BygqALSZIpUSEdQZMVn2rfvyPvOx8wIoCORjlhB2Ua2A== X-Received: by 2002:a25:9183:0:b0:d9a:ccdf:3873 with SMTP id w3-20020a259183000000b00d9accdf3873mr2438511ybl.48.1697559791244; Tue, 17 Oct 2023 09:23:11 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:10 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 1/9] bpf: refactory struct_ops type initialization to a function. Date: Tue, 17 Oct 2023 09:22:58 -0700 Message-Id: <20231017162306.176586-2-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Move the majority of the code to bpf_struct_ops_init_one(), which can then be utilized for the initialization of newly registered dynamically allocated struct_ops types in the following patches. Signed-off-by: Kui-Feng Lee --- kernel/bpf/bpf_struct_ops.c | 157 +++++++++++++++++++----------------- 1 file changed, 83 insertions(+), 74 deletions(-) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index db6176fb64dc..627cf1ea840a 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -110,102 +110,111 @@ const struct bpf_prog_ops bpf_struct_ops_prog_ops = { static const struct btf_type *module_type; -void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) +static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, + struct btf *btf, + struct bpf_verifier_log *log) { - s32 type_id, value_id, module_id; const struct btf_member *member; - struct bpf_struct_ops *st_ops; const struct btf_type *t; + s32 type_id, value_id; char value_name[128]; const char *mname; - u32 i, j; + int i; - /* Ensure BTF type is emitted for "struct bpf_struct_ops_##_name" */ -#define BPF_STRUCT_OPS_TYPE(_name) BTF_TYPE_EMIT(struct bpf_struct_ops_##_name); -#include "bpf_struct_ops_types.h" -#undef BPF_STRUCT_OPS_TYPE + if (strlen(st_ops->name) + VALUE_PREFIX_LEN >= + sizeof(value_name)) { + pr_warn("struct_ops name %s is too long\n", + st_ops->name); + return; + } + sprintf(value_name, "%s%s", VALUE_PREFIX, st_ops->name); - module_id = btf_find_by_name_kind(btf, "module", BTF_KIND_STRUCT); - if (module_id < 0) { - pr_warn("Cannot find struct module in btf_vmlinux\n"); + value_id = btf_find_by_name_kind(btf, value_name, + BTF_KIND_STRUCT); + if (value_id < 0) { + pr_warn("Cannot find struct %s in btf_vmlinux\n", + value_name); return; } - module_type = btf_type_by_id(btf, module_id); - for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { - st_ops = bpf_struct_ops[i]; + type_id = btf_find_by_name_kind(btf, st_ops->name, + BTF_KIND_STRUCT); + if (type_id < 0) { + pr_warn("Cannot find struct %s in btf_vmlinux\n", + st_ops->name); + return; + } + t = btf_type_by_id(btf, type_id); + if (btf_type_vlen(t) > BPF_STRUCT_OPS_MAX_NR_MEMBERS) { + pr_warn("Cannot support #%u members in struct %s\n", + btf_type_vlen(t), st_ops->name); + return; + } - if (strlen(st_ops->name) + VALUE_PREFIX_LEN >= - sizeof(value_name)) { - pr_warn("struct_ops name %s is too long\n", + for_each_member(i, t, member) { + const struct btf_type *func_proto; + + mname = btf_name_by_offset(btf, member->name_off); + if (!*mname) { + pr_warn("anon member in struct %s is not supported\n", st_ops->name); - continue; + break; } - sprintf(value_name, "%s%s", VALUE_PREFIX, st_ops->name); - value_id = btf_find_by_name_kind(btf, value_name, - BTF_KIND_STRUCT); - if (value_id < 0) { - pr_warn("Cannot find struct %s in btf_vmlinux\n", - value_name); - continue; + if (__btf_member_bitfield_size(t, member)) { + pr_warn("bit field member %s in struct %s is not supported\n", + mname, st_ops->name); + break; } - type_id = btf_find_by_name_kind(btf, st_ops->name, - BTF_KIND_STRUCT); - if (type_id < 0) { - pr_warn("Cannot find struct %s in btf_vmlinux\n", - st_ops->name); - continue; - } - t = btf_type_by_id(btf, type_id); - if (btf_type_vlen(t) > BPF_STRUCT_OPS_MAX_NR_MEMBERS) { - pr_warn("Cannot support #%u members in struct %s\n", - btf_type_vlen(t), st_ops->name); - continue; + func_proto = btf_type_resolve_func_ptr(btf, + member->type, + NULL); + if (func_proto && + btf_distill_func_proto(log, btf, + func_proto, mname, + &st_ops->func_models[i])) { + pr_warn("Error in parsing func ptr %s in struct %s\n", + mname, st_ops->name); + break; } + } - for_each_member(j, t, member) { - const struct btf_type *func_proto; + if (i == btf_type_vlen(t)) { + if (st_ops->init(btf)) { + pr_warn("Error in init bpf_struct_ops %s\n", + st_ops->name); + } else { + st_ops->type_id = type_id; + st_ops->type = t; + st_ops->value_id = value_id; + st_ops->value_type = btf_type_by_id(btf, + value_id); + } + } +} - mname = btf_name_by_offset(btf, member->name_off); - if (!*mname) { - pr_warn("anon member in struct %s is not supported\n", - st_ops->name); - break; - } +void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) +{ + struct bpf_struct_ops *st_ops; + s32 module_id; + u32 i; - if (__btf_member_bitfield_size(t, member)) { - pr_warn("bit field member %s in struct %s is not supported\n", - mname, st_ops->name); - break; - } + /* Ensure BTF type is emitted for "struct bpf_struct_ops_##_name" */ +#define BPF_STRUCT_OPS_TYPE(_name) BTF_TYPE_EMIT(struct bpf_struct_ops_##_name); +#include "bpf_struct_ops_types.h" +#undef BPF_STRUCT_OPS_TYPE - func_proto = btf_type_resolve_func_ptr(btf, - member->type, - NULL); - if (func_proto && - btf_distill_func_proto(log, btf, - func_proto, mname, - &st_ops->func_models[j])) { - pr_warn("Error in parsing func ptr %s in struct %s\n", - mname, st_ops->name); - break; - } - } + module_id = btf_find_by_name_kind(btf, "module", BTF_KIND_STRUCT); + if (module_id < 0) { + pr_warn("Cannot find struct module in btf_vmlinux\n"); + return; + } + module_type = btf_type_by_id(btf, module_id); - if (j == btf_type_vlen(t)) { - if (st_ops->init(btf)) { - pr_warn("Error in init bpf_struct_ops %s\n", - st_ops->name); - } else { - st_ops->type_id = type_id; - st_ops->type = t; - st_ops->value_id = value_id; - st_ops->value_type = btf_type_by_id(btf, - value_id); - } - } + for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { + st_ops = bpf_struct_ops[i]; + bpf_struct_ops_init_one(st_ops, btf, log); } } From patchwork Tue Oct 17 16:22:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425607 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43F674734E for ; Tue, 17 Oct 2023 16:23:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NOBKbfNO" Received: from mail-yb1-xb31.google.com (mail-yb1-xb31.google.com [IPv6:2607:f8b0:4864:20::b31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82132B0 for ; Tue, 17 Oct 2023 09:23:13 -0700 (PDT) Received: by mail-yb1-xb31.google.com with SMTP id 3f1490d57ef6-d9a4c0d89f7so6887106276.1 for ; Tue, 17 Oct 2023 09:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559792; x=1698164592; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2IFN/2HC9JygClV/G6CWtisNQ7HtXVVuY/UWh7b7UCo=; b=NOBKbfNOsskjHRhZYGfveQBY/ePY5NlfafVFcZNFWr+n5GGAP/FTjTAbiUZ9gxG/pl tBXUqpGKA65VSfAklOZEif/Neh07ZXv1cGWxfnLxpoE+xTCfPWA2iyBp+WHy02/M6VJr qSTA9vvpY3NYD6F0oxSAPa5OA/QkvRKDMU/sJ04ubojnHf6Ux0LdzVgTrEGi0QLnfVA0 J3IXBd51kzS3YEHmz8rpbi/C+Xratb0XLMLZn4gFISimW0kYENLYTxRRoWI1a36lFHvS xONWBDuo6D6tN74HN+9YA24S15pa2R7KGt+Msn+tcaQ0defNxBD1Z/wgtWoitT1To5Q3 2HkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559792; x=1698164592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2IFN/2HC9JygClV/G6CWtisNQ7HtXVVuY/UWh7b7UCo=; b=Yj+xgVXAcAIDnnveop4lGz53uDUG65BbAwKv8f64ozt2tExYQ5xTCp6EhfukT8tMAq OMwHoEeR6mgHCILidU2CKqCoMM5wZaoDKsi/OoQ5/NCVyOlrBYzh+mlOemnd64jvAnC0 X6O0Jw39MvCT1g/AkVFt9XRP/TF/C7HZIsXWChtR1JQtEQ5iFxGP1H/WWeT1aIdAIUG+ ceQDdCEML8+jCbALukWLsh4M9s+bZgCJbRg0u19V9okPtxEt+AGD3NL0LyHPJOC/e+uk f9cLRtLUwYJpOPIYQxKGKm7XnJ8+6Wy5KhqMyAHWoG6TRaphgaGXliK3//xHekxdGDzZ Tayg== X-Gm-Message-State: AOJu0Yx+eqTP0qMAIWO77UvsDZoHyQ8v2hIbztQJRx9dl6Ioc0OjoJZs 4c00MeF8Q1Gprg0/WhgNGXeLSGzpmjo= X-Google-Smtp-Source: AGHT+IGj6DNsckG6XRRw1wnFZ8yXeasydHAZlismm6IcJDt1CaGMgBzud8g6YGBz+fVBZI+3cPpYNA== X-Received: by 2002:a25:d250:0:b0:d9a:4839:68fe with SMTP id j77-20020a25d250000000b00d9a483968femr2614674ybg.43.1697559792377; Tue, 17 Oct 2023 09:23:12 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:12 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 2/9] bpf: add struct_ops_tab to btf. Date: Tue, 17 Oct 2023 09:22:59 -0700 Message-Id: <20231017162306.176586-3-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Maintain a registry of registered struct_ops types in the per-btf (module) struct_ops_tab. This registry allows for easy lookup of struct_ops types that are registered by a specific module. Every struct_ops type should have an associated module BTF to provide type information since we are going to allow modules to define and register new struct_ops types. Once this change is made, the bpf_struct_ops subsystem knows where to look up type info with just a bpf_struct_ops. The subsystem looks up struct_ops types from a given module BTF although it is always btf_vmlinux now. Once start using struct_ops_tab, btfs other than btf_vmlinux can be used as well. Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 5 +-- include/linux/btf.h | 6 ++++ kernel/bpf/bpf_struct_ops.c | 17 ++++----- kernel/bpf/btf.c | 70 +++++++++++++++++++++++++++++++++++++ kernel/bpf/verifier.c | 2 +- 5 files changed, 89 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 30063a760b5a..e6a648af2daa 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1626,6 +1626,7 @@ struct bpf_struct_ops { void (*unreg)(void *kdata); int (*update)(void *kdata, void *old_kdata); int (*validate)(void *kdata); + struct btf *btf; const struct btf_type *type; const struct btf_type *value_type; const char *name; @@ -1636,7 +1637,7 @@ struct bpf_struct_ops { #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL) #define BPF_MODULE_OWNER ((void *)((0xeB9FUL << 2) + POISON_POINTER_DELTA)) -const struct bpf_struct_ops *bpf_struct_ops_find(u32 type_id); +const struct bpf_struct_ops *bpf_struct_ops_find(struct btf *btf, u32 type_id); void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log); bool bpf_struct_ops_get(const void *kdata); void bpf_struct_ops_put(const void *kdata); @@ -1679,7 +1680,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr); #endif #else -static inline const struct bpf_struct_ops *bpf_struct_ops_find(u32 type_id) +static inline const struct bpf_struct_ops *bpf_struct_ops_find(struct btf *btf, u32 type_id) { return NULL; } diff --git a/include/linux/btf.h b/include/linux/btf.h index 928113a80a95..aa2ba77648be 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -571,4 +571,10 @@ static inline bool btf_type_is_struct_ptr(struct btf *btf, const struct btf_type return btf_type_is_struct(t); } +struct bpf_struct_ops; + +int btf_add_struct_ops(struct bpf_struct_ops *st_ops); +const struct bpf_struct_ops ** +btf_get_struct_ops(struct btf *btf, u32 *ret_cnt); + #endif diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 627cf1ea840a..7758f66ad734 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -185,6 +185,7 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, pr_warn("Error in init bpf_struct_ops %s\n", st_ops->name); } else { + st_ops->btf = btf; st_ops->type_id = type_id; st_ops->type = t; st_ops->value_id = value_id; @@ -221,7 +222,7 @@ void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) extern struct btf *btf_vmlinux; static const struct bpf_struct_ops * -bpf_struct_ops_find_value(u32 value_id) +bpf_struct_ops_find_value(struct btf *btf, u32 value_id) { unsigned int i; @@ -236,7 +237,7 @@ bpf_struct_ops_find_value(u32 value_id) return NULL; } -const struct bpf_struct_ops *bpf_struct_ops_find(u32 type_id) +const struct bpf_struct_ops *bpf_struct_ops_find(struct btf *btf, u32 type_id) { unsigned int i; @@ -316,7 +317,7 @@ static void bpf_struct_ops_map_put_progs(struct bpf_struct_ops_map *st_map) } } -static int check_zero_holes(const struct btf_type *t, void *data) +static int check_zero_holes(const struct btf *btf, const struct btf_type *t, void *data) { const struct btf_member *member; u32 i, moff, msize, prev_mend = 0; @@ -328,8 +329,8 @@ static int check_zero_holes(const struct btf_type *t, void *data) memchr_inv(data + prev_mend, 0, moff - prev_mend)) return -EINVAL; - mtype = btf_type_by_id(btf_vmlinux, member->type); - mtype = btf_resolve_size(btf_vmlinux, mtype, &msize); + mtype = btf_type_by_id(btf, member->type); + mtype = btf_resolve_size(btf, mtype, &msize); if (IS_ERR(mtype)) return PTR_ERR(mtype); prev_mend = moff + msize; @@ -395,12 +396,12 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, if (*(u32 *)key != 0) return -E2BIG; - err = check_zero_holes(st_ops->value_type, value); + err = check_zero_holes(st_ops->btf, st_ops->value_type, value); if (err) return err; uvalue = value; - err = check_zero_holes(t, uvalue->data); + err = check_zero_holes(st_ops->btf, t, uvalue->data); if (err) return err; @@ -671,7 +672,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) struct bpf_map *map; int ret; - st_ops = bpf_struct_ops_find_value(attr->btf_vmlinux_value_type_id); + st_ops = bpf_struct_ops_find_value(attr->btf_vmlinux_value_type_id, btf_vmlinux); if (!st_ops) return ERR_PTR(-ENOTSUPP); diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index f93e835d90af..be5144dbb53d 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -241,6 +241,12 @@ struct btf_id_dtor_kfunc_tab { struct btf_id_dtor_kfunc dtors[]; }; +struct btf_struct_ops_tab { + u32 cnt; + u32 capacity; + struct bpf_struct_ops *ops[]; +}; + struct btf { void *data; struct btf_type **types; @@ -258,6 +264,7 @@ struct btf { struct btf_kfunc_set_tab *kfunc_set_tab; struct btf_id_dtor_kfunc_tab *dtor_kfunc_tab; struct btf_struct_metas *struct_meta_tab; + struct btf_struct_ops_tab *struct_ops_tab; /* split BTF support */ struct btf *base_btf; @@ -1688,11 +1695,20 @@ static void btf_free_struct_meta_tab(struct btf *btf) btf->struct_meta_tab = NULL; } +static void btf_free_struct_ops_tab(struct btf *btf) +{ + struct btf_struct_ops_tab *tab = btf->struct_ops_tab; + + kfree(tab); + btf->struct_ops_tab = NULL; +} + static void btf_free(struct btf *btf) { btf_free_struct_meta_tab(btf); btf_free_dtor_kfunc_tab(btf); btf_free_kfunc_set_tab(btf); + btf_free_struct_ops_tab(btf); kvfree(btf->types); kvfree(btf->resolved_sizes); kvfree(btf->resolved_ids); @@ -8601,3 +8617,57 @@ bool btf_type_ids_nocast_alias(struct bpf_verifier_log *log, return !strncmp(reg_name, arg_name, cmp_len); } + +int btf_add_struct_ops(struct bpf_struct_ops *st_ops) +{ + struct btf_struct_ops_tab *tab, *new_tab; + struct btf *btf = st_ops->btf; + int i; + + if (!btf) + return -ENOENT; + + /* Assume this function is called for a module when the module is + * loading. + */ + + tab = btf->struct_ops_tab; + if (!tab) { + tab = kzalloc(offsetof(struct btf_struct_ops_tab, ops[4]), + GFP_KERNEL); + if (!tab) + return -ENOMEM; + tab->capacity = 4; + btf->struct_ops_tab = tab; + } + + for (i = 0; i < tab->cnt; i++) + if (tab->ops[i] == st_ops) + return -EEXIST; + + if (tab->cnt == tab->capacity) { + new_tab = krealloc(tab, sizeof(*tab) + + sizeof(struct bpf_struct_ops *) * + tab->capacity * 2, GFP_KERNEL); + if (!new_tab) + return -ENOMEM; + tab = new_tab; + tab->capacity *= 2; + btf->struct_ops_tab = tab; + } + + btf->struct_ops_tab->ops[btf->struct_ops_tab->cnt++] = st_ops; + + return 0; +} + +const struct bpf_struct_ops **btf_get_struct_ops(struct btf *btf, u32 *ret_cnt) +{ + if (!btf) + return NULL; + if (!btf->struct_ops_tab) + return NULL; + + *ret_cnt = btf->struct_ops_tab->cnt; + return (const struct bpf_struct_ops **)btf->struct_ops_tab->ops; +} diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a7178ecf676d..6564a03c425d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19631,7 +19631,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) } btf_id = prog->aux->attach_btf_id; - st_ops = bpf_struct_ops_find(btf_id); + st_ops = bpf_struct_ops_find(btf_vmlinux, btf_id); if (!st_ops) { verbose(env, "attach_btf_id %u is not a supported struct\n", btf_id); From patchwork Tue Oct 17 16:23:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425608 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E40B3B2B1 for ; Tue, 17 Oct 2023 16:23:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Nm1QzRvA" Received: from mail-yw1-x112a.google.com (mail-yw1-x112a.google.com [IPv6:2607:f8b0:4864:20::112a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0733F0 for ; Tue, 17 Oct 2023 09:23:14 -0700 (PDT) Received: by mail-yw1-x112a.google.com with SMTP id 00721157ae682-5a7afd45199so74400047b3.0 for ; Tue, 17 Oct 2023 09:23:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559793; x=1698164593; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CJG31ISWvKTgNfJuTjFh/5nAbkc59OAsCWWkmbBCCxY=; b=Nm1QzRvAo6HWgXQ6/rwG7qW5ZpPa/wMCZUh9V+hFx/E07C7lfdY2YmLW94kjYr6soC 6UiLK8mLEVt7hMBJkxd0qmV5ryu2mf6OmclI39efgvG8oWcSQdBd60En+72sZmb5+dM5 0dRhBNRBAxWdUFMrO9qydoeqjkRqMfW3way1FkXfkovPzGo/Apjr9tPSGEMVjr+X8iI+ CGN3Nj9c1vZDeBN8MdQKYXQmSK6CX8uv3yCo0xdTDufF++j8UjxMhd0LP3SF4lAlHlUN 0nC30lV/M9081iFi9//UsrC4dvvLth5IMesXRM50KT/gBDjI2wIIZ+Cy+OSr34xlvJUN DTqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559793; x=1698164593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CJG31ISWvKTgNfJuTjFh/5nAbkc59OAsCWWkmbBCCxY=; b=TjVjXpqgtxzsJYXIMyMs2i34CgAU+RRDUizOV80om4kZ65i7H/GEDyMvazc/FFcomg tEzGrUCngE9XKWaoeNTq+Q4RzzhfROompwXGSiJTA36tRHHwGnjo73q3FYz6ZBPbuc7w 8wwfKL/0+CnXVRZa4yXjUQHUgfR50U5ULTVn3u19oghgpsupMn0q8SD7Cgi6a0iCIo+N MNJpQRvYYBZ7l+nqyfwho5GKTZB62Rwf/r17Brr0GcTfAXNNuobcXkAkSX9Os/yn/+lP TdwsQSDaZb0h03KzgVIuztbGA/NIMbeLDEttgZLxraHZsqcXdcQMm+evuj4J6auJ1uRx T1BA== X-Gm-Message-State: AOJu0YwL27Fa1dZqhjCqg+zanAjwtqEoZ9BPiHztWXGWImaKTMkNuL0s Y9rumcSPYAJeCs6kYR80a4Lw2q/5W1k= X-Google-Smtp-Source: AGHT+IGlwgaAxFpPzUqhV0rnRtqXzD0Kp4GuKCmaWfrEpjI+tQ8FVRRbJyGaSh0tmm15J3hFjLwQPQ== X-Received: by 2002:a25:abac:0:b0:d9a:3ed1:e4ad with SMTP id v41-20020a25abac000000b00d9a3ed1e4admr2899215ybi.41.1697559793553; Tue, 17 Oct 2023 09:23:13 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:13 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 3/9] bpf: hold module for bpf_struct_ops_map. Date: Tue, 17 Oct 2023 09:23:00 -0700 Message-Id: <20231017162306.176586-4-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee To ensure that a module remains accessible whenever a struct_ops object of a struct_ops type provided by the module is still in use. Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 1 + kernel/bpf/bpf_struct_ops.c | 21 ++++++++++++++++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e6a648af2daa..1e1647c8b0ce 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1627,6 +1627,7 @@ struct bpf_struct_ops { int (*update)(void *kdata, void *old_kdata); int (*validate)(void *kdata); struct btf *btf; + struct module *owner; const struct btf_type *type; const struct btf_type *value_type; const char *name; diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 7758f66ad734..b561245fe235 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -112,6 +112,7 @@ static const struct btf_type *module_type; static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, struct btf *btf, + struct module *owner, struct bpf_verifier_log *log) { const struct btf_member *member; @@ -186,6 +187,7 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, st_ops->name); } else { st_ops->btf = btf; + st_ops->owner = owner; st_ops->type_id = type_id; st_ops->type = t; st_ops->value_id = value_id; @@ -193,6 +195,7 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, value_id); } } + } void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) @@ -215,7 +218,7 @@ void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { st_ops = bpf_struct_ops[i]; - bpf_struct_ops_init_one(st_ops, btf, log); + bpf_struct_ops_init_one(st_ops, btf, NULL, log); } } @@ -630,6 +633,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map) bpf_jit_uncharge_modmem(PAGE_SIZE); } bpf_map_area_free(st_map->uvalue); + module_put(st_map->st_ops->owner); bpf_map_area_free(st_map); } @@ -676,9 +680,18 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) if (!st_ops) return ERR_PTR(-ENOTSUPP); + /* If st_ops->owner is NULL, it means the struct_ops is + * statically defined in the kernel. We don't need to + * take a refcount on it. + */ + if (st_ops->owner && !btf_try_get_module(st_ops->btf)) + return ERR_PTR(-EINVAL); + vt = st_ops->value_type; - if (attr->value_size != vt->size) + if (attr->value_size != vt->size) { + module_put(st_ops->owner); return ERR_PTR(-EINVAL); + } t = st_ops->type; @@ -689,8 +702,10 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) (vt->size - sizeof(struct bpf_struct_ops_value)); st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE); - if (!st_map) + if (!st_map) { + module_put(st_ops->owner); return ERR_PTR(-ENOMEM); + } st_map->st_ops = st_ops; map = &st_map->map; From patchwork Tue Oct 17 16:23:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425609 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36D6B4735D for ; Tue, 17 Oct 2023 16:23:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UppGAfeW" Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D947A9E for ; Tue, 17 Oct 2023 09:23:15 -0700 (PDT) Received: by mail-yb1-xb34.google.com with SMTP id 3f1490d57ef6-d9beb865a40so2464072276.1 for ; Tue, 17 Oct 2023 09:23:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559795; x=1698164595; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uUcW83e55Bna8gBEv5q3MMNy5IQNTEE5zYJyBPrfHXs=; b=UppGAfeW6FNpy0f4f7umIzMm9RRNRurAxfmYr8mv+Tu1+1TcyjRzVNcMcbSk9hfNnX AV5DoEheKHnR8nD9I7wPufRSjohck3TxhyzXUHar1Wkj85/G5FuPSkUX9wmP8RGZR3U5 xuwyhfjVDHlJdScy1pZ2QBjYLpLLECFug3WwwTa0fl+Z8XO9SwT8v6JBQ46QbbuZKXar I7ga0b9+scp9lYdQMalsc1VQOr/SZXzxRDbV/AUNaGknLIJmC4JQh+Si6Oos+yZwM4NR 6iei4iVQJjITUSpcOQ2epnGguGeACvyCVp9j2bIC9etIytlmlQAz2RMjMvGrlVE1GXkz PwxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559795; x=1698164595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uUcW83e55Bna8gBEv5q3MMNy5IQNTEE5zYJyBPrfHXs=; b=BFNbxY7J38ThtLvuyDUOpg1xe3YirPqdrxJl7oqFfr8KEYZmccwb3X1ZLTlkuDXJaq EWJaImRm7rslRwKunLxC+le7MOaklsBsgHAZ6+FdzziI2Lws10iz1wV2Gye71AkgJuFm aL4HSpYtCvIqvKcPQXBRs/0B48Oge7mSqxjOtarSbKvPaebjMsOfoMANjlUBQSQ/ruR7 mUGReLTS7+JXbir2PKPo3iNm5W5cerrWUaN31Ks76oCFD/ipAN7CJtphTEsaSDcsy8S/ FHXeWrHIxnCmKV5+YVsRLJe95Pt4TcYS5ruWYOTh95YorqRBYgI33B0eF4IbbNgpT2Cw hsZQ== X-Gm-Message-State: AOJu0YxLQ1uk4QUU27h/0u4VVoexa52MxClMAVWOJhBeDgRpgcOKv8xl 06OspeQLF5/I3PYlBEQngveTP+GUuSk= X-Google-Smtp-Source: AGHT+IFa10qHzncqL3x4/ybcP9EWJ4Z/YH4iVwSWsKyvq6ECWpI95c0WYfTv4ReToAQWSvEuFhlvmQ== X-Received: by 2002:a25:b214:0:b0:d9a:63f5:10d6 with SMTP id i20-20020a25b214000000b00d9a63f510d6mr3181068ybj.26.1697559794703; Tue, 17 Oct 2023 09:23:14 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:14 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 4/9] bpf: validate value_type Date: Tue, 17 Oct 2023 09:23:01 -0700 Message-Id: <20231017162306.176586-5-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee A value_type should consist of three components: refcnt, state, and data. refcnt and state has been move to struct bpf_struct_ops_common_value to make it easier to check the value type. Signed-off-by: Kui-Feng Lee --- kernel/bpf/bpf_struct_ops.c | 88 +++++++++++++++++++++++++++---------- 1 file changed, 66 insertions(+), 22 deletions(-) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index b561245fe235..69703584fa4a 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -20,9 +20,11 @@ enum bpf_struct_ops_state { BPF_STRUCT_OPS_STATE_READY, }; -#define BPF_STRUCT_OPS_COMMON_VALUE \ - refcount_t refcnt; \ - enum bpf_struct_ops_state state +struct bpf_struct_ops_common_value { + refcount_t refcnt; + enum bpf_struct_ops_state state; +}; +#define BPF_STRUCT_OPS_COMMON_VALUE struct bpf_struct_ops_common_value common struct bpf_struct_ops_value { BPF_STRUCT_OPS_COMMON_VALUE; @@ -109,6 +111,38 @@ const struct bpf_prog_ops bpf_struct_ops_prog_ops = { }; static const struct btf_type *module_type; +static const struct btf_type *common_value_type; + +static bool is_valid_value_type(struct btf *btf, s32 value_id, + const struct btf_type *type, + const char *value_name) +{ + const struct btf_member *member; + const struct btf_type *vt, *mt; + + vt = btf_type_by_id(btf, value_id); + if (btf_vlen(vt) != 2) { + pr_warn("The number of %s's members should be 2, but we get %d\n", + value_name, btf_vlen(vt)); + return false; + } + member = btf_type_member(vt); + mt = btf_type_by_id(btf, member->type); + if (mt != common_value_type) { + pr_warn("The first member of %s should be bpf_struct_ops_common_value\n", + value_name); + return false; + } + member++; + mt = btf_type_by_id(btf, member->type); + if (mt != type) { + pr_warn("The second member of %s should be %s\n", + value_name, btf_name_by_offset(btf, type->name_off)); + return false; + } + + return true; +} static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, struct btf *btf, @@ -130,14 +164,6 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, } sprintf(value_name, "%s%s", VALUE_PREFIX, st_ops->name); - value_id = btf_find_by_name_kind(btf, value_name, - BTF_KIND_STRUCT); - if (value_id < 0) { - pr_warn("Cannot find struct %s in btf_vmlinux\n", - value_name); - return; - } - type_id = btf_find_by_name_kind(btf, st_ops->name, BTF_KIND_STRUCT); if (type_id < 0) { @@ -152,6 +178,16 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, return; } + value_id = btf_find_by_name_kind(btf, value_name, + BTF_KIND_STRUCT); + if (value_id < 0) { + pr_warn("Cannot find struct %s in btf_vmlinux\n", + value_name); + return; + } + if (!is_valid_value_type(btf, value_id, t, value_name)) + return; + for_each_member(i, t, member) { const struct btf_type *func_proto; @@ -201,7 +237,7 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) { struct bpf_struct_ops *st_ops; - s32 module_id; + s32 module_id, common_value_id; u32 i; /* Ensure BTF type is emitted for "struct bpf_struct_ops_##_name" */ @@ -215,6 +251,14 @@ void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) return; } module_type = btf_type_by_id(btf, module_id); + common_value_id = btf_find_by_name_kind(btf, + "bpf_struct_ops_common_value", + BTF_KIND_STRUCT); + if (common_value_id < 0) { + pr_warn("Cannot find struct common_value in btf_vmlinux\n"); + return; + } + common_value_type = btf_type_by_id(btf, common_value_id); for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { st_ops = bpf_struct_ops[i]; @@ -278,7 +322,7 @@ int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, kvalue = &st_map->kvalue; /* Pair with smp_store_release() during map_update */ - state = smp_load_acquire(&kvalue->state); + state = smp_load_acquire(&kvalue->common.state); if (state == BPF_STRUCT_OPS_STATE_INIT) { memset(value, 0, map->value_size); return 0; @@ -289,7 +333,7 @@ int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, */ uvalue = value; memcpy(uvalue, st_map->uvalue, map->value_size); - uvalue->state = state; + uvalue->common.state = state; /* This value offers the user space a general estimate of how * many sockets are still utilizing this struct_ops for TCP @@ -297,7 +341,7 @@ int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, * should sufficiently meet our present goals. */ refcnt = atomic64_read(&map->refcnt) - atomic64_read(&map->usercnt); - refcount_set(&uvalue->refcnt, max_t(s64, refcnt, 0)); + refcount_set(&uvalue->common.refcnt, max_t(s64, refcnt, 0)); return 0; } @@ -408,7 +452,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, if (err) return err; - if (uvalue->state || refcount_read(&uvalue->refcnt)) + if (uvalue->common.state || refcount_read(&uvalue->common.refcnt)) return -EINVAL; tlinks = kcalloc(BPF_TRAMP_MAX, sizeof(*tlinks), GFP_KERNEL); @@ -420,7 +464,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, mutex_lock(&st_map->lock); - if (kvalue->state != BPF_STRUCT_OPS_STATE_INIT) { + if (kvalue->common.state != BPF_STRUCT_OPS_STATE_INIT) { err = -EBUSY; goto unlock; } @@ -533,7 +577,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, * * Pair with smp_load_acquire() during lookup_elem(). */ - smp_store_release(&kvalue->state, BPF_STRUCT_OPS_STATE_READY); + smp_store_release(&kvalue->common.state, BPF_STRUCT_OPS_STATE_READY); goto unlock; } @@ -551,7 +595,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, * It ensures the above udata updates (e.g. prog->aux->id) * can be seen once BPF_STRUCT_OPS_STATE_INUSE is set. */ - smp_store_release(&kvalue->state, BPF_STRUCT_OPS_STATE_INUSE); + smp_store_release(&kvalue->common.state, BPF_STRUCT_OPS_STATE_INUSE); goto unlock; } @@ -582,7 +626,7 @@ static long bpf_struct_ops_map_delete_elem(struct bpf_map *map, void *key) if (st_map->map.map_flags & BPF_F_LINK) return -EOPNOTSUPP; - prev_state = cmpxchg(&st_map->kvalue.state, + prev_state = cmpxchg(&st_map->kvalue.common.state, BPF_STRUCT_OPS_STATE_INUSE, BPF_STRUCT_OPS_STATE_TOBEFREE); switch (prev_state) { @@ -676,7 +720,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) struct bpf_map *map; int ret; - st_ops = bpf_struct_ops_find_value(attr->btf_vmlinux_value_type_id, btf_vmlinux); + st_ops = bpf_struct_ops_find_value(btf_vmlinux, attr->btf_vmlinux_value_type_id); if (!st_ops) return ERR_PTR(-ENOTSUPP); @@ -805,7 +849,7 @@ static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map) return map->map_type == BPF_MAP_TYPE_STRUCT_OPS && map->map_flags & BPF_F_LINK && /* Pair with smp_store_release() during map_update */ - smp_load_acquire(&st_map->kvalue.state) == BPF_STRUCT_OPS_STATE_READY; + smp_load_acquire(&st_map->kvalue.common.state) == BPF_STRUCT_OPS_STATE_READY; } static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link) From patchwork Tue Oct 17 16:23:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425610 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 370089CA52 for ; Tue, 17 Oct 2023 16:23:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="g56KAa//" Received: from mail-yb1-xb32.google.com (mail-yb1-xb32.google.com [IPv6:2607:f8b0:4864:20::b32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EAC6DF7 for ; Tue, 17 Oct 2023 09:23:16 -0700 (PDT) Received: by mail-yb1-xb32.google.com with SMTP id 3f1490d57ef6-d9a389db3c7so5040908276.1 for ; Tue, 17 Oct 2023 09:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559796; x=1698164596; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zxMqaT8h6QY3nPgn2rnbtpHscHvTCIxN/Kv68wogCGY=; b=g56KAa//XNUbXRkHQy6MoAQjOkIujGKoDPn9BQtKB4BFGsXWACMc24TmRXuc3rCKz3 QFr0rFwf8NmRhSTg6hcubmKR0CubuvVLAd2boMmvuBc3Imx7Qm5mn67RLlftOxOzn8IP akj/KA87W/hacQs5rFtC8RLMC1wfXHHX53DDsBAYs/IeonOBH94A5eAEH0OzepvrHGOh 2r1CJVUVuiYQIKUfEX6q2okoBQ0BjUFFkAPPHy8rgWLWHQkhjf5WFd3wtk4YJNIi2PAo GvM1B/7qWkgCmZUh1NMgUfMbEZTAw9AqVvVP7dO+O0lyKcA7SCs4GPWKwmB03p5yFghU WNHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559796; x=1698164596; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zxMqaT8h6QY3nPgn2rnbtpHscHvTCIxN/Kv68wogCGY=; b=EhQwHwlJ7I/QmZskZjHr+PiEM3gyRz3v3yfPIuaVGme88RFdc8skXnuZSBUM1JWayX Tc4SnVEdJlILqE8dqhq7FtsaDY8ZfgLJMKOlhzNYzV9fskFSKL9d+HhprFWqeq2JJppW KwTsHfXaTzGeJ+XrdCaAY7adIlMaDHLNRh7O6e2s6BhcI09mVZB6G6EsV/5cRcKYD2Lf /UUjtV2xc+8VSnlqHY+J+dOnawhrbUiTFoqwN+oOo8SUbfx1vba6V8XifnKXGmMTCSjF DpXj8hZ3ZdNCXIsH3XgbV1fuen0ZlIxxX5AfX/c/RjI5go1DcHz/tHHHTf0rLkld9838 CHvg== X-Gm-Message-State: AOJu0Yw+jgzGe4p18kDnOSQ+zxRV3+Kd0ZjsPNNskysZu5tYUPIRM9bY PrdgkzRecJy0uz8rMKsu1zX7q3b9Jgo= X-Google-Smtp-Source: AGHT+IH0K8/pa575Zcv8DW2NzjjX/xqx2AyEJYFH+nEwWwhpVkyCyOmmu+OetyQGvgP+5Sf0TMFeJw== X-Received: by 2002:a25:7704:0:b0:cb2:7e6:191c with SMTP id s4-20020a257704000000b00cb207e6191cmr2005992ybc.20.1697559795851; Tue, 17 Oct 2023 09:23:15 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:15 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 5/9] bpf: pass attached BTF to the bpf_struct_ops subsystem Date: Tue, 17 Oct 2023 09:23:02 -0700 Message-Id: <20231017162306.176586-6-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Giving a BTF, the bpf_struct_ops knows the right place to look up type info associated with a type ID. This enables a user space program to load a struct_ops object linked to a struct_ops type defined by a module, by providing the module BTF (fd). The bpf_prog includes attach_btf in aux which is passed along with the bpf_attr when loading the program. The purpose of attach_btf is to determine the btf type of attach_btf_id. The attach_btf_id is then used to identify the traced function for a trace program. In the case of struct_ops programs, it is used to identify the struct_ops type of the struct_ops object that a program is attached to. Signed-off-by: Kui-Feng Lee --- include/uapi/linux/bpf.h | 5 +++++ kernel/bpf/bpf_struct_ops.c | 34 +++++++++++++++++++++++++++------- kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 4 +++- tools/include/uapi/linux/bpf.h | 5 +++++ 5 files changed, 41 insertions(+), 9 deletions(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 73b155e52204..b5ef22f65f35 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1390,6 +1390,11 @@ union bpf_attr { * to using 5 hash functions). */ __u64 map_extra; + + __u32 value_type_btf_obj_fd; /* fd pointing to a BTF + * type data for + * btf_vmlinux_value_type_id. + */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 69703584fa4a..60445ff32275 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -677,6 +677,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map) bpf_jit_uncharge_modmem(PAGE_SIZE); } bpf_map_area_free(st_map->uvalue); + btf_put(st_map->st_ops->btf); module_put(st_map->st_ops->owner); bpf_map_area_free(st_map); } @@ -718,23 +719,36 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) struct bpf_struct_ops_map *st_map; const struct btf_type *t, *vt; struct bpf_map *map; + struct btf *btf; int ret; - st_ops = bpf_struct_ops_find_value(btf_vmlinux, attr->btf_vmlinux_value_type_id); - if (!st_ops) + if (attr->value_type_btf_obj_fd) { + btf = btf_get_by_fd(attr->value_type_btf_obj_fd); + if (IS_ERR(btf)) + return ERR_PTR(PTR_ERR(btf)); + } else { + btf = btf_vmlinux; + btf_get(btf); + } + st_ops = bpf_struct_ops_find_value(btf, attr->btf_vmlinux_value_type_id); + if (!st_ops) { + btf_put(btf); return ERR_PTR(-ENOTSUPP); + } /* If st_ops->owner is NULL, it means the struct_ops is * statically defined in the kernel. We don't need to * take a refcount on it. */ - if (st_ops->owner && !btf_try_get_module(st_ops->btf)) + if (st_ops->owner && !btf_try_get_module(st_ops->btf)) { + btf_put(btf); return ERR_PTR(-EINVAL); + } vt = st_ops->value_type; if (attr->value_size != vt->size) { - module_put(st_ops->owner); - return ERR_PTR(-EINVAL); + ret = -EINVAL; + goto errout; } t = st_ops->type; @@ -747,8 +761,8 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map = bpf_map_area_alloc(st_map_size, NUMA_NO_NODE); if (!st_map) { - module_put(st_ops->owner); - return ERR_PTR(-ENOMEM); + ret = -ENOMEM; + goto errout; } st_map->st_ops = st_ops; @@ -784,6 +798,12 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) bpf_map_init_from_attr(map, attr); return map; + +errout: + btf_put(btf); + module_put(st_ops->owner); + + return ERR_PTR(ret); } static u64 bpf_struct_ops_map_mem_usage(const struct bpf_map *map) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 85c1d908f70f..5daf8a2c2bba 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1097,7 +1097,7 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, return ret; } -#define BPF_MAP_CREATE_LAST_FIELD map_extra +#define BPF_MAP_CREATE_LAST_FIELD value_type_btf_obj_fd /* called via syscall */ static int map_create(union bpf_attr *attr) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 6564a03c425d..ce4df24eb03b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19623,6 +19623,7 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) const struct btf_member *member; struct bpf_prog *prog = env->prog; u32 btf_id, member_idx; + struct btf *btf; const char *mname; if (!prog->gpl_compatible) { @@ -19630,8 +19631,9 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) return -EINVAL; } + btf = prog->aux->attach_btf; btf_id = prog->aux->attach_btf_id; - st_ops = bpf_struct_ops_find(btf_vmlinux, btf_id); + st_ops = bpf_struct_ops_find(btf, btf_id); if (!st_ops) { verbose(env, "attach_btf_id %u is not a supported struct\n", btf_id); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 73b155e52204..b5ef22f65f35 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1390,6 +1390,11 @@ union bpf_attr { * to using 5 hash functions). */ __u64 map_extra; + + __u32 value_type_btf_obj_fd; /* fd pointing to a BTF + * type data for + * btf_vmlinux_value_type_id. + */ }; struct { /* anonymous struct used by BPF_MAP_*_ELEM commands */ From patchwork Tue Oct 17 16:23:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425611 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9C564734F; Tue, 17 Oct 2023 16:23:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S1pjWRhd" Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27D19B0; Tue, 17 Oct 2023 09:23:18 -0700 (PDT) Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-5a7afd45199so74400687b3.0; Tue, 17 Oct 2023 09:23:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559797; x=1698164597; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NnWq+uYNn+SsjJw/E+I7Mb11OXcmP1oROjW1ThmoOQs=; b=S1pjWRhdWPRweHX4etqhy8DcIhPM0v0JcxS9IXmIVb7zC7CaYhk3as9Akt4AaGfIF0 ukC8f7NWgvxILPD9wNoey6kz/BpuO8nOsNw3itk5247pa7rYstSJ9Z1NvQUFiq/aq2E4 jDbIlRe52StjAn3DV59HQQE9Afj0NstFEz7Tu2X0j3pF+18nQz3nuIqdo993rYTZY5Pk DSwXK6GAqH4+Zh7+LxeggymcQrpvu8e63AtM2dY7B0mb7VARXtZE0aq37JEGs7T8/EZp y+6euTFmATYJd80NnSr9YtuDmfTbw5YKTeTvbHcd8s+tyM/LjlH5LqDET4AknumV1Uku keWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559797; x=1698164597; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NnWq+uYNn+SsjJw/E+I7Mb11OXcmP1oROjW1ThmoOQs=; b=RWqdu3AgFR/knQs+7s4Pqd0Tw/xZgB5NbkrqFVhe0RnO15pRdT4s2kRnB2zGQSLlBT HL196iC8mrSPI+dAwpDdQL2QsZ4Juq6sRhxN3EuC2c3jpH84PzQlMjkilClorB1MOwMf 8/CLBxUdWz3lAp5mDDhX0sNXNrIEvDOxKuEeoDPKJ/KWbgW4QZFSyo12iZzdpBBKFFlA wqTL7zAV9yiDygnjFIYSlRuCZGOXmAV66VF13sL1FCT/s2OgwtE6uh/pARqG4NIjIVng gzgWq9/ZousqeekS3ZCY5BnN5+1306CYWw9V3h3UIpB+mrTLxs/t4Oj2qjLSycai9rSc BllQ== X-Gm-Message-State: AOJu0YwSROd7eu7Ky1AavKZwkKjHiuc9n92GcIyaQ4c3IWo3/HNs+nrQ VbMVcXu2Tm9ZEW2Pn+9H8G573fWhN4Y= X-Google-Smtp-Source: AGHT+IEteWVlunfEUR41/DWe+unPX12TZNR8U4ttj2eux9Ty6fBOtfQ/FyC1SgKC8vbpjSecLI+kMg== X-Received: by 2002:a25:ae8e:0:b0:d9c:66fb:db9 with SMTP id b14-20020a25ae8e000000b00d9c66fb0db9mr1133388ybj.59.1697559797049; Tue, 17 Oct 2023 09:23:17 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:16 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee , netdev@vger.kernel.org Subject: [PATCH bpf-next v5 6/9] bpf, net: switch to dynamic registration Date: Tue, 17 Oct 2023 09:23:03 -0700 Message-Id: <20231017162306.176586-7-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Replace the static list of struct_ops types with pre-btf struct_ops_tab to enable dynamic registration. Both bpf_dummy_ops and bpf_tcp_ca now utilize the registration function instead of being listed in bpf_struct_ops_types.h. Cc: netdev@vger.kernel.org Signed-off-by: Kui-Feng Lee --- include/linux/bpf.h | 2 + include/linux/btf.h | 29 +++++++ kernel/bpf/bpf_struct_ops.c | 124 +++++++++++++++--------------- kernel/bpf/bpf_struct_ops_types.h | 12 --- kernel/bpf/btf.c | 2 +- net/bpf/bpf_dummy_struct_ops.c | 14 +++- net/ipv4/bpf_tcp_ca.c | 16 +++- 7 files changed, 119 insertions(+), 80 deletions(-) delete mode 100644 kernel/bpf/bpf_struct_ops_types.h diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1e1647c8b0ce..b0f33147aa93 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3207,4 +3207,6 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) return prog->aux->func_idx != 0; } +int register_bpf_struct_ops(struct bpf_struct_ops *st_ops); + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/btf.h b/include/linux/btf.h index aa2ba77648be..fdc83aa10462 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -12,6 +12,8 @@ #include #define BTF_TYPE_EMIT(type) ((void)(type *)0) +#define BTF_STRUCT_OPS_TYPE_EMIT(type) {((void)(struct type *)0); \ + ((void)(struct bpf_struct_ops_##type *)0); } #define BTF_TYPE_EMIT_ENUM(enum_val) ((void)enum_val) /* These need to be macros, as the expressions are used in assembler input */ @@ -200,6 +202,7 @@ u32 btf_obj_id(const struct btf *btf); bool btf_is_kernel(const struct btf *btf); bool btf_is_module(const struct btf *btf); struct module *btf_try_get_module(const struct btf *btf); +struct btf *btf_get_module_btf(const struct module *module); u32 btf_nr_types(const struct btf *btf); bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, const struct btf_member *m, @@ -577,4 +580,30 @@ int btf_add_struct_ops(struct bpf_struct_ops *st_ops); const struct bpf_struct_ops ** btf_get_struct_ops(struct btf *btf, u32 *ret_cnt); +enum bpf_struct_ops_state { + BPF_STRUCT_OPS_STATE_INIT, + BPF_STRUCT_OPS_STATE_INUSE, + BPF_STRUCT_OPS_STATE_TOBEFREE, + BPF_STRUCT_OPS_STATE_READY, +}; + +struct bpf_struct_ops_common_value { + refcount_t refcnt; + enum bpf_struct_ops_state state; +}; +#define BPF_STRUCT_OPS_COMMON_VALUE struct bpf_struct_ops_common_value common + +/* bpf_struct_ops_##_name (e.g. bpf_struct_ops_tcp_congestion_ops) is + * the map's value exposed to the userspace and its btf-type-id is + * stored at the map->btf_vmlinux_value_type_id. + * + */ +#define DEFINE_STRUCT_OPS_VALUE_TYPE(_name) \ +extern struct bpf_struct_ops bpf_##_name; \ + \ +struct bpf_struct_ops_##_name { \ + BPF_STRUCT_OPS_COMMON_VALUE; \ + struct _name data ____cacheline_aligned_in_smp; \ +} + #endif diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 60445ff32275..175068b083cb 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -13,19 +13,6 @@ #include #include -enum bpf_struct_ops_state { - BPF_STRUCT_OPS_STATE_INIT, - BPF_STRUCT_OPS_STATE_INUSE, - BPF_STRUCT_OPS_STATE_TOBEFREE, - BPF_STRUCT_OPS_STATE_READY, -}; - -struct bpf_struct_ops_common_value { - refcount_t refcnt; - enum bpf_struct_ops_state state; -}; -#define BPF_STRUCT_OPS_COMMON_VALUE struct bpf_struct_ops_common_value common - struct bpf_struct_ops_value { BPF_STRUCT_OPS_COMMON_VALUE; char data[] ____cacheline_aligned_in_smp; @@ -72,35 +59,6 @@ static DEFINE_MUTEX(update_mutex); #define VALUE_PREFIX "bpf_struct_ops_" #define VALUE_PREFIX_LEN (sizeof(VALUE_PREFIX) - 1) -/* bpf_struct_ops_##_name (e.g. bpf_struct_ops_tcp_congestion_ops) is - * the map's value exposed to the userspace and its btf-type-id is - * stored at the map->btf_vmlinux_value_type_id. - * - */ -#define BPF_STRUCT_OPS_TYPE(_name) \ -extern struct bpf_struct_ops bpf_##_name; \ - \ -struct bpf_struct_ops_##_name { \ - BPF_STRUCT_OPS_COMMON_VALUE; \ - struct _name data ____cacheline_aligned_in_smp; \ -}; -#include "bpf_struct_ops_types.h" -#undef BPF_STRUCT_OPS_TYPE - -enum { -#define BPF_STRUCT_OPS_TYPE(_name) BPF_STRUCT_OPS_TYPE_##_name, -#include "bpf_struct_ops_types.h" -#undef BPF_STRUCT_OPS_TYPE - __NR_BPF_STRUCT_OPS_TYPE, -}; - -static struct bpf_struct_ops * const bpf_struct_ops[] = { -#define BPF_STRUCT_OPS_TYPE(_name) \ - [BPF_STRUCT_OPS_TYPE_##_name] = &bpf_##_name, -#include "bpf_struct_ops_types.h" -#undef BPF_STRUCT_OPS_TYPE -}; - const struct bpf_verifier_ops bpf_struct_ops_verifier_ops = { }; @@ -234,16 +192,51 @@ static void bpf_struct_ops_init_one(struct bpf_struct_ops *st_ops, } +static int register_bpf_struct_ops_btf(struct bpf_struct_ops *st_ops, + struct btf *btf) +{ + struct bpf_verifier_log *log; + int err; + + if (st_ops == NULL) + return -EINVAL; + + log = kzalloc(sizeof(*log), GFP_KERNEL | __GFP_NOWARN); + if (!log) { + err = -ENOMEM; + goto errout; + } + + log->level = BPF_LOG_KERNEL; + + bpf_struct_ops_init_one(st_ops, btf, st_ops->owner, log); + + err = btf_add_struct_ops(st_ops); + +errout: + kfree(log); + + return err; +} + +int register_bpf_struct_ops(struct bpf_struct_ops *st_ops) +{ + struct btf *btf; + int err; + + btf = btf_get_module_btf(st_ops->owner); + if (!btf) + return -EINVAL; + err = register_bpf_struct_ops_btf(st_ops, btf); + btf_put(btf); + + return err; +} +EXPORT_SYMBOL_GPL(register_bpf_struct_ops); + void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) { - struct bpf_struct_ops *st_ops; s32 module_id, common_value_id; - u32 i; - - /* Ensure BTF type is emitted for "struct bpf_struct_ops_##_name" */ -#define BPF_STRUCT_OPS_TYPE(_name) BTF_TYPE_EMIT(struct bpf_struct_ops_##_name); -#include "bpf_struct_ops_types.h" -#undef BPF_STRUCT_OPS_TYPE module_id = btf_find_by_name_kind(btf, "module", BTF_KIND_STRUCT); if (module_id < 0) { @@ -259,11 +252,6 @@ void bpf_struct_ops_init(struct btf *btf, struct bpf_verifier_log *log) return; } common_value_type = btf_type_by_id(btf, common_value_id); - - for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { - st_ops = bpf_struct_ops[i]; - bpf_struct_ops_init_one(st_ops, btf, NULL, log); - } } extern struct btf *btf_vmlinux; @@ -271,32 +259,44 @@ extern struct btf *btf_vmlinux; static const struct bpf_struct_ops * bpf_struct_ops_find_value(struct btf *btf, u32 value_id) { + const struct bpf_struct_ops *st_ops = NULL; + const struct bpf_struct_ops **st_ops_list; unsigned int i; + u32 cnt = 0; if (!value_id || !btf_vmlinux) return NULL; - for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { - if (bpf_struct_ops[i]->value_id == value_id) - return bpf_struct_ops[i]; + st_ops_list = btf_get_struct_ops(btf, &cnt); + for (i = 0; i < cnt; i++) { + if (st_ops_list[i]->value_id == value_id) { + st_ops = st_ops_list[i]; + break; + } } - return NULL; + return st_ops; } const struct bpf_struct_ops *bpf_struct_ops_find(struct btf *btf, u32 type_id) { + const struct bpf_struct_ops *st_ops = NULL; + const struct bpf_struct_ops **st_ops_list; unsigned int i; + u32 cnt; if (!type_id || !btf_vmlinux) return NULL; - for (i = 0; i < ARRAY_SIZE(bpf_struct_ops); i++) { - if (bpf_struct_ops[i]->type_id == type_id) - return bpf_struct_ops[i]; + st_ops_list = btf_get_struct_ops(btf, &cnt); + for (i = 0; i < cnt; i++) { + if (st_ops_list[i]->type_id == type_id) { + st_ops = st_ops_list[i]; + break; + } } - return NULL; + return st_ops; } static int bpf_struct_ops_map_get_next_key(struct bpf_map *map, void *key, diff --git a/kernel/bpf/bpf_struct_ops_types.h b/kernel/bpf/bpf_struct_ops_types.h deleted file mode 100644 index 5678a9ddf817..000000000000 --- a/kernel/bpf/bpf_struct_ops_types.h +++ /dev/null @@ -1,12 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* internal file - do not include directly */ - -#ifdef CONFIG_BPF_JIT -#ifdef CONFIG_NET -BPF_STRUCT_OPS_TYPE(bpf_dummy_ops) -#endif -#ifdef CONFIG_INET -#include -BPF_STRUCT_OPS_TYPE(tcp_congestion_ops) -#endif -#endif diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index be5144dbb53d..990973d6057d 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -7532,7 +7532,7 @@ struct module *btf_try_get_module(const struct btf *btf) /* Returns struct btf corresponding to the struct module. * This function can return NULL or ERR_PTR. */ -static struct btf *btf_get_module_btf(const struct module *module) +struct btf *btf_get_module_btf(const struct module *module) { #ifdef CONFIG_DEBUG_INFO_BTF_MODULES struct btf_module *btf_mod, *tmp; diff --git a/net/bpf/bpf_dummy_struct_ops.c b/net/bpf/bpf_dummy_struct_ops.c index 5918d1b32e19..724bb7224079 100644 --- a/net/bpf/bpf_dummy_struct_ops.c +++ b/net/bpf/bpf_dummy_struct_ops.c @@ -7,7 +7,7 @@ #include #include -extern struct bpf_struct_ops bpf_bpf_dummy_ops; +static struct bpf_struct_ops bpf_bpf_dummy_ops; /* A common type for test_N with return value in bpf_dummy_ops */ typedef int (*dummy_ops_test_ret_fn)(struct bpf_dummy_ops_state *state, ...); @@ -216,11 +216,13 @@ static int bpf_dummy_reg(void *kdata) return -EOPNOTSUPP; } +DEFINE_STRUCT_OPS_VALUE_TYPE(bpf_dummy_ops); + static void bpf_dummy_unreg(void *kdata) { } -struct bpf_struct_ops bpf_bpf_dummy_ops = { +static struct bpf_struct_ops bpf_bpf_dummy_ops = { .verifier_ops = &bpf_dummy_verifier_ops, .init = bpf_dummy_init, .check_member = bpf_dummy_ops_check_member, @@ -228,4 +230,12 @@ struct bpf_struct_ops bpf_bpf_dummy_ops = { .reg = bpf_dummy_reg, .unreg = bpf_dummy_unreg, .name = "bpf_dummy_ops", + .owner = THIS_MODULE, }; + +static int __init bpf_dummy_struct_ops_init(void) +{ + BTF_STRUCT_OPS_TYPE_EMIT(bpf_dummy_ops); + return register_bpf_struct_ops(&bpf_bpf_dummy_ops); +} +late_initcall(bpf_dummy_struct_ops_init); diff --git a/net/ipv4/bpf_tcp_ca.c b/net/ipv4/bpf_tcp_ca.c index 39dcccf0f174..20c401c73dfb 100644 --- a/net/ipv4/bpf_tcp_ca.c +++ b/net/ipv4/bpf_tcp_ca.c @@ -12,7 +12,7 @@ #include /* "extern" is to avoid sparse warning. It is only used in bpf_struct_ops.c. */ -extern struct bpf_struct_ops bpf_tcp_congestion_ops; +static struct bpf_struct_ops bpf_tcp_congestion_ops; static u32 unsupported_ops[] = { offsetof(struct tcp_congestion_ops, get_info), @@ -271,7 +271,9 @@ static int bpf_tcp_ca_validate(void *kdata) return tcp_validate_congestion_control(kdata); } -struct bpf_struct_ops bpf_tcp_congestion_ops = { +DEFINE_STRUCT_OPS_VALUE_TYPE(tcp_congestion_ops); + +static struct bpf_struct_ops bpf_tcp_congestion_ops = { .verifier_ops = &bpf_tcp_ca_verifier_ops, .reg = bpf_tcp_ca_reg, .unreg = bpf_tcp_ca_unreg, @@ -281,10 +283,18 @@ struct bpf_struct_ops bpf_tcp_congestion_ops = { .init = bpf_tcp_ca_init, .validate = bpf_tcp_ca_validate, .name = "tcp_congestion_ops", + .owner = THIS_MODULE, }; static int __init bpf_tcp_ca_kfunc_init(void) { - return register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_tcp_ca_kfunc_set); + int ret; + + BTF_STRUCT_OPS_TYPE_EMIT(tcp_congestion_ops); + + ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_tcp_ca_kfunc_set); + ret = ret ?: register_bpf_struct_ops(&bpf_tcp_congestion_ops); + + return ret; } late_initcall(bpf_tcp_ca_kfunc_init); From patchwork Tue Oct 17 16:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425612 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3ECB147351 for ; Tue, 17 Oct 2023 16:23:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZkYcScy0" Received: from mail-yb1-xb35.google.com (mail-yb1-xb35.google.com [IPv6:2607:f8b0:4864:20::b35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D9C09E for ; Tue, 17 Oct 2023 09:23:19 -0700 (PDT) Received: by mail-yb1-xb35.google.com with SMTP id 3f1490d57ef6-d9abc069c8bso6108194276.3 for ; Tue, 17 Oct 2023 09:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559798; x=1698164598; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w0Xqt4IZ7Ie3LDmSaBrZpsj4oO1zjRL24L+Ne3vuTJo=; b=ZkYcScy0FAhe4ILtH8IPlhlz3p6HpNDXCB5Ns3dii5+fW/eQOUW/nhTq8vjBS1mldi QCJXy1ynHdJgMyRbtCYrCvJmf3QUziBIQ5bLnVIvT9MmmZdn5EkTL4VmGW+TrMKm5JvH LlhB9Uj0Auq2tlXxfVbHnIyJGCXJC1GhCYZXIIxqTZO8drpBjPMFmA9iVvQQlPF9Rt6x IIRqTkIyjjPje8WRJWmFOdHEkQfsvLl7jicAA0VarlCBNbXQ6JPWeSeLYBaTT6RcDppB xsWoCECtI7z3C95YL6mTV+H6reTZ1+x0mFccLurqN8kbjWvNK+Yyza79r8D5VZBc/uLU 76vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559798; x=1698164598; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w0Xqt4IZ7Ie3LDmSaBrZpsj4oO1zjRL24L+Ne3vuTJo=; b=DUxfiuijeX1XoZyHIRgDZ1FRQSflXk2rz+9eANtsjQLbNzbEcO9RBXOZsNOXSDCHvP kkCuLtRqRhvNeJJXfmbrlrc5nazSo4yPpIj7zg1liXZr2GwQ0afjDZkVvEtRXB04tcS5 hB0brPaEBK/VBNJfIy7qlrLdlV1Nx12oB4+nzGgjENcqSyRFRMrtCAGcVX66Q7u4JmTp FFKXbn27dvjmzmFuoGwdVSkY9YVvgiL1dNdlRPuRYRd43AdvLHc9lnZCNVs7YlZ3Feo3 9E55STBx+MQBRnkjlh/Jm+fu7wRenejdYPNIJaES+EmywIiAgRpNuUdcrRmq0G+RLG1I iggw== X-Gm-Message-State: AOJu0YynQcSJDMbQ6svWNyCPBwxoMK9oc5gmoRbvqz30T0zg+lvAjn3Z xiTzci6zSfkEsyS9mpD6IiaPjr597wg= X-Google-Smtp-Source: AGHT+IHrjdceRZG7H6oWHtELKDZtgevliK2hwR8M6kZi4H80DQTzm5cs4zEMra2O0vOlHrwRAYaMMw== X-Received: by 2002:a05:6902:603:b0:d9c:2b72:4fa5 with SMTP id d3-20020a056902060300b00d9c2b724fa5mr2458321ybt.48.1697559798224; Tue, 17 Oct 2023 09:23:18 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:17 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 7/9] libbpf: Find correct module BTFs for struct_ops maps and progs. Date: Tue, 17 Oct 2023 09:23:04 -0700 Message-Id: <20231017162306.176586-8-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Locate the module BTFs for struct_ops maps and progs and pass them to the kernel. This ensures that the kernel correctly resolves type IDs from the appropriate module BTFs. For the map of a struct_ops object, mod_btf is added to bpf_map to keep a reference to the module BTF. The FD of the module BTF is passed to the kernel as mod_btf_fd when the struct_ops object is loaded. For a bpf_struct_ops prog, attach_btf_obj_fd of bpf_prog is the FD of a module BTF in the kernel. Signed-off-by: Kui-Feng Lee Acked-by: Andrii Nakryiko --- tools/lib/bpf/bpf.c | 4 ++- tools/lib/bpf/bpf.h | 5 +++- tools/lib/bpf/libbpf.c | 68 +++++++++++++++++++++++++++--------------- 3 files changed, 51 insertions(+), 26 deletions(-) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index b0f1913763a3..af46488e4ea9 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -169,7 +169,8 @@ int bpf_map_create(enum bpf_map_type map_type, __u32 max_entries, const struct bpf_map_create_opts *opts) { - const size_t attr_sz = offsetofend(union bpf_attr, map_extra); + const size_t attr_sz = offsetofend(union bpf_attr, + value_type_btf_obj_fd); union bpf_attr attr; int fd; @@ -191,6 +192,7 @@ int bpf_map_create(enum bpf_map_type map_type, attr.btf_key_type_id = OPTS_GET(opts, btf_key_type_id, 0); attr.btf_value_type_id = OPTS_GET(opts, btf_value_type_id, 0); attr.btf_vmlinux_value_type_id = OPTS_GET(opts, btf_vmlinux_value_type_id, 0); + attr.value_type_btf_obj_fd = OPTS_GET(opts, value_type_btf_obj_fd, 0); attr.inner_map_fd = OPTS_GET(opts, inner_map_fd, 0); attr.map_flags = OPTS_GET(opts, map_flags, 0); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 74c2887cfd24..1733cdc21241 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -51,8 +51,11 @@ struct bpf_map_create_opts { __u32 numa_node; __u32 map_ifindex; + + __u32 value_type_btf_obj_fd; + size_t:0; }; -#define bpf_map_create_opts__last_field map_ifindex +#define bpf_map_create_opts__last_field value_type_btf_obj_fd LIBBPF_API int bpf_map_create(enum bpf_map_type map_type, const char *map_name, diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 3a6108e3238b..d8a60fb52f5c 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -519,6 +519,7 @@ struct bpf_map { struct bpf_map_def def; __u32 numa_node; __u32 btf_var_idx; + int mod_btf_fd; __u32 btf_key_type_id; __u32 btf_value_type_id; __u32 btf_vmlinux_value_type_id; @@ -893,6 +894,8 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, return 0; } +static int load_module_btfs(struct bpf_object *obj); + static const struct btf_member * find_member_by_offset(const struct btf_type *t, __u32 bit_offset) { @@ -922,22 +925,29 @@ find_member_by_name(const struct btf *btf, const struct btf_type *t, return NULL; } +static int find_module_btf_id(struct bpf_object *obj, const char *kern_name, + __u16 kind, struct btf **res_btf, + struct module_btf **res_mod_btf); + #define STRUCT_OPS_VALUE_PREFIX "bpf_struct_ops_" static int find_btf_by_prefix_kind(const struct btf *btf, const char *prefix, const char *name, __u32 kind); static int -find_struct_ops_kern_types(const struct btf *btf, const char *tname, +find_struct_ops_kern_types(struct bpf_object *obj, const char *tname, + struct module_btf **mod_btf, const struct btf_type **type, __u32 *type_id, const struct btf_type **vtype, __u32 *vtype_id, const struct btf_member **data_member) { const struct btf_type *kern_type, *kern_vtype; const struct btf_member *kern_data_member; + struct btf *btf; __s32 kern_vtype_id, kern_type_id; __u32 i; - kern_type_id = btf__find_by_name_kind(btf, tname, BTF_KIND_STRUCT); + kern_type_id = find_module_btf_id(obj, tname, BTF_KIND_STRUCT, + &btf, mod_btf); if (kern_type_id < 0) { pr_warn("struct_ops init_kern: struct %s is not found in kernel BTF\n", tname); @@ -991,14 +1001,16 @@ static bool bpf_map__is_struct_ops(const struct bpf_map *map) } /* Init the map's fields that depend on kern_btf */ -static int bpf_map__init_kern_struct_ops(struct bpf_map *map, - const struct btf *btf, - const struct btf *kern_btf) +static int bpf_map__init_kern_struct_ops(struct bpf_map *map) { const struct btf_member *member, *kern_member, *kern_data_member; const struct btf_type *type, *kern_type, *kern_vtype; __u32 i, kern_type_id, kern_vtype_id, kern_data_off; + struct bpf_object *obj = map->obj; + const struct btf *btf = obj->btf; struct bpf_struct_ops *st_ops; + const struct btf *kern_btf; + struct module_btf *mod_btf; void *data, *kern_data; const char *tname; int err; @@ -1006,16 +1018,19 @@ static int bpf_map__init_kern_struct_ops(struct bpf_map *map, st_ops = map->st_ops; type = st_ops->type; tname = st_ops->tname; - err = find_struct_ops_kern_types(kern_btf, tname, + err = find_struct_ops_kern_types(obj, tname, &mod_btf, &kern_type, &kern_type_id, &kern_vtype, &kern_vtype_id, &kern_data_member); if (err) return err; + kern_btf = mod_btf ? mod_btf->btf : obj->btf_vmlinux; + pr_debug("struct_ops init_kern %s: type_id:%u kern_type_id:%u kern_vtype_id:%u\n", map->name, st_ops->type_id, kern_type_id, kern_vtype_id); + map->mod_btf_fd = mod_btf ? mod_btf->fd : 0; map->def.value_size = kern_vtype->size; map->btf_vmlinux_value_type_id = kern_vtype_id; @@ -1091,6 +1106,8 @@ static int bpf_map__init_kern_struct_ops(struct bpf_map *map, return -ENOTSUP; } + if (mod_btf) + prog->attach_btf_obj_fd = mod_btf->fd; prog->attach_btf_id = kern_type_id; prog->expected_attach_type = kern_member_idx; @@ -1133,8 +1150,7 @@ static int bpf_object__init_kern_struct_ops_maps(struct bpf_object *obj) if (!bpf_map__is_struct_ops(map)) continue; - err = bpf_map__init_kern_struct_ops(map, obj->btf, - obj->btf_vmlinux); + err = bpf_map__init_kern_struct_ops(map); if (err) return err; } @@ -5193,8 +5209,10 @@ static int bpf_object__create_map(struct bpf_object *obj, struct bpf_map *map, b create_attr.numa_node = map->numa_node; create_attr.map_extra = map->map_extra; - if (bpf_map__is_struct_ops(map)) + if (bpf_map__is_struct_ops(map)) { create_attr.btf_vmlinux_value_type_id = map->btf_vmlinux_value_type_id; + create_attr.value_type_btf_obj_fd = map->mod_btf_fd; + } if (obj->btf && btf__fd(obj->btf) >= 0) { create_attr.btf_fd = btf__fd(obj->btf); @@ -7700,9 +7718,9 @@ static int bpf_object__read_kallsyms_file(struct bpf_object *obj) return libbpf_kallsyms_parse(kallsyms_cb, obj); } -static int find_ksym_btf_id(struct bpf_object *obj, const char *ksym_name, - __u16 kind, struct btf **res_btf, - struct module_btf **res_mod_btf) +static int find_module_btf_id(struct bpf_object *obj, const char *kern_name, + __u16 kind, struct btf **res_btf, + struct module_btf **res_mod_btf) { struct module_btf *mod_btf; struct btf *btf; @@ -7710,7 +7728,7 @@ static int find_ksym_btf_id(struct bpf_object *obj, const char *ksym_name, btf = obj->btf_vmlinux; mod_btf = NULL; - id = btf__find_by_name_kind(btf, ksym_name, kind); + id = btf__find_by_name_kind(btf, kern_name, kind); if (id == -ENOENT) { err = load_module_btfs(obj); @@ -7721,7 +7739,7 @@ static int find_ksym_btf_id(struct bpf_object *obj, const char *ksym_name, /* we assume module_btf's BTF FD is always >0 */ mod_btf = &obj->btf_modules[i]; btf = mod_btf->btf; - id = btf__find_by_name_kind_own(btf, ksym_name, kind); + id = btf__find_by_name_kind_own(btf, kern_name, kind); if (id != -ENOENT) break; } @@ -7744,7 +7762,7 @@ static int bpf_object__resolve_ksym_var_btf_id(struct bpf_object *obj, struct btf *btf = NULL; int id, err; - id = find_ksym_btf_id(obj, ext->name, BTF_KIND_VAR, &btf, &mod_btf); + id = find_module_btf_id(obj, ext->name, BTF_KIND_VAR, &btf, &mod_btf); if (id < 0) { if (id == -ESRCH && ext->is_weak) return 0; @@ -7798,8 +7816,8 @@ static int bpf_object__resolve_ksym_func_btf_id(struct bpf_object *obj, local_func_proto_id = ext->ksym.type_id; - kfunc_id = find_ksym_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf, - &mod_btf); + kfunc_id = find_module_btf_id(obj, ext->essent_name ?: ext->name, BTF_KIND_FUNC, &kern_btf, + &mod_btf); if (kfunc_id < 0) { if (kfunc_id == -ESRCH && ext->is_weak) return 0; @@ -9464,9 +9482,9 @@ static int libbpf_find_prog_btf_id(const char *name, __u32 attach_prog_fd) return err; } -static int find_kernel_btf_id(struct bpf_object *obj, const char *attach_name, - enum bpf_attach_type attach_type, - int *btf_obj_fd, int *btf_type_id) +static int find_kernel_attach_btf_id(struct bpf_object *obj, const char *attach_name, + enum bpf_attach_type attach_type, + int *btf_obj_fd, int *btf_type_id) { int ret, i; @@ -9531,7 +9549,9 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attac *btf_obj_fd = 0; *btf_type_id = 1; } else { - err = find_kernel_btf_id(prog->obj, attach_name, attach_type, btf_obj_fd, btf_type_id); + err = find_kernel_attach_btf_id(prog->obj, attach_name, + attach_type, btf_obj_fd, + btf_type_id); } if (err) { pr_warn("prog '%s': failed to find kernel BTF type ID of '%s': %d\n", @@ -12945,9 +12965,9 @@ int bpf_program__set_attach_target(struct bpf_program *prog, err = bpf_object__load_vmlinux_btf(prog->obj, true); if (err) return libbpf_err(err); - err = find_kernel_btf_id(prog->obj, attach_func_name, - prog->expected_attach_type, - &btf_obj_fd, &btf_id); + err = find_kernel_attach_btf_id(prog->obj, attach_func_name, + prog->expected_attach_type, + &btf_obj_fd, &btf_id); if (err) return libbpf_err(err); } From patchwork Tue Oct 17 16:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425613 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E7699CA61 for ; Tue, 17 Oct 2023 16:23:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="h2Cco82A" Received: from mail-yb1-xb2d.google.com (mail-yb1-xb2d.google.com [IPv6:2607:f8b0:4864:20::b2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78B25F2 for ; Tue, 17 Oct 2023 09:23:20 -0700 (PDT) Received: by mail-yb1-xb2d.google.com with SMTP id 3f1490d57ef6-d9ad67058fcso6133647276.1 for ; Tue, 17 Oct 2023 09:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559799; x=1698164599; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=erUOV4dNptkdGGqSELDU+CDiUZ9qDYiO5ZXsDCge60c=; b=h2Cco82AO+3MLk3o5tqlLj7AXSndxkLjF1ZLxKwmaE6XmtCFHc2icQUiJxp6+8SJ3M 1s87fj9vd4iR15zQHcMiyAZz6NWDTWRmuutXKLdQ+UwIGfQSEclImZruk3xvYo5ueIxr W+eP0G1ASOjrS/vG+qYOx3yq/UVM5yV+k/idC2thoZantQTqN0K5h+bnTjHFzsAUWFbY kVNGfwAA+NQUDlh6prZj5dOxXuhlIItaZAsSYUriBn/wNbXkKzz3Jd36lYzOmixaXPSj sQMJok2q/Ep/Wc5bnwRFQilvRrljJQCbHEkNuMLu3ETPZNhaSWQTfqaela70/mOXI+Lh DPGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559799; x=1698164599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=erUOV4dNptkdGGqSELDU+CDiUZ9qDYiO5ZXsDCge60c=; b=KQMueAR3PDL2oT52ihsBzHiF0aDJBql+6+TKGxmur4Uuf0tdH0iXnQCbJA0gV9lhHB XcswM6C7Nt1kc5JPIqUKa0/Q6xHrvrWLyLiv9alHUmfDW56h1Swlm/4InD8mlovoX1Sj M86Gzpq8in0iy/rkxDBjM7VlIp3AU9O5hLQNCsxCvWRuO3RYYfY0SRqft0w8Hv9eLhCP pc9AdUBF8saDr0Wj/emM5jVC5xgTzZ14vsfFhS9zeBGMa8c02q/dAzNeIbe90JcRdhA8 /+n/1emiOP1X1yI0TqZKaN4z33anroTZboaOSQ8Gltpk6wN2at1nt2zVx9oyaKL7ltH2 pd3A== X-Gm-Message-State: AOJu0YwGMUM8QDIPnns+G6xhvGiLtyiUOM7Fsts/me30Me+9z7+7j+cg CwGPIOusqUBZR/VfgzhUR8HQn1WBBSQ= X-Google-Smtp-Source: AGHT+IERxfJvwbInVrNSQE73LoybDeo/HtE/bVcErQgf4tM0AslP/eRD4rhkMYzi+jJO9O4G1JbKww== X-Received: by 2002:a25:f02:0:b0:d9a:fd15:82a3 with SMTP id 2-20020a250f02000000b00d9afd1582a3mr2844880ybp.24.1697559799426; Tue, 17 Oct 2023 09:23:19 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:19 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 8/9] bpf: export btf_ctx_access to modules. Date: Tue, 17 Oct 2023 09:23:05 -0700 Message-Id: <20231017162306.176586-9-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee The module requires the use of btf_ctx_access() to invoke bpf_tracing_btf_ctx_access() from a module. This function is valuable for implementing validation functions that ensure proper access to ctx. Signed-off-by: Kui-Feng Lee --- kernel/bpf/btf.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 990973d6057d..dc6d8fe9b63d 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6141,6 +6141,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, __btf_name_by_offset(btf, t->name_off)); return true; } +EXPORT_SYMBOL_GPL(btf_ctx_access); enum bpf_struct_walk_result { /* < 0 error */ From patchwork Tue Oct 17 16:23:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13425614 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 133649CA6F for ; Tue, 17 Oct 2023 16:23:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QguFCTRC" Received: from mail-yb1-xb32.google.com (mail-yb1-xb32.google.com [IPv6:2607:f8b0:4864:20::b32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92F5CF1 for ; Tue, 17 Oct 2023 09:23:21 -0700 (PDT) Received: by mail-yb1-xb32.google.com with SMTP id 3f1490d57ef6-d8a000f6a51so6562251276.3 for ; Tue, 17 Oct 2023 09:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697559800; x=1698164600; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hr4BB6xMm96dKQlIq1m6N5v/m3o/73pMeTPNbjiLvqo=; b=QguFCTRCGZ1rsFPssjEDvrYmOxA9fHRkPwlwDE5/3AKu5ad5alb4tKYNfwRP7a6K+z lTW/6CCLi8V5aKaaA7ohLzOiEP9flCHb9ICbBskG/bEfwG3mGOes3Y8ot2cd3cWVn8Md Rb/g+kK27kO8F5zhJoefKUew94ikWNNghYLK2+OHQXymNx0UJNlFkcNR2vP4iSN1HmFR 4U+F16ktsyXeVMItwXc0AuhYOWPergwDa6gUU0O1mmrvtCVa0TdaBSKrikTvJhzyDjmQ gq3TiXyesKcK3qL7ZnGsWIG/pAzhfNHNpdUq0tbm0gJ+dX1Rsv5hyEVhydwTEyra3ceB UWFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697559800; x=1698164600; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hr4BB6xMm96dKQlIq1m6N5v/m3o/73pMeTPNbjiLvqo=; b=jTzjrv4yduol6KxaMhnj018Mtk2cJj8+Wst04eog0/5Rd7ribAqm+rdmqVvH4U7WGq UEfheyz7ltgY7ZtUt8Hq7WRPL6+yKf7r9ULo3X23Sv4+o2/6ZDCLqGoRl7xNq7LSRbPx ZFXMrcakzbv7qAWlGowEwAc5yY6cQRjVa6KqSgFxZY+NfOYFnA9pTh0K0mqBQpQGM6m7 liR5zzD2DX0PBkAMh4oJVJjrkIZlcBG6FeMJmvIPkXJdyW4b5xtd3rhQl2l1ZkdSnaeV PveZSGh6jdUB/5xS2MmJLyV6MikbuyodhMZdnWoMk5xqv27lD4FsYvGa4y5FXxzSJxr/ 3a7Q== X-Gm-Message-State: AOJu0YxCZmmVjK1F5D0rtNp6SbfCCgxOJuN4o419uRL+252UvWLOmxe7 QE7fpU0UGmJW7Vnkd7WDH1kDiXoGhqU= X-Google-Smtp-Source: AGHT+IGxQQ7knUw53oK+pFod73kSDxbhPqbhZ5a44p5bPYtv0dn14m8NH9z4T28MrHaQCsCr+aJLDg== X-Received: by 2002:a25:32cd:0:b0:d9a:de96:3da2 with SMTP id y196-20020a2532cd000000b00d9ade963da2mr2810849yby.1.1697559800563; Tue, 17 Oct 2023 09:23:20 -0700 (PDT) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:ed01:b54a:4364:93cc]) by smtp.gmail.com with ESMTPSA id r189-20020a2544c6000000b00d814d8dfd69sm623645yba.27.2023.10.17.09.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 09:23:20 -0700 (PDT) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org, drosen@google.com Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v5 9/9] selftests/bpf: test case for register_bpf_struct_ops(). Date: Tue, 17 Oct 2023 09:23:06 -0700 Message-Id: <20231017162306.176586-10-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017162306.176586-1-thinker.li@gmail.com> References: <20231017162306.176586-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Create a new struct_ops type called bpf_testmod_ops within the bpf_testmod module. When a struct_ops object is registered, the bpf_testmod module will invoke test_2 from the module. Signed-off-by: Kui-Feng Lee --- tools/testing/selftests/bpf/Makefile | 2 + .../selftests/bpf/bpf_testmod/bpf_testmod.c | 59 +++++++++++++++++++ .../selftests/bpf/bpf_testmod/bpf_testmod.h | 5 ++ .../bpf/prog_tests/test_struct_ops_module.c | 38 ++++++++++++ .../selftests/bpf/progs/struct_ops_module.c | 30 ++++++++++ tools/testing/selftests/bpf/testing_helpers.c | 35 +++++++++++ 6 files changed, 169 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_module.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_module.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index caede9b574cb..dd7ff14e1fdf 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -706,6 +706,8 @@ $(OUTPUT)/uprobe_multi: uprobe_multi.c $(call msg,BINARY,,$@) $(Q)$(CC) $(CFLAGS) $(LDFLAGS) $^ $(LDLIBS) -o $@ +$(OUTPUT)/testing_helpers.o: $(OUTPUT)/rcu_tasks_trace_gp.skel.h + EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \ prog_tests/tests.h map_tests/tests.h verifier/tests.h \ feature bpftool \ diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index cefc5dd72573..f1a20669d884 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -1,5 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 /* Copyright (c) 2020 Facebook */ +#include #include #include #include @@ -517,11 +518,66 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test_static_unused_arg) BTF_ID_FLAGS(func, bpf_kfunc_call_test_offset) BTF_SET8_END(bpf_testmod_check_kfunc_ids) +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES + +DEFINE_STRUCT_OPS_VALUE_TYPE(bpf_testmod_ops); + +static int bpf_testmod_ops_init(struct btf *btf) +{ + return 0; +} + +static bool bpf_testmod_ops_is_valid_access(int off, int size, + enum bpf_access_type type, + const struct bpf_prog *prog, + struct bpf_insn_access_aux *info) +{ + return bpf_tracing_btf_ctx_access(off, size, type, prog, info); +} + +static int bpf_testmod_ops_init_member(const struct btf_type *t, + const struct btf_member *member, + void *kdata, const void *udata) +{ + return 0; +} + static const struct btf_kfunc_id_set bpf_testmod_kfunc_set = { .owner = THIS_MODULE, .set = &bpf_testmod_check_kfunc_ids, }; +static const struct bpf_verifier_ops bpf_testmod_verifier_ops = { + .is_valid_access = bpf_testmod_ops_is_valid_access, +}; + +static int bpf_dummy_reg(void *kdata) +{ + struct bpf_testmod_ops *ops = kdata; + int r; + + BTF_STRUCT_OPS_TYPE_EMIT(bpf_testmod_ops); + r = ops->test_2(4, 3); + + return 0; +} + +static void bpf_dummy_unreg(void *kdata) +{ +} + +struct bpf_struct_ops bpf_bpf_testmod_ops = { + .verifier_ops = &bpf_testmod_verifier_ops, + .init = bpf_testmod_ops_init, + .init_member = bpf_testmod_ops_init_member, + .reg = bpf_dummy_reg, + .unreg = bpf_dummy_unreg, + .name = "bpf_testmod_ops", + .owner = THIS_MODULE, +}; + +#endif /* CONFIG_DEBUG_INFO_BTF_MODULES */ + extern int bpf_fentry_test1(int a); static int bpf_testmod_init(void) @@ -532,6 +588,9 @@ static int bpf_testmod_init(void) ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_testmod_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_testmod_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set); +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES + ret = ret ?: register_bpf_struct_ops(&bpf_bpf_testmod_ops); +#endif if (ret < 0) return ret; if (bpf_fentry_test1(0) < 0) diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h index f32793efe095..ca5435751c79 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h @@ -28,4 +28,9 @@ struct bpf_iter_testmod_seq { int cnt; }; +struct bpf_testmod_ops { + int (*test_1)(void); + int (*test_2)(int a, int b); +}; + #endif /* _BPF_TESTMOD_H */ diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_module.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_module.c new file mode 100644 index 000000000000..7261fc6c377a --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_module.c @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ +#include +#include + +#include "rcu_tasks_trace_gp.skel.h" +#include "struct_ops_module.skel.h" + +static void test_regular_load(void) +{ + struct struct_ops_module *skel; + struct bpf_link *link; + DECLARE_LIBBPF_OPTS(bpf_object_open_opts, opts); + int err; + + skel = struct_ops_module__open_opts(&opts); + if (!ASSERT_OK_PTR(skel, "struct_ops_module_open")) + return; + err = struct_ops_module__load(skel); + if (!ASSERT_OK(err, "struct_ops_module_load")) + return; + + link = bpf_map__attach_struct_ops(skel->maps.testmod_1); + ASSERT_OK_PTR(link, "attach_test_mod_1"); + + ASSERT_EQ(skel->bss->test_2_result, 7, "test_2_result"); + + bpf_link__destroy(link); + + struct_ops_module__destroy(skel); +} + +void serial_test_struct_ops_module(void) +{ + if (test__start_subtest("regular_load")) + test_regular_load(); +} + diff --git a/tools/testing/selftests/bpf/progs/struct_ops_module.c b/tools/testing/selftests/bpf/progs/struct_ops_module.c new file mode 100644 index 000000000000..cb305d04342f --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_module.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "../bpf_testmod/bpf_testmod.h" + +char _license[] SEC("license") = "GPL"; + +int test_2_result = 0; + +SEC("struct_ops/test_1") +int BPF_PROG(test_1) +{ + return 0xdeadbeef; +} + +SEC("struct_ops/test_2") +int BPF_PROG(test_2, int a, int b) +{ + test_2_result = a + b; + return a + b; +} + +SEC(".struct_ops.link") +struct bpf_testmod_ops testmod_1 = { + .test_1 = (void *)test_1, + .test_2 = (void *)test_2, +}; + diff --git a/tools/testing/selftests/bpf/testing_helpers.c b/tools/testing/selftests/bpf/testing_helpers.c index 8d994884c7b4..05870cd62458 100644 --- a/tools/testing/selftests/bpf/testing_helpers.c +++ b/tools/testing/selftests/bpf/testing_helpers.c @@ -10,6 +10,7 @@ #include "test_progs.h" #include "testing_helpers.h" #include +#include "rcu_tasks_trace_gp.skel.h" int parse_num_list(const char *s, bool **num_set, int *num_set_len) { @@ -380,10 +381,44 @@ int load_bpf_testmod(bool verbose) return 0; } +/* This function will trigger call_rcu_tasks_trace() in the kernel */ +static int kern_sync_rcu_tasks_trace(void) +{ + struct rcu_tasks_trace_gp *rcu; + time_t start; + long gp_seq; + LIBBPF_OPTS(bpf_test_run_opts, opts); + + rcu = rcu_tasks_trace_gp__open_and_load(); + if (IS_ERR(rcu)) + return -EFAULT; + if (rcu_tasks_trace_gp__attach(rcu)) + return -EFAULT; + + gp_seq = READ_ONCE(rcu->bss->gp_seq); + + if (bpf_prog_test_run_opts(bpf_program__fd(rcu->progs.do_call_rcu_tasks_trace), + &opts)) + return -EFAULT; + if (opts.retval != 0) + return -EFAULT; + + start = time(NULL); + while ((start + 2) > time(NULL) && + gp_seq == READ_ONCE(rcu->bss->gp_seq)) + sched_yield(); + + rcu_tasks_trace_gp__destroy(rcu); + + return 0; +} + /* * Trigger synchronize_rcu() in kernel. */ int kern_sync_rcu(void) { + if (kern_sync_rcu_tasks_trace()) + return -EFAULT; return syscall(__NR_membarrier, MEMBARRIER_CMD_SHARED, 0, 0); }