From patchwork Mon Jan 23 17:15:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Vernet X-Patchwork-Id: 13112589 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 055BFC61D97 for ; Mon, 23 Jan 2023 17:15:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233395AbjAWRPY (ORCPT ); Mon, 23 Jan 2023 12:15:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232714AbjAWRPX (ORCPT ); Mon, 23 Jan 2023 12:15:23 -0500 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 26FBF2BF38; Mon, 23 Jan 2023 09:15:16 -0800 (PST) Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-4a2f8ad29d5so181294177b3.8; Mon, 23 Jan 2023 09:15:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xCvkxNfOd/aJ/4NyFm02XVnNNOSaCLc/2W9e18Wtpm0=; b=rSnF7pdTefxHrQUrjAJxbn7P8FuthoYjFDhf15CWDsgoJwzLKsKbCS2KXfv8s3S1ZX pKtPUrK6zwnIDosL8l4gw8P4a+oaq2Ep2kVvxoEMlSoFWjraf1QJjPN9XLUjrvr6xuRz Pv+en6w5waxzC34kJt/yAUNIveDKi+rYcYPnrZrLKsyG3onoGJTJ2M4RIq/lu86wWib6 ZIykQu4P6kU1DkRkkPAjiuWWEr0pNrvFDO9A3/gR0/P842eqSkBTnRscUjz5Kwx7nqQH zwOYLXLBcQa3aNKPZ3sEYA+eYVIM2VqAf79y9AueUSy+WBgzy1hIpjLHOrLflHQuq6W/ j7Sg== X-Gm-Message-State: AFqh2kpGxfD2SGHhxXL9ynC85k9Eu4Ecr38WyupHNxiN9t80zQlN+sI8 lXTgIatxXAg85u0MpYm1bzpA/2Z+tTWQ1tLP X-Google-Smtp-Source: AMrXdXvWzH+CTAQ1y/GbMr8m5Ub22Gzcl3qEjyFt9QAyTf1n9J5yoFfxbcoZKQsoaJFkhKnIUQc9pQ== X-Received: by 2002:a81:1b4c:0:b0:4ed:5f40:b293 with SMTP id b73-20020a811b4c000000b004ed5f40b293mr13140534ywb.1.1674494114895; Mon, 23 Jan 2023 09:15:14 -0800 (PST) Received: from localhost ([2620:10d:c091:480::1:93a0]) by smtp.gmail.com with ESMTPSA id f1-20020a05620a280100b00706b6ce386fsm10889646qkp.70.2023.01.23.09.15.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 09:15:14 -0800 (PST) From: David Vernet To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@meta.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, memxor@gmail.com Subject: [PATCH bpf-next v2 1/3] bpf: Add BPF_KFUNC macro for defining kfuncs Date: Mon, 23 Jan 2023 11:15:04 -0600 Message-Id: <20230123171506.71995-2-void@manifault.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230123171506.71995-1-void@manifault.com> References: <20230123171506.71995-1-void@manifault.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net kfuncs are functions defined in the kernel, which may be invoked by BPF programs. They may or may not also be used as regular kernel functions, implying that they may be static (in which case the compiler could e.g. inline it away), or it could have external linkage, but potentially be elided in an LTO build if a function is observed to never be used, and is stripped from the final kernel binary. We therefore require some convenience macro that kfunc developers can use when defining a kfunc, and which will prevent all of the above issues from happening. This is in contrast with what we have today, where some kfunc definitions have "noinline", some have "__used", and others are static and have neither. In addition to providing the obvious correctness benefits, having such a macro also provides an easy way to query for if kfuncs, as Christoph suggested at the kernel maintainers summit (https://lwn.net/Articles/908464/). This is currently possible by grepping for BTF_ID_FLAGS(func, but having something more self describing is useful. Note that checkpatch complains about this patch with the following: ERROR: Macros with multiple statements should be enclosed in a do - while loop +#define BPF_KFUNC(proto) \ + proto; \ + __used noinline proto Given that we're wrapping the function definition and also including its prototype, using do { } while(0) isn't an option. Hopefully it's therefore inappropriate to ignore the warning. Signed-off-by: David Vernet --- include/linux/btf.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/btf.h b/include/linux/btf.h index 5f628f323442..593a1c2615e7 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -72,6 +72,16 @@ #define KF_DESTRUCTIVE (1 << 6) /* kfunc performs destructive actions */ #define KF_RCU (1 << 7) /* kfunc only takes rcu pointer arguments */ +/* + * Macro for defining a kfunc. This is meant to minimize the amount of + * copy-paste that kfunc authors have to include for correctness so as to avoid + * issues such as the compiler inlining or eliding either a static kfunc, or a + * global kfunc in an LTO build. + */ +#define BPF_KFUNC(proto) \ + proto; \ + __used noinline proto + /* * Return the name of the passed struct, if exists, or halt the build if for * example the structure gets renamed. In this way, developers have to revisit From patchwork Mon Jan 23 17:15:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Vernet X-Patchwork-Id: 13112590 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41F2DC54E94 for ; Mon, 23 Jan 2023 17:15:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233450AbjAWRP0 (ORCPT ); Mon, 23 Jan 2023 12:15:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233403AbjAWRPY (ORCPT ); Mon, 23 Jan 2023 12:15:24 -0500 Received: from mail-vs1-f53.google.com (mail-vs1-f53.google.com [209.85.217.53]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECA1F2C675; Mon, 23 Jan 2023 09:15:17 -0800 (PST) Received: by mail-vs1-f53.google.com with SMTP id 187so13645753vsv.10; Mon, 23 Jan 2023 09:15:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qwfyXOwnFjSLejeyuihpTx6+nDSKI2AEDITjv1M4Kjs=; b=pwNSlB/mR/9/9aryavTNxExoS9X3Fvqn7A9vZGfbNKzq3X1IP17l87MwjOz4njG8FZ qqOlHZMkfOcMmSEBTMdsrSwxcQabD7qwufCnPOfXx4021kqZVq6gQ+QefVH274WH4um6 wvodPuwwuPKtj6ZrIC82aSkgH92pS1X9534Eg26EvAfNdFDhyuQbhOFn/ytNdNnPel5t 7bbzRFP1EaF088O9m9ZpzV7dqa9g+x1308qH9fBJdNPOmwznlmLhr++fuijJwj1cbVlV px9511Y3j0ESAfsopjUyGCIoUfvEqUslEk5nBWazhDvn7dXVGFrscbm5IB30oPGL1BYN VuyA== X-Gm-Message-State: AFqh2koataQIDEuj/PqHjRdOQ1hKTdk2I8cf6pdHD1Xo4AKm6EjMSxYx 8t3CeX0OKhJVs8JeTnPU9zWjGGaEyIQgvgnj X-Google-Smtp-Source: AMrXdXtROEP40fg4oY1pAvXEDVNW4RTiR/0tqbXveNNBsW4y/WW0e8cn1c2GcKMHdg9E5h60YJJihw== X-Received: by 2002:a05:6102:34d0:b0:3ce:dbc0:5a6d with SMTP id a16-20020a05610234d000b003cedbc05a6dmr15218951vst.21.1674494116451; Mon, 23 Jan 2023 09:15:16 -0800 (PST) Received: from localhost ([2620:10d:c091:480::1:93a0]) by smtp.gmail.com with ESMTPSA id w26-20020a05620a0e9a00b006fa22f0494bsm12217724qkm.117.2023.01.23.09.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 09:15:16 -0800 (PST) From: David Vernet To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@meta.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, memxor@gmail.com Subject: [PATCH bpf-next v2 2/3] bpf: Document usage of the new BPF_KFUNC macro Date: Mon, 23 Jan 2023 11:15:05 -0600 Message-Id: <20230123171506.71995-3-void@manifault.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230123171506.71995-1-void@manifault.com> References: <20230123171506.71995-1-void@manifault.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Now that the BPF_KFUNC macro has been added to linux/btf.h, include a description of how to use it in the kfuncs.rst file. Along the way, do some cleanup of kfuncs.rst which allows the description to be more cleanly integrated into the doc. Signed-off-by: David Vernet --- Documentation/bpf/kfuncs.rst | 93 +++++++++++++++++++----------------- 1 file changed, 50 insertions(+), 43 deletions(-) diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst index 9fd7fb539f85..372f65aa87f7 100644 --- a/Documentation/bpf/kfuncs.rst +++ b/Documentation/bpf/kfuncs.rst @@ -14,55 +14,56 @@ kernel. 2. Defining a kfunc =================== -There are two ways to expose a kernel function to BPF programs, either make an -existing function in the kernel visible, or add a new wrapper for BPF. In both -cases, care must be taken that BPF program can only call such function in a -valid context. To enforce this, visibility of a kfunc can be per program type. +There are two ways to expose a kernel function to BPF programs: either make an +existing function in the kernel visible, or add a new function that is only +invoked from BPF. In both cases, care must be taken that BPF program can only +call the function in a valid context. To enforce this, attributes may be set on +a kfunc which inform the verifier of how the kfunc should be invoked in order +to ensure safety. Additionally, when kfuncs are first registered, they are +registered for a specific program type. + +2.1 Using BPF_KFUNC macro +------------------------- + +When defining a kfunc, always wrap its signature in the ``BPF_KFUNC`` macro. +This macro adds the necessary attributes to prevent the compiler from +optimizing away dead code, as some kfuncs may not be invoked anywhere in the +kernel itself, or may e.g. be removed in LTO builds. If an additional +annotation is required to prevent such an issue with your kfunc, it is likely a +bug and should be added to the definition of the macro so that other kfuncs are +similarly protected. This macro also provides an easy way to grep for kfunc +definitions. + +An example is given below: -If you are not creating a BPF wrapper for existing kernel function, skip ahead -to :ref:`BPF_kfunc_nodef`. - -2.1 Creating a wrapper kfunc ----------------------------- - -When defining a wrapper kfunc, the wrapper function should have extern linkage. -This prevents the compiler from optimizing away dead code, as this wrapper kfunc -is not invoked anywhere in the kernel itself. It is not necessary to provide a -prototype in a header for the wrapper kfunc. - -An example is given below:: - - /* Disables missing prototype warnings */ - __diag_push(); - __diag_ignore_all("-Wmissing-prototypes", - "Global kfuncs as their definitions will be in BTF"); +.. code-block:: c - struct task_struct *bpf_find_get_task_by_vpid(pid_t nr) + BPF_KFUNC(void bpf_memzero(void *mem, int mem__sz)) { - return find_get_task_by_vpid(nr); + ... } - __diag_pop(); - -A wrapper kfunc is often needed when we need to annotate parameters of the -kfunc. Otherwise one may directly make the kfunc visible to the BPF program by -registering it with the BPF subsystem. See :ref:`BPF_kfunc_nodef`. +It is not necessary to provide a prototype in a header for a kfunc (the macro +also takes care of this), though you may provide one if the kfunc is invoked +elsewhere in the main kernel. 2.2 Annotating kfunc parameters ------------------------------- -Similar to BPF helpers, there is sometime need for additional context required -by the verifier to make the usage of kernel functions safer and more useful. -Hence, we can annotate a parameter by suffixing the name of the argument of the -kfunc with a __tag, where tag may be one of the supported annotations. +Similar to BPF helpers, the verifier sometimes requires additional context to +make the usage of kernel functions safer and more useful. Hence, we can +annotate a parameter by suffixing the name of the argument of the kfunc with a +__tag, where tag may be one of the following supported annotations. 2.2.1 __sz Annotation --------------------- This annotation is used to indicate a memory and size pair in the argument list. -An example is given below:: +An example is given below: + +.. code-block:: c - void bpf_memzero(void *mem, int mem__sz) + BPF_KFUNC(void bpf_memzero(void *mem, int mem__sz)) { ... } @@ -80,9 +81,11 @@ the verifier must check the scalar argument to be a known constant, which does not indicate a size parameter, and the value of the constant is relevant to the safety of the program. -An example is given below:: +An example is given below: + +.. code-block:: c - void *bpf_obj_new(u32 local_type_id__k, ...) + BPF_KFUNC(void *bpf_obj_new(u32 local_type_id__k, ...)) { ... } @@ -96,8 +99,6 @@ Hence, whenever a constant scalar argument is accepted by a kfunc which is not a size parameter, and the value of the constant matters for program safety, __k suffix should be used. -.. _BPF_kfunc_nodef: - 2.3 Using an existing kernel function ------------------------------------- @@ -109,17 +110,21 @@ and whether it is safe to do so. 2.4 Annotating kfuncs --------------------- -In addition to kfuncs' arguments, verifier may need more information about the +In addition to kfunc arguments, verifier may need more information about the type of kfunc(s) being registered with the BPF subsystem. To do so, we define -flags on a set of kfuncs as follows:: +flags on a set of kfuncs as follows: + +.. code-block:: c BTF_SET8_START(bpf_task_set) BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE) + BTF_ID_FLAGS(func, bpf_task_is_kthread) /* Flags are optional */ BTF_SET8_END(bpf_task_set) -This set encodes the BTF ID of each kfunc listed above, and encodes the flags -along with it. Ofcourse, it is also allowed to specify no flags. +This set encodes the BTF ID of each kfunc listed above, along with an optional +set of flags which provide context to the verifier about how the kfunc should +be invoked. 2.4.1 KF_ACQUIRE flag --------------------- @@ -205,7 +210,9 @@ into consideration. Once the kfunc is prepared for use, the final step to making it visible is registering it with the BPF subsystem. Registration is done per BPF program -type. An example is shown below:: +type. An example is shown below: + +.. code-block:: c BTF_SET8_START(bpf_task_set) BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL) From patchwork Mon Jan 23 17:15:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Vernet X-Patchwork-Id: 13112591 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 167CEC05027 for ; Mon, 23 Jan 2023 17:16:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233296AbjAWRQG (ORCPT ); Mon, 23 Jan 2023 12:16:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233523AbjAWRP4 (ORCPT ); Mon, 23 Jan 2023 12:15:56 -0500 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20FEA2CC42; Mon, 23 Jan 2023 09:15:19 -0800 (PST) Received: by mail-yb1-f169.google.com with SMTP id 66so15710149yba.4; Mon, 23 Jan 2023 09:15:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rHlyxHCEexg13DqSbeGy72WwioCUWH3UJdSrEkH1PeQ=; b=Y5K4sq4FA3VhVjYigQmT74pOiqz/8DD1rdy8iQPaLpLnFsTSRmN/GHT4pdNMcqqH4g yl8y8WZnunNd0fQf1XtdTKHDhiCbnbP3cre3/fJDQCTBAU2oIKNXZd+Y6gBj5K/Dv6Hf WvZ0QOIvesekqtzNN6riQO9J+dxaALAlGzC4w/jHpl41u90eiLfWI42yDHDT65p1ZDk9 /w06AR/7cwkpZk+XqF7LAeh8Pk2HP3ENwL0XR9gILJxM8LnYPizuCgfxvZnijbCxO8eK EK3Vbidg4Kwa25s7PURgYopCrFh1bx4x+GVp5DPo6SvrZJLjXgKsWrE1CGa8yMuMbPO2 MCYA== X-Gm-Message-State: AFqh2ko8pOEvzEeVH4hCM6yMjsT3oo0ye3D62NGw4Fa2E+6r2QiT/k73 5HDgvZK8J7j4d8IwXy3x/b6fvfwi09di4mrR X-Google-Smtp-Source: AMrXdXtLR6hmJyOIYZK8p09WiOvXA8w2JLcxvlufBJuceCZRcOuYgJ5SRQBKKiCCr+PMK+GCfxLDYA== X-Received: by 2002:a25:9849:0:b0:7b8:620c:d3e4 with SMTP id k9-20020a259849000000b007b8620cd3e4mr16988524ybo.26.1674494118319; Mon, 23 Jan 2023 09:15:18 -0800 (PST) Received: from localhost ([2620:10d:c091:480::1:93a0]) by smtp.gmail.com with ESMTPSA id y15-20020a05620a0e0f00b006f7ee901674sm31629527qkm.2.2023.01.23.09.15.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 09:15:17 -0800 (PST) From: David Vernet To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yhs@meta.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, memxor@gmail.com Subject: [PATCH bpf-next v2 3/3] bpf: Use BPF_KFUNC macro at all kfunc definitions Date: Mon, 23 Jan 2023 11:15:06 -0600 Message-Id: <20230123171506.71995-4-void@manifault.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230123171506.71995-1-void@manifault.com> References: <20230123171506.71995-1-void@manifault.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Now that we have the BPF_KFUNC macro, we should use it to define all existing kfuncs to ensure that they'll never be elided in LTO builds, and so that we can remove the __diag blocks around them. Suggested-by: Kumar Kartikeya Dwivedi Signed-off-by: David Vernet --- kernel/bpf/helpers.c | 44 ++++++------- kernel/cgroup/rstat.c | 4 +- kernel/kexec_core.c | 3 +- kernel/trace/bpf_trace.c | 18 ++---- net/bpf/test_run.c | 64 +++++++++---------- net/ipv4/tcp_bbr.c | 16 ++--- net/ipv4/tcp_cong.c | 10 +-- net/ipv4/tcp_cubic.c | 12 ++-- net/ipv4/tcp_dctcp.c | 12 ++-- net/netfilter/nf_conntrack_bpf.c | 34 ++++------ net/netfilter/nf_nat_bpf.c | 12 +--- net/xfrm/xfrm_interface_bpf.c | 14 +--- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 3 +- 13 files changed, 105 insertions(+), 141 deletions(-) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 458db2db2f81..51d102fdfc29 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1772,11 +1772,7 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, } } -__diag_push(); -__diag_ignore_all("-Wmissing-prototypes", - "Global functions as their definitions will be in vmlinux BTF"); - -void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign) +BPF_KFUNC(void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign)) { struct btf_struct_meta *meta = meta__ign; u64 size = local_type_id__k; @@ -1790,7 +1786,7 @@ void *bpf_obj_new_impl(u64 local_type_id__k, void *meta__ign) return p; } -void bpf_obj_drop_impl(void *p__alloc, void *meta__ign) +BPF_KFUNC(void bpf_obj_drop_impl(void *p__alloc, void *meta__ign)) { struct btf_struct_meta *meta = meta__ign; void *p = p__alloc; @@ -1811,12 +1807,12 @@ static void __bpf_list_add(struct bpf_list_node *node, struct bpf_list_head *hea tail ? list_add_tail(n, h) : list_add(n, h); } -void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) +BPF_KFUNC(void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node)) { return __bpf_list_add(node, head, false); } -void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) +BPF_KFUNC(void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node)) { return __bpf_list_add(node, head, true); } @@ -1834,12 +1830,12 @@ static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tai return (struct bpf_list_node *)n; } -struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) +BPF_KFUNC(struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head)) { return __bpf_list_del(head, false); } -struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) +BPF_KFUNC(struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head)) { return __bpf_list_del(head, true); } @@ -1850,7 +1846,7 @@ struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) * bpf_task_release(). * @p: The task on which a reference is being acquired. */ -struct task_struct *bpf_task_acquire(struct task_struct *p) +BPF_KFUNC(struct task_struct *bpf_task_acquire(struct task_struct *p)) { return get_task_struct(p); } @@ -1861,7 +1857,7 @@ struct task_struct *bpf_task_acquire(struct task_struct *p) * released by calling bpf_task_release(). * @p: The task on which a reference is being acquired. */ -struct task_struct *bpf_task_acquire_not_zero(struct task_struct *p) +BPF_KFUNC(struct task_struct *bpf_task_acquire_not_zero(struct task_struct *p)) { /* For the time being this function returns NULL, as it's not currently * possible to safely acquire a reference to a task with RCU protection @@ -1913,7 +1909,7 @@ struct task_struct *bpf_task_acquire_not_zero(struct task_struct *p) * be released by calling bpf_task_release(). * @pp: A pointer to a task kptr on which a reference is being acquired. */ -struct task_struct *bpf_task_kptr_get(struct task_struct **pp) +BPF_KFUNC(struct task_struct *bpf_task_kptr_get(struct task_struct **pp)) { /* We must return NULL here until we have clarity on how to properly * leverage RCU for ensuring a task's lifetime. See the comment above @@ -1926,7 +1922,7 @@ struct task_struct *bpf_task_kptr_get(struct task_struct **pp) * bpf_task_release - Release the reference acquired on a task. * @p: The task on which a reference is being released. */ -void bpf_task_release(struct task_struct *p) +BPF_KFUNC(void bpf_task_release(struct task_struct *p)) { if (!p) return; @@ -1941,7 +1937,7 @@ void bpf_task_release(struct task_struct *p) * calling bpf_cgroup_release(). * @cgrp: The cgroup on which a reference is being acquired. */ -struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp) +BPF_KFUNC(struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp)) { cgroup_get(cgrp); return cgrp; @@ -1953,7 +1949,7 @@ struct cgroup *bpf_cgroup_acquire(struct cgroup *cgrp) * be released by calling bpf_cgroup_release(). * @cgrpp: A pointer to a cgroup kptr on which a reference is being acquired. */ -struct cgroup *bpf_cgroup_kptr_get(struct cgroup **cgrpp) +BPF_KFUNC(struct cgroup *bpf_cgroup_kptr_get(struct cgroup **cgrpp)) { struct cgroup *cgrp; @@ -1985,7 +1981,7 @@ struct cgroup *bpf_cgroup_kptr_get(struct cgroup **cgrpp) * drops to 0. * @cgrp: The cgroup on which a reference is being released. */ -void bpf_cgroup_release(struct cgroup *cgrp) +BPF_KFUNC(void bpf_cgroup_release(struct cgroup *cgrp)) { if (!cgrp) return; @@ -2000,7 +1996,7 @@ void bpf_cgroup_release(struct cgroup *cgrp) * @cgrp: The cgroup for which we're performing a lookup. * @level: The level of ancestor to look up. */ -struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level) +BPF_KFUNC(struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level)) { struct cgroup *ancestor; @@ -2019,7 +2015,7 @@ struct cgroup *bpf_cgroup_ancestor(struct cgroup *cgrp, int level) * stored in a map, or released with bpf_task_release(). * @pid: The pid of the task being looked up. */ -struct task_struct *bpf_task_from_pid(s32 pid) +BPF_KFUNC(struct task_struct *bpf_task_from_pid(s32 pid)) { struct task_struct *p; @@ -2032,28 +2028,26 @@ struct task_struct *bpf_task_from_pid(s32 pid) return p; } -void *bpf_cast_to_kern_ctx(void *obj) +BPF_KFUNC(void *bpf_cast_to_kern_ctx(void *obj)) { return obj; } -void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k) +BPF_KFUNC(void *bpf_rdonly_cast(void *obj__ign, u32 btf_id__k)) { return obj__ign; } -void bpf_rcu_read_lock(void) +BPF_KFUNC(void bpf_rcu_read_lock(void)) { rcu_read_lock(); } -void bpf_rcu_read_unlock(void) +BPF_KFUNC(void bpf_rcu_read_unlock(void)) { rcu_read_unlock(); } -__diag_pop(); - BTF_SET8_START(generic_btf_ids) #ifdef CONFIG_KEXEC_CORE BTF_ID_FLAGS(func, crash_kexec, KF_DESTRUCTIVE) diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index 793ecff29038..fbf41b434e1b 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -26,7 +26,7 @@ static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) * rstat_cpu->updated_children list. See the comment on top of * cgroup_rstat_cpu definition for details. */ -void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +BPF_KFUNC(void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)) { raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -231,7 +231,7 @@ static void cgroup_rstat_flush_locked(struct cgroup *cgrp, bool may_sleep) * * This function may block. */ -void cgroup_rstat_flush(struct cgroup *cgrp) +BPF_KFUNC(void cgroup_rstat_flush(struct cgroup *cgrp)) { might_sleep(); diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index 969e8f52f7da..6ea662dbc053 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -6,6 +6,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include #include #include #include @@ -975,7 +976,7 @@ void __noclone __crash_kexec(struct pt_regs *regs) } STACK_FRAME_NON_STANDARD(__crash_kexec); -void crash_kexec(struct pt_regs *regs) +BPF_KFUNC(void crash_kexec(struct pt_regs *regs)) { int old_cpu, this_cpu; diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 8124f1ad0d4a..c30be8d3a6fc 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1204,10 +1204,6 @@ static const struct bpf_func_proto bpf_get_func_arg_cnt_proto = { }; #ifdef CONFIG_KEYS -__diag_push(); -__diag_ignore_all("-Wmissing-prototypes", - "kfuncs which will be used in BPF programs"); - /** * bpf_lookup_user_key - lookup a key by its serial * @serial: key handle serial number @@ -1233,7 +1229,7 @@ __diag_ignore_all("-Wmissing-prototypes", * Return: a bpf_key pointer with a valid key pointer if the key is found, a * NULL pointer otherwise. */ -struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags) +BPF_KFUNC(struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags)) { key_ref_t key_ref; struct bpf_key *bkey; @@ -1282,7 +1278,7 @@ struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags) * Return: a bpf_key pointer with an invalid key pointer set from the * pre-determined ID on success, a NULL pointer otherwise */ -struct bpf_key *bpf_lookup_system_key(u64 id) +BPF_KFUNC(struct bpf_key *bpf_lookup_system_key(u64 id)) { struct bpf_key *bkey; @@ -1306,7 +1302,7 @@ struct bpf_key *bpf_lookup_system_key(u64 id) * Decrement the reference count of the key inside *bkey*, if the pointer * is valid, and free *bkey*. */ -void bpf_key_put(struct bpf_key *bkey) +BPF_KFUNC(void bpf_key_put(struct bpf_key *bkey)) { if (bkey->has_ref) key_put(bkey->key); @@ -1326,9 +1322,9 @@ void bpf_key_put(struct bpf_key *bkey) * * Return: 0 on success, a negative value on error. */ -int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr, - struct bpf_dynptr_kern *sig_ptr, - struct bpf_key *trusted_keyring) +BPF_KFUNC(int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr, + struct bpf_dynptr_kern *sig_ptr, + struct bpf_key *trusted_keyring)) { int ret; @@ -1356,8 +1352,6 @@ int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr, } #endif /* CONFIG_SYSTEM_DATA_VERIFICATION */ -__diag_pop(); - BTF_SET8_START(key_sig_kfunc_set) BTF_ID_FLAGS(func, bpf_lookup_user_key, KF_ACQUIRE | KF_RET_NULL | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_lookup_system_key, KF_ACQUIRE | KF_RET_NULL) diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 2723623429ac..f024717b98a2 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -481,10 +481,7 @@ static int bpf_test_finish(const union bpf_attr *kattr, * architecture dependent calling conventions. 7+ can be supported in the * future. */ -__diag_push(); -__diag_ignore_all("-Wmissing-prototypes", - "Global functions as their definitions will be in vmlinux BTF"); -int noinline bpf_fentry_test1(int a) +BPF_KFUNC(int bpf_fentry_test1(int a)) { return a + 1; } @@ -529,23 +526,23 @@ int noinline bpf_fentry_test8(struct bpf_fentry_test_t *arg) return (long)arg->a; } -int noinline bpf_modify_return_test(int a, int *b) +BPF_KFUNC(int bpf_modify_return_test(int a, int *b)) { *b += 1; return a + *b; } -u64 noinline bpf_kfunc_call_test1(struct sock *sk, u32 a, u64 b, u32 c, u64 d) +BPF_KFUNC(u64 bpf_kfunc_call_test1(struct sock *sk, u32 a, u64 b, u32 c, u64 d)) { return a + b + c + d; } -int noinline bpf_kfunc_call_test2(struct sock *sk, u32 a, u32 b) +BPF_KFUNC(int bpf_kfunc_call_test2(struct sock *sk, u32 a, u32 b)) { return a + b; } -struct sock * noinline bpf_kfunc_call_test3(struct sock *sk) +BPF_KFUNC(struct sock *bpf_kfunc_call_test3(struct sock *sk)) { return sk; } @@ -574,21 +571,19 @@ static struct prog_test_ref_kfunc prog_test_struct = { .cnt = REFCOUNT_INIT(1), }; -noinline struct prog_test_ref_kfunc * -bpf_kfunc_call_test_acquire(unsigned long *scalar_ptr) +BPF_KFUNC(struct prog_test_ref_kfunc *bpf_kfunc_call_test_acquire(unsigned long *scalar_ptr)) { refcount_inc(&prog_test_struct.cnt); return &prog_test_struct; } -noinline struct prog_test_member * -bpf_kfunc_call_memb_acquire(void) +BPF_KFUNC(struct prog_test_member *bpf_kfunc_call_memb_acquire(void)) { WARN_ON_ONCE(1); return NULL; } -noinline void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) +BPF_KFUNC(void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p)) { if (!p) return; @@ -596,11 +591,11 @@ noinline void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) refcount_dec(&p->cnt); } -noinline void bpf_kfunc_call_memb_release(struct prog_test_member *p) +BPF_KFUNC(void bpf_kfunc_call_memb_release(struct prog_test_member *p)) { } -noinline void bpf_kfunc_call_memb1_release(struct prog_test_member1 *p) +BPF_KFUNC(void bpf_kfunc_call_memb1_release(struct prog_test_member1 *p)) { WARN_ON_ONCE(1); } @@ -613,12 +608,14 @@ static int *__bpf_kfunc_call_test_get_mem(struct prog_test_ref_kfunc *p, const i return (int *)p; } -noinline int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, const int rdwr_buf_size) +BPF_KFUNC(int *bpf_kfunc_call_test_get_rdwr_mem(struct prog_test_ref_kfunc *p, + const int rdwr_buf_size)) { return __bpf_kfunc_call_test_get_mem(p, rdwr_buf_size); } -noinline int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) +BPF_KFUNC(int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, + const int rdonly_buf_size)) { return __bpf_kfunc_call_test_get_mem(p, rdonly_buf_size); } @@ -628,17 +625,18 @@ noinline int *bpf_kfunc_call_test_get_rdonly_mem(struct prog_test_ref_kfunc *p, * Acquire functions must return struct pointers, so these ones are * failing. */ -noinline int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, const int rdonly_buf_size) +BPF_KFUNC(int *bpf_kfunc_call_test_acq_rdonly_mem(struct prog_test_ref_kfunc *p, + const int rdonly_buf_size)) { return __bpf_kfunc_call_test_get_mem(p, rdonly_buf_size); } -noinline void bpf_kfunc_call_int_mem_release(int *p) +BPF_KFUNC(void bpf_kfunc_call_int_mem_release(int *p)) { } -noinline struct prog_test_ref_kfunc * -bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **pp, int a, int b) +BPF_KFUNC(struct prog_test_ref_kfunc * +bpf_kfunc_call_test_kptr_get(struct prog_test_ref_kfunc **pp, int a, int b)) { struct prog_test_ref_kfunc *p = READ_ONCE(*pp); @@ -686,52 +684,50 @@ struct prog_test_fail3 { char arr2[]; }; -noinline void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb) +BPF_KFUNC(void bpf_kfunc_call_test_pass_ctx(struct __sk_buff *skb)) { } -noinline void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p) +BPF_KFUNC(void bpf_kfunc_call_test_pass1(struct prog_test_pass1 *p)) { } -noinline void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p) +BPF_KFUNC(void bpf_kfunc_call_test_pass2(struct prog_test_pass2 *p)) { } -noinline void bpf_kfunc_call_test_fail1(struct prog_test_fail1 *p) +BPF_KFUNC(void bpf_kfunc_call_test_fail1(struct prog_test_fail1 *p)) { } -noinline void bpf_kfunc_call_test_fail2(struct prog_test_fail2 *p) +BPF_KFUNC(void bpf_kfunc_call_test_fail2(struct prog_test_fail2 *p)) { } -noinline void bpf_kfunc_call_test_fail3(struct prog_test_fail3 *p) +BPF_KFUNC(void bpf_kfunc_call_test_fail3(struct prog_test_fail3 *p)) { } -noinline void bpf_kfunc_call_test_mem_len_pass1(void *mem, int mem__sz) +BPF_KFUNC(void bpf_kfunc_call_test_mem_len_pass1(void *mem, int mem__sz)) { } -noinline void bpf_kfunc_call_test_mem_len_fail1(void *mem, int len) +BPF_KFUNC(void bpf_kfunc_call_test_mem_len_fail1(void *mem, int len)) { } -noinline void bpf_kfunc_call_test_mem_len_fail2(u64 *mem, int len) +BPF_KFUNC(void bpf_kfunc_call_test_mem_len_fail2(u64 *mem, int len)) { } -noinline void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p) +BPF_KFUNC(void bpf_kfunc_call_test_ref(struct prog_test_ref_kfunc *p)) { } -noinline void bpf_kfunc_call_test_destructive(void) +BPF_KFUNC(void bpf_kfunc_call_test_destructive(void)) { } -__diag_pop(); - BTF_SET8_START(bpf_test_modify_return_ids) BTF_ID_FLAGS(func, bpf_modify_return_test) BTF_ID_FLAGS(func, bpf_fentry_test1, KF_SLEEPABLE) diff --git a/net/ipv4/tcp_bbr.c b/net/ipv4/tcp_bbr.c index d2c470524e58..a3f6d3ab4182 100644 --- a/net/ipv4/tcp_bbr.c +++ b/net/ipv4/tcp_bbr.c @@ -295,7 +295,7 @@ static void bbr_set_pacing_rate(struct sock *sk, u32 bw, int gain) } /* override sysctl_tcp_min_tso_segs */ -static u32 bbr_min_tso_segs(struct sock *sk) +BPF_KFUNC(static u32 bbr_min_tso_segs(struct sock *sk)) { return sk->sk_pacing_rate < (bbr_min_tso_rate >> 3) ? 1 : 2; } @@ -328,7 +328,7 @@ static void bbr_save_cwnd(struct sock *sk) bbr->prior_cwnd = max(bbr->prior_cwnd, tcp_snd_cwnd(tp)); } -static void bbr_cwnd_event(struct sock *sk, enum tcp_ca_event event) +BPF_KFUNC(static void bbr_cwnd_event(struct sock *sk, enum tcp_ca_event event)) { struct tcp_sock *tp = tcp_sk(sk); struct bbr *bbr = inet_csk_ca(sk); @@ -1023,7 +1023,7 @@ static void bbr_update_model(struct sock *sk, const struct rate_sample *rs) bbr_update_gains(sk); } -static void bbr_main(struct sock *sk, const struct rate_sample *rs) +BPF_KFUNC(static void bbr_main(struct sock *sk, const struct rate_sample *rs)) { struct bbr *bbr = inet_csk_ca(sk); u32 bw; @@ -1035,7 +1035,7 @@ static void bbr_main(struct sock *sk, const struct rate_sample *rs) bbr_set_cwnd(sk, rs, rs->acked_sacked, bw, bbr->cwnd_gain); } -static void bbr_init(struct sock *sk) +BPF_KFUNC(static void bbr_init(struct sock *sk)) { struct tcp_sock *tp = tcp_sk(sk); struct bbr *bbr = inet_csk_ca(sk); @@ -1077,7 +1077,7 @@ static void bbr_init(struct sock *sk) cmpxchg(&sk->sk_pacing_status, SK_PACING_NONE, SK_PACING_NEEDED); } -static u32 bbr_sndbuf_expand(struct sock *sk) +BPF_KFUNC(static u32 bbr_sndbuf_expand(struct sock *sk)) { /* Provision 3 * cwnd since BBR may slow-start even during recovery. */ return 3; @@ -1086,7 +1086,7 @@ static u32 bbr_sndbuf_expand(struct sock *sk) /* In theory BBR does not need to undo the cwnd since it does not * always reduce cwnd on losses (see bbr_main()). Keep it for now. */ -static u32 bbr_undo_cwnd(struct sock *sk) +BPF_KFUNC(static u32 bbr_undo_cwnd(struct sock *sk)) { struct bbr *bbr = inet_csk_ca(sk); @@ -1097,7 +1097,7 @@ static u32 bbr_undo_cwnd(struct sock *sk) } /* Entering loss recovery, so save cwnd for when we exit or undo recovery. */ -static u32 bbr_ssthresh(struct sock *sk) +BPF_KFUNC(static u32 bbr_ssthresh(struct sock *sk)) { bbr_save_cwnd(sk); return tcp_sk(sk)->snd_ssthresh; @@ -1125,7 +1125,7 @@ static size_t bbr_get_info(struct sock *sk, u32 ext, int *attr, return 0; } -static void bbr_set_state(struct sock *sk, u8 new_state) +BPF_KFUNC(static void bbr_set_state(struct sock *sk, u8 new_state)) { struct bbr *bbr = inet_csk_ca(sk); diff --git a/net/ipv4/tcp_cong.c b/net/ipv4/tcp_cong.c index d3cae40749e8..0a7ca941af65 100644 --- a/net/ipv4/tcp_cong.c +++ b/net/ipv4/tcp_cong.c @@ -403,7 +403,7 @@ int tcp_set_congestion_control(struct sock *sk, const char *name, bool load, * ABC caps N to 2. Slow start exits when cwnd grows over ssthresh and * returns the leftover acks to adjust cwnd in congestion avoidance mode. */ -u32 tcp_slow_start(struct tcp_sock *tp, u32 acked) +BPF_KFUNC(u32 tcp_slow_start(struct tcp_sock *tp, u32 acked)) { u32 cwnd = min(tcp_snd_cwnd(tp) + acked, tp->snd_ssthresh); @@ -417,7 +417,7 @@ EXPORT_SYMBOL_GPL(tcp_slow_start); /* In theory this is tp->snd_cwnd += 1 / tp->snd_cwnd (or alternative w), * for every packet that was ACKed. */ -void tcp_cong_avoid_ai(struct tcp_sock *tp, u32 w, u32 acked) +BPF_KFUNC(void tcp_cong_avoid_ai(struct tcp_sock *tp, u32 w, u32 acked)) { /* If credits accumulated at a higher w, apply them gently now. */ if (tp->snd_cwnd_cnt >= w) { @@ -443,7 +443,7 @@ EXPORT_SYMBOL_GPL(tcp_cong_avoid_ai); /* This is Jacobson's slow start and congestion avoidance. * SIGCOMM '88, p. 328. */ -void tcp_reno_cong_avoid(struct sock *sk, u32 ack, u32 acked) +BPF_KFUNC(void tcp_reno_cong_avoid(struct sock *sk, u32 ack, u32 acked)) { struct tcp_sock *tp = tcp_sk(sk); @@ -462,7 +462,7 @@ void tcp_reno_cong_avoid(struct sock *sk, u32 ack, u32 acked) EXPORT_SYMBOL_GPL(tcp_reno_cong_avoid); /* Slow start threshold is half the congestion window (min 2) */ -u32 tcp_reno_ssthresh(struct sock *sk) +BPF_KFUNC(u32 tcp_reno_ssthresh(struct sock *sk)) { const struct tcp_sock *tp = tcp_sk(sk); @@ -470,7 +470,7 @@ u32 tcp_reno_ssthresh(struct sock *sk) } EXPORT_SYMBOL_GPL(tcp_reno_ssthresh); -u32 tcp_reno_undo_cwnd(struct sock *sk) +BPF_KFUNC(u32 tcp_reno_undo_cwnd(struct sock *sk)) { const struct tcp_sock *tp = tcp_sk(sk); diff --git a/net/ipv4/tcp_cubic.c b/net/ipv4/tcp_cubic.c index 768c10c1f649..bad9f3adba7f 100644 --- a/net/ipv4/tcp_cubic.c +++ b/net/ipv4/tcp_cubic.c @@ -126,7 +126,7 @@ static inline void bictcp_hystart_reset(struct sock *sk) ca->sample_cnt = 0; } -static void cubictcp_init(struct sock *sk) +BPF_KFUNC(static void cubictcp_init(struct sock *sk)) { struct bictcp *ca = inet_csk_ca(sk); @@ -139,7 +139,7 @@ static void cubictcp_init(struct sock *sk) tcp_sk(sk)->snd_ssthresh = initial_ssthresh; } -static void cubictcp_cwnd_event(struct sock *sk, enum tcp_ca_event event) +BPF_KFUNC(static void cubictcp_cwnd_event(struct sock *sk, enum tcp_ca_event event)) { if (event == CA_EVENT_TX_START) { struct bictcp *ca = inet_csk_ca(sk); @@ -321,7 +321,7 @@ static inline void bictcp_update(struct bictcp *ca, u32 cwnd, u32 acked) ca->cnt = max(ca->cnt, 2U); } -static void cubictcp_cong_avoid(struct sock *sk, u32 ack, u32 acked) +BPF_KFUNC(static void cubictcp_cong_avoid(struct sock *sk, u32 ack, u32 acked)) { struct tcp_sock *tp = tcp_sk(sk); struct bictcp *ca = inet_csk_ca(sk); @@ -338,7 +338,7 @@ static void cubictcp_cong_avoid(struct sock *sk, u32 ack, u32 acked) tcp_cong_avoid_ai(tp, ca->cnt, acked); } -static u32 cubictcp_recalc_ssthresh(struct sock *sk) +BPF_KFUNC(static u32 cubictcp_recalc_ssthresh(struct sock *sk)) { const struct tcp_sock *tp = tcp_sk(sk); struct bictcp *ca = inet_csk_ca(sk); @@ -355,7 +355,7 @@ static u32 cubictcp_recalc_ssthresh(struct sock *sk) return max((tcp_snd_cwnd(tp) * beta) / BICTCP_BETA_SCALE, 2U); } -static void cubictcp_state(struct sock *sk, u8 new_state) +BPF_KFUNC(static void cubictcp_state(struct sock *sk, u8 new_state)) { if (new_state == TCP_CA_Loss) { bictcp_reset(inet_csk_ca(sk)); @@ -445,7 +445,7 @@ static void hystart_update(struct sock *sk, u32 delay) } } -static void cubictcp_acked(struct sock *sk, const struct ack_sample *sample) +BPF_KFUNC(static void cubictcp_acked(struct sock *sk, const struct ack_sample *sample)) { const struct tcp_sock *tp = tcp_sk(sk); struct bictcp *ca = inet_csk_ca(sk); diff --git a/net/ipv4/tcp_dctcp.c b/net/ipv4/tcp_dctcp.c index e0a2ca7456ff..3748099f014e 100644 --- a/net/ipv4/tcp_dctcp.c +++ b/net/ipv4/tcp_dctcp.c @@ -75,7 +75,7 @@ static void dctcp_reset(const struct tcp_sock *tp, struct dctcp *ca) ca->old_delivered_ce = tp->delivered_ce; } -static void dctcp_init(struct sock *sk) +BPF_KFUNC(static void dctcp_init(struct sock *sk)) { const struct tcp_sock *tp = tcp_sk(sk); @@ -104,7 +104,7 @@ static void dctcp_init(struct sock *sk) INET_ECN_dontxmit(sk); } -static u32 dctcp_ssthresh(struct sock *sk) +BPF_KFUNC(static u32 dctcp_ssthresh(struct sock *sk)) { struct dctcp *ca = inet_csk_ca(sk); struct tcp_sock *tp = tcp_sk(sk); @@ -113,7 +113,7 @@ static u32 dctcp_ssthresh(struct sock *sk) return max(tcp_snd_cwnd(tp) - ((tcp_snd_cwnd(tp) * ca->dctcp_alpha) >> 11U), 2U); } -static void dctcp_update_alpha(struct sock *sk, u32 flags) +BPF_KFUNC(static void dctcp_update_alpha(struct sock *sk, u32 flags)) { const struct tcp_sock *tp = tcp_sk(sk); struct dctcp *ca = inet_csk_ca(sk); @@ -169,7 +169,7 @@ static void dctcp_react_to_loss(struct sock *sk) tp->snd_ssthresh = max(tcp_snd_cwnd(tp) >> 1U, 2U); } -static void dctcp_state(struct sock *sk, u8 new_state) +BPF_KFUNC(static void dctcp_state(struct sock *sk, u8 new_state)) { if (new_state == TCP_CA_Recovery && new_state != inet_csk(sk)->icsk_ca_state) @@ -179,7 +179,7 @@ static void dctcp_state(struct sock *sk, u8 new_state) */ } -static void dctcp_cwnd_event(struct sock *sk, enum tcp_ca_event ev) +BPF_KFUNC(static void dctcp_cwnd_event(struct sock *sk, enum tcp_ca_event ev)) { struct dctcp *ca = inet_csk_ca(sk); @@ -229,7 +229,7 @@ static size_t dctcp_get_info(struct sock *sk, u32 ext, int *attr, return 0; } -static u32 dctcp_cwnd_undo(struct sock *sk) +BPF_KFUNC(static u32 dctcp_cwnd_undo(struct sock *sk)) { const struct dctcp *ca = inet_csk_ca(sk); struct tcp_sock *tp = tcp_sk(sk); diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c index 24002bc61e07..5512c8f563a7 100644 --- a/net/netfilter/nf_conntrack_bpf.c +++ b/net/netfilter/nf_conntrack_bpf.c @@ -230,10 +230,6 @@ static int _nf_conntrack_btf_struct_access(struct bpf_verifier_log *log, return 0; } -__diag_push(); -__diag_ignore_all("-Wmissing-prototypes", - "Global functions as their definitions will be in nf_conntrack BTF"); - /* bpf_xdp_ct_alloc - Allocate a new CT entry * * Parameters: @@ -249,9 +245,9 @@ __diag_ignore_all("-Wmissing-prototypes", * @opts__sz - Length of the bpf_ct_opts structure * Must be NF_BPF_CT_OPTS_SZ (12) */ -struct nf_conn___init * +BPF_KFUNC(struct nf_conn___init * bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple, - u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz) + u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)) { struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; struct nf_conn *nfct; @@ -283,9 +279,9 @@ bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple, * @opts__sz - Length of the bpf_ct_opts structure * Must be NF_BPF_CT_OPTS_SZ (12) */ -struct nf_conn * +BPF_KFUNC(struct nf_conn * bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple, - u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz) + u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)) { struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; struct net *caller_net; @@ -316,9 +312,9 @@ bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple, * @opts__sz - Length of the bpf_ct_opts structure * Must be NF_BPF_CT_OPTS_SZ (12) */ -struct nf_conn___init * +BPF_KFUNC(struct nf_conn___init * bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple, - u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz) + u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)) { struct sk_buff *skb = (struct sk_buff *)skb_ctx; struct nf_conn *nfct; @@ -351,9 +347,9 @@ bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple, * @opts__sz - Length of the bpf_ct_opts structure * Must be NF_BPF_CT_OPTS_SZ (12) */ -struct nf_conn * +BPF_KFUNC(struct nf_conn * bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple, - u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz) + u32 tuple__sz, struct bpf_ct_opts *opts, u32 opts__sz)) { struct sk_buff *skb = (struct sk_buff *)skb_ctx; struct net *caller_net; @@ -376,7 +372,7 @@ bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple, * @nfct - Pointer to referenced nf_conn___init object, obtained * using bpf_xdp_ct_alloc or bpf_skb_ct_alloc. */ -struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i) +BPF_KFUNC(struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i)) { struct nf_conn *nfct = (struct nf_conn *)nfct_i; int err; @@ -400,7 +396,7 @@ struct nf_conn *bpf_ct_insert_entry(struct nf_conn___init *nfct_i) * @nf_conn - Pointer to referenced nf_conn object, obtained using * bpf_xdp_ct_lookup or bpf_skb_ct_lookup. */ -void bpf_ct_release(struct nf_conn *nfct) +BPF_KFUNC(void bpf_ct_release(struct nf_conn *nfct)) { if (!nfct) return; @@ -417,7 +413,7 @@ void bpf_ct_release(struct nf_conn *nfct) * bpf_xdp_ct_alloc or bpf_skb_ct_alloc. * @timeout - Timeout in msecs. */ -void bpf_ct_set_timeout(struct nf_conn___init *nfct, u32 timeout) +BPF_KFUNC(void bpf_ct_set_timeout(struct nf_conn___init *nfct, u32 timeout)) { __nf_ct_set_timeout((struct nf_conn *)nfct, msecs_to_jiffies(timeout)); } @@ -432,7 +428,7 @@ void bpf_ct_set_timeout(struct nf_conn___init *nfct, u32 timeout) * bpf_ct_insert_entry, bpf_xdp_ct_lookup, or bpf_skb_ct_lookup. * @timeout - New timeout in msecs. */ -int bpf_ct_change_timeout(struct nf_conn *nfct, u32 timeout) +BPF_KFUNC(int bpf_ct_change_timeout(struct nf_conn *nfct, u32 timeout)) { return __nf_ct_change_timeout(nfct, msecs_to_jiffies(timeout)); } @@ -447,7 +443,7 @@ int bpf_ct_change_timeout(struct nf_conn *nfct, u32 timeout) * bpf_xdp_ct_alloc or bpf_skb_ct_alloc. * @status - New status value. */ -int bpf_ct_set_status(const struct nf_conn___init *nfct, u32 status) +BPF_KFUNC(int bpf_ct_set_status(const struct nf_conn___init *nfct, u32 status)) { return nf_ct_change_status_common((struct nf_conn *)nfct, status); } @@ -462,13 +458,11 @@ int bpf_ct_set_status(const struct nf_conn___init *nfct, u32 status) * bpf_ct_insert_entry, bpf_xdp_ct_lookup or bpf_skb_ct_lookup. * @status - New status value. */ -int bpf_ct_change_status(struct nf_conn *nfct, u32 status) +BPF_KFUNC(int bpf_ct_change_status(struct nf_conn *nfct, u32 status)) { return nf_ct_change_status_common(nfct, status); } -__diag_pop() - BTF_SET8_START(nf_ct_kfunc_set) BTF_ID_FLAGS(func, bpf_xdp_ct_alloc, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_xdp_ct_lookup, KF_ACQUIRE | KF_RET_NULL) diff --git a/net/netfilter/nf_nat_bpf.c b/net/netfilter/nf_nat_bpf.c index 0fa5a0bbb0ff..8de652043819 100644 --- a/net/netfilter/nf_nat_bpf.c +++ b/net/netfilter/nf_nat_bpf.c @@ -12,10 +12,6 @@ #include #include -__diag_push(); -__diag_ignore_all("-Wmissing-prototypes", - "Global functions as their definitions will be in nf_nat BTF"); - /* bpf_ct_set_nat_info - Set source or destination nat address * * Set source or destination nat address of the newly allocated @@ -30,9 +26,9 @@ __diag_ignore_all("-Wmissing-prototypes", * interpreted as select a random port. * @manip - NF_NAT_MANIP_SRC or NF_NAT_MANIP_DST */ -int bpf_ct_set_nat_info(struct nf_conn___init *nfct, - union nf_inet_addr *addr, int port, - enum nf_nat_manip_type manip) +BPF_KFUNC(int bpf_ct_set_nat_info(struct nf_conn___init *nfct, + union nf_inet_addr *addr, int port, + enum nf_nat_manip_type manip)) { struct nf_conn *ct = (struct nf_conn *)nfct; u16 proto = nf_ct_l3num(ct); @@ -54,8 +50,6 @@ int bpf_ct_set_nat_info(struct nf_conn___init *nfct, return nf_nat_setup_info(ct, &range, manip) == NF_DROP ? -ENOMEM : 0; } -__diag_pop() - BTF_SET8_START(nf_nat_kfunc_set) BTF_ID_FLAGS(func, bpf_ct_set_nat_info, KF_TRUSTED_ARGS) BTF_SET8_END(nf_nat_kfunc_set) diff --git a/net/xfrm/xfrm_interface_bpf.c b/net/xfrm/xfrm_interface_bpf.c index 1ef2162cebcf..b2b2ae603e4d 100644 --- a/net/xfrm/xfrm_interface_bpf.c +++ b/net/xfrm/xfrm_interface_bpf.c @@ -27,10 +27,6 @@ struct bpf_xfrm_info { int link; }; -__diag_push(); -__diag_ignore_all("-Wmissing-prototypes", - "Global functions as their definitions will be in xfrm_interface BTF"); - /* bpf_skb_get_xfrm_info - Get XFRM metadata * * Parameters: @@ -39,8 +35,7 @@ __diag_ignore_all("-Wmissing-prototypes", * @to - Pointer to memory to which the metadata will be copied * Cannot be NULL */ -__used noinline -int bpf_skb_get_xfrm_info(struct __sk_buff *skb_ctx, struct bpf_xfrm_info *to) +BPF_KFUNC(int bpf_skb_get_xfrm_info(struct __sk_buff *skb_ctx, struct bpf_xfrm_info *to)) { struct sk_buff *skb = (struct sk_buff *)skb_ctx; struct xfrm_md_info *info; @@ -62,9 +57,8 @@ int bpf_skb_get_xfrm_info(struct __sk_buff *skb_ctx, struct bpf_xfrm_info *to) * @from - Pointer to memory from which the metadata will be copied * Cannot be NULL */ -__used noinline -int bpf_skb_set_xfrm_info(struct __sk_buff *skb_ctx, - const struct bpf_xfrm_info *from) +BPF_KFUNC(int bpf_skb_set_xfrm_info(struct __sk_buff *skb_ctx, + const struct bpf_xfrm_info *from)) { struct sk_buff *skb = (struct sk_buff *)skb_ctx; struct metadata_dst *md_dst; @@ -96,8 +90,6 @@ int bpf_skb_set_xfrm_info(struct __sk_buff *skb_ctx, return 0; } -__diag_pop() - BTF_SET8_START(xfrm_ifc_kfunc_set) BTF_ID_FLAGS(func, bpf_skb_get_xfrm_info) BTF_ID_FLAGS(func, bpf_skb_set_xfrm_info) diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 5085fea3cac5..9392e5e406ec 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -59,8 +59,7 @@ bpf_testmod_test_struct_arg_5(void) { return bpf_testmod_test_struct_arg_result; } -noinline void -bpf_testmod_test_mod_kfunc(int i) +BPF_KFUNC(void bpf_testmod_test_mod_kfunc(int i)) { *(int *)this_cpu_ptr(&bpf_testmod_ksym_percpu) = i; }