From patchwork Thu Nov 18 11:24:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626653 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7143FC433F5 for ; Thu, 18 Nov 2021 11:26:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 53C3560D42 for ; Thu, 18 Nov 2021 11:26:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343896AbhKRL3B (ORCPT ); Thu, 18 Nov 2021 06:29:01 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:20122 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343858AbhKRL2H (ORCPT ); Thu, 18 Nov 2021 06:28:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234705; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2wMk9r+qTWeACxM5RbfD+NjYv4ihc192PDuS22iqDs0=; b=g2Zrfmg69h51LojPViBqEgO906Hn46EkXpKNbh0x+TEUdo8PRbkT4lPlPSMORQglbLlX3N gVR0Wmu7VjC0z04gtIM9+2Kxedt4dAeEkg7KYE4d2l3ucOHJwPSy8UcycXeoXR9YoE1hHZ aDjuIlHEy0rBMCUk8wctV/sSXRFbFyk= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-442-0WE6cpIiNrWLHtIdQy9Eag-1; Thu, 18 Nov 2021 06:25:04 -0500 X-MC-Unique: 0WE6cpIiNrWLHtIdQy9Eag-1 Received: by mail-ed1-f71.google.com with SMTP id t9-20020aa7d709000000b003e83403a5cbso3024065edq.19 for ; Thu, 18 Nov 2021 03:25:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2wMk9r+qTWeACxM5RbfD+NjYv4ihc192PDuS22iqDs0=; b=fW5w4IZmvR3nFCLA0VNvI59bhQNhBm5TvNZF/vy/1rDq+Z8ietu+Z1vvWvQl5Lq8ci 4E7EMVOpsE7gju1vmV7I70132f60JvEm210LTVEhrVzCUQwZUyF4zaCqCUK9cSEJFqm9 F4FrtcbmfBNV0eaZkXpzwPNiME9iNF98751c0IOcNO8jj6mVLu522vYAW5pB05hszTVy FfJjjcxxMUeRrWcdoSVjd0HSLYkbLOlTsziUfOo/CS30RL7xnAascSLxPVtYbqbBtMu5 Z0F2sCuT5sp/8GMzNkHRMIx5iPnYF6gz39Br5scvmarRRYYfdZYx6v94lghk2kq7C5lX C5Vw== X-Gm-Message-State: AOAM530D2ZSUgJ/wfThmRFEVo3sBoVkBtyJFJfLQdQKkD24b0N/UMKtB U1DL342K6SFjjypiqihZrsp4tmaMzJKm06lNLVQv5ZFlgmqjg9L3pqfpxFlwFd53WJNoss7SxBx xe7XL621gx1L4Mvti X-Received: by 2002:a05:6402:1d50:: with SMTP id dz16mr9945787edb.125.1637234703355; Thu, 18 Nov 2021 03:25:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJwJr+RYD7h54P4G64E9M6OGatzmCpJfI84kSBT6M5VO0kPsTYqcMYXdLI/HU70s+2zCj0Sd6w== X-Received: by 2002:a05:6402:1d50:: with SMTP id dz16mr9945762edb.125.1637234703224; Thu, 18 Nov 2021 03:25:03 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id ht7sm1199357ejc.27.2021.11.18.03.25.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:02 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: Steven Rostedt , netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 01/29] ftrace: Use direct_ops hash in unregister_ftrace_direct Date: Thu, 18 Nov 2021 12:24:27 +0100 Message-Id: <20211118112455.475349-2-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Now when we have *direct_multi interface the direct_functions hash is no longer owned just by direct_ops. It's also used by any other ftrace_ops passed to *direct_multi interface. Thus to find out that we are unregistering the last function from direct_ops, we need to check directly direct_ops's hash. Cc: Steven Rostedt Fixes: f64dd4627ec6 ("ftrace: Add multi direct register/unregister interface") Signed-off-by: Jiri Olsa --- kernel/trace/ftrace.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 30bc880c3849..7f0594e28226 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5217,6 +5217,7 @@ int unregister_ftrace_direct(unsigned long ip, unsigned long addr) { struct ftrace_direct_func *direct; struct ftrace_func_entry *entry; + struct ftrace_hash *hash; int ret = -ENODEV; mutex_lock(&direct_mutex); @@ -5225,7 +5226,8 @@ int unregister_ftrace_direct(unsigned long ip, unsigned long addr) if (!entry) goto out_unlock; - if (direct_functions->count == 1) + hash = direct_ops.func_hash->filter_hash; + if (hash->count == 1) unregister_ftrace_function(&direct_ops); ret = ftrace_set_filter_ip(&direct_ops, ip, 1, 0); From patchwork Thu Nov 18 11:24:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626655 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 720CEC433EF for ; Thu, 18 Nov 2021 11:26:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 59A6A61A89 for ; Thu, 18 Nov 2021 11:26:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343922AbhKRL3H (ORCPT ); Thu, 18 Nov 2021 06:29:07 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46329 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343882AbhKRL2N (ORCPT ); Thu, 18 Nov 2021 06:28:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234711; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zzOB74CJjcZMNcmR7SkVFYkvFIHJ/t8Pjrq9GPHfvLc=; b=UIMDaqBFoibn4uEuxonn4hBfaGBAhMhf/JnzdJuDkmxe3AFXiY29jOgLEfVC16AXQAS5cD e75r8Yd7zcD0jJHiBKl5ox2Bt/q72DqGfNkhOqNjSAgzacZaawzxawBCfa3XtiOrcOa1gd uwWThM3rtCjqF73X+phYZBaWzvX+4Xw= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-298-02c_Vr6MPuWa7iaUtABh1g-1; Thu, 18 Nov 2021 06:25:10 -0500 X-MC-Unique: 02c_Vr6MPuWa7iaUtABh1g-1 Received: by mail-ed1-f70.google.com with SMTP id w18-20020a056402071200b003e61cbafdb4so5000924edx.4 for ; Thu, 18 Nov 2021 03:25:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zzOB74CJjcZMNcmR7SkVFYkvFIHJ/t8Pjrq9GPHfvLc=; b=gCnYYsIAgy9v5x1NUPPpjbD7iQdogaym9zzgHd2XJA2QwDK+ZjucJWwXgs7t6ml2Jw PjDQB9DLbFDnqL9R95/fsXRdRJjnHVFKjlyoUjmJ0y4JmHv8BIWMqApQCariJ8j0q6zt iymbYYJF2VR/MT9VsEfSG6PO8yE9TIn1tDHnKBPQDoRxv+1RNGlrlQvfEB7BYAOYRD+E ArqbFEuoG3AfJeSVqxJoMCau2cc1fxCapHinlDpS1fqOOCA/O6oUBVOuliOdwO34gbgf OvEN3d1rtkbvv4XvCMGtlBJ7dhiNEI3v7FXt0VxG0DKmVse0ItQl/BZH8AYme+T8fzJl MzMg== X-Gm-Message-State: AOAM5302yZhrLibJucSeTUnRax1aJ73DF10IwXtyL/Dw7F+IEB/yfF2n L892CwmRhrBwyj46Im1FIS6yrQH60+a4KDrcu0WORKoZ5MRMivUhXcK9nFPXwvFuf7Fpy7yAkcn pHeH9it4CZxPMN8Oe X-Received: by 2002:a17:906:b254:: with SMTP id ce20mr33752020ejb.255.1637234709291; Thu, 18 Nov 2021 03:25:09 -0800 (PST) X-Google-Smtp-Source: ABdhPJyVTHqHxq258/wo2aBL1NVcFRVlFTjKLdyUouotZrmXT5aXHOVSBzQF1U+lUYINEmKsPxwBLg== X-Received: by 2002:a17:906:b254:: with SMTP id ce20mr33751990ejb.255.1637234709146; Thu, 18 Nov 2021 03:25:09 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id hr17sm1185249ejc.57.2021.11.18.03.25.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:08 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: Steven Rostedt , netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 02/29] ftrace: Add cleanup to unregister_ftrace_direct_multi Date: Thu, 18 Nov 2021 12:24:28 +0100 Message-Id: <20211118112455.475349-3-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding ops cleanup to unregister_ftrace_direct_multi, so it can be reused in another register call. Cc: Steven Rostedt Fixes: f64dd4627ec6 ("ftrace: Add multi direct register/unregister interface") Signed-off-by: Jiri Olsa --- kernel/trace/ftrace.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7f0594e28226..be5f6b32a012 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5542,6 +5542,10 @@ int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) err = unregister_ftrace_function(ops); remove_direct_functions_hash(hash, addr); mutex_unlock(&direct_mutex); + + /* cleanup for possible another register call */ + ops->func = NULL; + ops->trampoline = 0; return err; } EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi); From patchwork Thu Nov 18 11:24:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626657 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9509BC433EF for ; Thu, 18 Nov 2021 11:26:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8092E61AFB for ; Thu, 18 Nov 2021 11:26:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343961AbhKRL32 (ORCPT ); Thu, 18 Nov 2021 06:29:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46223 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343901AbhKRL2T (ORCPT ); Thu, 18 Nov 2021 06:28:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234717; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aOPx437JJnxykUVvS3fHDcXPxIroZ8aiAi9DUKmIWiE=; b=KNx5KDLyYIGiP+MOQvgs6dkCbFY+hJO2+ua1T3r8nfzmdzkKMlS/OzIf0lQ9I0iVGEPpA+ pJFYijDQ1zNn5eOrMQy5TNa37+9KTqhXpc8MR75KvszabSWjlV/soOMPF82mJgHCSIaEYZ TRM9AGJ7HRiSWEEY8Qpwywms59RyG0I= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-552-QGtM66PXNEO5_l7X59iV_g-1; Thu, 18 Nov 2021 06:25:16 -0500 X-MC-Unique: QGtM66PXNEO5_l7X59iV_g-1 Received: by mail-ed1-f70.google.com with SMTP id v10-20020aa7d9ca000000b003e7bed57968so4954517eds.23 for ; Thu, 18 Nov 2021 03:25:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aOPx437JJnxykUVvS3fHDcXPxIroZ8aiAi9DUKmIWiE=; b=bEeEN/1E0ZO1J3hxFTSHZLuoR5uH6v6upJGD8Ou6tfT1aG9mtH33VHQZlCpzwigYYk 1AJrr0f/nNj6V39XgsyUeBnXzlaPuDvs4Q3dW34L+BfVL3rXDyk9k5SuzQsjS+07g2MD ynCMlTm9sYLCQ2SN89RI1RSoB0yU3anLG1AaMqNUcrtjd6gWamC8vjo+Vhut76CvAFK1 qPRXZYtuXHYE/+DfJyrdogeQoHLCOD1cGsC/ZD/jcJ8fuzSUxIfPUtk8WXu0WF0II8QL V3HOfGHzgxyQuVM3YRQZD57+LOS2k0P49EvclTyXTrVhp6eRmaIDmWp2EpKdAN4E3abt Bb0A== X-Gm-Message-State: AOAM533PeSsAYq0bRwCZbc95O34umtKLbCTEHFElqvFKPfVLaBUQeLOC 8DemzVnVTOBknwUe4ndIjm4KwY4izwh5XZSvVFWLughQJ+eQwB+EGzihvTJgBbcGvPWBfGiwJsD po5QNKg/nFrDQL26M X-Received: by 2002:a17:906:974f:: with SMTP id o15mr32691472ejy.229.1637234715309; Thu, 18 Nov 2021 03:25:15 -0800 (PST) X-Google-Smtp-Source: ABdhPJyxOov2hZwwKPIGKlNsoDOMYKKO4GIWaL19bfPkqeSwudPArCIhLayqKvZSWhhQ/ORkx5MW3Q== X-Received: by 2002:a17:906:974f:: with SMTP id o15mr32691447ejy.229.1637234715148; Thu, 18 Nov 2021 03:25:15 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id ho17sm1140971ejc.111.2021.11.18.03.25.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:14 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: Steven Rostedt , netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 03/29] ftrace: Add ftrace_set_filter_ips function Date: Thu, 18 Nov 2021 12:24:29 +0100 Message-Id: <20211118112455.475349-4-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding ftrace_set_filter_ips function to be able to set filter on multiple ip addresses at once. With the *direct_multi interface we have cases where we need to initialize ftrace_ops object with thousands of functions, so having single function diving into ftrace_hash_move_and_update_ops with ftrace_lock is better. The functions ips are passed as unsigned ong array with count. Cc: Steven Rostedt Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 3 +++ kernel/trace/ftrace.c | 53 +++++++++++++++++++++++++++++++++++------- 2 files changed, 47 insertions(+), 9 deletions(-) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 9999e29187de..60847cbce0da 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -512,6 +512,8 @@ struct dyn_ftrace { int ftrace_set_filter_ip(struct ftrace_ops *ops, unsigned long ip, int remove, int reset); +int ftrace_set_filter_ips(struct ftrace_ops *ops, unsigned long *ips, + unsigned int cnt, int remove, int reset); int ftrace_set_filter(struct ftrace_ops *ops, unsigned char *buf, int len, int reset); int ftrace_set_notrace(struct ftrace_ops *ops, unsigned char *buf, @@ -802,6 +804,7 @@ static inline unsigned long ftrace_location(unsigned long ip) #define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; }) #define ftrace_set_early_filter(ops, buf, enable) do { } while (0) #define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; }) +#define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; }) #define ftrace_set_filter(ops, buf, len, reset) ({ -ENODEV; }) #define ftrace_set_notrace(ops, buf, len, reset) ({ -ENODEV; }) #define ftrace_free_filter(ops) do { } while (0) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index be5f6b32a012..39350aa38649 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -4958,7 +4958,7 @@ ftrace_notrace_write(struct file *file, const char __user *ubuf, } static int -ftrace_match_addr(struct ftrace_hash *hash, unsigned long ip, int remove) +__ftrace_match_addr(struct ftrace_hash *hash, unsigned long ip, int remove) { struct ftrace_func_entry *entry; @@ -4976,9 +4976,25 @@ ftrace_match_addr(struct ftrace_hash *hash, unsigned long ip, int remove) return add_hash_entry(hash, ip); } +static int +ftrace_match_addr(struct ftrace_hash *hash, unsigned long *ips, + unsigned int cnt, int remove) +{ + unsigned int i; + int err; + + for (i = 0; i < cnt; i++) { + err = __ftrace_match_addr(hash, ips[i], remove); + if (err) + return err; + } + return 0; +} + static int ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len, - unsigned long ip, int remove, int reset, int enable) + unsigned long *ips, unsigned int cnt, + int remove, int reset, int enable) { struct ftrace_hash **orig_hash; struct ftrace_hash *hash; @@ -5008,8 +5024,8 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len, ret = -EINVAL; goto out_regex_unlock; } - if (ip) { - ret = ftrace_match_addr(hash, ip, remove); + if (ips) { + ret = ftrace_match_addr(hash, ips, cnt, remove); if (ret < 0) goto out_regex_unlock; } @@ -5026,10 +5042,10 @@ ftrace_set_hash(struct ftrace_ops *ops, unsigned char *buf, int len, } static int -ftrace_set_addr(struct ftrace_ops *ops, unsigned long ip, int remove, - int reset, int enable) +ftrace_set_addr(struct ftrace_ops *ops, unsigned long *ips, unsigned int cnt, + int remove, int reset, int enable) { - return ftrace_set_hash(ops, NULL, 0, ip, remove, reset, enable); + return ftrace_set_hash(ops, NULL, 0, ips, cnt, remove, reset, enable); } #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS @@ -5634,10 +5650,29 @@ int ftrace_set_filter_ip(struct ftrace_ops *ops, unsigned long ip, int remove, int reset) { ftrace_ops_init(ops); - return ftrace_set_addr(ops, ip, remove, reset, 1); + return ftrace_set_addr(ops, &ip, 1, remove, reset, 1); } EXPORT_SYMBOL_GPL(ftrace_set_filter_ip); +/** + * ftrace_set_filter_ips - set a functions to filter on in ftrace by addresses + * @ops - the ops to set the filter with + * @ips - the array of addresses to add to or remove from the filter. + * @cnt - the number of addresses in @ips + * @remove - non zero to remove ips from the filter + * @reset - non zero to reset all filters before applying this filter. + * + * Filters denote which functions should be enabled when tracing is enabled + * If @ips array or any ip specified within is NULL , it fails to update filter. + */ +int ftrace_set_filter_ips(struct ftrace_ops *ops, unsigned long *ips, + unsigned int cnt, int remove, int reset) +{ + ftrace_ops_init(ops); + return ftrace_set_addr(ops, ips, cnt, remove, reset, 1); +} +EXPORT_SYMBOL_GPL(ftrace_set_filter_ips); + /** * ftrace_ops_set_global_filter - setup ops to use global filters * @ops - the ops which will use the global filters @@ -5659,7 +5694,7 @@ static int ftrace_set_regex(struct ftrace_ops *ops, unsigned char *buf, int len, int reset, int enable) { - return ftrace_set_hash(ops, buf, len, 0, 0, reset, enable); + return ftrace_set_hash(ops, buf, len, NULL, 0, 0, reset, enable); } /** From patchwork Thu Nov 18 11:24:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626659 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC8E6C433EF for ; Thu, 18 Nov 2021 11:26:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A846E61AFB for ; Thu, 18 Nov 2021 11:26:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343972AbhKRL3b (ORCPT ); Thu, 18 Nov 2021 06:29:31 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:30113 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343906AbhKRL2Y (ORCPT ); Thu, 18 Nov 2021 06:28:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234723; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lWJS1A2VobZl6UHVtIe8xy1WCp4+zjRZoPdV1CZlw20=; b=AvfrDfO/LtkL/vo6oW4XST7VHyAmSPdQRNuk/Y7XLcMVhoxwp9JNJUrCRNvDSyCMv3dAha l58/iQxsLRdktRXFQYaEBXamZDDMnHWU9gaBZuzdcpkE+n89BKEFoTLE9rNpwgYA0aZwvH fWIvloaV04XPmctlSr7wnTITJ4e3qdU= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-82-l65cDXg5P5--3zNtYzZiQg-1; Thu, 18 Nov 2021 06:25:22 -0500 X-MC-Unique: l65cDXg5P5--3zNtYzZiQg-1 Received: by mail-ed1-f69.google.com with SMTP id f4-20020a50e084000000b003db585bc274so4958471edl.17 for ; Thu, 18 Nov 2021 03:25:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lWJS1A2VobZl6UHVtIe8xy1WCp4+zjRZoPdV1CZlw20=; b=PYrcpWt3oUQi+sBz5rJB6YoklxYqQqs6qIyYnlLCrYZucrsk0R6sVh8YIadEHAjCo2 BXvQZFZk53VARYcv+jDYO1DKCkG8Il3sOsVKyWnArXIowuAFGaodYs+7s8ING2lIogsT yMvCjDGk6/4JAHrqa/DQzoc7/cPjDeJ9drnIvTDHXe9yxJqJpHbKgcpyM0CiI0cYltQx tUU2UJIxOz4L93C6JW9D+J219iN6YieoasiF8ONKw9j2eVtDrXrBkPoBl731wHk1frTk IVAotKBlvJF5tRXIPI68Q7oSGE4WuvGL1dqdma+T4VTLFo9kcPqWCCtvtqCWLgI+Sfgu nK8Q== X-Gm-Message-State: AOAM530URv6zaT6l5QkjLZn4lJ6Uq+TnivFPIpSOtFFo/R33OCrnXc7z UqhHZiZfxYF0g3l3KKEzUK63VliImXgW1ksQE9fcScZ90sEqGTsNJO/R3lE0lsEyFxe+mKxcriA RLx7BvD9J56MRSnba X-Received: by 2002:a17:907:868e:: with SMTP id qa14mr33652129ejc.564.1637234721268; Thu, 18 Nov 2021 03:25:21 -0800 (PST) X-Google-Smtp-Source: ABdhPJz/teIMbtVpauVKwlK8uwMZzmXtH92/ajUedOTRqBR2Kz0Du+ywf6IJq9LdpyGserDhW7HpIQ== X-Received: by 2002:a17:907:868e:: with SMTP id qa14mr33652095ejc.564.1637234721109; Thu, 18 Nov 2021 03:25:21 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id x14sm1140515ejs.124.2021.11.18.03.25.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:20 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 04/29] bpf: Factor bpf_check_attach_target function Date: Thu, 18 Nov 2021 12:24:30 +0100 Message-Id: <20211118112455.475349-5-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Separating the check itself from model distilling and address search into __bpf_check_attach_target function. This way we can easily add function in following patch that gets only function model without the address search, while using the same code as bpf_check_attach_target. Signed-off-by: Jiri Olsa --- kernel/bpf/verifier.c | 79 ++++++++++++++++++++++++++++++++----------- 1 file changed, 59 insertions(+), 20 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0763cca139a7..cbbbf47e1832 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13568,20 +13568,26 @@ static int check_non_sleepable_error_inject(u32 btf_id) return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id); } -int bpf_check_attach_target(struct bpf_verifier_log *log, - const struct bpf_prog *prog, - const struct bpf_prog *tgt_prog, - u32 btf_id, - struct bpf_attach_target_info *tgt_info) +struct attach_target { + const struct btf_type *t; + const char *tname; + int subprog; + struct btf *btf; +}; + +static int __bpf_check_attach_target(struct bpf_verifier_log *log, + const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct attach_target *target) { bool prog_extension = prog->type == BPF_PROG_TYPE_EXT; const char prefix[] = "btf_trace_"; - int ret = 0, subprog = -1, i; + int subprog = -1, i; const struct btf_type *t; bool conservative = true; const char *tname; struct btf *btf; - long addr = 0; if (!btf_id) { bpf_log(log, "Tracing programs must provide btf_id\n"); @@ -13706,9 +13712,6 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, t = btf_type_by_id(btf, t->type); if (!btf_type_is_func_proto(t)) return -EINVAL; - ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel); - if (ret) - return ret; break; default: if (!prog_extension) @@ -13737,22 +13740,57 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, if (tgt_prog && conservative) t = NULL; + } + + target->t = t; + target->tname = tname; + target->subprog = subprog; + target->btf = btf; + return 0; +} + +int bpf_check_attach_target(struct bpf_verifier_log *log, + const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct bpf_attach_target_info *tgt_info) +{ + struct attach_target target = { }; + long addr = 0; + int ret; - ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel); + ret = __bpf_check_attach_target(log, prog, tgt_prog, btf_id, &target); + if (ret) + return ret; + + switch (prog->expected_attach_type) { + case BPF_TRACE_RAW_TP: + break; + case BPF_TRACE_ITER: + ret = btf_distill_func_proto(log, target.btf, target.t, target.tname, &tgt_info->fmodel); + if (ret) + return ret; + break; + default: + case BPF_MODIFY_RETURN: + case BPF_LSM_MAC: + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: + ret = btf_distill_func_proto(log, target.btf, target.t, target.tname, &tgt_info->fmodel); if (ret < 0) return ret; if (tgt_prog) { - if (subprog == 0) + if (target.subprog == 0) addr = (long) tgt_prog->bpf_func; else - addr = (long) tgt_prog->aux->func[subprog]->bpf_func; + addr = (long) tgt_prog->aux->func[target.subprog]->bpf_func; } else { - addr = kallsyms_lookup_name(tname); + addr = kallsyms_lookup_name(target.tname); if (!addr) { bpf_log(log, "The address of function %s cannot be found\n", - tname); + target.tname); return -ENOENT; } } @@ -13779,7 +13817,7 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, break; } if (ret) { - bpf_log(log, "%s is not sleepable\n", tname); + bpf_log(log, "%s is not sleepable\n", target.tname); return ret; } } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) { @@ -13787,18 +13825,19 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, bpf_log(log, "can't modify return codes of BPF programs\n"); return -EINVAL; } - ret = check_attach_modify_return(addr, tname); + ret = check_attach_modify_return(addr, target.tname); if (ret) { - bpf_log(log, "%s() is not modifiable\n", tname); + bpf_log(log, "%s() is not modifiable\n", target.tname); return ret; } } break; } + tgt_info->tgt_addr = addr; - tgt_info->tgt_name = tname; - tgt_info->tgt_type = t; + tgt_info->tgt_name = target.tname; + tgt_info->tgt_type = target.t; return 0; } From patchwork Thu Nov 18 11:24:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626661 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC921C433EF for ; Thu, 18 Nov 2021 11:26:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 953DE61AFF for ; Thu, 18 Nov 2021 11:26:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343928AbhKRL3c (ORCPT ); Thu, 18 Nov 2021 06:29:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:47313 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343912AbhKRL23 (ORCPT ); Thu, 18 Nov 2021 06:28:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234729; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FziBM/Abk38ZsIcsX8WUe178lP8av41CXsWZwffEWF4=; b=dp87VThlV8r38SECUmklwAN0ziYusydSUGvtDFJO2XkUQkujqk3WC6C38LFfiH1jlyLHlY HnYBxNyBYa+ZW8nfgPT93fLtGOCV0+AA1EI6utX2dJc39jHoOAMTKbo15gc8A4loBZlYPC ZB6iZiNl6CqpNP84ZBeJc19qus2ldO4= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-123-mcBNma4UNfS3-AbA-yujPA-1; Thu, 18 Nov 2021 06:25:28 -0500 X-MC-Unique: mcBNma4UNfS3-AbA-yujPA-1 Received: by mail-ed1-f70.google.com with SMTP id m8-20020a056402510800b003e29de5badbso4961891edd.18 for ; Thu, 18 Nov 2021 03:25:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FziBM/Abk38ZsIcsX8WUe178lP8av41CXsWZwffEWF4=; b=fAzKGXMToJWPt/h7Val3nmJpjZ6Mzuhy6LZer0YsoUKxBxsU93ctAm/3gnSoy8H8mE e/irjGvQZJ072jbCLBxu8hrBFuaV5r5l7kP9XWjFS36ReslGOmO0iuoWKXzC2X4oWzYB 0iXIwlMJyW3Ly3AxgC240B/WJm07bSCzPRXyuKW40NGQOTE8OnAvW4CoikPA+sBpjc6i Ho5VZNipZU7jqM5ARDkWodYBKygyU9mOdfbxizoqxCVcGcuhUIn66uJ88KjDdGUrMyaR LMSKJZkjb3hgm1MkMi66KOMx6HWgeEbBjsJakmv226jn5rQKJtVq9uPuKnrMpgy0bJ4t zwkA== X-Gm-Message-State: AOAM532FsetFvMA2xtBZ8yWbt26CSxj44zxF0Vje1hsI6zbI3kes0uMd Q6TCCiuUTnTlJXTl9p46R166Y3YJmC5GoG1FRt5IXjq6l0QuckmqEIClHsMxwC93QJd51c/wPMN H6u5NWgVp9OrS+IK/ X-Received: by 2002:a50:ef02:: with SMTP id m2mr10171533eds.172.1637234727230; Thu, 18 Nov 2021 03:25:27 -0800 (PST) X-Google-Smtp-Source: ABdhPJwibvICcLQ2tYVo+N7H3O2OA40ycE0Lvf9i8jybkUJAsfZom/RYr7fy5OAjx+XkJ8L5+N/sOw== X-Received: by 2002:a50:ef02:: with SMTP id m2mr10171509eds.172.1637234727119; Thu, 18 Nov 2021 03:25:27 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id d3sm1443289edx.79.2021.11.18.03.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:26 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 05/29] bpf: Add bpf_check_attach_model function Date: Thu, 18 Nov 2021 12:24:31 +0100 Message-Id: <20211118112455.475349-6-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding bpf_check_attach_model function that returns model for function specified by btf_id. It will be used in following patches. Signed-off-by: Jiri Olsa --- include/linux/bpf_verifier.h | 4 ++++ kernel/bpf/verifier.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index c8a78e830fca..b561c0b08e68 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -527,6 +527,10 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *tgt_prog, u32 btf_id, struct bpf_attach_target_info *tgt_info); +int bpf_check_attach_model(const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct btf_func_model *fmodel); void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index cbbbf47e1832..fac0c3518add 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13749,6 +13749,35 @@ static int __bpf_check_attach_target(struct bpf_verifier_log *log, return 0; } +int bpf_check_attach_model(const struct bpf_prog *prog, + const struct bpf_prog *tgt_prog, + u32 btf_id, + struct btf_func_model *fmodel) +{ + struct attach_target target = { }; + int ret; + + ret = __bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, &target); + if (ret) + return ret; + + switch (prog->expected_attach_type) { + case BPF_TRACE_RAW_TP: + break; + case BPF_TRACE_ITER: + ret = btf_distill_func_proto(NULL, target.btf, target.t, target.tname, fmodel); + break; + default: + case BPF_MODIFY_RETURN: + case BPF_LSM_MAC: + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: + ret = btf_distill_func_proto(NULL, target.btf, target.t, target.tname, fmodel); + } + + return ret; +} + int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, From patchwork Thu Nov 18 11:24:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626663 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F4F4C433F5 for ; Thu, 18 Nov 2021 11:26:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7AAA661A89 for ; Thu, 18 Nov 2021 11:26:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343933AbhKRL3n (ORCPT ); Thu, 18 Nov 2021 06:29:43 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:24764 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343932AbhKRL2g (ORCPT ); Thu, 18 Nov 2021 06:28:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234736; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UExItWiG/aghgot1jOwsBbuLo+xTDuY4lfZqkstoXa0=; b=dAGphBc1E5hsjhcCPhHq9pmKEUAxeOkeYxXQX1xKOuo4cCXcSDITDENm7JtO5wccQI1oni N8XoH07JRKxgfIdJ/PMDYLUi/Q191UVvkHvN9yMMCNMmH8vz/r3g1LsXue3UC4qiabyKKC cOXlvZph8rW+Q0vIhGgE160dU3BlncY= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-197-gG8H2QqPPi-UxW2qMwEvtg-1; Thu, 18 Nov 2021 06:25:34 -0500 X-MC-Unique: gG8H2QqPPi-UxW2qMwEvtg-1 Received: by mail-ed1-f70.google.com with SMTP id v1-20020aa7cd41000000b003e80973378aso3453690edw.14 for ; Thu, 18 Nov 2021 03:25:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UExItWiG/aghgot1jOwsBbuLo+xTDuY4lfZqkstoXa0=; b=J+iah5wcVQxxujPWbHJEvP1JlFmErhDOGsH76G7h4XMpjj83IA62J+P8Az+mFKdOeL RLM/Z0fYLHLfpfLX831E5yUYr0VXKCML/WIlMk9kHqpC64oARsKRSDxWgDruSoY//WEr c8mf3hJyWvTMrmCZUY7nNcPWgRHu7j4s4e6PvWvYVJNLZWi1yQ85ZjW8AHoyjn8m3ofL dShsdyMukq4nr0NVDbgk0E2+dChVWo3Qv3f/CqjA4bj37igYPhrKxwYAjSH2CrkZ7wYp kR0NESae0C+FFo61oIg1LgRPO7X+HHR5U8Y24aAZ0dCzvsaTEinz2c87BOD5Ixk2zT0O 0O7A== X-Gm-Message-State: AOAM533q5VZO88qFUhsVf6mbIkE0+ruhaeb+Abl9GN7HoKcYn6HnFxtK dzy0JcSQsxTQ2CrJjFwBnwRzJBNdb/KK1HMZeXfHCkp2jzxfPNNzltxq5Fpu5514aeksXx+XG3H 90g9+5QTsMO9b7aRn X-Received: by 2002:a17:906:3b18:: with SMTP id g24mr32631661ejf.27.1637234733380; Thu, 18 Nov 2021 03:25:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJzZcB/WXAUH9R6GLkTBNbO00Tm3vdxO5uMqs2WUCXNQSpxdO8nFLE6w7EitJTSQZsOGZ2Twmg== X-Received: by 2002:a17:906:3b18:: with SMTP id g24mr32631601ejf.27.1637234733071; Thu, 18 Nov 2021 03:25:33 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id e19sm1451053edu.47.2021.11.18.03.25.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:32 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 06/29] bpf: Add bpf_arg/bpf_ret_value helpers for tracing programs Date: Thu, 18 Nov 2021 12:24:32 +0100 Message-Id: <20211118112455.475349-7-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding bpf_arg/bpf_ret_value helpers for tracing programs that returns traced function arguments. Get n-th argument of the traced function: long bpf_arg(void *ctx, int n) Get return value of the traced function: long bpf_ret_value(void *ctx) The trampoline now stores number of arguments on ctx-8 address, so it's easy to verify argument index and find return value argument. Moving function ip address on the trampoline stack behind the number of functions arguments, so it's now stored on ctx-16 address. Both helpers are inlined by verifier. Signed-off-by: Jiri Olsa --- arch/x86/net/bpf_jit_comp.c | 18 +++++++++++--- include/uapi/linux/bpf.h | 14 +++++++++++ kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++-- kernel/trace/bpf_trace.c | 38 +++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 14 +++++++++++ 5 files changed, 122 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 631847907786..67e8ac9aaf0d 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1941,7 +1941,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i void *orig_call) { int ret, i, nr_args = m->nr_args; - int stack_size = nr_args * 8; + int stack_size = nr_args * 8 + 8 /* nr_args */; struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY]; struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT]; struct bpf_tramp_progs *fmod_ret = &tprogs[BPF_TRAMP_MODIFY_RETURN]; @@ -1987,12 +1987,22 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i EMIT4(0x48, 0x83, 0xe8, X86_PATCH_SIZE); emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); - /* Continue with stack_size for regs storage, stack will - * be correctly restored with 'leave' instruction. - */ + /* Continue with stack_size for 'nr_args' storage */ stack_size -= 8; } + /* Store number of arguments of the traced function: + * mov rax, nr_args + * mov QWORD PTR [rbp - stack_size], rax + */ + emit_mov_imm64(&prog, BPF_REG_0, 0, (u32) nr_args); + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_size); + + /* Continue with stack_size for regs storage, stack will + * be correctly restored with 'leave' instruction. + */ + stack_size -= 8; + save_regs(m, &prog, nr_args, stack_size); if (flags & BPF_TRAMP_F_CALL_ORIG) { diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index a69e4b04ffeb..fc8b344eecba 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -4957,6 +4957,18 @@ union bpf_attr { * **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*. * **-EBUSY** if failed to try lock mmap_lock. * **-EINVAL** for invalid **flags**. + * + * long bpf_arg(void *ctx, int n) + * Description + * Get n-th argument of the traced function (for tracing programs). + * Return + * Value of the argument. + * + * long bpf_ret_value(void *ctx) + * Description + * Get return value of the traced function (for tracing programs). + * Return + * Return value of the traced function. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -5140,6 +5152,8 @@ union bpf_attr { FN(skc_to_unix_sock), \ FN(kallsyms_lookup_name), \ FN(find_vma), \ + FN(arg), \ + FN(ret_value), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index fac0c3518add..d4249ef6ca7e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13246,11 +13246,52 @@ static int do_misc_fixups(struct bpf_verifier_env *env) continue; } + /* Implement bpf_arg inline. */ + if (prog_type == BPF_PROG_TYPE_TRACING && + insn->imm == BPF_FUNC_arg) { + /* Load nr_args from ctx - 8 */ + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8); + insn_buf[1] = BPF_JMP32_REG(BPF_JGE, BPF_REG_2, BPF_REG_0, 4); + insn_buf[2] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8); + insn_buf[3] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1); + insn_buf[4] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0); + insn_buf[5] = BPF_JMP_A(1); + insn_buf[6] = BPF_MOV64_IMM(BPF_REG_0, 0); + + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 7); + if (!new_prog) + return -ENOMEM; + + delta += 6; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + continue; + } + + /* Implement bpf_ret_value inline. */ + if (prog_type == BPF_PROG_TYPE_TRACING && + insn->imm == BPF_FUNC_ret_value) { + /* Load nr_args from ctx - 8 */ + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_1, -8); + insn_buf[1] = BPF_ALU64_IMM(BPF_MUL, BPF_REG_2, 8); + insn_buf[2] = BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_1); + insn_buf[3] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, 0); + + new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 4); + if (!new_prog) + return -ENOMEM; + + delta += 3; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + continue; + } + /* Implement bpf_get_func_ip inline. */ if (prog_type == BPF_PROG_TYPE_TRACING && insn->imm == BPF_FUNC_get_func_ip) { - /* Load IP address from ctx - 8 */ - insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8); + /* Load IP address from ctx - 16 */ + insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -16); new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, 1); if (!new_prog) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 25ea521fb8f1..3844cfb45490 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1012,7 +1012,7 @@ const struct bpf_func_proto bpf_snprintf_btf_proto = { BPF_CALL_1(bpf_get_func_ip_tracing, void *, ctx) { /* This helper call is inlined by verifier. */ - return ((u64 *)ctx)[-1]; + return ((u64 *)ctx)[-2]; } static const struct bpf_func_proto bpf_get_func_ip_proto_tracing = { @@ -1091,6 +1091,38 @@ static const struct bpf_func_proto bpf_get_branch_snapshot_proto = { .arg2_type = ARG_CONST_SIZE_OR_ZERO, }; +BPF_CALL_2(bpf_arg, void *, ctx, int, n) +{ + /* This helper call is inlined by verifier. */ + u64 nr_args = ((u64 *)ctx)[-1]; + + if ((u64) n >= nr_args) + return 0; + return ((u64 *)ctx)[n]; +} + +static const struct bpf_func_proto bpf_arg_proto = { + .func = bpf_arg, + .gpl_only = true, + .ret_type = RET_INTEGER, + .arg1_type = ARG_PTR_TO_CTX, + .arg1_type = ARG_ANYTHING, +}; + +BPF_CALL_1(bpf_ret_value, void *, ctx) +{ + /* This helper call is inlined by verifier. */ + u64 nr_args = ((u64 *)ctx)[-1]; + + return ((u64 *)ctx)[nr_args]; +} + +static const struct bpf_func_proto bpf_ret_value_proto = { + .func = bpf_ret_value, + .gpl_only = true, + .ret_type = RET_INTEGER, +}; + static const struct bpf_func_proto * bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) { @@ -1212,6 +1244,10 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) return &bpf_find_vma_proto; case BPF_FUNC_trace_vprintk: return bpf_get_trace_vprintk_proto(); + case BPF_FUNC_arg: + return &bpf_arg_proto; + case BPF_FUNC_ret_value: + return &bpf_ret_value_proto; default: return bpf_base_func_proto(func_id); } diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index a69e4b04ffeb..fc8b344eecba 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -4957,6 +4957,18 @@ union bpf_attr { * **-ENOENT** if *task->mm* is NULL, or no vma contains *addr*. * **-EBUSY** if failed to try lock mmap_lock. * **-EINVAL** for invalid **flags**. + * + * long bpf_arg(void *ctx, int n) + * Description + * Get n-th argument of the traced function (for tracing programs). + * Return + * Value of the argument. + * + * long bpf_ret_value(void *ctx) + * Description + * Get return value of the traced function (for tracing programs). + * Return + * Return value of the traced function. */ #define __BPF_FUNC_MAPPER(FN) \ FN(unspec), \ @@ -5140,6 +5152,8 @@ union bpf_attr { FN(skc_to_unix_sock), \ FN(kallsyms_lookup_name), \ FN(find_vma), \ + FN(arg), \ + FN(ret_value), \ /* */ /* integer value in 'imm' field of BPF_CALL instruction selects which helper From patchwork Thu Nov 18 11:24:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626665 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3023C433EF for ; Thu, 18 Nov 2021 11:26:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A589960D42 for ; Thu, 18 Nov 2021 11:26:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344005AbhKRL3s (ORCPT ); Thu, 18 Nov 2021 06:29:48 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:37489 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343945AbhKRL2l (ORCPT ); Thu, 18 Nov 2021 06:28:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234741; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CFr165JXVPF9+YogBDBOFblIaG2/JxmtopNnbedaeug=; b=i9HhJpQ91OJnTkdcYBy34hlH99eaEbNPia6OJs34KpoLPKval3W2Gn8TTAxRSUZi4rBTWb bzLZKy6yfGoaInZQ+F/duFSSEsgEycYZXbAO9floy3Jl7SdnN6S/wVnky8aQFIYJulhB2D ZjjiCWk5pO2twjek+Y7bNcNq6M8/w6s= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-470-FRo2zJKTPiO9_BbRxlFpJA-1; Thu, 18 Nov 2021 06:25:40 -0500 X-MC-Unique: FRo2zJKTPiO9_BbRxlFpJA-1 Received: by mail-ed1-f70.google.com with SMTP id w18-20020a056402071200b003e61cbafdb4so5001923edx.4 for ; Thu, 18 Nov 2021 03:25:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CFr165JXVPF9+YogBDBOFblIaG2/JxmtopNnbedaeug=; b=JSMPDg8FK97EFiSfX3jKm0czguVp+sitR48K12q3nRmiQk5aHtet9g/+giABv94XXw 3dOMU5gDCy/GBRLy2ixdY06yXT5puZhQ37EKJC0AUOWvtEfJ04pReCGwZ4JEVi/s+5t2 NL8vfBz33bcdh41cqpxx/2NKph84RwRNs2sG4lL7c+7UXH4zyoJr74aLUmGb2oqsCuxG 2a7dzlDYFbezZFWgWrhO0bcmJtxzCGKJTIRPOxDFeqpomQB7fZ6CPuytYxHThJU6NU+h o0Ku0AmbkPtxX5XWDdxDIZ8lMCMxNB2ZtZcDo2RMqbQGKgXnuvoq1UJFSsP0gUU57JCr DMpA== X-Gm-Message-State: AOAM532VZvfawX+rUs1uzK1RGr3/R06uix5HWTQLc/H6zitSn2GH/48d FhjiBzZ4ympsApWWY51gTtYktZuUtqUA1GoACHgRprw8Q2Lw3Dmt9CN8pnR/RIMUvuPE9bs+t47 BUCRZr/pgSM7EMiZj X-Received: by 2002:a05:6402:278e:: with SMTP id b14mr10176262ede.354.1637234739240; Thu, 18 Nov 2021 03:25:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJwGAjtFRa/W1pUQDowtevr45KxolZl62BgECbGerO7oi7QNztVAFKosWxzDXCEbjwZNK6H+qg== X-Received: by 2002:a05:6402:278e:: with SMTP id b14mr10176236ede.354.1637234739120; Thu, 18 Nov 2021 03:25:39 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id q2sm1362615edh.44.2021.11.18.03.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:38 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 07/29] bpf, x64: Allow to use caller address from stack Date: Thu, 18 Nov 2021 12:24:33 +0100 Message-Id: <20211118112455.475349-8-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Currently we call the original function by using the absolute address given at the JIT generation. That's not usable when having trampoline attached to multiple functions. In this case we need to take the return address from the stack. Adding support to retrieve the original function address from the stack by adding new BPF_TRAMP_F_ORIG_STACK flag for arch_prepare_bpf_trampoline function. Basically we take the return address of the 'fentry' call: function + 0: call fentry # stores 'function + 5' address on stack function + 5: ... The 'function + 5' address will be used as the address for the original function to call. Signed-off-by: Jiri Olsa --- arch/x86/net/bpf_jit_comp.c | 13 +++++++++---- include/linux/bpf.h | 5 +++++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 67e8ac9aaf0d..d87001073033 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2035,10 +2035,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (flags & BPF_TRAMP_F_CALL_ORIG) { restore_regs(m, &prog, nr_args, stack_size); - /* call original function */ - if (emit_call(&prog, orig_call, prog)) { - ret = -EINVAL; - goto cleanup; + if (flags & BPF_TRAMP_F_ORIG_STACK) { + emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8); + EMIT2(0xff, 0xd0); /* call *rax */ + } else { + /* call original function */ + if (emit_call(&prog, orig_call, prog)) { + ret = -EINVAL; + goto cleanup; + } } /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index cc7a0c36e7df..77c76e2fa9ff 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -594,6 +594,11 @@ struct btf_func_model { /* Return the return value of fentry prog. Only used by bpf_struct_ops. */ #define BPF_TRAMP_F_RET_FENTRY_RET BIT(4) +/* Get original function from stack instead of from provided direct address. + * Makes sense for fexit programs only. + */ +#define BPF_TRAMP_F_ORIG_STACK BIT(5) + /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. Pick a number to fit into BPF_IMAGE_SIZE / 2 */ From patchwork Thu Nov 18 11:24:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626667 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51888C433F5 for ; Thu, 18 Nov 2021 11:26:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2D1BB61A89 for ; Thu, 18 Nov 2021 11:26:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343945AbhKRL3w (ORCPT ); Thu, 18 Nov 2021 06:29:52 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:53099 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343683AbhKRL2r (ORCPT ); Thu, 18 Nov 2021 06:28:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234747; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pX4ji75/Gajon/2J1LuYrNCwjAQvI0++spwtccRCPdk=; b=SZRTdPvgbcd9+A3fggLc9sKEATV4doR4LcUM5gNs0htdP1UI6uR0Brn7+qCX3i9Vbk2NJH mlBDHpUUJCyqDd2B+k6MpSvCHOleMRd+LtA4yKv6R4QgvUrmlGFbIEu3qFM8Ve0opcwnPj MUS+67ycyAZK2ClMnknv1VcCs3ZyHmI= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-316-wrvB6OLqM4-k1xrnT33GHA-1; Thu, 18 Nov 2021 06:25:46 -0500 X-MC-Unique: wrvB6OLqM4-k1xrnT33GHA-1 Received: by mail-ed1-f71.google.com with SMTP id v22-20020a50a456000000b003e7cbfe3dfeso4982240edb.11 for ; Thu, 18 Nov 2021 03:25:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pX4ji75/Gajon/2J1LuYrNCwjAQvI0++spwtccRCPdk=; b=L3tpnYTBWAnG21cSvm5xq//nCrk8VORj46ccWhMUb2kvyJkY2S/9XdaPTI5UhzfPZL q5g0SNiFtYsdwzCf8GRl2pCknSNf79ERl/8wupwXLpktEIU/DUpFStBrMmScEOWNgak7 jIBb1rqKnsuA1xv58MYOKUCjF7XhrUAgzObK1KPv4U9w0TI3aO/D+wZaGI2vPKPGdeMy jTIMSPXWYc9fEla29ZeABJIh75sOjHEiD8s+beUHcPUe8KeoWLyFtK7jeHmUEFsAdf8Z 3JQanm6//3TtjRzjreDjTZNtCpd7Ry+BU34Y9ualkaPPTPwnqlLMriLpMjGQWjanKBp0 tl5Q== X-Gm-Message-State: AOAM530qctb8C5f7+l1SGahgiA4b828wTbRZqg1yq7jFNueItl77k/ct LEyus/zAM+EzMgH2NF0pDNA7mMHp49zRh4mT+rcokissIgPS2ejp4UiwJsq7GBw/EBVTt5wr7Qd PI8O3jELwGb10YFkv X-Received: by 2002:a17:906:608:: with SMTP id s8mr32950799ejb.405.1637234745253; Thu, 18 Nov 2021 03:25:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJwXium1CcxTPtrQy+FrdiIZ6BCZsyJfc85NeUUE7XrI3XxCDHWOTUp2hMdOjCO6IiOvKBowSw== X-Received: by 2002:a17:906:608:: with SMTP id s8mr32950762ejb.405.1637234745054; Thu, 18 Nov 2021 03:25:45 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id s18sm1583011edd.15.2021.11.18.03.25.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:44 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 08/29] bpf: Keep active attached trampoline in bpf_prog Date: Thu, 18 Nov 2021 12:24:34 +0100 Message-Id: <20211118112455.475349-9-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Keeping active attached trampoline in bpf_prog so it can be used in following changes to account for multiple functions attachments in program. As EXT programs are not going to be supported in multiple functions attachment for now, I'm keeping them stored in link. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 1 + kernel/bpf/syscall.c | 34 +++++++++++++++++++++++++++++----- 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 77c76e2fa9ff..c93c629b5725 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -872,6 +872,7 @@ struct bpf_prog_aux { struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */ struct bpf_prog *dst_prog; struct bpf_trampoline *dst_trampoline; + struct bpf_trampoline *trampoline; enum bpf_prog_type saved_dst_prog_type; enum bpf_attach_type saved_dst_attach_type; bool verifier_zext; /* Zero extensions has been inserted by verifier. */ diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 50f96ea4452a..0df8b2f3d982 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2623,15 +2623,28 @@ struct bpf_tracing_link { struct bpf_prog *tgt_prog; }; +static struct bpf_trampoline *link_trampoline(struct bpf_tracing_link *link) +{ + struct bpf_prog *prog = link->link.prog; + + if (prog->type == BPF_PROG_TYPE_EXT) + return link->trampoline; + else + return prog->aux->trampoline; +} + static void bpf_tracing_link_release(struct bpf_link *link) { struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link); + struct bpf_trampoline *tr = link_trampoline(tr_link); + struct bpf_prog *prog = link->prog; - WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, - tr_link->trampoline)); + WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, tr)); - bpf_trampoline_put(tr_link->trampoline); + if (prog->type != BPF_PROG_TYPE_EXT) + prog->aux->trampoline = NULL; + bpf_trampoline_put(tr); /* tgt_prog is NULL if target is a kernel function */ if (tr_link->tgt_prog) @@ -2662,9 +2675,10 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, { struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link); + struct bpf_trampoline *tr = link_trampoline(tr_link); info->tracing.attach_type = tr_link->attach_type; - bpf_trampoline_unpack_key(tr_link->trampoline->key, + bpf_trampoline_unpack_key(tr->key, &info->tracing.target_obj_id, &info->tracing.target_btf_id); @@ -2682,6 +2696,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, int tgt_prog_fd, u32 btf_id) { + bool prog_extension = prog->type == BPF_PROG_TYPE_EXT; struct bpf_link_primer link_primer; struct bpf_prog *tgt_prog = NULL; struct bpf_trampoline *tr = NULL; @@ -2748,6 +2763,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, mutex_lock(&prog->aux->dst_mutex); + if (!prog_extension && prog->aux->trampoline) { + err = -EBUSY; + goto out_unlock; + } + /* There are a few possible cases here: * * - if prog->aux->dst_trampoline is set, the program was just loaded @@ -2824,7 +2844,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, } link->tgt_prog = tgt_prog; - link->trampoline = tr; + + if (prog_extension) + link->trampoline = tr; + else + prog->aux->trampoline = tr; /* Always clear the trampoline and target prog from prog->aux to make * sure the original attach destination is not kept alive after a From patchwork Thu Nov 18 11:24:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626669 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF2E5C433F5 for ; Thu, 18 Nov 2021 11:27:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C31AB61AFB for ; Thu, 18 Nov 2021 11:27:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343692AbhKRL37 (ORCPT ); Thu, 18 Nov 2021 06:29:59 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46190 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343772AbhKRL2y (ORCPT ); Thu, 18 Nov 2021 06:28:54 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234754; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z1ZiYznp4j141H60TZcw3n4CkxufbqGykQvRZP9LMQI=; b=YbJCxY73et7mwjf07jeOuae7xMzr8R3Odw8Rs4UI1lxcabhFiWvDXpx1M6HQLboqICU9lC 7LO4NFzxh/UFeLovS+N66i3qRpqDFYD5JpyhjG8a1RzLwW69QaLLX07m8gbIIZK6Rv8XeN zIy+i59eJetpMZBOiprPOyjBh2fLiRg= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-414-8qYkc8gKOFS4711_JGV-lw-1; Thu, 18 Nov 2021 06:25:52 -0500 X-MC-Unique: 8qYkc8gKOFS4711_JGV-lw-1 Received: by mail-ed1-f69.google.com with SMTP id b15-20020aa7c6cf000000b003e7cf0f73daso4940482eds.22 for ; Thu, 18 Nov 2021 03:25:52 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z1ZiYznp4j141H60TZcw3n4CkxufbqGykQvRZP9LMQI=; b=GUGj/sxPPMc95raUx+ClF15Zn7k6Sk9tQlEd8gZB3AhpQNDtDf4k/EhsOolWz74HRJ G3zeCkq4VzTnyBrAye62TloAdkkByS69PGM7NJWqXFpdU4YDQoFoEncs89goeehWyTmm I82IcodGA2cJW/uZ1gXi7RiN//r1a0cRLmGAG+layUTv+xZZfunrezKi20Yg4oKSQpip La3U6yWEKVKEKUTUJUnbciwIPF65TH0Z4rui4dOAGNaUQI6gBYxLznsOCnKH4op/vEYO XBsR6wa+QVTIgNgPN7a8fRXIM3zEF7XheeaYZrr/Hub+Gg24zQAVRjeZ1xAXudg+tlPo FXFA== X-Gm-Message-State: AOAM530ciS+7MRjTv9p70DVf2ChQsp2zAZ2GlDNaV8/DNsmddEex947f oTQN/iZxRkpNwc0qIwyRcJ9kkb2Y0M9tA3rt/y+UsMoiB/dc9FzQZbvbfCnE0QPiLA3TwNCk0hz PQgiK9KnbWRSC3D3B X-Received: by 2002:a50:d741:: with SMTP id i1mr10208640edj.37.1637234751204; Thu, 18 Nov 2021 03:25:51 -0800 (PST) X-Google-Smtp-Source: ABdhPJwMf9bx5Vl/rhIkaqyvY0xlKMIIuiWx7PfjU1RfalJ4dgrnynbQis07rnz2NXs0vtexXPh4HQ== X-Received: by 2002:a50:d741:: with SMTP id i1mr10208610edj.37.1637234751044; Thu, 18 Nov 2021 03:25:51 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id w3sm1422085edj.63.2021.11.18.03.25.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:50 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 09/29] bpf: Add support to load multi func tracing program Date: Thu, 18 Nov 2021 12:24:35 +0100 Message-Id: <20211118112455.475349-10-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to load tracing program with new BPF_F_MULTI_FUNC flag, that allows the program to be loaded without specific function to be attached to. Such program will be allowed to be attached to multiple functions in following patches. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 1 + include/uapi/linux/bpf.h | 7 +++++++ kernel/bpf/syscall.c | 35 +++++++++++++++++++++++++++++----- kernel/bpf/verifier.c | 3 ++- tools/include/uapi/linux/bpf.h | 7 +++++++ 5 files changed, 47 insertions(+), 6 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c93c629b5725..35f484f323f3 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -881,6 +881,7 @@ struct bpf_prog_aux { bool func_proto_unreliable; bool sleepable; bool tail_call_reachable; + bool multi_func; struct hlist_node tramp_hlist; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index fc8b344eecba..ca05e35e0478 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1111,6 +1111,13 @@ enum bpf_link_type { */ #define BPF_F_SLEEPABLE (1U << 4) +/* If BPF_F_MULTI_FUNC is used in BPF_PROG_LOAD command, the verifier does + * not expect BTF ID for the program, instead it assumes it's function + * with 6 u64 arguments. No trampoline is created for the program. Such + * program can be attached to multiple functions. + */ +#define BPF_F_MULTI_FUNC (1U << 5) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: * diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0df8b2f3d982..648155f0a4b1 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -31,6 +31,7 @@ #include #include #include +#include #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -2040,7 +2041,8 @@ static int bpf_prog_load_check_attach(enum bpf_prog_type prog_type, enum bpf_attach_type expected_attach_type, struct btf *attach_btf, u32 btf_id, - struct bpf_prog *dst_prog) + struct bpf_prog *dst_prog, + bool multi_func) { if (btf_id) { if (btf_id > BTF_MAX_TYPE) @@ -2060,6 +2062,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, } } + if (multi_func) { + if (prog_type != BPF_PROG_TYPE_TRACING) + return -EINVAL; + if (!attach_btf || btf_id) + return -EINVAL; + return 0; + } + if (attach_btf && (!btf_id || dst_prog)) return -EINVAL; @@ -2183,6 +2193,16 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) } } +#define DEFINE_BPF_MULTI_FUNC(args...) \ + extern int bpf_multi_func(args); \ + int __init bpf_multi_func(args) { return 0; } + +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2, + unsigned long a3, unsigned long a4, + unsigned long a5, unsigned long a6) + +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func) + /* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD fd_array @@ -2193,6 +2213,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) struct btf *attach_btf = NULL; int err; char license[128]; + bool multi_func; bool is_gpl; if (CHECK_ATTR(BPF_PROG_LOAD)) @@ -2202,7 +2223,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) BPF_F_ANY_ALIGNMENT | BPF_F_TEST_STATE_FREQ | BPF_F_SLEEPABLE | - BPF_F_TEST_RND_HI32)) + BPF_F_TEST_RND_HI32 | + BPF_F_MULTI_FUNC)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && @@ -2233,6 +2255,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) if (is_perfmon_prog_type(type) && !perfmon_capable()) return -EPERM; + multi_func = attr->prog_flags & BPF_F_MULTI_FUNC; + /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog * or btf, we need to check which one it is */ @@ -2251,7 +2275,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) return -ENOTSUPP; } } - } else if (attr->attach_btf_id) { + } else if (attr->attach_btf_id || multi_func) { /* fall back to vmlinux BTF, if BTF type ID is specified */ attach_btf = bpf_get_btf_vmlinux(); if (IS_ERR(attach_btf)) @@ -2264,7 +2288,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) bpf_prog_load_fixup_attach_type(attr); if (bpf_prog_load_check_attach(type, attr->expected_attach_type, attach_btf, attr->attach_btf_id, - dst_prog)) { + dst_prog, multi_func)) { if (dst_prog) bpf_prog_put(dst_prog); if (attach_btf) @@ -2284,10 +2308,11 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf = attach_btf; - prog->aux->attach_btf_id = attr->attach_btf_id; + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id; prog->aux->dst_prog = dst_prog; prog->aux->offload_requested = !!attr->prog_ifindex; prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE; + prog->aux->multi_func = multi_func; err = security_bpf_prog_alloc(prog->aux); if (err) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d4249ef6ca7e..af6a39bbb0dc 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13983,7 +13983,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) if (!bpf_iter_prog_supported(prog)) return -EINVAL; return 0; - } + } else if (prog->aux->multi_func) + return prog->type == BPF_PROG_TYPE_TRACING ? 0 : -EINVAL; if (prog->type == BPF_PROG_TYPE_LSM) { ret = bpf_lsm_verify_prog(&env->log, prog); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index fc8b344eecba..ca05e35e0478 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1111,6 +1111,13 @@ enum bpf_link_type { */ #define BPF_F_SLEEPABLE (1U << 4) +/* If BPF_F_MULTI_FUNC is used in BPF_PROG_LOAD command, the verifier does + * not expect BTF ID for the program, instead it assumes it's function + * with 6 u64 arguments. No trampoline is created for the program. Such + * program can be attached to multiple functions. + */ +#define BPF_F_MULTI_FUNC (1U << 5) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: * From patchwork Thu Nov 18 11:24:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626671 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1F49C433EF for ; Thu, 18 Nov 2021 11:27:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D555461A89 for ; Thu, 18 Nov 2021 11:27:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343846AbhKRLaI (ORCPT ); Thu, 18 Nov 2021 06:30:08 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:25990 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343873AbhKRL3A (ORCPT ); Thu, 18 Nov 2021 06:29:00 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ou9vYmCA8xtdoxix/C23Lw9M4Y94LjiNquU/4qPFDn4=; b=L8+CH/eX66ynFqT020zvg5qcd1SWzAmNH3kDHXSK/yK8tTXkQV47CLbZhpHbsrfMVdE6Mt 6OqgYnwR89MXvluHzpjpMHBFHWu8ZffEei8YkiWjkjeWyLvG514/IqCfT2CrLOXR65K48u TxCoHGz69++CL6szdhQM53L5PULS8ps= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-198-4KSYLCcyOBiFDd4cNDvAtA-1; Thu, 18 Nov 2021 06:25:58 -0500 X-MC-Unique: 4KSYLCcyOBiFDd4cNDvAtA-1 Received: by mail-ed1-f71.google.com with SMTP id r16-20020a056402019000b003e6cbb77ed2so4993488edv.10 for ; Thu, 18 Nov 2021 03:25:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ou9vYmCA8xtdoxix/C23Lw9M4Y94LjiNquU/4qPFDn4=; b=EiAAwiC5fBQMjrPFJ+sLpvW+pP3t3+o4M/UWt1shnz0hjoEu6VYj7cE0iRDLIL6Umf RN8Opxle16l7tHQ2+i3UdAO25A7Z6ZPBB4ieKZZ3pxO8Wvdd5W5jTgP/9XdR9kNz7smX WSB3Wk6LO2eRNcBlTmf/+WWdODLqJSQRfGaLfbPOLhtdggpbEO4JZ+b4iWwQs4ly+8iE jb2SC4WGC96JxMqeSbabBjgc5wXr3ZBj3Rpzi7deAveeoZhWOfSkOBQvDyAllJYogntE vXqVjz5bR4CqM+rUnmy1QXzrWqJ6N6FtxKkzhlzOcIQ27ethWVS2wr9YOs8mduQsE8vK nHaA== X-Gm-Message-State: AOAM533w4lCrlQq//tY+iTTC3890kfMsG3DdcNQB4oR8/3mU85XYVb+/ +axWI+kRD7uEn1aeMTGxQ0igFkWR6IM3Le3wz3cQScjjGoKkN8RfO6sXHAkYEiXDoIOm5L2Bz+A TMU000JpBQ/VBT68J X-Received: by 2002:a17:906:dc90:: with SMTP id cs16mr31926203ejc.432.1637234757192; Thu, 18 Nov 2021 03:25:57 -0800 (PST) X-Google-Smtp-Source: ABdhPJwEz7U2o5WUZUjYnrN1aPFkLKsHXVXO/dxfGOUDUALgO5wAoUf0tkvMHTcujM2ZGuZMYPiThA== X-Received: by 2002:a17:906:dc90:: with SMTP id cs16mr31926181ejc.432.1637234756997; Thu, 18 Nov 2021 03:25:56 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id jy28sm1146959ejc.118.2021.11.18.03.25.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:25:56 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 10/29] bpf: Add bpf_trampoline_id object Date: Thu, 18 Nov 2021 12:24:36 +0100 Message-Id: <20211118112455.475349-11-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Replacing the bpf_trampoline's key with struct bpf_tramp_id object, that currently holds only obj_id/btf_id, so same data as key. Having the key in the struct will allow us to add more ids (functions) to single trampoline in following patches. No functional change is intended. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 18 +++++++++-- include/linux/bpf_verifier.h | 19 ------------ kernel/bpf/syscall.c | 30 +++++++++++++----- kernel/bpf/trampoline.c | 59 +++++++++++++++++++++++++++++++----- kernel/bpf/verifier.c | 14 ++++++--- 5 files changed, 99 insertions(+), 41 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 35f484f323f3..2ce8b1c49af7 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -671,13 +671,18 @@ struct bpf_tramp_image { }; }; +struct bpf_tramp_id { + u32 obj_id; + u32 btf_id; +}; + struct bpf_trampoline { /* hlist for trampoline_table */ struct hlist_node hlist; /* serializes access to fields of this trampoline */ struct mutex mutex; refcount_t refcnt; - u64 key; + struct bpf_tramp_id *id; struct { struct btf_func_model model; void *addr; @@ -732,9 +737,16 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( return bpf_func(ctx, insnsi); } #ifdef CONFIG_BPF_JIT +struct bpf_tramp_id *bpf_tramp_id_alloc(void); +void bpf_tramp_id_free(struct bpf_tramp_id *id); +bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id); +int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b); +void bpf_tramp_id_init(struct bpf_tramp_id *id, + const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id); int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); -struct bpf_trampoline *bpf_trampoline_get(u64 key, +struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); #define BPF_DISPATCHER_INIT(_name) { \ @@ -792,7 +804,7 @@ static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, { return -ENOTSUPP; } -static inline struct bpf_trampoline *bpf_trampoline_get(u64 key, +static inline struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, struct bpf_attach_target_info *tgt_info) { return ERR_PTR(-EOPNOTSUPP); diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index b561c0b08e68..0c01a50180a0 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -503,25 +503,6 @@ int check_ctx_reg(struct bpf_verifier_env *env, int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, u32 regno, u32 mem_size); -/* this lives here instead of in bpf.h because it needs to dereference tgt_prog */ -static inline u64 bpf_trampoline_compute_key(const struct bpf_prog *tgt_prog, - struct btf *btf, u32 btf_id) -{ - if (tgt_prog) - return ((u64)tgt_prog->aux->id << 32) | btf_id; - else - return ((u64)btf_obj_id(btf) << 32) | 0x80000000 | btf_id; -} - -/* unpack the IDs from the key as constructed above */ -static inline void bpf_trampoline_unpack_key(u64 key, u32 *obj_id, u32 *btf_id) -{ - if (obj_id) - *obj_id = key >> 32; - if (btf_id) - *btf_id = key & 0x7FFFFFFF; -} - int bpf_check_attach_target(struct bpf_verifier_log *log, const struct bpf_prog *prog, const struct bpf_prog *tgt_prog, diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 648155f0a4b1..f99ea3237f9c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2703,9 +2703,8 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, struct bpf_trampoline *tr = link_trampoline(tr_link); info->tracing.attach_type = tr_link->attach_type; - bpf_trampoline_unpack_key(tr->key, - &info->tracing.target_obj_id, - &info->tracing.target_btf_id); + info->tracing.target_obj_id = tr->id->obj_id; + info->tracing.target_btf_id = tr->id->btf_id; return 0; } @@ -2726,7 +2725,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, struct bpf_prog *tgt_prog = NULL; struct bpf_trampoline *tr = NULL; struct bpf_tracing_link *link; - u64 key = 0; + struct bpf_tramp_id *id = NULL; int err; switch (prog->type) { @@ -2767,6 +2766,12 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } + id = bpf_tramp_id_alloc(); + if (!id) { + err = -ENOMEM; + goto out_put_prog; + } + tgt_prog = bpf_prog_get(tgt_prog_fd); if (IS_ERR(tgt_prog)) { err = PTR_ERR(tgt_prog); @@ -2774,7 +2779,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } - key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id); + bpf_tramp_id_init(id, tgt_prog, NULL, btf_id); } link = kzalloc(sizeof(*link), GFP_USER); @@ -2823,12 +2828,20 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, err = -EINVAL; goto out_unlock; } + + id = bpf_tramp_id_alloc(); + if (!id) { + err = -ENOMEM; + goto out_unlock; + } + btf_id = prog->aux->attach_btf_id; - key = bpf_trampoline_compute_key(NULL, prog->aux->attach_btf, btf_id); + bpf_tramp_id_init(id, NULL, prog->aux->attach_btf, btf_id); } if (!prog->aux->dst_trampoline || - (key && key != prog->aux->dst_trampoline->key)) { + (!bpf_tramp_id_is_empty(id) && + bpf_tramp_id_is_equal(id, prog->aux->dst_trampoline->id))) { /* If there is no saved target, or the specified target is * different from the destination specified at load time, we * need a new trampoline and a check for compatibility @@ -2840,7 +2853,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - tr = bpf_trampoline_get(key, &tgt_info); + tr = bpf_trampoline_get(id, &tgt_info); if (!tr) { err = -ENOMEM; goto out_unlock; @@ -2900,6 +2913,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, out_put_prog: if (tgt_prog_fd && tgt_prog) bpf_prog_put(tgt_prog); + bpf_tramp_id_free(id); return err; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index e98de5e73ba5..ae2573c36653 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -59,16 +59,57 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +static u64 bpf_tramp_id_key(struct bpf_tramp_id *id) +{ + return ((u64) id->obj_id << 32) | id->btf_id; +} + +bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id) +{ + return !id || (!id->obj_id && !id->btf_id); +} + +int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, + struct bpf_tramp_id *b) +{ + return !memcmp(a, b, sizeof(*a)); +} + +struct bpf_tramp_id *bpf_tramp_id_alloc(void) +{ + struct bpf_tramp_id *id; + + return kzalloc(sizeof(*id), GFP_KERNEL); +} + +void bpf_tramp_id_init(struct bpf_tramp_id *id, + const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id) +{ + if (tgt_prog) + id->obj_id = tgt_prog->aux->id; + else + id->obj_id = btf_obj_id(btf); + id->btf_id = btf_id; +} + +void bpf_tramp_id_free(struct bpf_tramp_id *id) +{ + kfree(id); +} + +static struct bpf_trampoline *bpf_trampoline_lookup(struct bpf_tramp_id *id) { struct bpf_trampoline *tr; struct hlist_head *head; + u64 key; int i; + key = bpf_tramp_id_key(id); mutex_lock(&trampoline_mutex); head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; hlist_for_each_entry(tr, head, hlist) { - if (tr->key == key) { + if (bpf_tramp_id_is_equal(tr->id, id)) { refcount_inc(&tr->refcnt); goto out; } @@ -77,7 +118,7 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) if (!tr) goto out; - tr->key = key; + tr->id = id; INIT_HLIST_NODE(&tr->hlist); hlist_add_head(&tr->hlist, head); refcount_set(&tr->refcnt, 1); @@ -291,12 +332,14 @@ static void bpf_tramp_image_put(struct bpf_tramp_image *im) call_rcu_tasks_trace(&im->rcu, __bpf_tramp_image_put_rcu_tasks); } -static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx) +static struct bpf_tramp_image* +bpf_tramp_image_alloc(struct bpf_tramp_id *id, u32 idx) { struct bpf_tramp_image *im; struct bpf_ksym *ksym; void *image; int err = -ENOMEM; + u64 key; im = kzalloc(sizeof(*im), GFP_KERNEL); if (!im) @@ -317,6 +360,7 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx) ksym = &im->ksym; INIT_LIST_HEAD_RCU(&ksym->lnode); + key = bpf_tramp_id_key(id); snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u", key, idx); bpf_image_ksym_add(image, ksym); return im; @@ -351,7 +395,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) goto out; } - im = bpf_tramp_image_alloc(tr->key, tr->selector); + im = bpf_tramp_image_alloc(tr->id, tr->selector); if (IS_ERR(im)) { err = PTR_ERR(im); goto out; @@ -482,12 +526,12 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) return err; } -struct bpf_trampoline *bpf_trampoline_get(u64 key, +struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, struct bpf_attach_target_info *tgt_info) { struct bpf_trampoline *tr; - tr = bpf_trampoline_lookup(key); + tr = bpf_trampoline_lookup(id); if (!tr) return NULL; @@ -521,6 +565,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) * multiple rcu callbacks. */ hlist_del(&tr->hlist); + bpf_tramp_id_free(tr->id); kfree(tr); out: mutex_unlock(&trampoline_mutex); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index af6a39bbb0dc..a1e4389b0e9e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13929,8 +13929,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) struct bpf_attach_target_info tgt_info = {}; u32 btf_id = prog->aux->attach_btf_id; struct bpf_trampoline *tr; + struct bpf_tramp_id *id; int ret; - u64 key; if (prog->type == BPF_PROG_TYPE_SYSCALL) { if (prog->aux->sleepable) @@ -13995,11 +13995,17 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -EINVAL; } - key = bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_id); - tr = bpf_trampoline_get(key, &tgt_info); - if (!tr) + id = bpf_tramp_id_alloc(); + if (!id) return -ENOMEM; + bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id); + tr = bpf_trampoline_get(id, &tgt_info); + if (!tr) { + bpf_tramp_id_free(id); + return -ENOMEM; + } + prog->aux->dst_trampoline = tr; return 0; } From patchwork Thu Nov 18 11:24:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626673 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A442C433F5 for ; Thu, 18 Nov 2021 11:27:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2C0D461A89 for ; Thu, 18 Nov 2021 11:27:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344054AbhKRLac (ORCPT ); Thu, 18 Nov 2021 06:30:32 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:57105 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343897AbhKRL3F (ORCPT ); Thu, 18 Nov 2021 06:29:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234765; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XypHiVckCtchSMS5k0zGXWKdvhQKbyQpumDiEMgd5B4=; b=BHb6zgQHf/ZzgH5m9Xl9VxPP0jsWElEhylqJPQNuRTIJCr9uBlm8iukF86AdNZDwOjYe+6 tOrvFRGz66dDgo6K1L85c0s9Ebk6nQTPyCmyHGvr3OUonG6ADjjUaruaBE2nXpHGoRItVh Ju9KuDiT5JOajaJiTZrlMk57SH3DSyY= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-3-TFwWG8_-P9mEX8ZKoKO53Q-1; Thu, 18 Nov 2021 06:26:04 -0500 X-MC-Unique: TFwWG8_-P9mEX8ZKoKO53Q-1 Received: by mail-ed1-f72.google.com with SMTP id m8-20020a056402510800b003e29de5badbso4963014edd.18 for ; Thu, 18 Nov 2021 03:26:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XypHiVckCtchSMS5k0zGXWKdvhQKbyQpumDiEMgd5B4=; b=1axSEXRfmXmfPEAsleQAwLzVTETCv/e/5HWzKECcwwsNDIawEw44ZqFmm8Dmd815aj pjaAzF1YZkFaEkABECWMcTHoDiLNbYLvs1xY7kkgMmrzQJDwj4wKpFjjR6XMf6fb0QxL UJhhPUpUITIagXTPcOGbfnB+Fba18Uzinkxm5UB7J8oQ3lrxATC9e9dgJ7Hj6Pu828Vu XQR4hZ1I4bolzWw5Lu/ctJKb94GEIqnkHUQizeJZ7ZBzrzShMwSNrEfU3NX92w7e8OiO YSTnFA+VrnNRhFY/USomVetlx5Pwnha80nrWEd9r27NVp9AmUsYCoVCkBQIg9MP9dV0D m8FQ== X-Gm-Message-State: AOAM530220vjVOsp3YiDeqfNJlVegjRSd50Dgd85BS5YQnEbXvBn8E4i WBRYbK+ojqrajV0dzKU3718l5eXK094psyTdLjncEJmPHukwIJcCvBHA2lWN8N5aBOaLXGnm9hB wwjjlioRbf0X9awvv X-Received: by 2002:a17:906:24ca:: with SMTP id f10mr32075642ejb.144.1637234763252; Thu, 18 Nov 2021 03:26:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJxBQI73h/Vg62/lRu8DgzrcfesdyULZXRPaNgUlfBPzFYq2vuDj6EuY7Fwvl9ivPQRcK8lGHw== X-Received: by 2002:a17:906:24ca:: with SMTP id f10mr32075597ejb.144.1637234763001; Thu, 18 Nov 2021 03:26:03 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id s3sm1146445ejm.49.2021.11.18.03.26.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:02 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 11/29] bpf: Add addr to bpf_trampoline_id object Date: Thu, 18 Nov 2021 12:24:37 +0100 Message-Id: <20211118112455.475349-12-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding addr to bpf_trampoline_id object so it's not associated directly with trampoline directly. This will help us to easily support multiple ids/addresses support for trampolines coming in following changes. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 4 ++-- kernel/bpf/trampoline.c | 18 +++++++++--------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2ce8b1c49af7..c57141a76e7b 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -674,6 +674,7 @@ struct bpf_tramp_image { struct bpf_tramp_id { u32 obj_id; u32 btf_id; + void *addr; }; struct bpf_trampoline { @@ -685,11 +686,10 @@ struct bpf_trampoline { struct bpf_tramp_id *id; struct { struct btf_func_model model; - void *addr; bool ftrace_managed; } func; /* if !NULL this is BPF_PROG_TYPE_EXT program that extends another BPF - * program by replacing one of its functions. func.addr is the address + * program by replacing one of its functions. id->addr is the address * of the function it replaced. */ struct bpf_prog *extension_prog; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index ae2573c36653..e19c5112be67 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -136,7 +136,7 @@ static int bpf_trampoline_module_get(struct bpf_trampoline *tr) int err = 0; preempt_disable(); - mod = __module_text_address((unsigned long) tr->func.addr); + mod = __module_text_address((unsigned long) tr->id->addr); if (mod && !try_module_get(mod)) err = -ENOENT; preempt_enable(); @@ -164,7 +164,7 @@ static int is_ftrace_location(void *ip) static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) { - void *ip = tr->func.addr; + void *ip = tr->id->addr; int ret; if (tr->func.ftrace_managed) @@ -179,7 +179,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_addr) { - void *ip = tr->func.addr; + void *ip = tr->id->addr; int ret; if (tr->func.ftrace_managed) @@ -192,7 +192,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad /* first time registering */ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) { - void *ip = tr->func.addr; + void *ip = tr->id->addr; int ret; ret = is_ftrace_location(ip); @@ -410,7 +410,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) err = arch_prepare_bpf_trampoline(im, im->image, im->image + PAGE_SIZE, &tr->func.model, flags, tprogs, - tr->func.addr); + tr->id->addr); if (err < 0) goto out; @@ -478,7 +478,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) goto out; } tr->extension_prog = prog; - err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL, + err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP, NULL, prog->bpf_func); goto out; } @@ -513,7 +513,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) mutex_lock(&tr->mutex); if (kind == BPF_TRAMP_REPLACE) { WARN_ON_ONCE(!tr->extension_prog); - err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, + err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP, tr->extension_prog->bpf_func, NULL); tr->extension_prog = NULL; goto out; @@ -536,11 +536,11 @@ struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, return NULL; mutex_lock(&tr->mutex); - if (tr->func.addr) + if (tr->id->addr) goto out; memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel)); - tr->func.addr = (void *)tgt_info->tgt_addr; + tr->id->addr = (void *)tgt_info->tgt_addr; out: mutex_unlock(&tr->mutex); return tr; From patchwork Thu Nov 18 11:24:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626675 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A475C433F5 for ; Thu, 18 Nov 2021 11:28:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 52D7360D42 for ; Thu, 18 Nov 2021 11:28:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344062AbhKRLa6 (ORCPT ); Thu, 18 Nov 2021 06:30:58 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22395 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343938AbhKRL3M (ORCPT ); Thu, 18 Nov 2021 06:29:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W/94uMDhmykTTrRJ70E8PPEqUT/kiBmU37LOMy0UW0k=; b=e6lkb8NaLPIx6Vms0DOCWPdoA1wP8UXbp4Uhn966fqi85lMIXMsfXsDyUQf2+gK1xI+4Fw n3dgkoUYSSUfRioScDkw7sIFR9+jLFMxY4qfsceV/6kv6DmgzqhsiYZG6dIw8AujFPNonn ABvUPsQc7/7Epc3dFcsb4wVF9337Wew= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-6-gDoV4YSVNo-zsa7vBYAdGA-1; Thu, 18 Nov 2021 06:26:11 -0500 X-MC-Unique: gDoV4YSVNo-zsa7vBYAdGA-1 Received: by mail-ed1-f71.google.com with SMTP id t9-20020aa7d709000000b003e83403a5cbso3026289edq.19 for ; Thu, 18 Nov 2021 03:26:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=W/94uMDhmykTTrRJ70E8PPEqUT/kiBmU37LOMy0UW0k=; b=NQjcyCWYZlUQVr6w+mE83Fnx1vHT0/Za6t9x9HdIkhGhAWE5F+UnWfThlaseKp9Iee hy+L6/1mbP7GovlA3ucuD+KtVW9/VKasCcnKMYC/cK2zwnVbrg1MaR5+tLdG3t3UUYQ6 2lI5aDhfnCLwhmq3WMCbDgFQUG9nwRd8oyIDOVEz6qCzZ79ft2EX9/HaKyp15QFW1vNR ZX/AIj1dLOoktn0qvuvcpnkeoZAZ6FrHd72Y3cRqHksBBqjLhoUXKz2sXDVoPCAgOxIh D0/dOd+mo6V3cF6AAYtMf7XxCklLs+k0KC8cgTJRYgJ9uERJebvEiswrVLOIrIsj1oiJ jrLA== X-Gm-Message-State: AOAM5324+4baCT2aZjMSJ/YDGBPAC4urKUkjDiR8WD/L9u5dM3nvhRx0 CU9LoAeeOm6PG1lpa/uYxLf+wTYW46+uVLVkY4SuFsu3BY3g9bGebRoeppsMfQX+3p5J6da9fv+ wh8lpvHbTkBl5B8EP X-Received: by 2002:a05:6402:44c:: with SMTP id p12mr10133424edw.234.1637234769182; Thu, 18 Nov 2021 03:26:09 -0800 (PST) X-Google-Smtp-Source: ABdhPJxiIhF7LjLYWcPNMq+AEu4aeJP6jDF460Wr6DkNxnkAsEmZPCMFB31tQ81YrNUZjw2dGUdMZA== X-Received: by 2002:a05:6402:44c:: with SMTP id p12mr10133389edw.234.1637234768986; Thu, 18 Nov 2021 03:26:08 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id n1sm1457459edf.45.2021.11.18.03.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:08 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 12/29] bpf: Add struct bpf_tramp_node layer Date: Thu, 18 Nov 2021 12:24:38 +0100 Message-Id: <20211118112455.475349-13-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Currently each trampoline holds a list of programs that are attached to it. With multi func attach support we need a way for a single program to be connected to multiple trampolines. Adding struct bpf_tramp_node object that holds bpf_prog pointer, so it can be resolved directly. We can now have multiple struct bpf_tramp_node being attached to different trampolines pointing to single bpf_prog. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 15 ++++++++++----- kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 4 ++-- kernel/bpf/trampoline.c | 22 ++++++++++++---------- 4 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index c57141a76e7b..21f8dbcf3f48 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -677,6 +677,11 @@ struct bpf_tramp_id { void *addr; }; +struct bpf_tramp_node { + struct hlist_node hlist_tramp; + struct bpf_prog *prog; +}; + struct bpf_trampoline { /* hlist for trampoline_table */ struct hlist_node hlist; @@ -744,8 +749,8 @@ int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b); void bpf_tramp_id_init(struct bpf_tramp_id *id, const struct bpf_prog *tgt_prog, struct btf *btf, u32 btf_id); -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); +int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); +int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); @@ -794,12 +799,12 @@ void bpf_ksym_del(struct bpf_ksym *ksym); int bpf_jit_charge_modmem(u32 pages); void bpf_jit_uncharge_modmem(u32 pages); #else -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, +static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { return -ENOTSUPP; } -static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { return -ENOTSUPP; @@ -894,7 +899,7 @@ struct bpf_prog_aux { bool sleepable; bool tail_call_reachable; bool multi_func; - struct hlist_node tramp_hlist; + struct bpf_tramp_node tramp_node; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index b52dc845ecea..2eed91153a3f 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -105,6 +105,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag fp->aux = aux; fp->aux->prog = fp; fp->jit_requested = ebpf_jit_enabled(); + fp->aux->tramp_node.prog = fp; INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); mutex_init(&fp->aux->used_maps_mutex); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index f99ea3237f9c..0d916e3b7676 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2665,7 +2665,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) struct bpf_trampoline *tr = link_trampoline(tr_link); struct bpf_prog *prog = link->prog; - WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, tr)); + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&link->prog->aux->tramp_node, tr)); if (prog->type != BPF_PROG_TYPE_EXT) prog->aux->trampoline = NULL; @@ -2874,7 +2874,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - err = bpf_trampoline_link_prog(prog, tr); + err = bpf_trampoline_link_prog(&prog->aux->tramp_node, tr); if (err) { bpf_link_cleanup(&link_primer); link = NULL; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index e19c5112be67..b6af3e0982d7 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -216,8 +216,8 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) static struct bpf_tramp_progs * bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) { - const struct bpf_prog_aux *aux; struct bpf_tramp_progs *tprogs; + struct bpf_tramp_node *node; struct bpf_prog **progs; int kind; @@ -231,9 +231,9 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a *total += tr->progs_cnt[kind]; progs = tprogs[kind].progs; - hlist_for_each_entry(aux, &tr->progs_hlist[kind], tramp_hlist) { - *ip_arg |= aux->prog->call_get_func_ip; - *progs++ = aux->prog; + hlist_for_each_entry(node, &tr->progs_hlist[kind], hlist_tramp) { + *ip_arg |= node->prog->call_get_func_ip; + *progs++ = node->prog; } } return tprogs; @@ -455,8 +455,9 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) +int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { + struct bpf_prog *prog = node->prog; enum bpf_tramp_prog_type kind; int err = 0; int cnt; @@ -486,16 +487,16 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) err = -E2BIG; goto out; } - if (!hlist_unhashed(&prog->aux->tramp_hlist)) { + if (!hlist_unhashed(&node->hlist_tramp)) { /* prog already linked */ err = -EBUSY; goto out; } - hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]); + hlist_add_head(&node->hlist_tramp, &tr->progs_hlist[kind]); tr->progs_cnt[kind]++; err = bpf_trampoline_update(tr); if (err) { - hlist_del_init(&prog->aux->tramp_hlist); + hlist_del_init(&node->hlist_tramp); tr->progs_cnt[kind]--; } out: @@ -504,8 +505,9 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) } /* bpf_trampoline_unlink_prog() should never fail. */ -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) +int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { + struct bpf_prog *prog = node->prog; enum bpf_tramp_prog_type kind; int err; @@ -518,7 +520,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) tr->extension_prog = NULL; goto out; } - hlist_del_init(&prog->aux->tramp_hlist); + hlist_del_init(&node->hlist_tramp); tr->progs_cnt[kind]--; err = bpf_trampoline_update(tr); out: From patchwork Thu Nov 18 11:24:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626677 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42993C433EF for ; Thu, 18 Nov 2021 11:28:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2D4A760D42 for ; Thu, 18 Nov 2021 11:28:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344069AbhKRLbN (ORCPT ); Thu, 18 Nov 2021 06:31:13 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:51715 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343949AbhKRL3S (ORCPT ); Thu, 18 Nov 2021 06:29:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=saUFETjyBrBLEue6scFG9UfDs+1LHN9Kj7f1do4qA6U=; b=RZRWvN6YN67hs3j7xTuVjDGqHpZeepEuyKK/wAudhBvWaZTx5y5AEgJwjjuu4su2orZX21 7zzjgK4vOJEH52mCgQcD5zr7vZQsJZOtIK3rJXZTb56aibzNfb24LLhQmLHiT7fHFo5gNJ j4M1OZQXIdjwf4whCnst4HgUsmwR6V8= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-475-3EM7ErLWPxarSiEht7sV7w-1; Thu, 18 Nov 2021 06:26:16 -0500 X-MC-Unique: 3EM7ErLWPxarSiEht7sV7w-1 Received: by mail-ed1-f72.google.com with SMTP id p4-20020aa7d304000000b003e7ef120a37so4972956edq.16 for ; Thu, 18 Nov 2021 03:26:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=saUFETjyBrBLEue6scFG9UfDs+1LHN9Kj7f1do4qA6U=; b=R56w72RMygyS3YBDkBE98cTuqUq2gsHxrcVvnwfaIq0M/sayuP/ElN9stzgQSE3j36 kzHuq+cTZg71j2xw/xEdrmbVpyjCJsYHlTjRGv0cVbzkp/oD3baWu3j8Dq+4pnSbL6SA YlAlRA4iy1VdC41e3AKtCUXRFtHsmKL3MC0lA15w9VxPVYpYBehHXVLG9tORBPOIOnEu S2ppFHxsnrg163ikXMa2FIHMb2vbljunSjIKcrSW4rDV6cqrRWbz7ULv1HK0Xz3Y5+aq uANfc1NetrfOCfLaPHpGkkZtjiPvDHe5/h1H18HQlFUPr5ifgxHjmEUmKEGrH2+o1wMr 4jWg== X-Gm-Message-State: AOAM531Ae4YSv8cB2uhMFrKQbv6hQraflGmW5iRzl2fmeD2L2mmK6Pjc +BQRVWx9hy1/vfsBSnj2zLQh8ogj5gRz/llY6y8WYtbmENR0jGQFcw0fNmq3wfnJfidWzz/g/UU yOUtpAk3PqBYFZrWF X-Received: by 2002:a05:6402:430e:: with SMTP id m14mr10206840edc.93.1637234775325; Thu, 18 Nov 2021 03:26:15 -0800 (PST) X-Google-Smtp-Source: ABdhPJzsH1mMFGHxfgayiYeeuaUjyG6xuSK7HKrO9rKbFyOF9P4GU3y1M+Rfv+jh1+A/WoSom5RcNA== X-Received: by 2002:a05:6402:430e:: with SMTP id m14mr10206785edc.93.1637234775050; Thu, 18 Nov 2021 03:26:15 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id l16sm567137edb.59.2021.11.18.03.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:14 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 13/29] bpf: Add bpf_tramp_attach layer for trampoline attachment Date: Thu, 18 Nov 2021 12:24:39 +0100 Message-Id: <20211118112455.475349-14-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding bpf_tramp_attach layer for trampoline attachment to have extra layer on top of the trampoline. The reason is that in following changes we will add multiple trampolines for single program and we need entity to hold them. The api in nutshell: - each bpf_prog holds 'bpf_tramp_attach' object, which holds list of 'struct bpf_tramp_node' objects: struct bpf_tramp_attach { struct bpf_tramp_id *id; struct hlist_head nodes; }; This allow us to hold multiple trampolines for each program. - bpf_tramp_attach returns 'bpf_tramp_attach' object that finds trampoline for given 'id' and adds it to the attach object, no actuall program attachment is done, just trampoline allocation - bpf_tramp_attach_link does the actual attachment of the program to trampoline - bpf_tramp_attach_unlink unlinks all the trampolines present in the attach object - bpf_tramp_detach frees all the trampolines in attach object Currently there'll be only single node added in attach object. Following patches add support for multiple id trampolines, and uses multiple nodes in attach object to hold trampoline for given program. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 21 +++++-- kernel/bpf/core.c | 5 +- kernel/bpf/syscall.c | 61 ++++++++++---------- kernel/bpf/trampoline.c | 122 ++++++++++++++++++++++++++++++++-------- kernel/bpf/verifier.c | 12 ++-- 5 files changed, 156 insertions(+), 65 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 21f8dbcf3f48..2dbc00904a84 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -679,7 +679,14 @@ struct bpf_tramp_id { struct bpf_tramp_node { struct hlist_node hlist_tramp; + struct hlist_node hlist_attach; struct bpf_prog *prog; + struct bpf_trampoline *tr; +}; + +struct bpf_tramp_attach { + struct bpf_tramp_id *id; + struct hlist_head nodes; }; struct bpf_trampoline { @@ -751,9 +758,14 @@ void bpf_tramp_id_init(struct bpf_tramp_id *id, struct btf *btf, u32 btf_id); int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); -struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, - struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); + +struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, + struct bpf_prog *tgt_prog, + struct bpf_prog *prog); +void bpf_tramp_detach(struct bpf_tramp_attach *attach); +int bpf_tramp_attach_link(struct bpf_tramp_attach *attach); +int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach); #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \ @@ -888,8 +900,8 @@ struct bpf_prog_aux { const struct bpf_ctx_arg_aux *ctx_arg_info; struct mutex dst_mutex; /* protects dst_* pointers below, *after* prog becomes visible */ struct bpf_prog *dst_prog; - struct bpf_trampoline *dst_trampoline; - struct bpf_trampoline *trampoline; + struct bpf_tramp_attach *dst_attach; + struct bpf_tramp_attach *attach; enum bpf_prog_type saved_dst_prog_type; enum bpf_attach_type saved_dst_attach_type; bool verifier_zext; /* Zero extensions has been inserted by verifier. */ @@ -899,7 +911,6 @@ struct bpf_prog_aux { bool sleepable; bool tail_call_reachable; bool multi_func; - struct bpf_tramp_node tramp_node; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2eed91153a3f..993ae224e371 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -105,7 +105,6 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag fp->aux = aux; fp->aux->prog = fp; fp->jit_requested = ebpf_jit_enabled(); - fp->aux->tramp_node.prog = fp; INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); mutex_init(&fp->aux->used_maps_mutex); @@ -2284,8 +2283,8 @@ static void bpf_prog_free_deferred(struct work_struct *work) if (aux->prog->has_callchain_buf) put_callchain_buffers(); #endif - if (aux->dst_trampoline) - bpf_trampoline_put(aux->dst_trampoline); + if (aux->dst_attach) + bpf_tramp_detach(aux->dst_attach); for (i = 0; i < aux->func_cnt; i++) { /* We can just unlink the subprog poke descriptor table as * it was originally linked to the main program and is also diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0d916e3b7676..a65c1862ab68 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2644,32 +2644,32 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) struct bpf_tracing_link { struct bpf_link link; enum bpf_attach_type attach_type; - struct bpf_trampoline *trampoline; + struct bpf_tramp_attach *attach; struct bpf_prog *tgt_prog; }; -static struct bpf_trampoline *link_trampoline(struct bpf_tracing_link *link) +static struct bpf_tramp_attach *link_attach(struct bpf_tracing_link *link) { struct bpf_prog *prog = link->link.prog; if (prog->type == BPF_PROG_TYPE_EXT) - return link->trampoline; + return link->attach; else - return prog->aux->trampoline; + return prog->aux->attach; } static void bpf_tracing_link_release(struct bpf_link *link) { struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link); - struct bpf_trampoline *tr = link_trampoline(tr_link); + struct bpf_tramp_attach *attach = link_attach(tr_link); struct bpf_prog *prog = link->prog; - WARN_ON_ONCE(bpf_trampoline_unlink_prog(&link->prog->aux->tramp_node, tr)); + WARN_ON_ONCE(bpf_tramp_attach_unlink(attach)); if (prog->type != BPF_PROG_TYPE_EXT) - prog->aux->trampoline = NULL; - bpf_trampoline_put(tr); + prog->aux->attach = NULL; + bpf_tramp_detach(attach); /* tgt_prog is NULL if target is a kernel function */ if (tr_link->tgt_prog) @@ -2700,11 +2700,11 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, { struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link); - struct bpf_trampoline *tr = link_trampoline(tr_link); + struct bpf_tramp_attach *attach = link_attach(tr_link); info->tracing.attach_type = tr_link->attach_type; - info->tracing.target_obj_id = tr->id->obj_id; - info->tracing.target_btf_id = tr->id->btf_id; + info->tracing.target_obj_id = attach->id->obj_id; + info->tracing.target_btf_id = attach->id->btf_id; return 0; } @@ -2721,9 +2721,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, u32 btf_id) { bool prog_extension = prog->type == BPF_PROG_TYPE_EXT; + struct bpf_tramp_attach *attach = NULL; struct bpf_link_primer link_primer; struct bpf_prog *tgt_prog = NULL; - struct bpf_trampoline *tr = NULL; struct bpf_tracing_link *link; struct bpf_tramp_id *id = NULL; int err; @@ -2793,7 +2793,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, mutex_lock(&prog->aux->dst_mutex); - if (!prog_extension && prog->aux->trampoline) { + if (!prog_extension && prog->aux->attach) { err = -EBUSY; goto out_unlock; } @@ -2816,7 +2816,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, * - if prog->aux->dst_trampoline and tgt_prog is NULL, the program * was detached and is going for re-attachment. */ - if (!prog->aux->dst_trampoline && !tgt_prog) { + if (!prog->aux->dst_attach && !tgt_prog) { /* * Allow re-attach for TRACING and LSM programs. If it's * currently linked, bpf_trampoline_link_prog will fail. @@ -2839,9 +2839,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, bpf_tramp_id_init(id, NULL, prog->aux->attach_btf, btf_id); } - if (!prog->aux->dst_trampoline || + if (!prog->aux->dst_attach || (!bpf_tramp_id_is_empty(id) && - bpf_tramp_id_is_equal(id, prog->aux->dst_trampoline->id))) { + bpf_tramp_id_is_equal(id, prog->aux->dst_attach->id))) { /* If there is no saved target, or the specified target is * different from the destination specified at load time, we * need a new trampoline and a check for compatibility @@ -2853,9 +2853,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - tr = bpf_trampoline_get(id, &tgt_info); - if (!tr) { - err = -ENOMEM; + id->addr = (void *) tgt_info.tgt_addr; + + attach = bpf_tramp_attach(id, tgt_prog, prog); + if (IS_ERR(attach)) { + err = PTR_ERR(attach); goto out_unlock; } } else { @@ -2866,7 +2868,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, * can only happen once for any program, as the saved values in * prog->aux are cleared below. */ - tr = prog->aux->dst_trampoline; + attach = prog->aux->dst_attach; tgt_prog = prog->aux->dst_prog; } @@ -2874,7 +2876,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - err = bpf_trampoline_link_prog(&prog->aux->tramp_node, tr); + err = bpf_tramp_attach_link(attach); if (err) { bpf_link_cleanup(&link_primer); link = NULL; @@ -2882,32 +2884,31 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, } link->tgt_prog = tgt_prog; - if (prog_extension) - link->trampoline = tr; + link->attach = attach; else - prog->aux->trampoline = tr; + prog->aux->attach = attach; /* Always clear the trampoline and target prog from prog->aux to make * sure the original attach destination is not kept alive after a * program is (re-)attached to another target. */ if (prog->aux->dst_prog && - (tgt_prog_fd || tr != prog->aux->dst_trampoline)) + (tgt_prog_fd || attach != prog->aux->dst_attach)) /* got extra prog ref from syscall, or attaching to different prog */ bpf_prog_put(prog->aux->dst_prog); - if (prog->aux->dst_trampoline && tr != prog->aux->dst_trampoline) + if (prog->aux->dst_attach && attach != prog->aux->dst_attach) /* we allocated a new trampoline, so free the old one */ - bpf_trampoline_put(prog->aux->dst_trampoline); + bpf_tramp_detach(prog->aux->dst_attach); prog->aux->dst_prog = NULL; - prog->aux->dst_trampoline = NULL; + prog->aux->dst_attach = NULL; mutex_unlock(&prog->aux->dst_mutex); return bpf_link_settle(&link_primer); out_unlock: - if (tr && tr != prog->aux->dst_trampoline) - bpf_trampoline_put(tr); + if (attach && attach != prog->aux->dst_attach) + bpf_tramp_detach(attach); mutex_unlock(&prog->aux->dst_mutex); kfree(link); out_put_prog: diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index b6af3e0982d7..16fc4c14319b 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -11,6 +11,7 @@ #include #include #include +#include /* dummy _ops. The verifier will operate on target program's ops. */ const struct bpf_verifier_ops bpf_extension_verifier_ops = { @@ -98,7 +99,7 @@ void bpf_tramp_id_free(struct bpf_tramp_id *id) kfree(id); } -static struct bpf_trampoline *bpf_trampoline_lookup(struct bpf_tramp_id *id) +static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id) { struct bpf_trampoline *tr; struct hlist_head *head; @@ -528,26 +529,6 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin return err; } -struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, - struct bpf_attach_target_info *tgt_info) -{ - struct bpf_trampoline *tr; - - tr = bpf_trampoline_lookup(id); - if (!tr) - return NULL; - - mutex_lock(&tr->mutex); - if (tr->id->addr) - goto out; - - memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel)); - tr->id->addr = (void *)tgt_info->tgt_addr; -out: - mutex_unlock(&tr->mutex); - return tr; -} - void bpf_trampoline_put(struct bpf_trampoline *tr) { if (!tr) @@ -567,12 +548,109 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) * multiple rcu callbacks. */ hlist_del(&tr->hlist); - bpf_tramp_id_free(tr->id); kfree(tr); out: mutex_unlock(&trampoline_mutex); } +static struct bpf_tramp_node *node_alloc(struct bpf_trampoline *tr, struct bpf_prog *prog) +{ + struct bpf_tramp_node *node; + + node = kzalloc(sizeof(*node), GFP_KERNEL); + if (!node) + return NULL; + + INIT_HLIST_NODE(&node->hlist_tramp); + INIT_HLIST_NODE(&node->hlist_attach); + node->prog = prog; + node->tr = tr; + return node; +} + +static void node_free(struct bpf_tramp_node *node) +{ + bpf_trampoline_put(node->tr); + kfree(node); +} + +struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, + struct bpf_prog *tgt_prog, + struct bpf_prog *prog) +{ + struct bpf_trampoline *tr = NULL; + struct bpf_tramp_attach *attach; + struct bpf_tramp_node *node; + int err; + + attach = kzalloc(sizeof(*attach), GFP_KERNEL); + if (!attach) + return ERR_PTR(-ENOMEM); + + tr = bpf_trampoline_get(id); + if (!tr) { + err = -ENOMEM; + goto out; + } + + node = node_alloc(tr, prog); + if (!node) + goto out; + + err = bpf_check_attach_model(prog, tgt_prog, id->btf_id, &tr->func.model); + if (err) + goto out; + + attach->id = id; + hlist_add_head(&node->hlist_attach, &attach->nodes); + return attach; + +out: + bpf_trampoline_put(tr); + kfree(attach); + return ERR_PTR(err); +} + +void bpf_tramp_detach(struct bpf_tramp_attach *attach) +{ + struct bpf_tramp_node *node; + struct hlist_node *n; + + hlist_for_each_entry_safe(node, n, &attach->nodes, hlist_attach) + node_free(node); + + bpf_tramp_id_free(attach->id); + kfree(attach); +} + +int bpf_tramp_attach_link(struct bpf_tramp_attach *attach) +{ + struct bpf_tramp_node *node; + int err; + + hlist_for_each_entry(node, &attach->nodes, hlist_attach) { + err = bpf_trampoline_link_prog(node, node->tr); + if (err) + return err; + } + + return 0; +} + +int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach) +{ + struct bpf_tramp_node *node; + int err; + + hlist_for_each_entry(node, &attach->nodes, hlist_attach) { + err = bpf_trampoline_unlink_prog(node, node->tr); + if (err) + return err; + } + + return 0; +} + #define NO_START_TIME 1 static __always_inline u64 notrace bpf_prog_start_time(void) { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a1e4389b0e9e..e05f39fd2708 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13928,7 +13928,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) struct bpf_prog *tgt_prog = prog->aux->dst_prog; struct bpf_attach_target_info tgt_info = {}; u32 btf_id = prog->aux->attach_btf_id; - struct bpf_trampoline *tr; + struct bpf_tramp_attach *attach; struct bpf_tramp_id *id; int ret; @@ -14000,13 +14000,15 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -ENOMEM; bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id); - tr = bpf_trampoline_get(id, &tgt_info); - if (!tr) { + id->addr = (void *) tgt_info.tgt_addr; + + attach = bpf_tramp_attach(id, tgt_prog, prog); + if (IS_ERR(attach)) { bpf_tramp_id_free(id); - return -ENOMEM; + return PTR_ERR(attach); } - prog->aux->dst_trampoline = tr; + prog->aux->dst_attach = attach; return 0; } From patchwork Thu Nov 18 11:24:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626681 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63FC9C433EF for ; Thu, 18 Nov 2021 11:28:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4EBB061A89 for ; Thu, 18 Nov 2021 11:28:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344071AbhKRLba (ORCPT ); Thu, 18 Nov 2021 06:31:30 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:27175 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343973AbhKRL3d (ORCPT ); Thu, 18 Nov 2021 06:29:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234793; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9S+q8pWMDFvdDxjJwU6rYEztkMBIcs++/6oEf507CaQ=; b=U5aiYyUC0vFqOAAgMRId8X5jfeoFnIM28TJaI3U2ZRL9BN+Y6qsb+9qbWRVKa/cKCYfeQb kbRpvGFZJXKTjt1a70Lx8VEI0ShljNzaFvPwXzhAUnxPwPhisiR1KlRk/afHb/qM+/aOSb H+GciTQB5LzkGnfmlFDJeD4bscnTrkg= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-219-X33CFTk5OEOL0YJ5Lrx-qQ-1; Thu, 18 Nov 2021 06:26:22 -0500 X-MC-Unique: X33CFTk5OEOL0YJ5Lrx-qQ-1 Received: by mail-ed1-f72.google.com with SMTP id p4-20020aa7d304000000b003e7ef120a37so4973163edq.16 for ; Thu, 18 Nov 2021 03:26:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9S+q8pWMDFvdDxjJwU6rYEztkMBIcs++/6oEf507CaQ=; b=V82Ne7gEfkafNZP1GYDbZQvrZE9pnnIxOjgni6EkFdMSwp/jBJGlrCYprwSbUAqwDJ B+dljYidc1YllChCxlAkS/iLiOPK1j5b6sLh+9z2aHJrTjcYedkq8Ec4csRal80JChXb mvDCB7bsz06dft5N7neNNS3LOjbvHA5GmYu1Df0Ig0m3hb2uUZCHKBLvLLJywMo5U7BW ZNA26dLlETX1CDq20+WvEJCvG8E5pIEut7Gd63it/GtBeDqOlljFCW3OpfPTZThSsB+p 6w/sCzXwUfpi46GOi65mG8xjud5tUf576F9ClT2kTfO2lIrWB6xvpZ663+zztmteAd/B PBxA== X-Gm-Message-State: AOAM5322CyBU3kcF6phiRkdY6VNpMtCe1vct/8wISX4Ee8+j80DtmG7f KbUrhgEwAMJzAzgRtvVdFk/O9vquB1x1p3/X+/GiBxj337mWSraKrT+vsS0sGWv6HtQgrSctYy9 xj8j0nIQ1qowd2zr0 X-Received: by 2002:a17:907:96aa:: with SMTP id hd42mr34545052ejc.385.1637234781215; Thu, 18 Nov 2021 03:26:21 -0800 (PST) X-Google-Smtp-Source: ABdhPJxgmIk56biArqymQrAnbeMLfZmEiUWksIFVac/+dL5uyCKQcFWyXprnJ/2JdubpH4kbDMoLgw== X-Received: by 2002:a17:907:96aa:: with SMTP id hd42mr34545021ejc.385.1637234781062; Thu, 18 Nov 2021 03:26:21 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id o12sm1418821edw.84.2021.11.18.03.26.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:20 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 14/29] bpf: Add support to store multiple ids in bpf_tramp_id object Date: Thu, 18 Nov 2021 12:24:40 +0100 Message-Id: <20211118112455.475349-15-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to store multiple ids in bpf_tramp_id object, to have id for trampolines with multiple functions assigned. Extra array of u32 values is allocated within bpf_tramp_id object allocation. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 6 ++++-- kernel/bpf/syscall.c | 6 +++--- kernel/bpf/trampoline.c | 39 +++++++++++++++++++++++++++++++-------- kernel/bpf/verifier.c | 2 +- 4 files changed, 39 insertions(+), 14 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 2dbc00904a84..47e25d8be600 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -672,8 +672,10 @@ struct bpf_tramp_image { }; struct bpf_tramp_id { + u32 max; + u32 cnt; u32 obj_id; - u32 btf_id; + u32 *id; void *addr; }; @@ -749,7 +751,7 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( return bpf_func(ctx, insnsi); } #ifdef CONFIG_BPF_JIT -struct bpf_tramp_id *bpf_tramp_id_alloc(void); +struct bpf_tramp_id *bpf_tramp_id_alloc(u32 cnt); void bpf_tramp_id_free(struct bpf_tramp_id *id); bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id); int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index a65c1862ab68..216fcce07326 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2704,7 +2704,7 @@ static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, info->tracing.attach_type = tr_link->attach_type; info->tracing.target_obj_id = attach->id->obj_id; - info->tracing.target_btf_id = attach->id->btf_id; + info->tracing.target_btf_id = attach->id->id[0]; return 0; } @@ -2766,7 +2766,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } - id = bpf_tramp_id_alloc(); + id = bpf_tramp_id_alloc(1); if (!id) { err = -ENOMEM; goto out_put_prog; @@ -2829,7 +2829,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_unlock; } - id = bpf_tramp_id_alloc(); + id = bpf_tramp_id_alloc(1); if (!id) { err = -ENOMEM; goto out_unlock; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 16fc4c14319b..d65f463c532d 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -60,27 +60,45 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } +static bool bpf_tramp_id_is_multi(struct bpf_tramp_id *id) +{ + return id->cnt > 1; +} + static u64 bpf_tramp_id_key(struct bpf_tramp_id *id) { - return ((u64) id->obj_id << 32) | id->btf_id; + if (bpf_tramp_id_is_multi(id)) + return (u64) &id; + else + return ((u64) id->obj_id << 32) | id->id[0]; } bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id) { - return !id || (!id->obj_id && !id->btf_id); + return !id || id->cnt == 0; } int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b) { - return !memcmp(a, b, sizeof(*a)); + return a->obj_id == b->obj_id && a->cnt == b->cnt && + !memcmp(a->id, b->id, a->cnt * sizeof(*a->id)); } -struct bpf_tramp_id *bpf_tramp_id_alloc(void) +struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max) { struct bpf_tramp_id *id; - return kzalloc(sizeof(*id), GFP_KERNEL); + id = kzalloc(sizeof(*id), GFP_KERNEL); + if (id) { + id->id = kzalloc(sizeof(u32) * max, GFP_KERNEL); + if (!id->id) { + kfree(id); + return NULL; + } + id->max = max; + } + return id; } void bpf_tramp_id_init(struct bpf_tramp_id *id, @@ -91,11 +109,15 @@ void bpf_tramp_id_init(struct bpf_tramp_id *id, id->obj_id = tgt_prog->aux->id; else id->obj_id = btf_obj_id(btf); - id->btf_id = btf_id; + id->id[0] = btf_id; + id->cnt = 1; } void bpf_tramp_id_free(struct bpf_tramp_id *id) { + if (!id) + return; + kfree(id->id); kfree(id); } @@ -362,7 +384,8 @@ bpf_tramp_image_alloc(struct bpf_tramp_id *id, u32 idx) ksym = &im->ksym; INIT_LIST_HEAD_RCU(&ksym->lnode); key = bpf_tramp_id_key(id); - snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u", key, idx); + snprintf(ksym->name, KSYM_NAME_LEN, "bpf_trampoline_%llu_%u%s", key, idx, + bpf_tramp_id_is_multi(id) ? "_multi" : ""); bpf_image_ksym_add(image, ksym); return im; @@ -597,7 +620,7 @@ struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, if (!node) goto out; - err = bpf_check_attach_model(prog, tgt_prog, id->btf_id, &tr->func.model); + err = bpf_check_attach_model(prog, tgt_prog, id->id[0], &tr->func.model); if (err) goto out; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e05f39fd2708..1903d5d256b6 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13995,7 +13995,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -EINVAL; } - id = bpf_tramp_id_alloc(); + id = bpf_tramp_id_alloc(1); if (!id) return -ENOMEM; From patchwork Thu Nov 18 11:24:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626679 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0C12C433F5 for ; Thu, 18 Nov 2021 11:28:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A716B61AFB for ; Thu, 18 Nov 2021 11:28:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343960AbhKRLb1 (ORCPT ); Thu, 18 Nov 2021 06:31:27 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52948 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343964AbhKRL3a (ORCPT ); Thu, 18 Nov 2021 06:29:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234789; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rJqHmk+ZMyp6K5b8ACnyHPg6ZEhvFJYuOs2AT1VL5MI=; b=fMb9m2x4tVYgMIenE/tOHs8nKTnX0XuDWLbkCgOiN0Z5N5MTn+AwCsabiPxbn4EdXInqEp Jl2xN2EVyWILSzWw7awQ76gghlQj0wdXhIXyvRv8+HAlurkY4y1Qd7i3TumKy+8pi5aDkw P/vEouKnZZ0OramkJDoRJNqymIGj0mU= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-590-hB28at2zMNOwwgbXlLTm-Q-1; Thu, 18 Nov 2021 06:26:28 -0500 X-MC-Unique: hB28at2zMNOwwgbXlLTm-Q-1 Received: by mail-ed1-f70.google.com with SMTP id v9-20020a50d849000000b003dcb31eabaaso5006945edj.13 for ; Thu, 18 Nov 2021 03:26:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rJqHmk+ZMyp6K5b8ACnyHPg6ZEhvFJYuOs2AT1VL5MI=; b=tXWxjQilCRKIFq1Y1I6f2OSDxuC/QRAwFiGyLNQzl/CG33xCiB7X3d+6/QREKF4+85 ybGSD3/ZmdDi7qTRf2y9vlp5lX8PX1n0OQRRWyUby1EqwSEZ7UkbadWfaMqgc8159Clp fwynyDjl07Q4aBgxI9JzUePx3FD5lkk/bkjsX3YGAG2L8/nE1KBk21R/5yR/UaWYod3v jqSV305Lx564Ws+i4UwWanHCi/zQ36wbcEKo8x88kEiWh8d4BqxJ/0IdxSe8hswOkBzp qjeCnFScz7nrNzn8hznLgbfOKD17otCpHh6KbeXL7YPd3zCg/78QHKviwrWUxc+OZxYQ HKUg== X-Gm-Message-State: AOAM532nHBFIQPZdu/oNK8rRprtTys4Dy3a3M3WQwiwv9KNIi375R52D 6lvvkB3EfylO9oGpVNwqnDF49iVFZFqqJbFXH/M5VdVw7juX2dMfyTQ2orhgWkHIwamATBjKQLD CK6umQQrB4ucv X-Received: by 2002:a50:e089:: with SMTP id f9mr10346386edl.290.1637234787260; Thu, 18 Nov 2021 03:26:27 -0800 (PST) X-Google-Smtp-Source: ABdhPJwjRG3EqPq6/SddZG4kWdHXZSl6WAkMfMmgkquhQDqfXX+KAqWQtWyPEbl+yp/g6O1neegfYw== X-Received: by 2002:a50:e089:: with SMTP id f9mr10346357edl.290.1637234787106; Thu, 18 Nov 2021 03:26:27 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id l26sm1403725eda.20.2021.11.18.03.26.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:26 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 15/29] bpf: Add support to store multiple addrs in bpf_tramp_id object Date: Thu, 18 Nov 2021 12:24:41 +0100 Message-Id: <20211118112455.475349-16-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to store multiple addrs in bpf_tramp_id object, to provide address values for id values stored in the object. The id->addr[idx] returns address value for id->id[idx] id. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 2 +- kernel/bpf/syscall.c | 2 +- kernel/bpf/trampoline.c | 20 ++++++++++++-------- kernel/bpf/verifier.c | 2 +- 4 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 47e25d8be600..13e9dcfd47e7 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -676,7 +676,7 @@ struct bpf_tramp_id { u32 cnt; u32 obj_id; u32 *id; - void *addr; + void **addr; }; struct bpf_tramp_node { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 216fcce07326..0ae3b5b7419a 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2853,7 +2853,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - id->addr = (void *) tgt_info.tgt_addr; + id->addr[0] = (void *) tgt_info.tgt_addr; attach = bpf_tramp_attach(id, tgt_prog, prog); if (IS_ERR(attach)) { diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d65f463c532d..d9675d619963 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -92,7 +92,10 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max) id = kzalloc(sizeof(*id), GFP_KERNEL); if (id) { id->id = kzalloc(sizeof(u32) * max, GFP_KERNEL); - if (!id->id) { + id->addr = kzalloc(sizeof(*id->addr) * max, GFP_KERNEL); + if (!id->id || !id->addr) { + kfree(id->id); + kfree(id->addr); kfree(id); return NULL; } @@ -117,6 +120,7 @@ void bpf_tramp_id_free(struct bpf_tramp_id *id) { if (!id) return; + kfree(id->addr); kfree(id->id); kfree(id); } @@ -159,7 +163,7 @@ static int bpf_trampoline_module_get(struct bpf_trampoline *tr) int err = 0; preempt_disable(); - mod = __module_text_address((unsigned long) tr->id->addr); + mod = __module_text_address((unsigned long) tr->id->addr[0]); if (mod && !try_module_get(mod)) err = -ENOENT; preempt_enable(); @@ -187,7 +191,7 @@ static int is_ftrace_location(void *ip) static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) { - void *ip = tr->id->addr; + void *ip = tr->id->addr[0]; int ret; if (tr->func.ftrace_managed) @@ -202,7 +206,7 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_addr) { - void *ip = tr->id->addr; + void *ip = tr->id->addr[0]; int ret; if (tr->func.ftrace_managed) @@ -215,7 +219,7 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad /* first time registering */ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) { - void *ip = tr->id->addr; + void *ip = tr->id->addr[0]; int ret; ret = is_ftrace_location(ip); @@ -434,7 +438,7 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) err = arch_prepare_bpf_trampoline(im, im->image, im->image + PAGE_SIZE, &tr->func.model, flags, tprogs, - tr->id->addr); + tr->id->addr[0]); if (err < 0) goto out; @@ -503,7 +507,7 @@ int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline goto out; } tr->extension_prog = prog; - err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP, NULL, + err = bpf_arch_text_poke(tr->id->addr[0], BPF_MOD_JUMP, NULL, prog->bpf_func); goto out; } @@ -539,7 +543,7 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin mutex_lock(&tr->mutex); if (kind == BPF_TRAMP_REPLACE) { WARN_ON_ONCE(!tr->extension_prog); - err = bpf_arch_text_poke(tr->id->addr, BPF_MOD_JUMP, + err = bpf_arch_text_poke(tr->id->addr[0], BPF_MOD_JUMP, tr->extension_prog->bpf_func, NULL); tr->extension_prog = NULL; goto out; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1903d5d256b6..56c518efa2d2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -14000,7 +14000,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -ENOMEM; bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id); - id->addr = (void *) tgt_info.tgt_addr; + id->addr[0] = (void *) tgt_info.tgt_addr; attach = bpf_tramp_attach(id, tgt_prog, prog); if (IS_ERR(attach)) { From patchwork Thu Nov 18 11:24:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626683 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B518EC433F5 for ; Thu, 18 Nov 2021 11:28:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A0B6661A89 for ; Thu, 18 Nov 2021 11:28:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344080AbhKRLbl (ORCPT ); Thu, 18 Nov 2021 06:31:41 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:58523 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343978AbhKRL3g (ORCPT ); Thu, 18 Nov 2021 06:29:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234795; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3epWTENbvCHtKUodXQN7x8PcsitZYsEe+46k8+JKkuw=; b=EqEyLAk8O0/WVdEAJ2VeTODrAzcobb94HE+bQptF1P2Mw8SJDcamdivQW81bTC2gonnCon p8MDQxD2+xSQFNnIqYVx1w5phYfYkUUbAKpHPsJ7uPA2mpTEE6il+HpmPd8O2owz4vwqC6 yaW0n0giq5nv7qjwOduZ0m43G3pR3iE= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-508-fvvMx_mtOcuFwW3UfQKFvA-1; Thu, 18 Nov 2021 06:26:34 -0500 X-MC-Unique: fvvMx_mtOcuFwW3UfQKFvA-1 Received: by mail-ed1-f72.google.com with SMTP id d11-20020a50cd4b000000b003da63711a8aso4952926edj.20 for ; Thu, 18 Nov 2021 03:26:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3epWTENbvCHtKUodXQN7x8PcsitZYsEe+46k8+JKkuw=; b=7+PQ42K7yeXwy/2V+1I+zO45OtwxpJUwLDGtn1aSCcpESlMXRrAgiKTE6cHMlGCr7C PMCyTY8YzCFm1fCLchxskBSXTejaNR/13LMdzd+CZD2QjSpKgKi1qpXupbhX87lWP0tt MjGaHo/pgybXgkNayjdnEiX08iiEmgdqCAMGc9fULPQ0jyLDEv8uXOkqZidSeyzowO4m DdziDaqqiWGfheMHw6c/LnBeqHlqpGXjGs6COs2wKGkdmywPjumvn1vvU7eetYKyTiLH knyQeksqSbF4ztGJ37MkYdkfSOa/STmPppbOhjruOimehhnBwzAH4oQV6b0+mM4NqpvP 0lgw== X-Gm-Message-State: AOAM533tgsH9x1fK1K3nC8nNXLdaMCgjZ7AZKCQAcZHUqT+i0yDDzSvX aI6ZRL8ix/NuGge6pC1N13/L/nOiG/3/ygVoI+VApDrCNeU2Ssml6ejeRTRRrcreJ7m/BH+R9fy FbAysVmNFLtmKt45q X-Received: by 2002:a17:906:1b1b:: with SMTP id o27mr32358069ejg.279.1637234793213; Thu, 18 Nov 2021 03:26:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJwwE0ncH346VRzvKTbIPM8QsddzsSDJe16UTcm5Gaf9xaOqpV6H2g4PE0AH7tj4e8F9DksE7A== X-Received: by 2002:a17:906:1b1b:: with SMTP id o27mr32358042ejg.279.1637234793065; Thu, 18 Nov 2021 03:26:33 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id ho30sm1186939ejc.30.2021.11.18.03.26.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:32 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 16/29] bpf: Add bpf_tramp_id_single function Date: Thu, 18 Nov 2021 12:24:42 +0100 Message-Id: <20211118112455.475349-17-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding bpf_tramp_id_single function as interface to create trampoline with single ID and grouping together the trampoline allocation with init that is used on several places and save us few lines. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 5 ++--- kernel/bpf/syscall.c | 18 +++++++----------- kernel/bpf/trampoline.c | 11 ++++++++--- kernel/bpf/verifier.c | 3 +-- 4 files changed, 18 insertions(+), 19 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 13e9dcfd47e7..894ee812e213 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -755,9 +755,8 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 cnt); void bpf_tramp_id_free(struct bpf_tramp_id *id); bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id); int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b); -void bpf_tramp_id_init(struct bpf_tramp_id *id, - const struct bpf_prog *tgt_prog, - struct btf *btf, u32 btf_id); +struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id); int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); void bpf_trampoline_put(struct bpf_trampoline *tr); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0ae3b5b7419a..8109b0fc7d2f 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2766,12 +2766,6 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } - id = bpf_tramp_id_alloc(1); - if (!id) { - err = -ENOMEM; - goto out_put_prog; - } - tgt_prog = bpf_prog_get(tgt_prog_fd); if (IS_ERR(tgt_prog)) { err = PTR_ERR(tgt_prog); @@ -2779,7 +2773,11 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } - bpf_tramp_id_init(id, tgt_prog, NULL, btf_id); + id = bpf_tramp_id_single(tgt_prog, NULL, btf_id); + if (!id) { + err = -ENOMEM; + goto out_put_prog; + } } link = kzalloc(sizeof(*link), GFP_USER); @@ -2829,14 +2827,12 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_unlock; } - id = bpf_tramp_id_alloc(1); + btf_id = prog->aux->attach_btf_id; + id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id); if (!id) { err = -ENOMEM; goto out_unlock; } - - btf_id = prog->aux->attach_btf_id; - bpf_tramp_id_init(id, NULL, prog->aux->attach_btf, btf_id); } if (!prog->aux->dst_attach || diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d9675d619963..4a4ef9396b7e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -104,16 +104,21 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max) return id; } -void bpf_tramp_id_init(struct bpf_tramp_id *id, - const struct bpf_prog *tgt_prog, - struct btf *btf, u32 btf_id) +struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, + struct btf *btf, u32 btf_id) { + struct bpf_tramp_id *id; + + id = bpf_tramp_id_alloc(1); + if (!id) + return NULL; if (tgt_prog) id->obj_id = tgt_prog->aux->id; else id->obj_id = btf_obj_id(btf); id->id[0] = btf_id; id->cnt = 1; + return id; } void bpf_tramp_id_free(struct bpf_tramp_id *id) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 56c518efa2d2..9914487f2281 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13995,11 +13995,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -EINVAL; } - id = bpf_tramp_id_alloc(1); + id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id); if (!id) return -ENOMEM; - bpf_tramp_id_init(id, tgt_prog, prog->aux->attach_btf, btf_id); id->addr[0] = (void *) tgt_info.tgt_addr; attach = bpf_tramp_attach(id, tgt_prog, prog); From patchwork Thu Nov 18 11:24:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626685 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9C97C433EF for ; Thu, 18 Nov 2021 11:28:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C533160FED for ; Thu, 18 Nov 2021 11:28:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1343983AbhKRLbp (ORCPT ); Thu, 18 Nov 2021 06:31:45 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:20211 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343989AbhKRL3l (ORCPT ); Thu, 18 Nov 2021 06:29:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234801; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/RSrlNmESC/yOXTmaT8HmTuDmJmtanZNQqjhrNcr48I=; b=XofaOF7Juy9NXUHKHYtbtn4oSQa+RkEGEwLFhY7pVc1NL4goUD/JepWk0Ave7CWvwrGOMC cdT3qttuDMOtnOqLMbOrYOOMBOt+hVDTIXhn1kMXnWjOqG3DC7XL3reDWC0jA4IT2OtAnE L2YDdUXARJ0fAhP3ygvMAGBR9JVk2w8= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-100-W53VLyQ_OtOsy2uOZdyf7g-1; Thu, 18 Nov 2021 06:26:40 -0500 X-MC-Unique: W53VLyQ_OtOsy2uOZdyf7g-1 Received: by mail-ed1-f70.google.com with SMTP id y9-20020aa7c249000000b003e7bf7a1579so5001902edo.5 for ; Thu, 18 Nov 2021 03:26:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/RSrlNmESC/yOXTmaT8HmTuDmJmtanZNQqjhrNcr48I=; b=ONH+psSIEcj2wIMmVPPgQwCSmT/hS+Nvw6ieEwAdAhe1wrCFTO2geEu16zfvW9xq4q iWMA+5W+gHltOYNfNBGX8KSiAmm1MwmEp8+xGeGt9TtoEeu9/1jovOH+FiVed4xb/9w5 bQKMhNFKDCj+sV3J/08ObRw6NzxCqP5clSSAe0wGFMBVcabxAizcj0MgrET8eFDU/bDv sjurA2llQLSOCfLTrxy6jv67u/g0oVNVGT2gqTGznwUGU8Jk23biQ5aBj4fobdSnOUx1 31dxbo/OYXKnRlmv0gsoa9HCLnIP4arxq/FXtL1w7Pv8LSr7E+KQ1qvExSmJOif3NoJp SvTQ== X-Gm-Message-State: AOAM531rY0CSYXS3CA8kpn3Nu+yVS2zttC6mtPFSQVe7DFMYle4oVBQc d0+qaMc96+UeUkPeELugjUUmswa9L5KM70RuRAa3vT81f065XKkYeOAbNuKD1r9A6l/MTvtTnM7 TE2M3dZ/Eov1QsX4b X-Received: by 2002:a17:907:6287:: with SMTP id nd7mr32143156ejc.152.1637234799200; Thu, 18 Nov 2021 03:26:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJwfgptxV/yt6rQtMUYJJlBglJe4JjttXNxcM3GrqP3qFWIlFIoxdWI3k6g3rjJEHp1aYxDTeA== X-Received: by 2002:a17:907:6287:: with SMTP id nd7mr32143126ejc.152.1637234799060; Thu, 18 Nov 2021 03:26:39 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id e16sm1706157edz.18.2021.11.18.03.26.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:38 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 17/29] bpf: Resolve id in bpf_tramp_id_single Date: Thu, 18 Nov 2021 12:24:43 +0100 Message-Id: <20211118112455.475349-18-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Moving the id resolving in the bpf_tramp_id_single function so it's centralized together with the trampoline's allocation and init. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 3 ++- kernel/bpf/syscall.c | 21 ++++++--------------- kernel/bpf/trampoline.c | 18 +++++++++++++++--- kernel/bpf/verifier.c | 4 +--- 4 files changed, 24 insertions(+), 22 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 894ee812e213..dda24339e4b1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -756,7 +756,8 @@ void bpf_tramp_id_free(struct bpf_tramp_id *id); bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id); int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b); struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, - struct btf *btf, u32 btf_id); + struct bpf_prog *prog, u32 btf_id, + struct bpf_attach_target_info *tgt_info); int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); void bpf_trampoline_put(struct bpf_trampoline *tr); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 8109b0fc7d2f..0acf6cb0fdc7 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2773,9 +2773,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_put_prog; } - id = bpf_tramp_id_single(tgt_prog, NULL, btf_id); - if (!id) { - err = -ENOMEM; + id = bpf_tramp_id_single(tgt_prog, prog, btf_id, NULL); + if (IS_ERR(id)) { + err = PTR_ERR(id); goto out_put_prog; } } @@ -2828,9 +2828,9 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, } btf_id = prog->aux->attach_btf_id; - id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id); - if (!id) { - err = -ENOMEM; + id = bpf_tramp_id_single(NULL, prog, btf_id, NULL); + if (IS_ERR(id)) { + err = PTR_ERR(id); goto out_unlock; } } @@ -2842,15 +2842,6 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, * different from the destination specified at load time, we * need a new trampoline and a check for compatibility */ - struct bpf_attach_target_info tgt_info = {}; - - err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, - &tgt_info); - if (err) - goto out_unlock; - - id->addr[0] = (void *) tgt_info.tgt_addr; - attach = bpf_tramp_attach(id, tgt_prog, prog); if (IS_ERR(attach)) { err = PTR_ERR(attach); diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 4a4ef9396b7e..e6a73088ecee 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -105,18 +105,30 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max) } struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, - struct btf *btf, u32 btf_id) + struct bpf_prog *prog, u32 btf_id, + struct bpf_attach_target_info *tgt_info) { struct bpf_tramp_id *id; + if (!tgt_info) { + struct bpf_attach_target_info __tgt_info = {}; + int err; + + tgt_info = &__tgt_info; + err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id, + tgt_info); + if (err) + return ERR_PTR(err); + } id = bpf_tramp_id_alloc(1); if (!id) - return NULL; + return ERR_PTR(-ENOMEM); if (tgt_prog) id->obj_id = tgt_prog->aux->id; else - id->obj_id = btf_obj_id(btf); + id->obj_id = btf_obj_id(prog->aux->attach_btf); id->id[0] = btf_id; + id->addr[0] = (void *) tgt_info->tgt_addr; id->cnt = 1; return id; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9914487f2281..8d56d43489aa 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13995,12 +13995,10 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -EINVAL; } - id = bpf_tramp_id_single(NULL, prog->aux->attach_btf, btf_id); + id = bpf_tramp_id_single(tgt_prog, prog, btf_id, &tgt_info); if (!id) return -ENOMEM; - id->addr[0] = (void *) tgt_info.tgt_addr; - attach = bpf_tramp_attach(id, tgt_prog, prog); if (IS_ERR(attach)) { bpf_tramp_id_free(id); From patchwork Thu Nov 18 11:24:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626687 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE2EFC433F5 for ; Thu, 18 Nov 2021 11:28:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C5F7F60FED for ; Thu, 18 Nov 2021 11:28:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344006AbhKRLbx (ORCPT ); Thu, 18 Nov 2021 06:31:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:37996 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344004AbhKRL3t (ORCPT ); Thu, 18 Nov 2021 06:29:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234808; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KPsGDeo6aCmpgmXjMoObHMVqT9QgbV/NPDn9sVokyBo=; b=hVRfM9IIUBI3gCTMaFYY77TWAmhPt/OPkoNFkeP9e8eV5L8qdbJrWdlbUJ2Ck5YDMUfet/ E0kjssYuU5klswMWdHzWpH4CqV0blqBliqfTuGbyrQtzlV2M7++oVVMzxZUrwBU73iZhb6 TjB+OBjX6/c9uDAWskIJINSVWdpqFKI= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-22-nlQ3pEvXMg2voc-iWlzfEg-1; Thu, 18 Nov 2021 06:26:47 -0500 X-MC-Unique: nlQ3pEvXMg2voc-iWlzfEg-1 Received: by mail-ed1-f69.google.com with SMTP id v22-20020a50a456000000b003e7cbfe3dfeso4984197edb.11 for ; Thu, 18 Nov 2021 03:26:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KPsGDeo6aCmpgmXjMoObHMVqT9QgbV/NPDn9sVokyBo=; b=7kpNI80OBdlecYICVm4KgiO2TbYA9kOUiM+g+HtLQUYFkIW2ciKuaaIp61Pi9Q7ZZ4 Nno+t9GA+WPxVEYLgVNlPqkz/ps/5wUod+05sNo0YWMZTHoa4NSSrfJ0gmM996ica/Tq meOe3xY557OQmiVh9587OJDJuti2vdjlLSIMUKsEF8bSevYhT5kULU4AWnfo5xXnYgYw 5L1wvJUv60rhMrAI6Gv4DLZWhG4XZUkVCJHu83Sj41/9SiCDWizx07RbqJuc5a/Gh/cx KtiKcWdJorUlZonWjTK+Povrudhpzs7zAPq2IoFI1bYmWo6uxP5cXZ/9aPXRvVIgc3/g DD1g== X-Gm-Message-State: AOAM5314xnpeSvtjv1bETzxwoyaTFxJ55h3XX0d8b5v+h7Vr6uwqs1u4 bLDdzeJze5O9owadwNKtUv41qrOB8aGRwQYFzmfx8rXGaOouS5ZdSu8gDFQGAJS/x7cfPJ7+zUK gNYqCMC1WdNynSd+n X-Received: by 2002:a17:906:e28b:: with SMTP id gg11mr32475322ejb.23.1637234805223; Thu, 18 Nov 2021 03:26:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJzgwQ6CQ0SzIvpMVKYD9a6jI2nTXJU1J4Gvi7jQflrEU22Z7fbPtedQcD2IlTdlp73ZuuVP1g== X-Received: by 2002:a17:906:e28b:: with SMTP id gg11mr32475289ejb.23.1637234805030; Thu, 18 Nov 2021 03:26:45 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id hc16sm1164557ejc.12.2021.11.18.03.26.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:44 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 18/29] bpf: Add refcount_t to struct bpf_tramp_id Date: Thu, 18 Nov 2021 12:24:44 +0100 Message-Id: <20211118112455.475349-19-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding refcount_t to struct bpf_tramp_id so we can track its allocation and safely use one object on more places in following changes. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 3 +++ kernel/bpf/syscall.c | 2 +- kernel/bpf/trampoline.c | 16 +++++++++++++--- kernel/bpf/verifier.c | 2 +- 4 files changed, 18 insertions(+), 5 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index dda24339e4b1..04ada1d2495e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -23,6 +23,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -677,6 +678,7 @@ struct bpf_tramp_id { u32 obj_id; u32 *id; void **addr; + refcount_t refcnt; }; struct bpf_tramp_node { @@ -753,6 +755,7 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( #ifdef CONFIG_BPF_JIT struct bpf_tramp_id *bpf_tramp_id_alloc(u32 cnt); void bpf_tramp_id_free(struct bpf_tramp_id *id); +void bpf_tramp_id_put(struct bpf_tramp_id *id); bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id); int bpf_tramp_id_is_equal(struct bpf_tramp_id *a, struct bpf_tramp_id *b); struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 0acf6cb0fdc7..bfbd81869818 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2901,7 +2901,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, out_put_prog: if (tgt_prog_fd && tgt_prog) bpf_prog_put(tgt_prog); - bpf_tramp_id_free(id); + bpf_tramp_id_put(id); return err; } diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index e6a73088ecee..39600fb78c9e 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -100,6 +100,7 @@ struct bpf_tramp_id *bpf_tramp_id_alloc(u32 max) return NULL; } id->max = max; + refcount_set(&id->refcnt, 1); } return id; } @@ -133,10 +134,18 @@ struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, return id; } -void bpf_tramp_id_free(struct bpf_tramp_id *id) +static struct bpf_tramp_id *bpf_tramp_id_get(struct bpf_tramp_id *id) +{ + refcount_inc(&id->refcnt); + return id; +} + +void bpf_tramp_id_put(struct bpf_tramp_id *id) { if (!id) return; + if (!refcount_dec_and_test(&id->refcnt)) + return; kfree(id->addr); kfree(id->id); kfree(id); @@ -162,7 +171,7 @@ static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id) if (!tr) goto out; - tr->id = id; + tr->id = bpf_tramp_id_get(id); INIT_HLIST_NODE(&tr->hlist); hlist_add_head(&tr->hlist, head); refcount_set(&tr->refcnt, 1); @@ -592,6 +601,7 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) * multiple rcu callbacks. */ hlist_del(&tr->hlist); + bpf_tramp_id_put(tr->id); kfree(tr); out: mutex_unlock(&trampoline_mutex); @@ -663,7 +673,7 @@ void bpf_tramp_detach(struct bpf_tramp_attach *attach) hlist_for_each_entry_safe(node, n, &attach->nodes, hlist_attach) node_free(node); - bpf_tramp_id_free(attach->id); + bpf_tramp_id_put(attach->id); kfree(attach); } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8d56d43489aa..6a87180ac2bb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -14001,7 +14001,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) attach = bpf_tramp_attach(id, tgt_prog, prog); if (IS_ERR(attach)) { - bpf_tramp_id_free(id); + bpf_tramp_id_put(id); return PTR_ERR(attach); } From patchwork Thu Nov 18 11:24:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626689 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E681C433F5 for ; Thu, 18 Nov 2021 11:29:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E643261251 for ; Thu, 18 Nov 2021 11:29:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344010AbhKRLb7 (ORCPT ); Thu, 18 Nov 2021 06:31:59 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:58932 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343683AbhKRL3y (ORCPT ); Thu, 18 Nov 2021 06:29:54 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TOfVinV1OwrahdFosss8ebQPODUryvfBCcrymxnr+48=; b=RKp+ka902JSCl1FWj+WchQta33qycBMeGxN+y1SvoR5OWx8/l90m47n21BS66hgLSi+n69 bqOQPxRAZFdFGAxHLKrt8CpRa2VNUKPzB4/RzL/qNzTl3FUBxwnAFunYVDNFKyLtDMBRZd 0zE8+nZea4BRzbmuJ9HcspzMk5BIuLQ= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-281-lX_AMYJIPa65JCWiZ3uI2w-1; Thu, 18 Nov 2021 06:26:53 -0500 X-MC-Unique: lX_AMYJIPa65JCWiZ3uI2w-1 Received: by mail-ed1-f72.google.com with SMTP id b15-20020aa7c6cf000000b003e7cf0f73daso4942553eds.22 for ; Thu, 18 Nov 2021 03:26:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TOfVinV1OwrahdFosss8ebQPODUryvfBCcrymxnr+48=; b=LVkZAblIxvLHIvvOEianMm1JIiKSroHnEkVgFywlCqGuqS/26ol8FK6I+dNlHn7aRi zKAAkU9573KQIYX+NdZ9dtj8tp6AFvu9xTXzFIrIs5M0jHOvreztBpRt8r5n8AILq1qB 0BHTimXwM4YZ0YcSJb70GuHTi+AJZVtES5/BuJNtDNWdPqwKSyWKHQPz5BxzaoWWhcdj nY0rjPyNbYI9/KD0fgqkdXlxXwPNz4FMMMXAckl0SSy0hbYzx65FHBy/X9+vImbrpVBH TQFeXgADNk5qSICSMTgijff/Ri85n4syMAtnOiJmUy64LBtGreTwuCst9mF9hf3CSUVe AAtw== X-Gm-Message-State: AOAM531keZsX9RNk80LY15ZRnQPCvOIueCNQIof7RweesXcxK7ckpsDD GNOVMZU1Sh70sj1NRy2ukuypNb76TsJ1hbXzjwLYO8wNNijpsLaH/nAWMV8pWNsIBQEpqPvzvNF ZCOCz1sTalQLuPnES X-Received: by 2002:a05:6402:84a:: with SMTP id b10mr10356566edz.285.1637234811523; Thu, 18 Nov 2021 03:26:51 -0800 (PST) X-Google-Smtp-Source: ABdhPJxdhtrH0x0C1LuZPAcXgxWcJKClEgNx7h9ri68aOBwa0f7ej6zn7komo/FHcd/h2kPyI6qfSA== X-Received: by 2002:a05:6402:84a:: with SMTP id b10mr10356500edz.285.1637234811081; Thu, 18 Nov 2021 03:26:51 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id y17sm170797edd.31.2021.11.18.03.26.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:50 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 19/29] bpf: Add support to attach trampolines with multiple IDs Date: Thu, 18 Nov 2021 12:24:45 +0100 Message-Id: <20211118112455.475349-20-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to attach trampolines with multiple IDs. This patch adds support to bpf_tramp_attach function to attach given program to bpf_tramp_id object that holds multiple BTF function IDs. The process of attaching in bpf_tramp_attach is as follows: - IDs in bpf_tramp_id object are sorted out to several new bpf_tramp_id objects based on number of arguments of each ID - so we end up with up to 6 bpf_tramp_id objects, that we will create or find suitable trampoline for - separating function IDs that have same number of arguments save us troubles of handling different argument counts within one trampoline - now for each such bpf_tramp_id object we do following: * search existing trampolines to find match or intersection * if there's full match on IDs, we add program to existing trampoline and we are done * if there's intersection with existing trampoline, we split it and add new program to the common part, the rest of the IDs are attached to new trampoline - we keep trampoline_table as place holder for all trampolines, (while the has works only for single ID trampolines) so in case there is no multi-id trampoline defined, we still use the fast hash trampoline lookup The bpf_tramp_attach assumes ID array is coming in sorted so it's possible to run bsearch on it to do all the needed searches. The splitting of the trampoline use the fact that we carry 'bpf_tramp_attach' object for each bpf_program, so when we split trampoline that the program is attached to, we just add new 'bpf_tramp_node' object to the program's attach 'nodes'. This way we keep track of all program's trampolines and it will be properly detached when the program goes away. The splitting of the trampoline is done with following steps: - lock the trampoline - unregister trampoline - alloc the duplicate, which means that for all attached programs of the original trampoline we create new bpf_tramp_node objects and add them to these programs' attach objects - then we assign new IDs (common and the rest) to both (original and the duplicated) trampolines - register both trampolines - unlock the original trampoline This patch only adds bpf_tramp_attach support to attach multiple ID bpf_tramp_id object. The actual user interface for that comes in following patch. Now when each call to bpf_tramp_attach can change any program's attach object, we need to take trampoline_mutex in both bpf_tramp_attach_link and bpf_tramp_attach_unlink functions. Perhaps we could add new lock to bpf_tramp_attach object to get rid of single lock for all. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 12 +- kernel/bpf/trampoline.c | 717 +++++++++++++++++++++++++++++++++++++--- 2 files changed, 674 insertions(+), 55 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 04ada1d2495e..6ceb3bb39e1d 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -24,6 +24,13 @@ #include #include #include +#ifdef CONFIG_FUNCTION_TRACER +#ifndef CC_USING_FENTRY +#define CC_USING_FENTRY +#endif +#endif +#include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -703,6 +710,9 @@ struct bpf_trampoline { struct { struct btf_func_model model; bool ftrace_managed; +#ifdef CONFIG_FUNCTION_TRACER + struct ftrace_ops ops; +#endif } func; /* if !NULL this is BPF_PROG_TYPE_EXT program that extends another BPF * program by replacing one of its functions. id->addr is the address @@ -763,7 +773,6 @@ struct bpf_tramp_id *bpf_tramp_id_single(const struct bpf_prog *tgt_prog, struct bpf_attach_target_info *tgt_info); int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); -void bpf_trampoline_put(struct bpf_trampoline *tr); struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, struct bpf_prog *tgt_prog, @@ -831,7 +840,6 @@ static inline struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id, { return ERR_PTR(-EOPNOTSUPP); } -static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {} #define DEFINE_BPF_DISPATCHER(name) #define DECLARE_BPF_DISPATCHER(name) #define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_nop_func diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 39600fb78c9e..7a9e3126e256 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -12,6 +12,8 @@ #include #include #include +#include +#include /* dummy _ops. The verifier will operate on target program's ops. */ const struct bpf_verifier_ops bpf_extension_verifier_ops = { @@ -24,8 +26,9 @@ const struct bpf_prog_ops bpf_extension_prog_ops = { #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS) static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE]; +static int nr_bpf_trampoline_multi; -/* serializes access to trampoline_table */ +/* serializes access to trampoline_table, nr_bpf_trampoline_multi */ static DEFINE_MUTEX(trampoline_mutex); void *bpf_jit_alloc_exec_page(void) @@ -62,15 +65,12 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) static bool bpf_tramp_id_is_multi(struct bpf_tramp_id *id) { - return id->cnt > 1; + return id && id->cnt > 1; } static u64 bpf_tramp_id_key(struct bpf_tramp_id *id) { - if (bpf_tramp_id_is_multi(id)) - return (u64) &id; - else - return ((u64) id->obj_id << 32) | id->id[0]; + return ((u64) id->obj_id << 32) | id->id[0]; } bool bpf_tramp_id_is_empty(struct bpf_tramp_id *id) @@ -151,26 +151,14 @@ void bpf_tramp_id_put(struct bpf_tramp_id *id) kfree(id); } -static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id) +static void bpf_trampoline_init(struct bpf_trampoline *tr, struct bpf_tramp_id *id) { - struct bpf_trampoline *tr; struct hlist_head *head; u64 key; int i; key = bpf_tramp_id_key(id); - mutex_lock(&trampoline_mutex); head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; - hlist_for_each_entry(tr, head, hlist) { - if (bpf_tramp_id_is_equal(tr->id, id)) { - refcount_inc(&tr->refcnt); - goto out; - } - } - tr = kzalloc(sizeof(*tr), GFP_KERNEL); - if (!tr) - goto out; - tr->id = bpf_tramp_id_get(id); INIT_HLIST_NODE(&tr->hlist); hlist_add_head(&tr->hlist, head); @@ -178,11 +166,39 @@ static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id) mutex_init(&tr->mutex); for (i = 0; i < BPF_TRAMP_MAX; i++) INIT_HLIST_HEAD(&tr->progs_hlist[i]); -out: - mutex_unlock(&trampoline_mutex); + if (bpf_tramp_id_is_multi(id)) + nr_bpf_trampoline_multi++; +} + +static struct bpf_trampoline *bpf_trampoline_alloc(struct bpf_tramp_id *id) +{ + struct bpf_trampoline *tr; + + tr = kzalloc(sizeof(*tr), GFP_KERNEL); + if (!tr) + return NULL; + + bpf_trampoline_init(tr, id); return tr; } +static struct bpf_trampoline *bpf_trampoline_get(struct bpf_tramp_id *id) +{ + struct bpf_trampoline *tr; + struct hlist_head *head; + u64 key; + + key = bpf_tramp_id_key(id); + head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; + hlist_for_each_entry(tr, head, hlist) { + if (bpf_tramp_id_is_equal(tr->id, id)) { + refcount_inc(&tr->refcnt); + return tr; + } + } + return bpf_trampoline_alloc(id); +} + static int bpf_trampoline_module_get(struct bpf_trampoline *tr) { struct module *mod; @@ -220,6 +236,9 @@ static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) void *ip = tr->id->addr[0]; int ret; + if (bpf_tramp_id_is_multi(tr->id)) + return unregister_ftrace_direct_multi(&tr->func.ops, (long) old_addr); + if (tr->func.ftrace_managed) ret = unregister_ftrace_direct((long)ip, (long)old_addr); else @@ -235,6 +254,9 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad void *ip = tr->id->addr[0]; int ret; + if (bpf_tramp_id_is_multi(tr->id)) + return modify_ftrace_direct_multi(&tr->func.ops, (long) new_addr); + if (tr->func.ftrace_managed) ret = modify_ftrace_direct((long)ip, (long)old_addr, (long)new_addr); else @@ -248,6 +270,9 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) void *ip = tr->id->addr[0]; int ret; + if (bpf_tramp_id_is_multi(tr->id)) + return register_ftrace_direct_multi(&tr->func.ops, (long) new_addr); + ret = is_ftrace_location(ip); if (ret < 0) return ret; @@ -435,17 +460,19 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) struct bpf_tramp_progs *tprogs; u32 flags = BPF_TRAMP_F_RESTORE_REGS; bool ip_arg = false; - int err, total; + int err = 0, total; tprogs = bpf_trampoline_get_progs(tr, &total, &ip_arg); if (IS_ERR(tprogs)) return PTR_ERR(tprogs); if (total == 0) { - err = unregister_fentry(tr, tr->cur_image->image); - bpf_tramp_image_put(tr->cur_image); - tr->cur_image = NULL; - tr->selector = 0; + if (tr->cur_image) { + err = unregister_fentry(tr, tr->cur_image->image); + bpf_tramp_image_put(tr->cur_image); + tr->cur_image = NULL; + tr->selector = 0; + } goto out; } @@ -456,8 +483,11 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) } if (tprogs[BPF_TRAMP_FEXIT].nr_progs || - tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs) + tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs) { flags = BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME; + if (bpf_tramp_id_is_multi(tr->id)) + flags |= BPF_TRAMP_F_ORIG_STACK; + } if (ip_arg) flags |= BPF_TRAMP_F_IP_ARG; @@ -582,29 +612,29 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin return err; } -void bpf_trampoline_put(struct bpf_trampoline *tr) +static void bpf_trampoline_put(struct bpf_trampoline *tr) { if (!tr) return; - mutex_lock(&trampoline_mutex); if (!refcount_dec_and_test(&tr->refcnt)) - goto out; + return; WARN_ON_ONCE(mutex_is_locked(&tr->mutex)); if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FENTRY]))) - goto out; + return; if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT]))) - goto out; + return; /* This code will be executed even when the last bpf_tramp_image * is alive. All progs are detached from the trampoline and the * trampoline image is patched with jmp into epilogue to skip * fexit progs. The fentry-only trampoline will be freed via * multiple rcu callbacks. */ + if (bpf_tramp_id_is_multi(tr->id)) + nr_bpf_trampoline_multi--; hlist_del(&tr->hlist); bpf_tramp_id_put(tr->id); + ftrace_free_filter(&tr->func.ops); kfree(tr); -out: - mutex_unlock(&trampoline_mutex); } static struct bpf_tramp_node *node_alloc(struct bpf_trampoline *tr, struct bpf_prog *prog) @@ -628,18 +658,442 @@ static void node_free(struct bpf_tramp_node *node) kfree(node); } -struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, - struct bpf_prog *tgt_prog, - struct bpf_prog *prog) +static void bpf_func_model_nargs(struct btf_func_model *m, int nr_args) +{ + int i; + + for (i = 0; i < nr_args; i++) + m->arg_size[i] = 8; + m->ret_size = 8; + m->nr_args = nr_args; +} + +struct attach_args { + int nr_args; + struct bpf_prog *tgt_prog; + struct bpf_prog *prog; +}; + +static int bpf_trampoline_setup(struct bpf_trampoline *tr, + struct attach_args *att) +{ + struct bpf_tramp_id *id = tr->id; + + if (bpf_tramp_id_is_multi(id)) { + bpf_func_model_nargs(&tr->func.model, att->nr_args); + return ftrace_set_filter_ips(&tr->func.ops, (long*) id->addr, + id->cnt, 0, 1); + } else { + return bpf_check_attach_model(att->prog, att->tgt_prog, + id->id[0], &tr->func.model); + } +} + +static int +bpf_trampoline_create(struct bpf_tramp_attach *attach, + struct bpf_tramp_id *id, struct attach_args *att) { struct bpf_trampoline *tr = NULL; - struct bpf_tramp_attach *attach; struct bpf_tramp_node *node; int err; - attach = kzalloc(sizeof(*attach), GFP_KERNEL); - if (!attach) - return ERR_PTR(-ENOMEM); + tr = bpf_trampoline_alloc(id); + if (!tr) { + err = -ENOMEM; + goto out; + } + + err = bpf_trampoline_setup(tr, att); + if (err) + goto out; + + node = node_alloc(tr, att->prog); + if (!node) { + err = -ENOMEM; + goto out; + } + + hlist_add_head(&node->hlist_attach, &attach->nodes); + return 0; + +out: + bpf_trampoline_put(tr); + return err; +} + +static void bpf_trampoline_dup_destroy(struct bpf_trampoline *tr) +{ + struct bpf_tramp_node *node; + struct hlist_node *n; + int kind; + + if (!tr) + return; + + for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { + hlist_for_each_entry_safe(node, n, &tr->progs_hlist[kind], + hlist_tramp) { + hlist_del(&node->hlist_tramp); + hlist_del(&node->hlist_attach); + node_free(node); + } + } + + WARN_ON_ONCE(refcount_read(&tr->refcnt) != 1); + bpf_trampoline_put(tr); +} + +static struct bpf_trampoline* +bpf_trampoline_dup(struct bpf_trampoline *tr, struct bpf_tramp_id *id) +{ + struct bpf_tramp_node *node, *iter; + struct bpf_trampoline *dup; + int kind; + + /* Allocate new trampoline and duplicate all + * the program attachments it has. + */ + dup = bpf_trampoline_alloc(id); + if (!dup) + return NULL; + + dup->refcnt = tr->refcnt; + + for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { + hlist_for_each_entry(iter, &tr->progs_hlist[kind], hlist_tramp) { + struct bpf_prog *prog = iter->prog; + + node = node_alloc(dup, prog); + if (!node) + goto out_free; + hlist_add_head(&node->hlist_tramp, &dup->progs_hlist[kind]); + hlist_add_head(&node->hlist_attach, &prog->aux->attach->nodes); + dup->progs_cnt[kind]++; + } + } + return dup; + +out_free: + bpf_trampoline_dup_destroy(dup); + return NULL; +} + +static int btf_id_cmp(const void *a, const void *b) +{ + const u32 *x = a; + const u32 *y = b; + + if (*x == *y) + return 0; + return *x < *y ? -1 : 1; +} + +static void id_add(struct bpf_tramp_id *id, u32 btf_id, void *addr) +{ + if (WARN_ON_ONCE(id->cnt >= id->max)) + return; + id->id[id->cnt] = btf_id; + id->addr[id->cnt] = addr; + id->cnt++; +} + +static struct bpf_tramp_id *id_check(struct bpf_tramp_id *id) +{ + if (bpf_tramp_id_is_empty(id)) { + bpf_tramp_id_put(id); + id = NULL; + } + return id; +} + +static int id_and(struct bpf_tramp_id *a, struct bpf_tramp_id *b, + struct bpf_tramp_id **pand, struct bpf_tramp_id **pother) +{ + struct bpf_tramp_id *and, *other; + u32 i, id; + + and = bpf_tramp_id_alloc(min(a->cnt, b->cnt)); + other = bpf_tramp_id_alloc(max(a->cnt, b->cnt)); + if (!and || !other) { + bpf_tramp_id_put(and); + bpf_tramp_id_put(other); + return -ENOMEM; + } + + and->obj_id = a->obj_id; + other->obj_id = a->obj_id; + + for (i = 0; i < a->cnt; i++) { + id = a->id[i]; + if (bsearch(&id, b->id, b->cnt, sizeof(u32), btf_id_cmp)) + id_add(and, id, a->addr[i]); + else + id_add(other, id, a->addr[i]); + } + + *pand = id_check(and); + *pother = id_check(other); + return 0; +} + +static int id_sub(struct bpf_tramp_id *a, struct bpf_tramp_id *b, + struct bpf_tramp_id **psub) +{ + struct bpf_tramp_id *sub; + u32 i, id; + + sub = bpf_tramp_id_alloc(max(a->cnt, b->cnt)); + if (!sub) + return -ENOMEM; + + sub->obj_id = a->obj_id; + + if (a->cnt < b->cnt) + swap(a, b); + + for (i = 0; i < a->cnt; i++) { + id = a->id[i]; + if (!bsearch(&id, b->id, b->cnt, sizeof(u32), btf_id_cmp)) + id_add(sub, id, a->addr[i]); + } + + *psub = id_check(sub); + return 0; +} + +struct tramp_state { + struct bpf_trampoline *tr_common; + struct bpf_trampoline *tr_other; + struct bpf_tramp_id *id_common; + struct bpf_tramp_id *id_other; + struct bpf_tramp_id *id; +}; + +#define MAX_TRAMP_STATE 20 + +struct attach_state { + struct tramp_state ts[MAX_TRAMP_STATE]; + int cnt; +}; + +static struct tramp_state* tramp_state_get(struct attach_state *state) +{ + if (state->cnt == MAX_TRAMP_STATE) + return NULL; + return &state->ts[state->cnt]; +} + +static void state_next(struct attach_state *state) +{ + state->cnt++; +} + +static void state_cleanup(struct attach_state *state) +{ + struct tramp_state *ts; + int i; + + for (i = 0; i < state->cnt; i++) { + ts = &state->ts[state->cnt]; + bpf_tramp_id_put(ts->id_common); + bpf_tramp_id_put(ts->id_other); + bpf_tramp_id_put(ts->id); + } +} + +static int tramp_state_compute(struct attach_state *state, + struct bpf_trampoline *tr, + struct bpf_tramp_id *id, + struct bpf_tramp_id **id_cont) +{ + struct bpf_tramp_id *id_new, *id_common, *id_other; + struct tramp_state *ts; + + ts = tramp_state_get(state); + if (!ts) + return -EBUSY; + + /* different playground.. bail out */ + if (tr->id->obj_id != id->obj_id) { + *id_cont = bpf_tramp_id_get(id); + return 0; + } + + /* complete match with trampoline */ + if (bpf_tramp_id_is_equal(tr->id, id)) { + ts->id_common = bpf_tramp_id_get(id); + *id_cont = NULL; + goto out; + } + + /* find out if there's common set of ids */ + if (id_and(id, tr->id, &id_common, &id_new)) + return -ENOMEM; + + /* nothing in common, bail out */ + if (!id_common) { + bpf_tramp_id_put(id_new); + *id_cont = bpf_tramp_id_get(id); + return 0; + } + + /* we have common set, let's get the rest of the matched + * trampoline ids as new id for split trampoline + */ + if (id_sub(id_common, tr->id, &id_other)) { + bpf_tramp_id_put(id_common); + bpf_tramp_id_put(id_new); + return -ENOMEM; + } + + ts->id_common = id_common; + ts->id_other = id_other; + ts->id = bpf_tramp_id_get(tr->id); + *id_cont = id_new; + +out: + ts->tr_common = tr; + state_next(state); + return 0; +} + +static int bpf_trampoline_register(struct bpf_trampoline *tr) +{ + return bpf_trampoline_update(tr); +} + +static int bpf_trampoline_unregister(struct bpf_trampoline *tr) +{ + int err; + + if (!tr->cur_image) + return 0; + err = unregister_fentry(tr, tr->cur_image->image); + bpf_tramp_image_put(tr->cur_image); + tr->cur_image = NULL; + tr->selector = 0; + return err; +} + +static void bpf_trampoline_id_assign(struct bpf_trampoline *tr, struct bpf_tramp_id *id) +{ + bool multi1 = bpf_tramp_id_is_multi(tr->id); + bool multi2 = bpf_tramp_id_is_multi(id); + + /* We can split into single ID trampolines and that + * might affect nr_bpf_trampoline_multi and the fast + * path trigger, so we need to check on that. + */ + if (multi1 && !multi2) + nr_bpf_trampoline_multi--; + if (!multi1 && multi2) + nr_bpf_trampoline_multi++; + + tr->id = id; +} + +static int bpf_trampoline_split(struct tramp_state *ts, struct attach_args *att) +{ + struct bpf_trampoline *tr_other, *tr_common = ts->tr_common; + struct bpf_tramp_id *id_common = ts->id_common; + struct bpf_tramp_id *id_other = ts->id_other; + int err; + + mutex_lock(&tr_common->mutex); + + err = bpf_trampoline_unregister(tr_common); + if (err) + goto out; + + tr_other = bpf_trampoline_dup(tr_common, id_other); + if (!tr_other) { + err = -ENOMEM; + goto out_free; + } + + err = bpf_trampoline_setup(tr_other, att); + if (err) + goto out_free; + + bpf_trampoline_id_assign(tr_common, id_common); + + err = bpf_trampoline_setup(tr_common, att); + if (err) + goto out_free; + + ts->tr_other = tr_other; + WARN_ON_ONCE(bpf_trampoline_register(tr_common)); + WARN_ON_ONCE(bpf_trampoline_register(tr_other)); + + mutex_unlock(&tr_common->mutex); + return 0; + +out_free: + bpf_trampoline_dup_destroy(tr_other); + tr_common->id = ts->id; + WARN_ON_ONCE(bpf_trampoline_register(tr_common)); +out: + mutex_unlock(&tr_common->mutex); + return err; +} + +static int tramp_state_apply(struct bpf_tramp_attach *attach, + struct tramp_state *ts, struct attach_args *att) +{ + struct bpf_tramp_node *node; + int err; + + /* The program will be attached to the common part. */ + node = node_alloc(ts->tr_common, att->prog); + if (!node) + return -ENOMEM; + + refcount_inc(&ts->tr_common->refcnt); + + /* If there are also 'other' IDs in the trampoline, + * we need to do the split. */ + if (ts->id_other) { + err = bpf_trampoline_split(ts, att); + if (err) { + node_free(node); + return err; + } + } + + hlist_add_head(&node->hlist_attach, &attach->nodes); + return 0; +} + +static int tramp_state_revert(struct tramp_state *ts, struct attach_args *att) +{ + struct bpf_trampoline *tr_common = ts->tr_common; + int err; + + bpf_trampoline_dup_destroy(ts->tr_other); + + mutex_lock(&tr_common->mutex); + err = bpf_trampoline_unregister(tr_common); + if (err) + goto out; + + tr_common->id = ts->id; + err = bpf_trampoline_setup(tr_common, att); + if (err) + goto out; + + WARN_ON_ONCE(bpf_trampoline_register(tr_common)); +out: + mutex_unlock(&tr_common->mutex); + return err; +} + +static int +bpf_tramp_attach_single(struct bpf_tramp_attach *attach, + struct bpf_tramp_id *id, struct attach_args *att) +{ + struct bpf_trampoline *tr = NULL; + struct bpf_tramp_node *node; + int err; tr = bpf_trampoline_get(id); if (!tr) { @@ -647,22 +1101,175 @@ struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, goto out; } - node = node_alloc(tr, prog); + node = node_alloc(tr, att->prog); if (!node) goto out; - err = bpf_check_attach_model(prog, tgt_prog, id->id[0], &tr->func.model); + err = bpf_check_attach_model(att->prog, att->tgt_prog, + id->id[0], &tr->func.model); if (err) goto out; - attach->id = id; hlist_add_head(&node->hlist_attach, &attach->nodes); - return attach; + return 0; out: bpf_trampoline_put(tr); - kfree(attach); - return ERR_PTR(err); + return err; +} + +#define list_for_each_trampoline(tr, i) \ + for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++) \ + hlist_for_each_entry(tr, &trampoline_table[i], hlist) + +static int __bpf_tramp_attach(struct bpf_tramp_attach *attach, + struct bpf_tramp_id *id, + struct attach_args *att) +{ + struct attach_state state = {}; + struct bpf_tramp_id *id_cont; + struct bpf_trampoline *tr; + bool id_put = false; + int err = 0, i, j; + + mutex_lock(&trampoline_mutex); + + /* If we are ataching single ID trampoline and there's no multi ID + * trampoline registered, there's no need to iterate all trampolines + * for intersection, we can do the fast path and use hash search. + * */ + if (!bpf_tramp_id_is_multi(id) && !nr_bpf_trampoline_multi) { + err = bpf_tramp_attach_single(attach, id, att); + goto out; + } + + /* Iterate all trampolines to find all the interesections. */ + list_for_each_trampoline(tr, i) { + err = tramp_state_compute(&state, tr, id, &id_cont); + if (err) + goto out_multi; + id_put = true; + id = id_cont; + if (!id) + goto out_break; + } +out_break: + + /* Do the actuall trampoline splits if there's any .. */ + for (i = 0; i < state.cnt; i++) { + err = tramp_state_apply(attach, &state.ts[i], att); + if (err) + goto revert; + } + + /* .. and create new trampoline if needed. */ + if (id) + err = bpf_trampoline_create(attach, id, att); + +revert: + /* Attach failed, let's revert already changed trampolines */ + if (err) { + for (j = 0; j < i; j++) + WARN_ON_ONCE(tramp_state_revert(&state.ts[j], att)); + } + +out_multi: + if (id_put) + bpf_tramp_id_put(id); +out: + mutex_unlock(&trampoline_mutex); + state_cleanup(&state); + return err; +} + +#define MAX_ARGS 7 + +static void put_args(struct bpf_tramp_id **args) +{ + int i; + + for (i = 0; i < MAX_ARGS; i++) + bpf_tramp_id_put(args[i]); +} + +static int get_args(struct bpf_tramp_id *id, struct bpf_tramp_id **args, + struct bpf_prog *tgt_prog, struct bpf_prog *prog) +{ + const struct btf_type *t; + struct bpf_tramp_id *a; + const struct btf *btf; + int err = -EINVAL; + u32 i, nargs; + + btf = tgt_prog ? tgt_prog->aux->btf : prog->aux->attach_btf; + if (!btf) + return -EINVAL; + + for (i = 0; i < id->cnt; i++){ + t = btf_type_by_id(btf, id->id[i]); + if (!btf_type_is_func(t)) + goto out_free; + t = btf_type_by_id(btf, t->type); + if (!btf_type_is_func_proto(t)) + goto out_free; + nargs = btf_type_vlen(t); + if (nargs >= MAX_ARGS) + goto out_free; + a = args[nargs]; + if (!a) { + a = bpf_tramp_id_alloc(id->cnt); + if (!a) { + err = -ENOMEM; + goto out_free; + } + a->obj_id = id->obj_id; + args[nargs] = a; + } + id_add(a, id->id[i], id->addr[i]); + } + err = 0; +out_free: + if (err) + put_args(args); + return err; +} + +struct bpf_tramp_attach *bpf_tramp_attach(struct bpf_tramp_id *id, + struct bpf_prog *tgt_prog, + struct bpf_prog *prog) +{ + struct bpf_tramp_id *args[MAX_ARGS] = {}; + struct bpf_tramp_attach *attach; + struct attach_args att = { + .tgt_prog = tgt_prog, + .prog = prog, + }; + int i, err; + + err = get_args(id, args, tgt_prog, prog); + if (err) + return ERR_PTR(err); + + attach = kzalloc(sizeof(*attach), GFP_KERNEL); + if (!attach) + return ERR_PTR(-ENOMEM); + + for (i = 0; i < MAX_ARGS; i++) { + if (!args[i] || !args[i]->cnt) + continue; + att.nr_args = i; + err = __bpf_tramp_attach(attach, args[i], &att); + if (err) + break; + } + + if (err) + bpf_tramp_detach(attach); + else + attach->id = id; + + put_args(args); + return err ? ERR_PTR(err) : attach; } void bpf_tramp_detach(struct bpf_tramp_attach *attach) @@ -670,8 +1277,10 @@ void bpf_tramp_detach(struct bpf_tramp_attach *attach) struct bpf_tramp_node *node; struct hlist_node *n; + mutex_lock(&trampoline_mutex); hlist_for_each_entry_safe(node, n, &attach->nodes, hlist_attach) node_free(node); + mutex_unlock(&trampoline_mutex); bpf_tramp_id_put(attach->id); kfree(attach); @@ -682,13 +1291,14 @@ int bpf_tramp_attach_link(struct bpf_tramp_attach *attach) struct bpf_tramp_node *node; int err; + mutex_lock(&trampoline_mutex); hlist_for_each_entry(node, &attach->nodes, hlist_attach) { err = bpf_trampoline_link_prog(node, node->tr); if (err) - return err; + break; } - - return 0; + mutex_unlock(&trampoline_mutex); + return err; } int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach) @@ -696,13 +1306,14 @@ int bpf_tramp_attach_unlink(struct bpf_tramp_attach *attach) struct bpf_tramp_node *node; int err; + mutex_lock(&trampoline_mutex); hlist_for_each_entry(node, &attach->nodes, hlist_attach) { err = bpf_trampoline_unlink_prog(node, node->tr); if (err) - return err; + break; } - - return 0; + mutex_unlock(&trampoline_mutex); + return err; } #define NO_START_TIME 1 From patchwork Thu Nov 18 11:24:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626691 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3091AC433EF for ; Thu, 18 Nov 2021 11:29:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 19D5A61246 for ; Thu, 18 Nov 2021 11:29:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344097AbhKRLcD (ORCPT ); Thu, 18 Nov 2021 06:32:03 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:32500 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343777AbhKRLaB (ORCPT ); Thu, 18 Nov 2021 06:30:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234820; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=trNwr7SPOVoQqmM8yn9ktA9tP84bxy9JXX7Q0ADRDic=; b=huVc9GXmlpQsMFxxdT5VzNwEjO6pnzh96RpNiOoBBuyQYjWJ/UEj8MtKua5WIfDP867Q/h xSFFFXfMmqaczaxhNvhGZp9wH5qwUSbVhD64aoWVD9ZHS5A3ho3dW6hcuT/CZMzlFfgWjT PfMoJOOuoDjpcAr3gAvsbq5YBvDcDDg= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-473-FTzmLfGBMlaDOurPOjn14w-1; Thu, 18 Nov 2021 06:26:59 -0500 X-MC-Unique: FTzmLfGBMlaDOurPOjn14w-1 Received: by mail-ed1-f72.google.com with SMTP id w4-20020aa7cb44000000b003e7c0f7cfffso5017123edt.2 for ; Thu, 18 Nov 2021 03:26:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=trNwr7SPOVoQqmM8yn9ktA9tP84bxy9JXX7Q0ADRDic=; b=Q4ofx/+D9eBD36VAzDgJVFSM2ZQ3h8EeYFxgnXrmZoPqItgn4JXB/i1m1JoZXLJk+X 3F+07qN+cZ0JX0qIgidq9QrzVuM0Uk4KHkEkiTIxeZeL5rfqiNsYNuVo2nUYWkwcyK7M OGXCiruL+L5g+zgigfExxZzR5hXAD0GGgu3n2s+o8NQMHsF+ktefM1vcl4c3Sek18n86 UdFR/hJd9ohGCEdIXFHXIx7CmhD1Xp0cr7Am0llY7PCAskg9n8SjtTW2hEc9X1PbzBHN eSgOQRze5Gqfw0l6Qzjt3qtMRiEwVI9ErkCim2wRuu2klzS/Q2lYtE7uZxrATw2bqe5C mi/g== X-Gm-Message-State: AOAM5320fPkN6rTyNfLFssH/LJbXTA8Bw0FWXwElArVL4uRS/4WH0Z62 2A1PrIN0LxMTsDvsAf94d8RcV74Ek3TOG7Hd/zf4w4X8sWuRdtYkpVlKJZbIcIGmKJYPH//z3Yo xYjAoXXSBb0zK4Bws X-Received: by 2002:a17:906:579a:: with SMTP id k26mr33154972ejq.250.1637234817398; Thu, 18 Nov 2021 03:26:57 -0800 (PST) X-Google-Smtp-Source: ABdhPJxKJHJ6x9ROA7IxQZgLLzVIxfF3mQIcI90cAxeAkLLzDgP1YWIht9JazeROo4R4KRo8iwygmg== X-Received: by 2002:a17:906:579a:: with SMTP id k26mr33154947ejq.250.1637234817188; Thu, 18 Nov 2021 03:26:57 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id v3sm1613657edc.69.2021.11.18.03.26.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:26:56 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 20/29] bpf: Add support for tracing multi link Date: Thu, 18 Nov 2021 12:24:46 +0100 Message-Id: <20211118112455.475349-21-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding new link to allow to attach program to multiple function BTF IDs. New fields are added to bpf_attr::link_create to pass array of BTF IDs: struct { __aligned_u64 btf_ids; /* addresses to attach */ __u32 btf_ids_cnt; /* addresses count */ } multi; The new link code will load these IDs into bpf_tramp_id and resolve their ips. The resolve itself is done as per Andrii's suggestion: - lookup all names for given IDs - store and sort them by name - go through all kallsyms symbols and use bsearch to find it in provided names - if name is found, store the address for the name - resort the names array based on ID If there are multi symbols of the same name the first one will be used to resolve the address. The new link will pass them to the bpf_tramp_attach that does all the work of attaching. Signed-off-by: Jiri Olsa --- include/uapi/linux/bpf.h | 5 + kernel/bpf/syscall.c | 252 +++++++++++++++++++++++++++++++++ kernel/kallsyms.c | 2 +- tools/include/uapi/linux/bpf.h | 5 + 4 files changed, 263 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index ca05e35e0478..a03a5bc1d141 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1009,6 +1009,7 @@ enum bpf_link_type { BPF_LINK_TYPE_NETNS = 5, BPF_LINK_TYPE_XDP = 6, BPF_LINK_TYPE_PERF_EVENT = 7, + BPF_LINK_TYPE_TRACING_MULTI = 8, MAX_BPF_LINK_TYPE, }; @@ -1470,6 +1471,10 @@ union bpf_attr { */ __u64 bpf_cookie; } perf_event; + struct { + __aligned_u64 btf_ids; /* addresses to attach */ + __u32 btf_ids_cnt; /* addresses count */ + } multi; }; } link_create; diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index bfbd81869818..e6f48dc9dd48 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -32,6 +32,9 @@ #include #include #include +#include +#include +#include #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -2905,6 +2908,251 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, return err; } +struct bpf_tracing_multi_link { + struct bpf_link link; + enum bpf_attach_type attach_type; + struct bpf_tramp_attach *attach; +}; + +static void bpf_tracing_multi_link_release(struct bpf_link *link) +{ + struct bpf_prog *prog = link->prog; + struct bpf_tramp_attach *attach = prog->aux->attach; + + WARN_ON_ONCE(bpf_tramp_attach_unlink(attach)); + + if (prog->type != BPF_PROG_TYPE_EXT) + prog->aux->attach = NULL; + bpf_tramp_detach(attach); +} + +static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + kfree(tr_link); +} + +static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + seq_printf(seq, "attach_type:\t%d\n", tr_link->attach_type); +} + +static int bpf_tracing_multi_link_fill_link_info(const struct bpf_link *link, + struct bpf_link_info *info) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + info->tracing.attach_type = tr_link->attach_type; + return 0; +} + +static const struct bpf_link_ops bpf_tracing_multi_link_lops = { + .release = bpf_tracing_multi_link_release, + .dealloc = bpf_tracing_multi_link_dealloc, + .show_fdinfo = bpf_tracing_multi_link_show_fdinfo, + .fill_link_info = bpf_tracing_multi_link_fill_link_info, +}; + +static int check_multi_prog_type(struct bpf_prog *prog) +{ + if (prog->expected_attach_type != BPF_TRACE_FENTRY && + prog->expected_attach_type != BPF_TRACE_FEXIT) + return -EINVAL; + return 0; +} + +static int btf_ids_cmp(const void *a, const void *b) +{ + const u32 *x = a; + const u32 *y = b; + + if (*x == *y) + return 0; + return *x < *y ? -1 : 1; +} + +struct resolve_id { + const char *name; + void *addr; + u32 id; +}; + +static int rid_name_cmp(const void *a, const void *b) +{ + const struct resolve_id *x = a; + const struct resolve_id *y = b; + + return strcmp(x->name, y->name); +} + +static int rid_id_cmp(const void *a, const void *b) +{ + const struct resolve_id *x = a; + const struct resolve_id *y = b; + + if (x->id == y->id) + return 0; + return x->id < y->id ? -1 : 1; +} + +struct kallsyms_data { + struct resolve_id *rid; + u32 cnt; + u32 found; +}; + +static int kallsyms_callback(void *data, const char *name, + struct module *mod, unsigned long addr) +{ + struct kallsyms_data *args = data; + struct resolve_id *rid, id = { + .name = name, + }; + + rid = bsearch(&id, args->rid, args->cnt, sizeof(*rid), rid_name_cmp); + if (rid && !rid->addr) { + rid->addr = (void *) addr; + args->found++; + } + return args->found == args->cnt ? 1 : 0; +} + +static int bpf_tramp_id_resolve(struct bpf_tramp_id *id, struct bpf_prog *prog) +{ + struct kallsyms_data args; + const struct btf_type *t; + struct resolve_id *rid; + const char *name; + struct btf *btf; + int err = 0; + u32 i; + + btf = prog->aux->attach_btf; + if (!btf) + return -EINVAL; + + rid = kzalloc(id->cnt * sizeof(*rid), GFP_KERNEL); + if (!rid) + return -ENOMEM; + + err = -EINVAL; + for (i = 0; i < id->cnt; i++) { + t = btf_type_by_id(btf, id->id[i]); + if (!t) + goto out_free; + + name = btf_name_by_offset(btf, t->name_off); + if (!name) + goto out_free; + + rid[i].name = name; + rid[i].id = id->id[i]; + } + + sort(rid, id->cnt, sizeof(*rid), rid_name_cmp, NULL); + + args.rid = rid; + args.cnt = id->cnt; + args.found = 0; + kallsyms_on_each_symbol(kallsyms_callback, &args); + + sort(rid, id->cnt, sizeof(*rid), rid_id_cmp, NULL); + + for (i = 0; i < id->cnt; i++) { + if (!rid[i].addr) { + err = -EINVAL; + goto out_free; + } + id->addr[i] = rid[i].addr; + } + err = 0; +out_free: + kfree(rid); + return err; +} + +static int bpf_tracing_multi_attach(struct bpf_prog *prog, + const union bpf_attr *attr) +{ + void __user *uids = u64_to_user_ptr(attr->link_create.multi.btf_ids); + u32 cnt_size, cnt = attr->link_create.multi.btf_ids_cnt; + struct bpf_tracing_multi_link *link = NULL; + struct bpf_link_primer link_primer; + struct bpf_tramp_attach *attach; + struct bpf_tramp_id *id = NULL; + int err = -EINVAL; + + if (check_multi_prog_type(prog)) + return -EINVAL; + if (!cnt || !uids) + return -EINVAL; + + id = bpf_tramp_id_alloc(cnt); + if (!id) + return -ENOMEM; + + err = -EFAULT; + cnt_size = cnt * sizeof(id->id[0]); + if (copy_from_user(id->id, uids, cnt_size)) + goto out_free_id; + + id->cnt = cnt; + id->obj_id = btf_obj_id(prog->aux->attach_btf); + + /* Sort user provided BTF ids, so we can use memcmp + * and bsearch on top of it later. + */ + sort(id->id, cnt, sizeof(u32), btf_ids_cmp, NULL); + + err = bpf_tramp_id_resolve(id, prog); + if (err) + goto out_free_id; + + attach = bpf_tramp_attach(id, NULL, prog); + if (IS_ERR(attach)) { + err = PTR_ERR(attach); + goto out_free_id; + } + + link = kzalloc(sizeof(*link), GFP_KERNEL); + if (!link) { + err = -ENOMEM; + goto out_detach; + } + + bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI, + &bpf_tracing_multi_link_lops, prog); + link->attach_type = prog->expected_attach_type; + + err = bpf_link_prime(&link->link, &link_primer); + if (err) { + kfree(link); + goto out_detach; + } + + err = bpf_tramp_attach_link(attach); + if (err) { + bpf_link_cleanup(&link_primer); + goto out_detach; + } + prog->aux->attach = attach; + return bpf_link_settle(&link_primer); + +out_detach: + bpf_tramp_detach(attach); + return err; +out_free_id: + bpf_tramp_id_put(id); + return err; +} + struct bpf_raw_tp_link { struct bpf_link link; struct bpf_raw_event_map *btp; @@ -3211,6 +3459,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) case BPF_CGROUP_SETSOCKOPT: return BPF_PROG_TYPE_CGROUP_SOCKOPT; case BPF_TRACE_ITER: + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: return BPF_PROG_TYPE_TRACING; case BPF_SK_LOOKUP: return BPF_PROG_TYPE_SK_LOOKUP; @@ -4270,6 +4520,8 @@ static int tracing_bpf_link_attach(const union bpf_attr *attr, bpfptr_t uattr, if (prog->expected_attach_type == BPF_TRACE_ITER) return bpf_iter_link_attach(attr, uattr, prog); + else if (prog->aux->multi_func) + return bpf_tracing_multi_attach(prog, attr); else if (prog->type == BPF_PROG_TYPE_EXT) return bpf_tracing_prog_attach(prog, attr->link_create.target_fd, diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c index 3011bc33a5ba..904e140c3491 100644 --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -224,7 +224,7 @@ unsigned long kallsyms_lookup_name(const char *name) return module_kallsyms_lookup_name(name); } -#ifdef CONFIG_LIVEPATCH +#if defined(CONFIG_LIVEPATCH) || defined(CONFIG_BPF) /* * Iterate over all symbols in vmlinux. For symbols from modules use * module_kallsyms_on_each_symbol instead. diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index ca05e35e0478..a03a5bc1d141 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1009,6 +1009,7 @@ enum bpf_link_type { BPF_LINK_TYPE_NETNS = 5, BPF_LINK_TYPE_XDP = 6, BPF_LINK_TYPE_PERF_EVENT = 7, + BPF_LINK_TYPE_TRACING_MULTI = 8, MAX_BPF_LINK_TYPE, }; @@ -1470,6 +1471,10 @@ union bpf_attr { */ __u64 bpf_cookie; } perf_event; + struct { + __aligned_u64 btf_ids; /* addresses to attach */ + __u32 btf_ids_cnt; /* addresses count */ + } multi; }; } link_create; From patchwork Thu Nov 18 11:24:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626693 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA0A8C433FE for ; Thu, 18 Nov 2021 11:29:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D3B0261246 for ; Thu, 18 Nov 2021 11:29:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344104AbhKRLcJ (ORCPT ); Thu, 18 Nov 2021 06:32:09 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:55660 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343868AbhKRLaG (ORCPT ); Thu, 18 Nov 2021 06:30:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Hi37+rv3u/Bm4bA8fo7TndVIQusjlyQQ5lAmm+EnXUU=; b=Ciki0JH64iXGJKvx26m9VR+ZOJ3FIVkkhd1iOMGklOU102ABwkbRaseFepEj+/UFdZ1TuW hM6M6WDhNWnKgmoba5NfvlxZ1Mn8ma4LlIQgUihGxXkUcJ6l1q1ujN2SsGqKM2AjLmAedA C50U3LFM3mGxK7dsGfgt5A9GlHR1tW0= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-181-iJjWuAqNMs-8Qi22Jh8WoA-1; Thu, 18 Nov 2021 06:27:05 -0500 X-MC-Unique: iJjWuAqNMs-8Qi22Jh8WoA-1 Received: by mail-ed1-f71.google.com with SMTP id b15-20020aa7c6cf000000b003e7cf0f73daso4942931eds.22 for ; Thu, 18 Nov 2021 03:27:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Hi37+rv3u/Bm4bA8fo7TndVIQusjlyQQ5lAmm+EnXUU=; b=gdjkNixIJChC8/fvEXBKebkCp7ce+CVk5DCC82pcknB7QY2AJvig6M0tYgUbPqnItk ZCWV/jJ7mdovlqvjcMMn6RbWGc/gkDgFrj2MBBW5r0M3Q8IL0+mxIlFUW+5E3RQj6qFI rE/k3bU5Hl8yeo+aLPcI5g/r2myZV2O780Iv4tP6JbPvldIZGtRpyMaiE4gREXmMp5rr bEEfKlkXnpq9ZEYvMKKvlaSfjoI2bu9zMcLuDeCxvWHqUEdGt7vOt9AKRNhVqXg0ihYK ubEBVGPl35qKfubL2q47tzzKm1qpplc5g1EELVVuPmuG1ufQLmfO1aTdc3iJtcg3aHWV EDRg== X-Gm-Message-State: AOAM531kumXM3WaWrh0mjZ8HtMmFB1UUCt2c67iiQ0BYQvF5YqRogTVH OXvQRtdkWwh6C5s1pskdcXJJLT+yVXtLfYKpwjgT1QikvA19UnkZwE18mWm23qjuqg8cOSBhHbE ldnG5V9J8l0v4Qawq X-Received: by 2002:aa7:d997:: with SMTP id u23mr10186220eds.164.1637234823313; Thu, 18 Nov 2021 03:27:03 -0800 (PST) X-Google-Smtp-Source: ABdhPJw5CZuVoobHUPvbd2K9Ul9ZEYH9a1jgeG1KjMB4ZHqm9OEPDgAz0puckb36VyPxan9tOL5pjA== X-Received: by 2002:aa7:d997:: with SMTP id u23mr10186197eds.164.1637234823141; Thu, 18 Nov 2021 03:27:03 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id l16sm568515edb.59.2021.11.18.03.27.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:02 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: Andrii Nakryiko , netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 21/29] libbpf: Add btf__find_by_glob_kind function Date: Thu, 18 Nov 2021 12:24:47 +0100 Message-Id: <20211118112455.475349-22-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding btf__find_by_glob_kind function that returns array of BTF ids that match given kind and allow/deny patterns. int btf__find_by_glob_kind(const struct btf *btf, __u32 kind, const char *allow_pattern, const char *deny_pattern, __u32 **__ids); The __ids array is allocated and needs to be manually freed. At the moment the supported pattern is '*' at the beginning or the end of the pattern. Kindly borrowed from retsnoop. Suggested-by: Andrii Nakryiko Signed-off-by: Jiri Olsa --- tools/lib/bpf/btf.c | 77 +++++++++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/btf.h | 3 ++ 2 files changed, 80 insertions(+) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index b6be579e0dc6..ebc02576390d 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -749,6 +749,83 @@ __s32 btf__find_by_name_kind(const struct btf *btf, const char *type_name, return btf_find_by_name_kind(btf, 1, type_name, kind); } +/* 'borrowed' from retsnoop */ +static bool glob_matches(const char *glob, const char *s) +{ + int n = strlen(glob); + + if (n == 1 && glob[0] == '*') + return true; + + if (glob[0] == '*' && glob[n - 1] == '*') { + const char *subs; + /* substring match */ + + /* this is hacky, but we don't want to allocate for no good reason */ + ((char *)glob)[n - 1] = '\0'; + subs = strstr(s, glob + 1); + ((char *)glob)[n - 1] = '*'; + + return subs != NULL; + } else if (glob[0] == '*') { + size_t nn = strlen(s); + /* suffix match */ + + /* too short for a given suffix */ + if (nn < n - 1) + return false; + + return strcmp(s + nn - (n - 1), glob + 1) == 0; + } else if (glob[n - 1] == '*') { + /* prefix match */ + return strncmp(s, glob, n - 1) == 0; + } else { + /* exact match */ + return strcmp(glob, s) == 0; + } +} + +int btf__find_by_glob_kind(const struct btf *btf, __u32 kind, + const char *allow_pattern, const char *deny_pattern, + __u32 **__ids) +{ + __u32 i, nr_types = btf__get_nr_types(btf); + int cnt = 0, alloc = 0; + __u32 *ids = NULL; + + for (i = 1; i <= nr_types; i++) { + const struct btf_type *t = btf__type_by_id(btf, i); + const char *name; + __u32 *p; + + if (btf_kind(t) != kind) + continue; + name = btf__name_by_offset(btf, t->name_off); + if (!name) + continue; + + if (deny_pattern && glob_matches(deny_pattern, name)) + continue; + if (allow_pattern && !glob_matches(allow_pattern, name)) + continue; + + if (cnt == alloc) { + alloc = max(16, alloc * 3 / 2); + p = libbpf_reallocarray(ids, alloc, sizeof(__u32)); + if (!p) { + free(ids); + return -ENOMEM; + } + ids = p; + } + ids[cnt] = i; + cnt++; + } + + *__ids = ids; + return cnt; +} + static bool btf_is_modifiable(const struct btf *btf) { return (void *)btf->hdr != btf->raw_data; diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h index 5c73a5b0a044..408b8e6d913b 100644 --- a/tools/lib/bpf/btf.h +++ b/tools/lib/bpf/btf.h @@ -572,6 +572,9 @@ static inline struct btf_decl_tag *btf_decl_tag(const struct btf_type *t) return (struct btf_decl_tag *)(t + 1); } +int btf__find_by_glob_kind(const struct btf *btf, __u32 kind, + const char *allow_pattern, const char *deny_pattern, + __u32 **__ids); #ifdef __cplusplus } /* extern "C" */ #endif From patchwork Thu Nov 18 11:24:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626695 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E55AAC433F5 for ; Thu, 18 Nov 2021 11:29:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CDA3061246 for ; Thu, 18 Nov 2021 11:29:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344034AbhKRLcO (ORCPT ); Thu, 18 Nov 2021 06:32:14 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:57435 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343894AbhKRLaL (ORCPT ); Thu, 18 Nov 2021 06:30:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bifxS4z5leMtv9W8fU/+OTbvzsJd1lcEkpE8E9OLG6I=; b=OAIZf4x/yaPDg+mMZgRWjAtxv/ADg5YGlbjOrUiieGZRayzMMyCvkOC6JJR9Gfm09mbkJj /3Gq5HRMy7gGFivX/Ou1pIDqGNFYFb/H8e7YtgImzP2MdxHd4NtfoeN7hqDbvmYI/rWjpl 4IWfFN8itOfEgJXDzxXNpeKD5myw3xc= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-536-hpS6kK2JPXaPk37fC68Tfw-1; Thu, 18 Nov 2021 06:27:10 -0500 X-MC-Unique: hpS6kK2JPXaPk37fC68Tfw-1 Received: by mail-ed1-f70.google.com with SMTP id m8-20020a056402510800b003e29de5badbso4965072edd.18 for ; Thu, 18 Nov 2021 03:27:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bifxS4z5leMtv9W8fU/+OTbvzsJd1lcEkpE8E9OLG6I=; b=HoFtOwGQZPB/Wy8WJ2zUFMFTifLKb3JwOdPFs0tQE6rxxYqvnObfxSfJdxU8fQKfeW 1QmR6+WhwYdNXJeGaAafL0fdy2IVxveV88NneQPaf5alQ64VxRr66+VQWYEpKEr6wdux BrseuJODrkjhatCVXagkdpAQ8kzo+AlYB5bfEYd8LrxxFLG4UwPpkApCBy8jMTRhz371 CS0N5sSe2zEWTyXfzS0obAQDlOO+VknKuARyRh3G7Cjls0bZPFu5QSZ3sjT+MONiBDV4 AcpX+dlLT3N2HG5ZsIc1/ZfAsjjzmi39AGdAXUCYYgWEgkEWM6XOT15BTDr2iOmTOgTP LHDQ== X-Gm-Message-State: AOAM532j45GigTTq+uMFTLgHtuvV/GWoDfvbvMbZrRGpMx+V/YznFuxO Rmz0ueXV36uJoaMPfpNYl6ONwZvgc3O+kp9VmcwgQBOK7YrmIHH6338Lbz0O5oJ09ls/OV9l5wL 92NRk7JaFG0c4VvqW X-Received: by 2002:a17:907:972a:: with SMTP id jg42mr33022580ejc.398.1637234829295; Thu, 18 Nov 2021 03:27:09 -0800 (PST) X-Google-Smtp-Source: ABdhPJzXNac21cC6ptXLwE8vb976WG6H1rW4zE4vFEKA+xATsZW9m+DG21K1kz5v58zfN/C4PnINUw== X-Received: by 2002:a17:907:972a:: with SMTP id jg42mr33022545ejc.398.1637234829081; Thu, 18 Nov 2021 03:27:09 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id bd12sm1464972edb.11.2021.11.18.03.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:08 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 22/29] libbpf: Add support to link multi func tracing program Date: Thu, 18 Nov 2021 12:24:48 +0100 Message-Id: <20211118112455.475349-23-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to link multi func tracing program through link_create interface. Adding special types for multi func programs: fentry.multi fexit.multi so you can define multi func programs like: SEC("fentry.multi/bpf_fentry_test*") int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) that defines test1 to be attached to bpf_fentry_test* functions. The test1 program is loaded with BPF_F_MULTI_FUNC flag. If functions are not specified the program needs to be attached manually. Adding new btf_ids/btf_ids_cnt fields to bpf_link_create_opts, that define functions to attach the program to. Signed-off-by: Jiri Olsa --- tools/lib/bpf/bpf.c | 7 +++++ tools/lib/bpf/bpf.h | 6 +++- tools/lib/bpf/libbpf.c | 66 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 1 deletion(-) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 94560ba31724..86a95419e501 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -784,6 +784,13 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, perf_event)) return libbpf_err(-EINVAL); break; + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: + attr.link_create.multi.btf_ids = (__u64) OPTS_GET(opts, multi.btf_ids, 0); + attr.link_create.multi.btf_ids_cnt = OPTS_GET(opts, multi.btf_ids_cnt, 0); + if (!OPTS_ZEROED(opts, multi)) + return libbpf_err(-EINVAL); + break; default: if (!OPTS_ZEROED(opts, flags)) return libbpf_err(-EINVAL); diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 079cc81ac51e..e55abf3528b3 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -249,10 +249,14 @@ struct bpf_link_create_opts { struct { __u64 bpf_cookie; } perf_event; + struct { + __u32 *btf_ids; + __u32 btf_ids_cnt; + } multi; }; size_t :0; }; -#define bpf_link_create_opts__last_field perf_event +#define bpf_link_create_opts__last_field multi LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, enum bpf_attach_type attach_type, diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index de7e09a6b5ec..4c11d38b1f92 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -246,6 +246,8 @@ enum sec_def_flags { SEC_SLEEPABLE = 8, /* allow non-strict prefix matching */ SEC_SLOPPY_PFX = 16, + /* BPF program type allows multiple functions attachment */ + SEC_MULTI_FUNC = 32, }; struct bpf_sec_def { @@ -6723,6 +6725,9 @@ static int bpf_object_init_progs(struct bpf_object *obj, const struct bpf_object continue; } + if (prog->sec_def->cookie & SEC_MULTI_FUNC) + prog->prog_flags |= BPF_F_MULTI_FUNC; + bpf_program__set_type(prog, prog->sec_def->prog_type); bpf_program__set_expected_attach_type(prog, prog->sec_def->expected_attach_type); @@ -8318,6 +8323,7 @@ static struct bpf_link *attach_kprobe(const struct bpf_program *prog, long cooki static struct bpf_link *attach_tp(const struct bpf_program *prog, long cookie); static struct bpf_link *attach_raw_tp(const struct bpf_program *prog, long cookie); static struct bpf_link *attach_trace(const struct bpf_program *prog, long cookie); +static struct bpf_link *attach_trace_multi(const struct bpf_program *prog, long cookie); static struct bpf_link *attach_lsm(const struct bpf_program *prog, long cookie); static struct bpf_link *attach_iter(const struct bpf_program *prog, long cookie); @@ -8345,6 +8351,8 @@ static const struct bpf_sec_def section_defs[] = { SEC_DEF("fentry.s/", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), SEC_DEF("fmod_ret.s/", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), SEC_DEF("fexit.s/", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), + SEC_DEF("fentry.multi/", TRACING, BPF_TRACE_FENTRY, SEC_MULTI_FUNC, attach_trace_multi), + SEC_DEF("fexit.multi/", TRACING, BPF_TRACE_FEXIT, SEC_MULTI_FUNC, attach_trace_multi), SEC_DEF("freplace/", EXT, 0, SEC_ATTACH_BTF, attach_trace), SEC_DEF("lsm/", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), SEC_DEF("lsm.s/", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm), @@ -8797,6 +8805,9 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attac __u32 attach_prog_fd = prog->attach_prog_fd; int err = 0; + if (prog->prog_flags & BPF_F_MULTI_FUNC) + return 0; + /* BPF program's BTF ID */ if (attach_prog_fd) { err = libbpf_find_prog_btf_id(attach_name, attach_prog_fd); @@ -10216,6 +10227,61 @@ static struct bpf_link *bpf_program__attach_btf_id(const struct bpf_program *pro return (struct bpf_link *)link; } +static struct bpf_link *bpf_program__attach_multi(const struct bpf_program *prog) +{ + char *pattern = prog->sec_name + strlen(prog->sec_def->sec); + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts); + enum bpf_attach_type attach_type; + int prog_fd, link_fd, cnt, err; + struct bpf_link *link = NULL; + __u32 *ids = NULL; + + prog_fd = bpf_program__fd(prog); + if (prog_fd < 0) { + pr_warn("prog '%s': can't attach before loaded\n", prog->name); + return ERR_PTR(-EINVAL); + } + + err = bpf_object__load_vmlinux_btf(prog->obj, true); + if (err) + return ERR_PTR(err); + + cnt = btf__find_by_glob_kind(prog->obj->btf_vmlinux, BTF_KIND_FUNC, + pattern, NULL, &ids); + if (cnt <= 0) + return ERR_PTR(-EINVAL); + + link = calloc(1, sizeof(*link)); + if (!link) { + err = -ENOMEM; + goto out_err; + } + link->detach = &bpf_link__detach_fd; + + opts.multi.btf_ids = ids; + opts.multi.btf_ids_cnt = cnt; + + attach_type = bpf_program__get_expected_attach_type(prog); + link_fd = bpf_link_create(prog_fd, 0, attach_type, &opts); + if (link_fd < 0) { + err = -errno; + goto out_err; + } + link->fd = link_fd; + free(ids); + return link; + +out_err: + free(link); + free(ids); + return ERR_PTR(err); +} + +static struct bpf_link *attach_trace_multi(const struct bpf_program *prog, long cookie) +{ + return bpf_program__attach_multi(prog); +} + struct bpf_link *bpf_program__attach_trace(const struct bpf_program *prog) { return bpf_program__attach_btf_id(prog); From patchwork Thu Nov 18 11:24:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626697 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49606C433EF for ; Thu, 18 Nov 2021 11:29:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3380361246 for ; Thu, 18 Nov 2021 11:29:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344117AbhKRLcW (ORCPT ); Thu, 18 Nov 2021 06:32:22 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:28877 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344042AbhKRLaT (ORCPT ); Thu, 18 Nov 2021 06:30:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234837; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x9XHLN69dMYL16w7embbO4+Aae5TE1ecNy2oyFj9C+I=; b=U/uTPB7wB8G12qwdQyeAqxpZTG+/KSr7FwORCdQ+ci9HXicknH1qJ71RSAA4op3rf3RnVg 1AYOpJTIRpd/rQMK1K7H3W6gi9P58PLv5zJZcmV4RS2tWGcUZsTt9w/mfbDGPE5y5xWugQ NpipQ6+yTKU6H2X/J59Dj7bC8DlGLR4= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-114-ASXVLc_MNUa2L_qV6j9nvA-1; Thu, 18 Nov 2021 06:27:16 -0500 X-MC-Unique: ASXVLc_MNUa2L_qV6j9nvA-1 Received: by mail-ed1-f69.google.com with SMTP id q17-20020aa7da91000000b003e7c0641b9cso4998816eds.12 for ; Thu, 18 Nov 2021 03:27:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x9XHLN69dMYL16w7embbO4+Aae5TE1ecNy2oyFj9C+I=; b=bzXYYoL6WxNSYYnFQfX9//+xJRdMdItRQJ/yHss9ciLave/q5OmblWgqeT29/azmGn BPrmfeyasxy7RDTCYFsnyWRi1+frUmRlqfGv/G+jc2w+gMaS0g0C1wVvIrbwlvwtstqV spkFDyPyD+h4Nt6cIFE0sHhg/o8ZUnMRrGuz5fpi9pKuBnp+WwVKdjatSyjkmaWgv77A lKxDK1L0Yl717/JnBBwMq/hdL+NAa9dMA5DmAyOuugqrQuJxhejG1/5UddveTrqXDcRm jYRJBqQYMGR4KzLAmZXXdrTcWtamroT0dMnRXtEqB0x/8aulMwTeZq6oPLwOq61zZp92 m7pg== X-Gm-Message-State: AOAM531x3MaT+0b9GQNmn19LJbT9ZpN2rtM/x3lgnxrnRyRLJwtfbeKS kqC5saI8r4WICgtpvqB1wOsX6su4M99lxnq/C21472gJXBK+FTNBfaWt29JYUJ2FEzT3VT8Z/uL rgMZhw0yrVY0bVtnw X-Received: by 2002:a17:907:7d89:: with SMTP id oz9mr32106289ejc.450.1637234835247; Thu, 18 Nov 2021 03:27:15 -0800 (PST) X-Google-Smtp-Source: ABdhPJxKZvEuUwu+FksxFjKTVn0La4xirfYOertt9DeRVat7py3YVHTo0B7rY1K6/fJpo/KnT+8RDg== X-Received: by 2002:a17:907:7d89:: with SMTP id oz9mr32106258ejc.450.1637234835060; Thu, 18 Nov 2021 03:27:15 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id hr11sm1230357ejc.108.2021.11.18.03.27.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:14 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 23/29] selftests/bpf: Add bpf_arg/bpf_ret_value test Date: Thu, 18 Nov 2021 12:24:49 +0100 Message-Id: <20211118112455.475349-24-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding tests for bpf_arg/bpf_ret_value helpers on both fentry and fexit programs. Signed-off-by: Jiri Olsa --- .../selftests/bpf/prog_tests/args_test.c | 34 +++++++++++++++++++ tools/testing/selftests/bpf/progs/args_test.c | 30 ++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/args_test.c create mode 100644 tools/testing/selftests/bpf/progs/args_test.c diff --git a/tools/testing/selftests/bpf/prog_tests/args_test.c b/tools/testing/selftests/bpf/prog_tests/args_test.c new file mode 100644 index 000000000000..1938f2616ee9 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/args_test.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "args_test.skel.h" + +void test_args_test(void) +{ + struct args_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = args_test__open(); + if (!ASSERT_OK_PTR(skel, "args_test__open")) + return; + + err = args_test__load(skel); + if (!ASSERT_OK(err, "args_test__load")) + goto cleanup; + + err = args_test__attach(skel); + if (!ASSERT_OK(err, "args_test__attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test1); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test1_result, 1, "test1_result"); + ASSERT_EQ(skel->bss->test2_result, 1, "test2_result"); + +cleanup: + args_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/args_test.c b/tools/testing/selftests/bpf/progs/args_test.c new file mode 100644 index 000000000000..7fc8e9fb41bd --- /dev/null +++ b/tools/testing/selftests/bpf/progs/args_test.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__u64 test1_result = 0; +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(test1) +{ + __u64 a = bpf_arg(ctx, 0); + __u64 x = bpf_arg(ctx, 1); + + test1_result = (int) a == 1 && x == 0; + return 0; +} + +__u64 test2_result = 0; +SEC("fexit/bpf_fentry_test2") +int BPF_PROG(test2) +{ + __u64 ret = bpf_ret_value(ctx); + __u64 a = bpf_arg(ctx, 0); + __u64 b = bpf_arg(ctx, 1); + __u64 x = bpf_arg(ctx, 2); + + test2_result = (int) a == 2 && b == 3 && ret == 5 && x == 0; + return 0; +} From patchwork Thu Nov 18 11:24:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626699 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97AB8C433FE for ; Thu, 18 Nov 2021 11:29:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8301261AFF for ; Thu, 18 Nov 2021 11:29:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344122AbhKRLca (ORCPT ); Thu, 18 Nov 2021 06:32:30 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:23174 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344046AbhKRLa1 (ORCPT ); Thu, 18 Nov 2021 06:30:27 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234845; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4w9z59EHFbXqy0WBMfhQAPvo2GwQkbl/oaYCrbS/5SI=; b=R5Imxqdok16BEfHDNRWA3hr7aWj/tqdd4tUeLA1UThtyhf9F507/Jc6QYvvB1irGaXD/wz QD2nzxD4ZTc+X708WMBEGCDDOsGiz/BaiAzAilsuLIGIp6c40T4/8wysK+I5vRasmA682W k2OLoG2RPIicsqUl4hHIKkKSOVYEMBE= Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-420-j3gFSWhINK--rSMBv8IyBg-1; Thu, 18 Nov 2021 06:27:22 -0500 X-MC-Unique: j3gFSWhINK--rSMBv8IyBg-1 Received: by mail-ed1-f72.google.com with SMTP id c1-20020aa7c741000000b003e7bf1da4bcso4954468eds.21 for ; Thu, 18 Nov 2021 03:27:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4w9z59EHFbXqy0WBMfhQAPvo2GwQkbl/oaYCrbS/5SI=; b=Wo8sACKFu5V8vHCWLV23dassSSzXoraj6RV8mb9mO5l0DSZnwK4s8QhB4Ch0eV7sK7 LnvQis7T3pkYLHZY2QFmLqyNQb0FsVNExwwQsauzg13qPAmKZUAIyymoA6DAPTe97mSg SXT8UuGL0D4+ALnpJ4mEfXxedRnWaN6kuYLYLDXwxRcJOREn5ECyLhOAIM4oY7UiZGni ZLt8luT11xcmlpYK1vf+IyN45A5lnDdkiFAiLpiDox+V8R4d9xxUYExh/DuZrVruLc/A X6nlL0ceHLvUc79L7gs23yF72/k5THRYa1AIgJfncXhaVN72x7apZ8zwbRd1XdjsBAqS dTeQ== X-Gm-Message-State: AOAM530+VGIoQa3JvUhbElwRlY3poHXFvfns7O0i50oPdUkR7cocIwBc 26SviAyQtBfOCb0naNXnbE7FONvPzlaHXYTmw2p+nS2Nn9fiOTB6VU8hFlFQOGeM9VXb9+xspxw IS1VCdZzpldyYRyUB X-Received: by 2002:a17:907:94d4:: with SMTP id dn20mr32268025ejc.379.1637234841218; Thu, 18 Nov 2021 03:27:21 -0800 (PST) X-Google-Smtp-Source: ABdhPJzpUiSvo4L4FdhFPvzMoyJYHsklqk2A5x6i6r+T8jxcf7dzlN5sh71H1PqoDctmR8fxgzeGzw== X-Received: by 2002:a17:907:94d4:: with SMTP id dn20mr32267998ejc.379.1637234841044; Thu, 18 Nov 2021 03:27:21 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id j3sm1157057ejo.2.2021.11.18.03.27.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:20 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 24/29] selftests/bpf: Add fentry multi func test Date: Thu, 18 Nov 2021 12:24:50 +0100 Message-Id: <20211118112455.475349-25-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fentry multi func test that attaches to bpf_fentry_test* functions and checks argument values based on the processed function. We need to cast to real arguments types in multi_arg_check, because the checked value can be shorter than u64. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/prog_tests/multi_fentry_test.c | 30 +++++++++ .../testing/selftests/bpf/progs/multi_check.c | 63 +++++++++++++++++++ .../selftests/bpf/progs/multi_fentry.c | 17 +++++ 4 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_check.c create mode 100644 tools/testing/selftests/bpf/progs/multi_fentry.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 35684d61aaeb..fa29ddd47cbe 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -322,7 +322,8 @@ endef SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ - linked_vars.skel.h linked_maps.skel.h + linked_vars.skel.h linked_maps.skel.h \ + multi_fentry_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c @@ -334,6 +335,7 @@ test_static_linked.skel.h-deps := test_static_linked1.o test_static_linked2.o linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o +multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c new file mode 100644 index 000000000000..8dc08c3e715f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_fentry_test.skel.h" +#include "trace_helpers.h" + +void test_multi_fentry_test(void) +{ + struct multi_fentry_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_fentry_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load")) + goto cleanup; + + err = multi_fentry_test__attach(skel); + if (!ASSERT_OK(err, "fentry_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test_result, 8, "test_result"); + +cleanup: + multi_fentry_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_check.c b/tools/testing/selftests/bpf/progs/multi_check.c new file mode 100644 index 000000000000..82acc9ee7715 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_check.c @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +extern const void bpf_fentry_test1 __ksym; +extern const void bpf_fentry_test2 __ksym; +extern const void bpf_fentry_test3 __ksym; +extern const void bpf_fentry_test4 __ksym; +extern const void bpf_fentry_test5 __ksym; +extern const void bpf_fentry_test6 __ksym; +extern const void bpf_fentry_test7 __ksym; +extern const void bpf_fentry_test8 __ksym; + +void multi_arg_check(__u64 *ctx, __u64 *test_result) +{ + void *ip = (void *) bpf_get_func_ip(ctx); + + if (ip == &bpf_fentry_test1) { + int a = (int) ctx[0]; + + *test_result += a == 1; + } else if (ip == &bpf_fentry_test2) { + int a = (int) bpf_arg(ctx, 0); + __u64 b = bpf_arg(ctx, 1); + + *test_result += a == 2 && b == 3; + } else if (ip == &bpf_fentry_test3) { + char a = (int) bpf_arg(ctx, 0); + int b = (int) bpf_arg(ctx, 1); + __u64 c = bpf_arg(ctx, 2); + + *test_result += a == 4 && b == 5 && c == 6; + } else if (ip == &bpf_fentry_test4) { + void *a = (void *) bpf_arg(ctx, 0); + char b = (char) bpf_arg(ctx, 1); + int c = (int) bpf_arg(ctx, 2); + __u64 d = bpf_arg(ctx, 3); + + *test_result += a == (void *) 7 && b == 8 && c == 9 && d == 10; + } else if (ip == &bpf_fentry_test5) { + __u64 a = bpf_arg(ctx, 0); + void *b = (void *) bpf_arg(ctx, 1); + short c = (short) bpf_arg(ctx, 2); + int d = (int) bpf_arg(ctx, 3); + __u64 e = bpf_arg(ctx, 4); + + *test_result += a == 11 && b == (void *) 12 && c == 13 && d == 14 && e == 15; + } else if (ip == &bpf_fentry_test6) { + __u64 a = bpf_arg(ctx, 0); + void *b = (void *) bpf_arg(ctx, 1); + short c = (short) bpf_arg(ctx, 2); + int d = (int) bpf_arg(ctx, 3); + void *e = (void *) bpf_arg(ctx, 4); + __u64 f = bpf_arg(ctx, 5); + + *test_result += a == 16 && b == (void *) 17 && c == 18 && d == 19 && e == (void *) 20 && f == 21; + } else if (ip == &bpf_fentry_test7) { + *test_result += 1; + } else if (ip == &bpf_fentry_test8) { + *test_result += 1; + } +} diff --git a/tools/testing/selftests/bpf/progs/multi_fentry.c b/tools/testing/selftests/bpf/progs/multi_fentry.c new file mode 100644 index 000000000000..b78d36772aa6 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_fentry.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__u64 test_result = 0; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); + +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test_result); + return 0; +} From patchwork Thu Nov 18 11:24:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626701 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AD81C433FE for ; Thu, 18 Nov 2021 11:29:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5629361246 for ; Thu, 18 Nov 2021 11:29:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344124AbhKRLcd (ORCPT ); Thu, 18 Nov 2021 06:32:33 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:39431 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344052AbhKRLaa (ORCPT ); Thu, 18 Nov 2021 06:30:30 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234849; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ErIJJtuMRUhbIPvfUeZOmACetWYxDXUh4cGQYrVEag4=; b=BKQOu9ViWFTJFIA8DdrtQrEasyCndVq49GlFV8DEWEMbIDpoU+3MO/8dD8nbIFPnB+PKXy a0gCOZttg4FtnJEm+88PCLCoKGjo7U6J43SnZUPTHhpZsWEG7x7WDZxwbDxppb8a0uk8C4 F1bTmha7JDuniM3h98nZ9lvXkESGENk= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-434-DyE4iWjYOo6ev42Rk1nlMw-1; Thu, 18 Nov 2021 06:27:28 -0500 X-MC-Unique: DyE4iWjYOo6ev42Rk1nlMw-1 Received: by mail-ed1-f69.google.com with SMTP id f4-20020a50e084000000b003db585bc274so4962337edl.17 for ; Thu, 18 Nov 2021 03:27:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ErIJJtuMRUhbIPvfUeZOmACetWYxDXUh4cGQYrVEag4=; b=t4JVAXTsp/e8viqIg+9csjW2ZmIeOtOQM2nNJlH7n8K/0u8myrmvHZZsOG8pa59N8R 2SQpbnaYyNi2dvdGkuk94jOWYacy6P+xPrdDBHT9wP5ZqfGJc/nfaUypu6Ysxbd7sild vbvMRtBS2m1O4XVZZR2PE+nBWVyI9U31XpMzcS9oVwxi1no6SL9SjG9Lo4k32WMp8gY9 MoKMp/tWcnsHJjOU6QxiDiWDyH6pK5Phk0RP+bpXXk5MIHyzrg/IUyuheSrv1QFLLPW5 ivjhWLjSidJd6ZDKsTmOC6FcBMsEoxSIhkDbW1cMdcctBs4IqkxRC5ex7bb6lIs7qdJq qPrQ== X-Gm-Message-State: AOAM532uBBPBJv1WpvE99ZhCdwX8oxg/LrWSm2ZRcmhfoqquYIhQJ7AH 9SyQkG3jDquHGpQcJdSlLQTAa4Z6Xv5IZZ6IuOZ0eTKZOmGKF+2GzczbQ7eUBAIqzyAVB+dbxiT 1x/c0DBA/W7ZlywS/ X-Received: by 2002:a50:9eca:: with SMTP id a68mr9846455edf.127.1637234847328; Thu, 18 Nov 2021 03:27:27 -0800 (PST) X-Google-Smtp-Source: ABdhPJzVH4BLqavpth5rV3X5MI3BbxRqmLyd3X53WvVpD0g4ofqtUoGZrAXTWYLTfvCekr4fdXLXoQ== X-Received: by 2002:a50:9eca:: with SMTP id a68mr9846425edf.127.1637234847146; Thu, 18 Nov 2021 03:27:27 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id hd18sm1109071ejc.84.2021.11.18.03.27.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:26 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 25/29] selftests/bpf: Add fexit multi func test Date: Thu, 18 Nov 2021 12:24:51 +0100 Message-Id: <20211118112455.475349-26-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fexit multi func test that attaches to bpf_fentry_test* functions and checks argument values based on the processed function. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/prog_tests/multi_fexit_test.c | 31 +++++++++++++++++++ .../testing/selftests/bpf/progs/multi_check.c | 23 ++++++++++++++ .../testing/selftests/bpf/progs/multi_fexit.c | 20 ++++++++++++ 4 files changed, 76 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_fexit.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index fa29ddd47cbe..42b67834d803 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -323,7 +323,7 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ - multi_fentry_test.skel.h + multi_fentry_test.skel.h multi_fexit_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c @@ -336,6 +336,7 @@ linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o +multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c new file mode 100644 index 000000000000..d9b0eedd9f45 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_fexit_test.skel.h" +#include "trace_helpers.h" + +void test_multi_fexit_test(void) +{ + struct multi_fexit_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_fexit_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fexit_multi_skel_load")) + goto cleanup; + + err = multi_fexit_test__attach(skel); + if (!ASSERT_OK(err, "fexit_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test_arg_result, 8, "fexit_multi_arg_result"); + ASSERT_EQ(skel->bss->test_ret_result, 8, "fexit_multi_ret_result"); + +cleanup: + multi_fexit_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_check.c b/tools/testing/selftests/bpf/progs/multi_check.c index 82acc9ee7715..2f14d232dd77 100644 --- a/tools/testing/selftests/bpf/progs/multi_check.c +++ b/tools/testing/selftests/bpf/progs/multi_check.c @@ -61,3 +61,26 @@ void multi_arg_check(__u64 *ctx, __u64 *test_result) *test_result += 1; } } + +void multi_ret_check(void *ctx, __u64 *test_result) +{ + void *ip = (void *) bpf_get_func_ip(ctx); + int ret = (int) bpf_ret_value(ctx); + + if (ip == &bpf_fentry_test1) + *test_result += ret == 2; + else if (ip == &bpf_fentry_test2) + *test_result += ret == 5; + else if (ip == &bpf_fentry_test3) + *test_result += ret == 15; + else if (ip == &bpf_fentry_test4) + *test_result += ret == 34; + else if (ip == &bpf_fentry_test5) + *test_result += ret == 65; + else if (ip == &bpf_fentry_test6) + *test_result += ret == 111; + else if (ip == &bpf_fentry_test7) + *test_result += ret == 0; + else if (ip == &bpf_fentry_test8) + *test_result += ret == 0; +} diff --git a/tools/testing/selftests/bpf/progs/multi_fexit.c b/tools/testing/selftests/bpf/progs/multi_fexit.c new file mode 100644 index 000000000000..54624acc7071 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_fexit.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, __u64 *test_result); + +__u64 test_arg_result = 0; +__u64 test_ret_result = 0; + +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test) +{ + multi_arg_check(ctx, &test_arg_result); + multi_ret_check(ctx, &test_ret_result); + return 0; +} From patchwork Thu Nov 18 11:24:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626703 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D9D2C433EF for ; Thu, 18 Nov 2021 11:29:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 71E8F61246 for ; Thu, 18 Nov 2021 11:29:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344129AbhKRLci (ORCPT ); Thu, 18 Nov 2021 06:32:38 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:29759 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343897AbhKRLaf (ORCPT ); Thu, 18 Nov 2021 06:30:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234855; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rsW/H1X5SXKb3SkfnLF8OThiEczj7whjd6wk1z+2Cn4=; b=QsqO70oPaNhy43N7ntWfsm3CPz3bhEwJrftR4Q1ho4vhNDzSMPItqcKs17bUh0CjGII+sw d2wkRCoY3kAyan/m8Ecs9Ewrg9BfETCH9fiJ1Yr8lKqRSYKYOvyqE/VbYFWYhMdcWmJfYj cUhKZgohp9SvIoG5kOjrBvbyk4sO2Co= Received: from mail-ed1-f69.google.com (mail-ed1-f69.google.com [209.85.208.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-323-bkoOao0uPEeGzpyEMLkHFQ-1; Thu, 18 Nov 2021 06:27:34 -0500 X-MC-Unique: bkoOao0uPEeGzpyEMLkHFQ-1 Received: by mail-ed1-f69.google.com with SMTP id k7-20020aa7c387000000b003e7ed87fb31so5041347edq.3 for ; Thu, 18 Nov 2021 03:27:34 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rsW/H1X5SXKb3SkfnLF8OThiEczj7whjd6wk1z+2Cn4=; b=i0O7Pza+2hXagBsNOL7jixB9n7K3Q1ZvWVpx4+e9RU3jnGRQJZCBt2eKvUjHgraJ0X M0541xxvov7zKHmMmrWSKfJKKdEAZzDoJc3QW8pqQm8S+JEIHPK9on+oLKpfn9qS4t62 uP5AI/UIa5t1hLjy4RT7Y99TRH9j9Y9d3LKt5XGoHSa9O9DxTa68R9DqXEAgTtvetcpL /X5hT8B9XiVpEJDsjBfriT0zE6BnnowqWn2KPBPIjRMNY3TKmvaptVIBBf4FjNaMVkXp hYfgp4zHZj53iljwyOjIzfgd1ps84r1y1evXGAP9+I5QmqE7JIo5/eVV70szr0az9ORz n0lA== X-Gm-Message-State: AOAM531X6hmMmVPsaphuJhnRiChezv9KbpHqtV2zscqdU0loykGJKN// 2bGTrNvYbSGRc0BIhYDliu8jz61G1b7WbW8U8V7Ukq4131j5F67pMrvmhq7SkiVBRIKI1K0tA8d IApWFpz4OeO5kJmUz X-Received: by 2002:a17:907:3d94:: with SMTP id he20mr33000604ejc.75.1637234853336; Thu, 18 Nov 2021 03:27:33 -0800 (PST) X-Google-Smtp-Source: ABdhPJzNIOqscLZtBmNLPGLo+AGS5074peYacS+Gr3CSw5E/dwY/58Gho08YE3Vj8B2JynfOt2rC2Q== X-Received: by 2002:a17:907:3d94:: with SMTP id he20mr33000570ejc.75.1637234853179; Thu, 18 Nov 2021 03:27:33 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id l18sm524029ejo.114.2021.11.18.03.27.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:32 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 26/29] selftests/bpf: Add fentry/fexit multi func test Date: Thu, 18 Nov 2021 12:24:52 +0100 Message-Id: <20211118112455.475349-27-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fentry/fexit multi func tests that attaches to bpf_fentry_test* functions and checks argument values based on the processed function. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 4 ++- .../bpf/prog_tests/multi_fentry_fexit_test.c | 32 +++++++++++++++++++ .../selftests/bpf/progs/multi_fentry_fexit.c | 28 ++++++++++++++++ 3 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_fentry_fexit.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 42b67834d803..236b6e0a36de 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -323,7 +323,8 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ - multi_fentry_test.skel.h multi_fexit_test.skel.h + multi_fentry_test.skel.h multi_fexit_test.skel.h \ + multi_fentry_fexit_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c @@ -337,6 +338,7 @@ linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o +multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c new file mode 100644 index 000000000000..d54abf36ab2f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_fentry_fexit_test.skel.h" + +void test_multi_fentry_fexit_test(void) +{ + DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts); + struct multi_fentry_fexit_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_fentry_fexit_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load")) + goto cleanup; + + err = multi_fentry_fexit_test__attach(skel); + if (!ASSERT_OK(err, "fentry_fexit_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test2); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test1_arg_result, 8, "test1_arg_result"); + ASSERT_EQ(skel->bss->test2_arg_result, 8, "test2_arg_result"); + ASSERT_EQ(skel->bss->test2_ret_result, 8, "test2_ret_result"); + +cleanup: + multi_fentry_fexit_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c b/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c new file mode 100644 index 000000000000..54ee94d060b2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__u64 test1_arg_result = 0; +__u64 test2_arg_result = 0; +__u64 test2_ret_result = 0; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, __u64 *test_result); + +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test1_arg_result); + return 0; +} + +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test2_arg_result); + multi_ret_check(ctx, &test2_ret_result); + return 0; +} From patchwork Thu Nov 18 11:24:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626705 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52C03C433EF for ; Thu, 18 Nov 2021 11:29:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3BAD761AFB for ; Thu, 18 Nov 2021 11:29:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344134AbhKRLcn (ORCPT ); Thu, 18 Nov 2021 06:32:43 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:54051 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344056AbhKRLal (ORCPT ); Thu, 18 Nov 2021 06:30:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234861; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rTnzLZH9St4nm5Qk+Gny9wdik+W3lc006QLR20VAmVI=; b=a2VegWR8rNeqATS/hdBDJbeaSK9ocFVUFfJMBTTjDtr1xPkbdffq93E5XrcNtsBD7mqnqH mMwQELuZdSx4hxGEA8df2xu3rnjiA6QTImsVKhNPOKPL6vlcnk3mAQHhv4v45rYy74PXFh tv3yLJkiawul07geucwXKn8FDE5Fv8w= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-151-IvOL1dThNfik-beEiQKdUQ-1; Thu, 18 Nov 2021 06:27:40 -0500 X-MC-Unique: IvOL1dThNfik-beEiQKdUQ-1 Received: by mail-ed1-f71.google.com with SMTP id d13-20020a056402516d00b003e7e67a8f93so5041529ede.0 for ; Thu, 18 Nov 2021 03:27:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rTnzLZH9St4nm5Qk+Gny9wdik+W3lc006QLR20VAmVI=; b=zUG45TS82WQt2Daz0Z2YX7re2XJyYq0uSAIbxRZeWrOo0sO/NiDRqDgwCuDjppw4NC 7Yzo6tdWBJ+U8MNZ0ckrco1MqSmPHPvQK0R+s7zmS8CODcgbTWaDKGFzVMOf40DcKiI0 oV4Np1qfzCIIG0DR076EzX9Tzsxh9lrl5bkzKTOhqNyyJ9/USwH6v9CgHQeWDWBvN3WD cWsmRNG5FCTio00zeLqPZox7040Wljc5FU17aiB9Ry6MpjLPveJ8P/1Sw0Wz3U0WStWd kGbHw+uvByE6fcHlbkvWi3ETCqmv6rK3oAwqgH4YXhTdl+xUe2eRZJ553m4mQT93GybQ QPlw== X-Gm-Message-State: AOAM530eW7RqPWCN8ZStPD+Vd379oDSyRBB4KWqcwkU8XvI3Llfl+3rf MTCxgxfcLKWcvdccsZDWlrNszX8LAzJ4TWLtsM+jyX4A1urWQeFpa1ou5h2D2771Ob9n90crHvB Kfy17wQ7dnUv73ZqB X-Received: by 2002:a05:6402:40d1:: with SMTP id z17mr10061168edb.340.1637234859290; Thu, 18 Nov 2021 03:27:39 -0800 (PST) X-Google-Smtp-Source: ABdhPJx34DKK1KicjwV0TlyXuE5zjue5xGNZ1DzlmaqJ8kIYKUGgW+EPe+5dluSTRQEa46z5KmYAvg== X-Received: by 2002:a05:6402:40d1:: with SMTP id z17mr10061137edb.340.1637234859120; Thu, 18 Nov 2021 03:27:39 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id lk22sm1163424ejb.83.2021.11.18.03.27.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:38 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 27/29] selftests/bpf: Add mixed multi func test Date: Thu, 18 Nov 2021 12:24:53 +0100 Message-Id: <20211118112455.475349-28-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fentry/fexit multi func tests that attaches to bpf_fentry_test* functions, where some of them have already attached trampoline. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/prog_tests/multi_mixed_test.c | 34 +++++++++++++++ .../testing/selftests/bpf/progs/multi_mixed.c | 43 +++++++++++++++++++ 3 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_mixed.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 236b6e0a36de..48970e983250 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -324,7 +324,7 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ multi_fentry_test.skel.h multi_fexit_test.skel.h \ - multi_fentry_fexit_test.skel.h + multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c @@ -339,6 +339,7 @@ linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o +multi_mixed_test.skel.h-deps := multi_mixed.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c b/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c new file mode 100644 index 000000000000..c9395b4eb5ac --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_mixed_test.skel.h" + +void test_multi_mixed_test(void) +{ + DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts); + struct multi_mixed_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_mixed_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load")) + goto cleanup; + + err = multi_mixed_test__attach(skel); + if (!ASSERT_OK(err, "fentry_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test1); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test1_result, 1, "test1_result"); + ASSERT_EQ(skel->bss->test2_result, 1, "test2_result"); + ASSERT_EQ(skel->bss->test3_arg_result, 8, "test3_arg_result"); + ASSERT_EQ(skel->bss->test4_arg_result, 8, "test4_arg_result"); + ASSERT_EQ(skel->bss->test4_ret_result, 8, "test4_ret_result"); + +cleanup: + multi_mixed_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_mixed.c b/tools/testing/selftests/bpf/progs/multi_mixed.c new file mode 100644 index 000000000000..468a044753e9 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_mixed.c @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, __u64 *test_result); + +__u64 test1_result = 0; +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(test1, int a) +{ + test1_result += a == 1; + return 0; +} + +__u64 test2_result = 0; +SEC("fexit/bpf_fentry_test2") +int BPF_PROG(test2, int a, __u64 b, int ret) +{ + test2_result += a == 2 && b == 3; + return 0; +} + +__u64 test3_arg_result = 0; +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test3, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test3_arg_result); + return 0; +} + +__u64 test4_arg_result = 0; +__u64 test4_ret_result = 0; +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test4, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test4_arg_result); + multi_ret_check(ctx, &test4_ret_result); + return 0; +} From patchwork Thu Nov 18 11:24:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626707 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE8E0C433F5 for ; Thu, 18 Nov 2021 11:29:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC4BF61246 for ; Thu, 18 Nov 2021 11:29:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344146AbhKRLcv (ORCPT ); Thu, 18 Nov 2021 06:32:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:54971 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344061AbhKRLau (ORCPT ); Thu, 18 Nov 2021 06:30:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234870; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5EJllEejN/3GIbtkSvycpDyvI7AcT8mw9dELbI2aqcU=; b=V1Z9vRIBKb5YGJoRglRZv10icukgm0C9grwBVlSnzvm58KB7rYTk8e6CMoNLngHsiM6CMt NKwOCmCYjYK7HvX767gmNH518D8NPxyyh/FuwUO/3TG4Ai4yGS2J410Az74uBuABe8GFLg NVM2ACxaB7z1bPYN0ij1NxoZMkjBjo0= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-489-K2XK5RfpNLuVbPlYT-5oaA-1; Thu, 18 Nov 2021 06:27:47 -0500 X-MC-Unique: K2XK5RfpNLuVbPlYT-5oaA-1 Received: by mail-ed1-f71.google.com with SMTP id k7-20020aa7c387000000b003e7ed87fb31so5041697edq.3 for ; Thu, 18 Nov 2021 03:27:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5EJllEejN/3GIbtkSvycpDyvI7AcT8mw9dELbI2aqcU=; b=DHNJ66sgB/rK7wg6eSJGQy3nXGYDfOKz0MOzXx0/B9kuTb9ABj2NZr0RBHdNQ1cSVe VRRCEJEdb4EwGTjYDSYL0PhNIBLlX11STsSsCxdajxglkIY7YaJYH12YCDkfSZNwRG0w HtLtDYr8QmQrJjbBYWAT66waFuDH4/qWG4YojfxIYhYXTP1rZIFyS/6zhn4YBnDbNvel FLR60rNQdYP2ZxRSZNqjuAhgfZITQmsfQ1N/uHFjPdRmhub3p4fVBd/1L9LTXqH9dUTm fgsn+/jKY3+gULEhrhfUkoYTpi467olS5rVY9+1sDVRWVEq0l+gmoEJZdYd0PwFAgzDU OihA== X-Gm-Message-State: AOAM530qYd1mzx+QF5YmvoaGHr7G5l5qXF95Gm1nv2w/818CI07UfS7Q 5z6ny7UpP9UoqtUERkNDgfOtRl85W6t1J9qBnGba5TK2EA8fASMZF9BYvUe9twS2n6c++MKs9vx rEuZ2On1oKikSafF2 X-Received: by 2002:a05:6402:5158:: with SMTP id n24mr9903601edd.230.1637234866207; Thu, 18 Nov 2021 03:27:46 -0800 (PST) X-Google-Smtp-Source: ABdhPJwjPu4vHCTfCb99ok+0tGXBYvYBwTiNwDraYicGV6XtI232nyS+/2MLQMyBi0gpPqy6ey/I7w== X-Received: by 2002:a05:6402:5158:: with SMTP id n24mr9903578edd.230.1637234866068; Thu, 18 Nov 2021 03:27:46 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id x7sm1054136edd.28.2021.11.18.03.27.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:45 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 28/29] selftests/bpf: Add ret_mod multi func test Date: Thu, 18 Nov 2021 12:24:54 +0100 Message-Id: <20211118112455.475349-29-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding extra test to existing modify_return test to test this with multi func program attached on top of the modify return program. Because the supported wildcards do not allow us to match both bpf_fentry_test* and bpf_modify_return_test, adding extra code to look it up in kernel's BTF. Signed-off-by: Jiri Olsa --- .../selftests/bpf/prog_tests/modify_return.c | 114 +++++++++++++++++- .../selftests/bpf/progs/multi_modify_return.c | 17 +++ 2 files changed, 128 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/multi_modify_return.c diff --git a/tools/testing/selftests/bpf/prog_tests/modify_return.c b/tools/testing/selftests/bpf/prog_tests/modify_return.c index b772fe30ce9b..ffb0be7ea5a5 100644 --- a/tools/testing/selftests/bpf/prog_tests/modify_return.c +++ b/tools/testing/selftests/bpf/prog_tests/modify_return.c @@ -5,13 +5,100 @@ */ #include +#include #include "modify_return.skel.h" +#include "multi_modify_return.skel.h" #define LOWER(x) ((x) & 0xffff) #define UPPER(x) ((x) >> 16) -static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) +struct multi_data { + struct multi_modify_return *skel; + int link_fentry; + int link_fexit; + __u32 btf_ids[9]; +}; + +static int multi_btf_ids(struct multi_data *md) +{ + __u32 i, nr_types, ids_cnt; + struct btf *btf; + + btf = btf__load_vmlinux_btf(); + if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf")) + return -1; + + nr_types = btf__get_nr_types(btf); + + for (i = 1; i <= nr_types; i++) { + const struct btf_type *t = btf__type_by_id(btf, i); + const char *name; + bool match; + + if (!btf_is_func(t)) + continue; + + name = btf__name_by_offset(btf, t->name_off); + if (!name) + continue; + match = strncmp(name, "bpf_modify_return_test", + sizeof("bpf_modify_return_test") - 1) == 0; + match |= strncmp(name, "bpf_fentry_test", + sizeof("bpf_fentry_test") - 1) == 0; + if (!match) + continue; + + md->btf_ids[ids_cnt] = i; + ids_cnt++; + } + + btf__free(btf); + return ASSERT_EQ(ids_cnt, 9, "multi_btf_ids") ? 0 : -1; +} + +static int multi_attach(struct multi_data *md) +{ + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts); + int prog_fd; + + md->skel = multi_modify_return__open_and_load(); + if (!ASSERT_OK_PTR(md->skel, "multi_attach_check__load")) + return -1; + + opts.multi.btf_ids = md->btf_ids; + opts.multi.btf_ids_cnt = 9; + + prog_fd = bpf_program__fd(md->skel->progs.test1); + + md->link_fentry = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, &opts); + if (!ASSERT_GE(md->link_fentry, 0, "bpf_link_create")) + goto cleanup; + + prog_fd = bpf_program__fd(md->skel->progs.test2); + + md->link_fexit = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, &opts); + if (!ASSERT_GE(md->link_fexit, 0, "bpf_link_create")) + goto cleanup_close; + + return 0; + +cleanup_close: + close(md->link_fentry); +cleanup: + multi_modify_return__destroy(md->skel); + return -1; +} + +static void multi_detach(struct multi_data *md) +{ + close(md->link_fentry); + close(md->link_fexit); + multi_modify_return__destroy(md->skel); +} + +static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret, + struct multi_data *md) { struct modify_return *skel = NULL; int err, prog_fd; @@ -27,6 +114,9 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) if (CHECK(err, "modify_return", "attach failed: %d\n", err)) goto cleanup; + if (md && !ASSERT_OK(multi_attach(md), "multi_attach")) + goto cleanup; + skel->bss->input_retval = input_retval; prog_fd = bpf_program__fd(skel->progs.fmod_ret_test); err = bpf_prog_test_run(prog_fd, 1, NULL, 0, NULL, 0, @@ -49,6 +139,8 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) CHECK(skel->bss->fmod_ret_result != 1, "modify_return", "fmod_ret failed\n"); + if (md) + multi_detach(md); cleanup: modify_return__destroy(skel); } @@ -56,11 +148,27 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) /* TODO: conflict with get_func_ip_test */ void serial_test_modify_return(void) { + struct multi_data data = {}; + + run_test(0 /* input_retval */, + 1 /* want_side_effect */, + 4 /* want_ret */, + NULL /* no multi func test */); + run_test(-EINVAL /* input_retval */, + 0 /* want_side_effect */, + -EINVAL /* want_ret */, + NULL /* no multi func test */); + + if (!ASSERT_OK(multi_btf_ids(&data), "multi_attach")) + return; + run_test(0 /* input_retval */, 1 /* want_side_effect */, - 4 /* want_ret */); + 4 /* want_ret */, + &data); run_test(-EINVAL /* input_retval */, 0 /* want_side_effect */, - -EINVAL /* want_ret */); + -EINVAL /* want_ret */, + &data); } diff --git a/tools/testing/selftests/bpf/progs/multi_modify_return.c b/tools/testing/selftests/bpf/progs/multi_modify_return.c new file mode 100644 index 000000000000..34754e438c96 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_modify_return.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + return 0; +} + +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + return 0; +} From patchwork Thu Nov 18 11:24:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12626709 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B6FEC433F5 for ; Thu, 18 Nov 2021 11:30:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 66D8661B73 for ; Thu, 18 Nov 2021 11:30:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344160AbhKRLc4 (ORCPT ); Thu, 18 Nov 2021 06:32:56 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27538 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1343920AbhKRLaz (ORCPT ); Thu, 18 Nov 2021 06:30:55 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1637234875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lXDO1Q0PT1LV+IkN9FZ4WM7egV17pWLfXKegiEUMZIc=; b=UC9jJV4le8Wk3oQcsm/r2HqLrvOeXhkSb/yctz4O0VbKCM81OYNi3Zg4zrWI/RCFm11SPQ neoPzab+/bZecX3C8lYO34mvo2YlESIF1e6cHNE2qZGoLftIQ8WDdUNtiM+2FMDOvoaKBr NldBTg2m18ejduwr/HMQPb3tRLfIbJQ= Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-496-MWYNFFnAN-6IA39h4yqBIw-1; Thu, 18 Nov 2021 06:27:54 -0500 X-MC-Unique: MWYNFFnAN-6IA39h4yqBIw-1 Received: by mail-ed1-f71.google.com with SMTP id a3-20020a05640213c300b003e7d12bb925so5025110edx.9 for ; Thu, 18 Nov 2021 03:27:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lXDO1Q0PT1LV+IkN9FZ4WM7egV17pWLfXKegiEUMZIc=; b=sdFVkYjUJP+0hrHHDXoU4Z5QaUZXfzsY8Aiopk/4SW7WI72qtkP9aakB8ouz3JxPeJ fX7We7UeCJLbpS1iedz/Lkndqgly1rbSqhBI/EYp/3xeDRDhb7AEyjrJD+0d0XqQwDD8 zan9UBHKgveDfv7FbujpLKv817MstRjIBqjdU3RJxEYyh2PeRR7U/cCYOsg3tY0+M7Mu CHNg/X/fi/PYoVZm2yeVJhioNI46f8IvFguhdm6e67ZaHAt2b3/KjpeEmLWec66UnPfE DFxDvwY+n0GWdbBoCYU5bi6PaR0fw0sbIxWVO66OqJpInerRuzoYwlyxfxmPX5apvrMl LfvQ== X-Gm-Message-State: AOAM530UGAlbMTyxuMNFTTr0rFiraCq7LAuM0zlsO3luJSdCtOTXOSuR CElD2wBK6c3p1VtcWNOIE9asmHLprTD8HSckSuZda2xkF8txVhAicwK7ypUxztViH04KLyi1UJB oV4OvmGXxjMgceXvr X-Received: by 2002:aa7:c3c8:: with SMTP id l8mr10099504edr.278.1637234872726; Thu, 18 Nov 2021 03:27:52 -0800 (PST) X-Google-Smtp-Source: ABdhPJxxp5GicqeoqaM5YlhbpBraXwMCTAKIvVszs2h9J1eZ9tAsFbu6eJl3k82B7jE9zwk0fcFhQQ== X-Received: by 2002:aa7:c3c8:: with SMTP id l8mr10099379edr.278.1637234872055; Thu, 18 Nov 2021 03:27:52 -0800 (PST) Received: from krava.redhat.com (nat-pool-brq-u.redhat.com. [213.175.37.12]) by smtp.gmail.com with ESMTPSA id em21sm1172772ejc.103.2021.11.18.03.27.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Nov 2021 03:27:51 -0800 (PST) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh Subject: [PATCH bpf-next 29/29] selftests/bpf: Add attach multi func test Date: Thu, 18 Nov 2021 12:24:55 +0100 Message-Id: <20211118112455.475349-30-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211118112455.475349-1-jolsa@kernel.org> References: <20211118112455.475349-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding test code to check on trampolines spliting. The tests attached various bpf_fetry_* functions in a way so there's always non trivial IDs intersection, that leads to trampoline splitting in kenrel code. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/prog_tests/multi_attach_test.c | 176 ++++++++++++++++++ .../selftests/bpf/progs/multi_attach.c | 105 +++++++++++ 3 files changed, 284 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_attach_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_attach.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 48970e983250..095d966a747a 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -324,7 +324,8 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ multi_fentry_test.skel.h multi_fexit_test.skel.h \ - multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h + multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h \ + multi_attach_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ringbuf.c atomics.c trace_printk.c trace_vprintk.c @@ -340,6 +341,7 @@ multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o multi_mixed_test.skel.h-deps := multi_mixed.o multi_check.o +multi_attach_test.skel.h-deps := multi_attach.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_attach_test.c b/tools/testing/selftests/bpf/prog_tests/multi_attach_test.c new file mode 100644 index 000000000000..c183941215a6 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_attach_test.c @@ -0,0 +1,176 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "multi_attach_test.skel.h" +#include + +static __u32 btf_ids[8]; + +static int load_btf_ids(void) +{ + __u32 i, nr_types, cnt; + struct btf *btf; + + btf = btf__load_vmlinux_btf(); + if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf")) + return -1; + + nr_types = btf__get_nr_types(btf); + + for (i = 1, cnt = 0; i <= nr_types && cnt < 8; i++) { + const struct btf_type *t = btf__type_by_id(btf, i); + const char *name; + + if (!btf_is_func(t)) + continue; + + name = btf__name_by_offset(btf, t->name_off); + if (!name) + continue; + if (strncmp(name, "bpf_fentry_test", sizeof("bpf_fentry_test") - 1)) + continue; + + btf_ids[cnt] = i; + cnt++; + } + + btf__free(btf); + return ASSERT_EQ(cnt, 8, "bpf_fentry_test_cnt") ? 0 : -1; +} + +static int link_prog_from_cnt(const struct bpf_program *prog, int from, int cnt) +{ + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts); + enum bpf_attach_type attach_type; + int prog_fd, link_fd; + + opts.multi.btf_ids = btf_ids + (from - 1); + opts.multi.btf_ids_cnt = cnt; + + prog_fd = bpf_program__fd(prog); + if (!ASSERT_GE(prog_fd, 0, "link_from_to_prog_fd")) + return -1; + attach_type = bpf_program__get_expected_attach_type(prog); + link_fd = bpf_link_create(prog_fd, 0, attach_type, &opts); + if (!ASSERT_GE(link_fd, 0, "link_from_to_link_fd")) + return -1; + return link_fd; +} + +static int prog_from_cnt(const struct bpf_program *prog, int *from, int *cnt) +{ + const char *sec; + int err, to; + + sec = bpf_program__section_name(prog); + sec = strchr(sec, '/'); + if (!sec) + return -1; + sec++; + err = sscanf(sec, "bpf_fentry_test%d-%d", from, &to); + if (err != 2) + return -1; + *cnt = to - *from + 1; + return 0; +} + +static int link_test(const struct bpf_program *prog1, + const struct bpf_program *prog2, + __u64 *test_result1, __u64 *test_result2, + bool do_close, int link_fd[2]) +{ + int from1, cnt1, from2, cnt2, err; + __u32 duration = 0, retval; + + if (!ASSERT_OK(prog_from_cnt(prog1, &from1, &cnt1), "prog_from_cnt__prog1")) + return -1; + + if (!ASSERT_OK(prog_from_cnt(prog2, &from2, &cnt2), "prog_from_cnt__prog2")) + return -1; + + link_fd[0] = link_prog_from_cnt(prog1, from1, cnt1); + if (link_fd[0] < 0) + return -1; + + link_fd[1] = link_prog_from_cnt(prog2, from2, cnt2); + if (link_fd[1] < 0) + return -1; + + *test_result1 = 0; + *test_result2 = 0; + + err = bpf_prog_test_run(bpf_program__fd(prog1), 1, NULL, 0, + NULL, NULL, &retval, &duration); + + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + ASSERT_EQ(*test_result1, cnt1, "test_result"); + ASSERT_EQ(*test_result2, cnt2, "test_result"); + + if (do_close) { + close(link_fd[0]); + close(link_fd[1]); + } + return err; +} + +void test_multi_attach_test(void) +{ + struct bpf_link *link7 = NULL, *link8 = NULL; + int link_fd[6] = { -1 }, i, err; + struct multi_attach_test *skel; + __u32 duration = 0, retval; + + for (i = 0; i < 6; i++) + link_fd[i] = -1; + + skel = multi_attach_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "multi_attach__load")) + return; + + if (!ASSERT_OK(load_btf_ids(), "load_btf_ids")) + goto cleanup; + +#define LINK_TEST(__prog1, __prog2, __close) \ + err = link_test(skel->progs.test ## __prog1, \ + skel->progs.test ## __prog2, \ + &skel->bss->test_result ## __prog1, \ + &skel->bss->test_result ## __prog2, \ + __close, link_fd + __prog1 - 1); \ + if (err) \ + goto cleanup; + + LINK_TEST(1, 2, true); + LINK_TEST(3, 4, true); + LINK_TEST(1, 3, true); + LINK_TEST(2, 4, true); + + LINK_TEST(1, 2, false); + LINK_TEST(3, 4, false); + LINK_TEST(5, 6, false); + +#undef LINK_TEST + + link7 = bpf_program__attach(skel->progs.test7); + if (!ASSERT_OK_PTR(link7, "multi_attach_check__test1_attach")) + goto cleanup; + + link8 = bpf_program__attach(skel->progs.test8); + if (!ASSERT_OK_PTR(link7, "multi_attach_check__test2_attach")) + goto cleanup; + + err = bpf_prog_test_run(bpf_program__fd(skel->progs.test7), 1, NULL, 0, + NULL, NULL, &retval, &duration); + + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + ASSERT_EQ(skel->bss->test_result7, 1, "test_result7"); + ASSERT_EQ(skel->bss->test_result8, 1, "test_result8"); + +cleanup: + bpf_link__destroy(link8); + bpf_link__destroy(link7); + for (i = 0; i < 6; i++) + close(link_fd[i]); + multi_attach_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_attach.c b/tools/testing/selftests/bpf/progs/multi_attach.c new file mode 100644 index 000000000000..d403f5f1f27e --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_attach.c @@ -0,0 +1,105 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, __u64 *test_result); + +__u64 test_result1 = 0; + +SEC("fentry.multi/bpf_fentry_test1-5") +int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test_result1); + return 0; +} + +__u64 test_result2 = 0; + +SEC("fentry.multi/bpf_fentry_test4-8") +int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test_result2); + return 0; +} + +__u64 test_result3 = 0; + +SEC("fexit.multi/bpf_fentry_test1-5") +int BPF_PROG(test3, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + __u64 arg_result = 0, ret_result = 0; + + multi_arg_check(ctx, &arg_result); + multi_ret_check(ctx, &ret_result); + + if (arg_result && ret_result) + test_result3 += 1; + return 0; +} + +__u64 test_result4 = 0; + +SEC("fexit.multi/bpf_fentry_test4-8") +int BPF_PROG(test4, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + __u64 arg_result = 0, ret_result = 0; + + multi_arg_check(ctx, &arg_result); + multi_ret_check(ctx, &ret_result); + + if (arg_result && ret_result) + test_result4 += 1; + return 0; +} + +__u64 test_result5 = 0; + +SEC("fentry.multi/bpf_fentry_test1-8") +int BPF_PROG(test5, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test_result5); + return 0; +} + +__u64 test_result6 = 0; + +SEC("fexit.multi/bpf_fentry_test1-8") +int BPF_PROG(test6, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + __u64 arg_result = 0, ret_result = 0; + + multi_arg_check(ctx, &arg_result); + multi_ret_check(ctx, &ret_result); + + if (arg_result && ret_result) + test_result6 += 1; + return 0; +} + +__u64 test_result7 = 0; + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(test7, int a) +{ + multi_arg_check(ctx, &test_result7); + return 0; +} + +__u64 test_result8 = 0; + +SEC("fexit/bpf_fentry_test2") +int BPF_PROG(test8, int a, __u64 b, int ret) +{ + __u64 arg_result = 0, ret_result = 0; + + multi_arg_check(ctx, &arg_result); + multi_ret_check(ctx, &ret_result); + + if (arg_result && ret_result) + test_result8 += 1; + return 0; +}