From patchwork Thu Aug 26 19:38:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460541 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A52CC4320E for ; Thu, 26 Aug 2021 19:39:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E2C9E6103E for ; Thu, 26 Aug 2021 19:39:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243488AbhHZTkU (ORCPT ); Thu, 26 Aug 2021 15:40:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:23693 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243485AbhHZTkU (ORCPT ); Thu, 26 Aug 2021 15:40:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8XYUgR9kHcOYFdSNrWNV6bYolumP9ymhRdSASBvoKM=; b=J4TD16JPNXes7ncQkhVlXA103mslDdrHwNveGgWlQL1CffLKe1ANTWj93MbmteKA9FHpUD tKW4tJaMbpdd2zIK0ads27ujFJPJBSvNm8STzIHwpJEV3F6yB6LcKXyxfE26yEsiuJIqyl 9wo7C8LpYJGu3CiebMOH8/NarcAfrlw= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-417-o3gdgBOSM_G5o2h1D6Rxsw-1; Thu, 26 Aug 2021 15:39:30 -0400 X-MC-Unique: o3gdgBOSM_G5o2h1D6Rxsw-1 Received: by mail-wm1-f69.google.com with SMTP id j135-20020a1c238d000000b002e87aa95b5aso3954000wmj.4 for ; Thu, 26 Aug 2021 12:39:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=t8XYUgR9kHcOYFdSNrWNV6bYolumP9ymhRdSASBvoKM=; b=oaiz2EqpzAoAyh7loKQaw0HsMI+ecg9xz5KFhZDi5uthec9VByvmZHTuK7Hj9z8clT VOewAfceixNn5+ZCartrena7V8wS0+hG1c8h5bdC5gWNOsyTXam4hhS687Cqrz/q87xQ P7a84WR32hbJogVrpZ3j7XvFucNSLvt1YyDwc1gR3VPudBJBsrrcHQM7wFfzGuO/v6IL b8gRk04Cl87Z5rXS2gfdiPF10gVtgMpP9mM/rRsQceaC1f6Z9BNhbkLUsQh5XaQAd/3R 3ul8o7UdhJXIKCoV9kClXJAtglL3nbztSZWTHjwtAcmNCyPwlhGLpJDLS87yiJvoc1Kd hkUQ== X-Gm-Message-State: AOAM5305ulveb487j/W+D2O0Ix34eaAXsANMTeuNg3WF47kdrZKh1GWT ig9e5KmxY5ZK57G1oque9libfNpCiXe1mPCtUsitnlS2iqlHByHwG/1g48e9cTTYtgOJI1myWo6 RJJzq8FL8NNR2 X-Received: by 2002:a5d:4142:: with SMTP id c2mr6091470wrq.340.1630006769759; Thu, 26 Aug 2021 12:39:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxBL1FafyptrQyoXeyUcyn0ennz0cwN+Ah0kJWNT5IknqO/e8yH2B5sBSVQLTvUBYm9pq1F/Q== X-Received: by 2002:a5d:4142:: with SMTP id c2mr6091460wrq.340.1630006769637; Thu, 26 Aug 2021 12:39:29 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id r1sm8012572wmn.46.2021.08.26.12.39.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:39:29 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 01/27] x86/ftrace: Remove extra orig rax move Date: Thu, 26 Aug 2021 21:38:56 +0200 Message-Id: <20210826193922.66204-2-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net There's identical move 2 lines earlier. Signed-off-by: Jiri Olsa --- arch/x86/kernel/ftrace_64.S | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 7c273846c687..a8eb084a7a9a 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -251,7 +251,6 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) * If ORIG_RAX is anything but zero, make this a call to that. * See arch_ftrace_set_direct_caller(). */ - movq ORIG_RAX(%rsp), %rax testq %rax, %rax SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) jnz 1f From patchwork Thu Aug 26 19:38:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460543 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18A8DC432BE for ; Thu, 26 Aug 2021 19:39:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E530461052 for ; Thu, 26 Aug 2021 19:39:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243464AbhHZTk0 (ORCPT ); Thu, 26 Aug 2021 15:40:26 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:45680 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243407AbhHZTk0 (ORCPT ); Thu, 26 Aug 2021 15:40:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006778; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eqAj4ZrmrtKDThZEc8WrAKFFOfUsJKx4HE/+LaO0ups=; b=Aqv01lJYjYD8nAuWAlTu4/fQk5EYYnyoRW2U3cBdRoPozjUCJ6H9tvdwQ2DnaI6f/M7aq2 /NbrS0Hy8uBgYpuj8lPnG1vzD/xiSpk3R/Pmd+92Gcr/MiNXf4l7CzH/fq26ZuN54bPacb uWXiBzIkRpqPXZRJVKMat6wLY+ygHC4= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-581-tyq5BFi2ONOjT_0mX_RgOw-1; Thu, 26 Aug 2021 15:39:37 -0400 X-MC-Unique: tyq5BFi2ONOjT_0mX_RgOw-1 Received: by mail-wm1-f71.google.com with SMTP id k5-20020a7bc3050000b02901e081f69d80so1265214wmj.8 for ; Thu, 26 Aug 2021 12:39:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eqAj4ZrmrtKDThZEc8WrAKFFOfUsJKx4HE/+LaO0ups=; b=TWOimxMPNJCawK0+tbcQHSt2/aRx/YIuyh1tZRbS38i3RtXhhbDLtTyNSAbHpBw6hz nWvRl/SarE00QGcCmtzzUvxd3oMe8lAG+jo0vPffWNQSZc4OmruOAjUS5rjCXyzJxjM0 LCfAXVcTB3p2WRQhoXYVKyYqoLlOWDVkx9LDXmMgzVu2mpWWx4bJERcpV4O3kwJ1XC1G DJ78mI3IB02CDsdnEhmWsUmYcXDCCtbJbMHDZTHxaAbL7q1EKRw6DY+z7/nBgMdXqVpG DQn0zLPrrC9sVtg7L0etr/LGCqAL/4oqdy0leVaoz4xGkvvmrpqZ0KOrQbyu0KYIQ364 vuCQ== X-Gm-Message-State: AOAM530LYVmalrfzmpzbNczVaWqnPL+wNC//vVk4shmBwgrx+ebgLyAc mPCXOQp887RUE02txkOjbWfyqStvaql9IU2+9kmRHOjjix7eFhu/uvgFwWCLJFaes0aon/xkI+O YGxClR9d9LfBw X-Received: by 2002:a5d:664b:: with SMTP id f11mr5900058wrw.39.1630006776008; Thu, 26 Aug 2021 12:39:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJynG5QoqDJvteuV2aFyjTpD5oRelW0Ocw2434p+kHB3WTHEdIOLbOwZ+xN6eQOLJnZF/BgGeQ== X-Received: by 2002:a5d:664b:: with SMTP id f11mr5900047wrw.39.1630006775853; Thu, 26 Aug 2021 12:39:35 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id o5sm3981177wrw.17.2021.08.26.12.39.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:39:35 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 02/27] x86/ftrace: Remove fault protection code in prepare_ftrace_return Date: Thu, 26 Aug 2021 21:38:57 +0200 Message-Id: <20210826193922.66204-3-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: "Steven Rostedt (VMware)" Removing the fault protection code when writing return_hooker to stack. As Steven noted: > That protection was there from the beginning due to being "paranoid", > considering ftrace was bricking network cards. But that protection > would not have even protected against that. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Jiri Olsa --- arch/x86/kernel/ftrace.c | 38 +++----------------------------------- 1 file changed, 3 insertions(+), 35 deletions(-) diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 1b3ce3b4a2a2..c555624da989 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -625,12 +625,10 @@ int ftrace_disable_ftrace_graph_caller(void) * Hook the return address and push it in the stack of return addrs * in current thread info. */ -void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, +void prepare_ftrace_return(unsigned long ip, unsigned long *parent, unsigned long frame_pointer) { unsigned long return_hooker = (unsigned long)&return_to_handler; - unsigned long old; - int faulted; /* * When resuming from suspend-to-ram, this function can be indirectly @@ -650,37 +648,7 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, if (unlikely(atomic_read(¤t->tracing_graph_pause))) return; - /* - * Protect against fault, even if it shouldn't - * happen. This tool is too much intrusive to - * ignore such a protection. - */ - asm volatile( - "1: " _ASM_MOV " (%[parent]), %[old]\n" - "2: " _ASM_MOV " %[return_hooker], (%[parent])\n" - " movl $0, %[faulted]\n" - "3:\n" - - ".section .fixup, \"ax\"\n" - "4: movl $1, %[faulted]\n" - " jmp 3b\n" - ".previous\n" - - _ASM_EXTABLE(1b, 4b) - _ASM_EXTABLE(2b, 4b) - - : [old] "=&r" (old), [faulted] "=r" (faulted) - : [parent] "r" (parent), [return_hooker] "r" (return_hooker) - : "memory" - ); - - if (unlikely(faulted)) { - ftrace_graph_stop(); - WARN_ON(1); - return; - } - - if (function_graph_enter(old, self_addr, frame_pointer, parent)) - *parent = old; + if (!function_graph_enter(*parent, ip, frame_pointer, parent)) + *parent = return_hooker; } #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ From patchwork Thu Aug 26 19:38:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460545 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63E5AC432BE for ; Thu, 26 Aug 2021 19:39:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4BF606108D for ; Thu, 26 Aug 2021 19:39:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243500AbhHZTkd (ORCPT ); Thu, 26 Aug 2021 15:40:33 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:23064 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243491AbhHZTkc (ORCPT ); Thu, 26 Aug 2021 15:40:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006784; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sLqTTRxVL8xfk55K+9eSTs+Utqi1uRVMWYGVEA7msME=; b=I5vHl+TxzayuRZERnEebo6t6iPNw5/fFuVJTlHM2sDPMMHOrUKKsbFdKNlSSI+HmzfTWmG xZwkSTxsg4evWhiBFm3lqcDHghzLHn+MpTDuoAYHbyAz+/iwzvNiAEfcJWopnTzmYijxZN wpxrG7dskX/5jhuI/3LU9vGkyFI8AO8= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-152-UuyF8gtbOTmsF0ObjJAPKg-1; Thu, 26 Aug 2021 15:39:43 -0400 X-MC-Unique: UuyF8gtbOTmsF0ObjJAPKg-1 Received: by mail-wm1-f69.google.com with SMTP id b207-20020a1c80d8000000b002ea321114f7so528258wmd.7 for ; Thu, 26 Aug 2021 12:39:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sLqTTRxVL8xfk55K+9eSTs+Utqi1uRVMWYGVEA7msME=; b=hswSZ1aqEuyKFRp0oi/Ecit5guAVRC56AtVnCaF+jeRKpuFXoSO9xRfMyrCEurYvjs Qe7J8FEmGfbugszeI6dtNIgrQMNE29oj6Bd9pnJXrKrWhtomVxQVnTP4TAaWJRPnhdWB swVi/XTvrYdyZvofsokrNzXBEE09L0/assbsv3z0wYmYvq1D5BgsxB3WERJfU0C02h4B 4648Jz6kSAZOwk38xJyGjoRBBItgG5qF2728gnyBo4arEY1qnGrfwpSKkICvsfivoNKA DfCOKet+O2PIB9EqwD4TrpSwufvQ96P0kY37sUGrX8D3fCXJmP158QTv7MSzYczt5/x+ Egug== X-Gm-Message-State: AOAM533H2wJSpsrV58OzHc/BsdBHqPnSjCdmzujMqSVCgdVORR+jsaM7 77xJGeX0gFMn/YEF9RZnpqmCU3cgoMz42/LWWFYKWmwP4fPJyZjolNFHkEOFfOBQOqvfBytMMnq cbrnZn//4tP1F X-Received: by 2002:a05:600c:1ca7:: with SMTP id k39mr5210875wms.162.1630006782215; Thu, 26 Aug 2021 12:39:42 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzVCSYcOLJeoO9HAhJSY06pVgY7DESSfJAk8Q4dTP9qnnmFI+O8vT7InR93U6Zg+d5eIZNF4A== X-Received: by 2002:a05:600c:1ca7:: with SMTP id k39mr5210858wms.162.1630006782038; Thu, 26 Aug 2021 12:39:42 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id g76sm8944197wme.16.2021.08.26.12.39.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:39:41 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 03/27] x86/ftrace: Make function graph use ftrace directly Date: Thu, 26 Aug 2021 21:38:58 +0200 Message-Id: <20210826193922.66204-4-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net From: "Steven Rostedt (VMware)" We don't need special hook for graph tracer entry point, but instead we can use graph_ops::func function to install the return_hooker. This moves the graph tracing setup _before_ the direct trampoline prepares the stack, so the return_hooker will be called when the direct trampoline is finished. This simplifies the code, because we don't need to take into account the direct trampoline setup when preparing the graph tracer hooker and we can allow function graph tracer on entries registered with direct trampoline. Signed-off-by: Steven Rostedt (VMware) Signed-off-by: Jiri Olsa --- arch/x86/include/asm/ftrace.h | 9 +++++++-- arch/x86/kernel/ftrace.c | 37 ++++++++++++++++++++++++++++++++--- arch/x86/kernel/ftrace_64.S | 29 +-------------------------- include/linux/ftrace.h | 6 ++++++ kernel/trace/fgraph.c | 6 ++++-- 5 files changed, 52 insertions(+), 35 deletions(-) diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index 9f3130f40807..024d9797646e 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -57,6 +57,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs) #define ftrace_instruction_pointer_set(fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0) + +struct ftrace_ops; +#define ftrace_graph_func ftrace_graph_func +void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, + struct ftrace_ops *op, struct ftrace_regs *fregs); +#else +#define FTRACE_GRAPH_TRAMP_ADDR FTRACE_GRAPH_ADDR #endif #ifdef CONFIG_DYNAMIC_FTRACE @@ -65,8 +72,6 @@ struct dyn_arch_ftrace { /* No extra data needed for x86 */ }; -#define FTRACE_GRAPH_TRAMP_ADDR FTRACE_GRAPH_ADDR - #endif /* CONFIG_DYNAMIC_FTRACE */ #endif /* __ASSEMBLY__ */ #endif /* CONFIG_FUNCTION_TRACER */ diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index c555624da989..804fcc6ef2c7 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -527,7 +527,7 @@ static void *addr_from_call(void *ptr) return ptr + CALL_INSN_SIZE + call.disp; } -void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent, +void prepare_ftrace_return(unsigned long ip, unsigned long *parent, unsigned long frame_pointer); /* @@ -541,7 +541,8 @@ static void *static_tramp_func(struct ftrace_ops *ops, struct dyn_ftrace *rec) void *ptr; if (ops && ops->trampoline) { -#ifdef CONFIG_FUNCTION_GRAPH_TRACER +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS) && \ + defined(CONFIG_FUNCTION_GRAPH_TRACER) /* * We only know about function graph tracer setting as static * trampoline. @@ -589,8 +590,9 @@ void arch_ftrace_trampoline_free(struct ftrace_ops *ops) #ifdef CONFIG_FUNCTION_GRAPH_TRACER #ifdef CONFIG_DYNAMIC_FTRACE -extern void ftrace_graph_call(void); +#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +extern void ftrace_graph_call(void); static const char *ftrace_jmp_replace(unsigned long ip, unsigned long addr) { return text_gen_insn(JMP32_INSN_OPCODE, (void *)ip, (void *)addr); @@ -618,7 +620,17 @@ int ftrace_disable_ftrace_graph_caller(void) return ftrace_mod_jmp(ip, &ftrace_stub); } +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ +int ftrace_enable_ftrace_graph_caller(void) +{ + return 0; +} +int ftrace_disable_ftrace_graph_caller(void) +{ + return 0; +} +#endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ #endif /* !CONFIG_DYNAMIC_FTRACE */ /* @@ -629,6 +641,7 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent, unsigned long frame_pointer) { unsigned long return_hooker = (unsigned long)&return_to_handler; + int bit; /* * When resuming from suspend-to-ram, this function can be indirectly @@ -648,7 +661,25 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent, if (unlikely(atomic_read(¤t->tracing_graph_pause))) return; + bit = ftrace_test_recursion_trylock(ip, *parent); + if (bit < 0) + return; + if (!function_graph_enter(*parent, ip, frame_pointer, parent)) *parent = return_hooker; + + ftrace_test_recursion_unlock(bit); +} + +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, + struct ftrace_ops *op, struct ftrace_regs *fregs) +{ + struct pt_regs *regs = &fregs->regs; + unsigned long *stack = (unsigned long *)kernel_stack_pointer(regs); + + prepare_ftrace_return(ip, (unsigned long *)stack, 0); } +#endif + #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index a8eb084a7a9a..7a879901f103 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -174,11 +174,6 @@ SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) SYM_FUNC_END(ftrace_caller); SYM_FUNC_START(ftrace_epilogue) -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) - jmp ftrace_stub -#endif - /* * This is weak to keep gas from relaxing the jumps. * It is also used to copy the retq for trampolines. @@ -288,15 +283,6 @@ SYM_FUNC_START(__fentry__) cmpq $ftrace_stub, ftrace_trace_function jnz trace -fgraph_trace: -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - cmpq $ftrace_stub, ftrace_graph_return - jnz ftrace_graph_caller - - cmpq $ftrace_graph_entry_stub, ftrace_graph_entry - jnz ftrace_graph_caller -#endif - SYM_INNER_LABEL(ftrace_stub, SYM_L_GLOBAL) retq @@ -314,25 +300,12 @@ trace: CALL_NOSPEC r8 restore_mcount_regs - jmp fgraph_trace + jmp ftrace_stub SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) #endif /* CONFIG_DYNAMIC_FTRACE */ #ifdef CONFIG_FUNCTION_GRAPH_TRACER -SYM_FUNC_START(ftrace_graph_caller) - /* Saves rbp into %rdx and fills first parameter */ - save_mcount_regs - - leaq MCOUNT_REG_SIZE+8(%rsp), %rsi - movq $0, %rdx /* No framepointers needed */ - call prepare_ftrace_return - - restore_mcount_regs - - retq -SYM_FUNC_END(ftrace_graph_caller) - SYM_FUNC_START(return_to_handler) subq $24, %rsp diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index a69f363b61bf..9b218e59a608 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -614,6 +614,12 @@ void ftrace_modify_all_code(int command); extern void ftrace_graph_caller(void); extern int ftrace_enable_ftrace_graph_caller(void); extern int ftrace_disable_ftrace_graph_caller(void); +#ifndef ftrace_graph_func +#define ftrace_graph_func ftrace_stub +#define FTRACE_OPS_GRAPH_STUB FTRACE_OPS_FL_STUB +#else +#define FTRACE_OPS_GRAPH_STUB 0 +#endif #else static inline int ftrace_enable_ftrace_graph_caller(void) { return 0; } static inline int ftrace_disable_ftrace_graph_caller(void) { return 0; } diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index b8a0d1d564fb..22061d38fc00 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -115,6 +115,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, { struct ftrace_graph_ent trace; +#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS /* * Skip graph tracing if the return location is served by direct trampoline, * since call sequence and return addresses are unpredictable anyway. @@ -124,6 +125,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, if (ftrace_direct_func_count && ftrace_find_rec_direct(ret - MCOUNT_INSN_SIZE)) return -EBUSY; +#endif trace.func = func; trace.depth = ++current->curr_ret_depth; @@ -333,10 +335,10 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, #endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ static struct ftrace_ops graph_ops = { - .func = ftrace_stub, + .func = ftrace_graph_func, .flags = FTRACE_OPS_FL_INITIALIZED | FTRACE_OPS_FL_PID | - FTRACE_OPS_FL_STUB, + FTRACE_OPS_GRAPH_STUB, #ifdef FTRACE_GRAPH_TRAMP_ADDR .trampoline = FTRACE_GRAPH_TRAMP_ADDR, /* trampoline_size is only needed for dynamically allocated tramps */ From patchwork Thu Aug 26 19:38:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460547 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C06C9C4320A for ; Thu, 26 Aug 2021 19:39:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A1DFF6108F for ; Thu, 26 Aug 2021 19:39:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243514AbhHZTkl (ORCPT ); Thu, 26 Aug 2021 15:40:41 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:30425 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243507AbhHZTkk (ORCPT ); Thu, 26 Aug 2021 15:40:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cg/efFypXdgTTjnQMuhwdn9EcUQgpWsXBxL2SLIRi+8=; b=J+Mc6Uu8pc7u6/+vyZSxsyxPQLkiqP6vO9fT/OZ2rdogni4XGcNj30JQLS5kcyHsinTwhi ibEA0r5WYfXjTvoCcA0pQV3ETTdmxYcq9rfCornFA/YtgXoA2ld3H2007+gu7z1tXSl2+C +/mA9Xgjkpgq+V33MOfAt2INdNlxC3A= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-571-BctiasJlPpOFFMnSWXKJbA-1; Thu, 26 Aug 2021 15:39:49 -0400 X-MC-Unique: BctiasJlPpOFFMnSWXKJbA-1 Received: by mail-wm1-f69.google.com with SMTP id v2-20020a7bcb420000b02902e6b108fcf1so4748562wmj.8 for ; Thu, 26 Aug 2021 12:39:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Cg/efFypXdgTTjnQMuhwdn9EcUQgpWsXBxL2SLIRi+8=; b=c9zjP8hgFB1QZCTJqPXDuDlwmrAdtriVdsmRrEOI5UsGa4SfRGdRUdOmu/BOSbV7D+ 5eMfdmiiZ4jUQ0PNgPNdd4QjuGbXVTPss3dhie7KJVq0S+mIbK6uSnYiMUvcwbPOWJfh yGGAMwMC+8ZvixcsOnG6auItRTwXp/cEZg2M+dRKOklCCGexUnBzBKEZhAPVcmpNnCJz BO//Sc7h/nVT6tP9X88RDkg9/5+AAh06b0+scGvsQATxrPIScA8Gem8oakvMRBrbp1Lr heAQpIfW00snqyHBYT41iKSnU8lysz8IMWESEP9do1js3QsXQREaWunvVs54hfPmT6mY N0FA== X-Gm-Message-State: AOAM531t/+p5zs7BIvlJZAczvMzqmtNb2GJ+tau4LJicfk71KcByAI+1 dUnjxXprklI75wORDXWNVhrnnL6mRGX6X2afjTj/WQrgFCP0nwHGtC+wzOKH3j9gxZS4B88rs+Z 6ycH5aO5+W3al X-Received: by 2002:a05:600c:1c9c:: with SMTP id k28mr16247263wms.148.1630006788444; Thu, 26 Aug 2021 12:39:48 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzErSIb9da71+bd4eM+gpeR8ymg1W/1JdN8FsOYvZ/k0CPgNeUBQMNrL2cqD8lXC1MrP68OqQ== X-Received: by 2002:a05:600c:1c9c:: with SMTP id k28mr16247245wms.148.1630006788257; Thu, 26 Aug 2021 12:39:48 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id i14sm3327180wmq.40.2021.08.26.12.39.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:39:48 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 04/27] tracing: Add trampoline/graph selftest Date: Thu, 26 Aug 2021 21:38:59 +0200 Message-Id: <20210826193922.66204-5-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for checking that direct trampoline can co-exist together with graph tracer on same function. This is supported for CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS config option, which is defined only for x86_64 for now. Signed-off-by: Jiri Olsa --- kernel/trace/trace_selftest.c | 49 ++++++++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c index adf7ef194005..f8e55b949cdd 100644 --- a/kernel/trace/trace_selftest.c +++ b/kernel/trace/trace_selftest.c @@ -750,6 +750,8 @@ static struct fgraph_ops fgraph_ops __initdata = { .retfunc = &trace_graph_return, }; +noinline __noclone static void trace_direct_tramp(void) { } + /* * Pretty much the same than for the function tracer from which the selftest * has been borrowed. @@ -760,6 +762,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, { int ret; unsigned long count; + char *func_name __maybe_unused; #ifdef CONFIG_DYNAMIC_FTRACE if (ftrace_filter_param) { @@ -808,8 +811,52 @@ trace_selftest_startup_function_graph(struct tracer *trace, goto out; } - /* Don't test dynamic tracing, the function tracer already did */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS + tracing_reset_online_cpus(&tr->array_buffer); + set_graph_array(tr); + /* + * Some archs *cough*PowerPC*cough* add characters to the + * start of the function names. We simply put a '*' to + * accommodate them. + */ + func_name = "*" __stringify(DYN_FTRACE_TEST_NAME); + ftrace_set_global_filter(func_name, strlen(func_name), 1); + + /* + * Register direct function together with graph tracer + * and make sure we get graph trace. + */ + ret = register_ftrace_direct((unsigned long) DYN_FTRACE_TEST_NAME, + (unsigned long) trace_direct_tramp); + if (ret) + goto out; + + ret = register_ftrace_graph(&fgraph_ops); + if (ret) { + warn_failed_init_tracer(trace, ret); + goto out; + } + + DYN_FTRACE_TEST_NAME(); + + count = 0; + + tracing_stop(); + /* check the trace buffer */ + ret = trace_test_buffer(&tr->array_buffer, &count); + + unregister_ftrace_graph(&fgraph_ops); + + tracing_start(); + + if (!ret && !count) { + ret = -1; + goto out; + } +#endif + + /* Don't test dynamic tracing, the function tracer already did */ out: /* Stop it if we failed */ if (ret) From patchwork Thu Aug 26 19:39:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460549 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B53B2C432BE for ; Thu, 26 Aug 2021 19:40:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9CA236108F for ; Thu, 26 Aug 2021 19:40:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243520AbhHZTks (ORCPT ); Thu, 26 Aug 2021 15:40:48 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:22030 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243527AbhHZTkp (ORCPT ); Thu, 26 Aug 2021 15:40:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006797; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uK5Gw3luJndbFn1Fa9N5OtQXbRTUWbF3Z2jHUj38PoM=; b=JgdpZARtUwyw7ldeEnZIwFaeGUpSAl5jSTsPRlj2NFResSU6bM1Y+ko82atoMrNG0yyQ7k y6xMQlfWsYnq+ZO9AO8kUVNGPBWvUXZ6jkvIqoeO64AYJhrm18cc4+AZxeL4ulQwdCTHr0 g1ypXfq8CsfRPUIyHMFbri5dDk+SGOY= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-284-Czpd3H1hOU2kB5QIBn4TlA-1; Thu, 26 Aug 2021 15:39:55 -0400 X-MC-Unique: Czpd3H1hOU2kB5QIBn4TlA-1 Received: by mail-wm1-f69.google.com with SMTP id w25-20020a1cf6190000b0290252505ddd56so1273969wmc.3 for ; Thu, 26 Aug 2021 12:39:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uK5Gw3luJndbFn1Fa9N5OtQXbRTUWbF3Z2jHUj38PoM=; b=eII2ypKlWiyfrgP2Q8OuRnD9nAR9D6WcoCpI0d+2BSX/IswyTuikjIBLnK9D/fKKY+ FAYZGEahE3GzRKNKoHJ+jSbtXEXGNSwGfQK/eVmOX8A8DJGPgqst4Ig3psGxLDK9eKUW 5EeH47Uj97RI0ORfy8wbVlczlQKFO+WaF4oIXGfcIl0hT0+sjNUkMg3O/J2ZqPAH8Ome kscVqwqJUhpm8NTV/ac/TXbiVXjHzLvXdTVNKnt5HTl6sPMX4OHQXiBbvL22P2b5zVXn WlSgeghkRKxFQe/rbPl0dfq23jnW2TR9zbHs8MD2nFQBQg38EFP/712ylyNG9OKWeCwg MnyQ== X-Gm-Message-State: AOAM5303aTkPGb3V5uwH0LDAxCYUFCA3OC7jO+9Xp+VRTo/VvXAotCPZ RCmMfsIo5QLGgMUR7908QFKrAyP0AHgqNutgMOl1vk62VzT2HDp+i8xuPAy//zKKLdPtcK4rnWS V/8vfIGpdi6tj X-Received: by 2002:adf:fb8f:: with SMTP id a15mr6051138wrr.92.1630006794546; Thu, 26 Aug 2021 12:39:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyFi9NXcOpnIV36WGLOJUq/THYNfhH3OgZ+38gkY2C+jDnkQzuS/bc0x0iDcjDVsSvlBt7Aug== X-Received: by 2002:adf:fb8f:: with SMTP id a15mr6051116wrr.92.1630006794343; Thu, 26 Aug 2021 12:39:54 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id b18sm4107493wrr.89.2021.08.26.12.39.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:39:54 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 05/27] ftrace: Add ftrace_add_rec_direct function Date: Thu, 26 Aug 2021 21:39:00 +0200 Message-Id: <20210826193922.66204-6-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Factor out the code that adds (ip, addr) tuple to direct_functions hash in new ftrace_add_rec_direct function. It will be used in following patches. Signed-off-by: Jiri Olsa --- kernel/trace/ftrace.c | 60 ++++++++++++++++++++++++++----------------- 1 file changed, 36 insertions(+), 24 deletions(-) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7b180f61e6d3..c60217d81040 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -2394,6 +2394,39 @@ unsigned long ftrace_find_rec_direct(unsigned long ip) return entry->direct; } +static struct ftrace_func_entry* +ftrace_add_rec_direct(unsigned long ip, unsigned long addr, + struct ftrace_hash **free_hash) +{ + struct ftrace_func_entry *entry; + + if (ftrace_hash_empty(direct_functions) || + direct_functions->count > 2 * (1 << direct_functions->size_bits)) { + struct ftrace_hash *new_hash; + int size = ftrace_hash_empty(direct_functions) ? 0 : + direct_functions->count + 1; + + if (size < 32) + size = 32; + + new_hash = dup_hash(direct_functions, size); + if (!new_hash) + return NULL; + + *free_hash = direct_functions; + direct_functions = new_hash; + } + + entry = kmalloc(sizeof(*entry), GFP_KERNEL); + if (!entry) + return NULL; + + entry->ip = ip; + entry->direct = addr; + __add_hash_entry(direct_functions, entry); + return entry; +} + static void call_direct_funcs(unsigned long ip, unsigned long pip, struct ftrace_ops *ops, struct ftrace_regs *fregs) { @@ -5110,27 +5143,6 @@ int register_ftrace_direct(unsigned long ip, unsigned long addr) } ret = -ENOMEM; - if (ftrace_hash_empty(direct_functions) || - direct_functions->count > 2 * (1 << direct_functions->size_bits)) { - struct ftrace_hash *new_hash; - int size = ftrace_hash_empty(direct_functions) ? 0 : - direct_functions->count + 1; - - if (size < 32) - size = 32; - - new_hash = dup_hash(direct_functions, size); - if (!new_hash) - goto out_unlock; - - free_hash = direct_functions; - direct_functions = new_hash; - } - - entry = kmalloc(sizeof(*entry), GFP_KERNEL); - if (!entry) - goto out_unlock; - direct = ftrace_find_direct_func(addr); if (!direct) { direct = ftrace_alloc_direct_func(addr); @@ -5140,9 +5152,9 @@ int register_ftrace_direct(unsigned long ip, unsigned long addr) } } - entry->ip = ip; - entry->direct = addr; - __add_hash_entry(direct_functions, entry); + entry = ftrace_add_rec_direct(ip, addr, &free_hash); + if (!entry) + goto out_unlock; ret = ftrace_set_filter_ip(&direct_ops, ip, 0, 0); if (ret) From patchwork Thu Aug 26 19:39:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460551 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C7AC432BE for ; Thu, 26 Aug 2021 19:40:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 822826108F for ; Thu, 26 Aug 2021 19:40:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243508AbhHZTkw (ORCPT ); Thu, 26 Aug 2021 15:40:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:39352 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243507AbhHZTkv (ORCPT ); Thu, 26 Aug 2021 15:40:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006803; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jJQ/tMNBXaGFHSvueMGSUoswOun/bpw7UK/P+WAvXSI=; b=B/u17cE6WAypTHOzp0YLTXxETYezaVPbhOn0FFdZdowrkGGJJiebWTl10XKYNuWJu7yFHT 4Yw3rgQs8B+EmXX9YbccuV9HVY1DecXWQ+s3iRRxuhGCZYXUqOFKExmjaJkjP955SSB3XS N09kZdACRWv8v5z9AR+JM1o+LvnMTQQ= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-576-7fgcI2KENWO5jHuT76ir-Q-1; Thu, 26 Aug 2021 15:40:02 -0400 X-MC-Unique: 7fgcI2KENWO5jHuT76ir-Q-1 Received: by mail-wm1-f71.google.com with SMTP id j33-20020a05600c1c21b02902e6828f7a20so1124694wms.7 for ; Thu, 26 Aug 2021 12:40:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jJQ/tMNBXaGFHSvueMGSUoswOun/bpw7UK/P+WAvXSI=; b=rVCI83ZSVqEO686KjLoo/H6G6ZBzBiKs4fMZr3fvTRp88DKzhJACfJFKpm9OiVyxag 2lP8Ao//AT3Xmvza8KnDltat/U4ju9D3KFfDmXDp76YPfIZuXvu9JNpcf5d/9fexOSBc bUrHLJIEkAwRKUQmHtvsku7c6HGlTG1DA8Bg1C00/j27ThW1Yydrmx6zWxn0HKwGhc1I AAZjDe+g8YaW1u0eCJFI6bMstnyWuzqgkD4cX/egfdBB8UxFmDAeGVsrlYhYS7nWEhhU QUssSAjUIGYD3IELvAOwOoWZeaFzY7HYcBhc/RzDiIfZSb6pGwKEn2hx/+EGrD6CU9Lc +6JQ== X-Gm-Message-State: AOAM533nFrvu1oDIywS9ty6K9dc4larkg9UTYidGQFqNtHt7TI7Bj9MK 8cCFQr6C6nF8D39JJQVOCaIDE/Rznsv767/S7eAlaTk48NIrsXtJ/AZaPj3ca9jU9ccsQbfMMKp vWovGUp6dKZuR X-Received: by 2002:a05:6000:259:: with SMTP id m25mr6083246wrz.53.1630006800787; Thu, 26 Aug 2021 12:40:00 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwkPTBgsURejuxJTIjuestEHFAy+9/r3Y8oB2z0Dt6mIxe7/DOx/+W3andgX1YyrhGriFbOQw== X-Received: by 2002:a05:6000:259:: with SMTP id m25mr6083221wrz.53.1630006800591; Thu, 26 Aug 2021 12:40:00 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id t5sm2944929wra.95.2021.08.26.12.39.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:00 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 06/27] ftrace: Add multi direct register/unregister interface Date: Thu, 26 Aug 2021 21:39:01 +0200 Message-Id: <20210826193922.66204-7-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding interface to register multiple direct functions within single call. Adding following functions: register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) unregister_ftrace_direct_multi(struct ftrace_ops *ops) The register_ftrace_direct_multi registers direct function (addr) with all functions in ops filter. The ops filter can be updated before with ftrace_set_filter_ip calls. All requested functions must not have direct function currently registered, otherwise register_ftrace_direct_multi will fail. The unregister_ftrace_direct_multi unregisters ops related direct functions. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 10 ++++ kernel/trace/ftrace.c | 111 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 9b218e59a608..93d8f12e70b3 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -316,6 +316,8 @@ int ftrace_modify_direct_caller(struct ftrace_func_entry *entry, unsigned long old_addr, unsigned long new_addr); unsigned long ftrace_find_rec_direct(unsigned long ip); +int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); +int unregister_ftrace_direct_multi(struct ftrace_ops *ops); #else # define ftrace_direct_func_count 0 static inline int register_ftrace_direct(unsigned long ip, unsigned long addr) @@ -346,6 +348,14 @@ static inline unsigned long ftrace_find_rec_direct(unsigned long ip) { return 0; } +int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +{ + return -ENODEV; +} +int unregister_ftrace_direct_multi(struct ftrace_ops *ops) +{ + return -ENODEV; +} #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index c60217d81040..7243769493c9 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5407,6 +5407,117 @@ int modify_ftrace_direct(unsigned long ip, return ret; } EXPORT_SYMBOL_GPL(modify_ftrace_direct); + +#define MULTI_FLAGS (FTRACE_OPS_FL_IPMODIFY | FTRACE_OPS_FL_DIRECT | \ + FTRACE_OPS_FL_SAVE_REGS) + +static int check_direct_multi(struct ftrace_ops *ops) +{ + if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) + return -EINVAL; + if ((ops->flags & MULTI_FLAGS) != MULTI_FLAGS) + return -EINVAL; + return 0; +} + +int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +{ + struct ftrace_hash *hash, *free_hash = NULL; + struct ftrace_func_entry *entry, *new; + int err = -EBUSY, size, i; + + if (ops->func || ops->trampoline) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) + return -EINVAL; + if (ops->flags & FTRACE_OPS_FL_ENABLED) + return -EINVAL; + + hash = ops->func_hash->filter_hash; + if (ftrace_hash_empty(hash)) + return -EINVAL; + + mutex_lock(&direct_mutex); + + /* Make sure requested entries are not already registered.. */ + size = 1 << hash->size_bits; + for (i = 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + if (ftrace_find_rec_direct(entry->ip)) + goto out_unlock; + } + } + + /* ... and insert them to direct_functions hash. */ + err = -ENOMEM; + for (i = 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + new = ftrace_add_rec_direct(entry->ip, addr, &free_hash); + if (!new) + goto out_remove; + entry->direct = addr; + } + } + + ops->func = call_direct_funcs; + ops->flags = MULTI_FLAGS; + ops->trampoline = FTRACE_REGS_ADDR; + + err = register_ftrace_function(ops); + + out_remove: + if (err) { + for (i = 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + new = __ftrace_lookup_ip(direct_functions, entry->ip); + if (new) { + remove_hash_entry(direct_functions, new); + kfree(new); + } + } + } + } + + out_unlock: + mutex_unlock(&direct_mutex); + + if (free_hash) { + synchronize_rcu_tasks(); + free_ftrace_hash(free_hash); + } + return err; +} +EXPORT_SYMBOL_GPL(register_ftrace_direct_multi); + +int unregister_ftrace_direct_multi(struct ftrace_ops *ops) +{ + struct ftrace_hash *hash = ops->func_hash->filter_hash; + struct ftrace_func_entry *entry, *new; + int err, size, i; + + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + + mutex_lock(&direct_mutex); + err = unregister_ftrace_function(ops); + + size = 1 << hash->size_bits; + for (i = 0; i < size; i++) { + hlist_for_each_entry(entry, &hash->buckets[i], hlist) { + new = __ftrace_lookup_ip(direct_functions, entry->ip); + if (new) { + remove_hash_entry(direct_functions, new); + kfree(new); + } + } + } + + mutex_unlock(&direct_mutex); + return err; +} +EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi); #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ /** From patchwork Thu Aug 26 19:39:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460553 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AF5DC4320A for ; Thu, 26 Aug 2021 19:40:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 849C9610A7 for ; Thu, 26 Aug 2021 19:40:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243528AbhHZTk6 (ORCPT ); Thu, 26 Aug 2021 15:40:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:59873 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243535AbhHZTk5 (ORCPT ); Thu, 26 Aug 2021 15:40:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cz5FUM/QUQ2HTCjVFzxMUjYkqvve+4iMlPHPGJoE+bY=; b=g5TPky87oEMa902vleuwTw67ErNjaz9WlHlKdtcVGuSE9Jmldb9mIigb6JpDN80lvI/iGb A9gOeocO1aM5ZdL1YEg4AH3KkonA/IWM6T2S9HEEQYVrxwfZN1qaTmy22N8b/mgoa0JDHb 5lCSJbUed1AzxgDsSE09GxvNHb08tH8= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-43-P8HyeHacNl6pwM4MzlpjfA-1; Thu, 26 Aug 2021 15:40:08 -0400 X-MC-Unique: P8HyeHacNl6pwM4MzlpjfA-1 Received: by mail-wr1-f69.google.com with SMTP id n18-20020adfe792000000b00156ae576abdso1190932wrm.9 for ; Thu, 26 Aug 2021 12:40:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Cz5FUM/QUQ2HTCjVFzxMUjYkqvve+4iMlPHPGJoE+bY=; b=W5ORW4crjXIULRR+Q9iatooD3v8fljc3S+pV+5rTVkqEWNq66eWGE+KHSoPlXj4XXt Kh83KLPje6pg4XTAhV3NEpAEYM0Bbz6mATxpCVIGtw8ThG81Ty0LIhxATz1I3EGILrr+ IDbgvuLahBRYav3I14xwGVLssZYhnQEUV6LwslGvtsq5SemZqqHUIuNUmwpCNsWbKEB3 /sbRws6g+c7a2iMCOakRPDj3QXnWEmW94AyiUrFLXLEJ0AgC1M5I4hFes0iXbAlbTYr0 bPYijEXNtMG2aPj4mvAipMjvYq867JwHroHVzRE7FJtb0bPZo7gAXuxSkYEoIQKqE6Od XJpw== X-Gm-Message-State: AOAM533RUtT8FoGU3TyPq4B7czBG8MycjSy/gNnKZiSJnuk/U8sqRuFz OXPyBKANFZUOCmytY8gsqVYdtStlbZNwIVB7yACmC9SqsyT3eTl6Wsbdj1nJpomsPodNV5TTvfn HdTiI4rcOPaOO X-Received: by 2002:adf:e4c5:: with SMTP id v5mr6082126wrm.1.1630006806980; Thu, 26 Aug 2021 12:40:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw+oapw3UzB0tUkEstf0tZaeuwgr3wd1znhLKDrE5AJuTWzq9xPA/RbcYKzFpfOJzGc1FS3SA== X-Received: by 2002:adf:e4c5:: with SMTP id v5mr6082103wrm.1.1630006806810; Thu, 26 Aug 2021 12:40:06 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id f2sm4002494wru.31.2021.08.26.12.40.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:06 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 07/27] ftrace: Add multi direct modify interface Date: Thu, 26 Aug 2021 21:39:02 +0200 Message-Id: <20210826193922.66204-8-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding interface to modify registered direct function for ftrace_ops. Adding following function: modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) The function changes the currently registered direct function for all attached functions. Signed-off-by: Jiri Olsa --- include/linux/ftrace.h | 6 ++++++ kernel/trace/ftrace.c | 43 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 93d8f12e70b3..63ca0a424947 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -318,6 +318,8 @@ int ftrace_modify_direct_caller(struct ftrace_func_entry *entry, unsigned long ftrace_find_rec_direct(unsigned long ip); int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); int unregister_ftrace_direct_multi(struct ftrace_ops *ops); +int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr); + #else # define ftrace_direct_func_count 0 static inline int register_ftrace_direct(unsigned long ip, unsigned long addr) @@ -356,6 +358,10 @@ int unregister_ftrace_direct_multi(struct ftrace_ops *ops) { return -ENODEV; } +int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +{ + return -ENODEV; +} #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 7243769493c9..59940a6a907c 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -5518,6 +5518,49 @@ int unregister_ftrace_direct_multi(struct ftrace_ops *ops) return err; } EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi); + +int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) +{ + struct ftrace_hash *hash = ops->func_hash->filter_hash; + struct ftrace_func_entry *entry, *iter; + int i, size; + int err; + + if (check_direct_multi(ops)) + return -EINVAL; + if (!(ops->flags & FTRACE_OPS_FL_ENABLED)) + return -EINVAL; + + mutex_lock(&direct_mutex); + mutex_lock(&ftrace_lock); + + /* + * Shutdown the ops, change 'direct' pointer for each + * ops entry in direct_functions hash and startup the + * ops back again. + */ + err = ftrace_shutdown(ops, 0); + if (err) + goto out_unlock; + + size = 1 << hash->size_bits; + for (i = 0; i < size; i++) { + hlist_for_each_entry(iter, &hash->buckets[i], hlist) { + entry = __ftrace_lookup_ip(direct_functions, iter->ip); + if (!entry) + continue; + entry->direct = addr; + } + } + + err = ftrace_startup(ops, 0); + + out_unlock: + mutex_unlock(&ftrace_lock); + mutex_unlock(&direct_mutex); + return err; +} +EXPORT_SYMBOL_GPL(modify_ftrace_direct_multi); #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ /** From patchwork Thu Aug 26 19:39:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460555 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12FAEC432BE for ; Thu, 26 Aug 2021 19:40:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F13AA610A6 for ; Thu, 26 Aug 2021 19:40:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243540AbhHZTlF (ORCPT ); Thu, 26 Aug 2021 15:41:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:24663 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243546AbhHZTlD (ORCPT ); Thu, 26 Aug 2021 15:41:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006816; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AjVYZzchk8zj3iEUBSEoZP/smDG6NngufYLxpwikn0s=; b=LDNy8AyIgJGgYdCtOnNvu2fbl/YeFgMR2I3CRU7Tv71qk69PxTb0OJykkx/Ly0nFbgo3M5 fa28TGRKKasUvoCNSuKDyaf37d64uydMzpQXXC0OBar54P1X7TJaW+iRCz2BBpwCZ0jlm4 B1BlSCz/4adEauXT5+0ivuFGlPaCbDU= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-453-FgTzSsuYM0-z4EJliwEPzA-1; Thu, 26 Aug 2021 15:40:14 -0400 X-MC-Unique: FgTzSsuYM0-z4EJliwEPzA-1 Received: by mail-wr1-f70.google.com with SMTP id t15-20020a5d42cf000000b001565f9c9ee8so1211175wrr.2 for ; Thu, 26 Aug 2021 12:40:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AjVYZzchk8zj3iEUBSEoZP/smDG6NngufYLxpwikn0s=; b=tBlc8INzhkMobBIJerE3MgRUmwby+os6v2Rk5cd5JVWSBhge01ORmxiX2dEEf0akR4 S8w8WpL8RJI9S+yerBaPcWJV9KzRUrBvObMjLIiJiOWBv/pHPldb3maC5HRDX+RPw4R+ JZeNQpyyvdBRpdeGxi4gDrzrMExjHCOq4csUcQFNHs8Yoicq30bjd3+5Xq3mSY2QlXni l2KlkUyB5rHXENYbsV8/BykHqO/jL/pGPOOxi/lZ/5XIdIn/Ax99QpLRkjK04KtMRrwY knmC1IdcCJuwlnr4va/jnI6bHjX7d8R5rgnAP2qKEHqayMaH+BvyMJEa4eEc1utHhX6D nwfw== X-Gm-Message-State: AOAM532Fe/+PfgldDaZd0/xgnRPdaoFhMpDdrOW8UT1hZHFghDyUmLby sY8x5vBOnNbHzzY2yJ3rHGsSWLJG/aNYqn4kKlYmTfClwcVjeOaLn7SNNlN86ncvLrxwMB+pdiq Uo6oHieZLkN1o X-Received: by 2002:adf:82b0:: with SMTP id 45mr6112045wrc.161.1630006813095; Thu, 26 Aug 2021 12:40:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwXWI8rRU0SR10XdK68IEbBWJLmfDc+ENkRYWVSvF2hIZkTc2PTWjhIOWFCG/NSRX9EmnJFaA== X-Received: by 2002:adf:82b0:: with SMTP id 45mr6112033wrc.161.1630006812955; Thu, 26 Aug 2021 12:40:12 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id z5sm9243836wmp.26.2021.08.26.12.40.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:12 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 08/27] ftrace/samples: Add multi direct interface test module Date: Thu, 26 Aug 2021 21:39:03 +0200 Message-Id: <20210826193922.66204-9-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding simple module that uses multi direct interface: register_ftrace_direct_multi unregister_ftrace_direct_multi The init function registers trampoline for 2 functions, and exit function unregisters them. Signed-off-by: Jiri Olsa --- samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-direct-multi.c | 52 ++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 samples/ftrace/ftrace-direct-multi.c diff --git a/samples/ftrace/Makefile b/samples/ftrace/Makefile index 4ce896e10b2e..ab1d1c05c288 100644 --- a/samples/ftrace/Makefile +++ b/samples/ftrace/Makefile @@ -3,6 +3,7 @@ obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-too.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-modify.o +obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-multi.o CFLAGS_sample-trace-array.o := -I$(src) obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += sample-trace-array.o diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c new file mode 100644 index 000000000000..76b34d46d11c --- /dev/null +++ b/samples/ftrace/ftrace-direct-multi.c @@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include + +#include /* for handle_mm_fault() */ +#include +#include + +void my_direct_func(unsigned long ip) +{ + trace_printk("ip %lx\n", ip); +} + +extern void my_tramp(void *); + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp, @function\n" +" .globl my_tramp\n" +" my_tramp:" +" pushq %rbp\n" +" movq %rsp, %rbp\n" +" pushq %rdi\n" +" movq 8(%rbp), %rdi\n" +" call my_direct_func\n" +" popq %rdi\n" +" leave\n" +" ret\n" +" .size my_tramp, .-my_tramp\n" +" .popsection\n" +); + +static struct ftrace_ops direct; + +static int __init ftrace_direct_multi_init(void) +{ + ftrace_set_filter_ip(&direct, (unsigned long) wake_up_process, 0, 0); + ftrace_set_filter_ip(&direct, (unsigned long) schedule, 0, 0); + + return register_ftrace_direct_multi(&direct, (unsigned long) my_tramp); +} + +static void __exit ftrace_direct_multi_exit(void) +{ + unregister_ftrace_direct_multi(&direct); +} + +module_init(ftrace_direct_multi_init); +module_exit(ftrace_direct_multi_exit); + +MODULE_AUTHOR("Jiri Olsa"); +MODULE_DESCRIPTION("Example use case of using register_ftrace_direct_multi()"); +MODULE_LICENSE("GPL"); From patchwork Thu Aug 26 19:39:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460557 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80586C4320A for ; Thu, 26 Aug 2021 19:40:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 69E4A610C8 for ; Thu, 26 Aug 2021 19:40:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243539AbhHZTlO (ORCPT ); Thu, 26 Aug 2021 15:41:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:23997 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243541AbhHZTlK (ORCPT ); Thu, 26 Aug 2021 15:41:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v00APJ6meHaZsL60TP0qAulcqbDujycB2Ey6wbdiEio=; b=ZQM3UD9L8j7OkZnunNNDNHYNbKPoQYjffFQuASNDjH8zfmHjGz5nZ3K5rXSW/VAVkZJso5 JLQK5fxEtueue28VdCunygKIATzh28+c0UrY/lBoYLzOqVZyc685baDbqA/Issp7kvBlAs fOeZrFjBZXVRtnLs4wJDjzYz8C/eTe8= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-4-djb-LbAZOdK3eLEgt1Yn-A-1; Thu, 26 Aug 2021 15:40:20 -0400 X-MC-Unique: djb-LbAZOdK3eLEgt1Yn-A-1 Received: by mail-wm1-f71.google.com with SMTP id y188-20020a1c7dc5000000b002e80e0b2f87so4777668wmc.1 for ; Thu, 26 Aug 2021 12:40:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v00APJ6meHaZsL60TP0qAulcqbDujycB2Ey6wbdiEio=; b=KSNK+wxRGEh5nr5sD8ap3a1Mb0rFr/saRJGu9h31dw4N+nlZZTqHU3CrHm0b6iRsO6 bnNNqHRz1lDR8CXProOBBCzhqqpZfMGVRwPHjWVVG0M6E+yFXe712ufrTfVAYpZNMwzZ spD+0sCX8gZSNJa90kEaTwLZsZRzDp4PdjBcPstuymnr1AfISsJKiuKI3gzeiMlMTzlm aqd/3VYlapXBVvsZ1xT5qUPv2/CgDLVSHqaXROoMKV/kQCMQj/ATR5OCRlzNBJ1jSJ7u weuGtd1yxpZaHcZw3FOK0OMlbVwGnZhUPZfAEoeVUbzS5wGC0Rhq6Vmab+2/aAI7jO38 lSNw== X-Gm-Message-State: AOAM532zVodJAFeKhhMXprEegKG85GJgbkzj6g8uLVxjum5kOElY5dVc KugwMtfdv4eJZE0aFPWk5h7GRI9ABK6Vgt6B3WZNFt6wywVKjV53b6DktxMkZv1QUSRe8f3ZY4o JEADR/4N98yWl X-Received: by 2002:a05:600c:154e:: with SMTP id f14mr16035639wmg.162.1630006819453; Thu, 26 Aug 2021 12:40:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxMzN3kbJMLDo29fhXdu4MghVFwWLrCtOYz0OBBkS+zzgjXQaVK4AOPIsKHCwBqu0N6kFRmgw== X-Received: by 2002:a05:600c:154e:: with SMTP id f14mr16035614wmg.162.1630006819193; Thu, 26 Aug 2021 12:40:19 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id n14sm4016308wrx.10.2021.08.26.12.40.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:18 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 09/27] bpf: Add support to load multi func tracing program Date: Thu, 26 Aug 2021 21:39:04 +0200 Message-Id: <20210826193922.66204-10-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to load tracing program with new BPF_F_MULTI_FUNC flag, that allows the program to be loaded without specific function to be attached to. Such program will be allowed to be attached to multiple functions in following patches. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 1 + include/uapi/linux/bpf.h | 7 +++++++ kernel/bpf/syscall.c | 35 +++++++++++++++++++++++++++++----- kernel/bpf/verifier.c | 3 ++- tools/include/uapi/linux/bpf.h | 7 +++++++ 5 files changed, 47 insertions(+), 6 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f4c16f19f83e..dc9838d741ac 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -865,6 +865,7 @@ struct bpf_prog_aux { bool func_proto_unreliable; bool sleepable; bool tail_call_reachable; + bool multi_func; struct hlist_node tramp_hlist; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 791f31dd0abe..1f9d336861f0 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1110,6 +1110,13 @@ enum bpf_link_type { */ #define BPF_F_SLEEPABLE (1U << 4) +/* If BPF_F_MULTI_FUNC is used in BPF_PROG_LOAD command, the verifier does + * not expect BTF ID for the program, instead it assumes it's function + * with 6 u64 arguments. No trampoline is created for the program. Such + * program can be attached to multiple functions. + */ +#define BPF_F_MULTI_FUNC (1U << 5) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: * diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 4e50c0bfdb7d..fa3f93c423d8 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -31,6 +31,7 @@ #include #include #include +#include #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -2012,7 +2013,8 @@ static int bpf_prog_load_check_attach(enum bpf_prog_type prog_type, enum bpf_attach_type expected_attach_type, struct btf *attach_btf, u32 btf_id, - struct bpf_prog *dst_prog) + struct bpf_prog *dst_prog, + bool multi_func) { if (btf_id) { if (btf_id > BTF_MAX_TYPE) @@ -2032,6 +2034,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, } } + if (multi_func) { + if (prog_type != BPF_PROG_TYPE_TRACING) + return -EINVAL; + if (!attach_btf || btf_id) + return -EINVAL; + return 0; + } + if (attach_btf && (!btf_id || dst_prog)) return -EINVAL; @@ -2155,6 +2165,16 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) } } +#define DEFINE_BPF_MULTI_FUNC(args...) \ + extern int bpf_multi_func(args); \ + int __init bpf_multi_func(args) { return 0; } + +DEFINE_BPF_MULTI_FUNC(unsigned long a1, unsigned long a2, + unsigned long a3, unsigned long a4, + unsigned long a5, unsigned long a6) + +BTF_ID_LIST_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func) + /* last field in 'union bpf_attr' used by this command */ #define BPF_PROG_LOAD_LAST_FIELD fd_array @@ -2165,6 +2185,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) struct btf *attach_btf = NULL; int err; char license[128]; + bool multi_func; bool is_gpl; if (CHECK_ATTR(BPF_PROG_LOAD)) @@ -2174,7 +2195,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) BPF_F_ANY_ALIGNMENT | BPF_F_TEST_STATE_FREQ | BPF_F_SLEEPABLE | - BPF_F_TEST_RND_HI32)) + BPF_F_TEST_RND_HI32 | + BPF_F_MULTI_FUNC)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && @@ -2205,6 +2227,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) if (is_perfmon_prog_type(type) && !perfmon_capable()) return -EPERM; + multi_func = attr->prog_flags & BPF_F_MULTI_FUNC; + /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog * or btf, we need to check which one it is */ @@ -2223,7 +2247,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) return -ENOTSUPP; } } - } else if (attr->attach_btf_id) { + } else if (attr->attach_btf_id || multi_func) { /* fall back to vmlinux BTF, if BTF type ID is specified */ attach_btf = bpf_get_btf_vmlinux(); if (IS_ERR(attach_btf)) @@ -2236,7 +2260,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) bpf_prog_load_fixup_attach_type(attr); if (bpf_prog_load_check_attach(type, attr->expected_attach_type, attach_btf, attr->attach_btf_id, - dst_prog)) { + dst_prog, multi_func)) { if (dst_prog) bpf_prog_put(dst_prog); if (attach_btf) @@ -2256,10 +2280,11 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr) prog->expected_attach_type = attr->expected_attach_type; prog->aux->attach_btf = attach_btf; - prog->aux->attach_btf_id = attr->attach_btf_id; + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id; prog->aux->dst_prog = dst_prog; prog->aux->offload_requested = !!attr->prog_ifindex; prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE; + prog->aux->multi_func = multi_func; err = security_bpf_prog_alloc(prog->aux); if (err) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 206c221453cf..e9e84adfb974 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13628,7 +13628,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) if (!bpf_iter_prog_supported(prog)) return -EINVAL; return 0; - } + } else if (prog->aux->multi_func) + return prog->type == BPF_PROG_TYPE_TRACING ? 0 : -EINVAL; if (prog->type == BPF_PROG_TYPE_LSM) { ret = bpf_lsm_verify_prog(&env->log, prog); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 791f31dd0abe..1f9d336861f0 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1110,6 +1110,13 @@ enum bpf_link_type { */ #define BPF_F_SLEEPABLE (1U << 4) +/* If BPF_F_MULTI_FUNC is used in BPF_PROG_LOAD command, the verifier does + * not expect BTF ID for the program, instead it assumes it's function + * with 6 u64 arguments. No trampoline is created for the program. Such + * program can be attached to multiple functions. + */ +#define BPF_F_MULTI_FUNC (1U << 5) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: * From patchwork Thu Aug 26 19:39:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460559 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB832C432BE for ; Thu, 26 Aug 2021 19:40:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A6559610D0 for ; Thu, 26 Aug 2021 19:40:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243564AbhHZTlS (ORCPT ); Thu, 26 Aug 2021 15:41:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:29521 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243552AbhHZTlQ (ORCPT ); Thu, 26 Aug 2021 15:41:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006828; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XL7EGdnRDnjROM7SJL/BtZcDUjirQCAXAUSG27AzLfg=; b=Whvi/GPUV1G0RsYoAfag96W8yZVC4kfCTcbGWTU12q22QFP3BLbYMwhJRzWxGV2/HR4lp4 69pGVmZTVILLJKGllrN5ypSjbxptc6KRm3xV6tR5iYDb2GpfFy4DyIlM+JLbHaphRkQJCK 7VXOdXzuBZuaWvY6PyaxXU8awEwv3Hk= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-226-5u7IVD18Pjq3TuhKVxPCPA-1; Thu, 26 Aug 2021 15:40:27 -0400 X-MC-Unique: 5u7IVD18Pjq3TuhKVxPCPA-1 Received: by mail-wr1-f71.google.com with SMTP id r17-20020adfda510000b02901526f76d738so1222329wrl.0 for ; Thu, 26 Aug 2021 12:40:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XL7EGdnRDnjROM7SJL/BtZcDUjirQCAXAUSG27AzLfg=; b=Z/l2+YdQDzE5UxujLHTP2hCVgUoJeWSvp4GXxBbPyjTqvGhliZMIuk28wp2qdy0mCJ Tt9fxZD/38AXDdF9gbvKBYfa/l0OPtXHDToj+CIKy8Ab0P5et0pb8+4XEnLD5EMJPFPX JpAlfq1nIKsgyOktTPvEwT9ZKQv560L9yxYWoGIVcjyr5n9V1EwlLT1CINYn/gxNekZN 6XIicuB055eXE0SYzMQ3jcG0XxSuV/EAaDe74CnvDZecDtNtI/DcadyXd7rYmnfcisa8 RaVYpHNcKmCRhHEN6ANLbtp+3XKcBaJ3W7rZKlXrH2cfIKfan697t6A+f+GW0o1wvqFq U2ng== X-Gm-Message-State: AOAM531G5H4f1wMB7wB6lMVyAg0dP3vd3wcMaA4LSWJHpNsDWNOF/CgI nZzdxEKCgxVsiaOq9Gd2Cuvyp1q1eLHJxEmnj7fPOwj2wW0ZlNCS8/Ky58Q/tosbhAqZcMxYNgK 49tIVmaWCAzx3 X-Received: by 2002:adf:c442:: with SMTP id a2mr6178143wrg.228.1630006825870; Thu, 26 Aug 2021 12:40:25 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz4ThkcoabbL1tyL3cMS54Eqzkexu+Sh11dVw2PpPP3/j80quT+iIshaVRguUeFAaNNH5Joaw== X-Received: by 2002:adf:c442:: with SMTP id a2mr6178127wrg.228.1630006825695; Thu, 26 Aug 2021 12:40:25 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id 18sm10264579wmv.27.2021.08.26.12.40.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:25 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 10/27] bpf: Add struct bpf_tramp_node layer Date: Thu, 26 Aug 2021 21:39:05 +0200 Message-Id: <20210826193922.66204-11-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Currently each trampoline holds a list of programs that are attached to it. With multi func attach support we need a way for a single program to be connected to multiple trampolines. Adding struct bpf_tramp_node object that holds bpf_prog pointer, so it can be resolved directly. We can now have multiple struct bpf_tramp_node being attached to different trampolines pointing to single bpf_prog. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 15 ++++++++++----- kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 4 ++-- kernel/bpf/trampoline.c | 22 ++++++++++++---------- 4 files changed, 25 insertions(+), 17 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index dc9838d741ac..f0f548f8f391 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -656,6 +656,11 @@ struct bpf_tramp_image { }; }; +struct bpf_tramp_node { + struct hlist_node hlist; + struct bpf_prog *prog; +}; + struct bpf_trampoline { /* hlist for trampoline_table */ struct hlist_node hlist; @@ -717,8 +722,8 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( return bpf_func(ctx, insnsi); } #ifdef CONFIG_BPF_JIT -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr); +int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); +int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr); struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); @@ -767,12 +772,12 @@ void bpf_ksym_del(struct bpf_ksym *ksym); int bpf_jit_charge_modmem(u32 pages); void bpf_jit_uncharge_modmem(u32 pages); #else -static inline int bpf_trampoline_link_prog(struct bpf_prog *prog, +static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { return -ENOTSUPP; } -static inline int bpf_trampoline_unlink_prog(struct bpf_prog *prog, +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { return -ENOTSUPP; @@ -866,7 +871,7 @@ struct bpf_prog_aux { bool sleepable; bool tail_call_reachable; bool multi_func; - struct hlist_node tramp_hlist; + struct bpf_tramp_node tramp_node; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 9f4636d021b1..bad03dde97a2 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -104,6 +104,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag fp->aux = aux; fp->aux->prog = fp; fp->jit_requested = ebpf_jit_enabled(); + fp->aux->tramp_node.prog = fp; INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); mutex_init(&fp->aux->used_maps_mutex); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index fa3f93c423d8..e667d392cc33 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2625,7 +2625,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link); - WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog, + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&link->prog->aux->tramp_node, tr_link->trampoline)); bpf_trampoline_put(tr_link->trampoline); @@ -2813,7 +2813,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - err = bpf_trampoline_link_prog(prog, tr); + err = bpf_trampoline_link_prog(&prog->aux->tramp_node, tr); if (err) { bpf_link_cleanup(&link_primer); link = NULL; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index fe1e857324e6..525fa74c2f62 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -174,8 +174,8 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) static struct bpf_tramp_progs * bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) { - const struct bpf_prog_aux *aux; struct bpf_tramp_progs *tprogs; + struct bpf_tramp_node *node; struct bpf_prog **progs; int kind; @@ -189,9 +189,9 @@ bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_a *total += tr->progs_cnt[kind]; progs = tprogs[kind].progs; - hlist_for_each_entry(aux, &tr->progs_hlist[kind], tramp_hlist) { - *ip_arg |= aux->prog->call_get_func_ip; - *progs++ = aux->prog; + hlist_for_each_entry(node, &tr->progs_hlist[kind], hlist) { + *ip_arg |= node->prog->call_get_func_ip; + *progs++ = node->prog; } } return tprogs; @@ -410,8 +410,9 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) +int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { + struct bpf_prog *prog = node->prog; enum bpf_tramp_prog_type kind; int err = 0; int cnt; @@ -441,16 +442,16 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) err = -E2BIG; goto out; } - if (!hlist_unhashed(&prog->aux->tramp_hlist)) { + if (!hlist_unhashed(&node->hlist)) { /* prog already linked */ err = -EBUSY; goto out; } - hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]); + hlist_add_head(&node->hlist, &tr->progs_hlist[kind]); tr->progs_cnt[kind]++; err = bpf_trampoline_update(tr); if (err) { - hlist_del_init(&prog->aux->tramp_hlist); + hlist_del_init(&node->hlist); tr->progs_cnt[kind]--; } out: @@ -459,8 +460,9 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) } /* bpf_trampoline_unlink_prog() should never fail. */ -int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) +int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampoline *tr) { + struct bpf_prog *prog = node->prog; enum bpf_tramp_prog_type kind; int err; @@ -473,7 +475,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr) tr->extension_prog = NULL; goto out; } - hlist_del_init(&prog->aux->tramp_hlist); + hlist_del_init(&node->hlist); tr->progs_cnt[kind]--; err = bpf_trampoline_update(tr); out: From patchwork Thu Aug 26 19:39:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460561 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FC09C432BE for ; Thu, 26 Aug 2021 19:40:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6BD26610C8 for ; Thu, 26 Aug 2021 19:40:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243560AbhHZTlm (ORCPT ); Thu, 26 Aug 2021 15:41:42 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:26584 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243568AbhHZTlW (ORCPT ); Thu, 26 Aug 2021 15:41:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006834; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PSIDNuxSNboWGja6JJJ//OQqFnAgGxrRV3fj8oXduYs=; b=JAZ2tTRDSxMLX26HYqGRw7tJDdH6QN0amlb2gbT2+79JIPbkxUtd2SIZ64dsYsy3gOEf4E 8dJyDHC9XOW5tLASQ15rMxq0brZl79Fo0Ex/N3J9cNJe3MQprvUhGzeBz+yL8/NghBMOL9 6X/bfdBg1KL5t2wBmz9bsVV3ot0tZU8= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-105-0rta5N5oMOmhsphMBj5R-w-1; Thu, 26 Aug 2021 15:40:33 -0400 X-MC-Unique: 0rta5N5oMOmhsphMBj5R-w-1 Received: by mail-wm1-f69.google.com with SMTP id 5-20020a1c00050000b02902e67111d9f0so1269078wma.4 for ; Thu, 26 Aug 2021 12:40:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PSIDNuxSNboWGja6JJJ//OQqFnAgGxrRV3fj8oXduYs=; b=YKVQSQ/mf2JGpYQ6l/AgWIET0nXIDnYiXaJ4/W2iIIqkl3NrHVkimntHPXhhvZ8893 7Gz6eZMOLK5BPnU4FfELyOrrK+P28oh6UsdBqkh25rWwUo/5dgfz9fyg0FN2NhuH/xFt J4CVUn70Fa7gmrN4xCxCtTDApAUbDjjvYs9qOpbvAyjksDzquS9DEG/MEkP1tXzivQsB ew1J/q7yYmI2Ho5p/DHA/YaUnsKnBCB7SJos71axORbOrOz7j6+u2ltqY1gd5anFUQCL BcQ9Wio7NEpU6UfFrlvoiJHzV1w7RJ4oYEGMjhnoVVjH2KHeqHFD4vWF8C9tEWImjO5t ux0A== X-Gm-Message-State: AOAM530Gao0ifnA1qL46UOtY7EF66I70nVm3amkZ8OwDRJFkqAib3urR xa2wdZ+RNtscenMGu0vvD1E5Qd+UZYK4dI7CROJJWcqYc5vcWQdwAyCzrhdz7vrNlZCRFQlaNHi DR0i5PVc06J6i X-Received: by 2002:a5d:4d03:: with SMTP id z3mr6008079wrt.229.1630006832018; Thu, 26 Aug 2021 12:40:32 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyuQjRMB3r13599nxIcsa8dPQw9RicOFY+s2ocnwBE1cxvcGUhEESYSLd5EulO1V4gbP89M8g== X-Received: by 2002:a5d:4d03:: with SMTP id z3mr6008069wrt.229.1630006831880; Thu, 26 Aug 2021 12:40:31 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id g11sm4016811wrx.30.2021.08.26.12.40.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:31 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 11/27] bpf: Factor out bpf_trampoline_init function Date: Thu, 26 Aug 2021 21:39:06 +0200 Message-Id: <20210826193922.66204-12-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Separating out bpf_trampoline_init function, so it can be used from other places in following patches. Signed-off-by: Jiri Olsa --- kernel/bpf/trampoline.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 525fa74c2f62..f44899c9698a 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -58,11 +58,22 @@ void bpf_image_ksym_del(struct bpf_ksym *ksym) PAGE_SIZE, true, ksym->name); } +static void bpf_trampoline_init(struct bpf_trampoline *tr, u64 key) +{ + int i; + + tr->key = key; + INIT_HLIST_NODE(&tr->hlist); + refcount_set(&tr->refcnt, 1); + mutex_init(&tr->mutex); + for (i = 0; i < BPF_TRAMP_MAX; i++) + INIT_HLIST_HEAD(&tr->progs_hlist[i]); +} + static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) { struct bpf_trampoline *tr; struct hlist_head *head; - int i; mutex_lock(&trampoline_mutex); head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; @@ -75,14 +86,8 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) tr = kzalloc(sizeof(*tr), GFP_KERNEL); if (!tr) goto out; - - tr->key = key; - INIT_HLIST_NODE(&tr->hlist); + bpf_trampoline_init(tr, key); hlist_add_head(&tr->hlist, head); - refcount_set(&tr->refcnt, 1); - mutex_init(&tr->mutex); - for (i = 0; i < BPF_TRAMP_MAX; i++) - INIT_HLIST_HEAD(&tr->progs_hlist[i]); out: mutex_unlock(&trampoline_mutex); return tr; From patchwork Thu Aug 26 19:39:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460563 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 942ACC432BE for ; Thu, 26 Aug 2021 19:40:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7A08D60F4C for ; Thu, 26 Aug 2021 19:40:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243573AbhHZTln (ORCPT ); Thu, 26 Aug 2021 15:41:43 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:46557 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243586AbhHZTl2 (ORCPT ); Thu, 26 Aug 2021 15:41:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gXpJHZO1cihtLBArG/eE/Ics75WBgBQPWIW5tbtMZsM=; b=Zmid3RIHz2Jxmur7Linz/Ppuo/7yLy+HZyfvRUsAHfvWcq8XBtPH0zM+YyOYE2V05oMDYx +MFIx9nwlTcF4CFA6xH9AZ5QPMnAk+zOnmvE4hVmhB53FPU9k1afAcqS5gm37FfkgrLlHQ F28a36+j7hhg1GPtWgCkwPBZvQrQoGs= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-49-SYyEA1E3OvO9V8y9-4Jfyw-1; Thu, 26 Aug 2021 15:40:39 -0400 X-MC-Unique: SYyEA1E3OvO9V8y9-4Jfyw-1 Received: by mail-wr1-f71.google.com with SMTP id i16-20020adfded0000000b001572ebd528eso1176983wrn.19 for ; Thu, 26 Aug 2021 12:40:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gXpJHZO1cihtLBArG/eE/Ics75WBgBQPWIW5tbtMZsM=; b=K/JughXegVNWnMXv8usq+0/Dj45zLwP6mOqEwshFmiV8lJktOI7q5ehmub8+PsHdrN 6d6zDjKvaSV8RQ2paN1hGnv90JjfrS34npCNGHg7u2philM5sc75yvp0WrClUnka7MKe WYGXxzgF2CwFYU4O/sjihlClNRtodGPh0FU1B8HP3O9JGh2Mp3mglwDfu1VdpIHabPkn hg8D1digt0H/26zh8onqPTyKT1sDuLg7iFU9zr3bVcr3TndhyOxM6/QzV+y3nsL0buBh JO86tkIjlJHSUudC0z3mz5KVyUhd7H7kdJ/FPYsK8uZaPUxPyRpvnlt6A7UFs7HHyYE9 rGdA== X-Gm-Message-State: AOAM533d6R9ZbZBc4B2S6MEEVSL5nZoveWLVn3fpDYhj84vgPZjwAm5Q 37kUZ+hPkG5GRC4qBtXaXxO6EFXV5G+s7B0HLAxPre3HDUKqMEV8R0FXaE8wU6JezjCdOOo+KJr OhRfhnj5hElK0 X-Received: by 2002:adf:d0cf:: with SMTP id z15mr6017472wrh.356.1630006838126; Thu, 26 Aug 2021 12:40:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx03/eQ8lgmsE2bT+LDzqvydNywb8x2w37h8HYfEYEsVbKsve6VGBDJGD7R+bXh6oEAsZ3oyg== X-Received: by 2002:adf:d0cf:: with SMTP id z15mr6017455wrh.356.1630006837992; Thu, 26 Aug 2021 12:40:37 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id r12sm4234429wrv.96.2021.08.26.12.40.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:37 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 12/27] bpf: Factor out __bpf_trampoline_lookup function Date: Thu, 26 Aug 2021 21:39:07 +0200 Message-Id: <20210826193922.66204-13-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Separating out __bpf_trampoline_lookup function, so it can be used from other places in following patches. Signed-off-by: Jiri Olsa --- kernel/bpf/trampoline.c | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index f44899c9698a..6dba43266e0b 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -70,23 +70,37 @@ static void bpf_trampoline_init(struct bpf_trampoline *tr, u64 key) INIT_HLIST_HEAD(&tr->progs_hlist[i]); } -static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +static struct bpf_trampoline *__bpf_trampoline_lookup(u64 key) { struct bpf_trampoline *tr; struct hlist_head *head; - mutex_lock(&trampoline_mutex); + lockdep_assert_held(&trampoline_mutex); + head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; hlist_for_each_entry(tr, head, hlist) { - if (tr->key == key) { - refcount_inc(&tr->refcnt); - goto out; - } + if (tr->key == key) + return tr; + } + return NULL; +} + +static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) +{ + struct bpf_trampoline *tr; + struct hlist_head *head; + + mutex_lock(&trampoline_mutex); + tr = __bpf_trampoline_lookup(key); + if (tr) { + refcount_inc(&tr->refcnt); + goto out; } tr = kzalloc(sizeof(*tr), GFP_KERNEL); if (!tr) goto out; bpf_trampoline_init(tr, key); + head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; hlist_add_head(&tr->hlist, head); out: mutex_unlock(&trampoline_mutex); From patchwork Thu Aug 26 19:39:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460565 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 730FEC4320E for ; Thu, 26 Aug 2021 19:40:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 536B160F4C for ; Thu, 26 Aug 2021 19:40:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243583AbhHZTlp (ORCPT ); Thu, 26 Aug 2021 15:41:45 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:24094 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243476AbhHZTle (ORCPT ); Thu, 26 Aug 2021 15:41:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006846; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z2XITBrOxDpGRyjAJ/GeVEmFFxm+Vz1j/C47GlqXHzU=; b=CocDHnl9xXT2Mnawh/vuRjEM2/yahjy32SkyjftiNfIlmBPyzUnw47HUIZLTdlH98R50Qj 3wiBGKc9xpog+yXWKx3DEfUDvTNCTfWxRAniCXive3EPEqnRtDNycEE1iQBwdYrwhJa2IE G4E/C9Av834Q5/TEQiKf+XOwaDwdIvI= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-513-LABtArzFMqOzaGati3I0Mw-1; Thu, 26 Aug 2021 15:40:45 -0400 X-MC-Unique: LABtArzFMqOzaGati3I0Mw-1 Received: by mail-wm1-f70.google.com with SMTP id x125-20020a1c3183000000b002e73f079eefso3448637wmx.0 for ; Thu, 26 Aug 2021 12:40:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z2XITBrOxDpGRyjAJ/GeVEmFFxm+Vz1j/C47GlqXHzU=; b=Qh+hoM3zzkmWGsbJEuaRah3n+hkzz6tsq4uvPJNzqAIr7Ofx4EqVeU4lYJFcrNj2da qNe8iGzNMhq+oozkrm/CIpfSoDZAw1AeUEnaU+5UMowFTJETffM1eclqCtQDiSTYRgGt viLL1cq7WEVxSzoW2qczhtWUzEPJmQ7nGenEbxlEanVDZC3fZZhdavi+7naqFsoRhsec 6PAJ+m7j7BrrKSzwKw/40wE9znIPaEZ4dquISjHYjkOUv8yoiTZrecwsThK5r6zWlKlL qjjPfYaFMfVr9VcQXRvGJcgZPcgD//hoTbQWZ1UQEJHeRjc3lqfVamhdA5o+F1OVvfXZ kzlQ== X-Gm-Message-State: AOAM530hnRRUpc1IlvfX00oTPLKaswFBcgG7i+DlQD1c00M5C7Q8LoiJ 6uz17wVqDP5JsfS8Idz2rXBVTzi+TiDF5+QysM9XLRHryeH2uGevJi7ItN8K7KUnJxswI9wsxZA u+4GGEIUIuVsN X-Received: by 2002:a1c:ed10:: with SMTP id l16mr15751501wmh.8.1630006844415; Thu, 26 Aug 2021 12:40:44 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzitQXnasWLsuJudMdZeXK+TS2WTVT2UhHDl/35tfbJIVjoTYJHVpOxEFwQ00OumMXLgAsD+g== X-Received: by 2002:a1c:ed10:: with SMTP id l16mr15751486wmh.8.1630006844263; Thu, 26 Aug 2021 12:40:44 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id f7sm9136006wmh.20.2021.08.26.12.40.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:43 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 13/27] bpf: Factor out __bpf_trampoline_put function Date: Thu, 26 Aug 2021 21:39:08 +0200 Message-Id: <20210826193922.66204-14-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Separating out __bpf_trampoline_put function, so it can be used from other places in following patches. Signed-off-by: Jiri Olsa --- kernel/bpf/trampoline.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 6dba43266e0b..8aa0aca38b3a 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -522,18 +522,16 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, return tr; } -void bpf_trampoline_put(struct bpf_trampoline *tr) +static void __bpf_trampoline_put(struct bpf_trampoline *tr) { - if (!tr) - return; - mutex_lock(&trampoline_mutex); + lockdep_assert_held(&trampoline_mutex); if (!refcount_dec_and_test(&tr->refcnt)) - goto out; + return; WARN_ON_ONCE(mutex_is_locked(&tr->mutex)); if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FENTRY]))) - goto out; + return; if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT]))) - goto out; + return; /* This code will be executed even when the last bpf_tramp_image * is alive. All progs are detached from the trampoline and the * trampoline image is patched with jmp into epilogue to skip @@ -542,7 +540,14 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) */ hlist_del(&tr->hlist); kfree(tr); -out: +} + +void bpf_trampoline_put(struct bpf_trampoline *tr) +{ + if (!tr) + return; + mutex_lock(&trampoline_mutex); + __bpf_trampoline_put(tr); mutex_unlock(&trampoline_mutex); } From patchwork Thu Aug 26 19:39:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460567 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 003F8C43214 for ; Thu, 26 Aug 2021 19:41:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E1E9360FDA for ; Thu, 26 Aug 2021 19:41:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243566AbhHZTlq (ORCPT ); Thu, 26 Aug 2021 15:41:46 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:52809 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243598AbhHZTll (ORCPT ); Thu, 26 Aug 2021 15:41:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006853; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zTg+zzMWSEF+4A/U8wBHLOWp42sO4HXvxr5/nTc1gqE=; b=a5KBP3g/y7ytiteK1wwxDfet8sMJpZGZvE1kAkPSqdLkjQpebBp7HYEwKOxmQWQfSrh0US GKIywtS750X+rWzi5nNa+9NWwkkpfxPpwSO3sYglB4nne0dWfnzxHXyIvUOW1M7St4BWRR thCytPk/yZ2MBOmTigW3ahnl6MaQzxo= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-560-zoXyT7GLMUO3s08CKTOZOA-1; Thu, 26 Aug 2021 15:40:51 -0400 X-MC-Unique: zoXyT7GLMUO3s08CKTOZOA-1 Received: by mail-wm1-f72.google.com with SMTP id y188-20020a1c7dc5000000b002e80e0b2f87so4778221wmc.1 for ; Thu, 26 Aug 2021 12:40:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zTg+zzMWSEF+4A/U8wBHLOWp42sO4HXvxr5/nTc1gqE=; b=Fzu91QUauxKfXeITr57c2G3XPHRUDAsl/aQz43WFy4kSKVOaXc/rO4vJIX4laUnRtQ dYMuQTtM6zmDtxOZo52XIPRWFK/CB90H5xoAZ0zH6jPyAe0OIWCryAEQDF70PVGFdi/G UvTHoq4pvkJVY/ig51hbbV5DuYWapGkFVKbzYL+oBmFapGQVDhs0ZT4gXyKFJ9baWfki DSn3ywT5RiyGG4S9S2rpsQobN7vSYGFNLXP/NEidejwE4vN+JSynfj2bw3xUcIAxxuM1 ADgzVXuMAORf7aVaK+lkGRmSHmfsbpxwHEPFrGpLsMk6DrTaF3SuKJn1Ye3vmusEeujJ wFPA== X-Gm-Message-State: AOAM533hgRVxUT4wa8YltOSHIMub/gyQ+/zYq7xnsbBwvB7WcoHYkFxl Q/EdUg1JUwyu1X7IXT+xa6iJmfii0sjzr1WBIiMPw765qh+dEF0RLCzncVyrvzz+JghNNJOt2mn 6fwhpDlUQfWrO X-Received: by 2002:a5d:58e7:: with SMTP id f7mr6128985wrd.51.1630006850529; Thu, 26 Aug 2021 12:40:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwynqmTqvsqW9iFYuNy6lebFZRAvc8nScDmoxg+QC4b5TbglOriSiC/Vk1eFd/doAeMYyC5tA== X-Received: by 2002:a5d:58e7:: with SMTP id f7mr6128964wrd.51.1630006850387; Thu, 26 Aug 2021 12:40:50 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id h12sm3568113wmm.29.2021.08.26.12.40.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:50 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 14/27] bpf: Change bpf_trampoline_get to return error pointer Date: Thu, 26 Aug 2021 21:39:09 +0200 Message-Id: <20210826193922.66204-15-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Changing bpf_trampoline_get to return error pointer, so we can return other than ENOMEM error in following changes. Signed-off-by: Jiri Olsa --- kernel/bpf/syscall.c | 4 ++-- kernel/bpf/trampoline.c | 8 +++++--- kernel/bpf/verifier.c | 4 ++-- 3 files changed, 9 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index e667d392cc33..537687664bdf 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2793,8 +2793,8 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, goto out_unlock; tr = bpf_trampoline_get(key, &tgt_info); - if (!tr) { - err = -ENOMEM; + if (IS_ERR(tr)) { + err = PTR_ERR(tr); goto out_unlock; } } else { diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 8aa0aca38b3a..c9794e9f24ee 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -97,8 +97,10 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) goto out; } tr = kzalloc(sizeof(*tr), GFP_KERNEL); - if (!tr) + if (!tr) { + tr = ERR_PTR(-ENOMEM); goto out; + } bpf_trampoline_init(tr, key); head = &trampoline_table[hash_64(key, TRAMPOLINE_HASH_BITS)]; hlist_add_head(&tr->hlist, head); @@ -508,8 +510,8 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_trampoline *tr; tr = bpf_trampoline_lookup(key); - if (!tr) - return NULL; + if (IS_ERR(tr)) + return tr; mutex_lock(&tr->mutex); if (tr->func.addr) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e9e84adfb974..ad410e1222e4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13642,8 +13642,8 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) key = bpf_trampoline_compute_key(tgt_prog, prog->aux->attach_btf, btf_id); tr = bpf_trampoline_get(key, &tgt_info); - if (!tr) - return -ENOMEM; + if (IS_ERR(tr)) + return PTR_ERR(tr); prog->aux->dst_trampoline = tr; return 0; From patchwork Thu Aug 26 19:39:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460569 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EF7CC432BE for ; Thu, 26 Aug 2021 19:41:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 79F2060F11 for ; Thu, 26 Aug 2021 19:41:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243554AbhHZTlz (ORCPT ); Thu, 26 Aug 2021 15:41:55 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:43968 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243589AbhHZTlr (ORCPT ); Thu, 26 Aug 2021 15:41:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aB69aifL1zI8vrW5XNajMWuHmZ5MVDONIdA/rCm6EsU=; b=MM/yGe4J5Qw3TSVXWp93tQ5cWeXzJHyP3qr1xIjQim0RI3pSPUYJVIAXiP5+ACQiGfhoe9 KMTpMLdmbA31xVxnxtBs45KOBXu5+cgQOwaGPcK2KbLK8ojABH4MAcShBkBEJhlZMEXkiE DGH/OKcETCLoHJrHAUKavN5v+Alwgp8= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-440-QyQ-obSbOp6nhS7EXNwf5Q-1; Thu, 26 Aug 2021 15:40:58 -0400 X-MC-Unique: QyQ-obSbOp6nhS7EXNwf5Q-1 Received: by mail-wr1-f72.google.com with SMTP id h15-20020adff18f000000b001574654fbc2so1202058wro.10 for ; Thu, 26 Aug 2021 12:40:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aB69aifL1zI8vrW5XNajMWuHmZ5MVDONIdA/rCm6EsU=; b=H0mudDtb1QgcJ9i5Awfq49nZE1lE77G+ogDcRCLEWBJiW7xkvVu7io57VczK0WXa3W soETps0m9LzC+pQHrBU6kzeds2FXzbNxupTKO315sVLOo/g1qL4w+aCnaUTW+821cWYl m6Tn/s1kG7hxAnyBemGZJKxI970lrg2+KEGmImZFC/9TrRkcsnSXNTi+bjWtUw+XT1vi x13MAhlzwR4vTOXCbEHldmjj7Fo0eKBeioM9HODyL2hh7W7m0EJ2goee8wN1TF/Q3wBe K9yNLkEe3DWIxwioQgGmHgdzVeG5gIbU4bESvNTSgH+xyIAFbKhUXfl8T8vDHSqMvAbL tFeA== X-Gm-Message-State: AOAM533bW122qm4LF+fzLW90fy4ZPIw2QvFLUIPfG2bnEyS1DB7x/J4q MbG1G1eov09aTOrJhacpaXjgzTQkXCSSvN7kIyqEbup7PUtvejY2cZBoHudnupMh7deKTfSVKKM IJnXnOes1Ex5I X-Received: by 2002:a1c:27c2:: with SMTP id n185mr5120207wmn.20.1630006856789; Thu, 26 Aug 2021 12:40:56 -0700 (PDT) X-Google-Smtp-Source: ABdhPJximqozMr0/Jm9W8qaJJ+TOp3iu2uH+xdipTy7sLT+cli5O3p4fnfXEZvgGgFYa01EeiwtxAQ== X-Received: by 2002:a1c:27c2:: with SMTP id n185mr5120188wmn.20.1630006856586; Thu, 26 Aug 2021 12:40:56 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id x21sm2507342wmi.15.2021.08.26.12.40.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:40:56 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 15/27] bpf, x64: Allow to use caller address from stack Date: Thu, 26 Aug 2021 21:39:10 +0200 Message-Id: <20210826193922.66204-16-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Currently we call the original function by using the absolute address given at the JIT generation. That's not usable when having trampoline attached to multiple functions. In this case we need to take the return address from the stack. Adding support to retrieve the original function address from the stack by adding new BPF_TRAMP_F_ORIG_STACK flag for arch_prepare_bpf_trampoline function. Basically we take the return address of the 'fentry' call: function + 0: call fentry # stores 'function + 5' address on stack function + 5: ... The 'function + 5' address will be used as the address for the original function to call. Signed-off-by: Jiri Olsa --- arch/x86/net/bpf_jit_comp.c | 13 +++++++++---- include/linux/bpf.h | 5 +++++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 0fe6aacef3db..9f31197780ae 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2024,10 +2024,15 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i if (flags & BPF_TRAMP_F_CALL_ORIG) { restore_regs(m, &prog, nr_args, stack_size); - /* call original function */ - if (emit_call(&prog, orig_call, prog)) { - ret = -EINVAL; - goto cleanup; + if (flags & BPF_TRAMP_F_ORIG_STACK) { + emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, 8); + EMIT2(0xff, 0xd0); /* call *rax */ + } else { + /* call original function */ + if (emit_call(&prog, orig_call, prog)) { + ret = -EINVAL; + goto cleanup; + } } /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f0f548f8f391..a5c3307d49c6 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -584,6 +584,11 @@ struct btf_func_model { */ #define BPF_TRAMP_F_IP_ARG BIT(3) +/* Get original function from stack instead of from provided direct address. + * Makes sense for fexit programs only. + */ +#define BPF_TRAMP_F_ORIG_STACK BIT(4) + /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. Pick a number to fit into BPF_IMAGE_SIZE / 2 */ From patchwork Thu Aug 26 19:39:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460571 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFD56C432BE for ; Thu, 26 Aug 2021 19:41:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99B2B60F11 for ; Thu, 26 Aug 2021 19:41:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243594AbhHZTl6 (ORCPT ); Thu, 26 Aug 2021 15:41:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:51846 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243360AbhHZTlx (ORCPT ); Thu, 26 Aug 2021 15:41:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006865; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ISOz03pxm8/IDfssr2+wjRRzJVFbEDf7IJBjDKpD7YI=; b=DgLxdtcmc0UlerEG9mYQdlZSpFi+ppY1cJgl9IOxW3PfPwTywA8CACodrCMn2UnG3LEHIa B+stxRGQyEe+oMnJ0bMupdV83oQNfXnKIcmueuqSgK7dAF2SIsN4yfVUqgf5oOKVLwCZiU ErSx+WAswdemEJ3FvVh+k/RsUmWbaRc= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-563-iT5-9Mo7NqqP3ROjAUoILQ-1; Thu, 26 Aug 2021 15:41:04 -0400 X-MC-Unique: iT5-9Mo7NqqP3ROjAUoILQ-1 Received: by mail-wm1-f72.google.com with SMTP id n16-20020a1c7210000000b002ea2ed60dc6so240247wmc.0 for ; Thu, 26 Aug 2021 12:41:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ISOz03pxm8/IDfssr2+wjRRzJVFbEDf7IJBjDKpD7YI=; b=IYzASB+yXll/YU/VOIvWkgI66kSdaCldHbvZkBqTeJIJOLXVhKQRHRrTs1NjyvPDdF mx77ZkIO2Dk6YNrIUfrpAhwwbZm7Qydg7wKC12N7QFr/i3z+QuURf5TQ2jF1NZDib6S1 HSPkraJVEe4siWP0zq6XLioTMRPbKdP/uaMpMc086ddJqQ/ZrxWb/P19hjmahPhFzifw RXBC/pfHpAb/W5dLNvBaHxQQa0FvUaCN+TAo+XFl8j7tp8qRovY0CH5Qnxnmoxjdv0Iu 0a2znh948CcLR6Xe2Wz/NNMkYpZtWhgtHE3D08DGfiEMxBHcp+e7Kcw+BjvPDjM1yZGv VQHw== X-Gm-Message-State: AOAM533hGAgl/2rWeSBBFqA97or1u8mXFANY71OJooh9lixMBkBfmHOx nz/GamLryuUJ6ygyDpQHC8iUx15ad5/ah8g9t3TAjlVvJSl/T9HlwICG6OW0+QKTEsACMePwivb Cw0s6/sgNsg4Y X-Received: by 2002:a5d:6707:: with SMTP id o7mr6103977wru.307.1630006863210; Thu, 26 Aug 2021 12:41:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxuAcnoH9r3dNMdYGKPXN+YTRJdweWec0LDdFWr33uFOwzn9/YBcxwHxSUU5VsuJnmi3PlETA== X-Received: by 2002:a5d:6707:: with SMTP id o7mr6103956wru.307.1630006862972; Thu, 26 Aug 2021 12:41:02 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id j7sm9112722wmi.37.2021.08.26.12.41.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:02 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 16/27] bpf: Add bpf_trampoline_multi_get/put functions Date: Thu, 26 Aug 2021 21:39:11 +0200 Message-Id: <20210826193922.66204-17-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding struct bpf_trampoline_multi object and API to allocate and free it. The multi bpf trampoline is defined by BTF ids that represents functions that the trampoline will be attached to. By calling bpf_trampoline_multi_get you'll allocate new or get existing bpf_trampoline_multi object with following rules: - multi trampolines BTF ids can't intersect - multi trampoline can attach to functions that have standard program attached - standard programs can't attach to functions that have multi trampoline attached The multi trampoline contains pointers to all 'nested' standard trampolines and 'main' standard trampoline object (with key == 0) that represents the rest of the functions. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 15 ++++ kernel/bpf/trampoline.c | 179 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 194 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index a5c3307d49c6..678b9cd2fa21 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -691,6 +691,18 @@ struct bpf_trampoline { struct bpf_tramp_image *cur_image; u64 selector; struct module *mod; + struct { + struct bpf_trampoline *tr; + } multi; +}; + +struct bpf_trampoline_multi { + struct bpf_trampoline main; + struct list_head list; + u32 *ids; + u32 ids_cnt; + int tr_cnt; + struct bpf_trampoline *tr[]; }; struct bpf_attach_target_info { @@ -732,6 +744,9 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); +struct bpf_trampoline_multi *bpf_trampoline_multi_get(struct bpf_prog *prog, u32 *ids, + u32 ids_cnt); +void bpf_trampoline_multi_put(struct bpf_trampoline_multi *multi); #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \ diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index c9794e9f24ee..d66b76c23d74 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -10,6 +10,9 @@ #include #include #include +#include +#include +#include /* dummy _ops. The verifier will operate on target program's ops. */ const struct bpf_verifier_ops bpf_extension_verifier_ops = { @@ -22,6 +25,7 @@ const struct bpf_prog_ops bpf_extension_prog_ops = { #define TRAMPOLINE_TABLE_SIZE (1 << TRAMPOLINE_HASH_BITS) static struct hlist_head trampoline_table[TRAMPOLINE_TABLE_SIZE]; +static LIST_HEAD(trampoline_multi); /* serializes access to trampoline_table */ static DEFINE_MUTEX(trampoline_mutex); @@ -85,12 +89,41 @@ static struct bpf_trampoline *__bpf_trampoline_lookup(u64 key) return NULL; } +static int btf_ids_cmp(const void *a, const void *b) +{ + const u32 *x = a; + const u32 *y = b; + + if (*x == *y) + return 0; + return *x < *y ? -1 : 1; +} + +static bool is_in_multi(u64 key) +{ + struct bpf_trampoline_multi *multi; + u32 id; + + bpf_trampoline_unpack_key(key, NULL, &id); + + list_for_each_entry(multi, &trampoline_multi, list) { + if (bsearch(&id, multi->ids, multi->ids_cnt, + sizeof(u32), btf_ids_cmp)) + return true; + } + return false; +} + static struct bpf_trampoline *bpf_trampoline_lookup(u64 key) { struct bpf_trampoline *tr; struct hlist_head *head; mutex_lock(&trampoline_mutex); + if (is_in_multi(key)) { + tr = ERR_PTR(-EBUSY); + goto out; + } tr = __bpf_trampoline_lookup(key); if (tr) { refcount_inc(&tr->refcnt); @@ -553,6 +586,152 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) mutex_unlock(&trampoline_mutex); } +static void bpf_func_model_nargs(struct btf_func_model *m, int nr_args) +{ + int i; + + for (i = 0; i < nr_args; i++) + m->arg_size[i] = 8; + m->ret_size = 8; + m->nr_args = nr_args; +} + +static struct bpf_trampoline *lookup_trampoline(struct bpf_prog *prog, u32 id) +{ + u64 key = bpf_trampoline_compute_key(NULL, prog->aux->attach_btf, id); + + return __bpf_trampoline_lookup(key); +} + +struct bpf_trampoline_multi *bpf_trampoline_multi_get(struct bpf_prog *prog, + u32 *ids, u32 ids_cnt) +{ + int i, j, tr_cnt = 0, err = 0; + struct bpf_trampoline_multi *multi; + struct bpf_trampoline *tr; + u8 nr_args = 0; + size_t size; + + /* Sort user provided BTF ids, so we can use memcpy + * and bsearch below. + */ + sort(ids, ids_cnt, sizeof(u32), btf_ids_cmp, NULL); + + mutex_lock(&trampoline_mutex); + /* Check if the requested multi trampoline already exists. */ + list_for_each_entry(multi, &trampoline_multi, list) { + if (ids_cnt == multi->ids_cnt && !memcmp(ids, multi->ids, ids_cnt)) { + refcount_inc(&multi->main.refcnt); + kfree(ids); + goto out; + } + for (i = 0; i < ids_cnt; i++) { + if (bsearch(&ids[i], multi->ids, multi->ids_cnt, + sizeof(u32), btf_ids_cmp)) { + multi = ERR_PTR(-EINVAL); + goto out; + } + } + } + + /* Check if any of the requested functions have already standard + * trampoline attached. + */ + for (i = 0; i < ids_cnt; i++) { + tr = lookup_trampoline(prog, ids[i]); + if (!tr) + continue; + if (tr->multi.tr) { + multi = ERR_PTR(-EBUSY); + goto out; + } + tr_cnt++; + } + + /* Create new multi trampoline ... */ + size = sizeof(*multi) + tr_cnt * sizeof(multi->tr[0]); + multi = kzalloc(size, GFP_KERNEL); + if (!multi) { + multi = ERR_PTR(-ENOMEM); + goto out; + } + + bpf_trampoline_init(&multi->main, 0); + multi->tr_cnt = tr_cnt; + multi->ids = ids; + multi->ids_cnt = ids_cnt; + list_add_tail(&multi->list, &trampoline_multi); + + for (i = 0; i < ids_cnt; i++) { + struct bpf_attach_target_info tgt_info = {}; + + tr = lookup_trampoline(prog, ids[i]); + if (tr) + continue; + + err = bpf_check_attach_target(NULL, prog, NULL, ids[i], &tgt_info); + if (err) + goto out_free; + + err = -EINVAL; + if (!is_ftrace_location((void *) tgt_info.tgt_addr)) + goto out_free; + + if (nr_args < tgt_info.fmodel.nr_args) + nr_args = tgt_info.fmodel.nr_args; + } + + bpf_func_model_nargs(&multi->main.func.model, nr_args); + + /* ... and attach already existing standard trampolines. */ + for (i = 0, j = 0; i < ids_cnt && j < tr_cnt; i++) { + tr = lookup_trampoline(prog, ids[i]); + if (tr) { + refcount_inc(&tr->refcnt); + tr->multi.tr = &multi->main; + multi->tr[j++] = tr; + } + } + +out_free: + if (err) { + list_del(&multi->list); + kfree(multi); + multi = ERR_PTR(err); + } +out: + mutex_unlock(&trampoline_mutex); + return multi; +} + +void bpf_trampoline_multi_put(struct bpf_trampoline_multi *multi) +{ + int i; + + if (!multi) + return; + + mutex_lock(&trampoline_mutex); + if (!refcount_dec_and_test(&multi->main.refcnt)) + goto out; + + if (WARN_ON_ONCE(!hlist_empty(&multi->main.progs_hlist[BPF_TRAMP_FENTRY]))) + goto out; + if (WARN_ON_ONCE(!hlist_empty(&multi->main.progs_hlist[BPF_TRAMP_FEXIT]))) + goto out; + + list_del(&multi->list); + + for (i = 0; i < multi->tr_cnt; i++) { + multi->tr[i]->multi.tr = NULL; + __bpf_trampoline_put(multi->tr[i]); + } + kfree(multi->ids); + kfree(multi); +out: + mutex_unlock(&trampoline_mutex); +} + #define NO_START_TIME 1 static u64 notrace bpf_prog_start_time(void) { From patchwork Thu Aug 26 19:39:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460573 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01A44C432BE for ; Thu, 26 Aug 2021 19:41:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF00660F4C for ; Thu, 26 Aug 2021 19:41:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243608AbhHZTmF (ORCPT ); Thu, 26 Aug 2021 15:42:05 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44824 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243605AbhHZTmA (ORCPT ); Thu, 26 Aug 2021 15:42:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k7Y1Ok4cyox0DsYgmFPhX/i9rcG3AZ5Q61TPBm3wVYA=; b=bGk3jkt8mcTUrsG/oL/7yHE0xkKK9P1/wjvKMEOZKmNmluwDvQCNkkKZhmsxwCndTslaD+ qOdOqkBLG8I1hp8l20FyF8d0/U27DHne4TIvh8OcYBFELxxai5wZmWl1Tq/99cm1jzKlMK 1SgvBb1Fu9mUI02BGkwRTUcDO8aCBDs= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-502-jX11GqZPM8Gn3s1V6l4KVg-1; Thu, 26 Aug 2021 15:41:11 -0400 X-MC-Unique: jX11GqZPM8Gn3s1V6l4KVg-1 Received: by mail-wr1-f69.google.com with SMTP id q11-20020a5d61cb0000b02901550c3fccb5so1193728wrv.14 for ; Thu, 26 Aug 2021 12:41:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=k7Y1Ok4cyox0DsYgmFPhX/i9rcG3AZ5Q61TPBm3wVYA=; b=mlHiewsIvgajVw4QR0yvV7XKhYKqVKFkq3lFRnNv/iszDhC8JE2tng5oy9KdCXdg5O ntjKFPATW+3xH5abNUzVD3OfD8oKPeB145KFl6qeV/RweAB5awPQgiI0J/XM8X6tfmyX vULcbuXIblxYRHfiBKvsqBTErFC01mhzMmUVbbSlFOiS9C0ZIyqFWepbbFWWIhtork/k sIXtLhk0mBeOKpMfl0E8cbdi0ZA3od9WoMWnA53Usf46j/+jDdcVHcKA+i+Wn3Z863ay IQlumjAF0dUge+c2YIxUlGLoOZ610Ul+fVAwGILCHtNlGqu0pAAbzFk1icBeeuDukfBx okRQ== X-Gm-Message-State: AOAM530zchG0xTjmmjhuLQoUjcoq8PtSlUXFWkGGivLL6d1ZwbW1SsbK oRpOug6jsYgfOqEbZXDa6HtlTeK3cjhsJZCd8z21ddmMEBSGNEzfx1R3ogDSZLdJhPIrvlCAbU+ p1GBNBe7kPy/5 X-Received: by 2002:adf:fb8f:: with SMTP id a15mr6055951wrr.92.1630006869551; Thu, 26 Aug 2021 12:41:09 -0700 (PDT) X-Google-Smtp-Source: ABdhPJykopySwjKrdyJqFYHgjK0fph8uXk7OdmrjR6/62gjo9cD7ouKp4mlKRx/OXYmQv49Cpaib6w== X-Received: by 2002:adf:fb8f:: with SMTP id a15mr6055922wrr.92.1630006869317; Thu, 26 Aug 2021 12:41:09 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id p9sm9445594wmq.40.2021.08.26.12.41.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:09 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 17/27] bpf: Add multi trampoline attach support Date: Thu, 26 Aug 2021 21:39:12 +0200 Message-Id: <20210826193922.66204-18-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding new multi trampoline link (BPF_LINK_TYPE_TRACING_MULTI) as an interface to attach program to multiple functions. The link_create bpf_attr interface already has 'bpf_prog' file descriptor, that defines the program to be attached. It must be loaded with BPF_F_MULTI_FUNC flag. Adding new multi_btf_ids/multi_btf_ids_cnt link_create bpf_attr fields that provides BTF ids. The new link gets multi trampoline (via bpf_trampoline_multi_get) and links the provided program with embedded trampolines and the 'main' trampoline with new multi link/unlink functions: int bpf_trampoline_multi_link_prog(struct bpf_prog *prog, struct bpf_trampoline_multi *tr); int bpf_trampoline_multi_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline_multi *tr); If embedded trampoline contains fexit programs, we need to switch its model to the multi trampoline model (because of the final 'ret' argument). We keep the count of attached multi func programs for each trampoline, so we can tell when to switch the model. Signed-off-by: Jiri Olsa --- include/linux/bpf.h | 5 ++ include/uapi/linux/bpf.h | 5 ++ kernel/bpf/core.c | 1 + kernel/bpf/syscall.c | 120 +++++++++++++++++++++++++++++++++ kernel/bpf/trampoline.c | 87 ++++++++++++++++++++++-- tools/include/uapi/linux/bpf.h | 5 ++ 6 files changed, 219 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 678b9cd2fa21..3ce4656e2057 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -693,6 +693,7 @@ struct bpf_trampoline { struct module *mod; struct { struct bpf_trampoline *tr; + int count; } multi; }; @@ -747,6 +748,8 @@ void bpf_trampoline_put(struct bpf_trampoline *tr); struct bpf_trampoline_multi *bpf_trampoline_multi_get(struct bpf_prog *prog, u32 *ids, u32 ids_cnt); void bpf_trampoline_multi_put(struct bpf_trampoline_multi *multi); +int bpf_trampoline_multi_link_prog(struct bpf_prog *prog, struct bpf_trampoline_multi *tr); +int bpf_trampoline_multi_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline_multi *tr); #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \ @@ -892,6 +895,8 @@ struct bpf_prog_aux { bool tail_call_reachable; bool multi_func; struct bpf_tramp_node tramp_node; + struct bpf_tramp_node *multi_node; + struct mutex multi_node_mutex; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 1f9d336861f0..9533200ffadf 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -1008,6 +1008,7 @@ enum bpf_link_type { BPF_LINK_TYPE_NETNS = 5, BPF_LINK_TYPE_XDP = 6, BPF_LINK_TYPE_PERF_EVENT = 7, + BPF_LINK_TYPE_TRACING_MULTI = 8, MAX_BPF_LINK_TYPE, }; @@ -1462,6 +1463,10 @@ union bpf_attr { */ __u64 bpf_cookie; } perf_event; + struct { + __aligned_u64 multi_btf_ids; /* addresses to attach */ + __u32 multi_btf_ids_cnt; /* addresses count */ + }; }; } link_create; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index bad03dde97a2..6c16ac43dd91 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -109,6 +109,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); mutex_init(&fp->aux->used_maps_mutex); mutex_init(&fp->aux->dst_mutex); + mutex_init(&fp->aux->multi_node_mutex); return fp; } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 537687664bdf..8f1f934a8f26 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -32,6 +32,7 @@ #include #include #include +#include #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \ (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \ @@ -2851,6 +2852,121 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, return err; } +struct bpf_tracing_multi_link { + struct bpf_link link; + enum bpf_attach_type attach_type; + struct bpf_trampoline_multi *multi; +}; + +static void bpf_tracing_multi_link_release(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + bpf_trampoline_multi_unlink_prog(link->prog, tr_link->multi); +} + +static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + bpf_trampoline_multi_put(tr_link->multi); + kfree(tr_link); +} + +static void bpf_tracing_multi_link_show_fdinfo(const struct bpf_link *link, + struct seq_file *seq) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + seq_printf(seq, "attach_type:\t%d\n", tr_link->attach_type); +} + +static int bpf_tracing_multi_link_fill_link_info(const struct bpf_link *link, + struct bpf_link_info *info) +{ + struct bpf_tracing_multi_link *tr_link = + container_of(link, struct bpf_tracing_multi_link, link); + + info->tracing.attach_type = tr_link->attach_type; + return 0; +} + +static int check_multi_prog_type(struct bpf_prog *prog) +{ + if (prog->expected_attach_type != BPF_TRACE_FENTRY && + prog->expected_attach_type != BPF_TRACE_FEXIT) + return -EINVAL; + return 0; +} + +static const struct bpf_link_ops bpf_tracing_multi_link_lops = { + .release = bpf_tracing_multi_link_release, + .dealloc = bpf_tracing_multi_link_dealloc, + .show_fdinfo = bpf_tracing_multi_link_show_fdinfo, + .fill_link_info = bpf_tracing_multi_link_fill_link_info, +}; + +static int bpf_tracing_multi_attach(struct bpf_prog *prog, + const union bpf_attr *attr) +{ + void __user *ubtf_ids = u64_to_user_ptr(attr->link_create.multi_btf_ids); + u32 size, cnt = attr->link_create.multi_btf_ids_cnt; + struct bpf_tracing_multi_link *link = NULL; + struct bpf_link_primer link_primer; + struct bpf_trampoline_multi *multi = NULL; + int err = -EINVAL; + u32 *btf_ids; + + if (check_multi_prog_type(prog)) + return -EINVAL; + if (!cnt) + return -EINVAL; + + size = cnt * sizeof(*btf_ids); + btf_ids = kmalloc(size, GFP_USER | __GFP_NOWARN); + if (!btf_ids) + return -ENOMEM; + + err = -EFAULT; + if (ubtf_ids && copy_from_user(btf_ids, ubtf_ids, size)) + goto out_free_ids; + + multi = bpf_trampoline_multi_get(prog, btf_ids, cnt); + if (IS_ERR(multi)) { + err = PTR_ERR(multi); + goto out_free_ids; + } + + link = kzalloc(sizeof(*link), GFP_USER); + if (!link) { + err = -ENOMEM; + goto out_free; + } + + bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI, + &bpf_tracing_multi_link_lops, prog); + link->attach_type = prog->expected_attach_type; + link->multi = multi; + + err = bpf_link_prime(&link->link, &link_primer); + if (err) + goto out_free; + err = bpf_trampoline_multi_link_prog(prog, multi); + if (err) + goto out_free; + return bpf_link_settle(&link_primer); + +out_free: + bpf_trampoline_multi_put(multi); + kfree(link); +out_free_ids: + kfree(btf_ids); + return err; +} + struct bpf_raw_tp_link { struct bpf_link link; struct bpf_raw_event_map *btp; @@ -3157,6 +3273,8 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) case BPF_CGROUP_SETSOCKOPT: return BPF_PROG_TYPE_CGROUP_SOCKOPT; case BPF_TRACE_ITER: + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: return BPF_PROG_TYPE_TRACING; case BPF_SK_LOOKUP: return BPF_PROG_TYPE_SK_LOOKUP; @@ -4213,6 +4331,8 @@ static int tracing_bpf_link_attach(const union bpf_attr *attr, bpfptr_t uattr, if (prog->expected_attach_type == BPF_TRACE_ITER) return bpf_iter_link_attach(attr, uattr, prog); + else if (prog->aux->multi_func) + return bpf_tracing_multi_attach(prog, attr); else if (prog->type == BPF_PROG_TYPE_EXT) return bpf_tracing_prog_attach(prog, attr->link_create.target_fd, diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index d66b76c23d74..6ff5c2512f91 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -30,6 +30,11 @@ static LIST_HEAD(trampoline_multi); /* serializes access to trampoline_table */ static DEFINE_MUTEX(trampoline_mutex); +static bool is_multi_trampoline(struct bpf_trampoline *tr) +{ + return tr->key == 0; +} + void *bpf_jit_alloc_exec_page(void) { void *image; @@ -384,8 +389,21 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key, u32 idx) return ERR_PTR(err); } +static bool needs_multi_model(struct bpf_trampoline *tr, struct btf_func_model *new) +{ + struct bpf_trampoline *multi = tr->multi.tr; + + if (!tr->multi.count || !multi) + return false; + if (tr->func.model.nr_args >= multi->func.model.nr_args) + return false; + memcpy(new, &multi->func.model, sizeof(*new)); + return true; +} + static int bpf_trampoline_update(struct bpf_trampoline *tr) { + struct btf_func_model model_multi, *model = &tr->func.model; struct bpf_tramp_image *im; struct bpf_tramp_progs *tprogs; u32 flags = BPF_TRAMP_F_RESTORE_REGS; @@ -411,15 +429,19 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) } if (tprogs[BPF_TRAMP_FEXIT].nr_progs || - tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs) + tprogs[BPF_TRAMP_MODIFY_RETURN].nr_progs) { flags = BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_SKIP_FRAME; + if (is_multi_trampoline(tr)) + flags |= BPF_TRAMP_F_ORIG_STACK; + if (needs_multi_model(tr, &model_multi)) + model = &model_multi; + } if (ip_arg) flags |= BPF_TRAMP_F_IP_ARG; err = arch_prepare_bpf_trampoline(im, im->image, im->image + PAGE_SIZE, - &tr->func.model, flags, tprogs, - tr->func.addr); + model, flags, tprogs, tr->func.addr); if (err < 0) goto out; @@ -507,7 +529,8 @@ int bpf_trampoline_link_prog(struct bpf_tramp_node *node, struct bpf_trampoline if (err) { hlist_del_init(&node->hlist); tr->progs_cnt[kind]--; - } + } else if (prog->aux->multi_func) + tr->multi.count++; out: mutex_unlock(&tr->mutex); return err; @@ -531,6 +554,8 @@ int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, struct bpf_trampolin } hlist_del_init(&node->hlist); tr->progs_cnt[kind]--; + if (prog->aux->multi_func) + tr->multi.count--; err = bpf_trampoline_update(tr); out: mutex_unlock(&tr->mutex); @@ -732,6 +757,60 @@ void bpf_trampoline_multi_put(struct bpf_trampoline_multi *multi) mutex_unlock(&trampoline_mutex); } +int bpf_trampoline_multi_link_prog(struct bpf_prog *prog, struct bpf_trampoline_multi *multi) +{ + struct bpf_tramp_node *multi_node = NULL; + int i, j, err = 0; + + multi_node = kzalloc(sizeof(*multi_node) * multi->tr_cnt, GFP_KERNEL); + if (!multi_node) + return -ENOMEM; + + mutex_lock(&prog->aux->multi_node_mutex); + if (prog->aux->multi_node) + err = -EBUSY; + else + prog->aux->multi_node = multi_node; + mutex_unlock(&prog->aux->multi_node_mutex); + if (err) + goto out_free; + + for (i = 0; i < multi->tr_cnt; i++) { + multi_node[i].prog = prog; + err = bpf_trampoline_link_prog(&multi_node[i], multi->tr[i]); + if (err) + goto out_unlink; + } + + err = bpf_trampoline_link_prog(&prog->aux->tramp_node, &multi->main); + if (!err) + return 0; + +out_unlink: + for (j = 0; j < i; j++) + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&multi_node[j], multi->tr[j])); + +out_free: + kfree(multi_node); + return err; +} + +int bpf_trampoline_multi_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline_multi *multi) +{ + struct bpf_tramp_node *multi_node = prog->aux->multi_node; + int i; + + for (i = 0; i < multi->tr_cnt; i++) + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&multi_node[i], multi->tr[i])); + + mutex_lock(&prog->aux->multi_node_mutex); + prog->aux->multi_node = NULL; + mutex_unlock(&prog->aux->multi_node_mutex); + + kfree(multi_node); + return bpf_trampoline_unlink_prog(&prog->aux->tramp_node, &multi->main); +} + #define NO_START_TIME 1 static u64 notrace bpf_prog_start_time(void) { diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 1f9d336861f0..9533200ffadf 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -1008,6 +1008,7 @@ enum bpf_link_type { BPF_LINK_TYPE_NETNS = 5, BPF_LINK_TYPE_XDP = 6, BPF_LINK_TYPE_PERF_EVENT = 7, + BPF_LINK_TYPE_TRACING_MULTI = 8, MAX_BPF_LINK_TYPE, }; @@ -1462,6 +1463,10 @@ union bpf_attr { */ __u64 bpf_cookie; } perf_event; + struct { + __aligned_u64 multi_btf_ids; /* addresses to attach */ + __u32 multi_btf_ids_cnt; /* addresses count */ + }; }; } link_create; From patchwork Thu Aug 26 19:39:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460575 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2008BC432BE for ; Thu, 26 Aug 2021 19:41:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 09E4B60FDA for ; Thu, 26 Aug 2021 19:41:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243356AbhHZTmH (ORCPT ); Thu, 26 Aug 2021 15:42:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:22335 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243485AbhHZTmG (ORCPT ); Thu, 26 Aug 2021 15:42:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IZxI6wOoaWT81ALa8QkvdxqAQKY5lKnGUs6cJrU25Fs=; b=RRgn1bKlERf8XUV/0EPJaSSTK9ZiwOWppz0HKpv5Vg6SEw4hd7aEJM3kA5B661quNMCPiW jPV/4wJg6bMs08qknVPmPQ4/9erUF75egxAJA3/6nv1J8XnlWoJX3sdXZ88aWmmGSuCDPb StP2/y95AhwQmdoqwLdLh4r9Kweaucw= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-563-ZTiVIGMuNnOq-jEMHPesCg-1; Thu, 26 Aug 2021 15:41:17 -0400 X-MC-Unique: ZTiVIGMuNnOq-jEMHPesCg-1 Received: by mail-wr1-f72.google.com with SMTP id b7-20020a5d4d87000000b001575d1053d2so1180704wru.23 for ; Thu, 26 Aug 2021 12:41:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IZxI6wOoaWT81ALa8QkvdxqAQKY5lKnGUs6cJrU25Fs=; b=FlOEonjOs+3ProEJJXJanv9DRajuCoJgmi+4G1l3phr1A1GaByjvOt+1vEnaFxHrgV eGs8sXPilMCLKX26gRwSxzLCFwzcOw2jtrR1Ob0MLZHlUjvrd927NpVVAZEdR9kufgKM Li3U+8E5q3qEymIokZ5rjMDxeTDfHnHetiVHtpN2eXos6h2/hGFoIlXcmEQhDIOQ3Yve gVme53EXPqSSiE4/SwvVdWFm3LRcVTYT9CYcxR8o8ryIa6OGWjWq/qheaQcr072yy2vx eL8QcvXzSGdldq64RX4+s69BNmZydaB+zqfyKEJ8vW3uoHQEDGL7t8ymkt+HpqyuewbC C4KQ== X-Gm-Message-State: AOAM53387xncoqCgjYd+Y4G/ge97pGR9sL1dgE9EARWbCbEv9VYak2Dg F6D2fdC6AKnnkP4Ed/U4B3pn9tbK+c519tQBXbeeKRao13vO9l+uxxjlWjXssUMQ4mg6AjoJeur UWbtkLiaeDIMk X-Received: by 2002:adf:9f0d:: with SMTP id l13mr848694wrf.328.1630006875831; Thu, 26 Aug 2021 12:41:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwo67KkXMdAPOOGMrz24uHf7y9/tWyscdaPfYgn96pwyBfYz/iJlHc46O9G4R2MLdO7VZ/buQ== X-Received: by 2002:adf:9f0d:: with SMTP id l13mr848672wrf.328.1630006875603; Thu, 26 Aug 2021 12:41:15 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id n14sm4018343wrx.10.2021.08.26.12.41.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:15 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 18/27] bpf, x64: Store properly return value for trampoline with multi func programs Date: Thu, 26 Aug 2021 21:39:13 +0200 Message-Id: <20210826193922.66204-19-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net When we have multi func program attached, the trampoline switched to the function model of the multi func program. This breaks already attached standard programs, for example when we attach following program: SEC("fexit/bpf_fentry_test2") int BPF_PROG(test1, int a, __u64 b, int ret) the trampoline pushes on stack args 'a' and 'b' and return value 'ret'. When following multi func program is attached to bpf_fentry_test2: SEC("fexit.multi/bpf_fentry_test*") int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) the trampoline takes this program model and pushes all 6 args and return value on stack. But we still have the original 'test1' program attached, that expects 'ret' value where there's 'c' argument now: test1(a, b, c) To fix that we simply overwrite 'c' argument with 'ret' value, so test1 is called as expected and test2 gets called as: test2(a, b, ret, d, e, f, ret) which is ok, because 'c' is not defined for bpf_fentry_test2 anyway. Signed-off-by: Jiri Olsa --- arch/x86/net/bpf_jit_comp.c | 40 ++++++++++++++++++++++++++++++------- include/linux/bpf.h | 1 + kernel/bpf/trampoline.c | 1 + 3 files changed, 35 insertions(+), 7 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 9f31197780ae..3f7911a92d62 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -1744,7 +1744,7 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, int nr_args, } static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, - struct bpf_prog *p, int stack_size, bool mod_ret) + struct bpf_prog *p, int stack_size, bool mod_ret, int args_off) { u8 *prog = *pprog; u8 *jmp_insn; @@ -1780,9 +1780,14 @@ static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, /* BPF_TRAMP_MODIFY_RETURN trampolines can modify the return * of the previous call which is then passed on the stack to * the next BPF program. + * Store the return value also to original args' end in case + * we have multi func programs in trampoline. */ - if (mod_ret) + if (mod_ret) { emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + if (args_off) + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -args_off); + } /* replace 2 nops with JE insn, since jmp target is known */ jmp_insn[0] = X86_JE; @@ -1834,7 +1839,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, u8 *prog = *pprog; for (i = 0; i < tp->nr_progs; i++) { - if (invoke_bpf_prog(m, &prog, tp->progs[i], stack_size, false)) + if (invoke_bpf_prog(m, &prog, tp->progs[i], stack_size, false, 0)) return -EINVAL; } *pprog = prog; @@ -1843,7 +1848,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, struct bpf_tramp_progs *tp, int stack_size, - u8 **branches) + u8 **branches, int args_off) { u8 *prog = *pprog; int i; @@ -1853,8 +1858,15 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, */ emit_mov_imm32(&prog, false, BPF_REG_0, 0); emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + + /* Store the return value also to original args' end in case + * we have multi func programs in trampoline. + */ + if (args_off) + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -args_off); + for (i = 0; i < tp->nr_progs; i++) { - if (invoke_bpf_prog(m, &prog, tp->progs[i], stack_size, true)) + if (invoke_bpf_prog(m, &prog, tp->progs[i], stack_size, true, args_off)) return -EINVAL; /* mod_ret prog stored return value into [rbp - 8]. Emit: @@ -1942,7 +1954,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i struct bpf_tramp_progs *tprogs, void *orig_call) { - int ret, i, nr_args = m->nr_args; + int ret, i, args_off = 0, nr_args = m->nr_args; int stack_size = nr_args * 8; struct bpf_tramp_progs *fentry = &tprogs[BPF_TRAMP_FENTRY]; struct bpf_tramp_progs *fexit = &tprogs[BPF_TRAMP_FEXIT]; @@ -1958,6 +1970,13 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i (flags & BPF_TRAMP_F_SKIP_FRAME)) return -EINVAL; + /* if m->nr_args_orig != 0, then we have multi prog model and + * we need to also store return value at the end of standard + * trampoline's arguments + */ + if (m->nr_args_orig && m->nr_args > m->nr_args_orig) + args_off = (m->nr_args - m->nr_args_orig) * 8 + 8; + if (flags & BPF_TRAMP_F_CALL_ORIG) stack_size += 8; /* room for return value of orig_call */ @@ -2015,7 +2034,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i return -ENOMEM; if (invoke_bpf_mod_ret(m, &prog, fmod_ret, stack_size, - branches)) { + branches, args_off)) { ret = -EINVAL; goto cleanup; } @@ -2036,6 +2055,13 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i } /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + + /* store return value also to original args' end in case we have + * multi func programs in trampoline + */ + if (args_off) + emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -args_off); + im->ip_after_call = prog; memcpy(prog, x86_nops[5], X86_PATCH_SIZE); prog += X86_PATCH_SIZE; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 3ce4656e2057..373f45ae7dce 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -563,6 +563,7 @@ struct btf_func_model { u8 ret_size; u8 nr_args; u8 arg_size[MAX_BPF_FUNC_ARGS]; + u8 nr_args_orig; }; /* Restore arguments before returning from trampoline to let original function diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 6ff5c2512f91..308d58e698be 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -398,6 +398,7 @@ static bool needs_multi_model(struct bpf_trampoline *tr, struct btf_func_model * if (tr->func.model.nr_args >= multi->func.model.nr_args) return false; memcpy(new, &multi->func.model, sizeof(*new)); + new->nr_args_orig = tr->func.model.nr_args; return true; } From patchwork Thu Aug 26 19:39:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460577 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87FC7C432BE for ; Thu, 26 Aug 2021 19:41:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7077D60F11 for ; Thu, 26 Aug 2021 19:41:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243607AbhHZTmO (ORCPT ); Thu, 26 Aug 2021 15:42:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:50126 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243611AbhHZTmN (ORCPT ); Thu, 26 Aug 2021 15:42:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/wdRRB3BbU0zTebKEzNMBHGnZzjSnqwp9yN08+1iPjU=; b=XTCAJin1neNHFLzJvlySZ/iYT40ae8hFSZJkZURFEHf6vx9IVA08Yk6gXyNzsdp/hXFjPf HS8+ckPLzzqvCR7Tsjsm/usVX0QU5nEvkHHHvsslnZnLPq3jFoqeV1zaxmONl5KIn25uRj W8StGtpN6vLep/qQ8VemdSyCFsAKgFU= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-496-2Jc8iAuTONSSkLalopdd8g-1; Thu, 26 Aug 2021 15:41:24 -0400 X-MC-Unique: 2Jc8iAuTONSSkLalopdd8g-1 Received: by mail-wm1-f72.google.com with SMTP id r4-20020a1c4404000000b002e728beb9fbso4752910wma.9 for ; Thu, 26 Aug 2021 12:41:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/wdRRB3BbU0zTebKEzNMBHGnZzjSnqwp9yN08+1iPjU=; b=L8xPM99zVBcZFs9cp0z1ozuMhcENb0in52LAULTcK9CFB/toSev+8MaLLLQ/VX6ErL Dr7GqKNeXE93NvPYdiWsrU0sc501GjYorxUBVcVYPEFnqS6uDiG+fVNpoyHhSzWciERs Xsneuo97G04WY6wfKFedGfPGJfrQgxoNp7tmhds7dXLhHHupsKpNo0K7GobMQFapdMQ7 lZbdC9RMjddbM9EG97UzsS5QzbA76l2x6e/bk2CyjB4hdTjdrrBiyHRZtYljNcaAonkQ WsEyhqBnmuadmARD9yZRqCscrwUnX2muD376grrmmCswASQe8+X05KXdCLaFkiFNJnF+ DmdA== X-Gm-Message-State: AOAM5309cc4Vb0JKfR3lcFOwOYnEK9N1pkAqSbGTkjMNqqe82ncmwbSW 4BfRv33IrC+bl4yiVRTteR1TZay/rLTjqwmuCpMvX3dCk9RcLmjPv6ELVSOlQobETUaorYEUkco iWZt9j3lfK1x1 X-Received: by 2002:a05:6000:1808:: with SMTP id m8mr5842525wrh.272.1630006882351; Thu, 26 Aug 2021 12:41:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzhi28xOo8akRuWYI1q+XExG6E1iNF4UXfPNErBo+zAHkd2rbEd372edPEF5IYB7n7bBmLlLA== X-Received: by 2002:a05:6000:1808:: with SMTP id m8mr5842503wrh.272.1630006882147; Thu, 26 Aug 2021 12:41:22 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id l21sm3356742wmh.31.2021.08.26.12.41.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:21 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 19/27] bpf: Attach multi trampoline with ftrace_ops Date: Thu, 26 Aug 2021 21:39:14 +0200 Message-Id: <20210826193922.66204-20-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding struct ftrace_ops object to bpf_trampoline_multi struct and setting it up with all the requested function addresses. Adding is_multi_trampoline(tr) hooks to installing functions to actually install multiple bpf trampoline via ftrace_ops. I had to add -DCC_USING_FENTRY to several places because arch/x86/include/asm/ftrace.h would fail the compilation if it's not defined. Perhaps there's a better way. Signed-off-by: Jiri Olsa --- arch/x86/Makefile | 7 +++++ arch/x86/boot/compressed/Makefile | 4 +++ drivers/firmware/efi/libstub/Makefile | 3 +++ include/linux/bpf.h | 2 ++ kernel/bpf/trampoline.c | 37 +++++++++++++++++++++++++++ 5 files changed, 53 insertions(+) diff --git a/arch/x86/Makefile b/arch/x86/Makefile index 307fd0000a83..72986eb38d0b 100644 --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -34,6 +34,9 @@ REALMODE_CFLAGS += -fno-stack-protector REALMODE_CFLAGS += $(call __cc-option, $(CC), $(REALMODE_CFLAGS), -Wno-address-of-packed-member) REALMODE_CFLAGS += $(call __cc-option, $(CC), $(REALMODE_CFLAGS), $(cc_stack_align4)) REALMODE_CFLAGS += $(CLANG_FLAGS) +ifdef CONFIG_DYNAMIC_FTRACE + REALMODE_CFLAGS += -DCC_USING_FENTRY +endif export REALMODE_CFLAGS # BITS is used as extension for files which are available in a 32 bit @@ -54,6 +57,10 @@ KBUILD_CFLAGS += $(call cc-option,-mno-avx,) # Intel CET isn't enabled in the kernel KBUILD_CFLAGS += $(call cc-option,-fcf-protection=none) +ifdef CONFIG_DYNAMIC_FTRACE + KBUILD_CFLAGS += -DCC_USING_FENTRY +endif + ifeq ($(CONFIG_X86_32),y) BITS := 32 UTS_MACHINE := i386 diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile index 431bf7f846c3..a93dcbda4de2 100644 --- a/arch/x86/boot/compressed/Makefile +++ b/arch/x86/boot/compressed/Makefile @@ -49,6 +49,10 @@ KBUILD_CFLAGS += $(call as-option,-Wa$(comma)-mrelax-relocations=no) KBUILD_CFLAGS += -include $(srctree)/include/linux/hidden.h KBUILD_CFLAGS += $(CLANG_FLAGS) +ifdef CONFIG_DYNAMIC_FTRACE + KBUILD_CFLAGS += -DCC_USING_FENTRY +endif + # sev.c indirectly inludes inat-table.h which is generated during # compilation and stored in $(objtree). Add the directory to the includes so # that the compiler finds it even with out-of-tree builds (make O=/some/path). diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index d0537573501e..2fd5d6f55e24 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -15,6 +15,9 @@ cflags-$(CONFIG_X86) += -m$(BITS) -D__KERNEL__ \ $(call cc-disable-warning, gnu) \ -fno-asynchronous-unwind-tables \ $(CLANG_FLAGS) +ifdef CONFIG_DYNAMIC_FTRACE + cflags-$(CONFIG_X86) += -DCC_USING_FENTRY +endif # arm64 uses the full KBUILD_CFLAGS so it's necessary to explicitly # disable the stackleak plugin diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 373f45ae7dce..52cbec23665c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -23,6 +23,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -703,6 +704,7 @@ struct bpf_trampoline_multi { struct list_head list; u32 *ids; u32 ids_cnt; + struct ftrace_ops ops; int tr_cnt; struct bpf_trampoline *tr[]; }; diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 308d58e698be..82e7545a7426 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -179,11 +179,38 @@ static int is_ftrace_location(void *ip) return 1; } +static int register_ftrace_multi(struct bpf_trampoline *tr, void *new_addr) +{ + struct bpf_trampoline_multi *multi; + + multi = container_of(tr, struct bpf_trampoline_multi, main); + return register_ftrace_direct_multi(&multi->ops, (long)new_addr); +} + +static int unregister_ftrace_multi(struct bpf_trampoline *tr, void *old_addr) +{ + struct bpf_trampoline_multi *multi; + + multi = container_of(tr, struct bpf_trampoline_multi, main); + return unregister_ftrace_direct_multi(&multi->ops); +} + +static int modify_ftrace_multi(struct bpf_trampoline *tr, void *new_addr) +{ + struct bpf_trampoline_multi *multi; + + multi = container_of(tr, struct bpf_trampoline_multi, main); + return modify_ftrace_direct_multi(&multi->ops, (long)new_addr); +} + static int unregister_fentry(struct bpf_trampoline *tr, void *old_addr) { void *ip = tr->func.addr; int ret; + if (is_multi_trampoline(tr)) + return unregister_ftrace_multi(tr, old_addr); + if (tr->func.ftrace_managed) ret = unregister_ftrace_direct((long)ip, (long)old_addr); else @@ -199,6 +226,9 @@ static int modify_fentry(struct bpf_trampoline *tr, void *old_addr, void *new_ad void *ip = tr->func.addr; int ret; + if (is_multi_trampoline(tr)) + return modify_ftrace_multi(tr, new_addr); + if (tr->func.ftrace_managed) ret = modify_ftrace_direct((long)ip, (long)old_addr, (long)new_addr); else @@ -212,6 +242,9 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) void *ip = tr->func.addr; int ret; + if (is_multi_trampoline(tr)) + return register_ftrace_multi(tr, new_addr); + ret = is_ftrace_location(ip); if (ret < 0) return ret; @@ -703,6 +736,10 @@ struct bpf_trampoline_multi *bpf_trampoline_multi_get(struct bpf_prog *prog, if (!is_ftrace_location((void *) tgt_info.tgt_addr)) goto out_free; + err = ftrace_set_filter_ip(&multi->ops, tgt_info.tgt_addr, 0, 0); + if (err) + goto out_free; + if (nr_args < tgt_info.fmodel.nr_args) nr_args = tgt_info.fmodel.nr_args; } From patchwork Thu Aug 26 19:39:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460579 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A126DC432BE for ; Thu, 26 Aug 2021 19:41:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8997C60F4C for ; Thu, 26 Aug 2021 19:41:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243485AbhHZTmW (ORCPT ); Thu, 26 Aug 2021 15:42:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47311 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243616AbhHZTmU (ORCPT ); Thu, 26 Aug 2021 15:42:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006892; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+3b9pUdzhGyKoDaWX2s5ijJmxL2rpVCQT5XLwXyJeOo=; b=PQ2m8lVchFfD1LdldJTGj3chjXaDdaDSLj9QC9/HSMFD+Ejx5atd49EO46gjgPxEX95A15 3P1H/k+w/CWuZ5ffdK+WvdQAKWBR88QYovfQT9DxIYPqC3Jh61Rdn2RpCwNlsD896ROhb7 LVAKXWmEF/1zxsLAtphvrU7dau2R34A= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-402-fOsbiRfnPLSE4UAqg8JSFQ-1; Thu, 26 Aug 2021 15:41:31 -0400 X-MC-Unique: fOsbiRfnPLSE4UAqg8JSFQ-1 Received: by mail-wr1-f69.google.com with SMTP id z15-20020adff74f000000b001577d70c98dso1196024wrp.12 for ; Thu, 26 Aug 2021 12:41:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+3b9pUdzhGyKoDaWX2s5ijJmxL2rpVCQT5XLwXyJeOo=; b=BF6caG2pqwuRiY0S4M1/l1dH+qzn9idvMdLVWnqme1SFYNaa8Dxw1bkYHAnR/Ntpis HfLd9nuRvZLs2GrwFG+EXGxtj6nMmPUcXSsDrzL1F/V8ZIqX5I+gGoT2IYlGU6UaHLPQ M4dSAyewiPdPGPFgZ8+HlKdVZ9NJvddJYIRzoJcmx84Vf3ETlnanD8/88mWzaR28Sh4v 07pGUSHorabM2FYEmILsqjRpbxiBHf+Z+e/VOeQerK4X5cxKFLllh7aNB/ypD8kwWnkh Rla5PLXLtEERnVFVgpwgAm8em9p/c2cZPOJ5B02ZGhkmZ4UNfWG08DEZRszO9dE/bP4T v1qQ== X-Gm-Message-State: AOAM531HRnWTSlrJ5C2vX/ZFFY1RaFlyHO71Cyh/jFB04CvEH7xfAZyN 76SapJ0LrRvAQuQQMuHagYuHKlf5R6IixmwKcDW2Kxa2BFKAqd9Nkobfe7FLzNse35qt9XYd0tc O4FrcgS275mQi X-Received: by 2002:adf:fa82:: with SMTP id h2mr6020802wrr.195.1630006888611; Thu, 26 Aug 2021 12:41:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz8AA9YdhAgPNPAGj1JNJ0gytWblM+WJehjDHzc5IbNAoSE+WpYNT8IVVFqve3IdWXLaRtpqg== X-Received: by 2002:adf:fa82:: with SMTP id h2mr6020793wrr.195.1630006888354; Thu, 26 Aug 2021 12:41:28 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id w14sm3956093wrt.23.2021.08.26.12.41.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:28 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: Andrii Nakryiko , netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 20/27] libbpf: Add btf__find_by_glob_kind function Date: Thu, 26 Aug 2021 21:39:15 +0200 Message-Id: <20210826193922.66204-21-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding btf__find_by_glob_kind function that returns array of BTF ids that match given kind and allow/deny patterns. int btf__find_by_glob_kind(const struct btf *btf, __u32 kind, const char *allow_pattern, const char *deny_pattern, __u32 **__ids); The __ids array is allocated and needs to be manually freed. At the moment the supported pattern is '*' at the beginning or the end of the pattern. Kindly borrowed from retsnoop. Suggested-by: Andrii Nakryiko Signed-off-by: Jiri Olsa --- tools/lib/bpf/btf.c | 80 +++++++++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/btf.h | 3 ++ 2 files changed, 83 insertions(+) diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c index 77dc24d58302..5baaca6c3134 100644 --- a/tools/lib/bpf/btf.c +++ b/tools/lib/bpf/btf.c @@ -711,6 +711,86 @@ __s32 btf__find_by_name_kind(const struct btf *btf, const char *type_name, return libbpf_err(-ENOENT); } +/* 'borrowed' from retsnoop */ +static bool glob_matches(const char *glob, const char *s) +{ + int n = strlen(glob); + + if (n == 1 && glob[0] == '*') + return true; + + if (glob[0] == '*' && glob[n - 1] == '*') { + const char *subs; + /* substring match */ + + /* this is hacky, but we don't want to allocate for no good reason */ + ((char *)glob)[n - 1] = '\0'; + subs = strstr(s, glob + 1); + ((char *)glob)[n - 1] = '*'; + + return subs != NULL; + } else if (glob[0] == '*') { + size_t nn = strlen(s); + /* suffix match */ + + /* too short for a given suffix */ + if (nn < n - 1) + return false; + + return strcmp(s + nn - (n - 1), glob + 1) == 0; + } else if (glob[n - 1] == '*') { + /* prefix match */ + return strncmp(s, glob, n - 1) == 0; + } else { + /* exact match */ + return strcmp(glob, s) == 0; + } +} + +int btf__find_by_glob_kind(const struct btf *btf, __u32 kind, + const char *allow_pattern, const char *deny_pattern, + __u32 **__ids) +{ + __u32 i, nr_types = btf__get_nr_types(btf); + int cnt = 0, alloc = 0; + __u32 *ids = NULL; + + for (i = 1; i <= nr_types; i++) { + const struct btf_type *t = btf__type_by_id(btf, i); + bool match = false; + const char *name; + __u32 *p; + + if (btf_kind(t) != kind) + continue; + name = btf__name_by_offset(btf, t->name_off); + if (!name) + continue; + + if (allow_pattern && glob_matches(allow_pattern, name)) + match = true; + if (deny_pattern && !glob_matches(deny_pattern, name)) + match = true; + if (!match) + continue; + + if (cnt == alloc) { + alloc = max(100, alloc * 3 / 2); + p = realloc(ids, alloc * sizeof(__u32)); + if (!p) { + free(ids); + return -ENOMEM; + } + ids = p; + } + ids[cnt] = i; + cnt++; + } + + *__ids = ids; + return cnt ?: -ENOENT; +} + static bool btf_is_modifiable(const struct btf *btf) { return (void *)btf->hdr != btf->raw_data; diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h index 4a711f990904..b288211770c3 100644 --- a/tools/lib/bpf/btf.h +++ b/tools/lib/bpf/btf.h @@ -396,6 +396,9 @@ btf_var_secinfos(const struct btf_type *t) return (struct btf_var_secinfo *)(t + 1); } +int btf__find_by_glob_kind(const struct btf *btf, __u32 kind, + const char *allow_pattern, const char *deny_pattern, + __u32 **__ids); #ifdef __cplusplus } /* extern "C" */ #endif From patchwork Thu Aug 26 19:39:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460581 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BB98C4320E for ; Thu, 26 Aug 2021 19:41:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 23A3B610CB for ; Thu, 26 Aug 2021 19:41:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243489AbhHZTm0 (ORCPT ); Thu, 26 Aug 2021 15:42:26 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:39338 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243615AbhHZTm0 (ORCPT ); Thu, 26 Aug 2021 15:42:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006898; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sNOph7fDuFs97Eha5R8U+AbH+ZbOQ2fdynr7Ppgl9xo=; b=b57Bi3IzL+hQo7tH1pim1ZRafH0OZ+Ur5OqbZLuFFIdV33wbpJTm1ch8ob86XkjG2u8QX7 QdYO71wA07dlvrwNOb08zbC6++8FQ5bwLZPu+dqtjp3PF8NtxGU1F/p1Msx8MkL9ePtref DpHg4ENm8eK+n8OM1VoB5niUued1GYo= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-602-ZMnDMkuQMby9068he42_Kw-1; Thu, 26 Aug 2021 15:41:36 -0400 X-MC-Unique: ZMnDMkuQMby9068he42_Kw-1 Received: by mail-wm1-f71.google.com with SMTP id 201-20020a1c01d2000000b002e72ba822dcso4763425wmb.6 for ; Thu, 26 Aug 2021 12:41:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sNOph7fDuFs97Eha5R8U+AbH+ZbOQ2fdynr7Ppgl9xo=; b=SzaOFhdN8pnmK38itpye8PkqEcFipjdD35EcABGdb+nMdsAhxXX73jP4eXna1djy7d 3vMWZ38ATrVDcRqJNXbHitmza3CWdIT0d7dMdddNQqRNW4ta/nP2ve8jDwzOmNtbz6sL X6h2kriKttPgu+hQNaNZ6YMj8HSC07kVLKLgRhTFETKkTEONOWoMbay/vaTriyOaF+KT kzLvT1VLgsxthvmSUjYj35YQlErcAdyvA5kHzHRXJ7UKNDLDsl8k/OmZ62KyZa22HUeC XFqeZ1EMTQ850wTT9pQoMg7wX9/8Ns59hCuemC9oex5YgSilprEs0WTrIQvDlZkUOSB6 9DEg== X-Gm-Message-State: AOAM532Td2qL4CPPzF2TCmSi3HtZLqKufhk/L7JyAhRKlqtp/T5iWQ4n aM7GCaCHKIcOtfRoaGyVCGY4AO/7GeLUjbfJlS3t0EBBohjkaofDzwfEfVIAVRCgzWIJviTh/IS F/Ti03vbnDQzH X-Received: by 2002:a5d:65cc:: with SMTP id e12mr6218294wrw.266.1630006894649; Thu, 26 Aug 2021 12:41:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz1VfBS9lN1U/9ZDy+tjTNLSfUPE3NMeOcXS6gjf3AKHcPmgkRW6/Zzyv3AhZ1r8BdB6dgFyg== X-Received: by 2002:a5d:65cc:: with SMTP id e12mr6218280wrw.266.1630006894474; Thu, 26 Aug 2021 12:41:34 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id q11sm9133336wmc.41.2021.08.26.12.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:34 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 21/27] libbpf: Add support to link multi func tracing program Date: Thu, 26 Aug 2021 21:39:16 +0200 Message-Id: <20210826193922.66204-22-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding support to link multi func tracing program through link_create interface. Adding special types for multi func programs: fentry.multi fexit.multi so you can define multi func programs like: SEC("fentry.multi/bpf_fentry_test*") int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) that defines test1 to be attached to bpf_fentry_test* functions. The test1 program is loaded with BPF_F_MULTI_FUNC flag. If functions are not specified the program needs to be attached manually. Adding new btf_ids/btf_ids_cnt fields to bpf_link_create_opts, that define functions to attach the program to. Signed-off-by: Jiri Olsa --- tools/lib/bpf/bpf.c | 8 +++++ tools/lib/bpf/bpf.h | 6 +++- tools/lib/bpf/libbpf.c | 72 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 85 insertions(+), 1 deletion(-) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index 2401fad090c5..20422d58b945 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -713,12 +713,20 @@ int bpf_link_create(int prog_fd, int target_fd, if (!OPTS_ZEROED(opts, perf_event)) return libbpf_err(-EINVAL); break; + case BPF_TRACE_FENTRY: + case BPF_TRACE_FEXIT: + attr.link_create.multi_btf_ids = (__u64) OPTS_GET(opts, multi.btf_ids, 0); + attr.link_create.multi_btf_ids_cnt = OPTS_GET(opts, multi.btf_ids_cnt, 0); + if (!OPTS_ZEROED(opts, multi)) + return libbpf_err(-EINVAL); + break; default: if (!OPTS_ZEROED(opts, flags)) return libbpf_err(-EINVAL); break; } proceed: + fd = sys_bpf(BPF_LINK_CREATE, &attr, sizeof(attr)); return libbpf_err_errno(fd); } diff --git a/tools/lib/bpf/bpf.h b/tools/lib/bpf/bpf.h index 6fffb3cdf39b..2eff99be7069 100644 --- a/tools/lib/bpf/bpf.h +++ b/tools/lib/bpf/bpf.h @@ -181,10 +181,14 @@ struct bpf_link_create_opts { struct { __u64 bpf_cookie; } perf_event; + struct { + __u32 *btf_ids; + __u32 btf_ids_cnt; + } multi; }; size_t :0; }; -#define bpf_link_create_opts__last_field perf_event +#define bpf_link_create_opts__last_field multi LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, enum bpf_attach_type attach_type, diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 88d8825fc6f6..7f717b7755be 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -230,6 +230,7 @@ struct bpf_sec_def { bool is_attachable; bool is_attach_btf; bool is_sleepable; + bool is_multi_func; attach_fn_t attach_fn; }; @@ -6446,6 +6447,8 @@ __bpf_object__open(const char *path, const void *obj_buf, size_t obj_buf_sz, if (prog->sec_def->is_sleepable) prog->prog_flags |= BPF_F_SLEEPABLE; + if (prog->sec_def->is_multi_func) + prog->prog_flags |= BPF_F_MULTI_FUNC; bpf_program__set_type(prog, prog->sec_def->prog_type); bpf_program__set_expected_attach_type(prog, prog->sec_def->expected_attach_type); @@ -7915,6 +7918,8 @@ static struct bpf_link *attach_raw_tp(const struct bpf_sec_def *sec, struct bpf_program *prog); static struct bpf_link *attach_trace(const struct bpf_sec_def *sec, struct bpf_program *prog); +static struct bpf_link *attach_trace_multi(const struct bpf_sec_def *sec, + struct bpf_program *prog); static struct bpf_link *attach_lsm(const struct bpf_sec_def *sec, struct bpf_program *prog); static struct bpf_link *attach_iter(const struct bpf_sec_def *sec, @@ -7991,6 +7996,14 @@ static const struct bpf_sec_def section_defs[] = { .attach_fn = attach_iter), SEC_DEF("syscall", SYSCALL, .is_sleepable = true), + SEC_DEF("fentry.multi/", TRACING, + .expected_attach_type = BPF_TRACE_FENTRY, + .is_multi_func = true, + .attach_fn = attach_trace_multi), + SEC_DEF("fexit.multi/", TRACING, + .expected_attach_type = BPF_TRACE_FEXIT, + .is_multi_func = true, + .attach_fn = attach_trace_multi), BPF_EAPROG_SEC("xdp_devmap/", BPF_PROG_TYPE_XDP, BPF_XDP_DEVMAP), BPF_EAPROG_SEC("xdp_cpumap/", BPF_PROG_TYPE_XDP, @@ -8435,6 +8448,9 @@ static int libbpf_find_attach_btf_id(struct bpf_program *prog, int *btf_obj_fd, if (!name) return -EINVAL; + if (prog->prog_flags & BPF_F_MULTI_FUNC) + return 0; + for (i = 0; i < ARRAY_SIZE(section_defs); i++) { if (!section_defs[i].is_attach_btf) continue; @@ -9523,6 +9539,62 @@ static struct bpf_link *bpf_program__attach_btf_id(struct bpf_program *prog) return (struct bpf_link *)link; } +static struct bpf_link *bpf_program__attach_multi(struct bpf_program *prog) +{ + char *pattern = prog->sec_name + prog->sec_def->len; + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts); + enum bpf_attach_type attach_type; + int prog_fd, link_fd, cnt, err; + struct bpf_link *link = NULL; + __u32 *ids = NULL; + + prog_fd = bpf_program__fd(prog); + if (prog_fd < 0) { + pr_warn("prog '%s': can't attach before loaded\n", prog->name); + return ERR_PTR(-EINVAL); + } + + err = bpf_object__load_vmlinux_btf(prog->obj, true); + if (err) + return ERR_PTR(err); + + cnt = btf__find_by_glob_kind(prog->obj->btf_vmlinux, BTF_KIND_FUNC, + pattern, NULL, &ids); + if (cnt <= 0) + return ERR_PTR(-EINVAL); + + link = calloc(1, sizeof(*link)); + if (!link) { + err = -ENOMEM; + goto out_err; + } + link->detach = &bpf_link__detach_fd; + + opts.multi.btf_ids = ids; + opts.multi.btf_ids_cnt = cnt; + + attach_type = bpf_program__get_expected_attach_type(prog); + link_fd = bpf_link_create(prog_fd, 0, attach_type, &opts); + if (link_fd < 0) { + err = -errno; + goto out_err; + } + link->fd = link_fd; + free(ids); + return link; + +out_err: + free(link); + free(ids); + return ERR_PTR(err); +} + +static struct bpf_link *attach_trace_multi(const struct bpf_sec_def *sec, + struct bpf_program *prog) +{ + return bpf_program__attach_multi(prog); +} + struct bpf_link *bpf_program__attach_trace(struct bpf_program *prog) { return bpf_program__attach_btf_id(prog); From patchwork Thu Aug 26 19:39:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460583 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEBE5C432BE for ; Thu, 26 Aug 2021 19:41:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D95AA610D1 for ; Thu, 26 Aug 2021 19:41:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243494AbhHZTmc (ORCPT ); Thu, 26 Aug 2021 15:42:32 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:45663 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243628AbhHZTmb (ORCPT ); Thu, 26 Aug 2021 15:42:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006903; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IdiH+hWlAfCXXR5vse4KVzlNX30ztiZTZ3F2EBv6jwY=; b=Az8xrVD5yTCgdpUpNxNMD5mWoAHvZRZnzmgTZVqtBLJ4+UWRgLO13flHNs1cmw6IKr+fR4 C9IAbA+i5iRieQLJBMqwLuMmuUTk6pkxNzVxUXlWSbnOZXDhI9gwejVpRFlFf/h0OkqZ/1 Cq2ygbCFr21tpD88pqwW/j1R8pJ7Mj8= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-571-JrO2XGPNPmybBYeL6-TP9w-1; Thu, 26 Aug 2021 15:41:42 -0400 X-MC-Unique: JrO2XGPNPmybBYeL6-TP9w-1 Received: by mail-wr1-f69.google.com with SMTP id a13-20020adfed0d000000b00156fd70137aso1203213wro.8 for ; Thu, 26 Aug 2021 12:41:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IdiH+hWlAfCXXR5vse4KVzlNX30ztiZTZ3F2EBv6jwY=; b=BoaTsxIli5l+9V1vLw/myn+9XkkK7BILX3xoWTvvsR0JsiobG3aCmULxhIUxZRwZ0+ v+ktfFcpkWSvZvKJ0vozdqluc4cwRPKaXHGoLnwc7JQEHzqeOG82Z8sSol/dxlK/IIuV 0WzU5hHfSYfysh6BIOE3S1R5xA5N1rSiBhVIEZUheeKAxc34J//YoDK1UYLKzRSn2f/c PmVo5Ev7mvak1v6b9kabedj9GXCIYwNO/q4KeRuoc3ElRLPgpANakkU04o47qKtd2yTA rwebv3OsciAACAM/PLnc6VGmQdelUK7BH1NXiiqn2GHZIEwjoNzU0RG2QFrm5B50ygCK ejWA== X-Gm-Message-State: AOAM532XhsacjjmHY/IIn8ERwgUT1BwHVm4h/Lq/9iJFNyQLT5kMbp+P WhrQHY+Crk8y5YdgruFv6Z/b6uCZTThPRn2Jylmt3SqX3NxrmY+OkQCZnRXj879krbYCGf+YypH MW6D2K8/S3cUY X-Received: by 2002:adf:fa82:: with SMTP id h2mr6021560wrr.195.1630006900893; Thu, 26 Aug 2021 12:41:40 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzFG/7XNDG/mpqKRrO1b4pQ3vAuhPnl321lfpkEfEA5CRQvEFWrP8pYl4CQGSc0eHH1sSwbAw== X-Received: by 2002:adf:fa82:: with SMTP id h2mr6021547wrr.195.1630006900694; Thu, 26 Aug 2021 12:41:40 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id v28sm4013930wrv.93.2021.08.26.12.41.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:40 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 22/27] selftests/bpf: Add fentry multi func test Date: Thu, 26 Aug 2021 21:39:17 +0200 Message-Id: <20210826193922.66204-23-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fentry multi func test that attaches to bpf_fentry_test* functions and checks argument values based on the processed function. We need to cast to real arguments types in multi_arg_check, because the checked value can be shorter than u64. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 4 +- .../bpf/prog_tests/multi_fentry_test.c | 30 +++++++++ .../testing/selftests/bpf/progs/multi_check.c | 63 +++++++++++++++++++ .../selftests/bpf/progs/multi_fentry.c | 17 +++++ 4 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_check.c create mode 100644 tools/testing/selftests/bpf/progs/multi_fentry.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 866531c08e4f..013d41c8edae 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -312,7 +312,8 @@ endef SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ - linked_vars.skel.h linked_maps.skel.h + linked_vars.skel.h linked_maps.skel.h \ + multi_fentry_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ksyms_module.c test_ringbuf.c atomics.c trace_printk.c @@ -322,6 +323,7 @@ test_static_linked.skel.h-deps := test_static_linked1.o test_static_linked2.o linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o +multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c new file mode 100644 index 000000000000..8dc08c3e715f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_fentry_test.c @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_fentry_test.skel.h" +#include "trace_helpers.h" + +void test_multi_fentry_test(void) +{ + struct multi_fentry_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_fentry_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load")) + goto cleanup; + + err = multi_fentry_test__attach(skel); + if (!ASSERT_OK(err, "fentry_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test_result, 8, "test_result"); + +cleanup: + multi_fentry_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_check.c b/tools/testing/selftests/bpf/progs/multi_check.c new file mode 100644 index 000000000000..415c96684a30 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_check.c @@ -0,0 +1,63 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +extern const void bpf_fentry_test1 __ksym; +extern const void bpf_fentry_test2 __ksym; +extern const void bpf_fentry_test3 __ksym; +extern const void bpf_fentry_test4 __ksym; +extern const void bpf_fentry_test5 __ksym; +extern const void bpf_fentry_test6 __ksym; +extern const void bpf_fentry_test7 __ksym; +extern const void bpf_fentry_test8 __ksym; + +void multi_arg_check(__u64 *ctx, __u64 *test_result) +{ + void *ip = (void *) bpf_get_func_ip(ctx); + + if (ip == &bpf_fentry_test1) { + int a = (int) ctx[0]; + + *test_result += a == 1; + } else if (ip == &bpf_fentry_test2) { + int a = (int) ctx[0]; + __u64 b = ctx[1]; + + *test_result += a == 2 && b == 3; + } else if (ip == &bpf_fentry_test3) { + int a = (int) ctx[0]; + int b = (int) ctx[1]; + __u64 c = ctx[2]; + + *test_result += a == 4 && b == 5 && c == 6; + } else if (ip == &bpf_fentry_test4) { + void *a = (void *) ctx[0]; + int b = (int) ctx[1]; + int c = (int) ctx[2]; + __u64 d = ctx[3]; + + *test_result += a == (void *) 7 && b == 8 && c == 9 && d == 10; + } else if (ip == &bpf_fentry_test5) { + __u64 a = ctx[0]; + void *b = (void *) ctx[1]; + short c = (short) ctx[2]; + int d = (int) ctx[3]; + __u64 e = ctx[4]; + + *test_result += a == 11 && b == (void *) 12 && c == 13 && d == 14 && e == 15; + } else if (ip == &bpf_fentry_test6) { + __u64 a = ctx[0]; + void *b = (void *) ctx[1]; + short c = (short) ctx[2]; + int d = (int) ctx[3]; + void *e = (void *) ctx[4]; + __u64 f = ctx[5]; + + *test_result += a == 16 && b == (void *) 17 && c == 18 && d == 19 && e == (void *) 20 && f == 21; + } else if (ip == &bpf_fentry_test7) { + *test_result += 1; + } else if (ip == &bpf_fentry_test8) { + *test_result += 1; + } +} diff --git a/tools/testing/selftests/bpf/progs/multi_fentry.c b/tools/testing/selftests/bpf/progs/multi_fentry.c new file mode 100644 index 000000000000..b78d36772aa6 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_fentry.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__u64 test_result = 0; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); + +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test_result); + return 0; +} From patchwork Thu Aug 26 19:39:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460585 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4809CC432BE for ; Thu, 26 Aug 2021 19:41:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 316FD610CB for ; Thu, 26 Aug 2021 19:41:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243628AbhHZTmi (ORCPT ); Thu, 26 Aug 2021 15:42:38 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:25317 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243630AbhHZTmh (ORCPT ); Thu, 26 Aug 2021 15:42:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006909; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L4uQfRC0gVRi3xhE8qT4qNuckZwO4Pkcm7ceoMb3k98=; b=ghh22P8zIihrI4MNK0mLuZ/2Gkic5DmGe8gjn7qQ68yZO5OJ/etx5bwk5QqFyCozc4Vav5 krXNuP2+79t6Xue87pqmQCL6ASEWiFX8Q7CQT9xneq0BuqVO4UcxmDkij4Q02d9xHUqIyZ WCZcfmbwhM1SULGCQBZb94qV1maDBBg= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-545-F5tjQCBnOEqCTn2Yl7vKfA-1; Thu, 26 Aug 2021 15:41:48 -0400 X-MC-Unique: F5tjQCBnOEqCTn2Yl7vKfA-1 Received: by mail-wm1-f72.google.com with SMTP id z186-20020a1c7ec30000b02902e6a27a9962so4769437wmc.3 for ; Thu, 26 Aug 2021 12:41:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L4uQfRC0gVRi3xhE8qT4qNuckZwO4Pkcm7ceoMb3k98=; b=CHq4ZfDhKnIAEHHkI/7wiKxgnK+qn1v39qVmUudwIERUrOa4/8PU7XyD8jV5T5d+Mw opPdZ4yNxNYsM0O8Pz3r7udXB7eh8f8/3H2cFZCVWcQIA9c0rOGyYLkYWl2/hx5V6zPt 838nDiDMgOIFcMwv8VQZECOUzaqx2E0bESJtbqwN/73+XWTc6xnMNpiacvJxxqXJgjCd wrUWFBtJ8gRKMjpaCEdrvKySL1lPGd4sYCjAKD7dqbrGX+Xx1h4ov/nHmYIXgELBljJB cCuiEWhGtuVh02EGqYgVjwsJ5JOsfK9VDrD8NanoLZYzJD1KKNfIZgcGIKtabZ+GlLJ7 ZwKA== X-Gm-Message-State: AOAM533ErBgk55w5mYUFb6UOAA5kRLxuYshF19X9kTQtOTHbETDpW54/ tIpD2PbF/9MYhf0WaD9MlbYSZJqJuuODaMoo20G1cfTrGQXMkpQWSfstQ1QDaxeFFhB4Vec0FML 8b4X2epsaViwH X-Received: by 2002:adf:916f:: with SMTP id j102mr2112223wrj.422.1630006907161; Thu, 26 Aug 2021 12:41:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzVnhRZe4NqfZXpIVeTTiL8K0dGI7pHdPxTHEUchEkXPBEeXA2GfmitEE+opVev1Ae+keafIw== X-Received: by 2002:adf:916f:: with SMTP id j102mr2112205wrj.422.1630006906931; Thu, 26 Aug 2021 12:41:46 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id u10sm4137538wrt.14.2021.08.26.12.41.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:46 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 23/27] selftests/bpf: Add fexit multi func test Date: Thu, 26 Aug 2021 21:39:18 +0200 Message-Id: <20210826193922.66204-24-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fexit multi func test that attaches to bpf_fentry_test* functions and checks argument values based on the processed function. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/prog_tests/multi_fexit_test.c | 31 +++++++++++++++++++ .../testing/selftests/bpf/progs/multi_check.c | 22 +++++++++++++ .../testing/selftests/bpf/progs/multi_fexit.c | 20 ++++++++++++ 4 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_fexit.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 013d41c8edae..8da3be0972de 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -313,7 +313,7 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ - multi_fentry_test.skel.h + multi_fentry_test.skel.h multi_fexit_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ksyms_module.c test_ringbuf.c atomics.c trace_printk.c @@ -324,6 +324,7 @@ linked_funcs.skel.h-deps := linked_funcs1.o linked_funcs2.o linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o +multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c new file mode 100644 index 000000000000..d9b0eedd9f45 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_fexit_test.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_fexit_test.skel.h" +#include "trace_helpers.h" + +void test_multi_fexit_test(void) +{ + struct multi_fexit_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_fexit_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fexit_multi_skel_load")) + goto cleanup; + + err = multi_fexit_test__attach(skel); + if (!ASSERT_OK(err, "fexit_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test_arg_result, 8, "fexit_multi_arg_result"); + ASSERT_EQ(skel->bss->test_ret_result, 8, "fexit_multi_ret_result"); + +cleanup: + multi_fexit_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_check.c b/tools/testing/selftests/bpf/progs/multi_check.c index 415c96684a30..80ebbefaa8cd 100644 --- a/tools/testing/selftests/bpf/progs/multi_check.c +++ b/tools/testing/selftests/bpf/progs/multi_check.c @@ -61,3 +61,25 @@ void multi_arg_check(__u64 *ctx, __u64 *test_result) *test_result += 1; } } + +void multi_ret_check(void *ctx, int ret, __u64 *test_result) +{ + void *ip = (void *) bpf_get_func_ip(ctx); + + if (ip == &bpf_fentry_test1) + *test_result += ret == 2; + else if (ip == &bpf_fentry_test2) + *test_result += ret == 5; + else if (ip == &bpf_fentry_test3) + *test_result += ret == 15; + else if (ip == &bpf_fentry_test4) + *test_result += ret == 34; + else if (ip == &bpf_fentry_test5) + *test_result += ret == 65; + else if (ip == &bpf_fentry_test6) + *test_result += ret == 111; + else if (ip == &bpf_fentry_test7) + *test_result += ret == 0; + else if (ip == &bpf_fentry_test8) + *test_result += ret == 0; +} diff --git a/tools/testing/selftests/bpf/progs/multi_fexit.c b/tools/testing/selftests/bpf/progs/multi_fexit.c new file mode 100644 index 000000000000..29c49aa3bfc2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_fexit.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, int ret, __u64 *test_result); + +__u64 test_arg_result = 0; +__u64 test_ret_result = 0; + +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test_arg_result); + multi_ret_check(ctx, ret, &test_ret_result); + return 0; +} From patchwork Thu Aug 26 19:39:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460593 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A252C432BE for ; Thu, 26 Aug 2021 19:43:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F018560F4C for ; Thu, 26 Aug 2021 19:43:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243476AbhHZTnt (ORCPT ); Thu, 26 Aug 2021 15:43:49 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:44899 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243503AbhHZTnt (ORCPT ); Thu, 26 Aug 2021 15:43:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006981; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cg3b6pCUYgk1DCEc/G+FSd+XekVAu5S9ccZ75etSNB8=; b=AkYK2YxQHAxyo69U8XHC9X4RwLhXjPp4jzVROzSaIbid0Z3g3Y4q0yFndsblkYhsnSD+6q 8AO11e0qz1qUtRD2ZjZ1+ICFAfcspdEsSJQzFKwUb5DA5L7PKP/sArUZnIUpqJuiyGV5K3 +FdQDoPUmCxFIeZIjMB3RUV708AnXYQ= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-7-MgOu0cLYOUepSGsOr90-ZQ-1; Thu, 26 Aug 2021 15:41:54 -0400 X-MC-Unique: MgOu0cLYOUepSGsOr90-ZQ-1 Received: by mail-wr1-f70.google.com with SMTP id v18-20020adfe2920000b029013bbfb19640so1186161wri.17 for ; Thu, 26 Aug 2021 12:41:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cg3b6pCUYgk1DCEc/G+FSd+XekVAu5S9ccZ75etSNB8=; b=VrorhrKr6CFgJ0D+e4l6+vQPk9wBDoWt9JI8IlXTUtaMQOVyc/V9naqewT90cP4kiK PcmX8AWTnkh2Mu7k1nX6OLaTlQnhDNUJWIPqubwO1t/UTa4403oO9efIVIrSuAU1npAL miMU4Hys++zHKtVlvGl3b9gJ143Eqm9YQZUQ7u6VzLShAAqV7YRzhsCQF4J25SwNQ8/z BHSNv0deaHTfpdniZvRSscvn2p/s2cb+tg/ReIRulTGBPwkphK3WANPSIEhIt8t0SrDM SlIjnhNbdhyLOPwXJwQ2ZW/b03TBc8Dcwi/ROcyeliVxovI8IOF9t8hUdT/qBTBd5qRN twTA== X-Gm-Message-State: AOAM531JYaS9BJ7FISzRIcoMFI4I+SWOYJMQbKf2C7gIjsnF2rnGtlYb Ijz2auxLTd2JE2RkJK9lO+yAUTatynRDE25JyTu4lskpOxMQVMCeEnaPXALlUx1WP6B7slE0BoB yLptgVcRQgCwQ X-Received: by 2002:a7b:cb44:: with SMTP id v4mr5315777wmj.169.1630006913397; Thu, 26 Aug 2021 12:41:53 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz9WybcY+dg6NCte2Tc5bpGkGSGSc32LlQkKDXkHz9yizHUZoqK2YAG3KJWGGtS9Cqwax+XXA== X-Received: by 2002:a7b:cb44:: with SMTP id v4mr5315763wmj.169.1630006913198; Thu, 26 Aug 2021 12:41:53 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id l2sm10371542wmi.1.2021.08.26.12.41.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:52 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 24/27] selftests/bpf: Add fentry/fexit multi func test Date: Thu, 26 Aug 2021 21:39:19 +0200 Message-Id: <20210826193922.66204-25-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fentry/fexit multi func tests that attaches to bpf_fentry_test* functions and checks argument values based on the processed function. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 4 ++- .../bpf/prog_tests/multi_fentry_fexit_test.c | 32 +++++++++++++++++++ .../selftests/bpf/progs/multi_fentry_fexit.c | 28 ++++++++++++++++ 3 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_fentry_fexit.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 8da3be0972de..6272d9c166f9 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -313,7 +313,8 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ - multi_fentry_test.skel.h multi_fexit_test.skel.h + multi_fentry_test.skel.h multi_fexit_test.skel.h \ + multi_fentry_fexit_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ksyms_module.c test_ringbuf.c atomics.c trace_printk.c @@ -325,6 +326,7 @@ linked_vars.skel.h-deps := linked_vars1.o linked_vars2.o linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o +multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c b/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c new file mode 100644 index 000000000000..d54abf36ab2f --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_fentry_fexit_test.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_fentry_fexit_test.skel.h" + +void test_multi_fentry_fexit_test(void) +{ + DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts); + struct multi_fentry_fexit_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_fentry_fexit_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load")) + goto cleanup; + + err = multi_fentry_fexit_test__attach(skel); + if (!ASSERT_OK(err, "fentry_fexit_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test2); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test1_arg_result, 8, "test1_arg_result"); + ASSERT_EQ(skel->bss->test2_arg_result, 8, "test2_arg_result"); + ASSERT_EQ(skel->bss->test2_ret_result, 8, "test2_ret_result"); + +cleanup: + multi_fentry_fexit_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c b/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c new file mode 100644 index 000000000000..aac7df8c9211 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_fentry_fexit.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__u64 test1_arg_result = 0; +__u64 test2_arg_result = 0; +__u64 test2_ret_result = 0; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, int ret, __u64 *test_result); + +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test1_arg_result); + return 0; +} + +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test2_arg_result); + multi_ret_check(ctx, ret, &test2_ret_result); + return 0; +} From patchwork Thu Aug 26 19:39:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460587 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EFD0C432BE for ; Thu, 26 Aug 2021 19:42:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E65B360F11 for ; Thu, 26 Aug 2021 19:42:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243511AbhHZTmu (ORCPT ); Thu, 26 Aug 2021 15:42:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:26981 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243632AbhHZTmt (ORCPT ); Thu, 26 Aug 2021 15:42:49 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006922; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XvwmNaf8LZrHvZynooCcnzAt3fHTMCuVZNqKAgXRXAQ=; b=DHZN+t5s3Ly+MVybrF93HbMIx6NOvMELtPkNrWMOmnaW+aaKet48I401FW9Xr3p4Tdr/xc NNMPkJ7LXl37LmVezlHzfKLtl3lKhvsq5MweYZFf5drn9V/QoScfESeA0tTKkSwNa/Epfm L+Z7NzF6il/7mqiT0ubVT4CHHuYvS4U= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-537-8aUS3IT2Pq-bWosCXPz4Wg-1; Thu, 26 Aug 2021 15:42:00 -0400 X-MC-Unique: 8aUS3IT2Pq-bWosCXPz4Wg-1 Received: by mail-wm1-f71.google.com with SMTP id u14-20020a7bcb0e0000b0290248831d46e4so1127061wmj.6 for ; Thu, 26 Aug 2021 12:42:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XvwmNaf8LZrHvZynooCcnzAt3fHTMCuVZNqKAgXRXAQ=; b=Ga+7YfAFxMXLW0zeVEhU/ZDa+QavjwCt4tpizUYTbb6lusm/GGmoW48ky9mYEbM56S ZwLXMz/sIJdSPChERCQ2459vm8PaOaFSLBFPcC/5I/VNsvkMblHqernmvX+ZrCuZzrI2 DguiwJq882Ow4MRkmHMdiJ8lM5wHGb1UxdARuPORV8Nh4abUBOFSnH2l8FsYaKDqQKDS s0qb7LyvMnwWtSWK2ZQwVJlpNOHTashof+p3+vbtIwVBtufls5CrUHPWkzmPO8lROJPV xij9uoKNGDKf5FYyjUXoGbhVT+JVHdXghf9BJ2jRT6F/3por5ieWuvcRpn4MXnSvjImH yvjA== X-Gm-Message-State: AOAM531xV97bzdCaBV6NQ8PhW0VIElI4XYP04uotCEjuRdCe+mMTPa8L YGYP++Knc5cDxzdoljcgoTvEb530vktt02coMoQOQE9kc6+a0+eRO9LWEyP70vGexQEq5Ri0AVN 7Hb25ZpZ0y+zi X-Received: by 2002:adf:ba08:: with SMTP id o8mr6045288wrg.234.1630006919528; Thu, 26 Aug 2021 12:41:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxbSIcNWq3pNhPfh5PAZx+X4g/ugKXyanA20urzEHBZSEH0UpdYlBiliDK4HUiYLPJSI0J1qQ== X-Received: by 2002:adf:ba08:: with SMTP id o8mr6045267wrg.234.1630006919384; Thu, 26 Aug 2021 12:41:59 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id r129sm3599334wmr.7.2021.08.26.12.41.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:41:59 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 25/27] selftests/bpf: Add mixed multi func test Date: Thu, 26 Aug 2021 21:39:20 +0200 Message-Id: <20210826193922.66204-26-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest for fentry/fexit multi func tests that attaches to bpf_fentry_test* functions, where some of them have already attached trampoline. Signed-off-by: Jiri Olsa --- tools/testing/selftests/bpf/Makefile | 3 +- .../bpf/prog_tests/multi_mixed_test.c | 34 +++++++++++++++ .../testing/selftests/bpf/progs/multi_mixed.c | 43 +++++++++++++++++++ 3 files changed, 79 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_mixed.c diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 6272d9c166f9..1c80d76ebc70 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -314,7 +314,7 @@ SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \ linked_vars.skel.h linked_maps.skel.h \ multi_fentry_test.skel.h multi_fexit_test.skel.h \ - multi_fentry_fexit_test.skel.h + multi_fentry_fexit_test.skel.h multi_mixed_test.skel.h LSKELS := kfunc_call_test.c fentry_test.c fexit_test.c fexit_sleep.c \ test_ksyms_module.c test_ringbuf.c atomics.c trace_printk.c @@ -327,6 +327,7 @@ linked_maps.skel.h-deps := linked_maps1.o linked_maps2.o multi_fentry_test.skel.h-deps := multi_fentry.o multi_check.o multi_fexit_test.skel.h-deps := multi_fexit.o multi_check.o multi_fentry_fexit_test.skel.h-deps := multi_fentry_fexit.o multi_check.o +multi_mixed_test.skel.h-deps := multi_mixed.o multi_check.o LINKED_BPF_SRCS := $(patsubst %.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(skel)-deps))) diff --git a/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c b/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c new file mode 100644 index 000000000000..c9395b4eb5ac --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_mixed_test.c @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include "multi_mixed_test.skel.h" + +void test_multi_mixed_test(void) +{ + DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts); + struct multi_mixed_test *skel = NULL; + __u32 duration = 0, retval; + int err, prog_fd; + + skel = multi_mixed_test__open_and_load(); + if (!ASSERT_OK_PTR(skel, "fentry_multi_skel_load")) + goto cleanup; + + err = multi_mixed_test__attach(skel); + if (!ASSERT_OK(err, "fentry_attach")) + goto cleanup; + + prog_fd = bpf_program__fd(skel->progs.test1); + err = bpf_prog_test_run(prog_fd, 1, NULL, 0, + NULL, NULL, &retval, &duration); + ASSERT_OK(err, "test_run"); + ASSERT_EQ(retval, 0, "test_run"); + + ASSERT_EQ(skel->bss->test1_result, 1, "test1_result"); + ASSERT_EQ(skel->bss->test2_result, 1, "test2_result"); + ASSERT_EQ(skel->bss->test3_arg_result, 8, "test3_arg_result"); + ASSERT_EQ(skel->bss->test4_arg_result, 8, "test4_arg_result"); + ASSERT_EQ(skel->bss->test4_ret_result, 8, "test4_ret_result"); + +cleanup: + multi_mixed_test__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_mixed.c b/tools/testing/selftests/bpf/progs/multi_mixed.c new file mode 100644 index 000000000000..2ccd507747c7 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_mixed.c @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +__hidden extern void multi_arg_check(__u64 *ctx, __u64 *test_result); +__hidden extern void multi_ret_check(void *ctx, int ret, __u64 *test_result); + +__u64 test1_result = 0; +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(test1, int a) +{ + test1_result += a == 1; + return 0; +} + +__u64 test2_result = 0; +SEC("fexit/bpf_fentry_test2") +int BPF_PROG(test2, int a, __u64 b, int ret) +{ + test2_result += a == 2 && b == 3; + return 0; +} + +__u64 test3_arg_result = 0; +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test3, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + multi_arg_check(ctx, &test3_arg_result); + return 0; +} + +__u64 test4_arg_result = 0; +__u64 test4_ret_result = 0; +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test4, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + multi_arg_check(ctx, &test4_arg_result); + multi_ret_check(ctx, ret, &test4_ret_result); + return 0; +} From patchwork Thu Aug 26 19:39:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460589 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A4D0C432BE for ; Thu, 26 Aug 2021 19:42:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0480D60F11 for ; Thu, 26 Aug 2021 19:42:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243634AbhHZTm4 (ORCPT ); Thu, 26 Aug 2021 15:42:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:37147 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243522AbhHZTm4 (ORCPT ); Thu, 26 Aug 2021 15:42:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006928; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bokwZZdBG1B1kLTNIjydZjB2m8CruKIWTf2DTwAUs3E=; b=i9qKK11CdsE0bvDecLOA8Bi4HJCl/L8viGm5yGD0BENngzRVwt6r47Hn3x3Ff0AzBCMRp/ 3qGMA/mOkPtNrB4rHoyXSBQmdPIEEvIKhhWfYFmW3W6TAtTDLIDcJ6nfCQLbcBZARIXW9S AUzhWL+62Jcpk0FS/Sq7D72kVm4+e/M= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-318-Nlnp21dzNUW5Z3KoKzh0YQ-1; Thu, 26 Aug 2021 15:42:07 -0400 X-MC-Unique: Nlnp21dzNUW5Z3KoKzh0YQ-1 Received: by mail-wm1-f72.google.com with SMTP id h1-20020a05600c350100b002e751bf6733so1125898wmq.8 for ; Thu, 26 Aug 2021 12:42:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bokwZZdBG1B1kLTNIjydZjB2m8CruKIWTf2DTwAUs3E=; b=bfKRTETrOumAA2y+X9pDsCuQeV47lb5OZf89F2x89kWu15TSgTJAyBHwyBSesYH5k6 Xr/N2+MUyjhicuOXFodbD0CRVr83wXn12SEiqUr5lNKZi8Dlbk2LZinjIsExS4iZY6Fh h1e/ZgQ/4zcvXwU1LkaOUJkZiI6qGtyQipxEqMz4rICUQB2jaxwaYxR6Wqst5HBAUPEq BiD6OJZcOdFnX/40zagPCmTx6RvFJ6Ua/2jINM3cHK0fnwYAQsG4szMqu4jgix0sHGgG QBawZkqC3DPRudh7y93yfDfbTQqxFTF4nqxQxF/B/kBt/UsxfZoNCTprlUhWzoA7sEqO hsuw== X-Gm-Message-State: AOAM532l1Q4StOYc09n5MVCJ+WTBAaUVgA+y/3Cp4QSW6ntOZPtNFwaI bPG6reUbV9EXb4Ee6JUQhruJd32SsDT33xv5RPmTJ50J6Bx2bQICp52RhWGUks8LJBedtcf6SPY GW4+TGaulyieh X-Received: by 2002:adf:e28a:: with SMTP id v10mr5805326wri.289.1630006925723; Thu, 26 Aug 2021 12:42:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzFN+ZboqGkMatDOw0B/DLmI+FQdpqDxpOj7fXgQ9Xb7hPg5qMWaB/EhxUaqgiLxJEfevUbow== X-Received: by 2002:adf:e28a:: with SMTP id v10mr5805306wri.289.1630006925500; Thu, 26 Aug 2021 12:42:05 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id c1sm9108499wml.33.2021.08.26.12.42.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:42:05 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 26/27] selftests/bpf: Add attach multi func test Date: Thu, 26 Aug 2021 21:39:21 +0200 Message-Id: <20210826193922.66204-27-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding selftest to check attaching rules for multi func programs. - attach 2 programs: fentry/bpf_fentry_test1 fexit/bpf_fentry_test2 - check that we can attach multi func program on top of them: fentry.multi/bpf_fentry_test* - check that we cannot attach another multi funct program that does not cover the same BTF ids (one less): fentry.multi/bpf_fentry_test[1-7] fexit.multi/bpf_fentry_test[1-7] - check that we can no longer attach standard trampoline programs (below) on top of attached multi func program: fentry/bpf_fentry_test1 fexit/bpf_fentry_test3 Because the supported wildcards do not allow us to match just limited set of bpf_fentry_test*, adding extra code to look it up in kernel's BTF. Signed-off-by: Jiri Olsa --- .../bpf/prog_tests/multi_attach_check_test.c | 115 ++++++++++++++++++ .../selftests/bpf/progs/multi_attach_check.c | 36 ++++++ .../bpf/progs/multi_attach_check_extra1.c | 12 ++ .../bpf/progs/multi_attach_check_extra2.c | 12 ++ 4 files changed, 175 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/multi_attach_check_test.c create mode 100644 tools/testing/selftests/bpf/progs/multi_attach_check.c create mode 100644 tools/testing/selftests/bpf/progs/multi_attach_check_extra1.c create mode 100644 tools/testing/selftests/bpf/progs/multi_attach_check_extra2.c diff --git a/tools/testing/selftests/bpf/prog_tests/multi_attach_check_test.c b/tools/testing/selftests/bpf/prog_tests/multi_attach_check_test.c new file mode 100644 index 000000000000..32b23718437d --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/multi_attach_check_test.c @@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "multi_attach_check.skel.h" +#include "multi_attach_check_extra1.skel.h" +#include "multi_attach_check_extra2.skel.h" +#include + +static __u32 btf_ids[7]; + +static int load_btf_ids(void) +{ + __u32 i, nr_types, cnt; + struct btf *btf; + + btf = btf__load_vmlinux_btf(); + if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf")) + return -1; + + nr_types = btf__get_nr_types(btf); + + for (i = 1, cnt = 0; i <= nr_types && cnt < 7; i++) { + const struct btf_type *t = btf__type_by_id(btf, i); + const char *name; + + if (!btf_is_func(t)) + continue; + + name = btf__name_by_offset(btf, t->name_off); + if (!name) + continue; + if (strncmp(name, "bpf_fentry_test", sizeof("bpf_fentry_test") - 1)) + continue; + + btf_ids[cnt] = i; + cnt++; + } + + btf__free(btf); + return ASSERT_EQ(cnt, 7, "bpf_fentry_test_cnt") ? 0 : -1; +} + +void test_multi_attach_check_test(void) +{ + struct bpf_link *link1 = NULL, *link2 = NULL, *link3 = NULL; + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts); + struct multi_attach_check_extra1 *skel_extra1 = NULL; + struct multi_attach_check_extra2 *skel_extra2 = NULL; + struct multi_attach_check *skel; + int link_fd, prog_fd; + + /* Load/attach standard trampolines and on top of it multi + * func program. It should succeed. + */ + skel = multi_attach_check__open_and_load(); + if (!ASSERT_OK_PTR(skel, "multi_attach_check__load")) + return; + + link1 = bpf_program__attach(skel->progs.test1); + if (!ASSERT_OK_PTR(link1, "multi_attach_check__test1_attach")) + goto cleanup; + + link2 = bpf_program__attach(skel->progs.test2); + if (!ASSERT_OK_PTR(link2, "multi_attach_check__test2_attach")) + goto cleanup; + + link3 = bpf_program__attach(skel->progs.test3); + if (!ASSERT_OK_PTR(link3, "multi_attach_check__test3_attach")) + goto cleanup; + + if (!ASSERT_OK(load_btf_ids(), "load_btf_ids")) + goto cleanup; + + /* There's 8 bpf_fentry_test* functions, get BTF ids for 7 of them + * and try to load/link multi func program with them. It should fail + * both for fentry.multi ... + */ + opts.multi.btf_ids = btf_ids; + opts.multi.btf_ids_cnt = 7; + + prog_fd = bpf_program__fd(skel->progs.test4); + + link_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, &opts); + if (!ASSERT_LT(link_fd, 0, "bpf_link_create")) + goto cleanup; + + close(link_fd); + + /* ... and fexit.multi */ + prog_fd = bpf_program__fd(skel->progs.test5); + + link_fd = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, &opts); + if (!ASSERT_LT(link_fd, 0, "bpf_link_create")) + goto cleanup; + + close(link_fd); + + /* Try to load/attach extra programs on top of multi func programs, + * it should fail for both fentry ... + */ + skel_extra1 = multi_attach_check_extra1__open_and_load(); + if (!ASSERT_ERR_PTR(skel_extra1, "multi_attach_check_extra1__load")) + multi_attach_check_extra1__destroy(skel_extra1); + + /* ... and fexit */ + skel_extra2 = multi_attach_check_extra2__open_and_load(); + if (!ASSERT_ERR_PTR(skel_extra2, "multi_attach_check_extra2__load")) + multi_attach_check_extra2__destroy(skel_extra2); + +cleanup: + bpf_link__destroy(link1); + bpf_link__destroy(link2); + bpf_link__destroy(link3); + multi_attach_check__destroy(skel); +} diff --git a/tools/testing/selftests/bpf/progs/multi_attach_check.c b/tools/testing/selftests/bpf/progs/multi_attach_check.c new file mode 100644 index 000000000000..cf0f25c69556 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_attach_check.c @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(test1, int a) +{ + return 0; +} + +SEC("fexit/bpf_fentry_test2") +int BPF_PROG(test2, int a, __u64 b, int ret) +{ + return 0; +} + +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test3, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + return 0; +} + +SEC("fentry.multi/bpf_fentry_test1-7") +int BPF_PROG(test4, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + return 0; +} + +SEC("fexit.multi/bpf_fentry_test1-7") +int BPF_PROG(test5, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/multi_attach_check_extra1.c b/tools/testing/selftests/bpf/progs/multi_attach_check_extra1.c new file mode 100644 index 000000000000..c1d816a0206a --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_attach_check_extra1.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +SEC("fentry/bpf_fentry_test1") +int BPF_PROG(test1, int a) +{ + return 0; +} diff --git a/tools/testing/selftests/bpf/progs/multi_attach_check_extra2.c b/tools/testing/selftests/bpf/progs/multi_attach_check_extra2.c new file mode 100644 index 000000000000..cd66abb4b848 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_attach_check_extra2.c @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; + +SEC("fexit/bpf_fentry_test3") +int BPF_PROG(test3, int a, __u64 b, int ret) +{ + return 0; +} From patchwork Thu Aug 26 19:39:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiri Olsa X-Patchwork-Id: 12460591 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1CCFC4320E for ; Thu, 26 Aug 2021 19:42:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C561060F4C for ; Thu, 26 Aug 2021 19:42:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243637AbhHZTnE (ORCPT ); Thu, 26 Aug 2021 15:43:04 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:46421 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243632AbhHZTnE (ORCPT ); Thu, 26 Aug 2021 15:43:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1630006936; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s1ZTGe/UoYnoQvh13a27N1ppHSSL1B6a+t6brCKLLcE=; b=if83h5QsWY+l1+BQimlcy279ljo2pjV4fU8WehdGO//L38mMuZxYGOa6wgdBnzC8oQ+6VA wZjDH/HAht8UVWQn1VUb2neewvD3N+XfXDenPcRUTWqPCXhbPrPYybCXxC+7Xau4+0/cEV Gr8J4yP3EoL51WPHoDrlunESInFbfIw= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-330-FT84-1_LO86SNdFG0zjxMw-1; Thu, 26 Aug 2021 15:42:13 -0400 X-MC-Unique: FT84-1_LO86SNdFG0zjxMw-1 Received: by mail-wr1-f72.google.com with SMTP id o9-20020a5d6849000000b001574518a85aso1194916wrw.11 for ; Thu, 26 Aug 2021 12:42:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s1ZTGe/UoYnoQvh13a27N1ppHSSL1B6a+t6brCKLLcE=; b=smMJJdLTMQbxynpZ9FmE1HRvkA8ob3CRw6rlPNMWXNGmtr9NWadpo8mISlAbwvPxEF WNTYl+fHQ9430p3ZuYAEWQIBBFL+7Q4z08+OVg6tgjtKBR/T0gQs7Skpn19vHS0ifj9Z o1sneys+Mzu7nNhRtMITjIHY4RKsmlhTiqTv8IdvUEuygbu/5tWnETJB/YI4c1W4VHvT nw7AJNVq5+mQhlmAYlw7Cn4TcpMkugTvwinV5e42wYTvPuNwXfRPZ55NTPTUNF/xTJSs bgSDyVsZ+0UwRiumX9d/dgUdlBTXFRBRYLYKU1m9nyRuXNHlggXfBlWur9+JDVVtZ3m0 mtMg== X-Gm-Message-State: AOAM531a2tR1k1/rn9pwQCdrdpDOkKNxjKK+km2jbsG2yuZolzkymjgr R+i8eROCV0M5l+8fS3xQMTGvJ5BecVagyIEW9n1o2u5TOHsM0yhSQUtiEEnwu40/Qo55t9FCGYq TWYwgh3llq8PF X-Received: by 2002:a05:600c:a49:: with SMTP id c9mr15575546wmq.159.1630006931864; Thu, 26 Aug 2021 12:42:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyCKcQiJSVwruqkLwerb4RepMP4yAha4C4WOLcw9+AfBwsUrdhovlrq4XZznn+xhUvmxg7e6A== X-Received: by 2002:a05:600c:a49:: with SMTP id c9mr15575526wmq.159.1630006931617; Thu, 26 Aug 2021 12:42:11 -0700 (PDT) Received: from krava.redhat.com ([83.240.63.86]) by smtp.gmail.com with ESMTPSA id f20sm3372228wml.38.2021.08.26.12.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 26 Aug 2021 12:42:11 -0700 (PDT) From: Jiri Olsa X-Google-Original-From: Jiri Olsa To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Steven Rostedt (VMware)" Cc: netdev@vger.kernel.org, bpf@vger.kernel.org, Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Daniel Xu , Viktor Malik Subject: [PATCH bpf-next v4 27/27] selftests/bpf: Add ret_mod multi func test Date: Thu, 26 Aug 2021 21:39:22 +0200 Message-Id: <20210826193922.66204-28-jolsa@kernel.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210826193922.66204-1-jolsa@kernel.org> References: <20210826193922.66204-1-jolsa@kernel.org> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org X-Patchwork-Delegate: bpf@iogearbox.net Adding extra test to existing modify_return test to test this with multi func program attached on top of the modify return program. Because the supported wildcards do not allow us to match both bpf_fentry_test* and bpf_modify_return_test, adding extra code to look it up in kernel's BTF. Signed-off-by: Jiri Olsa --- .../selftests/bpf/prog_tests/modify_return.c | 114 +++++++++++++++++- .../selftests/bpf/progs/multi_modify_return.c | 17 +++ 2 files changed, 128 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/multi_modify_return.c diff --git a/tools/testing/selftests/bpf/prog_tests/modify_return.c b/tools/testing/selftests/bpf/prog_tests/modify_return.c index 97fec70c600b..9876104ad5b2 100644 --- a/tools/testing/selftests/bpf/prog_tests/modify_return.c +++ b/tools/testing/selftests/bpf/prog_tests/modify_return.c @@ -5,13 +5,100 @@ */ #include +#include #include "modify_return.skel.h" +#include "multi_modify_return.skel.h" #define LOWER(x) ((x) & 0xffff) #define UPPER(x) ((x) >> 16) -static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) +struct multi_data { + struct multi_modify_return *skel; + int link_fentry; + int link_fexit; + __u32 btf_ids[9]; +}; + +static int multi_btf_ids(struct multi_data *md) +{ + __u32 i, nr_types, ids_cnt; + struct btf *btf; + + btf = btf__load_vmlinux_btf(); + if (!ASSERT_OK_PTR(btf, "btf__load_vmlinux_btf")) + return -1; + + nr_types = btf__get_nr_types(btf); + + for (i = 1; i <= nr_types; i++) { + const struct btf_type *t = btf__type_by_id(btf, i); + const char *name; + bool match; + + if (!btf_is_func(t)) + continue; + + name = btf__name_by_offset(btf, t->name_off); + if (!name) + continue; + match = strncmp(name, "bpf_modify_return_test", + sizeof("bpf_modify_return_test") - 1) == 0; + match |= strncmp(name, "bpf_fentry_test", + sizeof("bpf_fentry_test") - 1) == 0; + if (!match) + continue; + + md->btf_ids[ids_cnt] = i; + ids_cnt++; + } + + btf__free(btf); + return ASSERT_EQ(ids_cnt, 9, "multi_btf_ids") ? 0 : -1; +} + +static int multi_attach(struct multi_data *md) +{ + DECLARE_LIBBPF_OPTS(bpf_link_create_opts, opts); + int prog_fd; + + md->skel = multi_modify_return__open_and_load(); + if (!ASSERT_OK_PTR(md->skel, "multi_attach_check__load")) + return -1; + + opts.multi.btf_ids = md->btf_ids; + opts.multi.btf_ids_cnt = 9; + + prog_fd = bpf_program__fd(md->skel->progs.test1); + + md->link_fentry = bpf_link_create(prog_fd, 0, BPF_TRACE_FENTRY, &opts); + if (!ASSERT_GE(md->link_fentry, 0, "bpf_link_create")) + goto cleanup; + + prog_fd = bpf_program__fd(md->skel->progs.test2); + + md->link_fexit = bpf_link_create(prog_fd, 0, BPF_TRACE_FEXIT, &opts); + if (!ASSERT_GE(md->link_fexit, 0, "bpf_link_create")) + goto cleanup_close; + + return 0; + +cleanup_close: + close(md->link_fentry); +cleanup: + multi_modify_return__destroy(md->skel); + return -1; +} + +static void multi_detach(struct multi_data *md) +{ + close(md->link_fentry); + close(md->link_fexit); + multi_modify_return__destroy(md->skel); +} + +static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret, + struct multi_data *md) { struct modify_return *skel = NULL; int err, prog_fd; @@ -27,6 +114,9 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) if (CHECK(err, "modify_return", "attach failed: %d\n", err)) goto cleanup; + if (md && !ASSERT_OK(multi_attach(md), "multi_attach")) + goto cleanup; + skel->bss->input_retval = input_retval; prog_fd = bpf_program__fd(skel->progs.fmod_ret_test); err = bpf_prog_test_run(prog_fd, 1, NULL, 0, NULL, 0, @@ -49,17 +139,35 @@ static void run_test(__u32 input_retval, __u16 want_side_effect, __s16 want_ret) CHECK(skel->bss->fmod_ret_result != 1, "modify_return", "fmod_ret failed\n"); + if (md) + multi_detach(md); cleanup: modify_return__destroy(skel); } void test_modify_return(void) { + struct multi_data data = {}; + + run_test(0 /* input_retval */, + 1 /* want_side_effect */, + 4 /* want_ret */, + NULL /* no multi func test */); + run_test(-EINVAL /* input_retval */, + 0 /* want_side_effect */, + -EINVAL /* want_ret */, + NULL /* no multi func test */); + + if (!ASSERT_OK(multi_btf_ids(&data), "multi_attach")) + return; + run_test(0 /* input_retval */, 1 /* want_side_effect */, - 4 /* want_ret */); + 4 /* want_ret */, + &data); run_test(-EINVAL /* input_retval */, 0 /* want_side_effect */, - -EINVAL /* want_ret */); + -EINVAL /* want_ret */, + &data); } diff --git a/tools/testing/selftests/bpf/progs/multi_modify_return.c b/tools/testing/selftests/bpf/progs/multi_modify_return.c new file mode 100644 index 000000000000..34754e438c96 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/multi_modify_return.c @@ -0,0 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +char _license[] SEC("license") = "GPL"; +SEC("fentry.multi/bpf_fentry_test*") +int BPF_PROG(test1, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f) +{ + return 0; +} + +SEC("fexit.multi/bpf_fentry_test*") +int BPF_PROG(test2, __u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f, int ret) +{ + return 0; +}