From patchwork Wed Aug 9 11:41:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347817 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44CE317AAB for ; Wed, 9 Aug 2023 11:41:39 +0000 (UTC) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D771F1FDE for ; Wed, 9 Aug 2023 04:41:37 -0700 (PDT) Received: by mail-ed1-x544.google.com with SMTP id 4fb4d7f45d1cf-52307552b03so9181438a12.0 for ; Wed, 09 Aug 2023 04:41:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581296; x=1692186096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OgCPWqiXQwFRSb/lWNuSAajZm2LK6U9DM3Wtj4eKKQc=; b=Y3m4rSCTWPn93+g3X2Bs+fktPej342y7vEpLBk8tF3gbs9jeymFpKk7kR0ROpsZ6KA O5zS0SAv7RSaG0+5ZC3HGf7fB+iTzYwjUPFNmauUwedZtu9IFEkHko+zlse+mALAp18t H+06WhAD2CUczWFb4m+xrbKYfPpSyFCjxu5KDgvDvZnARdFcAPMd2yAc6Zc48HiXRPxX sn3oIDdg3YjCwPeHfiBmGImvcPqLMoOUyCLT4IfaZxAa2GbJWHBolGjboN87A4nrZbA1 xyxAmU+XpdPBqUeRYHFKH+xIe3KmPYU+Cy2FDN3y2vqHHp327dUAskjrmystFjRbqMgH f0ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581296; x=1692186096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OgCPWqiXQwFRSb/lWNuSAajZm2LK6U9DM3Wtj4eKKQc=; b=de3Z1n84IiBtE/No8ID8jnzzD5UEP7J0Cik33nV/nPrMLy7aHlfH6lgxevxE+6mYH+ W6JuLD67XzSfnQidDnQi3zYefIZyQuGWbFw1y3u1LSgmKk9tOPNdGqZ8ZnI6hMeuHmTL oM3TrznSQ9bXRj6eLFAEw2ftYfjuKt7PtsB47HtDgKT4kelPo6Ivt4x+f39JDFI1Gyrb 0/gW1k3mRGftAK8GH7Qsgi4wKgf2WZdN+D/lRDtjldTvF6vSsp4ODfoGAwVF6BokmuE3 9WGyUBN62AxEfmkR2y1ZZq0iIBxZEYExe++5E4paOaDGk5jTCeAHEAoEyPx+Oc1d0dot YD/A== X-Gm-Message-State: AOJu0Yyo1Eh4JS8Wg6sZFuYE/159E2AIlK/BhiazfHfSBlGbQVDCwoWq o4QP4o7EPR5fXJUQyPGj4eV0NUb1sthNHeCh1gs= X-Google-Smtp-Source: AGHT+IF3iBrirqGTSOJFGjx5Hum/Pf+fSLQ9zoiASnCu3HINXrcqsRVJxlY0N0vL2ZyM/nB1iE2iAQ== X-Received: by 2002:aa7:df82:0:b0:523:4bfa:b450 with SMTP id b2-20020aa7df82000000b005234bfab450mr1679392edy.27.1691581295551; Wed, 09 Aug 2023 04:41:35 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id k24-20020a056402049800b005236b47116asm350808edv.70.2023.08.09.04.41.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:41:34 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 01/14] arch/x86: Implement arch_bpf_stack_walk Date: Wed, 9 Aug 2023 17:11:03 +0530 Message-ID: <20230809114116.3216687-2-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4040; i=memxor@gmail.com; h=from:subject; bh=4m7z5DYbfWvxVqsPDw9pgoNOX0kO5cOAvWd6mcbLs8M=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rI2qah9PJ77xfT5vRzBVHfsjMRNZvJvVBhP LCerXVx/waJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yAAKCRBM4MiGSL8R yiK1D/90dZCkDWvHPX2/zPWKxvtkvW4SgqjiFv0WW7YNADiShaoasfhSFLitdJRA69dLE5zVWyK PsjHDUsTY4RkTtTU3RXpWaLbb9+YJEfBbJkzScJP06i4Q/vOisDxmg9MAdHtHVeKckPDmL1ZuEe aYhOiBo1yt+9+CiY+9oZpvL1Mo28B6WJIWw0BDbNMrstPA2LUGTEXIVhgey/5Din0tLyykYhRkj 8Qqa8BX6vtryJl5XSRcyF4PaWskldhnT2mIqHmCALtKqP/3mtYJ44drzWk2Gu3fZwFguVY/ybKE ljFIqbaNOWIZ4Sfpy44BNah1bFmpMpbecKyWs9rkirj8HbFNQdYXDHrCWAt8FO9ion2FACIJ64o G1oPKQd0OKVnfwNrOVhHVQHGSyJawvmviQdov8o7NovwhcUi4TbeAcgJO/juhDqmUYw5frsbbBu FHgBx6q5wyp32Cdk2STktAUNcbPr0lPyvGwUDgu/iGmGLZDXUPXNSY6OtpMGs3V4y+xa8cMv/qg vTvmlW55/FloA6Q0CL2lQjYVFWM7gOpLjYeKvfyyHVh27mykbgjzyFp98cPNfedCp47vjc1fFPD LfUsGY5eUcWnNL9RkgVt+aCelnUyvbH+iC57U3OVH68iLDS7aowUfHNb+S/WQRT2MX2UIwadPgH s0rS1RbFL+bsGwA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net The plumbing for offline unwinding when we throw an exception in programs would require walking the stack, hence introduce a new arch_bpf_stack_walk function. This is provided when the JIT supports exceptions, i.e. bpf_jit_supports_exceptions is true. The arch-specific code is really minimal, hence it should be straightforward to extend this support to other architectures as well, as it reuses the logic of arch_stack_walk, but allowing access to unwind_state data. Once the stack pointer and frame pointer are known for the main subprog during the unwinding, we know the stack layout and location of any callee-saved registers which must be restored before we return back to the kernel. This handling will be added in the subsequent patches. Note that while we primarily unwind through BPF frames, which are effectively CONFIG_UNWINDER_FRAME_POINTER, we still need one of this or CONFIG_UNWINDER_ORC to be able to unwind through the bpf_throw frame from which we begin walking the stack. We also require both sp and bp (stack and frame pointers) from the unwind_state structure, which are only available when one of these two options are enabled. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 28 ++++++++++++++++++++++++++++ include/linux/filter.h | 2 ++ kernel/bpf/core.c | 9 +++++++++ 3 files changed, 39 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a5930042139d..a0a0054014e0 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -16,6 +16,7 @@ #include #include #include +#include static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { @@ -2913,3 +2914,30 @@ void bpf_jit_free(struct bpf_prog *prog) bpf_prog_unlock_free(prog); } + +bool bpf_jit_supports_exceptions(void) +{ + /* We unwind through both kernel frames (starting from within bpf_throw + * call) and BPF frames. Therefore we require one of ORC or FP unwinder + * to be enabled to walk kernel frames and reach BPF frames in the stack + * trace. + */ + return IS_ENABLED(CONFIG_UNWINDER_ORC) || IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER); +} + +void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) +{ +#if defined(CONFIG_UNWINDER_ORC) || defined(CONFIG_UNWINDER_FRAME_POINTER) + struct unwind_state state; + unsigned long addr; + + for (unwind_start(&state, current, NULL, NULL); !unwind_done(&state); + unwind_next_frame(&state)) { + addr = unwind_get_return_address(&state); + if (!addr || !consume_fn(cookie, (u64)addr, (u64)state.sp, (u64)state.bp)) + break; + } + return; +#endif + WARN(1, "verification of programs using bpf_throw should have failed\n"); +} diff --git a/include/linux/filter.h b/include/linux/filter.h index 761af6b3cf2b..9fd8f0dc4077 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -912,6 +912,8 @@ bool bpf_jit_needs_zext(void); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_far_kfunc_call(void); +bool bpf_jit_supports_exceptions(void); +void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie); bool bpf_helper_changes_pkt_data(void *func); static inline bool bpf_dump_raw_ok(const struct cred *cred) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 0f8f036d8bd1..526059386e9d 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2914,6 +2914,15 @@ int __weak bpf_arch_text_invalidate(void *dst, size_t len) return -ENOTSUPP; } +bool __weak bpf_jit_supports_exceptions(void) +{ + return false; +} + +void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) +{ +} + #ifdef CONFIG_BPF_SYSCALL static int __init bpf_global_ma_init(void) { From patchwork Wed Aug 9 11:41:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347818 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6AD8E17AAB for ; Wed, 9 Aug 2023 11:41:50 +0000 (UTC) Received: from mail-ed1-x541.google.com (mail-ed1-x541.google.com [IPv6:2a00:1450:4864:20::541]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9BDB1FD2 for ; Wed, 9 Aug 2023 04:41:48 -0700 (PDT) Received: by mail-ed1-x541.google.com with SMTP id 4fb4d7f45d1cf-5230a22cfd1so8930370a12.1 for ; Wed, 09 Aug 2023 04:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581307; x=1692186107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4R4rA62kVSnDZWKBc7RUYNS8basysfRTud29ctuoGuU=; b=XupG/Hx9zEdFmy3gtjLT4W1aMC6Nfr85ZLChbCveBmCQ/P5MI0/eIfStnjZF3n7/wm uEBtx1stsEL+IJEQe7HIoF9QSdSOeA81py+QR8TtbhvEQxnMYWkBWVg2gY3DE5QKq46n 8qwhfEY9vvBqbdaHN2p1e9mo4x0eJtTDmQR72PHsbFg77p8En3ZgofAaJDUaGLCWVgNX G5RULiusSvQQGyBjJw4UM7Qm20mNYcrht+dWmhgX6gkv30ORI4vw2ZKFA6TcD9uMsHH+ rkCI+rwyu8sn/TwM4nlTi4wkSNt2WTypSIMmpQ2LFdfedR4fU5T6UVD/CV1vM8D9S2Of x7bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581307; x=1692186107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4R4rA62kVSnDZWKBc7RUYNS8basysfRTud29ctuoGuU=; b=jyI9fNpGcUPgbMWxJUta59SY1iqgEqm5xXnt9aQTwpWP9KU/xLcclBAz2vlcdMj5DU R64iCOyEiKUI0wNDDEXTrxmQys12C7dv93vsfxs0FNKOdn4FVUm6R+Tv8i533uMzrJVD 4eICCzMiZqVkb3/h72ycCGF/OtJtk+2abRE13Bg6WNDLVNdCpYsVCy5MezDvUMGDYGHn Od/9uTX11with6Ze+q/x3JcX07ojy2B6mLu4nmnpapqENCI4kTtCkf+fvZvDILBmiYNN bhE066ioMiYePAwmCQ8xtRts6w6XJIinTsyL4f/PSvDCOHh9GOUm7Zu2yTmsEcTRPnL7 Nf3g== X-Gm-Message-State: AOJu0YzVVTF82p4zmUa3DH9mLNVVT469Z0GHVrcgENex8hYQCrglzAwv VTL31/0WQ/ONvvKqCXkzg4+zHuAE5yJYgYsd9p0= X-Google-Smtp-Source: AGHT+IEdutLpWWPvLxpxnncslh1DzD1jVO+SVYzrxIEGmy8MnWkWjeQoF91ELN1O5pRkCFVUm8gpAA== X-Received: by 2002:a17:906:2097:b0:991:c842:2ca2 with SMTP id 23-20020a170906209700b00991c8422ca2mr2155723ejq.15.1691581306733; Wed, 09 Aug 2023 04:41:46 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id m8-20020a1709066d0800b0099cadcf13cesm6977876ejr.66.2023.08.09.04.41.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:41:45 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 02/14] bpf: Implement support for adding hidden subprogs Date: Wed, 9 Aug 2023 17:11:04 +0530 Message-ID: <20230809114116.3216687-3-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8798; i=memxor@gmail.com; h=from:subject; bh=KFQ9FWW0oxq2pX5bNBd6GsvJzMUT24XEyjPEvCLV0HA=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rI0gyMCy72lU88JrAdEyJbKWftIXk0si2Gk 4ez8YU/HfuJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yAAKCRBM4MiGSL8R yv1ID/9qN/dUl4I3VDbTi42ARnofWs5Wq4x4/b7SsAhTjN5BQIrIc8LtjlT11KPozmmYhQpWXyc rtL5F7zYGuL9UxyfHVLSSt3fYg8ETo4QL9duhczX7DXSSIIN7S/pI9Um9veYSeX5XBmcBMrBPN0 s/NOclfIIz6Tf4qVhUdwgINCVYqhkG30d0uGuyPvlazK+7Pn0QY+sjMUS+A3atF84G5zNjoOzfl K6XDtlTBsS6roM3I267dJ49T+iNWaPdcibyU7bCRcPFHk9kOO05oVJ92TmDWskKYlH7GLPOUico TgboHa0VV+cfex00m7hQa462ANS05h6YNx2XbBbBe9tGOUK5PPqbgy8Q8FVc/L83j/rLGKx7jnI cxSOAqfUepfuqwrivqTOx1NdbPFLdv6NgiIBliUv6VxhWG2hW4itRZFEiOjRYl3L4KJsJCulx+4 Poe0rbe33iWxrTBHGUIXlzq2cP2I6imOOoFx+Pjr/NSHkWEOSd0p3lXLmUB1AUh00PinrqCe+uz LfAETNu5ZYwU3VY+0iqvBu2JVwKy79pidJm7KdIKVDygyMzfRRHiHvon6S26HYQsJ+li5vI9pWe I6s1VFskkvPWANdK6o8L/rSuuImAsl7FTt8kuaZYgcKyCjhhhKRhs59VggV+wdrsML9tC6d22tN 2aCm6IbXRTuFrlg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net Introduce support in the verifier for generating a subprogram and include it as part of a BPF program dynamically after the do_check phase is complete. The first user will be the next patch which generates default exception callbacks if none are set for the program. The phase of invocation will be do_misc_fixups. Note that this is an internal verifier function, and should be used with instruction blocks which uphold the invariants stated in check_subprogs. Since these subprogs are always appended to the end of the instruction sequence of the program, it becomes relatively inexpensive to do the related adjustments to the subprog_info of the program. Only the fake exit subprogram is shifted forward, making room for our new subprog. This is useful to insert a new subprogram, get it JITed, and obtain its function pointer. The next patch will use this functionality to insert a default exception callback which will be invoked after unwinding the stack. Note that these added subprograms are invisible to userspace, and never reported in BPF_OBJ_GET_INFO_BY_ID etc. For now, only a single subprogram is supported, but more can be easily supported in the future. To this end, two function counts are introduced now, the existing func_cnt, and real_func_cnt, the latter including hidden programs. This allows us to conver the JIT code to use the real_func_cnt for management of resources while syscall path continues working with existing func_cnt. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 3 ++- kernel/bpf/core.c | 12 ++++++------ kernel/bpf/syscall.c | 2 +- kernel/bpf/verifier.c | 36 +++++++++++++++++++++++++++++++++--- 5 files changed, 43 insertions(+), 11 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index db3fe5a61b05..751f565037f9 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1377,6 +1377,7 @@ struct bpf_prog_aux { u32 stack_depth; u32 id; u32 func_cnt; /* used by non-func prog as the number of func progs */ + u32 real_func_cnt; /* includes hidden progs, only used for JIT and freeing progs */ u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */ u32 attach_btf_id; /* in-kernel BTF type id to attach to */ u32 ctx_arg_info_size; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index f70f9ac884d2..beb0e9e01bd5 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -587,6 +587,7 @@ struct bpf_verifier_env { u32 used_map_cnt; /* number of used maps */ u32 used_btf_cnt; /* number of used BTF objects */ u32 id_gen; /* used to generate unique reg IDs */ + u32 hidden_subprog_cnt; /* number of hidden subprogs */ bool explore_alu_limits; bool allow_ptr_leaks; bool allow_uninit_stack; @@ -597,7 +598,7 @@ struct bpf_verifier_env { struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; - struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 1]; + struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */ union { struct bpf_idmap idmap_scratch; struct bpf_idset idset_scratch; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 526059386e9d..2e5907d15118 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -212,7 +212,7 @@ void bpf_prog_fill_jited_linfo(struct bpf_prog *prog, const struct bpf_line_info *linfo; void **jited_linfo; - if (!prog->aux->jited_linfo) + if (!prog->aux->jited_linfo || prog->aux->func_idx > prog->aux->func_cnt) /* Userspace did not provide linfo */ return; @@ -539,7 +539,7 @@ static void bpf_prog_kallsyms_del_subprogs(struct bpf_prog *fp) { int i; - for (i = 0; i < fp->aux->func_cnt; i++) + for (i = 0; i < fp->aux->real_func_cnt; i++) bpf_prog_kallsyms_del(fp->aux->func[i]); } @@ -589,7 +589,7 @@ bpf_prog_ksym_set_name(struct bpf_prog *prog) sym = bin2hex(sym, prog->tag, sizeof(prog->tag)); /* prog->aux->name will be ignored if full btf name is available */ - if (prog->aux->func_info_cnt) { + if (prog->aux->func_info_cnt && prog->aux->func_idx < prog->aux->func_info_cnt) { type = btf_type_by_id(prog->aux->btf, prog->aux->func_info[prog->aux->func_idx].type_id); func_name = btf_name_by_offset(prog->aux->btf, type->name_off); @@ -1208,7 +1208,7 @@ int bpf_jit_get_func_addr(const struct bpf_prog *prog, if (!extra_pass) addr = NULL; else if (prog->aux->func && - off >= 0 && off < prog->aux->func_cnt) + off >= 0 && off < prog->aux->real_func_cnt) addr = (u8 *)prog->aux->func[off]->bpf_func; else return -EINVAL; @@ -2721,7 +2721,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) #endif if (aux->dst_trampoline) bpf_trampoline_put(aux->dst_trampoline); - for (i = 0; i < aux->func_cnt; i++) { + for (i = 0; i < aux->real_func_cnt; i++) { /* We can just unlink the subprog poke descriptor table as * it was originally linked to the main program and is also * released along with it. @@ -2729,7 +2729,7 @@ static void bpf_prog_free_deferred(struct work_struct *work) aux->func[i]->aux->poke_tab = NULL; bpf_jit_free(aux->func[i]); } - if (aux->func_cnt) { + if (aux->real_func_cnt) { kfree(aux->func); bpf_prog_unlock_free(aux->prog); } else { diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 7f4e8c357a6a..d90f5001da83 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -2746,7 +2746,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) * period before we can tear down JIT memory since symbols * are already exposed under kallsyms. */ - __bpf_prog_put_noref(prog, prog->aux->func_cnt); + __bpf_prog_put_noref(prog, prog->aux->real_func_cnt); return err; free_prog_sec: free_uid(prog->aux->user); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4ccca1f6c998..ed90e22d7600 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15108,7 +15108,8 @@ static void adjust_btf_func(struct bpf_verifier_env *env) if (!aux->func_info) return; - for (i = 0; i < env->subprog_cnt; i++) + /* func_info is not available for hidden subprogs */ + for (i = 0; i < env->subprog_cnt - env->hidden_subprog_cnt; i++) aux->func_info[i].insn_off = env->subprog_info[i].start; } @@ -18053,7 +18054,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) * the call instruction, as an index for this list */ func[i]->aux->func = func; - func[i]->aux->func_cnt = env->subprog_cnt; + func[i]->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; + func[i]->aux->real_func_cnt = env->subprog_cnt; } for (i = 0; i < env->subprog_cnt; i++) { old_bpf_func = func[i]->bpf_func; @@ -18099,7 +18101,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->extable = func[0]->aux->extable; prog->aux->num_exentries = func[0]->aux->num_exentries; prog->aux->func = func; - prog->aux->func_cnt = env->subprog_cnt; + prog->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; + prog->aux->real_func_cnt = env->subprog_cnt; bpf_prog_jit_attempt_done(prog); return 0; out_free: @@ -18307,6 +18310,33 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return 0; } +/* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */ +static __maybe_unused int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) +{ + struct bpf_subprog_info *info = env->subprog_info; + int cnt = env->subprog_cnt; + struct bpf_prog *prog; + + /* We only reserve one slot for hidden subprogs in subprog_info. */ + if (env->hidden_subprog_cnt) { + verbose(env, "verifier internal error: only one hidden subprog supported\n"); + return -EFAULT; + } + /* We're not patching any existing instruction, just appending the new + * ones for the hidden subprog. Hence all of the adjustment operations + * in bpf_patch_insn_data are no-ops. + */ + prog = bpf_patch_insn_data(env, env->prog->len - 1, patch, len); + if (!prog) + return -ENOMEM; + env->prog = prog; + info[cnt + 1].start = info[cnt].start; + info[cnt].start = prog->len - len + 1; + env->subprog_cnt++; + env->hidden_subprog_cnt++; + return 0; +} + /* Do various post-verification rewrites in a single program pass. * These rewrites simplify JIT and interpreter implementations. */ From patchwork Wed Aug 9 11:41:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347819 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2BDA17ABC for ; Wed, 9 Aug 2023 11:42:06 +0000 (UTC) Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C8C01FD7 for ; Wed, 9 Aug 2023 04:42:04 -0700 (PDT) Received: by mail-pf1-x442.google.com with SMTP id d2e1a72fcca58-686ed1d2594so6381648b3a.2 for ; Wed, 09 Aug 2023 04:42:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581323; x=1692186123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fZfqJZRpQ4loiBQ1rjAhGfqvUiqnK3afeERj0EN7cKo=; b=jVLWp2P0+iPGgGAM6U0lgWcrOR2HviZdDFTYKH3X6433cvlHygsSKvG6mFsufrUFIU iohzwV9f5SEeaYsRO4DE32R+5FGaZj/5Q2r5EX1FLFET9pFiAWSxAExYFDFoIvE2gp39 QjDLEuP1vj4I2FoD/aPlBJKjVjPTYaOdDkLKBaYb+Sdi+5Wij58DCrofSTLOrICEGRZf D9x8gy401aYGPkkttBeNOLfGnvSLQ+P9ByBHhmU1KAm8XFG9WHX2Ckj6xGQyElqJ2tDK pM1ST8e1DLiud8FHVEbLI8gol3gB6CPjZekbQEF0WjS670NhlBhlPwBGVl6PCkjSfYq1 xiZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581323; x=1692186123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fZfqJZRpQ4loiBQ1rjAhGfqvUiqnK3afeERj0EN7cKo=; b=HJXh82qqbm0kKhyLI7zmacRExMpXf467azgnBDfbPKp+r1ShIkaFGp3/rSr8aRwvF/ ++BoST494iXJCGcGYlXDRTxYxpTHBZE3crCsq9WDTQwHONmC+GNLUKaV3Y6x9f8A/h6O fPvp6iuzlNgJOp8CzxH09WGbTHfFgr1TaDCo/m5MN81+31gV6Md1r2O5rfhh/VDfRjD/ g4eJUUATBAmSw4iP39PruFpfpqq6THk5KS39Nj46mt9LEUrXm/Ke+FCaoZg7jAzwwROR 39Zmne2P1seSIQ+BOP/vVY8ad9nTkkL9h+Jv6yt5jUImDg0dl1UtctvqmrcIrw3QgTfP amqA== X-Gm-Message-State: AOJu0YzjudMnzsE2dn1ZYUkvu8s3ZSgH/BbpTi7pTk9MBLNfpOHz1skA czGrpe/mKFPqC0kBVYc/8y41ZRqP8JlQJ1gkd9o= X-Google-Smtp-Source: AGHT+IFVrIvGxipAGjmEjerYB2BW3yEo1GK/Corr3wa//2rz6RqpEQL99zWr5CyhlTWIn/yj/iSFGg== X-Received: by 2002:a05:6a20:3d0f:b0:138:5a28:e8cc with SMTP id y15-20020a056a203d0f00b001385a28e8ccmr3105741pzi.37.1691581323132; Wed, 09 Aug 2023 04:42:03 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id n27-20020a638f1b000000b00551df489590sm7896278pgd.12.2023.08.09.04.42.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:42:02 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 03/14] bpf: Implement BPF exceptions Date: Wed, 9 Aug 2023 17:11:05 +0530 Message-ID: <20230809114116.3216687-4-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=25772; i=memxor@gmail.com; h=from:subject; bh=OO0/h9Z6L9b5bQjlrldPOYjazkjvjLJmbUvuffE9coI=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rIu0MaKgPKLILFrtae5cbYQIbXCMtC9CJvi GEvGQWsPD2JAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yAAKCRBM4MiGSL8R ynWwD/48iEj9UUgsOzLrDWZOKDb4GrvizzTBegt8vzWgzgSYpc/3CgnupBn0WqgbgU0gbbnDC1Z a4Q0nvPqMaj6u/3Tks6JSqgyDN13W43dogiWTDtckzOaN+5SooDDK9yPP4wzviAN/xpRjgZf+2X pkztlUYNEGmV3crep0/EoQWpzZO24Ycf/HCq/5VwvyG5dcSuQ9+AHc0uGavuvndp6/iIMmWz8iM tzlaiJ93qfw19PyeEJ5YgPn89hLJggl/o3oYcT/I4RaBPhEepHNrYwJ1S0mneAMeOHyq5Zdu76s ViM0GQ+iBabn9wp8Nfeiy7pHG2iXj4iz2q/zLVruwdT/xF5BsIRfIfMmjfeRwtFr8i/jnJQYtJc T94gD9PYJlh1oqLlfI86TZqXsmaPgXXbffoxb3i8+Olp6JM0ZlBNf5hZ17rWZF7nBMmJD/wkT8w yX3iVmr8z0cvtoEBh3tIUxl/UmnowTbCcQ/p6Q0OKT/klteqJ2Nd/NFflV0MemMgZ1OzzGD8w4Z MrM+D8/bxgwO0AdRc5wCgsQsli4MkIjTTiFJ92Yd8uw4llkn+6WWdCWEGxtVeqkjJ2Bx5jcAWi0 dD86tEib7j/P5ep1AxjVZNIq//hUVHE4o3YcfjzoLu0Wjtc/UekRfyvpqEjd2b55yOoyfLFOByd N2rDi2fQ2HtEHfA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net This patch implements BPF exceptions, and introduces a bpf_throw kfunc to allow programs to throw exceptions during their execution at runtime. A bpf_throw invocation is treated as an immediate termination of the program, returning back to its caller within the kernel, unwinding all stack frames. This allows the program to simplify its implementation, by testing for runtime conditions which the verifier has no visibility into, and assert that they are true. In case they are not, the program can simply throw an exception from the other branch. BPF exceptions are explicitly *NOT* an unlikely slowpath error handling primitive, and this objective has guided design choices of the implementation of the them within the kernel (with the bulk of the cost for unwinding the stack offloaded to the bpf_throw kfunc). The implementation of this mechanism requires use of add_hidden_subprog mechanism introduced in the previous patch, which generates a couple of instructions to move R1 to R0 and exit. The JIT then rewrites the prologue of this subprog to take the stack pointer and frame pointer as inputs and reset the stack frame, popping all callee-saved registers saved by the main subprog. The bpf_throw function then walks the stack at runtime, and invokes this exception subprog with the stack and frame pointers as parameters. Reviewers must take note that currently the main program is made to save all callee-saved registers on x86_64 during entry into the program. This is because we must do an equivalent of a lightweight context switch when unwinding the stack, therefore we need the callee-saved registers of the caller of the BPF program to be able to return with a sane state. Note that we have to additionally handle r12, even though it is not used by the program, because when throwing the exception the program makes an entry into the kernel which could clobber r12 after saving it on the stack. To be able to preserve the value we received on program entry, we push r12 and restore it from the generated subprogram when unwinding the stack. For now, bpf_throw invocation fails when lingering resources or locks exist in that path of the program. In a future followup, bpf_throw will be extended to perform frame-by-frame unwinding to release lingering resources for each stack frame, removing this limitation. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 90 ++++++++++++-- include/linux/bpf.h | 3 + include/linux/bpf_verifier.h | 4 + include/linux/filter.h | 6 + kernel/bpf/core.c | 2 +- kernel/bpf/helpers.c | 38 ++++++ kernel/bpf/verifier.c | 116 +++++++++++++++--- .../testing/selftests/bpf/bpf_experimental.h | 16 +++ 8 files changed, 248 insertions(+), 27 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a0a0054014e0..7f28705da26e 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -18,6 +18,8 @@ #include #include +static bool all_callee_regs_used[4] = {true, true, true, true}; + static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { if (len == 1) @@ -256,6 +258,14 @@ struct jit_context { /* Number of bytes that will be skipped on tailcall */ #define X86_TAIL_CALL_OFFSET (11 + ENDBR_INSN_SIZE) +static void push_r12(u8 **pprog) +{ + u8 *prog = *pprog; + + EMIT2(0x41, 0x54); /* push r12 */ + *pprog = prog; +} + static void push_callee_regs(u8 **pprog, bool *callee_regs_used) { u8 *prog = *pprog; @@ -271,6 +281,14 @@ static void push_callee_regs(u8 **pprog, bool *callee_regs_used) *pprog = prog; } +static void pop_r12(u8 **pprog) +{ + u8 *prog = *pprog; + + EMIT2(0x41, 0x5C); /* pop r12 */ + *pprog = prog; +} + static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) { u8 *prog = *pprog; @@ -292,7 +310,8 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) * while jumping to another program */ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf, - bool tail_call_reachable, bool is_subprog) + bool tail_call_reachable, bool is_subprog, + bool is_exception_cb) { u8 *prog = *pprog; @@ -308,8 +327,22 @@ static void emit_prologue(u8 **pprog, u32 stack_depth, bool ebpf_from_cbpf, else EMIT2(0x66, 0x90); /* nop2 */ } - EMIT1(0x55); /* push rbp */ - EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ + /* Exception callback receives FP as second parameter */ + if (is_exception_cb) { + EMIT3(0x48, 0x89, 0xF4); /* mov rsp, rsi */ + EMIT3(0x48, 0x89, 0xD5); /* mov rbp, rdx */ + /* The main frame must have exception_boundary as true, so we + * first restore those callee-saved regs from stack, before + * reusing the stack frame. + */ + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + /* Reset the stack frame. */ + EMIT3(0x48, 0x89, 0xEC); /* mov rsp, rbp */ + } else { + EMIT1(0x55); /* push rbp */ + EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ + } /* X86_TAIL_CALL_OFFSET is here */ EMIT_ENDBR(); @@ -468,7 +501,8 @@ static void emit_return(u8 **pprog, u8 *ip) * goto *(prog->bpf_func + prologue_size); * out: */ -static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, +static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog, + u8 **pprog, bool *callee_regs_used, u32 stack_depth, u8 *ip, struct jit_context *ctx) { @@ -518,7 +552,12 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, offset = ctx->tail_call_indirect_label - (prog + 2 - start); EMIT2(X86_JE, offset); /* je out */ - pop_callee_regs(&prog, callee_regs_used); + if (bpf_prog->aux->exception_boundary) { + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + } else { + pop_callee_regs(&prog, callee_regs_used); + } EMIT1(0x58); /* pop rax */ if (stack_depth) @@ -542,7 +581,8 @@ static void emit_bpf_tail_call_indirect(u8 **pprog, bool *callee_regs_used, *pprog = prog; } -static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke, +static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog, + struct bpf_jit_poke_descriptor *poke, u8 **pprog, u8 *ip, bool *callee_regs_used, u32 stack_depth, struct jit_context *ctx) @@ -571,7 +611,13 @@ static void emit_bpf_tail_call_direct(struct bpf_jit_poke_descriptor *poke, emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE, poke->tailcall_bypass); - pop_callee_regs(&prog, callee_regs_used); + if (bpf_prog->aux->exception_boundary) { + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + } else { + pop_callee_regs(&prog, callee_regs_used); + } + EMIT1(0x58); /* pop rax */ if (stack_depth) EMIT3_off32(0x48, 0x81, 0xC4, round_up(stack_depth, 8)); @@ -1042,8 +1088,21 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image emit_prologue(&prog, bpf_prog->aux->stack_depth, bpf_prog_was_classic(bpf_prog), tail_call_reachable, - bpf_prog->aux->func_idx != 0); - push_callee_regs(&prog, callee_regs_used); + bpf_prog->aux->func_idx != 0, bpf_prog->aux->exception_cb); + /* Exception callback will clobber callee regs for its own use, and + * restore the original callee regs from main prog's stack frame. + */ + if (bpf_prog->aux->exception_boundary) { + /* We also need to save r12, which is not mapped to any BPF + * register, as we throw after entry into the kernel, which may + * overwrite r12. + */ + push_r12(&prog); + push_callee_regs(&prog, all_callee_regs_used); + } else { + push_callee_regs(&prog, callee_regs_used); + } + ilen = prog - temp; if (rw_image) @@ -1642,13 +1701,15 @@ st: if (is_imm8(insn->off)) case BPF_JMP | BPF_TAIL_CALL: if (imm32) - emit_bpf_tail_call_direct(&bpf_prog->aux->poke_tab[imm32 - 1], + emit_bpf_tail_call_direct(bpf_prog, + &bpf_prog->aux->poke_tab[imm32 - 1], &prog, image + addrs[i - 1], callee_regs_used, bpf_prog->aux->stack_depth, ctx); else - emit_bpf_tail_call_indirect(&prog, + emit_bpf_tail_call_indirect(bpf_prog, + &prog, callee_regs_used, bpf_prog->aux->stack_depth, image + addrs[i - 1], @@ -1901,7 +1962,12 @@ st: if (is_imm8(insn->off)) seen_exit = true; /* Update cleanup_addr */ ctx->cleanup_addr = proglen; - pop_callee_regs(&prog, callee_regs_used); + if (bpf_prog->aux->exception_boundary) { + pop_callee_regs(&prog, all_callee_regs_used); + pop_r12(&prog); + } else { + pop_callee_regs(&prog, callee_regs_used); + } EMIT1(0xC9); /* leave */ emit_return(&prog, image + addrs[i - 1] + (prog - temp)); break; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 751f565037f9..e938e75b0998 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1398,6 +1398,8 @@ struct bpf_prog_aux { bool sleepable; bool tail_call_reachable; bool xdp_has_frags; + bool exception_cb; + bool exception_boundary; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ @@ -1420,6 +1422,7 @@ struct bpf_prog_aux { int cgroup_atype; /* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]; char name[BPF_OBJ_NAME_LEN]; + unsigned int (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp); #ifdef CONFIG_SECURITY void *security; #endif diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index beb0e9e01bd5..9e6c25ecac9f 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -540,7 +540,9 @@ struct bpf_subprog_info { bool has_tail_call; bool tail_call_reachable; bool has_ld_abs; + bool is_cb; bool is_async_cb; + bool is_exception_cb; }; struct bpf_verifier_env; @@ -588,6 +590,7 @@ struct bpf_verifier_env { u32 used_btf_cnt; /* number of used BTF objects */ u32 id_gen; /* used to generate unique reg IDs */ u32 hidden_subprog_cnt; /* number of hidden subprogs */ + int exception_callback_subprog; bool explore_alu_limits; bool allow_ptr_leaks; bool allow_uninit_stack; @@ -595,6 +598,7 @@ struct bpf_verifier_env { bool bypass_spec_v1; bool bypass_spec_v4; bool seen_direct_write; + bool seen_exception; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; diff --git a/include/linux/filter.h b/include/linux/filter.h index 9fd8f0dc4077..389e550a6a25 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1129,6 +1129,7 @@ const char *__bpf_address_lookup(unsigned long addr, unsigned long *size, bool is_bpf_text_address(unsigned long addr); int bpf_get_kallsym(unsigned int symnum, unsigned long *value, char *type, char *sym); +struct bpf_prog *bpf_prog_ksym_find(unsigned long addr); static inline const char * bpf_address_lookup(unsigned long addr, unsigned long *size, @@ -1196,6 +1197,11 @@ static inline int bpf_get_kallsym(unsigned int symnum, unsigned long *value, return -ERANGE; } +static inline struct bpf_prog *bpf_prog_ksym_find(unsigned long addr) +{ + return NULL; +} + static inline const char * bpf_address_lookup(unsigned long addr, unsigned long *size, unsigned long *off, char **modname, char *sym) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2e5907d15118..ef362d7b09a5 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -733,7 +733,7 @@ bool is_bpf_text_address(unsigned long addr) return ret; } -static struct bpf_prog *bpf_prog_ksym_find(unsigned long addr) +struct bpf_prog *bpf_prog_ksym_find(unsigned long addr) { struct bpf_ksym *ksym = bpf_ksym_find(addr); diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index eb91cae0612a..af4add1e3a31 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2429,6 +2429,43 @@ __bpf_kfunc void bpf_rcu_read_unlock(void) rcu_read_unlock(); } +struct bpf_throw_ctx { + struct bpf_prog_aux *aux; + u64 sp; + u64 bp; + int cnt; +}; + +static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp) +{ + struct bpf_throw_ctx *ctx = cookie; + struct bpf_prog *prog; + + if (!is_bpf_text_address(ip)) + return !ctx->cnt; + prog = bpf_prog_ksym_find(ip); + ctx->cnt++; + if (!prog->aux->id) + return true; + ctx->aux = prog->aux; + ctx->sp = sp; + ctx->bp = bp; + return false; +} + +__bpf_kfunc void bpf_throw(u64 cookie) +{ + struct bpf_throw_ctx ctx = {}; + + arch_bpf_stack_walk(bpf_stack_walker, &ctx); + WARN_ON_ONCE(!ctx.aux); + if (ctx.aux) + WARN_ON_ONCE(!ctx.aux->exception_boundary); + WARN_ON_ONCE(!ctx.bp); + WARN_ON_ONCE(!ctx.cnt); + ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); +} + __diag_pop(); BTF_SET8_START(generic_btf_ids) @@ -2456,6 +2493,7 @@ BTF_ID_FLAGS(func, bpf_cgroup_from_id, KF_ACQUIRE | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_task_under_cgroup, KF_RCU) #endif BTF_ID_FLAGS(func, bpf_task_from_pid, KF_ACQUIRE | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_throw) BTF_SET8_END(generic_btf_ids) static const struct btf_kfunc_id_set generic_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ed90e22d7600..2ac0be088dd5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -543,6 +543,7 @@ static bool is_dynptr_ref_function(enum bpf_func_id func_id) } static bool is_callback_calling_kfunc(u32 btf_id); +static bool is_bpf_throw_kfunc(struct bpf_insn *insn); static bool is_callback_calling_function(enum bpf_func_id func_id) { @@ -1748,7 +1749,9 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state, return -ENOMEM; dst_state->jmp_history_cnt = src->jmp_history_cnt; - /* if dst has more stack frames then src frame, free them */ + /* if dst has more stack frames then src frame, free them, this is also + * necessary in case of exceptional exits using bpf_throw. + */ for (i = src->curframe + 1; i <= dst_state->curframe; i++) { free_func_state(dst_state->frame[i]); dst_state->frame[i] = NULL; @@ -2868,7 +2871,7 @@ static int check_subprogs(struct bpf_verifier_env *env) if (i == subprog_end - 1) { /* to avoid fall-through from one subprog into another * the last insn of the subprog should be either exit - * or unconditional jump back + * or unconditional jump back or bpf_throw call */ if (code != (BPF_JMP | BPF_EXIT) && code != (BPF_JMP32 | BPF_JA) && @@ -5645,6 +5648,27 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) for (; i < subprog_end; i++) { int next_insn, sidx; + if (bpf_pseudo_kfunc_call(insn + i) && !insn[i].off) { + bool err = false; + + if (!is_bpf_throw_kfunc(insn + i)) + continue; + if (subprog[idx].is_cb) + err = true; + for (int c = 0; c < frame && !err; c++) { + if (subprog[ret_prog[c]].is_cb) { + err = true; + break; + } + } + if (!err) + continue; + verbose(env, + "bpf_throw kfunc (insn %d) cannot be called from callback subprog %d\n", + i, idx); + return -EINVAL; + } + if (!bpf_pseudo_call(insn + i) && !bpf_pseudo_func(insn + i)) continue; /* remember insn and function to return to */ @@ -8905,6 +8929,7 @@ static int __check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn * callbacks */ if (set_callee_state_cb != set_callee_state) { + env->subprog_info[subprog].is_cb = true; if (bpf_pseudo_kfunc_call(insn) && !is_callback_calling_kfunc(insn->imm)) { verbose(env, "verifier bug: kfunc %s#%d not marked as callback-calling\n", @@ -9294,7 +9319,8 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) verbose(env, "to caller at %d:\n", *insn_idx); print_verifier_state(env, caller, true); } - /* clear everything in the callee */ + /* clear everything in the callee. In case of exceptional exits using + * bpf_throw, this will be done by copy_verifier_state for extra frames. */ free_func_state(callee); state->frame[state->curframe--] = NULL; return 0; @@ -9418,17 +9444,17 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, return 0; } -static int check_reference_leak(struct bpf_verifier_env *env) +static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exit) { struct bpf_func_state *state = cur_func(env); bool refs_lingering = false; int i; - if (state->frameno && !state->in_callback_fn) + if (!exception_exit && state->frameno && !state->in_callback_fn) return 0; for (i = 0; i < state->acquired_refs; i++) { - if (state->in_callback_fn && state->refs[i].callback_ref != state->frameno) + if (!exception_exit && state->in_callback_fn && state->refs[i].callback_ref != state->frameno) continue; verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", state->refs[i].id, state->refs[i].insn_idx); @@ -9662,7 +9688,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn switch (func_id) { case BPF_FUNC_tail_call: - err = check_reference_leak(env); + err = check_reference_leak(env, false); if (err) { verbose(env, "tail_call would lead to reference leak\n"); return err; @@ -10271,6 +10297,7 @@ enum special_kfunc_type { KF_bpf_dynptr_slice, KF_bpf_dynptr_slice_rdwr, KF_bpf_dynptr_clone, + KF_bpf_throw, }; BTF_SET_START(special_kfunc_set) @@ -10291,6 +10318,7 @@ BTF_ID(func, bpf_dynptr_from_xdp) BTF_ID(func, bpf_dynptr_slice) BTF_ID(func, bpf_dynptr_slice_rdwr) BTF_ID(func, bpf_dynptr_clone) +BTF_ID(func, bpf_throw) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -10313,6 +10341,7 @@ BTF_ID(func, bpf_dynptr_from_xdp) BTF_ID(func, bpf_dynptr_slice) BTF_ID(func, bpf_dynptr_slice_rdwr) BTF_ID(func, bpf_dynptr_clone) +BTF_ID(func, bpf_throw) static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) { @@ -10626,6 +10655,12 @@ static bool is_callback_calling_kfunc(u32 btf_id) return btf_id == special_kfunc_list[KF_bpf_rbtree_add_impl]; } +static bool is_bpf_throw_kfunc(struct bpf_insn *insn) +{ + return bpf_pseudo_kfunc_call(insn) && insn->off == 0 && + insn->imm == special_kfunc_list[KF_bpf_throw]; +} + static bool is_rbtree_lock_required_kfunc(u32 btf_id) { return is_bpf_rbtree_api_kfunc(btf_id); @@ -11400,6 +11435,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } } + if (meta.func_id == special_kfunc_list[KF_bpf_throw]) { + if (!bpf_jit_supports_exceptions()) { + verbose(env, "JIT does not support calling kfunc %s#%d\n", + func_name, meta.func_id); + return -ENOTSUPP; + } + env->seen_exception = true; + } + for (i = 0; i < CALLER_SAVED_REGS; i++) mark_reg_not_init(env, regs, caller_saved[i]); @@ -14423,7 +14467,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * gen_ld_abs() may terminate the program at runtime, leading to * reference leak. */ - err = check_reference_leak(env); + err = check_reference_leak(env, false); if (err) { verbose(env, "BPF_LD_[ABS|IND] cannot be mixed with socket references\n"); return err; @@ -16437,6 +16481,7 @@ static int do_check(struct bpf_verifier_env *env) int prev_insn_idx = -1; for (;;) { + bool exception_exit = false; struct bpf_insn *insn; u8 class; int err; @@ -16651,12 +16696,17 @@ static int do_check(struct bpf_verifier_env *env) return -EINVAL; } } - if (insn->src_reg == BPF_PSEUDO_CALL) + if (insn->src_reg == BPF_PSEUDO_CALL) { err = check_func_call(env, insn, &env->insn_idx); - else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) + } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { err = check_kfunc_call(env, insn, &env->insn_idx); - else + if (!err && is_bpf_throw_kfunc(insn)) { + exception_exit = true; + goto process_bpf_exit_full; + } + } else { err = check_helper_call(env, insn, &env->insn_idx); + } if (err) return err; @@ -16686,7 +16736,7 @@ static int do_check(struct bpf_verifier_env *env) verbose(env, "BPF_EXIT uses reserved fields\n"); return -EINVAL; } - +process_bpf_exit_full: if (env->cur_state->active_lock.ptr && !in_rbtree_lock_required_cb(env)) { verbose(env, "bpf_spin_unlock is missing\n"); @@ -16704,10 +16754,23 @@ static int do_check(struct bpf_verifier_env *env) * function, for which reference_state must * match caller reference state when it exits. */ - err = check_reference_leak(env); + err = check_reference_leak(env, exception_exit); if (err) return err; + /* The side effect of the prepare_func_exit + * which is being skipped is that it frees + * bpf_func_state. Typically, process_bpf_exit + * will only be hit with outermost exit. + * copy_verifier_state in pop_stack will handle + * freeing of any extra bpf_func_state left over + * from not processing all nested function + * exits. We also skip return code checks as + * they are not needed for exceptional exits. + */ + if (exception_exit) + goto process_bpf_exit; + if (state->curframe) { /* exit from nested function */ err = prepare_func_exit(env, &env->insn_idx); @@ -18015,6 +18078,9 @@ static int jit_subprogs(struct bpf_verifier_env *env) } func[i]->aux->num_exentries = num_exentries; func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable; + func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb; + if (!i) + func[i]->aux->exception_boundary = env->seen_exception; func[i] = bpf_int_jit_compile(func[i]); if (!func[i]->jited) { err = -ENOTSUPP; @@ -18103,6 +18169,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->func = func; prog->aux->func_cnt = env->subprog_cnt - env->hidden_subprog_cnt; prog->aux->real_func_cnt = env->subprog_cnt; + prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; + prog->aux->exception_boundary = func[0]->aux->exception_boundary; bpf_prog_jit_attempt_done(prog); return 0; out_free: @@ -18311,7 +18379,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, } /* The function requires that first instruction in 'patch' is insnsi[prog->len - 1] */ -static __maybe_unused int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) +static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *patch, int len) { struct bpf_subprog_info *info = env->subprog_info; int cnt = env->subprog_cnt; @@ -18355,6 +18423,26 @@ static int do_misc_fixups(struct bpf_verifier_env *env) struct bpf_map *map_ptr; int i, ret, cnt, delta = 0; + if (env->seen_exception && !env->exception_callback_subprog) { + struct bpf_insn patch[] = { + env->prog->insnsi[insn_cnt - 1], + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }; + + ret = add_hidden_subprog(env, patch, ARRAY_SIZE(patch)); + if (ret < 0) + return ret; + prog = env->prog; + insn = prog->insnsi; + + env->exception_callback_subprog = env->subprog_cnt - 1; + /* Don't update insn_cnt, as invent_subprog always appends insns */ + env->subprog_info[env->exception_callback_subprog].is_cb = true; + env->subprog_info[env->exception_callback_subprog].is_async_cb = true; + env->subprog_info[env->exception_callback_subprog].is_exception_cb = true; + } + for (i = 0; i < insn_cnt; i++, insn++) { /* Make divide-by-zero exceptions impossible. */ if (insn->code == (BPF_ALU64 | BPF_MOD | BPF_X) || diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 209811b1993a..952a40cbe09c 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -131,4 +131,20 @@ extern int bpf_rbtree_add_impl(struct bpf_rb_root *root, struct bpf_rb_node *nod */ extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym; +/* Description + * Throw a BPF exception from the program, immediately terminating its + * execution and unwinding the stack. The supplied 'cookie' parameter + * will be the return value of the program when an exception is thrown. + * + * Note that throwing an exception with lingering resources (locks, + * references, etc.) will lead to a verification error. + * + * Note that callbacks *cannot* call this helper. + * Returns + * Never. + * Throws + * An exception with the specified 'cookie' value. + */ +extern void bpf_throw(u64 cookie) __ksym; + #endif From patchwork Wed Aug 9 11:41:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347820 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF8C017AAB for ; Wed, 9 Aug 2023 11:42:22 +0000 (UTC) Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2CBE1FEE for ; Wed, 9 Aug 2023 04:42:21 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id d9443c01a7336-1bc6bfc4b58so20662635ad.1 for ; Wed, 09 Aug 2023 04:42:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581341; x=1692186141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HUvv6Os60HAuReQRcsxzrHCKFcspSZmwRcjvI8hvxp8=; b=IkyD973lEYKAyevTIVociX0nWWUTH/Rfm9u9Jqvc/oYX70Ky9W2bYWky0InQZg+e67 2yq3ddxMQGYOovqL5fF02w5V+NOQroVC20C/oYAUqOUWH63znP8/YoCqwelvNoXWdye3 U4gqKFPVAyWWXlJVGjWpoJE7+WLxXyrK7nBHuPk2d/V88++z4UsHiuKdDFncaLzklOnG 76ZG+TaPxUjrcg/dn2msj++outMB9ff6gzTwP4jk/wa4104AjlnxrkB5woj+jZGQA+/x X9aqSqWb8/H5xLgUVdQorElVCDYsnUh/qAa9yakYUnjBPfGJReHNf2b0LOvYiNHRyG39 1axA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581341; x=1692186141; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HUvv6Os60HAuReQRcsxzrHCKFcspSZmwRcjvI8hvxp8=; b=W+LmnrvuGNPwQMZwgIlPJRrz2k1DIDheWX3FISAuNPB1ka5cizLRmxLyyJvuX5J9Z+ M1bBvid/da1RbbgcUWtCpeWukkL4Xtj4d7Xh3EhCVJa7JanlmffwEaNvZf0JRkRjmsTk SYuMha9M9sBuvDKsMqC64zPsLw73hGeP9jQUAhqwaP+GoTFmzoxhpkW0ENIbbIqYlvEc 8yE1vrhZbkszDf+UUJ3/SOU6SMj4vOguwlbNPXeQScUNreCBcXnHAfNBfbLEY2FMSADa jKcOjIrNYgyeZfh+FmL1pDGgPSZnnCHCqaO2s1HGZF/6RL7ufUUwYMKSvV2lnA3fDUE+ 3nRg== X-Gm-Message-State: AOJu0Yw5Ps5BswITVNpZhAwBlG27+huhwRjSPJMtFZ1ILGE7kzun8gLR ApKX4+yRuU8bsKdplC9WC+z4TAAPY91S9MnHQkY= X-Google-Smtp-Source: AGHT+IEsUS89Z0GXi81R35KMMZFHS7hoF0cRZ6m22BQn85vmBu0AABhcd1wkLL85xvpZ2x3wlsdMVA== X-Received: by 2002:a17:902:ec8a:b0:1b9:dea2:800f with SMTP id x10-20020a170902ec8a00b001b9dea2800fmr2817335plg.8.1691581340606; Wed, 09 Aug 2023 04:42:20 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id y5-20020a1709029b8500b001b8943b37a5sm10952157plp.24.2023.08.09.04.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:42:19 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 04/14] bpf: Refactor check_btf_func and split into two phases Date: Wed, 9 Aug 2023 17:11:06 +0530 Message-ID: <20230809114116.3216687-5-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7961; i=memxor@gmail.com; h=from:subject; bh=s6wIRf1n0bide5l//ymRv094SUp/1VDFnOriLs97C6A=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rImXWteajf3kSbJfQM+5x6N6GGf6Blgn1kv kZVboOnTEyJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yAAKCRBM4MiGSL8R yvJrD/0bu2R8j+IHn+t3fh0XsWVVL70OLAfn0/iPQk37RfXu6FiI84YvXmQAUrqbv640K+k+/LQ qnvmPqw4+6Uu0IZLDaTl1b5jGhBv+s+4lWgGwTJLacHEopsbr4IMfzUeiCtPnZyQbFV0qRdK9sc IqhTVmyTpl9WXqVEftbHlxijClvKLcUpM790WyR9hrvXLrFiuFeRgRmonLfaqZf3Z+NejABYPQi RLBOFCthUqe1D1uUYOVEJ1YDJJOv4OCrGf7trAyQkHnIQ2WIXoaVKrq4MnFZPQSzQ5zGZyQHHcP VQYnQEy9pGpFgXrZztdUq7A1Rbq77z6zwiNGgp+jlHl3iRKSHQqZDe6oYmcg/Foa5eIUqOh1Sjt XgSITRlagP3DI9ID/n0Dh+ZfEhlwFZUo1f33rZlqvhcDqsu8L1Rrq4X69a1dGcyBySzkg66Mo1F 1j/Um29ZFUa/qNAjENKa3ULqdKUNzO8+Y2fu0dceHXScMiwAdbUezYQhN2lktlSHt8R5aBhN856 DcmWqT8WFOQMFI45fLoTxZ63QgGYW7P4QIjNIscl9gwTfaUMrv7dsaUszJPW6/l8aXNdxEiAfPB P622VDlQ3XNU7kygY2zbw7aJgr2+O6m719dOO2HQR0YRZ19T2pP8SDfgppYrGxFue8RweDiwOvm edbKQG/HcFa6UQg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net This patch splits the check_btf_info's check_btf_func check into two separate phases. The first phase sets up the BTF and prepares func_info, but does not perform any validation of required invariants for subprogs just yet. This is left to the second phase, which happens where check_btf_info executes currently, and performs the line_info and CO-RE relocation. The reason to perform this split is to obtain the userspace supplied func_info information before we perform the add_subprog call, where we would now require finding and adding subprogs that may not have a bpf_pseudo_call or bpf_pseudo_func instruction in the program. We require this as we want to enable userspace to supply exception callbacks that can override the default hidden subprogram generated by the verifier (which performs a hardcoded action). In such a case, the exception callback may never be referenced in an instruction, but will still be suitably annotated (by way of BTF declaration tags). For finding this exception callback, we would require the program's BTF information, and the supplied func_info information which maps BTF type IDs to subprograms. Since the exception callback won't actually be referenced through instructions, later checks in check_cfg and do_check_subprogs will not verify the subprog. This means that add_subprog needs to add them in the add_subprog_and_kfunc phase before we move forward, which is why the BTF and func_info are required at that point. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 128 +++++++++++++++++++++++++++++++++--------- 1 file changed, 100 insertions(+), 28 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2ac0be088dd5..d0f6c984272b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15013,20 +15013,18 @@ static int check_abnormal_return(struct bpf_verifier_env *env) #define MIN_BPF_FUNCINFO_SIZE 8 #define MAX_FUNCINFO_REC_SIZE 252 -static int check_btf_func(struct bpf_verifier_env *env, - const union bpf_attr *attr, - bpfptr_t uattr) +static int check_btf_func_early(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) { - const struct btf_type *type, *func_proto, *ret_type; - u32 i, nfuncs, urec_size, min_size; u32 krec_size = sizeof(struct bpf_func_info); + const struct btf_type *type, *func_proto; + u32 i, nfuncs, urec_size, min_size; struct bpf_func_info *krecord; - struct bpf_func_info_aux *info_aux = NULL; struct bpf_prog *prog; const struct btf *btf; - bpfptr_t urecord; u32 prev_offset = 0; - bool scalar_return; + bpfptr_t urecord; int ret = -ENOMEM; nfuncs = attr->func_info_cnt; @@ -15036,11 +15034,6 @@ static int check_btf_func(struct bpf_verifier_env *env, return 0; } - if (nfuncs != env->subprog_cnt) { - verbose(env, "number of funcs in func_info doesn't match number of subprogs\n"); - return -EINVAL; - } - urec_size = attr->func_info_rec_size; if (urec_size < MIN_BPF_FUNCINFO_SIZE || urec_size > MAX_FUNCINFO_REC_SIZE || @@ -15058,9 +15051,6 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord = kvcalloc(nfuncs, krec_size, GFP_KERNEL | __GFP_NOWARN); if (!krecord) return -ENOMEM; - info_aux = kcalloc(nfuncs, sizeof(*info_aux), GFP_KERNEL | __GFP_NOWARN); - if (!info_aux) - goto err_free; for (i = 0; i < nfuncs; i++) { ret = bpf_check_uarg_tail_zero(urecord, krec_size, urec_size); @@ -15099,11 +15089,6 @@ static int check_btf_func(struct bpf_verifier_env *env, goto err_free; } - if (env->subprog_info[i].start != krecord[i].insn_off) { - verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n"); - goto err_free; - } - /* check type_id */ type = btf_type_by_id(btf, krecord[i].type_id); if (!type || !btf_type_is_func(type)) { @@ -15111,12 +15096,80 @@ static int check_btf_func(struct bpf_verifier_env *env, krecord[i].type_id); goto err_free; } - info_aux[i].linkage = BTF_INFO_VLEN(type->info); func_proto = btf_type_by_id(btf, type->type); if (unlikely(!func_proto || !btf_type_is_func_proto(func_proto))) /* btf_func_check() already verified it during BTF load */ goto err_free; + + prev_offset = krecord[i].insn_off; + bpfptr_add(&urecord, urec_size); + } + + prog->aux->func_info = krecord; + prog->aux->func_info_cnt = nfuncs; + return 0; + +err_free: + kvfree(krecord); + return ret; +} + +static int check_btf_func(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) +{ + const struct btf_type *type, *func_proto, *ret_type; + u32 i, nfuncs, urec_size, min_size; + u32 krec_size = sizeof(struct bpf_func_info); + struct bpf_func_info *krecord; + struct bpf_func_info_aux *info_aux = NULL; + struct bpf_prog *prog; + const struct btf *btf; + bpfptr_t urecord; + u32 prev_offset = 0; + bool scalar_return; + int ret = -ENOMEM; + + nfuncs = attr->func_info_cnt; + if (!nfuncs) { + if (check_abnormal_return(env)) + return -EINVAL; + return 0; + } + if (nfuncs != env->subprog_cnt) { + verbose(env, "number of funcs in func_info doesn't match number of subprogs\n"); + return -EINVAL; + } + + urec_size = attr->func_info_rec_size; + + prog = env->prog; + btf = prog->aux->btf; + + urecord = make_bpfptr(attr->func_info, uattr.is_kernel); + min_size = min_t(u32, krec_size, urec_size); + + krecord = prog->aux->func_info; + info_aux = kcalloc(nfuncs, sizeof(*info_aux), GFP_KERNEL | __GFP_NOWARN); + if (!info_aux) + return -ENOMEM; + + for (i = 0; i < nfuncs; i++) { + /* check insn_off */ + ret = -EINVAL; + + if (env->subprog_info[i].start != krecord[i].insn_off) { + verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n"); + goto err_free; + } + + /* Already checked type_id */ + type = btf_type_by_id(btf, krecord[i].type_id); + info_aux[i].linkage = BTF_INFO_VLEN(type->info); + /* Already checked func_proto */ + func_proto = btf_type_by_id(btf, type->type); + ret_type = btf_type_skip_modifiers(btf, func_proto->type, NULL); scalar_return = btf_type_is_small_int(ret_type) || btf_is_any_enum(ret_type); @@ -15133,13 +15186,10 @@ static int check_btf_func(struct bpf_verifier_env *env, bpfptr_add(&urecord, urec_size); } - prog->aux->func_info = krecord; - prog->aux->func_info_cnt = nfuncs; prog->aux->func_info_aux = info_aux; return 0; err_free: - kvfree(krecord); kfree(info_aux); return ret; } @@ -15357,9 +15407,9 @@ static int check_core_relo(struct bpf_verifier_env *env, return err; } -static int check_btf_info(struct bpf_verifier_env *env, - const union bpf_attr *attr, - bpfptr_t uattr) +static int check_btf_info_early(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) { struct btf *btf; int err; @@ -15379,6 +15429,24 @@ static int check_btf_info(struct bpf_verifier_env *env, } env->prog->aux->btf = btf; + err = check_btf_func_early(env, attr, uattr); + if (err) + return err; + return 0; +} + +static int check_btf_info(struct bpf_verifier_env *env, + const union bpf_attr *attr, + bpfptr_t uattr) +{ + int err; + + if (!attr->func_info_cnt && !attr->line_info_cnt) { + if (check_abnormal_return(env)) + return -EINVAL; + return 0; + } + err = check_btf_func(env, attr, uattr); if (err) return err; @@ -19842,6 +19910,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 if (!env->explored_states) goto skip_full_check; + ret = check_btf_info_early(env, attr, uattr); + if (ret < 0) + goto skip_full_check; + ret = add_subprog_and_kfunc(env); if (ret < 0) goto skip_full_check; From patchwork Wed Aug 9 11:41:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347821 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C049D17AB8 for ; Wed, 9 Aug 2023 11:42:57 +0000 (UTC) Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AEF18210C for ; Wed, 9 Aug 2023 04:42:52 -0700 (PDT) Received: by mail-ej1-x642.google.com with SMTP id a640c23a62f3a-99c353a395cso927761766b.2 for ; Wed, 09 Aug 2023 04:42:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581370; x=1692186170; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UXcswezpT6mGJeYdhmBOBcDAXUG/QIQG0ZIMU88BxEU=; b=QKa6vLaTDAnevI2KuK0buVeHN2eL1jzM8n+NGCW9W7VQQS0U3O6GVEXkkDgicHbSKR j1l8WOgLw7Cf6tCUHoAYldo4mEZYP1G/CjETwLyNG8UjPh8J0QIoVcswfdLVG90fIWQN /I0hVh2/XvkXSOc+p5O2svA3mDHTaOYucYYnH+ypXxDr3/QvEc9pTHOqanMhIbWZSDSV OC8BLVB8wHA45x2IPV6QRbPAlH+3Ke/hRt62h1cXmwaW+hUWogoGAQg9/zEScr+IRPo8 U8r12VleoYmkXKPE1wvwFFGspvGzkVQgotcIrVcgWsrTpIR86PGGC8PvFpVvGmuGBMtf ROtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581370; x=1692186170; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UXcswezpT6mGJeYdhmBOBcDAXUG/QIQG0ZIMU88BxEU=; b=Tf3HCynajw1zS+25b4Db5gZKgi8finQ0B5a/KFO/gmdwAK9BoLkYWUysT4oyzj99XZ OHH0s7o26k8a4iTO9DeUVQGVb58z1p4pup0KKxRCFAsWvqb1+sofXbtKscpEkIQnpiDB Pwm7skBnW1Gk8qyMwyC++cUHw9k6GjpI0TwNyCQ8ptCebQdJkGicLd8gesWl6WP/eupq j9aZoQ/xfzKcGO3G/RpfMYapHCx6Z5msN2porJM8/3KS476lb0xVuVI56/oo5g77gzZ8 z4wX0ebNZ80RE/0xKy7UeuVSkF9gmcEtFfreEB7DqwwtmCoW/YXL7rAxQgAJ/GdTKrn5 Nbuw== X-Gm-Message-State: AOJu0Yzd0dr06wyY8r3yr6D3Dy7U5ttOe75CN5VJ+L4LhAkAB2FVhz0U UNN/7RT+AIub83iNit+aHqZMcy9qYCSxQdCDQYY= X-Google-Smtp-Source: AGHT+IEwfzGRH8aRChEN2gBKHbWqaI7WvDc53PZgDw9clXokBT8BvN9RtCfzQ26URP761vrSiMfdZg== X-Received: by 2002:a17:906:8a77:b0:99b:f03d:de37 with SMTP id hy23-20020a1709068a7700b0099bf03dde37mr1793712ejc.60.1691581369932; Wed, 09 Aug 2023 04:42:49 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id h4-20020a1709062dc400b0099bc2d1429csm7917345eji.72.2023.08.09.04.42.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:42:49 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 05/14] bpf: Add support for custom exception callbacks Date: Wed, 9 Aug 2023 17:11:07 +0530 Message-ID: <20230809114116.3216687-6-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=15119; i=memxor@gmail.com; h=from:subject; bh=yqSxAz0oCDyglBWsfGPLR+MGNEhMCs4XTlknsgcty00=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rIjUHN61Ov4JN9rFYz+h7JEBcWumPPXsZN/ g8FpcvCdZiJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yAAKCRBM4MiGSL8R ygP1D/oCML2IlBzUzyuRXYhDR8/5YocCsW4YxlAjtLOQvS72b1K9WOC5z4qsEhNSA19d/MUBLcu Va/PjVlXhlH79h8pDfKiWIhGaEic6vCoG+X/fTsDsxxVERWa87DOD7OQwdbIQDBiz/Ph2HuEUEg fxFngeaLicmC4rp6pNY9wCebda+uFCbPX+MlispjEg58YBuTHMf3AK3FIsvEK44p81QXpV3HwXG W6LeX5bSoog5E1FrWB7B6bAj+6EoCzAu41EPnCdh92/shbJHL2diC3Y/mUlzBSrSaEAX3y6EGwu d/QZIIWUTsOp4/3fDaX9FejgsL83lF0BZ2TPzzrosydiAX6dejuwodcYTSMoM1J0gAgdE0VQPLH 26yEkCRBMs8Fv0w8Os05sxdSLmYNasKrR6CUAd+Dzjuw2JyiSv43uN+FG9HICRh6hdhgwe7favp VRjcTZmK2j5tbNpC7RBuS/AXtwyDnogCjyHDIIK8SLz19g8qq71hmFkMyDxYzrZrOJF9/tKQaNh AKg8Qrj/EL5K3bCITyh4sLbpTX7f0Yk/zheaef5mZqomsF7vz/pewpzXaUOEPaDtaKOUYVy6Csc 6UfZVVr0Isp06aSsW5L/utE1KJ0kt71ZHxriMp/7kHQDpzrP4IRtOo6X2crxV7TmjXnAMgjf3x7 53Bow+rFJHS3/8A== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net By default, the subprog generated by the verifier to handle a thrown exception hardcodes a return value of 0. To allow user-defined logic and modification of the return value when an exception is thrown, introduce the 'exception_callback:' declaration tag, which marks a callback as the default exception handler for the program. The format of the declaration tag is 'exception_callback:', where is the name of the exception callback. Each main program can be tagged using this BTF declaratiion tag to associate it with an exception callback. In case the tag is absent, the default callback is used. As such, the exception callback cannot be modified at runtime, only set during verification. Allowing modification of the callback for the current program execution at runtime leads to issues when the programs begin to nest, as any per-CPU state maintaing this information will have to be saved and restored. We don't want it to stay in bpf_prog_aux as this takes a global effect for all programs. An alternative solution is spilling the callback pointer at a known location on the program stack on entry, and then passing this location to bpf_throw as a parameter. However, since exceptions are geared more towards a use case where they are ideally never invoked, optimizing for this use case and adding to the complexity has diminishing returns. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 4 +- include/linux/bpf_verifier.h | 1 + kernel/bpf/btf.c | 29 +++-- kernel/bpf/verifier.c | 118 ++++++++++++++++-- .../testing/selftests/bpf/bpf_experimental.h | 31 ++++- 5 files changed, 165 insertions(+), 18 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e938e75b0998..2125d77ce2e1 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -2364,9 +2364,11 @@ int btf_check_subprog_arg_match(struct bpf_verifier_env *env, int subprog, int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog, struct bpf_reg_state *regs); int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, - struct bpf_reg_state *reg); + struct bpf_reg_state *reg, bool is_ex_cb); int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog, struct btf *btf, const struct btf_type *t); +const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt, + int comp_idx, const char *tag_key); struct bpf_prog *bpf_prog_by_id(u32 id); struct bpf_link *bpf_link_by_id(u32 id); diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 9e6c25ecac9f..801ada8e614e 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -300,6 +300,7 @@ struct bpf_func_state { bool in_callback_fn; struct tnum callback_ret_range; bool in_async_callback_fn; + bool in_exception_callback_fn; /* The following fields should be last. See copy_func_state() */ int acquired_refs; diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 249657c466dd..8da0eac3dcbd 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3308,10 +3308,10 @@ static int btf_find_kptr(const struct btf *btf, const struct btf_type *t, return BTF_FIELD_FOUND; } -static const char *btf_find_decl_tag_value(const struct btf *btf, - const struct btf_type *pt, - int comp_idx, const char *tag_key) +const char *btf_find_decl_tag_value(const struct btf *btf, const struct btf_type *pt, + int comp_idx, const char *tag_key) { + const char *value = NULL; int i; for (i = 1; i < btf_nr_types(btf); i++) { @@ -3325,9 +3325,14 @@ static const char *btf_find_decl_tag_value(const struct btf *btf, continue; if (strncmp(__btf_name_by_offset(btf, t->name_off), tag_key, len)) continue; - return __btf_name_by_offset(btf, t->name_off) + len; + /* Prevent duplicate entries for same type */ + if (value) + return ERR_PTR(-EEXIST); + value = __btf_name_by_offset(btf, t->name_off) + len; } - return NULL; + if (!value) + return ERR_PTR(-ENOENT); + return value; } static int @@ -3345,7 +3350,7 @@ btf_find_graph_root(const struct btf *btf, const struct btf_type *pt, if (t->size != sz) return BTF_FIELD_IGNORE; value_type = btf_find_decl_tag_value(btf, pt, comp_idx, "contains:"); - if (!value_type) + if (IS_ERR(value_type)) return -EINVAL; node_field_name = strstr(value_type, ":"); if (!node_field_name) @@ -6949,7 +6954,7 @@ int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog, * (either PTR_TO_CTX or SCALAR_VALUE). */ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, - struct bpf_reg_state *regs) + struct bpf_reg_state *regs, bool is_ex_cb) { struct bpf_verifier_log *log = &env->log; struct bpf_prog *prog = env->prog; @@ -7006,7 +7011,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, tname, nargs, MAX_BPF_FUNC_REG_ARGS); return -EINVAL; } - /* check that function returns int */ + /* check that function returns int, exception cb also requires this */ t = btf_type_by_id(btf, t->type); while (btf_type_is_modifier(t)) t = btf_type_by_id(btf, t->type); @@ -7055,6 +7060,14 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, i, btf_type_str(t), tname); return -EINVAL; } + /* We have already ensured that the callback returns an integer, just + * like all global subprogs. We need to determine it only has a single + * scalar argument. + */ + if (is_ex_cb && (nargs != 1 || regs[BPF_REG_1].type != SCALAR_VALUE)) { + bpf_log(log, "exception cb only supports single integer argument\n"); + return -EINVAL; + } return 0; } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d0f6c984272b..9d67d0633c59 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2457,6 +2457,73 @@ static int add_subprog(struct bpf_verifier_env *env, int off) return env->subprog_cnt - 1; } +static int bpf_find_exception_callback_insn_off(struct bpf_verifier_env *env) +{ + struct bpf_prog_aux *aux = env->prog->aux; + struct btf *btf = aux->btf; + const struct btf_type *t; + const char *name; + u32 main_btf_id; + int ret, i, j; + + /* Non-zero func_info_cnt implies valid btf */ + if (!aux->func_info_cnt) + return 0; + main_btf_id = aux->func_info[0].type_id; + + t = btf_type_by_id(btf, main_btf_id); + if (!t) { + verbose(env, "invalid btf id for main subprog in func_info\n"); + return -EINVAL; + } + + name = btf_find_decl_tag_value(btf, t, -1, "exception_callback:"); + if (IS_ERR(name)) { + ret = PTR_ERR(name); + /* If there is no tag present, there is no exception callback */ + if (ret == -ENOENT) + ret = 0; + else if (ret == -EEXIST) + verbose(env, "multiple exception callback tags for main subprog\n"); + return ret; + } + + ret = -ENOENT; + for (i = 0; i < btf_nr_types(btf); i++) { + t = btf_type_by_id(btf, i); + if (!btf_type_is_func(t)) + continue; + if (strcmp(name, btf_name_by_offset(btf, t->name_off))) + continue; + if (btf_func_linkage(t) != BTF_FUNC_GLOBAL) { + verbose(env, "exception callback '%s' must have global linkage\n", name); + return -EINVAL; + } + + ret = 0; + for (j = 0; j < aux->func_info_cnt; j++) { + if (aux->func_info[j].type_id != i) + continue; + ret = aux->func_info[j].insn_off; + /* Further func_info and subprog checks will also happen + * later, so assume this is the right insn_off for now. + */ + if (!ret) { + verbose(env, "invalid exception callback insn_off in func_info: 0\n"); + ret = -EINVAL; + } + } + if (!ret) { + verbose(env, "exception callback type id not found in func_info\n"); + ret = -EINVAL; + } + break; + } + if (ret == -ENOENT) + verbose(env, "exception callback '%s' could not be found in BTF\n", name); + return ret; +} + #define MAX_KFUNC_DESCS 256 #define MAX_KFUNC_BTFS 256 @@ -2796,8 +2863,8 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog, static int add_subprog_and_kfunc(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprog = env->subprog_info; + int i, ret, insn_cnt = env->prog->len, ex_cb_insn; struct bpf_insn *insn = env->prog->insnsi; - int i, ret, insn_cnt = env->prog->len; /* Add entry function. */ ret = add_subprog(env, 0); @@ -2823,6 +2890,26 @@ static int add_subprog_and_kfunc(struct bpf_verifier_env *env) return ret; } + ret = bpf_find_exception_callback_insn_off(env); + if (ret < 0) + return ret; + ex_cb_insn = ret; + + /* If ex_cb_insn > 0, this means that the main program has a subprog + * marked using BTF decl tag to serve as the exception callback. + */ + if (ex_cb_insn) { + ret = add_subprog(env, ex_cb_insn); + if (ret < 0) + return ret; + for (i = 1; i < env->subprog_cnt; i++) { + if (env->subprog_info[i].start != ex_cb_insn) + continue; + env->exception_callback_subprog = i; + break; + } + } + /* Add a fake 'exit' subprog which could simplify subprog iteration * logic. 'subprog_cnt' should not be increased. */ @@ -5691,6 +5778,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) /* async callbacks don't increase bpf prog stack size unless called directly */ if (!bpf_pseudo_call(insn + i)) continue; + if (subprog[sidx].is_exception_cb) { + verbose(env, "insn %d cannot call exception cb directly\n", i); + return -EINVAL; + } } i = next_insn; idx = sidx; @@ -5712,8 +5803,13 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) * tail call counter throughout bpf2bpf calls combined with tailcalls */ if (tail_call_reachable) - for (j = 0; j < frame; j++) + for (j = 0; j < frame; j++) { + if (subprog[ret_prog[j]].is_exception_cb) { + verbose(env, "cannot tail call within exception cb\n"); + return -EINVAL; + } subprog[ret_prog[j]].tail_call_reachable = true; + } if (subprog[0].tail_call_reachable) env->prog->aux->tail_call_reachable = true; @@ -14528,7 +14624,7 @@ static int check_return_code(struct bpf_verifier_env *env) const bool is_subprog = frame->subprogno; /* LSM and struct_ops func-ptr's return type could be "void" */ - if (!is_subprog) { + if (!is_subprog || frame->in_exception_callback_fn) { switch (prog_type) { case BPF_PROG_TYPE_LSM: if (prog->expected_attach_type == BPF_LSM_CGROUP) @@ -14576,7 +14672,7 @@ static int check_return_code(struct bpf_verifier_env *env) return 0; } - if (is_subprog) { + if (is_subprog && !frame->in_exception_callback_fn) { if (reg->type != SCALAR_VALUE) { verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", reg_type_str(env, reg->type)); @@ -19189,7 +19285,7 @@ static void free_states(struct bpf_verifier_env *env) } } -static int do_check_common(struct bpf_verifier_env *env, int subprog) +static int do_check_common(struct bpf_verifier_env *env, int subprog, bool is_ex_cb) { bool pop_log = !(env->log.level & BPF_LOG_LEVEL2); struct bpf_verifier_state *state; @@ -19220,7 +19316,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) regs = state->frame[state->curframe]->regs; if (subprog || env->prog->type == BPF_PROG_TYPE_EXT) { - ret = btf_prepare_func_args(env, subprog, regs); + ret = btf_prepare_func_args(env, subprog, regs, is_ex_cb); if (ret) goto out; for (i = BPF_REG_1; i <= BPF_REG_5; i++) { @@ -19236,6 +19332,12 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog) regs[i].id = ++env->id_gen; } } + if (is_ex_cb) { + state->frame[0]->in_exception_callback_fn = true; + env->subprog_info[subprog].is_cb = true; + env->subprog_info[subprog].is_async_cb = true; + env->subprog_info[subprog].is_exception_cb = true; + } } else { /* 1st arg to a function */ regs[BPF_REG_1].type = PTR_TO_CTX; @@ -19300,7 +19402,7 @@ static int do_check_subprogs(struct bpf_verifier_env *env) continue; env->insn_idx = env->subprog_info[i].start; WARN_ON_ONCE(env->insn_idx == 0); - ret = do_check_common(env, i); + ret = do_check_common(env, i, env->exception_callback_subprog == i); if (ret) { return ret; } else if (env->log.level & BPF_LOG_LEVEL) { @@ -19317,7 +19419,7 @@ static int do_check_main(struct bpf_verifier_env *env) int ret; env->insn_idx = 0; - ret = do_check_common(env, 0); + ret = do_check_common(env, 0, false); if (!ret) env->prog->aux->stack_depth = env->subprog_info[0].stack_depth; return ret; diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 952a40cbe09c..612ac86873af 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -134,7 +134,16 @@ extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym; /* Description * Throw a BPF exception from the program, immediately terminating its * execution and unwinding the stack. The supplied 'cookie' parameter - * will be the return value of the program when an exception is thrown. + * will be the return value of the program when an exception is thrown, + * and the default exception callback is used. Otherwise, if an exception + * callback is set using the '__exception_cb(callback)' declaration tag + * on the main program, the 'cookie' parameter will be the callback's only + * input argument. + * + * Thus, in case of default exception callback, 'cookie' is subjected to + * constraints on the program's return value (as with R0 on exit). + * Otherwise, the return value of the marked exception callback will be + * subjected to the same checks. * * Note that throwing an exception with lingering resources (locks, * references, etc.) will lead to a verification error. @@ -147,4 +156,24 @@ extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym; */ extern void bpf_throw(u64 cookie) __ksym; +/* This macro must be used to mark the exception callback corresponding to the + * main program. For example: + * + * int exception_cb(u64 cookie) { + * return cookie; + * } + * + * SEC("tc") + * __exception_cb(exception_cb) + * int main_prog(struct __sk_buff *ctx) { + * ... + * return TC_ACT_OK; + * } + * + * Here, exception callback for the main program will be 'exception_cb'. Note + * that this attribute can only be used once, and multiple exception callbacks + * specified for the main program will lead to verification error. + */ +#define __exception_cb(name) __attribute__((btf_decl_tag("exception_callback:" #name))) + #endif From patchwork Wed Aug 9 11:41:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347822 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1212917744 for ; Wed, 9 Aug 2023 11:43:05 +0000 (UTC) Received: from mail-ej1-x641.google.com (mail-ej1-x641.google.com [IPv6:2a00:1450:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EFAB31FEB for ; Wed, 9 Aug 2023 04:43:00 -0700 (PDT) Received: by mail-ej1-x641.google.com with SMTP id a640c23a62f3a-99c10ba30afso172514366b.1 for ; Wed, 09 Aug 2023 04:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581379; x=1692186179; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BSsvbI7ahExjrKursMU/KjRLhJR0LGF6w7wqZb7DClE=; b=MwnF17OxdlSApLkjGfg/tSJRCBBhJRX1+NqDHdn6EIUlBtLd8PbaraOVOHKMrEu27i OwqmDdgwqfElk/z2OUzwcC5TkpC9erIlHyyGbQfhg7FMDRqSXsg0eQUVu7S6/TvILDf0 2dCFzXDabv3p7JovNA5G9ai9ly40cdgrMU/8PMPVuOaWFiqsiWahVOTwodMhDeTdKW11 N+8knHVi+cdIhqqc9BQyl4+zSJSSgmLxPiO+6LsCLoFfOmK+wr6WeqZ+5D4T+5tBbOWA MIfJBGgT+99rQjp2SpB235BNCDyHztOAyXZaJaPaPVUBVfnLXQeQc6iyermXmdkztaXp ZSeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581379; x=1692186179; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BSsvbI7ahExjrKursMU/KjRLhJR0LGF6w7wqZb7DClE=; b=M69mLceXvYK4GMZc4Vo7JR0TezI73KaC52EOXAZ9atwaOboOgLCG0Q+Xt7M9m1Yegq xWGrAUSrfS3/UFhUII+Kjn9LopOnnuT+OBI4S4qfGMAZuJsv11NCxkYriNdT0JGRI37T 2FD18AKKiNqKsvmBYvtEKfsEIabd8quKRe5oK0t7j/L6aAhx3cn1ciwu2xE8MmmAI77P VqsG8I5MHxxCv+XaHmj4PO5gh+V7gnqHS8SQTihqrCUaJ5dfRJe4SkeyGD6G/O3giG0x pm4eUjNj9HSAsmYpbtUITeN2ob8zuF1hnTcTQ7k+/Lv5gtoTeWeqQybxUaLt4ecf+wbT uLJg== X-Gm-Message-State: AOJu0Yw6KIxASwBZnLMwmdFBsQBM2Y8+w0mu7KV5HITo92eYxsts+Ukp asBT0/RNHD4rfHppA9xNe+koNi9vlCru0QFDsfk= X-Google-Smtp-Source: AGHT+IGJRAjlSLkxcQW4atBAZZ5yzl6GRQ2guDKPMf6r+ykTHxto1dhhDM02QtkkwCxuLDNUC20log== X-Received: by 2002:a17:907:b17:b0:99c:ae54:6654 with SMTP id h23-20020a1709070b1700b0099cae546654mr2612066ejl.14.1691581378770; Wed, 09 Aug 2023 04:42:58 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id h11-20020a170906854b00b0099cb1a2cab0sm6512869ejy.28.2023.08.09.04.42.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:42:58 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 06/14] bpf: Perform CFG walk for exception callback Date: Wed, 9 Aug 2023 17:11:08 +0530 Message-ID: <20230809114116.3216687-7-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1909; i=memxor@gmail.com; h=from:subject; bh=LCuy1hed9QK0LVfEql4qQ5HH4jsTZAd6/XT0xD0ZFDY=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rILTjNeyyjihAo6DRyCHUOgdyPQWjE58yxZ VoNZXZ9S1iJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yAAKCRBM4MiGSL8R yvPKD/0TTRlAV07UMeF9+txR1RgAunci/eEnl13fQQ/uqgvNfD56PeN8yFgzLlpSASSOwiGweYA srwzC9V4xDGYhM+aiHjUEgLi1e8FnD7vcdjiyHEMy1p69wmqNOIRxazeq7wftzrZCpIx+luug3G peHzF02RZFX6CyvqMKFFEYAc+JqxiOKX3y9v8VI7VRapYn6ztZ1OpDfSq1+HRg/+09CUtwHZ2qr hpX8+VQI5A3kPT9F5l2ck/xmkS91BZb7TaHGLw5mgJyTtfEWcm5BokhbnVKRvGAmaUC/0aJ/c/s 3+fyauMLoShZJE653YBPVxn8JKWM8xaYvkWWF1Gzx++yasfiF7Fd7zpfxq5U2zeEQyqKzZ0tkYB U3ns/pod18nGFatAaSOwF0DmkxkgU1Qe8x5StNNrOUIANP3gfVAJTox+SH0lkSoII1wvb+JARbm un+MPwYnCHR+ysyI08FLdzq1LxWkwc3exM+DCn7tldwa0SlFHs1ymkTef0A9lnRJ0etO39xiLu3 +xVX0qfTIB0rE0M9tpY4MGfpwZzLCL/vbFdAxoE/1f0lRHBqR3rTxgHdvpc/JkIi0vNIqCK3AgB +td3Vw9nSdD34juN3aE54edlKTHSH18Wj+LYMp9DPlUAGO3fEuV3PL9QFF9Q1iB8lb4Eg4VLWFt +hgDg8dzBG84txg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net Since exception callbacks are not referenced using bpf_pseudo_func and bpf_pseudo_call instructions, check_cfg traversal will never explore instructions of the exception callback. Even after adding the subprog, the program will then fail with a 'unreachable insn' error. We thus need to begin walking from the start of the exception callback again in check_cfg after a complete CFG traversal finishes, so as to explore the CFG rooted at the exception callback. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9d67d0633c59..c22ba0423d27 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -15029,8 +15029,8 @@ static int check_cfg(struct bpf_verifier_env *env) { int insn_cnt = env->prog->len; int *insn_stack, *insn_state; - int ret = 0; - int i; + int ex_insn_beg, i, ret = 0; + bool ex_done = false; insn_state = env->cfg.insn_state = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL); if (!insn_state) @@ -15046,6 +15046,7 @@ static int check_cfg(struct bpf_verifier_env *env) insn_stack[0] = 0; /* 0 is the first instruction */ env->cfg.cur_stack = 1; +walk_cfg: while (env->cfg.cur_stack > 0) { int t = insn_stack[env->cfg.cur_stack - 1]; @@ -15072,6 +15073,16 @@ static int check_cfg(struct bpf_verifier_env *env) goto err_free; } + if (env->exception_callback_subprog && !ex_done) { + ex_insn_beg = env->subprog_info[env->exception_callback_subprog].start; + + insn_state[ex_insn_beg] = DISCOVERED; + insn_stack[0] = ex_insn_beg; + env->cfg.cur_stack = 1; + ex_done = true; + goto walk_cfg; + } + for (i = 0; i < insn_cnt; i++) { if (insn_state[i] != EXPLORED) { verbose(env, "unreachable insn %d\n", i); From patchwork Wed Aug 9 11:41:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347823 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2A1717744 for ; Wed, 9 Aug 2023 11:43:10 +0000 (UTC) Received: from mail-ed1-x542.google.com (mail-ed1-x542.google.com [IPv6:2a00:1450:4864:20::542]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6930518E for ; Wed, 9 Aug 2023 04:43:09 -0700 (PDT) Received: by mail-ed1-x542.google.com with SMTP id 4fb4d7f45d1cf-52364e9daceso795634a12.2 for ; Wed, 09 Aug 2023 04:43:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581387; x=1692186187; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7RQCLZLUwoM8WdFwA8DnuZn0TAGNZQI5hb8I+mfQGA4=; b=Ncxyacyly6uz0VU1hJK7dtSeHjhaCQdT3B8yvgzhRNcmeWzv7/E/xTos0WT/KxSOaW ZwgMOywsYg2swEYw/YhwF6qjwQYnLsfoE41jimpUsvEdfT+ysdXUGCjWLvosJ4MrfK+B sL4mwGA6psSfbDhbO2ovcfEqDoXMeZmaLWXRDIgRZk/wIP+vTiW8rW4wiNyWz2eOE6sK HjWZI+/GumhD09CKXBKS23w0e4WcUGRNvICVo/bvmmDpNDAsFanZWq+6h29GxBH0DkTq wBIFdrOw3z1vBXzamIM3kdc7xMW6V1FuKAgFfMZPQQigVhLPYTTJ1sOYH3u5BHcWpgx9 x61g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581387; x=1692186187; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7RQCLZLUwoM8WdFwA8DnuZn0TAGNZQI5hb8I+mfQGA4=; b=LjpV8CpZK+yTyNb8XnXxiqxdauxQvmR9KD6l+NkBMRzTWcWJTIi9nmVcKUGAcuJGu/ nVIytOxfJFxsj+do+g+QVzhKD0uqKxNm1zjWCw+Q9axuompqPuXwW1r4lZ+YjzzApJEl z6lI/RPKubWm+Q67qmD0gv1LzMDEPNF4JvND0ks+DFXXkEwmkOhaKPpZLM10C1bDtR33 NpL/kQ3lfQ5tWZKku5WFXEtFf2GsAVyqZNoWGIKMvsTDrGOtce/GYTwj3SKpzxnD/FOn Kr4DnlIkKuR0xmehUK5RnutW9dIACD6Sczr2AOcgysMC7AgKcQxTkE0crPP4Do4Uh4cn C1KA== X-Gm-Message-State: AOJu0Ywmz5Oid8QXe+iz1ZfpIWosMtBJKZ2hTEVGZzrqYF2fMF6HZOV4 K2KzMxRjq8p1BXrVsRfk5NwFQLfu3V0fM5ErKYs= X-Google-Smtp-Source: AGHT+IF7MrE8lKvGeGtZg6/sefQWO+Iwz2U2XJVu/Zj73XsDRjGKeXPBGOjJ9qu6x0JtQK3tXKUQtA== X-Received: by 2002:a17:906:1c7:b0:997:c377:b41f with SMTP id 7-20020a17090601c700b00997c377b41fmr2121702ejj.64.1691581386728; Wed, 09 Aug 2023 04:43:06 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id a14-20020a17090682ce00b0099d0c0bb92bsm809631ejy.80.2023.08.09.04.43.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:43:06 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 07/14] bpf: Treat first argument as return value for bpf_throw Date: Wed, 9 Aug 2023 17:11:09 +0530 Message-ID: <20230809114116.3216687-8-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4675; i=memxor@gmail.com; h=from:subject; bh=12P3B/yYYq41xWHnsCoi5CZR1+UBgMaPfc02X/5Pb7w=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJ+OTSgB1Qy1gGKT963T+b6XCHwbs4qjqkO Abjzv0wR4KJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R ypVGD/4lsgY2+/M2+LTKK0k3axTZ63/CBVh+1nJ5VmPCY47sP3McMVGekxN9HBnw3f6j+fOA6w7 2wrjEgVB8osIQfEYIgDJUrgfRFwyi6WnbOLkG2VkvFngTOlpqfnHMSXvyuAFepI0Bjb7S8DzeoW 7GNoiXUHs8NDqsz0OrWPO9iK3WOTdPCvwlbkbZ1RgqLu2aefcagKOADUbwpKKxwTewQ3UwPPtMD KhuQ+a++ed+dljbdiIWpn1P+VcUWYtvZonYrldVL0V4Do52e5cs27Oc/0YwkJzifeZuBE1M7OYH 8DqCJoYkRQmNZDv/7LWJHIgZtUUGZ7BiMNf8g7BExU5LYs4W2nzzlljhkclqFh8GhNf7gGjEXhA Ts8Z8K2D11cxd+07VbRBiu70A1BPD7GzdnGjMgDbY/jTi/v3IdPKRwGIqIH2/RxdLiCw33lcjVj mIOMT96tulT8ZceyRnkxRj/bhKQruolntkLhF0AAtQqV+uCKx0WFOoKXACKlNNVCrBsgsBes3BB HwTvh925q9i5h4HB5J/MegeVolZUQo7HwvBCc/CvW3iuw4EhcIgbZ2Fvc7dbRAck2fZIf9xO8cC 7vqx7lnWhKb5hNdavL2TI6uvYbZNb1QfHTStdarOmq8DjK6KRYxl+4YwRjNE3WJI3n6lq0bRIev 49cEwZkvG/fbH7g== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net In case of the default exception callback, change the behavior of bpf_throw, where the passed cookie value is no longer ignored, but is instead the return value of the default exception callback. As such, we need to place restrictions on the value being passed into bpf_throw in such a case, only allowing those permitted by the check_return_code function. Thus, bpf_throw can now control the return value of the program from each call site without having the user install a custom exception callback just to override the return value when an exception is thrown. We also modify the hidden subprog instructions to now move BPF_REG_1 to BPF_REG_0, so as to set the return value before exit in the default callback. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index c22ba0423d27..a0e1a1d1f5d3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11415,6 +11415,8 @@ static int fetch_kfunc_meta(struct bpf_verifier_env *env, return 0; } +static int check_return_code(struct bpf_verifier_env *env, int regno); + static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx_p) { @@ -11538,6 +11540,15 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return -ENOTSUPP; } env->seen_exception = true; + + /* In the case of the default callback, the cookie value passed + * to bpf_throw becomes the return value of the program. + */ + if (!env->exception_callback_subprog) { + err = check_return_code(env, BPF_REG_1); + if (err < 0) + return err; + } } for (i = 0; i < CALLER_SAVED_REGS; i++) @@ -14612,7 +14623,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return 0; } -static int check_return_code(struct bpf_verifier_env *env) +static int check_return_code(struct bpf_verifier_env *env, int regno) { struct tnum enforce_attach_type_range = tnum_unknown; const struct bpf_prog *prog = env->prog; @@ -14646,22 +14657,22 @@ static int check_return_code(struct bpf_verifier_env *env) * of bpf_exit, which means that program wrote * something into it earlier */ - err = check_reg_arg(env, BPF_REG_0, SRC_OP); + err = check_reg_arg(env, regno, SRC_OP); if (err) return err; - if (is_pointer_value(env, BPF_REG_0)) { - verbose(env, "R0 leaks addr as return value\n"); + if (is_pointer_value(env, regno)) { + verbose(env, "R%d leaks addr as return value\n", regno); return -EACCES; } - reg = cur_regs(env) + BPF_REG_0; + reg = cur_regs(env) + regno; if (frame->in_async_callback_fn) { /* enforce return zero from async callbacks like timer */ if (reg->type != SCALAR_VALUE) { - verbose(env, "In async callback the register R0 is not a known value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "In async callback the register R%d is not a known value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } @@ -14674,8 +14685,8 @@ static int check_return_code(struct bpf_verifier_env *env) if (is_subprog && !frame->in_exception_callback_fn) { if (reg->type != SCALAR_VALUE) { - verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } return 0; @@ -14757,8 +14768,8 @@ static int check_return_code(struct bpf_verifier_env *env) } if (reg->type != SCALAR_VALUE) { - verbose(env, "At program exit the register R0 is not a known value (%s)\n", - reg_type_str(env, reg->type)); + verbose(env, "At program exit the register R%d is not a known value (%s)\n", + regno, reg_type_str(env, reg->type)); return -EINVAL; } @@ -16955,7 +16966,7 @@ static int do_check(struct bpf_verifier_env *env) continue; } - err = check_return_code(env); + err = check_return_code(env, BPF_REG_0); if (err) return err; process_bpf_exit: @@ -18601,7 +18612,7 @@ static int do_misc_fixups(struct bpf_verifier_env *env) if (env->seen_exception && !env->exception_callback_subprog) { struct bpf_insn patch[] = { env->prog->insnsi[insn_cnt - 1], - BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_MOV64_REG(BPF_REG_0, BPF_REG_1), BPF_EXIT_INSN(), }; From patchwork Wed Aug 9 11:41:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347824 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA16B17744 for ; Wed, 9 Aug 2023 11:43:24 +0000 (UTC) Received: from mail-ej1-x644.google.com (mail-ej1-x644.google.com [IPv6:2a00:1450:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 889EF10C0 for ; Wed, 9 Aug 2023 04:43:21 -0700 (PDT) Received: by mail-ej1-x644.google.com with SMTP id a640c23a62f3a-99bf1f632b8so996315666b.1 for ; Wed, 09 Aug 2023 04:43:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581399; x=1692186199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=axxrwc04eHcvGTpC0q/DJGjJwM+C0owU3pNAm66KQhk=; b=WyDBZQMcWKnJ1XVLqTQfo6l59lCmeBQL4xd0mXI7J3hSQX393X6ximAKe/J6I/EtB2 UIuXYUYLpm4pkShdENh7ms+Db5eyj1yOssiMbcfveDHWXqUhcoixr7cGMqOZUBfSC9Vf 2HYVXYYWEjvI5ndlcQM4zYgtLt6OlVzqVJAimUYK6GVrboQ1UJICw7dMJGQWPfGkEzW+ xjx7VIL3Q+Pa+rcrAvACd8MbF5ZkDjnKorc2CqTL44lqhxhoAA1Ka89oBdvTevtw8AKg H5s6eO13vexGKAcpf7e2cqWZpbxWMLEbG9caFf1jbqMuyTKJQfULtWpF38yBP2LPSg/y YyFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581399; x=1692186199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=axxrwc04eHcvGTpC0q/DJGjJwM+C0owU3pNAm66KQhk=; b=POiYb/9Iyr71WrgGimZe9ZKdG9KPmleNIE3SIV5GI2gJvwRUfC90dVizSc5oZB5V5K Rc1ERviROxz3K96VXZr9iGVQSH24NCLNo0nLRht+GUVzc+OH0o6yCtnSlbHN5JqwPjAZ UF112GRsdcMKjqcu50LO6vW6W+O1KnFER10GBBQA6056x0Y+iomrAGGoKQX5LnvB5+/w K1/ZnEVIywRgIM5MlQjNcDH5JrVqRRaJYCn1HVIZ9QbxZ+1kgsPqBEFPZVazP09QCshH G0ZQDkbv6t+Z4EPbQE0g6SEq3xbwdF3qsgDEF27xWQRhvkHKRCWmmHaAd8u6ExkDJdeo y79g== X-Gm-Message-State: AOJu0YwFuO6pKFOdiaeUROpO4zX67Z3wvMQ9FD1xi813iWemWgPunrOK txzRZL6xT2lnJQkouosAD6yOEL6JjgFJ3TEMu1g= X-Google-Smtp-Source: AGHT+IHC21KiuUPuPvpbaEXhWuzm4AG60VyrA7gFE9yRWY4PDfkO7n3NkWAPLbu3T64nZ7SLQ/iKDg== X-Received: by 2002:a17:907:75d4:b0:99c:3bb:fd96 with SMTP id jl20-20020a17090775d400b0099c03bbfd96mr1851965ejc.52.1691581399386; Wed, 09 Aug 2023 04:43:19 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id s12-20020a1709067b8c00b00992b50fbbe9sm7949863ejo.90.2023.08.09.04.43.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:43:18 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 08/14] bpf: Prevent KASAN false positive with bpf_throw Date: Wed, 9 Aug 2023 17:11:10 +0530 Message-ID: <20230809114116.3216687-9-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=6190; i=memxor@gmail.com; h=from:subject; bh=NaiGOoo5s//gcTZipLf7STpGdrTi5Ez9y9d5703fRv8=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJl1+9xnrE15CRT3c8NliD+b9w1U/HQ4d5x qPUEY1CbgaJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R yguiEAC3d/ji8X+7OnXnWnBTWy9VcaoC7qV4VcDrOkAoiPQJPdnpHC7syFcivsm41QCGQddg1r/ 4Ra2s42KjFBVlrwiJ1jktHK/6YNb76jc5Nw2TcWYEZfqSHsQ+1P2ib/I3dU8Ancr3hE6guo4IMd b+G+1b06EG6AYbdD4ryEfPjgFQPJn94AmMnN7UZjASvJjwsrQvMeMZCV/MjLVOOjMF1U4CIZbnz y1qqHxl1jlT81f98lrugK36sdUbrCkquwZKNIpm8XfRKdC21MwypzG6D/eHR31criP89YA3CIOZ ZdkJthaspTeSYpmVoHMFDZZCMyf4RW4/dCZT5ZrsqOo8Gxsfeh4shm7PYix50GjTm2k3djBFHwF TK5bcpmD1FagCqRTzUMf2hUGPlS2+IS4shoy5r2hEOvEaB9K0sYdHTCQW7SPPOs+5S3dxMPd1ck 7Zpoq721TNpbbA+ccdOuBT9mb9dU3B39EZyYmthhBd5rlYOhXtEMNGdo2g82pyY98pHG1ntU1oD dxNZyyUPiLh/emM9+QldWkCjzy8hPr4aJn1AkNMgcbeA83hNsHpKwLe06HqeiUHujlQ9QkRnLB5 PnVGBKf4zx85seXeMBtAEh/X6ILp5T6ISuVmhe9fzjcdqRDA4h7qOvl5OpGYCfWLaBYdLYCfF5c +47L+z7OCC/LbZQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net The KASAN stack instrumentation when CONFIG_KASAN_STACK is true poisons the stack of a function when it is entered and unpoisons it when leaving. However, in the case of bpf_throw, we will never return as we switch our stack frame to the BPF exception callback. Later, this discrepancy will lead to confusing KASAN splats when kernel resumes execution on return from the BPF program. Fix this by unpoisoning everything below the stack pointer of the BPF program, which should cover the range that would not be unpoisoned. An example splat is below: BUG: KASAN: stack-out-of-bounds in stack_trace_consume_entry+0x14e/0x170 Write of size 8 at addr ffffc900013af958 by task test_progs/227 CPU: 0 PID: 227 Comm: test_progs Not tainted 6.5.0-rc2-g43f1c6c9052a-dirty #26 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-2.fc39 04/01/2014 Call Trace: dump_stack_lvl+0x4a/0x80 print_report+0xcf/0x670 ? arch_stack_walk+0x79/0x100 kasan_report+0xda/0x110 ? stack_trace_consume_entry+0x14e/0x170 ? stack_trace_consume_entry+0x14e/0x170 ? __pfx_stack_trace_consume_entry+0x10/0x10 stack_trace_consume_entry+0x14e/0x170 ? __sys_bpf+0xf2e/0x41b0 arch_stack_walk+0x8b/0x100 ? __sys_bpf+0xf2e/0x41b0 ? bpf_prog_test_run_skb+0x341/0x1c70 ? bpf_prog_test_run_skb+0x341/0x1c70 stack_trace_save+0x9b/0xd0 ? __pfx_stack_trace_save+0x10/0x10 ? __kasan_slab_free+0x109/0x180 ? bpf_prog_test_run_skb+0x341/0x1c70 ? __sys_bpf+0xf2e/0x41b0 ? __x64_sys_bpf+0x78/0xc0 ? do_syscall_64+0x3c/0x90 ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8 kasan_save_stack+0x33/0x60 ? kasan_save_stack+0x33/0x60 ? kasan_set_track+0x25/0x30 ? kasan_save_free_info+0x2b/0x50 ? __kasan_slab_free+0x109/0x180 ? kmem_cache_free+0x191/0x460 ? bpf_prog_test_run_skb+0x341/0x1c70 kasan_set_track+0x25/0x30 kasan_save_free_info+0x2b/0x50 __kasan_slab_free+0x109/0x180 kmem_cache_free+0x191/0x460 bpf_prog_test_run_skb+0x341/0x1c70 ? __pfx_bpf_prog_test_run_skb+0x10/0x10 ? __fget_light+0x51/0x220 __sys_bpf+0xf2e/0x41b0 ? __might_fault+0xa2/0x170 ? __pfx___sys_bpf+0x10/0x10 ? lock_release+0x1de/0x620 ? __might_fault+0xcd/0x170 ? __pfx_lock_release+0x10/0x10 ? __pfx_blkcg_maybe_throttle_current+0x10/0x10 __x64_sys_bpf+0x78/0xc0 ? syscall_enter_from_user_mode+0x20/0x50 do_syscall_64+0x3c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 RIP: 0033:0x7f0fbb38880d Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d f3 45 12 00 f7 d8 64 89 01 48 RSP: 002b:00007ffe13907de8 EFLAGS: 00000206 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007ffe13908708 RCX: 00007f0fbb38880d RDX: 0000000000000050 RSI: 00007ffe13907e20 RDI: 000000000000000a RBP: 00007ffe13907e00 R08: 0000000000000000 R09: 00007ffe13907e20 R10: 0000000000000064 R11: 0000000000000206 R12: 0000000000000003 R13: 0000000000000000 R14: 00007f0fbb532000 R15: 0000000000cfbd90 The buggy address belongs to stack of task test_progs/227 KASAN internal error: frame info validation failed; invalid marker: 0 The buggy address belongs to the virtual mapping at [ffffc900013a8000, ffffc900013b1000) created by: kernel_clone+0xcd/0x600 The buggy address belongs to the physical page: page:00000000b70f4332 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x11418f flags: 0x2fffe0000000000(node=0|zone=2|lastcpupid=0x7fff) page_type: 0xffffffff() raw: 02fffe0000000000 0000000000000000 dead000000000122 0000000000000000 raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000 page dumped because: kasan: bad access detected Memory state around the buggy address: ffffc900013af800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffc900013af880: 00 00 00 f1 f1 f1 f1 00 00 00 f3 f3 f3 f3 f3 00 >ffffc900013af900: 00 00 00 00 00 00 00 00 00 00 00 f1 00 00 00 00 ^ ffffc900013af980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ffffc900013afa00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ================================================================== Disabling lock debugging due to kernel taint Cc: Andrey Ryabinin Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Dmitry Vyukov Cc: Vincenzo Frascino Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/kasan.h | 2 ++ kernel/bpf/helpers.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 819b6bc8ac08..7a463f814db2 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -283,8 +283,10 @@ static inline bool kasan_check_byte(const void *address) #if defined(CONFIG_KASAN) && defined(CONFIG_KASAN_STACK) void kasan_unpoison_task_stack(struct task_struct *task); +asmlinkage void kasan_unpoison_task_stack_below(const void *watermark); #else static inline void kasan_unpoison_task_stack(struct task_struct *task) {} +static inline void kasan_unpoison_task_stack_below(const void *watermark) {} #endif #ifdef CONFIG_KASAN_GENERIC diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index af4add1e3a31..64a07232c58f 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -22,6 +22,7 @@ #include #include #include +#include #include "../../lib/kstrtox.h" @@ -2463,6 +2464,11 @@ __bpf_kfunc void bpf_throw(u64 cookie) WARN_ON_ONCE(!ctx.aux->exception_boundary); WARN_ON_ONCE(!ctx.bp); WARN_ON_ONCE(!ctx.cnt); + /* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning + * deeper stack depths than ctx.sp as we do not return from bpf_throw, + * which skips compiler generated instrumentation to do the same. + */ + kasan_unpoison_task_stack_below((void *)ctx.sp); ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); } From patchwork Wed Aug 9 11:41:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347825 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 025C917744 for ; Wed, 9 Aug 2023 11:43:36 +0000 (UTC) Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4BFE210B for ; Wed, 9 Aug 2023 04:43:29 -0700 (PDT) Received: by mail-ej1-x642.google.com with SMTP id a640c23a62f3a-99bc9e3cbf1so177195666b.0 for ; Wed, 09 Aug 2023 04:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581408; x=1692186208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+fkgAn42hNtPJ3k0142tc+x/r9LU5OQhsN2tuFGn7vo=; b=Tjx9tk+7OngKag2G+2BrZKmSuA2ktqxjrxjpvivx4R3ccQvKTRRtzLL853rBYqXnUN 1WKthUV2/V8AaOTg0+q3HjpM7Vw8dz7kSxOy+G2abcDQK7Co0L1pgIJgEkTZyalNbLU4 jS3MtEQgFKPdgd97muapoWYj+dkAYtrA+VFHhblae+mDpvib23/oVVe9KzTmx3M+XKJC 0lZrBTAtB52zDBVLdgO/OvOmxENY7gj0lgjankbEAm9sTfKGFUTZv+PmyNClp+6pVXjQ f9k2aAxCGTPwNTj4c3dtgnPDIM1E+IZKcBX6DaCDBcRNVZO0S1Az0mVDpmEH+dudecpu rG0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581408; x=1692186208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+fkgAn42hNtPJ3k0142tc+x/r9LU5OQhsN2tuFGn7vo=; b=GUewB5jUSIq6hBu7EaVEaW0Wtr6qLMkZODWn2y3crJQQYGwW1bHk4pMJ24JT7DBwXv X7LkKq+b+Yu8L6Nv0PaBucsCZ+mqk+YePR84SzfrnKS3Jj+ex+Xbywf8eYZGkCdcuLsB 1uCOnYSJTK6x8jCn/HKuzZNYtagMcnCPQ2SdeRLXgRoWIoE7CyZlaute0Cwm2iScLsrE Xf6m8BPvsJJm57WLVZiRsg5lqR68oIegxyAj0S/K8LJJR2UZRoXMhm6UMqgMxh8gCOQV OSTuowdLXaq6rRzjQuseLl2eS8bySAt29STMf8oHRus0tgaWiI67jAcNzVD7bNYxpePb oEnA== X-Gm-Message-State: AOJu0YyNMkg2PoFX6IZEATHD5ZfdxYYQdi6Xg69GdEsJepjwUTZ8vkG5 kiYHVTDDZe/4IWW/DcZ1ryDbHH4f/02LMUP5nPY= X-Google-Smtp-Source: AGHT+IEG/DHNSvbGx/h1PSlnKx/ln77gEKtQwCiKlLZA0HLuf+q87n+6Vql68NJtuKcWWz1t0vs7ig== X-Received: by 2002:a17:907:86a8:b0:98d:f2c9:a1eb with SMTP id qa40-20020a17090786a800b0098df2c9a1ebmr1532154ejc.24.1691581407666; Wed, 09 Aug 2023 04:43:27 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id n8-20020a1709067b4800b00992b66e54e9sm7805924ejo.214.2023.08.09.04.43.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:43:26 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 09/14] bpf: Detect IP == ksym.end as part of BPF program Date: Wed, 9 Aug 2023 17:11:11 +0530 Message-ID: <20230809114116.3216687-10-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2617; i=memxor@gmail.com; h=from:subject; bh=++XY7wq9bBeKTUhVWFcVuyxVIrhNp7v6Xqks8rPgVAQ=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJQU3cyuG0kAPGov/+td5qxhg43JxrsZ9PV sR+KDnRsIyJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R ynQzD/4xNmpA958LEqi0KcFOlG5RIZh42VVtYZFv7f/OnUyPM81ys2ONyRDaf4UrzVposBK19R+ jROEUPQGdu3hPhclZT6cAIpqvzEG12H2EwWuLXNtgukbqd5KOjsPCrmErwpLcDMjQQIAmmFfNBk gOP+0ohgQxng5tAYJAJ80xLJaNWmsvX30yJy/vSQ11ZFiU1IHpeGZdrIyYIikU3fEQkWYIvO4JK /fTVXOFzEDj2OeE2flOoSH5FwLzEbf6l0zH8l7HZA2RKsnjy+5BlFdrADJCtVHRAZZDiIkp5aHo y22T4Tr2P3zW5gCoDLzhLmUgH99wZFHvOL8D59ZrjKHvlStnIswyhHTy+i+jt18ahyvxXv6CD4s hwTcc2EPaq1fcTE4hMQhLGcGg/MoXpNzvNlPOzdtic+Uqt+F4OqoR6TvijHVCeyxNxsuYb65Cne fbXXQ1hAmYdDOhb5aGGtInEcpAFcGjtqHIZfNbptLRJeaq/NO6nrql8zWr73pGhwkvV2f1pB3ob BAXWds+VXmxRZuhvbm2i8yaVDCPtV2GbYk8YlioL0bTd93HlbN0s9E7Dxrx6MS7qhm2mKQYp04a Bg4dnEAvRPRiouQu4IDTp4gHI+RtUwSdn0fDWi6x3kpt19TqgDJGSlErS9xl657loLvLpYYPVD0 Q8EqmRv9FzWjLHA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net Now that bpf_throw kfunc is the first such call instruction that has noreturn semantics within the verifier, this also kicks in dead code elimination in unprecedented ways. For one, any instruction following a bpf_throw call will never be marked as seen. Moreover, if a callchain ends up throwing, any instructions after the call instruction to the eventually throwing subprog in callers will also never be marked as seen. The tempting way to fix this would be to emit extra 'int3' instructions which bump the jited_len of a program, and ensure that during runtime when a program throws, we can discover its boundaries even if the call instruction to bpf_throw (or to subprogs that always throw) is emitted as the final instruction in the program. An example of such a program would be this: do_something(): ... r0 = 0 exit foo(): r1 = 0 call bpf_throw r0 = 0 exit bar(cond): if r1 != 0 goto pc+2 call do_something exit call foo r0 = 0 // Never seen by verifier exit // main(ctx): r1 = ... call bar r0 = 0 exit Here, if we do end up throwing, the stacktrace would be the following: bpf_throw foo bar main In bar, the final instruction emitted will be the call to foo, as such, the return address will be the subsequent instruction (which the JIT emits as int3 on x86). This will end up lying outside the jited_len of the program, thus, when unwinding, we will fail to discover the return address as belonging to any program and end up in a panic due to the unreliable stack unwinding of BPF programs that we never expect. To remedy this case, make bpf_prog_ksym_find treat IP == ksym.end as part of the BPF program, so that is_bpf_text_address returns true when such a case occurs, and we are able to unwind reliably when the final instruction ends up being a call instruction. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/core.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ef362d7b09a5..08d52059655c 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -623,7 +623,11 @@ static __always_inline int bpf_tree_comp(void *key, struct latch_tree_node *n) if (val < ksym->start) return -1; - if (val >= ksym->end) + /* Ensure that we detect return addresses as part of the program, when + * the final instruction is a call for a program part of the stack + * trace. Therefore, do val > ksym->end instead of val >= ksym->end. + */ + if (val > ksym->end) return 1; return 0; From patchwork Wed Aug 9 11:41:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347826 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A1AA17744 for ; Wed, 9 Aug 2023 11:43:42 +0000 (UTC) Received: from mail-lf1-x143.google.com (mail-lf1-x143.google.com [IPv6:2a00:1450:4864:20::143]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFEBB210B for ; Wed, 9 Aug 2023 04:43:39 -0700 (PDT) Received: by mail-lf1-x143.google.com with SMTP id 2adb3069b0e04-4fe21e7f3d1so11107045e87.3 for ; Wed, 09 Aug 2023 04:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581417; x=1692186217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uOpe+KLiU31oQSG0sOAigLOphpYYwxqbP41P4lMfoBc=; b=AGLf4ylofyoM4yOwItvDDzFBw1dZyVNt0w0Z8iHJ+JjhE1HuzvmrXRKnR1foBQHbR8 zGvcn3Ifn50q68gMSdh3T8zz1ddfx7BaGsLh0+E5U6vWYxHHopE55EMArFsDAWaJlrot x9CgLamhOsqsnY8zS2bOYMDryzjGqBDglW3Qmi78YwsLFDldQ0O5G7vnUkQlEtka4zt4 8rT9zHSisPPCSOI1KdbcI2XD+kZIMZ9dkUlwo2F+6Pg4PUsYSHFpNeeA0J6jAsOCk/cC YLEoYCFwqzw2qDmUESxgijBL0IeOkS6rFRbLQItp6Lncgq+dud1avMewB7BEfyw+sG8J lnHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581417; x=1692186217; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uOpe+KLiU31oQSG0sOAigLOphpYYwxqbP41P4lMfoBc=; b=CQuasDnL1vBJOsmArVasYg5CSj8rEaG+JOn31w3Jmvh4Fm5XT9vnWF/oy+dWXDoHQR A50EVZNcGnuf39kXvfQm3En5CKoWh7+959e1OWKDtMANBOXz7ZJ4fSE2ZXUczJFsm/Uk 0INR8I6Es9YzxevpGYqjRQa+lNDUquVI24i7uptxFpXVEgFy31aWpOegnK9B4OvEILfB kcU0dj1K+UhU4i3riolinhCaFdqiFXxfnZjlcq89K0DGiiGVC9LekR+SDyh/Y2UEoqzl kHru8oU4vGk2gSX0nBHi3wxxbyEYvbRenQ1wIfjk6qFdIuuimvNMexeaKjs+daWaVjBM WlkA== X-Gm-Message-State: AOJu0Yz+Ai7PW6TpR0DaNrsSWr4QoP5g1eVxiKd5VagesFBOq7XhbIPK 4w3uNrKjfXxyZ2GP24xbaUwoX2z8ydeIfk1xReE= X-Google-Smtp-Source: AGHT+IG3nMZD7OKoFLJhAwBnGv+G8AAYQQyB6pUwVlNUdBg2GoMAvukpDC8l7drgwKcfmyJSA6wz1Q== X-Received: by 2002:a19:6507:0:b0:4fb:8939:d95c with SMTP id z7-20020a196507000000b004fb8939d95cmr1513259lfb.30.1691581416947; Wed, 09 Aug 2023 04:43:36 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id e14-20020a056402104e00b005227e53cec2sm7837421edu.50.2023.08.09.04.43.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:43:36 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 10/14] bpf: Disallow extensions to exception callbacks Date: Wed, 9 Aug 2023 17:11:12 +0530 Message-ID: <20230809114116.3216687-11-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2960; i=memxor@gmail.com; h=from:subject; bh=+FH1ASLYTMFo7b517rED6jaPFYVZ8DbNm8TQbKjCl1M=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJMJpVmU7EL/Wi42Nihq4WoWMPM7rR7Q0P+ u7A3iWL9/yJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R yptaD/9UyuPKlxGAIUOsl3H4wo9pWdQ7qDbeqhXJh6rf4jeezcrHyTOr+gj0Dr3fXCjVIioc8lQ gcZnFhoiNfci4I0UlRbpUUS5mwxKcYiom8Icv6yDKfko/xWvjP5CQu0c6xyPe6/Gr7LQfdy2EyP kJbVg7fqHw9JRoeqHBFY1VhmgieGG6zcpwZz7oD1+j+EM6YG74dKizXFz+1GvTRRJFsjHOr6+AT StQlAr0UlMdI58jkZvLYxcLaC7lPc1ZPYgkSjn0G3Ltn/psgFyskQwUGEeQ4QRb4nJqktvQO4nd N3SLVcIn/GOFs3wkbqnxoNwjZqw8E7B7UQgMl0QZaMvxIlIZobtE8QVGCyfAPC3kBq3DXbVgfwb BW9FWmXaNZkDpz4ask/IIab/YtclIzhv0uuHCLYbz6qpOidZId0UOKkWrEKXeoPLq+3FDOuYpia hPyniDSIuUGNetDhe9FzC6wNqe1WT4tB73cWOLFHkCI5DJ4Is5Uht1lJ8H/fq+fgc0M0W8isk3g CgQZFoAp/kEbNci74Ufhz1Oxm7aF9woQt5C+wlV8YH+IjoePUYG1mcQrcYbXERSHBpnw7qA3UMq Xg4Kjj0AVxY1ZLZgOxD0yy5zZXnCHlYg5uI3c9cT1Pq+m5YZJNp326W3Yme+q7sjpSasQbMEA98 dSvVTlI0/WAjVyQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net During testing, it was discovered that extensions to exception callbacks had no checks, upon running a testcase, the kernel ended up running off the end of a program having final call as bpf_throw, and hitting int3 instructions. The reason is that while the default exception callback would have reset the stack frame to return back to the main program's caller, the replacing extension program will simply return back to bpf_throw, which will instead return back to the program and the program will continue execution, now in an undefined state where anything could happen. The way to support extensions to an exception callback would be to mark the BPF_PROG_TYPE_EXT main subprog as an exception_cb, and prevent it from calling bpf_throw. This would make the JIT produce a prologue that restores saved registers and reset the stack frame. But let's not do that until there is a concrete use case for this, and simply disallow this for now. One key point here to note is that currently X86_TAIL_CALL_OFFSET didn't require any modifications, even though we emit instructions before the corresponding endbr64 instruction. This is because we ensure that a main subprog never serves as an exception callback, and therefore the exception callback (which will be a global subprog) can never serve as the tail call target, eliminating any discrepancies. However, once we support a BPF_PROG_TYPE_EXT to also act as an exception callback, it will end up requiring change to the tail call offset to account for the extra instructions. For simplicitly, tail calls could be disabled for such targets. Noting the above, it appears better to wait for a concrete use case before choosing to permit extension programs to replace exception callbacks. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/helpers.c | 1 + kernel/bpf/verifier.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 64a07232c58f..a04eff53354c 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2470,6 +2470,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) */ kasan_unpoison_task_stack_below((void *)ctx.sp); ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); + WARN(1, "A call to BPF exception callback should never return\n"); } __diag_pop(); diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index a0e1a1d1f5d3..13db1fa4163c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19622,6 +19622,11 @@ int bpf_check_attach_target(struct bpf_verifier_log *log, "Extension programs should be JITed\n"); return -EINVAL; } + if (aux->func && aux->func[subprog]->aux->exception_cb) { + bpf_log(log, + "Extension programs cannot replace exception callback\n"); + return -EINVAL; + } } if (!tgt_prog->jited) { bpf_log(log, "Can attach to only JITed progs\n"); From patchwork Wed Aug 9 11:41:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347827 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0716F17744 for ; Wed, 9 Aug 2023 11:43:55 +0000 (UTC) Received: from mail-lf1-x142.google.com (mail-lf1-x142.google.com [IPv6:2a00:1450:4864:20::142]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E425310C for ; Wed, 9 Aug 2023 04:43:53 -0700 (PDT) Received: by mail-lf1-x142.google.com with SMTP id 2adb3069b0e04-4fe463420fbso10463713e87.3 for ; Wed, 09 Aug 2023 04:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581431; x=1692186231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5jt4MD6otCHDvVVH7w0i/nPhgzdFoqeKcIRFnJUhleg=; b=Us5lsvwiycgZ7xp39wPhWyfudOhB5/uAWqZdmj29rGH5IwCa3NpZewmDCYfIQI8/mQ aKv7HchVsY9sIsvOEGkNYq7zCgbmx970JIrGk4N72sK5GVVXvosBxfDVrRke2YfxJ/ou Gb6wR6IZY720cCxCRqk+pRksSLPQ2WjShNdfW9AiAACLVrwYJyVdDpN2Cv9HlddEZFiN lLoVTkDW8gl3b440o7488FJPnRCZJNFzf3W/I8ikWBhPCiTT285++jXmIIwpcKjcWpGf MAvxa0EKycclUCSWspPB+4YDxlZIMc2tJECBrM5EgPHi+TSuRbLUBnui6vrZN8we1+hF VXEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581431; x=1692186231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5jt4MD6otCHDvVVH7w0i/nPhgzdFoqeKcIRFnJUhleg=; b=MM+lp8F3Uxva4i0+AEewRYE7K7P6JwzYWEgQnuQo+DyU0vtq76zqqScd9x+43e8xhB draRvTpFif/gOFwZeN6LK3C0uGz/S+k9Edglx6HXLGwa1OL+JSfsdvHxah8dIlpZc1w0 gosUg3D5kbxzLTfJ/Xzh8IyIBM0uvXZh+l2EQ7/T8+283UXyRKMAGuzMbAEYxsHxoLgz 90C1QKzlE0PyklVLwUu9rLU6RzPO5T67jMH65VDAoIJrZqsoPRaxgQxJgCH4XSPsuACv rZvWZ52W/hcg+IL599J4zZx3SJDbyYzkMt5zaIF8tzMohriCoyc21wUcZbL6rQAQC+nZ SssQ== X-Gm-Message-State: AOJu0YyIxnRL2DZJZFuU0ZYC1Ym4ug91PMSXhNpNn7Hyx8OZXtGMoSgd T4hGx+qAHhSpVqaJniQGwy2j86wLE6pDTq4vUCA= X-Google-Smtp-Source: AGHT+IE0WecKzYMrC2QgJMkfDJg4SbP1JM9o280v3cw0rgr+Titv36CadEHu6n5QRIOO6WZ1Ftj9pg== X-Received: by 2002:ac2:5b5e:0:b0:4f8:d385:41bd with SMTP id i30-20020ac25b5e000000b004f8d38541bdmr1742016lfp.8.1691581431239; Wed, 09 Aug 2023 04:43:51 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id n15-20020aa7d04f000000b0052227c4384esm7939959edo.34.2023.08.09.04.43.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:43:50 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Dave Marchevsky , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 11/14] bpf: Fix kfunc callback register type handling Date: Wed, 9 Aug 2023 17:11:13 +0530 Message-ID: <20230809114116.3216687-12-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1178; i=memxor@gmail.com; h=from:subject; bh=uTfZBPCaLcBY2DK9GK066GGX6NJIOUwA5y0+iLSaY0g=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJNTDeBWOVkqE3t7JYDz6fgJ52iTv0wJ7AB Gd+DWN3kRSJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R yqWlD/9F9Yg72IeQPAroj38ptxQd0jHcyKk3MHdSwDVx9av/xz+/EcM+x9+YcRoQrCEmpqVvxep NtxP3h3EYK86Z4Svogq5sFSMJNkvULv/KDuOQz9g2Y4Jno4RlJdJcJmwqAxTVihynTAtvhejnwV 77LbPdUnwGJfmGPw0Ug3+twYhx5sTOAZRmu7ij+gB7o3vQhAau+JLJ6cIg7etSlqLSHHrr42OXH zEkOdaV6xvP4hICWwJrVcNPOweIzvQ6QNDgHObAAcfUAyDRl4Fj8VzlcWN+4RuQG57zMbA4ucyo P4+gRtk2ygjmvJd7NKWCVhJY2BUU+hBGM4eEcjeN6hGoR4a2ZaVF0Dkojw3oY5mLUEWIVjM/n4T ZEXwz1V5ODJE4EqLXVFSAppr9J3HM5l4CmdvptbQCYmWOkQxboHKt0SDQ9W/rxjjWzmAZH/WJPH y/kX0nf/v4g2Jn9z3zPykw11awaQYV0iHiZdV++bQUDyhYyWP9cC7oqnO+iPvw/y52DiDuWq40I azMLZKrMvMV218Om428Lu/d2rZNM2Ug6rERB5k0FuT9KOVqu1UGfcfqBnfkFX4aba7TD8yR3/14 VZhs78ancKNvXzdSpDJY6vDFFZOH+dX0TZPe1NrZjSxsZc6O6iQffXVX9o0SWiozmdate7h6ltK AKR1p1J+CDeuGNA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net The kfunc code to handle KF_ARG_PTR_TO_CALLBACK does not check the reg type before using reg->subprogno. This can accidently permit invalid pointers from being passed into callback helpers (e.g. silently from different paths). Likewise, reg->subprogno from the per-register type union may not be meaningful either. We need to reject any other type except PTR_TO_FUNC. Cc: Dave Marchevsky Fixes: 5d92ddc3de1b ("bpf: Add callback validation to kfunc verifier logic") Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Dave Marchevsky --- kernel/bpf/verifier.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 13db1fa4163c..1c9a7a6ef906 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11334,6 +11334,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ break; } case KF_ARG_PTR_TO_CALLBACK: + if (reg->type != PTR_TO_FUNC) { + verbose(env, "arg%d expected pointer to func\n", i); + return -EINVAL; + } meta->subprogno = reg->subprogno; break; case KF_ARG_PTR_TO_REFCOUNTED_KPTR: From patchwork Wed Aug 9 11:41:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347828 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5F0E17744 for ; Wed, 9 Aug 2023 11:44:09 +0000 (UTC) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50C0D9B for ; Wed, 9 Aug 2023 04:44:08 -0700 (PDT) Received: by mail-ed1-x544.google.com with SMTP id 4fb4d7f45d1cf-522dd6b6438so8525848a12.0 for ; Wed, 09 Aug 2023 04:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581446; x=1692186246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=z2lNDuvhkm/BrRXLbYSnkEGTXFzIoZ+ukDW4N75Y24Y=; b=fnd5XPZVrT7qylun1Y3F3sT8DfNx9wTFPIZgQz4qnkBeUAjbN2A7e7Way6FwnnRxNl Hz64D8V+VL5jwYokt2SZCTIrT1WCkG01vtZCdirGRiWw1KTPogg7NAirhWrCx/N/8l9x C4B0Pxq4wpm2rUETQkkJU1ylK/6tV2sl2bKlAz+n/NJpkYBhXThbWnarSeNc4Hk40DkT gjncXcQXK+U/pQMoJtm780MJOciSFRYlrq+rxTtQ5tkqHcYVH0UyGLN2FTY0HKx0QfMG KgrB8om0Wee6e9p0JJUZHkdJBFZaROzmxPupCS1exmzb8UzxfPcL2ttdV4DE4RlZ52F+ hn6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581446; x=1692186246; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z2lNDuvhkm/BrRXLbYSnkEGTXFzIoZ+ukDW4N75Y24Y=; b=j3C3E996en/piCKDv0mX5Qi9F3oYVSp3IBSxpBcY28HK14vcMDbAR9LvBuVKClrx4L pMLONmaaUJ//2IRyEP5Ng+PUUU9uON37TE+kG7p5cyXV762/CdMjatAka/qz/OZ3R7kC KtkHX+uZJZwcCrkEBMz8GgOpZOqPMdyQxyeuTJxzLtzt54WO9Yxwpi3ohZ65SJIlTiVh +LrgLpaDcjWtjkGzppnlMbNr2Lugs8VAX6C6aU91RyOreCi5a1HYl+0/zU7FD6O7q//k eNwfQ5fvhhZsoF/Iejsm53N053BLxhN8N3ZV85ACVRCAuKwUFoezoYgAHVSP0AiwHLa9 3f0w== X-Gm-Message-State: AOJu0YyMvBGbHt9FboKoz5SwgqVRKs1/N+zq84d1+zC4IOnPo6bWWFuP yu2BdLMkFHTIvORWl6PySRe1hBw+DyZ5nVDWIoI= X-Google-Smtp-Source: AGHT+IEUuWPnLtT1YBMS61CqO6XHgAEHAenAOfYlmgm1ZcXEE8kgoHEVqvQtQv02bA0EcaAiVvQGMA== X-Received: by 2002:a05:6402:2050:b0:522:ca7c:df78 with SMTP id bc16-20020a056402205000b00522ca7cdf78mr2314003edb.0.1691581445892; Wed, 09 Aug 2023 04:44:05 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id k12-20020aa7c04c000000b0052348d74865sm1978633edo.61.2023.08.09.04.44.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:44:05 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 12/14] libbpf: Add support for custom exception callbacks Date: Wed, 9 Aug 2023 17:11:14 +0530 Message-ID: <20230809114116.3216687-13-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8764; i=memxor@gmail.com; h=from:subject; bh=2v+GgMkFFxB87Ltd7kGCtWMTzsuSjB6PC0H1kn78B6s=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJcP2m5bxyGw0tnqRcf/ooCswdtKsXRtDVC 45kTDi8OSqJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R yk/BEACcOsdK2FEQ1hGxWRjOXdUdp2KLRmWa9veVdiCAG1or2Z4JQPCgQOQXUWRsJ6gcVgkGKpC xWa/ZyqSDsCNAhvIqgBd/19dlk/IqupzVYEEBTNcjYwRy1F2AsxoEF/wB+Wza47JPIFvs4a80VP YvUHOUlgn9XjPtSdJiryYyVjbeZC+bj3TGbjjSPnb6mHD9YD5AKjnLwn1qgKUNhPRvNQtkvYIJl Jy1LiSW0ceD782SBxQedRgvh1/mvoULW/6C/PcrBnu1YdW+pXXO7LoFv/M2WkImD4PxXQql2iRN UscSCsDdzwHF8mcRSYP0x0PAvrSnejbD4DXDjEVwNIFUVQuAUOPcO9NycxqaSAYJCwSfD7x1aum a9pF1t9el2WAs23M9aeQ/lY4+AMKvsVY1fHZi3j8exEdGwANrWR/6/NUa53QEqpRqXZIoztJ9+v al6RszWlPEoJjeZmLPg4iYRpj4tM+wSxIMuL3QLUPMeueSYAkb+SChNvZBy/s1M2VHRoyUb5XJH bjuo/b4XFx8SKpBiDAZYHYFGxzoIGJFL5Y+C0ObAYZGExwBuY1fySX2gSoiaKKn674z3ljjtlcf Tjr4q2sRF2/w+VucYBcbYhUNZv8C1mGyiJ0YueHPZj3DjLBK1U79FVS1ErmPnrXoFbXZ6ZVmutl sAuO9QxO5d8z79Q== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net Add support to libbpf to append exception callbacks when loading a program. The exception callback is found by discovering the declaration tag 'exception_callback:' and finding the callback in the value of the tag. The process is done in two steps. First, for each main program, the bpf_object__sanitize_and_load_btf function finds and marks its corresponding exception callback as defined by the declaration tag on it. Second, bpf_object__reloc_code is modified to append the indicated exception callback at the end of the instruction iteration (since exception callback will never be appended in that loop, as it is not directly referenced). Signed-off-by: Kumar Kartikeya Dwivedi --- tools/lib/bpf/libbpf.c | 166 +++++++++++++++++++++++++++++++++++------ 1 file changed, 142 insertions(+), 24 deletions(-) diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 17883f5a44b9..7c607bac8204 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -432,9 +432,11 @@ struct bpf_program { int fd; bool autoload; bool autoattach; + bool sym_global; bool mark_btf_static; enum bpf_prog_type type; enum bpf_attach_type expected_attach_type; + int exception_cb_idx; int prog_ifindex; __u32 attach_btf_obj_fd; @@ -760,6 +762,7 @@ bpf_object__init_prog(struct bpf_object *obj, struct bpf_program *prog, prog->type = BPF_PROG_TYPE_UNSPEC; prog->fd = -1; + prog->exception_cb_idx = -1; /* libbpf's convention for SEC("?abc...") is that it's just like * SEC("abc...") but the corresponding bpf_program starts out with @@ -866,20 +869,28 @@ bpf_object__add_programs(struct bpf_object *obj, Elf_Data *sec_data, if (err) return err; + if (ELF64_ST_BIND(sym->st_info) != STB_LOCAL) + prog->sym_global = true; + /* if function is a global/weak symbol, but has restricted * (STV_HIDDEN or STV_INTERNAL) visibility, mark its BTF FUNC * as static to enable more permissive BPF verification mode * with more outside context available to BPF verifier */ - if (ELF64_ST_BIND(sym->st_info) != STB_LOCAL - && (ELF64_ST_VISIBILITY(sym->st_other) == STV_HIDDEN - || ELF64_ST_VISIBILITY(sym->st_other) == STV_INTERNAL)) + if (prog->sym_global && (ELF64_ST_VISIBILITY(sym->st_other) == STV_HIDDEN + || ELF64_ST_VISIBILITY(sym->st_other) == STV_INTERNAL)) prog->mark_btf_static = true; nr_progs++; obj->nr_programs = nr_progs; } + /* After adding all programs, now pair them with their exception + * callbacks if specified. + */ + if (!kernel_supports(obj, FEAT_BTF_DECL_TAG)) + goto out; +out: return 0; } @@ -3137,6 +3148,80 @@ static int bpf_object__sanitize_and_load_btf(struct bpf_object *obj) } } + if (!kernel_supports(obj, FEAT_BTF_DECL_TAG)) + goto skip_exception_cb; + for (i = 0; i < obj->nr_programs; i++) { + struct bpf_program *prog = &obj->programs[i]; + int j, k, n; + + if (prog_is_subprog(obj, prog)) + continue; + n = btf__type_cnt(obj->btf); + for (j = 1; j < n; j++) { + const char *str = "exception_callback:", *name; + size_t len = strlen(str); + struct btf_type *t; + + t = btf_type_by_id(obj->btf, j); + if (!btf_is_decl_tag(t) || btf_decl_tag(t)->component_idx != -1) + continue; + + name = btf__str_by_offset(obj->btf, t->name_off); + if (strncmp(name, str, len)) + continue; + + t = btf_type_by_id(obj->btf, t->type); + if (!btf_is_func(t) || btf_func_linkage(t) != BTF_FUNC_GLOBAL) { + pr_warn("prog '%s': exception_callback: decl tag not applied to the main program\n", + prog->name); + return -EINVAL; + } + if (strcmp(prog->name, btf__str_by_offset(obj->btf, t->name_off))) + continue; + /* Multiple callbacks are specified for the same prog, + * the verifier will eventually return an error for this + * case, hence simply skip appending a subprog. + */ + if (prog->exception_cb_idx >= 0) { + prog->exception_cb_idx = -1; + break; + } + + name += len; + if (str_is_empty(name)) { + pr_warn("prog '%s': exception_callback: decl tag contains empty value\n", + prog->name); + return -EINVAL; + } + + for (k = 0; k < obj->nr_programs; k++) { + struct bpf_program *subprog = &obj->programs[k]; + + if (!prog_is_subprog(obj, subprog)) + continue; + if (strcmp(name, subprog->name)) + continue; + /* Enforce non-hidden, as from verifier point of + * view it expects global functions, whereas the + * mark_btf_static fixes up linkage as static. + */ + if (!subprog->sym_global || subprog->mark_btf_static) { + pr_warn("prog '%s': exception callback %s must be a global non-hidden function\n", + prog->name, subprog->name); + return -EINVAL; + } + prog->exception_cb_idx = k; + break; + } + + if (prog->exception_cb_idx >= 0) + continue; + pr_warn("prog '%s': cannot find exception callback '%s'\n", prog->name, name); + return -ENOENT; + } + } +skip_exception_cb: + sanitize = btf_needs_sanitization(obj); if (sanitize) { const void *raw_data; @@ -6184,14 +6269,46 @@ static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_progra return 0; } +static int +bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main_prog, + struct bpf_program *subprog) +{ + struct bpf_insn *insns; + size_t new_cnt; + int err; + + subprog->sub_insn_off = main_prog->insns_cnt; + + new_cnt = main_prog->insns_cnt + subprog->insns_cnt; + insns = libbpf_reallocarray(main_prog->insns, new_cnt, sizeof(*insns)); + if (!insns) { + pr_warn("prog '%s': failed to realloc prog code\n", main_prog->name); + return -ENOMEM; + } + main_prog->insns = insns; + main_prog->insns_cnt = new_cnt; + + memcpy(main_prog->insns + subprog->sub_insn_off, subprog->insns, + subprog->insns_cnt * sizeof(*insns)); + + pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n", + main_prog->name, subprog->insns_cnt, subprog->name); + + /* The subprog insns are now appended. Append its relos too. */ + err = append_subprog_relos(main_prog, subprog); + if (err) + return err; + return 0; +} + static int bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, struct bpf_program *prog) { - size_t sub_insn_idx, insn_idx, new_cnt; + size_t sub_insn_idx, insn_idx; struct bpf_program *subprog; - struct bpf_insn *insns, *insn; struct reloc_desc *relo; + struct bpf_insn *insn; int err; err = reloc_prog_func_and_line_info(obj, main_prog, prog); @@ -6266,25 +6383,7 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, * and relocate. */ if (subprog->sub_insn_off == 0) { - subprog->sub_insn_off = main_prog->insns_cnt; - - new_cnt = main_prog->insns_cnt + subprog->insns_cnt; - insns = libbpf_reallocarray(main_prog->insns, new_cnt, sizeof(*insns)); - if (!insns) { - pr_warn("prog '%s': failed to realloc prog code\n", main_prog->name); - return -ENOMEM; - } - main_prog->insns = insns; - main_prog->insns_cnt = new_cnt; - - memcpy(main_prog->insns + subprog->sub_insn_off, subprog->insns, - subprog->insns_cnt * sizeof(*insns)); - - pr_debug("prog '%s': added %zu insns from sub-prog '%s'\n", - main_prog->name, subprog->insns_cnt, subprog->name); - - /* The subprog insns are now appended. Append its relos too. */ - err = append_subprog_relos(main_prog, subprog); + err = bpf_object__append_subprog_code(obj, main_prog, subprog); if (err) return err; err = bpf_object__reloc_code(obj, main_prog, subprog); @@ -6308,6 +6407,25 @@ bpf_object__reloc_code(struct bpf_object *obj, struct bpf_program *main_prog, prog->name, insn_idx, insn->imm, subprog->name, subprog->sub_insn_off); } + /* Now, also append exception callback if it has not been done already. */ + if (main_prog == prog && main_prog->exception_cb_idx >= 0) { + subprog = &obj->programs[main_prog->exception_cb_idx]; + + /* Calling exception callback directly is disallowed, which the + * verifier will reject later. In case it was processed already, + * we can skip this step, otherwise for all other valid cases we + * have to append exception callback now. + */ + if (subprog->sub_insn_off == 0) { + err = bpf_object__append_subprog_code(obj, main_prog, subprog); + if (err) + return err; + err = bpf_object__reloc_code(obj, main_prog, subprog); + if (err) + return err; + } + } + return 0; } From patchwork Wed Aug 9 11:41:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347829 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51AF117744 for ; Wed, 9 Aug 2023 11:44:19 +0000 (UTC) Received: from mail-ej1-x643.google.com (mail-ej1-x643.google.com [IPv6:2a00:1450:4864:20::643]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E02AE9B for ; Wed, 9 Aug 2023 04:44:17 -0700 (PDT) Received: by mail-ej1-x643.google.com with SMTP id a640c23a62f3a-99bcfe28909so922632166b.3 for ; Wed, 09 Aug 2023 04:44:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581456; x=1692186256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PoQfVAUwXkJ1W9+4QVXdiQlLVeOCQMqFcmTK991KBtI=; b=jsEjWzohm9+TLr/o4BjYuZ0JrtceBMXiK2So5OHdwyoV6x8t/UjNrmsr6fBTERbooH E6RPKzZPkvw6hqgbTy7IyuZFiNR5EnICFLn6whCqBFTDmx6kdR1dAGKzH/X3CzVI7yts mGQq0ji9j9/X8aRG69++1AZYX4vmP9dH3Ai60Cx+dSPeJgZBmOezZXQTnVTF+lqrDRcE FfA5GziTZU+dol50fm/WMtETi/RXPZ8EF55UmaHRCK/zgjXfxs9QG2WrYP3gbcKO9/9d ksU4jgkiMjjh6cnT0Oe+x+mtDGW4vJKMDLe2FlTa21MXekffb6CmSzGj/rcvUq8RFjr5 MUNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581456; x=1692186256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PoQfVAUwXkJ1W9+4QVXdiQlLVeOCQMqFcmTK991KBtI=; b=H0HaJcp7uOy0azLOHAsfEacyP/Qh++PfiFFJcemJt7g1vXzo+G69Fg2XEB3A6eMJwO MbmG2FOqePR06bUfBw39mixZAJXisU45fzcraDvX1Yvrlmr7cZ64i9kSyOQYkW5czIbm +IwSGMP/SXiqaotzIJOuadng9a5K4RCjsfLKC2h8/4m4UH4Oo1PQqxoMx+2DVzoqtS2m hJK+O4Y5LukkPa5Pgd24BskCIZF9wWJVBylACOQu6fIupoA9lvGnUV0RdqKJQD7YALMp sh3TgutwEa28ZkQJbFg41ZwUmG9h0+tjSsyqLUkTqc1EY5GzD13wj+iTC6obaJCuBxi6 E/6w== X-Gm-Message-State: AOJu0Yz0VNIu0VnWuZE2l98jCHKHB/NFxN55zAEj5T8UUshtZfwoT46W 1/jjUFeSf76rsbIMSbC8mW9TnPIdZ2rR0K9ytlo= X-Google-Smtp-Source: AGHT+IHZC+tpNd9kw7LREoXTSQ/onAlB6FLhAmLw/De+F0wqFsdQefYIN6P0Edn8+bTHc3V93Nt0lA== X-Received: by 2002:a17:906:3019:b0:982:45ca:ac06 with SMTP id 25-20020a170906301900b0098245caac06mr1872561ejz.60.1691581455676; Wed, 09 Aug 2023 04:44:15 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id y9-20020a17090668c900b009828e26e519sm7822081ejr.122.2023.08.09.04.44.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:44:15 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 13/14] selftests/bpf: Add BPF assertion macros Date: Wed, 9 Aug 2023 17:11:15 +0530 Message-ID: <20230809114116.3216687-14-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9707; i=memxor@gmail.com; h=from:subject; bh=QURVWXu/ZaKDcpy8ugNyPXoVE8I3kd5SyHBr6uVxLrA=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJHHIs3FJbaMOf+V/nXrMj1S1M5302ICJVm mV1+J6FdrCJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R yjrsD/4p/fUtEZAOaey+xvl/lMx07dSc/mxq156gc3rNUDY6ZJa39rqhRJDC7zZuePGXxulhqZc RV54o1/o3CGgqNJ9Si/qsUO4wctDkHkGSZ/kVWZ2XW4GNA6TDRlfnDI4pJWdIllS2E2EZ4boTo7 hddY2NHNNkJrMq9xHSoyJ55LYgvSITTXCGcOGp6pmgGzjOf4Mmzfgj4J9lj8sBFoHr/1BhZCbHE 7lRm68yEcnMB9WDc6V7zqk1QRukMPXci/wEuYu0gawnhM3ZlJXv7TIz4Wh9wPfO7eHQEgHyvDfp P4jN7p6SHoQmuy3bu+H9HbPKZAFLw+oiiVK7pBRBaqXHmE/JhMBL9opX16bCBnEM+HWC4XHg7VV svuCrOdgtnHlY17NOH9zR5AP5mAk1zmqJZ4qMwruKy05OiiAdTeEoKoeX0S4CjuFaU4hPgwdy2M yaJNXaK5NLq7alZWayGAf3B31Gh6ryuzFU8CZfnC51FvNz2Zx7qOlXG3cgojY+91GXnR/f+rrj2 qKNvJNC1n2hHNm8nUNhvdQ71i1zqgj1x9COSBHPqNNbN09zcRRhegYVBxsheDzWOuVq0d9y/EV+ MTcvl3lPzARlgk8XblRyVSIKvbmhU9L3HXEYyldT9GU4STnxjCiAzUnumoYYI81SVWbdZwnVwtd DUL01zh39HcZRvQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net Add macros implementing an 'assert' statement primitive using macros, built on top of the BPF exceptions support introduced in previous patches. The bpf_assert_*_with variants allow supplying a value which can the be inspected within the exception handler to signify the assert statement that led to the program being terminated abruptly, or be returned by the default exception handler. Note that only 64-bit scalar values are supported with these assertion macros, as during testing I found other cases quite unreliable in presence of compiler shifts/manipulations extracting the value of the right width from registers scrubbing the verifier's bounds information and knowledge about the value in the register. Thus, it is easier to reliably support this feature with only the full register width, and support both signed and unsigned variants. The bpf_assert_range is interesting in particular, which clamps the value in the [begin, end] (both inclusive) range within verifier state, and emits a check for the same at runtime. Signed-off-by: Kumar Kartikeya Dwivedi --- .../testing/selftests/bpf/bpf_experimental.h | 242 ++++++++++++++++++ 1 file changed, 242 insertions(+) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 612ac86873af..faa0e785a331 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -176,4 +176,246 @@ extern void bpf_throw(u64 cookie) __ksym; */ #define __exception_cb(name) __attribute__((btf_decl_tag("exception_callback:" #name))) +#define __bpf_assert_signed(x) _Generic((x), \ + unsigned long: 0, \ + unsigned long long: 0, \ + signed long: 1, \ + signed long long: 1 \ +) + +#define __bpf_assert_check(LHS, op, RHS) \ + _Static_assert(sizeof(&(LHS)), "1st argument must be an lvalue expression"); \ + _Static_assert(sizeof(LHS) == 8, "Only 8-byte integers are supported\n"); \ + _Static_assert(__builtin_constant_p(__bpf_assert_signed(LHS)), "internal static assert"); \ + _Static_assert(__builtin_constant_p((RHS)), "2nd argument must be a constant expression") + +#define __bpf_assert(LHS, op, cons, RHS, VAL) \ + ({ \ + asm volatile ("if %[lhs] " op " %[rhs] goto +2; r1 = %[value]; call bpf_throw" \ + : : [lhs] "r"(LHS), [rhs] cons(RHS), [value] "ri"(VAL) : ); \ + }) + +#define __bpf_assert_op_sign(LHS, op, cons, RHS, VAL, supp_sign) \ + ({ \ + __bpf_assert_check(LHS, op, RHS); \ + if (__bpf_assert_signed(LHS) && !(supp_sign)) \ + __bpf_assert(LHS, "s" #op, cons, RHS, VAL); \ + else \ + __bpf_assert(LHS, #op, cons, RHS, VAL); \ + }) + +#define __bpf_assert_op(LHS, op, RHS, VAL, supp_sign) \ + ({ \ + if (sizeof(typeof(RHS)) == 8) { \ + const typeof(RHS) rhs_var = (RHS); \ + __bpf_assert_op_sign(LHS, op, "r", rhs_var, VAL, supp_sign); \ + } else { \ + __bpf_assert_op_sign(LHS, op, "i", RHS, VAL, supp_sign); \ + } \ + }) + +/* Description + * Assert that a conditional expression is true. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert(cond) if (!(cond)) bpf_throw(0); + +/* Description + * Assert that a conditional expression is true. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_with(cond, value) if (!(cond)) bpf_throw(value); + +/* Description + * Assert that LHS is equal to RHS. This statement updates the known value + * of LHS during verification. Note that RHS must be a constant value, and + * must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_eq(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, ==, RHS, 0, true); \ + }) + +/* Description + * Assert that LHS is equal to RHS. This statement updates the known value + * of LHS during verification. Note that RHS must be a constant value, and + * must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_eq_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, ==, RHS, value, true); \ + }) + +/* Description + * Assert that LHS is less than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_lt(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is less than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_lt_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is greater than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_gt(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is greater than RHS. This statement updates the known + * bounds of LHS during verification. Note that RHS must be a constant + * value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_gt_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is less than or equal to RHS. This statement updates the + * known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_le(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <=, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is less than or equal to RHS. This statement updates the + * known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_le_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, <=, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is greater than or equal to RHS. This statement updates + * the known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_ge(LHS, RHS) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, RHS, 0, false); \ + }) + +/* Description + * Assert that LHS is greater than or equal to RHS. This statement updates + * the known bounds of LHS during verification. Note that RHS must be a + * constant value, and must fit within the data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_ge_with(LHS, RHS, value) \ + ({ \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, RHS, value, false); \ + }) + +/* Description + * Assert that LHS is in the range [BEG, END] (inclusive of both). This + * statement updates the known bounds of LHS during verification. Note + * that both BEG and END must be constant values, and must fit within the + * data type of LHS. + * Returns + * Void. + * Throws + * An exception with the value zero when the assertion fails. + */ +#define bpf_assert_range(LHS, BEG, END) \ + ({ \ + _Static_assert(BEG <= END, "BEG must be <= END"); \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, BEG, 0, false); \ + __bpf_assert_op(LHS, <=, END, 0, false); \ + }) + +/* Description + * Assert that LHS is in the range [BEG, END] (inclusive of both). This + * statement updates the known bounds of LHS during verification. Note + * that both BEG and END must be constant values, and must fit within the + * data type of LHS. + * Returns + * Void. + * Throws + * An exception with the specified value when the assertion fails. + */ +#define bpf_assert_range_with(LHS, BEG, END, value) \ + ({ \ + _Static_assert(BEG <= END, "BEG must be <= END"); \ + barrier_var(LHS); \ + __bpf_assert_op(LHS, >=, BEG, value, false); \ + __bpf_assert_op(LHS, <=, END, value, false); \ + }) + #endif From patchwork Wed Aug 9 11:41:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13347830 X-Patchwork-Delegate: bpf@iogearbox.net Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AA97154B4 for ; Wed, 9 Aug 2023 11:44:37 +0000 (UTC) Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C25B09B for ; Wed, 9 Aug 2023 04:44:34 -0700 (PDT) Received: by mail-ej1-x642.google.com with SMTP id a640c23a62f3a-99bc9e3cbf1so177397866b.0 for ; Wed, 09 Aug 2023 04:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691581472; x=1692186272; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MLXU92UH2ZvHSkyu6SiYNwHbPWtUAEB+uZUUM4BB628=; b=Nes576yikwYwQC6nxYlKX0fsVNt6idny/ZAHc+2Z6gwFgk1Brg2nmTccCbAL5BE/r/ srzapPCVh+0T/vGv57fGzAYe1tZ/IvDjmCD4GUijd5R846+NJnWfcBFSJ91MYBryd4l0 fWD8+4UZ5L9Vx0YvYResrWFzV3tMufxi2Pl+AlZkvRY1VzStEYotsoGhIuakS4H4/FF2 9fJRYkYHqOKjKD9iBeU7fOF/XMrP0SSzJHT8VWAWSkmH1gA0uPrx90wZTzRlgQWV0lvn DXyhSfPmQq/MuhU1VMR8JI+2aWU3KhpOs59e+gzuLRNgmquzCofOiDb7vQ1zQsOXHGHv EbOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691581472; x=1692186272; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MLXU92UH2ZvHSkyu6SiYNwHbPWtUAEB+uZUUM4BB628=; b=F1se0bfbY7sP3HGPwG+RluG6WQagD0+S0yjLysSiHdmDYRlbiYrunFp9pAPNTZJ5f1 r8RuSIJULExfB2qjyT7U8K6GoaVGyU75YlO1av0hHXu8qwfn1V2kB1mxvYT6nRURA+OG 3XIfZsJUOWtsueeexGuWQsqkCIVUoay4vdpwYz4/SdiRkCHLObhYPeYmgHvpoZGIZotT tk4xuUUuau4p4WCAKAc7OrwyfrUc7rOZy4bxN6erE3TH9ETwM0XYvtQf1NpiWV478XoR STU/fNaW7L2iD6k/23skbDgHThcKAZjdVlI5wM3pHOZcpX9qY4xgPtrTPBdq7KjQ4daL JlzA== X-Gm-Message-State: AOJu0YzU14Pn1x9S6aUD+E+2oNFND/gjcDdR8N2T+FD6LKhzJIHLGpoo UEzroITEkJzUUmOoR/boTzfT8u+99iOlIduLPOQ= X-Google-Smtp-Source: AGHT+IHVZps0tBC/yEP/8OttK8JS/CW18gJGzsv42Rl94KlgB8n5H1oK695WjQP0LcFhlb57k6FoQw== X-Received: by 2002:a17:907:6096:b0:974:fb94:8067 with SMTP id ht22-20020a170907609600b00974fb948067mr14903278ejc.23.1691581472372; Wed, 09 Aug 2023 04:44:32 -0700 (PDT) Received: from localhost ([2405:201:6014:dae3:1f3a:4dfc:39ef:546b]) by smtp.gmail.com with ESMTPSA id i15-20020a17090685cf00b0099d4bfa87a7sm272609ejy.199.2023.08.09.04.44.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Aug 2023 04:44:31 -0700 (PDT) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Yonghong Song , David Vernet Subject: [PATCH bpf-next v2 14/14] selftests/bpf: Add tests for BPF exceptions Date: Wed, 9 Aug 2023 17:11:16 +0530 Message-ID: <20230809114116.3216687-15-memxor@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230809114116.3216687-1-memxor@gmail.com> References: <20230809114116.3216687-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=37831; i=memxor@gmail.com; h=from:subject; bh=6AbUaD+cL0MCUKe2a8zqLMJkKunZj53iars50LCf2Dc=; b=owEBbQKS/ZANAwAIAUzgyIZIvxHKAcsmYgBk03rJ9dqD89oEfPbcr+eHBZxL5t4jqH3JboNVj YKU/tUVxNyJAjMEAAEIAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZNN6yQAKCRBM4MiGSL8R ymasD/9VfTTvJdHCDaq7sjiTHptI3Cj9PUEE0Kdkkeshkp6xQ8YVS+fIzSECXekYCRplcMSzkuJ rDLUb4f0FNmI7y1BLILaJK6IM25sK4yjCb4aguV+UZ6hQDbmTWe9Mdth0ciAVIdp5+xpokVlQ0T AIRa5yGAVHJQW2TOdoAS149OiRjBSp5SqDy67ntD6gbC+1Ogix6bTWUywZan63no9pIx2RQN8QC PqK3RB7vidtiBls7KnrDK+BUpeswo2YaI61Hcu6LePq0FOhv+RG6oa5HGF5fHzH+8eHbA1EFRfU M87cbP1ZBSsOFSA1NkRLi0EdOXPT61vMuFkpstxkfArUPHBbb9tnRYBmS60zk9XB6ptA2ETeYOF vzTJEH8GE12iZ/GEfyYK1E7wiNB4d0pDTIj7+HkHdVpXRh2765yIiU7aVE2gQpthOKbNKUn9twS DzuLcO8DaYl1qCkIF9oZAtVvqCuqwNBpImh1w7SyA893csoCZB1iJ0zfgd0MSphY5wGSW0w86Sc 2r5oIl28MOQ7udDcUxpytxjGodAOBIl3QgdEmwhSR9vUPyntRTji9JjMFpwFi8VE65aDtpO1IPT 0f6AelN46KpRuislCuTNuRshhdPLyaJ39VKvzY/0G4q0H2uT0tlR4aw0vEVH9z5OQ1YmGre71VK rIx0EyzEFs1rusw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net X-Patchwork-Delegate: bpf@iogearbox.net Add selftests to cover success and failure cases of API usage, runtime behavior and invariants that need to be maintained for implementation correctness. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../selftests/bpf/prog_tests/exceptions.c | 324 +++++++++++++++ .../testing/selftests/bpf/progs/exceptions.c | 368 ++++++++++++++++++ .../selftests/bpf/progs/exceptions_assert.c | 135 +++++++ .../selftests/bpf/progs/exceptions_ext.c | 59 +++ .../selftests/bpf/progs/exceptions_fail.c | 347 +++++++++++++++++ 7 files changed, 1235 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_assert.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ext.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_fail.c diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 3b61e8b35d62..432af22973b1 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -1,5 +1,6 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 +exceptions # JIT does not support calling kfunc bpf_throw: -524 fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # bpf_program__attach_kprobe_multi_opts unexpected error: -95 kprobe_multi_test/attach_api_addrs # bpf_program__attach_kprobe_multi_opts unexpected error: -95 diff --git a/tools/testing/selftests/bpf/DENYLIST.s390x b/tools/testing/selftests/bpf/DENYLIST.s390x index 5061d9e24c16..ce6f291665cf 100644 --- a/tools/testing/selftests/bpf/DENYLIST.s390x +++ b/tools/testing/selftests/bpf/DENYLIST.s390x @@ -6,6 +6,7 @@ bpf_loop # attaches to __x64_sys_nanosleep cgrp_local_storage # prog_attach unexpected error: -524 (trampoline) dynptr/test_dynptr_skb_data dynptr/test_skb_readonly +exceptions # JIT does not support calling kfunc bpf_throw (exceptions) fexit_sleep # fexit_skel_load fexit skeleton failed (trampoline) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) iters/testmod_seq* # s390x doesn't support kfuncs in modules yet diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions.c b/tools/testing/selftests/bpf/prog_tests/exceptions.c new file mode 100644 index 000000000000..4a0b9910dab3 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/exceptions.c @@ -0,0 +1,324 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +#include "exceptions.skel.h" +#include "exceptions_ext.skel.h" +#include "exceptions_fail.skel.h" +#include "exceptions_assert.skel.h" + +static char log_buf[1024 * 1024]; + +static void test_exceptions_failure(void) +{ + RUN_TESTS(exceptions_fail); +} + +static void test_exceptions_success(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, ropts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct exceptions_ext *eskel = NULL; + struct exceptions *skel; + int ret; + + skel = exceptions__open(); + if (!ASSERT_OK_PTR(skel, "exceptions__open")) + return; + + ret = exceptions__load(skel); + if (!ASSERT_OK(ret, "exceptions__load")) + goto done; + + if (!ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.jmp_table), &(int){0}, + &(int){bpf_program__fd(skel->progs.exception_tail_call_target)}, BPF_ANY), + "bpf_map_update_elem jmp_table")) + goto done; + +#define RUN_SUCCESS(_prog, return_val) \ + if (!test__start_subtest(#_prog)) goto _prog##_##return_val; \ + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs._prog), &ropts); \ + ASSERT_OK(ret, #_prog " prog run ret"); \ + ASSERT_EQ(ropts.retval, return_val, #_prog " prog run retval"); \ + _prog##_##return_val: + + RUN_SUCCESS(exception_throw_always_1, 64); + RUN_SUCCESS(exception_throw_always_2, 32); + RUN_SUCCESS(exception_throw_unwind_1, 16); + RUN_SUCCESS(exception_throw_unwind_2, 32); + RUN_SUCCESS(exception_throw_default, 0); + RUN_SUCCESS(exception_throw_default_value, 5); + RUN_SUCCESS(exception_tail_call, 24); + RUN_SUCCESS(exception_ext, 0); + RUN_SUCCESS(exception_ext_mod_cb_runtime, 35); + RUN_SUCCESS(exception_throw_subprog, 1); + RUN_SUCCESS(exception_assert_nz_gfunc, 1); + RUN_SUCCESS(exception_assert_zero_gfunc, 1); + RUN_SUCCESS(exception_assert_neg_gfunc, 1); + RUN_SUCCESS(exception_assert_pos_gfunc, 1); + RUN_SUCCESS(exception_assert_negeq_gfunc, 1); + RUN_SUCCESS(exception_assert_poseq_gfunc, 1); + RUN_SUCCESS(exception_assert_nz_gfunc_with, 1); + RUN_SUCCESS(exception_assert_zero_gfunc_with, 1); + RUN_SUCCESS(exception_assert_neg_gfunc_with, 1); + RUN_SUCCESS(exception_assert_pos_gfunc_with, 1); + RUN_SUCCESS(exception_assert_negeq_gfunc_with, 1); + RUN_SUCCESS(exception_assert_poseq_gfunc_with, 1); + RUN_SUCCESS(exception_bad_assert_nz_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_zero_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_neg_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_pos_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_negeq_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_poseq_gfunc, 0); + RUN_SUCCESS(exception_bad_assert_nz_gfunc_with, 100); + RUN_SUCCESS(exception_bad_assert_zero_gfunc_with, 105); + RUN_SUCCESS(exception_bad_assert_neg_gfunc_with, 200); + RUN_SUCCESS(exception_bad_assert_pos_gfunc_with, 0); + RUN_SUCCESS(exception_bad_assert_negeq_gfunc_with, 101); + RUN_SUCCESS(exception_bad_assert_poseq_gfunc_with, 99); + RUN_SUCCESS(exception_assert_range, 1); + RUN_SUCCESS(exception_assert_range_with, 1); + RUN_SUCCESS(exception_bad_assert_range, 0); + RUN_SUCCESS(exception_bad_assert_range_with, 10); + +#define RUN_EXT(load_ret, attach_err, expr, msg, after_link) \ + { \ + LIBBPF_OPTS(bpf_object_open_opts, o, .kernel_log_buf = log_buf, \ + .kernel_log_size = sizeof(log_buf), \ + .kernel_log_level = 2); \ + exceptions_ext__destroy(eskel); \ + eskel = exceptions_ext__open_opts(&o); \ + struct bpf_program *prog = NULL; \ + struct bpf_link *link = NULL; \ + if (!ASSERT_OK_PTR(eskel, "exceptions_ext__open")) \ + goto done; \ + (expr); \ + ASSERT_OK_PTR(bpf_program__name(prog), bpf_program__name(prog)); \ + if (!ASSERT_EQ(exceptions_ext__load(eskel), load_ret, \ + "exceptions_ext__load")) { \ + printf("%s\n", log_buf); \ + goto done; \ + } \ + if (load_ret != 0) { \ + printf("%s\n", log_buf); \ + if (!ASSERT_OK_PTR(strstr(log_buf, msg), "strstr")) \ + goto done; \ + } \ + if (!load_ret && attach_err) { \ + if (!ASSERT_ERR_PTR(link = bpf_program__attach(prog), "attach err")) \ + goto done; \ + } else if (!load_ret) { \ + if (!ASSERT_OK_PTR(link = bpf_program__attach(prog), "attach ok")) \ + goto done; \ + (void)(after_link); \ + bpf_link__destroy(link); \ + } \ + } + + if (test__start_subtest("throwing extension with exception_cb")) + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext), + "exception_ext_global"), "set_attach_target")) + goto done; + }), "", ({ RUN_SUCCESS(exception_ext, 128); })); + + if (test__start_subtest("throwing exception_cb extension with exception_cb")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.throwing_exception_cb_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod"), "set_attach_target")) + goto done; + }), "Extension programs cannot replace exception callback", 0); + + if (test__start_subtest("throwing extension with exception_cb global")) + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_exception_cb_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_ext_mod_cb_runtime), + "exception_cb_mod_global"), "set_attach_target")) + goto done; + }), "", ({ RUN_SUCCESS(exception_ext_mod_cb_runtime, 131); })); + + if (test__start_subtest("non-throwing fexit -> non-throwing subprog")) + /* non-throwing fexit -> non-throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.pfexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing fexit -> non-throwing subprog")) + /* throwing fexit -> non-throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_fexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing fexit -> throwing subprog")) + /* non-throwing fexit -> throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.pfexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing fexit -> throwing subprog")) + /* throwing fexit -> throwing subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_fexit; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + /* fmod_ret not allowed for subprog - Check so we remember to handle its + * throwing specification compatibility with target when supported. + */ + if (test__start_subtest("non-throwing fmod_ret -> non-throwing subprog")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.pfmod_ret; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "can't modify return codes of BPF program", 0); + + /* fmod_ret not allowed for subprog - Check so we remember to handle its + * throwing specification compatibility with target when supported. + */ + if (test__start_subtest("non-throwing fmod_ret -> non-throwing global subprog")) + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.pfmod_ret; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "global_subprog"), "set_attach_target")) + goto done; + }), "can't modify return codes of BPF program", 0); + + if (test__start_subtest("non-throwing extension -> non-throwing subprog")) + /* non-throwing extension -> non-throwing subprog : BAD (!global) */ + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "subprog"), "set_attach_target")) + goto done; + }), "subprog() is not a global function", 0); + + if (test__start_subtest("non-throwing extension -> throwing subprog")) + /* non-throwing extension -> throwing subprog : BAD (!global) */ + RUN_EXT(-EINVAL, true, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_subprog"), "set_attach_target")) + goto done; + }), "throwing_subprog() is not a global function", 0); + + if (test__start_subtest("non-throwing extension -> non-throwing subprog")) + /* non-throwing extension -> non-throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing extension -> throwing global subprog")) + /* non-throwing extension -> throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing extension -> throwing global subprog")) + /* throwing extension -> throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "throwing_global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing extension -> non-throwing global subprog")) + /* throwing extension -> non-throwing global subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "global_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("non-throwing extension -> main subprog")) + /* non-throwing extension -> main subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "exception_throw_subprog"), "set_attach_target")) + goto done; + }), "", 0); + + if (test__start_subtest("throwing extension -> main subprog")) + /* throwing extension -> main subprog : OK */ + RUN_EXT(0, false, ({ + prog = eskel->progs.throwing_extension; + bpf_program__set_autoload(prog, true); + if (!ASSERT_OK(bpf_program__set_attach_target(prog, + bpf_program__fd(skel->progs.exception_throw_subprog), + "exception_throw_subprog"), "set_attach_target")) + goto done; + }), "", 0); + +done: + exceptions_ext__destroy(eskel); + exceptions__destroy(skel); +} + +static void test_exceptions_assertions(void) +{ + RUN_TESTS(exceptions_assert); +} + +void test_exceptions(void) +{ + test_exceptions_success(); + test_exceptions_failure(); + test_exceptions_assertions(); +} diff --git a/tools/testing/selftests/bpf/progs/exceptions.c b/tools/testing/selftests/bpf/progs/exceptions.c new file mode 100644 index 000000000000..2811ee842b01 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions.c @@ -0,0 +1,368 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" + +#ifndef ETH_P_IP +#define ETH_P_IP 0x0800 +#endif + +struct { + __uint(type, BPF_MAP_TYPE_PROG_ARRAY); + __uint(max_entries, 4); + __uint(key_size, sizeof(__u32)); + __uint(value_size, sizeof(__u32)); +} jmp_table SEC(".maps"); + +static __noinline int static_func(u64 i) +{ + bpf_throw(32); + return i; +} + +__noinline int global2static_simple(u64 i) +{ + static_func(i + 2); + return i - 1; +} + +__noinline int global2static(u64 i) +{ + if (i == ETH_P_IP) + bpf_throw(16); + return static_func(i); +} + +static __noinline int static2global(u64 i) +{ + return global2static(i) + i; +} + +SEC("tc") +int exception_throw_always_1(struct __sk_buff *ctx) +{ + bpf_throw(64); + return 0; +} + +/* In this case, the global func will never be seen executing after call to + * static subprog, hence verifier will DCE the remaining instructions. Ensure we + * are resilient to that. + */ +SEC("tc") +int exception_throw_always_2(struct __sk_buff *ctx) +{ + return global2static_simple(ctx->protocol); +} + +SEC("tc") +int exception_throw_unwind_1(struct __sk_buff *ctx) +{ + return static2global(bpf_ntohs(ctx->protocol)); +} + +SEC("tc") +int exception_throw_unwind_2(struct __sk_buff *ctx) +{ + return static2global(bpf_ntohs(ctx->protocol) - 1); +} + +SEC("tc") +int exception_throw_default(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 1; +} + +SEC("tc") +int exception_throw_default_value(struct __sk_buff *ctx) +{ + bpf_throw(5); + return 1; +} + +SEC("tc") +int exception_tail_call_target(struct __sk_buff *ctx) +{ + bpf_throw(16); + return 0; +} + +static __noinline +int exception_tail_call_subprog(struct __sk_buff *ctx) +{ + volatile int ret = 10; + + bpf_tail_call_static(ctx, &jmp_table, 0); + return ret; +} + +SEC("tc") +int exception_tail_call(struct __sk_buff *ctx) { + volatile int ret = 0; + + ret = exception_tail_call_subprog(ctx); + return ret + 8; +} + +__noinline int exception_ext_global(struct __sk_buff *ctx) +{ + volatile int ret = 0; + + return ret; +} + +static __noinline int exception_ext_static(struct __sk_buff *ctx) +{ + return exception_ext_global(ctx); +} + +SEC("tc") +int exception_ext(struct __sk_buff *ctx) +{ + return exception_ext_static(ctx); +} + +__noinline int exception_cb_mod_global(u64 cookie) +{ + volatile int ret = 0; + + return ret; +} + +/* Example of how the exception callback supplied during verification can still + * introduce extensions by calling to dummy global functions, and alter runtime + * behavior. + * + * Right now we don't allow freplace attachment to exception callback itself, + * but if the need arises this restriction is technically feasible to relax in + * the future. + */ +__noinline int exception_cb_mod(u64 cookie) +{ + return exception_cb_mod_global(cookie) + cookie + 10; +} + +SEC("tc") +__exception_cb(exception_cb_mod) +int exception_ext_mod_cb_runtime(struct __sk_buff *ctx) +{ + bpf_throw(25); + return 0; +} + +__noinline static int subprog(struct __sk_buff *ctx) +{ + return bpf_ktime_get_ns(); +} + +__noinline static int throwing_subprog(struct __sk_buff *ctx) +{ + if (ctx->tstamp) + bpf_throw(0); + return bpf_ktime_get_ns(); +} + +__noinline int global_subprog(struct __sk_buff *ctx) +{ + return bpf_ktime_get_ns(); +} + +__noinline int throwing_global_subprog(struct __sk_buff *ctx) +{ + if (ctx->tstamp) + bpf_throw(0); + return bpf_ktime_get_ns(); +} + +SEC("tc") +int exception_throw_subprog(struct __sk_buff *ctx) +{ + switch (ctx->protocol) { + case 1: + return subprog(ctx); + case 2: + return global_subprog(ctx); + case 3: + return throwing_subprog(ctx); + case 4: + return throwing_global_subprog(ctx); + default: + break; + } + bpf_throw(1); + return 0; +} + +__noinline int assert_nz_gfunc(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert(cookie != 0); + return 0; +} + +__noinline int assert_zero_gfunc(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert_eq(cookie, 0); + return 0; +} + +__noinline int assert_neg_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_lt(cookie, 0); + return 0; +} + +__noinline int assert_pos_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_gt(cookie, 0); + return 0; +} + +__noinline int assert_negeq_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_le(cookie, -1); + return 0; +} + +__noinline int assert_poseq_gfunc(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_ge(cookie, 1); + return 0; +} + +__noinline int assert_nz_gfunc_with(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert_with(cookie != 0, cookie + 100); + return 0; +} + +__noinline int assert_zero_gfunc_with(u64 c) +{ + volatile u64 cookie = c; + + bpf_assert_eq_with(cookie, 0, cookie + 100); + return 0; +} + +__noinline int assert_neg_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_lt_with(cookie, 0, cookie + 100); + return 0; +} + +__noinline int assert_pos_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_gt_with(cookie, 0, cookie + 100); + return 0; +} + +__noinline int assert_negeq_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_le_with(cookie, -1, cookie + 100); + return 0; +} + +__noinline int assert_poseq_gfunc_with(s64 c) +{ + volatile s64 cookie = c; + + bpf_assert_ge_with(cookie, 1, cookie + 100); + return 0; +} + +#define check_assert(name, cookie, tag) \ +SEC("tc") \ +int exception##tag##name(struct __sk_buff *ctx) \ +{ \ + return name(cookie) + 1; \ +} + +check_assert(assert_nz_gfunc, 5, _); +check_assert(assert_zero_gfunc, 0, _); +check_assert(assert_neg_gfunc, -100, _); +check_assert(assert_pos_gfunc, 100, _); +check_assert(assert_negeq_gfunc, -1, _); +check_assert(assert_poseq_gfunc, 1, _); + +check_assert(assert_nz_gfunc_with, 5, _); +check_assert(assert_zero_gfunc_with, 0, _); +check_assert(assert_neg_gfunc_with, -100, _); +check_assert(assert_pos_gfunc_with, 100, _); +check_assert(assert_negeq_gfunc_with, -1, _); +check_assert(assert_poseq_gfunc_with, 1, _); + +check_assert(assert_nz_gfunc, 0, _bad_); +check_assert(assert_zero_gfunc, 5, _bad_); +check_assert(assert_neg_gfunc, 100, _bad_); +check_assert(assert_pos_gfunc, -100, _bad_); +check_assert(assert_negeq_gfunc, 1, _bad_); +check_assert(assert_poseq_gfunc, -1, _bad_); + +check_assert(assert_nz_gfunc_with, 0, _bad_); +check_assert(assert_zero_gfunc_with, 5, _bad_); +check_assert(assert_neg_gfunc_with, 100, _bad_); +check_assert(assert_pos_gfunc_with, -100, _bad_); +check_assert(assert_negeq_gfunc_with, 1, _bad_); +check_assert(assert_poseq_gfunc_with, -1, _bad_); + +SEC("tc") +int exception_assert_range(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range(time, 0, ~0ULL); + return 1; +} + +SEC("tc") +int exception_assert_range_with(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range_with(time, 0, ~0ULL, 10); + return 1; +} + +SEC("tc") +int exception_bad_assert_range(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range(time, -100, 100); + return 1; +} + +SEC("tc") +int exception_bad_assert_range_with(struct __sk_buff *ctx) +{ + u64 time = bpf_ktime_get_ns(); + + bpf_assert_range_with(time, -1000, 1000, 10); + return 1; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_assert.c b/tools/testing/selftests/bpf/progs/exceptions_assert.c new file mode 100644 index 000000000000..fa35832e6748 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_assert.c @@ -0,0 +1,135 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include "bpf_misc.h" +#include "bpf_experimental.h" + +#define check_assert(type, op, name, value) \ + SEC("?tc") \ + __log_level(2) __failure \ + int check_assert_##op##_##name(void *ctx) \ + { \ + type num = bpf_ktime_get_ns(); \ + bpf_assert_##op(num, value); \ + return *(u64 *)num; \ + } + +__msg(": R0_w=-2147483648 R10=fp0") +check_assert(s64, eq, int_min, INT_MIN); +__msg(": R0_w=2147483647 R10=fp0") +check_assert(s64, eq, int_max, INT_MAX); +__msg(": R0_w=0 R10=fp0") +check_assert(s64, eq, zero, 0); +__msg(": R0_w=-9223372036854775808 R1_w=-9223372036854775808 R10=fp0") +check_assert(s64, eq, llong_min, LLONG_MIN); +__msg(": R0_w=9223372036854775807 R1_w=9223372036854775807 R10=fp0") +check_assert(s64, eq, llong_max, LLONG_MAX); + +__msg(": R0_w=scalar(smax=2147483646) R10=fp0") +check_assert(s64, lt, pos, INT_MAX); +__msg(": R0_w=scalar(umin=9223372036854775808,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +check_assert(s64, lt, zero, 0); +__msg(": R0_w=scalar(umin=9223372036854775808,umax=18446744071562067967,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +check_assert(s64, lt, neg, INT_MIN); + +__msg(": R0_w=scalar(smax=2147483647) R10=fp0") +check_assert(s64, le, pos, INT_MAX); +__msg(": R0_w=scalar(smax=0) R10=fp0") +check_assert(s64, le, zero, 0); +__msg(": R0_w=scalar(umin=9223372036854775808,umax=18446744071562067968,var_off=(0x8000000000000000; 0x7fffffffffffffff))") +check_assert(s64, le, neg, INT_MIN); + +__msg(": R0_w=scalar(umin=2147483648,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))") +check_assert(s64, gt, pos, INT_MAX); +__msg(": R0_w=scalar(umin=1,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))") +check_assert(s64, gt, zero, 0); +__msg(": R0_w=scalar(smin=-2147483647) R10=fp0") +check_assert(s64, gt, neg, INT_MIN); + +__msg(": R0_w=scalar(umin=2147483647,umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff))") +check_assert(s64, ge, pos, INT_MAX); +__msg(": R0_w=scalar(umax=9223372036854775807,var_off=(0x0; 0x7fffffffffffffff)) R10=fp0") +check_assert(s64, ge, zero, 0); +__msg(": R0_w=scalar(smin=-2147483648) R10=fp0") +check_assert(s64, ge, neg, INT_MIN); + +SEC("?tc") +__log_level(2) __failure +__msg(": R0=0 R1=ctx(off=0,imm=0) R2=scalar(smin=-2147483646,smax=2147483645) R10=fp0") +int check_assert_range_s64(struct __sk_buff *ctx) +{ + struct bpf_sock *sk = ctx->sk; + s64 num; + + _Static_assert(_Generic((sk->rx_queue_mapping), s32: 1, default: 0), "type match"); + if (!sk) + return 0; + num = sk->rx_queue_mapping; + bpf_assert_range(num, INT_MIN + 2, INT_MAX - 2); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R1=ctx(off=0,imm=0) R2=scalar(umin=4096,umax=8192,var_off=(0x0; 0x3fff))") +int check_assert_range_u64(struct __sk_buff *ctx) +{ + u64 num = ctx->len; + + bpf_assert_range(num, 4096, 8192); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R0=0 R1=ctx(off=0,imm=0) R2=4096 R10=fp0") +int check_assert_single_range_s64(struct __sk_buff *ctx) +{ + struct bpf_sock *sk = ctx->sk; + s64 num; + + _Static_assert(_Generic((sk->rx_queue_mapping), s32: 1, default: 0), "type match"); + if (!sk) + return 0; + num = sk->rx_queue_mapping; + + bpf_assert_range(num, 4096, 4096); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R1=ctx(off=0,imm=0) R2=4096 R10=fp0") +int check_assert_single_range_u64(struct __sk_buff *ctx) +{ + u64 num = ctx->len; + + bpf_assert_range(num, 4096, 4096); + return *((u8 *)ctx + num); +} + +SEC("?tc") +__log_level(2) __failure +__msg(": R1=pkt(off=64,r=64,imm=0) R2=pkt_end(off=0,imm=0) R6=pkt(off=0,r=64,imm=0) R10=fp0") +int check_assert_generic(struct __sk_buff *ctx) +{ + u8 *data_end = (void *)(long)ctx->data_end; + u8 *data = (void *)(long)ctx->data; + + bpf_assert(data + 64 <= data_end); + return data[128]; +} + +SEC("?fentry/bpf_check") +__failure __msg("At program exit the register R0 has value (0x40; 0x0)") +int check_assert_with_return(void *ctx) +{ + bpf_assert_with(!ctx, 64); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_ext.c b/tools/testing/selftests/bpf/progs/exceptions_ext.c new file mode 100644 index 000000000000..e37f05a03d86 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_ext.c @@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include "bpf_experimental.h" + +__noinline int exception_cb(u64 cookie) +{ + return cookie + 64; +} + +SEC("?freplace") +int extension(struct __sk_buff *ctx) +{ + return 0; +} + +SEC("?freplace") +__exception_cb(exception_cb) +int throwing_exception_cb_extension(u64 cookie) +{ + bpf_throw(32); + return 0; +} + +SEC("?freplace") +__exception_cb(exception_cb) +int throwing_extension(struct __sk_buff *ctx) +{ + bpf_throw(64); + return 0; +} + +SEC("?fexit") +int pfexit(void *ctx) +{ + return 0; +} + +SEC("?fexit") +int throwing_fexit(void *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?fmod_ret") +int pfmod_ret(void *ctx) +{ + return 0; +} + +SEC("?fmod_ret") +int throwing_fmod_ret(void *ctx) +{ + bpf_throw(0); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c new file mode 100644 index 000000000000..4c39e920dac2 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -0,0 +1,347 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include "bpf_misc.h" +#include "bpf_experimental.h" + +extern void bpf_rcu_read_lock(void) __ksym; + +#define private(name) SEC(".bss." #name) __hidden __attribute__((aligned(8))) + +struct foo { + struct bpf_rb_node node; +}; + +struct hmap_elem { + struct bpf_timer timer; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 64); + __type(key, int); + __type(value, struct hmap_elem); +} hmap SEC(".maps"); + +private(A) struct bpf_spin_lock lock; +private(A) struct bpf_rb_root rbtree __contains(foo, node); + +__noinline void *exception_cb_bad_ret_type(u64 cookie) +{ + return NULL; +} + +__noinline int exception_cb_bad_arg_0(void) +{ + return 0; +} + +__noinline int exception_cb_bad_arg_2(int a, int b) +{ + return 0; +} + +__noinline int exception_cb_ok_arg_small(int a) +{ + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_bad_ret_type) +__failure __msg("Global function exception_cb_bad_ret_type() doesn't return scalar.") +int reject_exception_cb_type_1(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_bad_arg_0) +__failure __msg("exception cb only supports single integer argument") +int reject_exception_cb_type_2(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_bad_arg_2) +__failure __msg("exception cb only supports single integer argument") +int reject_exception_cb_type_3(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb_ok_arg_small) +__success +int reject_exception_cb_type_4(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 0; +} + +__noinline +static int timer_cb(void *map, int *key, struct bpf_timer *timer) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback subprog") +int reject_async_callback_throw(struct __sk_buff *ctx) +{ + struct hmap_elem *elem; + + elem = bpf_map_lookup_elem(&hmap, &(int){0}); + if (!elem) + return 0; + return bpf_timer_set_callback(&elem->timer, timer_cb); +} + +__noinline static int subprog_lock(struct __sk_buff *ctx) +{ + volatile int ret = 0; + + bpf_spin_lock(&lock); + if (ctx->len) + bpf_throw(0); + return ret; +} + +SEC("?tc") +__failure __msg("function calls are not allowed while holding a lock") +int reject_with_lock(void *ctx) +{ + bpf_spin_lock(&lock); + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("function calls are not allowed while holding a lock") +int reject_subprog_with_lock(void *ctx) +{ + return subprog_lock(ctx); +} + +SEC("?tc") +__failure __msg("bpf_rcu_read_unlock is missing") +int reject_with_rcu_read_lock(void *ctx) +{ + bpf_rcu_read_lock(); + bpf_throw(0); + return 0; +} + +__noinline static int throwing_subprog(struct __sk_buff *ctx) +{ + if (ctx->len) + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("bpf_rcu_read_unlock is missing") +int reject_subprog_with_rcu_read_lock(void *ctx) +{ + bpf_rcu_read_lock(); + return throwing_subprog(ctx); +} + +static bool rbless(struct bpf_rb_node *n1, const struct bpf_rb_node *n2) +{ + bpf_throw(0); + return true; +} + +SEC("?tc") +__failure __msg("function calls are not allowed while holding a lock") +int reject_with_rbtree_add_throw(void *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_spin_lock(&lock); + bpf_rbtree_add(&rbtree, &f->node, rbless); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_reference(void *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_throw(0); + return 0; +} + +__noinline static int subprog_ref(struct __sk_buff *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_throw(0); + return 0; +} + +__noinline static int subprog_cb_ref(u32 i, void *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_cb_reference(void *ctx) +{ + struct foo *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_loop(5, subprog_cb_ref, NULL, 0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback") +int reject_with_cb(void *ctx) +{ + bpf_loop(5, subprog_cb_ref, NULL, 0); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_subprog_reference(void *ctx) +{ + return subprog_ref(ctx) + 1; +} + +__noinline int throwing_exception_cb(u64 c) +{ + bpf_throw(0); + return c; +} + +__noinline int exception_cb1(u64 c) +{ + return c; +} + +__noinline int exception_cb2(u64 c) +{ + return c; +} + +static __noinline int static_func(struct __sk_buff *ctx) +{ + return exception_cb1(ctx->tstamp); +} + +__noinline int global_func(struct __sk_buff *ctx) +{ + return exception_cb1(ctx->tstamp); +} + +SEC("?tc") +__exception_cb(throwing_exception_cb) +__failure __msg("cannot be called from callback subprog") +int reject_throwing_exception_cb(struct __sk_buff *ctx) +{ + return 0; +} + +SEC("?tc") +__exception_cb(exception_cb1) +__failure __msg("cannot call exception cb directly") +int reject_exception_cb_call_global_func(struct __sk_buff *ctx) +{ + return global_func(ctx); +} + +SEC("?tc") +__exception_cb(exception_cb1) +__failure __msg("cannot call exception cb directly") +int reject_exception_cb_call_static_func(struct __sk_buff *ctx) +{ + return static_func(ctx); +} + +SEC("?tc") +__exception_cb(exception_cb1) +__exception_cb(exception_cb2) +__failure __msg("multiple exception callback tags for main subprog") +int reject_multiple_exception_cb(struct __sk_buff *ctx) +{ + bpf_throw(0); + return 16; +} + +__noinline int exception_cb_bad_ret(u64 c) +{ + return c; +} + +SEC("?fentry/bpf_check") +__exception_cb(exception_cb_bad_ret) +__failure __msg("At program exit the register R0 has unknown scalar value should") +int reject_set_exception_cb_bad_ret1(void *ctx) +{ + return 0; +} + +SEC("?fentry/bpf_check") +__failure __msg("At program exit the register R0 has value (0x40; 0x0) should") +int reject_set_exception_cb_bad_ret2(void *ctx) +{ + bpf_throw(64); + return 0; +} + +__noinline static int loop_cb1(u32 index, int *ctx) +{ + bpf_throw(0); + return 0; +} + +__noinline static int loop_cb2(u32 index, int *ctx) +{ + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback") +int reject_exception_throw_cb(struct __sk_buff *ctx) +{ + bpf_loop(5, loop_cb1, NULL, 0); + return 0; +} + +SEC("?tc") +__failure __msg("cannot be called from callback") +int reject_exception_throw_cb_diff(struct __sk_buff *ctx) +{ + if (ctx->protocol) + bpf_loop(5, loop_cb1, NULL, 0); + else + bpf_loop(5, loop_cb2, NULL, 0); + return 0; +} + +char _license[] SEC("license") = "GPL";