From patchwork Sun Mar 24 21:15:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13600947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E58DC47DD9 for ; Sun, 24 Mar 2024 21:15:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=B+RJWoAU3dvh87xO45jXfjehG7KX/hrEne1VQjT3hIk=; b=MiTYOVn2epFpXU x8YFpeLntvBZA5nDQWxSXlRv1cOYTt+CqQpi08xw6tdUKs8TwDrG+FPSvi1Zth8T2joX3o6aZS8vN SKb0+ffsCb0rBwa5NJJMaM1ZMFUl8tsm+4G0lfjjNSHVAKKhLxHV5/JpWp0WSIRXHCPiEAbSqd2Yu iGrhHWLEa7scO4ezBjyg+naSKG2+qPhBzesK6XkEPLRSrRoxNjBQvRfF21Jd+aXmIeNSZ5eh6CMU7 i+2AjfEgVyX9qc9ezjL328gLPHMzabqbm4yeXpCjUdaTm4N6X1B6HrbpVZJNb6rnyq6Z6UPpA6tSb 494AJCWkq7dldMpAJmHA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1roVBo-0000000DTg0-3QJJ; Sun, 24 Mar 2024 21:15:36 +0000 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1roVBg-0000000DTeS-2QEo for linux-arm-kernel@lists.infradead.org; Sun, 24 Mar 2024 21:15:29 +0000 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-33ececeb19eso2397311f8f.3 for ; Sun, 24 Mar 2024 14:15:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711314927; x=1711919727; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r+hQBZmnSVN9HewOiblrQK3C2T98xP0zStDNINtIws0=; b=lVclE3n3usaLVdqAezLOn05EEEvTp70K+qDWrtVIsBlaQjALXnWyYcPmpHchaxWdET w0hVik0UNdTZ0DHwLe+d44TfIoSWS0l7cuW8g7TA+lwKRy2hD5LZCG4QJq/TLULijHTU 5N42HeCAiNqfGlAkjGTiElvF4n2qMFST/+VFafOq9DWH+nuPnZq7BxISeHCgqetr3kDn 8DLZbcjebmb/pfkcY7Wl4X+7LML5n8lDJVRvtXlRVesZygAxHNe+LUrm8IAPmqEcs+zy guV8gXnAuN3GT+dGH9UmTx0zbsQ4fwfmzJZ70ZEahBWH1i8wWHgVj3lWHhHwCn9tdja4 aR8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711314927; x=1711919727; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r+hQBZmnSVN9HewOiblrQK3C2T98xP0zStDNINtIws0=; b=KFz46Ddh+ikQmlXmOTDXJLjLNnfTw5n3JOFBWhUAB2ZV+aU/hNrlV95GhNtwyAtSny L7tRMMwexPRSbQeD+XNPxN6u7cYS/wa7K/rRAGUIDxSFs4xBkkD+/wFdCUYOgqypgiGj +tg/EfnqbaYftxxfxGMf5i0/8PkBnzoJqI++CMjvt6tA9Yx3oRUzoDI6UJ183ZRAseuk robspM+mSlAgb2MZqjhOP0F4xa1Hp3VeYvR4SYoC1Mx6IhXCSslUfHr9CMBOC6AExKM3 aM7P+ekvOIlWeMnLz6vysUVRGLJ5AiVNF6w+WQYO9rdi1VGcc5wGN91AAmH3GhaWnkSI TFDA== X-Forwarded-Encrypted: i=1; AJvYcCXxeP70teurHExLJrDYfCXONzyjkuTvPFIMzumHYLZOLbUAcbRZI291Uj4P7Eu0p6ZzopTeIvhHvvkJvQUOnTNcCM+2ipHe4ubEA+LNhmVnr4ZlDwY= X-Gm-Message-State: AOJu0YwWLk6/loZr0k19gSm7v63b4pnsHrEpkGGmvRIO4BTiiryBkBEQ achNn1tr5WQjMI1CEAZUkH5cCgPK74OJFx2iwsFPoCtQ1b3wA2/8 X-Google-Smtp-Source: AGHT+IF09u55LK04HZ2m7C725taCQ/WMxwE6LLOn76lGa8Iy6EkDzdDPuIlI/2LpVPSHkKzFCPGiOw== X-Received: by 2002:adf:e887:0:b0:341:bfe2:4509 with SMTP id d7-20020adfe887000000b00341bfe24509mr3183574wrm.42.1711314926668; Sun, 24 Mar 2024 14:15:26 -0700 (PDT) Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id t13-20020a5d42cd000000b0033ec312cd8asm7625890wrr.33.2024.03.24.14.15.26 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 24 Mar 2024 14:15:26 -0700 (PDT) From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Mark Rutland , Suzuki K Poulose , Mark Brown , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org (open list), bpf@vger.kernel.org (open list:BPF [GENERAL] (Safe Dynamic Programs and Tools)), Josh Poimboeuf Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v2 1/1] arm64/cfi,bpf: Support kCFI + BPF on arm64 Date: Sun, 24 Mar 2024 21:15:18 +0000 Message-Id: <20240324211518.93892-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240324211518.93892-1-puranjay12@gmail.com> References: <20240324211518.93892-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240324_141528_645787_4AB80026 X-CRM114-Status: GOOD ( 25.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, bpf_dispatcher_*_func() is marked with `__nocfi` therefore calling BPF programs from this interface doesn't cause CFI warnings. When BPF programs are called directly from C: from BPF helpers or struct_ops, CFI warnings are generated. Implement proper CFI prologues for the BPF programs and callbacks and drop __nocfi for arm64. Fix the trampoline generation code to emit kCFI prologue when a struct_ops trampoline is being prepared. Signed-off-by: Puranjay Mohan Signed-off-by: Mark Rutland --- arch/arm64/include/asm/cfi.h | 23 ++++++++++++++ arch/arm64/kernel/alternative.c | 54 +++++++++++++++++++++++++++++++++ arch/arm64/net/bpf_jit_comp.c | 28 +++++++++++++---- 3 files changed, 99 insertions(+), 6 deletions(-) create mode 100644 arch/arm64/include/asm/cfi.h diff --git a/arch/arm64/include/asm/cfi.h b/arch/arm64/include/asm/cfi.h new file mode 100644 index 000000000000..670e191f8628 --- /dev/null +++ b/arch/arm64/include/asm/cfi.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_ARM64_CFI_H +#define _ASM_ARM64_CFI_H + +#ifdef CONFIG_CFI_CLANG +#define __bpfcall +static inline int cfi_get_offset(void) +{ + return 4; +} +#define cfi_get_offset cfi_get_offset +extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; +extern u32 cfi_get_func_hash(void *func); +#else +#define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash(void *func) +{ + return 0; +} +#endif /* CONFIG_CFI_CLANG */ +#endif /* _ASM_ARM64_CFI_H */ diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c index 8ff6610af496..1715da7df137 100644 --- a/arch/arm64/kernel/alternative.c +++ b/arch/arm64/kernel/alternative.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -298,3 +299,56 @@ noinstr void alt_cb_patch_nops(struct alt_instr *alt, __le32 *origptr, updptr[i] = cpu_to_le32(aarch64_insn_gen_nop()); } EXPORT_SYMBOL(alt_cb_patch_nops); + +#ifdef CONFIG_CFI_CLANG +struct bpf_insn; + +/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ +extern unsigned int __bpf_prog_runX(const void *ctx, + const struct bpf_insn *insn); + +/* + * Force a reference to the external symbol so the compiler generates + * __kcfi_typid. + */ +__ADDRESSABLE(__bpf_prog_runX); + +/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_hash,@object \n" +" .globl cfi_bpf_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_hash: \n" +" .word __kcfi_typeid___bpf_prog_runX \n" +" .size cfi_bpf_hash, 4 \n" +" .popsection \n" +); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); + +__ADDRESSABLE(__bpf_callback_fn); + +/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_subprog_hash,@object \n" +" .globl cfi_bpf_subprog_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_subprog_hash: \n" +" .word __kcfi_typeid___bpf_callback_fn \n" +" .size cfi_bpf_subprog_hash, 4 \n" +" .popsection \n" +); + +u32 cfi_get_func_hash(void *func) +{ + u32 hash; + + if (get_kernel_nofault(hash, func - cfi_get_offset())) + return 0; + + return hash; +} +#endif diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index bc16eb694657..2372812bb47c 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -158,6 +159,12 @@ static inline void emit_bti(u32 insn, struct jit_ctx *ctx) emit(insn, ctx); } +static inline void emit_kcfi(u32 hash, struct jit_ctx *ctx) +{ + if (IS_ENABLED(CONFIG_CFI_CLANG)) + emit(hash, ctx); +} + /* * Kernel addresses in the vmalloc space use at most 48 bits, and the * remaining bits are guaranteed to be 0x1. So we can compose the address @@ -295,7 +302,7 @@ static bool is_lsi_offset(int offset, int scale) #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8) static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, - bool is_exception_cb) + bool is_exception_cb, bool is_subprog) { const struct bpf_prog *prog = ctx->prog; const bool is_main_prog = !bpf_is_subprog(prog); @@ -306,7 +313,6 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, const u8 fp = bpf2a64[BPF_REG_FP]; const u8 tcc = bpf2a64[TCALL_CNT]; const u8 fpb = bpf2a64[FP_BOTTOM]; - const int idx0 = ctx->idx; int cur_offset; /* @@ -332,6 +338,8 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, * */ + emit_kcfi(is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash, ctx); + const int idx0 = ctx->idx; /* bpf function may be invoked by 3 instruction types: * 1. bl, attached via freplace to bpf prog via short jump * 2. br, attached via freplace to bpf prog via long jump @@ -1648,7 +1656,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) * BPF line info needs ctx->offset[i] to be the offset of * instruction[i] in jited image, so build prologue first. */ - if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) { + if (build_prologue(&ctx, was_classic, prog->aux->exception_cb, + bpf_is_subprog(prog))) { prog = orig_prog; goto out_off; } @@ -1696,7 +1705,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.idx = 0; ctx.exentry_idx = 0; - build_prologue(&ctx, was_classic, prog->aux->exception_cb); + build_prologue(&ctx, was_classic, prog->aux->exception_cb, + bpf_is_subprog(prog)); if (build_body(&ctx, extra_pass)) { prog = orig_prog; @@ -1745,9 +1755,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) jit_data->ro_header = ro_header; } - prog->bpf_func = (void *)ctx.ro_image; + prog->bpf_func = (void *)ctx.ro_image + cfi_get_offset(); prog->jited = 1; - prog->jited_len = prog_size; + prog->jited_len = prog_size - cfi_get_offset(); if (!prog->is_func || extra_pass) { int i; @@ -2011,6 +2021,12 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, /* return address locates above FP */ retaddr_off = stack_size + 8; + if (flags & BPF_TRAMP_F_INDIRECT) { + /* + * Indirect call for bpf_struct_ops + */ + emit_kcfi(cfi_get_func_hash(func_addr), ctx); + } /* bpf trampoline may be invoked by 3 instruction types: * 1. bl, attached to bpf prog or kernel function via short jump * 2. br, attached to bpf prog or kernel function via long jump