From patchwork Fri Apr 26 12:13:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13644694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE3C4C4345F for ; Fri, 26 Apr 2024 12:14:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8PtGAdaIsqrNvDVGnPzqelH9snApl+sIyYUNioLG8Wg=; b=b2Osae2ve2h7Ft fZh4Ko5fQwTB+SeHA0nR+hJ1BEIr1Tcut1o/14Goq3mIezkmA/+rmuZpLbahJutPJbqO4Hl7XGKot nf4Ov0a+oVoAIfFSSxfzgGEG77keJ18fa61axeQP7EX3BDVzpfMFok9uqzpaZk4/3DAWNGFaxDKTN 8oL0p6r2K/3GAcrX4mWwRsBYleDi2MfxsYpCP7up6G2Eo0/viHfrzTDq6A+pqCY9imxOv1jfQlqj1 vgmrEg3eiNclyEp0noxFz4/rLeF9eyFNhBJP2cB3NiEk+yL+ClSjkg0xgl4omFXARFIdNUcEMR07V QW4J1CNd2PySvvgcvtlg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0KT4-0000000CQsl-2BKt; Fri, 26 Apr 2024 12:14:18 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0KSt-0000000CQo6-0pgK for linux-arm-kernel@lists.infradead.org; Fri, 26 Apr 2024 12:14:12 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 20CECCE1C31; Fri, 26 Apr 2024 12:14:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF1E2C116B1; Fri, 26 Apr 2024 12:14:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714133644; bh=TbPOTBmS82ynsL8h5jbyPu2ZW+Geh2onSqdzZPRv0KM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=misLKGhSz7D7/HB5koSgsOhfKKRCFUwuQxfqLAwA3MPbiSCJqYBbBSrqKb6jOh3Fv wkcr/1yw7TSMrfNKB0p8PM/0n1aDwNWjTJZ2UjFJ0on1enlEJDKC+0lMpV6bQqrkPh e+A8DyPrDkIeWOO4Cl80IpKa+INEtNApt5ARjZeNgDeOpsC0BB3SvpPaJM1dkUWYfd 0CPJmksC2w3YRvebNajBPYSZhMdC6K6vVAwHp824DjgO9QdyjuWzmHAtf+XXEOXFcq KI+Vm5+D5dvy8jY7P733NXd4qyfY9WCnKGTicPP7yFHqRpJV7PoqFC/B/6AKpspZ4I pkhHN79NlkOAQ== From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v3 1/2] arm64, bpf: add internal-only MOV instruction to resolve per-CPU addrs Date: Fri, 26 Apr 2024 12:13:48 +0000 Message-Id: <20240426121349.97651-2-puranjay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240426121349.97651-1-puranjay@kernel.org> References: <20240426121349.97651-1-puranjay@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240426_051407_604896_CCC9B746 X-CRM114-Status: GOOD ( 13.56 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Puranjay Mohan Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs. Since commit 7158627686f0 ("arm64: percpu: implement optimised pcpu access using tpidr_el1"), the per-cpu offset for the CPU is stored in the tpidr_el1/2 register of that CPU. To support this BPF instruction in the ARM64 JIT, the following ARM64 instructions are emitted: mov dst, src // Move src to dst, if src != dst mrs tmp, tpidr_el1/2 // Move per-cpu offset of the current cpu in tmp. add dst, dst, tmp // Add the per cpu offset to the dst. To measure the performance improvement provided by this change, the benchmark in [1] was used: Before: glob-arr-inc : 23.597 ± 0.012M/s arr-inc : 23.173 ± 0.019M/s hash-inc : 12.186 ± 0.028M/s After: glob-arr-inc : 23.819 ± 0.034M/s arr-inc : 23.285 ± 0.017M/s hash-inc : 12.419 ± 0.011M/s [1] https://github.com/anakryiko/linux/commit/8dec900975ef Signed-off-by: Puranjay Mohan Acked-by: Andrii Nakryiko --- arch/arm64/include/asm/insn.h | 7 +++++++ arch/arm64/lib/insn.c | 11 +++++++++++ arch/arm64/net/bpf_jit.h | 6 ++++++ arch/arm64/net/bpf_jit_comp.c | 14 ++++++++++++++ 4 files changed, 38 insertions(+) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index db1aeacd4cd9..8de0e39b29f3 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -135,6 +135,11 @@ enum aarch64_insn_special_register { AARCH64_INSN_SPCLREG_SP_EL2 = 0xF210 }; +enum aarch64_insn_system_register { + AARCH64_INSN_SYSREG_TPIDR_EL1 = 0x4684, + AARCH64_INSN_SYSREG_TPIDR_EL2 = 0x6682, +}; + enum aarch64_insn_variant { AARCH64_INSN_VARIANT_32BIT, AARCH64_INSN_VARIANT_64BIT @@ -686,6 +691,8 @@ u32 aarch64_insn_gen_cas(enum aarch64_insn_register result, } #endif u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type); +u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result, + enum aarch64_insn_system_register sysreg); s32 aarch64_get_branch_offset(u32 insn); u32 aarch64_set_branch_offset(u32 insn, s32 offset); diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c index a635ab83fee3..b008a9b46a7f 100644 --- a/arch/arm64/lib/insn.c +++ b/arch/arm64/lib/insn.c @@ -1515,3 +1515,14 @@ u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type) return insn; } + +u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result, + enum aarch64_insn_system_register sysreg) +{ + u32 insn = aarch64_insn_get_mrs_value(); + + insn &= ~GENMASK(19, 0); + insn |= sysreg << 5; + return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, + insn, result); +} diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index 23b1b34db088..b627ef7188c7 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -297,4 +297,10 @@ #define A64_ADR(Rd, offset) \ aarch64_insn_gen_adr(0, offset, Rd, AARCH64_INSN_ADR_TYPE_ADR) +/* MRS */ +#define A64_MRS_TPIDR_EL1(Rt) \ + aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL1) +#define A64_MRS_TPIDR_EL2(Rt) \ + aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL2) + #endif /* _BPF_JIT_H */ diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 76b91f36c729..ed8f9716d9d5 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -877,6 +877,15 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit(A64_ORR(1, tmp, dst, tmp), ctx); emit(A64_MOV(1, dst, tmp), ctx); break; + } else if (insn_is_mov_percpu_addr(insn)) { + if (dst != src) + emit(A64_MOV(1, dst, src), ctx); + if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + emit(A64_MRS_TPIDR_EL2(tmp), ctx); + else + emit(A64_MRS_TPIDR_EL1(tmp), ctx); + emit(A64_ADD(1, dst, dst, tmp), ctx); + break; } switch (insn->off) { case 0: @@ -2527,6 +2536,11 @@ bool bpf_jit_supports_arena(void) return true; } +bool bpf_jit_supports_percpu_insn(void) +{ + return true; +} + void bpf_jit_free(struct bpf_prog *prog) { if (prog->jited) { From patchwork Fri Apr 26 12:13:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13644693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4E4EC4345F for ; Fri, 26 Apr 2024 12:14:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=e9Dq+anMe644bZ99XkCrvajpU/c9OIjirelDt1hdUMQ=; b=YPBPeS+Q/WyMNL DWkEMfulcVbliKAJUpACtjN8jRl2H05LGsVkLKOCQm4MiA6LjUbH4hi7183tp7GiNgWwQJ8Tz2ERT jiX/ibcY0kWotUHkfanX0Kj6im0YUOuIb7pusnt9IxvsokUmmGJfBhFns5iD5JqWv3L+VNjKKRTsq zfJLysBpnee9ZIR1Yvbb0O7ZuvAujzld1hHTWjglay4lMhZ7oea9n1yW3sLRanSIQtbbyhtpRHcrq gq//3K2Ujf42EPNpzYkfxdqcaKzC2anR+w5aDCIiDyEEDEDldNbOe1a3jMtf98aQK3ztfyaDLT6+S XK/ypGTM8J86Y6NNOgfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0KT3-0000000CQs6-38gE; Fri, 26 Apr 2024 12:14:17 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0KSv-0000000CQop-07A4 for linux-arm-kernel@lists.infradead.org; Fri, 26 Apr 2024 12:14:11 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 5F2FB62068; Fri, 26 Apr 2024 12:14:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BBFA4C113CD; Fri, 26 Apr 2024 12:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714133647; bh=wwOo7Qalbj5LGH+Uol2Kxq1QroiMAQbRn89m80qF1hk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wqf1Cp+LfX9tULU0HyRGnDeMCjKen3b6QQDxRgGRuwAOuj5zbDCwVCKJhd6Auo08o B8pZrXZMAP5IDBLwE4CD69u9MCDX+5lGz3aytu4cFT6NezNCUyof9nt0CGFJ4LdqqH OSzFvR0EG1Jq5AwOg97W36gELTUZaDggc9TxZ+cniSes9RMNhAaYl/p/vr7t8VNKRL Fqm0tEbdObc+W3DUtSF7mCNOR3m/AvCsEvQSaA+VUlsTzIOzkO7SVTB/Ou4VK2ROF9 OQA1ipw7x+LCI+/G2ni7UaGuPcCUc9PlDEZil5BL3pGoMFnQuhkZ4yZSQNnkHzt9Df 1qa2PZ05giUcA== From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v3 2/2] bpf, arm64: inline bpf_get_smp_processor_id() helper Date: Fri, 26 Apr 2024 12:13:49 +0000 Message-Id: <20240426121349.97651-3-puranjay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240426121349.97651-1-puranjay@kernel.org> References: <20240426121349.97651-1-puranjay@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240426_051409_179838_D6D0A690 X-CRM114-Status: GOOD ( 12.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As ARM64 JIT now implements BPF_MOV64_PERCPU_REG instruction, inline bpf_get_smp_processor_id(). ARM64 uses the per-cpu variable cpu_number to store the cpu id. Here is how the BPF and ARM64 JITed assembly changes after this commit: BPF ===== BEFORE AFTER -------- ------- int cpu = bpf_get_smp_processor_id(); int cpu = bpf_get_smp_processor_id(); (85) call bpf_get_smp_processor_id#229032 (18) r0 = 0xffff800082072008 (bf) r0 = &(void __percpu *)(r0) (61) r0 = *(u32 *)(r0 +0) ARM64 JIT =========== BEFORE AFTER -------- ------- int cpu = bpf_get_smp_processor_id(); int cpu = bpf_get_smp_processor_id(); mov x10, #0xfffffffffffff4d0 mov x7, #0xffff8000ffffffff movk x10, #0x802b, lsl #16 movk x7, #0x8207, lsl #16 movk x10, #0x8000, lsl #32 movk x7, #0x2008 blr x10 mrs x10, tpidr_el1 add x7, x0, #0x0 add x7, x7, x10 ldr w7, [x7] Performance improvement using benchmark[1] BEFORE AFTER -------- ------- glob-arr-inc : 23.817 ± 0.019M/s glob-arr-inc : 24.631 ± 0.027M/s arr-inc : 23.253 ± 0.019M/s arr-inc : 23.742 ± 0.023M/s hash-inc : 12.258 ± 0.010M/s hash-inc : 12.625 ± 0.004M/s [1] https://github.com/anakryiko/linux/commit/8dec900975ef Signed-off-by: Puranjay Mohan Acked-by: Andrii Nakryiko --- kernel/bpf/verifier.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4e474ef44e9c..6ff4e63b2ef2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20273,20 +20273,31 @@ static int do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } -#ifdef CONFIG_X86_64 /* Implement bpf_get_smp_processor_id() inline. */ if (insn->imm == BPF_FUNC_get_smp_processor_id && prog->jit_requested && bpf_jit_supports_percpu_insn()) { /* BPF_FUNC_get_smp_processor_id inlining is an - * optimization, so if pcpu_hot.cpu_number is ever + * optimization, so if cpu_number_addr is ever * changed in some incompatible and hard to support * way, it's fine to back out this inlining logic */ - insn_buf[0] = BPF_MOV32_IMM(BPF_REG_0, (u32)(unsigned long)&pcpu_hot.cpu_number); - insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); - insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); - cnt = 3; + u64 cpu_number_addr; +#if defined(CONFIG_X86_64) + cpu_number_addr = (u64)&pcpu_hot.cpu_number; +#elif defined(CONFIG_ARM64) + cpu_number_addr = (u64)&cpu_number; +#else + goto next_insn; +#endif + struct bpf_insn ld_cpu_number_addr[2] = { + BPF_LD_IMM64(BPF_REG_0, cpu_number_addr) + }; + insn_buf[0] = ld_cpu_number_addr[0]; + insn_buf[1] = ld_cpu_number_addr[1]; + insn_buf[2] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); + insn_buf[3] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); + cnt = 4; new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); if (!new_prog) return -ENOMEM; @@ -20296,7 +20307,6 @@ static int do_misc_fixups(struct bpf_verifier_env *env) insn = new_prog->insnsi + i + delta; goto next_insn; } -#endif /* Implement bpf_get_func_arg inline. */ if (prog_type == BPF_PROG_TYPE_TRACING && insn->imm == BPF_FUNC_get_func_arg) {