From patchwork Fri Apr 26 12:13:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13644693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4E4EC4345F for ; Fri, 26 Apr 2024 12:14:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=e9Dq+anMe644bZ99XkCrvajpU/c9OIjirelDt1hdUMQ=; b=YPBPeS+Q/WyMNL DWkEMfulcVbliKAJUpACtjN8jRl2H05LGsVkLKOCQm4MiA6LjUbH4hi7183tp7GiNgWwQJ8Tz2ERT jiX/ibcY0kWotUHkfanX0Kj6im0YUOuIb7pusnt9IxvsokUmmGJfBhFns5iD5JqWv3L+VNjKKRTsq zfJLysBpnee9ZIR1Yvbb0O7ZuvAujzld1hHTWjglay4lMhZ7oea9n1yW3sLRanSIQtbbyhtpRHcrq gq//3K2Ujf42EPNpzYkfxdqcaKzC2anR+w5aDCIiDyEEDEDldNbOe1a3jMtf98aQK3ztfyaDLT6+S XK/ypGTM8J86Y6NNOgfQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0KT3-0000000CQs6-38gE; Fri, 26 Apr 2024 12:14:17 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s0KSv-0000000CQop-07A4 for linux-arm-kernel@lists.infradead.org; Fri, 26 Apr 2024 12:14:11 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 5F2FB62068; Fri, 26 Apr 2024 12:14:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BBFA4C113CD; Fri, 26 Apr 2024 12:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714133647; bh=wwOo7Qalbj5LGH+Uol2Kxq1QroiMAQbRn89m80qF1hk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Wqf1Cp+LfX9tULU0HyRGnDeMCjKen3b6QQDxRgGRuwAOuj5zbDCwVCKJhd6Auo08o B8pZrXZMAP5IDBLwE4CD69u9MCDX+5lGz3aytu4cFT6NezNCUyof9nt0CGFJ4LdqqH OSzFvR0EG1Jq5AwOg97W36gELTUZaDggc9TxZ+cniSes9RMNhAaYl/p/vr7t8VNKRL Fqm0tEbdObc+W3DUtSF7mCNOR3m/AvCsEvQSaA+VUlsTzIOzkO7SVTB/Ou4VK2ROF9 OQA1ipw7x+LCI+/G2ni7UaGuPcCUc9PlDEZil5BL3pGoMFnQuhkZ4yZSQNnkHzt9Df 1qa2PZ05giUcA== From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v3 2/2] bpf, arm64: inline bpf_get_smp_processor_id() helper Date: Fri, 26 Apr 2024 12:13:49 +0000 Message-Id: <20240426121349.97651-3-puranjay@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240426121349.97651-1-puranjay@kernel.org> References: <20240426121349.97651-1-puranjay@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240426_051409_179838_D6D0A690 X-CRM114-Status: GOOD ( 12.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As ARM64 JIT now implements BPF_MOV64_PERCPU_REG instruction, inline bpf_get_smp_processor_id(). ARM64 uses the per-cpu variable cpu_number to store the cpu id. Here is how the BPF and ARM64 JITed assembly changes after this commit: BPF ===== BEFORE AFTER -------- ------- int cpu = bpf_get_smp_processor_id(); int cpu = bpf_get_smp_processor_id(); (85) call bpf_get_smp_processor_id#229032 (18) r0 = 0xffff800082072008 (bf) r0 = &(void __percpu *)(r0) (61) r0 = *(u32 *)(r0 +0) ARM64 JIT =========== BEFORE AFTER -------- ------- int cpu = bpf_get_smp_processor_id(); int cpu = bpf_get_smp_processor_id(); mov x10, #0xfffffffffffff4d0 mov x7, #0xffff8000ffffffff movk x10, #0x802b, lsl #16 movk x7, #0x8207, lsl #16 movk x10, #0x8000, lsl #32 movk x7, #0x2008 blr x10 mrs x10, tpidr_el1 add x7, x0, #0x0 add x7, x7, x10 ldr w7, [x7] Performance improvement using benchmark[1] BEFORE AFTER -------- ------- glob-arr-inc : 23.817 ± 0.019M/s glob-arr-inc : 24.631 ± 0.027M/s arr-inc : 23.253 ± 0.019M/s arr-inc : 23.742 ± 0.023M/s hash-inc : 12.258 ± 0.010M/s hash-inc : 12.625 ± 0.004M/s [1] https://github.com/anakryiko/linux/commit/8dec900975ef Signed-off-by: Puranjay Mohan Acked-by: Andrii Nakryiko --- kernel/bpf/verifier.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 4e474ef44e9c..6ff4e63b2ef2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -20273,20 +20273,31 @@ static int do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } -#ifdef CONFIG_X86_64 /* Implement bpf_get_smp_processor_id() inline. */ if (insn->imm == BPF_FUNC_get_smp_processor_id && prog->jit_requested && bpf_jit_supports_percpu_insn()) { /* BPF_FUNC_get_smp_processor_id inlining is an - * optimization, so if pcpu_hot.cpu_number is ever + * optimization, so if cpu_number_addr is ever * changed in some incompatible and hard to support * way, it's fine to back out this inlining logic */ - insn_buf[0] = BPF_MOV32_IMM(BPF_REG_0, (u32)(unsigned long)&pcpu_hot.cpu_number); - insn_buf[1] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); - insn_buf[2] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); - cnt = 3; + u64 cpu_number_addr; +#if defined(CONFIG_X86_64) + cpu_number_addr = (u64)&pcpu_hot.cpu_number; +#elif defined(CONFIG_ARM64) + cpu_number_addr = (u64)&cpu_number; +#else + goto next_insn; +#endif + struct bpf_insn ld_cpu_number_addr[2] = { + BPF_LD_IMM64(BPF_REG_0, cpu_number_addr) + }; + insn_buf[0] = ld_cpu_number_addr[0]; + insn_buf[1] = ld_cpu_number_addr[1]; + insn_buf[2] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); + insn_buf[3] = BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); + cnt = 4; new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt); if (!new_prog) return -ENOMEM; @@ -20296,7 +20307,6 @@ static int do_misc_fixups(struct bpf_verifier_env *env) insn = new_prog->insnsi + i + delta; goto next_insn; } -#endif /* Implement bpf_get_func_arg inline. */ if (prog_type == BPF_PROG_TYPE_TRACING && insn->imm == BPF_FUNC_get_func_arg) {