From patchwork Fri Apr 5 09:17:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13618768 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCCEDC67861 for ; Fri, 5 Apr 2024 09:17:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=Ue+e2BOxay2IwWP4sYhfLK1F1KG9tpzGKTAYo4JBYgs=; b=LR+lc+doiAk1E8 qEVCOPPmmqIlZLIZhzGYWWd9Ve2mFj6HEr1jI83sxeCuB052IQOabdfjnn/fVZQ9uL3iTGuruFEzM m5i6Io3/KHWyB/Sq2yq79SdP6eD8uWMICn1coJYn7u5fnHw8NOQKxdhahz0TMdOv9r7O9QJuiGHgy SMhnHrCn3qWmHUNcgHQBb/8Ieay6XwlJ6aDu2AIe7Bj4OIhQHFTRHY+71UfcY4016VMpJ7xWPOuLZ 3688CuZ9O1kabwqlCwzlo2GKrt7sCjG2HjQz0wlgImHih07FI6Qu/nHzQ9P2LlNNsmLEg0oOjUWXF zT8dNLqtQsFULhLe17dA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rsfhZ-000000069Cf-26wS; Fri, 05 Apr 2024 09:17:37 +0000 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rsfhV-0000000699v-34pP for linux-arm-kernel@lists.infradead.org; Fri, 05 Apr 2024 09:17:35 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-41632010478so1119775e9.1 for ; Fri, 05 Apr 2024 02:17:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712308649; x=1712913449; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=LX/ma26TDvYVdpCIWg6iUhgi/+t6jX0tkNjGdFErDfk=; b=duPLpImEtRk3jWJdNgLM7FSmF4cb1YO96i+jzWyyUWseAi6FPBTczpo1dQwTzwqeKd RySkx5B5hEA1alxLefNVzBNhmORMlQoPl7cPp27gCJDIwbmPpnvqTwy90nHNIZstOQbS l4F7HDgK/crltAqVzhxnHeXBw9WdaL5tdpPCvo+BVAdZzfZAwyYzKgutJAD22ViDfNNf +G/9NNSQnsvbQl7u0YnrAJi9TdNH7jEAhImpevnZiongvzZpNDLxMdzlkZEtLewfFUYl Kd0HbyLTdAur+DP+eVy5nScVPahkd21IR3oWcNNtk6ajhXZ8Qo057nBjKQE79MKPATFa jHvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712308649; x=1712913449; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LX/ma26TDvYVdpCIWg6iUhgi/+t6jX0tkNjGdFErDfk=; b=eneqxqRX5aXPSMhmeH4RlhhDuQfw9woz2+a9WDRFU5XRhCnY5jCI4byAp8HA1+3xvz Fi4q/5uP4/WVKg7GUp5oFPr2x0nFb/Fe3E9VpDR8ukiURNJ/CcJhBqbAmxRQZsoMRTFh 9KNGbyw1pWFn1axr/QZznQs61zVHQQwFpVEED6iftdMbSzzfYyFL0oDHcQKaHoSma0Df 4nCMLnz2E37aXjOMFdoork5LnDITRD7TQ8bMQvBkiO5vjnUGoKAUfGuJWM4cNrjZLc+F awY93PVZp5KFh+kJGyA7XM02H6FIuZzevd9x76j5RODcM/SR4HS8BwGziSZq/JCVUIdi xDDw== X-Forwarded-Encrypted: i=1; AJvYcCXFsMl1y+Dr8MrLl7HEqQZGPy2wMzODc5B9NSzqZdb1Cbhkb3R0JyepzkW4EihTFC3UrCk7M3P2QFWvru06gkrqRT6dBKxvz0EWdacMudeBQLOvuAU= X-Gm-Message-State: AOJu0YzIqS/2BGPQi0UE35lfUv0i3yCv8BNF9LCTPkfXe4CIObCQSkA6 8WmFVeANj9o21Vyrjd67/i8GjTCjnV3/1PijmeCYVh2c8Zkb0/bc X-Google-Smtp-Source: AGHT+IGgfHTLTIsVgQ3B5HauM2cnUDg4HorY6H22kp7XWlCO7vVHucxahGea+Fkf1Sjbk1Up4r//pg== X-Received: by 2002:a05:600c:4f41:b0:416:1df0:7c06 with SMTP id m1-20020a05600c4f4100b004161df07c06mr933170wmq.6.1712308648855; Fri, 05 Apr 2024 02:17:28 -0700 (PDT) Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id v13-20020a05600c444d00b00416306c17basm1017286wmn.14.2024.04.05.02.17.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Apr 2024 02:17:28 -0700 (PDT) From: Puranjay Mohan To: Catalin Marinas , Will Deacon , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Xu Kuohai , Florent Revest , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Cc: puranjay12@gmail.com Subject: [PATCH bpf-next] arm64, bpf: add internal-only MOV instruction to resolve per-CPU addrs Date: Fri, 5 Apr 2024 09:17:07 +0000 Message-Id: <20240405091707.66675-1-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240405_021734_051748_EBCC450A X-CRM114-Status: GOOD ( 15.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs. Since commit 7158627686f0 ("arm64: percpu: implement optimised pcpu access using tpidr_el1"), the per-cpu offset for the CPU is stored in the tpidr_el1/2 register of that CPU. To support this BPF instruction in the ARM64 JIT, the following ARM64 instructions are emitted: mov dst, src // Move src to dst, if src != dst mrs tmp, tpidr_el1/2 // Move per-cpu offset of the current cpu in tmp. add dst, dst, tmp // Add the per cpu offset to the dst. If CONFIG_SMP is not defined, then nothing is emitted if src == dst, and mov dst, src is emitted if dst != src. To measure the performance improvement provided by this change, the benchmark in [1] was used: Before: glob-arr-inc : 23.597 ± 0.012M/s arr-inc : 23.173 ± 0.019M/s hash-inc : 12.186 ± 0.028M/s After: glob-arr-inc : 23.819 ± 0.034M/s arr-inc : 23.285 ± 0.017M/s hash-inc : 12.419 ± 0.011M/s [1] https://github.com/anakryiko/linux/commit/8dec900975ef Signed-off-by: Puranjay Mohan --- arch/arm64/include/asm/insn.h | 7 +++++++ arch/arm64/lib/insn.c | 11 +++++++++++ arch/arm64/net/bpf_jit.h | 6 ++++++ arch/arm64/net/bpf_jit_comp.c | 16 ++++++++++++++++ 4 files changed, 40 insertions(+) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index db1aeacd4cd9..d16d68550c22 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -135,6 +135,11 @@ enum aarch64_insn_special_register { AARCH64_INSN_SPCLREG_SP_EL2 = 0xF210 }; +enum aarch64_insn_system_register { + AARCH64_INSN_SYSREG_TPIDR_EL1 = 0xC684, + AARCH64_INSN_SYSREG_TPIDR_EL2 = 0xE682, +}; + enum aarch64_insn_variant { AARCH64_INSN_VARIANT_32BIT, AARCH64_INSN_VARIANT_64BIT @@ -686,6 +691,8 @@ u32 aarch64_insn_gen_cas(enum aarch64_insn_register result, } #endif u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type); +u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result, + enum aarch64_insn_system_register sysreg); s32 aarch64_get_branch_offset(u32 insn); u32 aarch64_set_branch_offset(u32 insn, s32 offset); diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c index a635ab83fee3..b008a9b46a7f 100644 --- a/arch/arm64/lib/insn.c +++ b/arch/arm64/lib/insn.c @@ -1515,3 +1515,14 @@ u32 aarch64_insn_gen_dmb(enum aarch64_insn_mb_type type) return insn; } + +u32 aarch64_insn_gen_mrs(enum aarch64_insn_register result, + enum aarch64_insn_system_register sysreg) +{ + u32 insn = aarch64_insn_get_mrs_value(); + + insn &= ~GENMASK(19, 0); + insn |= sysreg << 5; + return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, + insn, result); +} diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index 23b1b34db088..b627ef7188c7 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -297,4 +297,10 @@ #define A64_ADR(Rd, offset) \ aarch64_insn_gen_adr(0, offset, Rd, AARCH64_INSN_ADR_TYPE_ADR) +/* MRS */ +#define A64_MRS_TPIDR_EL1(Rt) \ + aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL1) +#define A64_MRS_TPIDR_EL2(Rt) \ + aarch64_insn_gen_mrs(Rt, AARCH64_INSN_SYSREG_TPIDR_EL2) + #endif /* _BPF_JIT_H */ diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 76b91f36c729..e9ad9f257a18 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -877,6 +877,17 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, emit(A64_ORR(1, tmp, dst, tmp), ctx); emit(A64_MOV(1, dst, tmp), ctx); break; + } else if (insn_is_mov_percpu_addr(insn)) { + if (dst != src) + emit(A64_MOV(1, dst, src), ctx); +#ifdef CONFIG_SMP + if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + emit(A64_MRS_TPIDR_EL2(tmp), ctx); + else + emit(A64_MRS_TPIDR_EL1(tmp), ctx); + emit(A64_ADD(1, dst, dst, tmp), ctx); +#endif + break; } switch (insn->off) { case 0: @@ -2527,6 +2538,11 @@ bool bpf_jit_supports_arena(void) return true; } +bool bpf_jit_supports_percpu_insn(void) +{ + return true; +} + void bpf_jit_free(struct bpf_prog *prog) { if (prog->jited) {