From patchwork Fri Apr 5 12:43:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13619030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A8D5CD11C2 for ; Fri, 5 Apr 2024 12:44:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=t0K+N2yoKCqcFuPLvWgxqio48ZUof0T1QCoh8G5xJo8=; b=kFe7L6ztGiQhWX 4JmbWMbAcZB53+dupAOWMip9CMWvM9Xo63mVWArnsaqGLhqpQ1BgNvV9B8FOF5eBu71EoLRpUP9Pn TODPjr1M48bCYTI6SL8N8GWrHstgKFBM76ktEUR1vvv6hILTDF+EmfAd0j5zix3JUbon83s/F2AZg cAN62baPo8zVl7Aukbm6hJha+PQnl9KOIkJj6RzJ9lNMvAyaAg7OT7wC1qBOlurhdmmqhETkikbqy 6rAyan6DAXGFdR14sGyu8+TPaBTR/stscdZb9iJRRJ5DrOqGHAxxrP1u7kD6AAOYUdVlTYgrMBC6o DYPP8GtbX/mlH1H1cOMQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rsivN-000000070hv-0zRB; Fri, 05 Apr 2024 12:44:05 +0000 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rsivJ-000000070dN-1Ubd for linux-riscv@lists.infradead.org; Fri, 05 Apr 2024 12:44:03 +0000 Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-41551639550so15220015e9.2 for ; Fri, 05 Apr 2024 05:43:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712321038; x=1712925838; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=pULb2/SEKmMEonTMi0DJm45CRye8jTdoFzr8IafrgxM=; b=QTXKvwGsMexTdDgJJe3nRmGXjWRKehCpQ6aI0ZrHgcbjIzHepYob3w+DgV0f3c/AIb 6/KDSr5zbdOaMZdifgTF7RGwB/FRwDvU1nJdfQA1sT7jlZzvHe3in/7nF+i24MhvVYPB sHBGp8Qz8PJttIiBxsMPK+JsUuWsIldV7STFcQAi6nfOsI3BTNX16nXy38Y7bklzJgay Qhd9LJVaEwbLzI2YLDeLL7MLx/SC3l//gGpi45GG8d11W0Y1Cq2gVGXlsaiw0cMGMmpZ NgZCFoL3JfkWOYtVs9mCjJyOuf/pFDV6APLggRltllFVbF4BSY4hPQbQLydhbUeUnI6J Elug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712321038; x=1712925838; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pULb2/SEKmMEonTMi0DJm45CRye8jTdoFzr8IafrgxM=; b=eNMHFe0yeHp5T7NAcKLJ4x9BHKs5Bc8VcUPhtih8B9yYObDvxz6LfKlK2PJdRmnym4 rICtiuPOjBa1UbO01oFfaQRMMxHHX9KXSu+FBHcXThVhbCIUeMn5S1Lw/lNJwrse0UAW qjJnVKM1RaL74sQ5zUJvIfKdWqkHpfgJFNvRuF0fIcn9tVLhyXX5a204V66rqMBok+ey zVwEBJqLgUDsRbd5OJV6RycpBiBIwUJXLtxKXMz3S4/McGGO1oDE5lHhB4SuM2jisn8l 3JC8Pl+vHHVL9wmcCXc8n9RNjAl8jZhv1V7hsYKBffU0RaZu7qrQTGXeLE8zsn12xTXf Pkug== X-Forwarded-Encrypted: i=1; AJvYcCXMbkV7DO/8rhvYWtSLvZvOlR7b3d0+cjxMKmku5ohNRIKt7Alz2AYkHfcAfNK5aLhxSgBJkHOcoyirQe+4cIhivVITNmIm6P9M44VOLRXn X-Gm-Message-State: AOJu0YwUUiNPYiZtOErGlv2kH0DWuIWpR5YA2TzkbNBJ54RG5+E1cUFE O6FpY7z5mfgNqjv2bC/ZoQircV4ApaOzQn9rvdR3ks+Kyq3v85t4 X-Google-Smtp-Source: AGHT+IHUW+bDfIBmn2iekT8yv3sAv6IIGKq2Tu5MgysTRgeCJ5Z1WV+/TuTm7uyrmNDypSyF3mGedA== X-Received: by 2002:a05:600c:a0d:b0:416:331f:d3b3 with SMTP id z13-20020a05600c0a0d00b00416331fd3b3mr53944wmp.38.1712321037814; Fri, 05 Apr 2024 05:43:57 -0700 (PDT) Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id n32-20020a05600c3ba000b00415612e29a1sm2887862wms.30.2024.04.05.05.43.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 05 Apr 2024 05:43:57 -0700 (PDT) From: Puranjay Mohan To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Paul Walmsley , Palmer Dabbelt , Albert Ou , bpf@vger.kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Pu Lehui Cc: puranjay12@gmail.com Subject: [PATCH bpf-next] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs Date: Fri, 5 Apr 2024 12:43:48 +0000 Message-Id: <20240405124348.27644-1-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240405_054401_965471_22EBADA3 X-CRM114-Status: GOOD ( 13.60 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs. RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu). Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register. To measure the improvement from this change, the benchmark in [1] was used on Qemu: Before: glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/s After: glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s [1] https://github.com/anakryiko/linux/commit/8dec900975ef Signed-off-by: Puranjay Mohan --- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 15e482f2c657..e95bd1d459a4 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -12,6 +12,7 @@ #include #include #include +#include #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2 @@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_or(RV_REG_T1, rd, RV_REG_T1, ctx); emit_mv(rd, RV_REG_T1, ctx); break; + } else if (insn_is_mov_percpu_addr(insn)) { + if (rd != rs) + emit_mv(rd, rs, ctx); +#ifdef CONFIG_SMP + /* Load current CPU number in T1 */ + emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP, + ctx); + /* << 3 because offsets are 8 bytes */ + emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx); + /* Load address of __per_cpu_offset array in T2 */ + emit_imm(RV_REG_T2, (u64)&__per_cpu_offset, ctx); + /* Add offset of current CPU to __per_cpu_offset */ + emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx); + /* Load __per_cpu_offset[cpu] in T1 */ + emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx); + /* Add the offset to Rd */ + emit_add(rd, rd, RV_REG_T1, ctx); +#endif } if (imm == 1) { /* Special mov32 for zext */ @@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void) { return true; } + +bool bpf_jit_supports_percpu_insn(void) +{ + return true; +}