From patchwork Sat Nov 9 00:41:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 13869120 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71B10D2FB for ; Sat, 9 Nov 2024 00:42:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.145.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112965; cv=none; b=AX8DsG/SMzGtS+l9Bfy9DvJm9wUmXOqc9oNQs1IgKFeoePSAQIGAUhi6Hg3j87sNL+t8WClkaGUo7uiVKJlA8vmN22popPKbN7hVXcPJCYDXBt53GyJkUiMQD6jk+110Jt1nYl+2B4CoPAuOzEUtf2osXrCqlNGaR1lGfpIZ9sQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112965; c=relaxed/simple; bh=FC8OeIxBlGVyD3KmNtL1d1NOWPH8xmLzYFcRs519h8M=; h=From:To:CC:Subject:Date:Message-ID:MIME-Version:Content-Type; b=iwjgvoVqjpvtudfQ8VGpV6fG1maFtPI2Lbm/dTVKGpZFKkiCjkbVtVaB8Oh1ux2fh822PgU4Ux+e2Hn4P0X6tQ4lD6guiWdNBoboryF2XMLuLHVCC8hSeTeRfWUiE0BV/3Hd3MCx6zXk3gC2aiO5PGfqALKm71V8ZvqeP1UEysA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=FKXskP7J; arc=none smtp.client-ip=67.231.145.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="FKXskP7J" Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4A8LFGTd023316; Fri, 8 Nov 2024 16:42:09 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to; s=s2048-2021-q4; bh=EYzQ1pHzoHnEVrijzk BBLtGrjKwtKv52fx5MPaCAesY=; b=FKXskP7JIYG0/sK5GoKwo6XUfKDxAAY9Na f+Y5ZtFTDscqTpM6OBX7tReh+MRq+Jqt2s5gq3NzL1qO+t6nXABOQA7YnFpsxeoo b4Xnjc2Y6TNpy3GPSTwK6oDfxHLwjVCrR0hMc3dFWUYByhrDTUcu1n+Ro2/FcUy/ aeQCkBESxzKWecbXE1IKblFXShxhI7zUOFTHQqYZ9Ljav1iE5YKhT61k4jCGwVUs 82C0paiThybPomhnPpA/Sub2eCiW3oRfxpniCmbLuhLqZtvBjsvV9KYf98xhGp0t 6OECBbrgvmouMbrpt4FV+nx545DSPjhEQrbTojSAgVKx4v1LNfVg== Received: from mail.thefacebook.com ([163.114.134.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 42sn58bstp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 08 Nov 2024 16:42:09 -0800 (PST) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.11; Sat, 9 Nov 2024 00:42:06 +0000 From: Vadim Fedorenko To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Vadim Fedorenko , Mykola Lysenko , Jakub Kicinski CC: , , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v5 1/4] bpf: add bpf_get_cpu_cycles kfunc Date: Fri, 8 Nov 2024 16:41:54 -0800 Message-ID: <20241109004158.2259301-1-vadfed@meta.com> X-Mailer: git-send-email 2.43.5 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: 8ie3mUxZiEy2DeNWFXLD_8lVFw7jBz05 X-Proofpoint-GUID: 8ie3mUxZiEy2DeNWFXLD_8lVFw7jBz05 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-05_03,2024-10-04_01,2024-09-30_01 X-Patchwork-Delegate: bpf@iogearbox.net New kfunc to return ARCH-specific timecounter. For x86 BPF JIT converts it into rdtsc ordered call. Other architectures will get JIT implementation too if supported. The fallback is to __arch_get_hw_counter(). Signed-off-by: Vadim Fedorenko Acked-by: Eduard Zingerman --- v4 -> v5: * use if instead of ifdef with IS_ENABLED v3 -> v4: * change name of the helper to bpf_get_cpu_cycles (Andrii) * Hide the helper behind CONFIG_GENERIC_GETTIMEOFDAY to avoid exposing it on architectures which do not have vDSO functions and data * reduce the scope of check of inlined functions in verifier to only 2, which are actually inlined. v2 -> v3: * change name of the helper to bpf_get_cpu_cycles_counter to explicitly mention what counter it provides (Andrii) * move kfunc definition to bpf.h to use it in JIT. * introduce another kfunc to convert cycles into nanoseconds as more meaningful time units for generic tracing use case (Andrii) v1 -> v2: * Fix incorrect function return value type to u64 * Introduce bpf_jit_inlines_kfunc_call() and use it in mark_fastcall_pattern_for_call() to avoid clobbering in case of running programs with no JIT (Eduard) * Avoid rewriting instruction and check function pointer directly in JIT (Alexei) * Change includes to fix compile issues on non x86 architectures --- arch/x86/net/bpf_jit_comp.c | 28 ++++++++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 14 ++++++++++++++ include/linux/bpf.h | 5 +++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 +++++++++++ kernel/bpf/helpers.c | 13 +++++++++++++ kernel/bpf/verifier.c | 30 +++++++++++++++++++++++++++++- 7 files changed, 101 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 06b080b61aa5..4f78ed93ee7f 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2126,6 +2126,26 @@ st: if (is_imm8(insn->off)) case BPF_JMP | BPF_CALL: { u8 *ip = image + addrs[i - 1]; + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + imm32 == BPF_CALL_IMM(bpf_get_cpu_cycles)) { + /* Save RDX because RDTSC will use EDX:EAX to return u64 */ + emit_mov_reg(&prog, true, AUX_REG, BPF_REG_3); + if (boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) + EMIT_LFENCE(); + EMIT2(0x0F, 0x31); + + /* shl RDX, 32 */ + maybe_emit_1mod(&prog, BPF_REG_3, true); + EMIT3(0xC1, add_1reg(0xE0, BPF_REG_3), 32); + /* or RAX, RDX */ + maybe_emit_mod(&prog, BPF_REG_0, BPF_REG_3, true); + EMIT2(0x09, add_2reg(0xC0, BPF_REG_0, BPF_REG_3)); + /* restore RDX from R11 */ + emit_mov_reg(&prog, true, BPF_REG_3, AUX_REG); + + break; + } + func = (u8 *) __bpf_call_base + imm32; if (tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth); @@ -3652,3 +3672,11 @@ u64 bpf_arch_uaddress_limit(void) { return 0; } + +/* x86-64 JIT can inline kfunc */ +bool bpf_jit_inlines_kfunc_call(s32 imm) +{ + if (imm == BPF_CALL_IMM(bpf_get_cpu_cycles)) + return true; + return false; +} diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index de0f9e5f9f73..e6097a371b69 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -2094,6 +2094,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { int err; + if (imm32 == BPF_CALL_IMM(bpf_get_cpu_cycles)) { + if (boot_cpu_has(X86_FEATURE_LFENCE_RDTSC)) + EMIT3(0x0F, 0xAE, 0xE8); + EMIT2(0x0F, 0x31); + break; + } + err = emit_kfunc_call(bpf_prog, image + addrs[i], insn, &prog); @@ -2621,3 +2628,10 @@ bool bpf_jit_supports_kfunc_call(void) { return true; } + +bool bpf_jit_inlines_kfunc_call(s32 imm) +{ + if (imm == BPF_CALL_IMM(bpf_get_cpu_cycles)) + return true; + return false; +} diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1b84613b10ac..fed5f36d387a 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3328,6 +3328,11 @@ void bpf_user_rnd_init_once(void); u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); +/* Inlined kfuncs */ +#if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) +u64 bpf_get_cpu_cycles(void); +#endif + #if defined(CONFIG_NET) bool bpf_sock_common_is_valid_access(int off, int size, enum bpf_access_type type, diff --git a/include/linux/filter.h b/include/linux/filter.h index 7d7578a8eac1..8bdd5e6b2a65 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1111,6 +1111,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); bool bpf_jit_needs_zext(void); bool bpf_jit_inlines_helper_call(s32 imm); +bool bpf_jit_inlines_kfunc_call(s32 imm); bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_percpu_insn(void); bool bpf_jit_supports_kfunc_call(void); diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 233ea78f8f1b..ab6a2452ade0 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2965,6 +2965,17 @@ bool __weak bpf_jit_inlines_helper_call(s32 imm) return false; } +/* Return true if the JIT inlines the call to the kfunc corresponding to + * the imm. + * + * The verifier will not patch the insn->imm for the call to the helper if + * this returns true. + */ +bool __weak bpf_jit_inlines_kfunc_call(s32 imm) +{ + return false; +} + /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls(void) { diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 395221e53832..5c6c0383ebf4 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -23,6 +23,9 @@ #include #include #include +#if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) +#include +#endif #include "../../lib/kstrtox.h" @@ -3023,6 +3026,13 @@ __bpf_kfunc int bpf_copy_from_user_str(void *dst, u32 dst__sz, const void __user return ret + 1; } +#if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) +__bpf_kfunc u64 bpf_get_cpu_cycles(void) +{ + return __arch_get_hw_counter(1, NULL); +} +#endif + __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -3115,6 +3125,9 @@ BTF_ID_FLAGS(func, bpf_get_kmem_cache) BTF_ID_FLAGS(func, bpf_iter_kmem_cache_new, KF_ITER_NEW | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLEEPABLE) BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) +#if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) +BTF_ID_FLAGS(func, bpf_get_cpu_cycles, KF_FASTCALL) +#endif BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7958d6ff6b73..b5220d996231 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16273,6 +16273,24 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } } +/* True if fixup_kfunc_call() replaces calls to kfunc number 'imm', + * replacement patch is presumed to follow bpf_fastcall contract + * (see mark_fastcall_pattern_for_call() below). + */ +static bool verifier_inlines_kfunc_call(struct bpf_verifier_env *env, s32 imm) +{ + const struct bpf_kfunc_desc *desc = find_kfunc_desc(env->prog, imm, 0); + + if (!env->prog->jit_requested) + return false; + + if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) + return true; + + return false; +} + /* Same as helper_fastcall_clobber_mask() but for kfuncs, see comment above */ static u32 kfunc_fastcall_clobber_mask(struct bpf_kfunc_call_arg_meta *meta) { @@ -16400,7 +16418,10 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env, return; clobbered_regs_mask = kfunc_fastcall_clobber_mask(&meta); - can_be_inlined = is_fastcall_kfunc_call(&meta); + can_be_inlined = is_fastcall_kfunc_call(&meta) && + (verifier_inlines_kfunc_call(env, call->imm) || + (meta.btf == btf_vmlinux && + bpf_jit_inlines_kfunc_call(call->imm))); } if (clobbered_regs_mask == ALL_CALLER_SAVED_REGS) @@ -20402,6 +20423,7 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, struct bpf_insn *insn_buf, int insn_idx, int *cnt) { const struct bpf_kfunc_desc *desc; + s32 imm = insn->imm; if (!insn->imm) { verbose(env, "invalid kernel function call not eliminated in verifier pass\n"); @@ -20488,6 +20510,12 @@ static int fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, node_offset_reg, insn, insn_buf, cnt); } else if (desc->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || desc->func_id == special_kfunc_list[KF_bpf_rdonly_cast]) { + if (!verifier_inlines_kfunc_call(env, imm)) { + verbose(env, "verifier internal error: kfunc id %d is not defined in checker\n", + desc->func_id); + return -EFAULT; + } + insn_buf[0] = BPF_MOV64_REG(BPF_REG_0, BPF_REG_1); *cnt = 1; } else if (is_bpf_wq_set_callback_impl_kfunc(desc->func_id)) { From patchwork Sat Nov 9 00:41:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 13869118 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1E798C07 for ; Sat, 9 Nov 2024 00:42:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112959; cv=none; b=rM2aPM4uJpman0vuiWBU3OXavInPztciowj+43Nj57OTw72OfqfZXL7SWZponEGeCMnzFw9M9Dqxxb/chb8SIFBgVLBwt+TTxD/tPWRwbwnAdx+WBAV7sPNFRZMOnrM+G4YFo61H1jq9XTE4fVLLuEhZnJVxITLwz1Y9jBg5kpw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112959; c=relaxed/simple; bh=lCA7kE8eJPbbby/OWYXwaSgudjElro42JjrwaPz8fko=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=hpc7L5/Ef3Hr4o+pfuDM4ucWG9HwRZIWEHjRELfVFZGwqpgH/G0BHIVbcJi4xvCbqI33quelRzCYFmYdhmDAYFDqKfBEY56Nf4TWpKB5aMkF3KUtqyjVrOSUTvHyEmVFsh4/Vws6iZzRLOKLOS9KzxPPd57oesI1uAMa3qhAkEQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=QVgvVCTb; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="QVgvVCTb" Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 4A905Rcd031770; Fri, 8 Nov 2024 16:42:12 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=KSj1dXaY4HBmCHkzSYAKNoaKkZ7HsCKIEK6LG7RtK+U=; b=QVgvVCTb/QvV A7YVp5WJdWUj0SpTMdnZtp/NOAp7umAE8ekwQKVgnMADiZuXzleqTC6HQMIDF6i7 XXl3ewzVYCHdujNL+sx90TMjEywEiVBDFCl5SjWz5IaA4kbqxwqrrMPhax+zNPl0 hjhqtOA6QVQ2YQjOTDUtoMYYVgoTov3928wV8ATX1i4ypyFJq/S2qguUxUOeC2C2 RcsrR7a4Vsye1CIEEuNQemQVRNdrZQV9EnHmH1pCo2Tp+pzADcv5/PcEFJWRguae CA3SlvczStP9qXD9GcTTKA5p/M/rRSPu/lvyNsnslFH6H9hG5H4LNflZShi0CGrk 4dn9Mv/6ig== Received: from mail.thefacebook.com ([163.114.134.16]) by m0001303.ppops.net (PPS) with ESMTPS id 42svvsg65e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 08 Nov 2024 16:42:12 -0800 (PST) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.11; Sat, 9 Nov 2024 00:42:08 +0000 From: Vadim Fedorenko To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Vadim Fedorenko , Mykola Lysenko , Jakub Kicinski CC: , , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v5 2/4] bpf: add bpf_cpu_cycles_to_ns helper Date: Fri, 8 Nov 2024 16:41:55 -0800 Message-ID: <20241109004158.2259301-2-vadfed@meta.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241109004158.2259301-1-vadfed@meta.com> References: <20241109004158.2259301-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: X2YPAdLbcbBEVufkNP00XKR47RdL772g X-Proofpoint-GUID: X2YPAdLbcbBEVufkNP00XKR47RdL772g X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-05_03,2024-10-04_01,2024-09-30_01 X-Patchwork-Delegate: bpf@iogearbox.net The new helper should be used to convert cycles received by bpf_get_cpu_cycle() into nanoseconds. Signed-off-by: Vadim Fedorenko Reviewed-by: Eduard Zingerman --- v5: * no changes v4: * change helper name to bpf_cpu_cycles_to_ns. * hide it behind CONFIG_GENERIC_GETTIMEOFDAY to avoid exposing on unsupported architectures. --- arch/x86/net/bpf_jit_comp.c | 22 ++++++++++++++++++++++ arch/x86/net/bpf_jit_comp32.c | 19 +++++++++++++++++++ include/linux/bpf.h | 1 + kernel/bpf/helpers.c | 10 +++++++++- 4 files changed, 51 insertions(+), 1 deletion(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 4f78ed93ee7f..ddc73d9a90f4 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -2146,6 +2147,24 @@ st: if (is_imm8(insn->off)) break; } + if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && + imm32 == BPF_CALL_IMM(bpf_cpu_cycles_to_ns) && + boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) { + u32 mult, shift; + + clocks_calc_mult_shift(&mult, &shift, tsc_khz, USEC_PER_SEC, 0); + /* imul RAX, RDI, mult */ + maybe_emit_mod(&prog, BPF_REG_1, BPF_REG_0, true); + EMIT2_off32(0x69, add_2reg(0xC0, BPF_REG_1, BPF_REG_0), + mult); + + /* shr RAX, shift (which is less than 64) */ + maybe_emit_1mod(&prog, BPF_REG_0, true); + EMIT3(0xC1, add_1reg(0xE8, BPF_REG_0), shift); + + break; + } + func = (u8 *) __bpf_call_base + imm32; if (tail_call_reachable) { LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth); @@ -3678,5 +3697,8 @@ bool bpf_jit_inlines_kfunc_call(s32 imm) { if (imm == BPF_CALL_IMM(bpf_get_cpu_cycles)) return true; + if (imm == BPF_CALL_IMM(bpf_cpu_cycles_to_ns) && + boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) + return true; return false; } diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index e6097a371b69..34f762f28c82 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -2100,6 +2101,24 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT2(0x0F, 0x31); break; } + if (imm32 == BPF_CALL_IMM(bpf_cpu_cycles_to_ns) && + boot_cpu_has(X86_FEATURE_CONSTANT_TSC)) { + u32 mult, shift; + + clocks_calc_mult_shift(&mult, &shift, tsc_khz, + USEC_PER_SEC, 0); + + /* move parameter to BPF_REG_0 */ + emit_ia32_mov_r64(true, bpf2ia32[BPF_REG_0], + bpf2ia32[BPF_REG_1], true, true, + &prog, bpf_prog->aux); + /* multiply parameter by mut */ + emit_ia32_mul_i64(bpf2ia32[BPF_REG_0], + mult, true, &prog); + /* shift parameter by shift which is less than 64 */ + emit_ia32_rsh_i64(bpf2ia32[BPF_REG_0], + shift, true, &prog); + } err = emit_kfunc_call(bpf_prog, image + addrs[i], diff --git a/include/linux/bpf.h b/include/linux/bpf.h index fed5f36d387a..46fa662d95e4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3331,6 +3331,7 @@ u64 bpf_get_raw_cpu_id(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); /* Inlined kfuncs */ #if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) u64 bpf_get_cpu_cycles(void); +u64 bpf_cpu_cycles_to_ns(u64 cycles); #endif #if defined(CONFIG_NET) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 5c6c0383ebf4..72d5819e5df2 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -25,6 +25,7 @@ #include #if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) #include +#include #endif #include "../../lib/kstrtox.h" @@ -3031,8 +3032,14 @@ __bpf_kfunc u64 bpf_get_cpu_cycles(void) { return __arch_get_hw_counter(1, NULL); } -#endif +__bpf_kfunc u64 bpf_cpu_cycles_to_ns(u64 cycles) +{ + const struct vdso_data *vd = __arch_get_k_vdso_data(); + + return mul_u64_u32_shr(cycles, vd->mult, vd->shift); +} +#endif __bpf_kfunc_end_defs(); BTF_KFUNCS_START(generic_btf_ids) @@ -3127,6 +3134,7 @@ BTF_ID_FLAGS(func, bpf_iter_kmem_cache_next, KF_ITER_NEXT | KF_RET_NULL | KF_SLE BTF_ID_FLAGS(func, bpf_iter_kmem_cache_destroy, KF_ITER_DESTROY | KF_SLEEPABLE) #if IS_ENABLED(CONFIG_GENERIC_GETTIMEOFDAY) BTF_ID_FLAGS(func, bpf_get_cpu_cycles, KF_FASTCALL) +BTF_ID_FLAGS(func, bpf_cpu_cycles_to_ns, KF_FASTCALL) #endif BTF_KFUNCS_END(common_btf_ids) From patchwork Sat Nov 9 00:41:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 13869121 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79DA18C0B for ; Sat, 9 Nov 2024 00:42:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.145.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112970; cv=none; b=eyijPT704qC2NTzjKlk2BJyk6P3EXHXXHjTrN5guSs3u8Rr12cSJuyvIkQ/jhp1RsJaWIOD7qs6kfB2V74MnPu67ZiMFHizGnfWw3AiJgpd5FLkhpaV42JVkWxywKZurJzx2ndxjHedFPLYXlCrkGcJlThIZZmsq0GvxJJAaEk4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112970; c=relaxed/simple; bh=WGsV1SaFfLC/K3ND2x1htghQvqRTeu5V6tnV/ziQL4Y=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=H54jj6nZNM4+J+fTkK5tmlzpLIcwW30U1AtN5IjGYK0/VIL8X48sZ5tuZAWre+KQlLFReRN7xQ9z/walv5Rminljv9tcsIt+Mpae75C71ab4WT5dFLyiAN40m0rGoigU8MghKdgKM8H4rNy6ZXz0Dz9nimnqksGTaqkJOj0mfEk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=XmKgOkQt; arc=none smtp.client-ip=67.231.145.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="XmKgOkQt" Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4A8LFFAU023237; Fri, 8 Nov 2024 16:42:14 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=dcI28znQL45vPqQFKE1MV54vNgiEo9OBLY6TCGqUKGs=; b=XmKgOkQtV4Qk 7lv3pG8SCORw894q/56JLFP9rAWy9oAzOnLbjTfKKim3qj2FjOYckcvJYQQwshma tWK1az7xFCyjCXpn0+tF0bADROccy2MjXUfTThvWm19YNPH32alr5DbZHivL7Trn FqweW9/ZcsTKOMCXzcgtqlFjM7HXEGkak9fIZHmfzz19bleq1pItGikT33skcuUA qexx7Na0c3WC2NvUAwDWZZzijb8JuvMogtMYVsk7kM0OS+OhB7Bf5t2zQCF2uXRP br0/a9wJWzRGsXeEYbbIQHV1sdbKvkXT1TV/DvYn5q7b0mllxgY5k0PGOonBA+oP Ihlojd7Ppw== Received: from mail.thefacebook.com ([163.114.134.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 42sn58bsu2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 08 Nov 2024 16:42:14 -0800 (PST) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.11; Sat, 9 Nov 2024 00:42:11 +0000 From: Vadim Fedorenko To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Vadim Fedorenko , Mykola Lysenko , Jakub Kicinski CC: , , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v5 3/4] selftests/bpf: add selftest to check rdtsc jit Date: Fri, 8 Nov 2024 16:41:56 -0800 Message-ID: <20241109004158.2259301-3-vadfed@meta.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241109004158.2259301-1-vadfed@meta.com> References: <20241109004158.2259301-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: zykTJXMSLZjWqG1RwYnTt80TNgEtOdP2 X-Proofpoint-GUID: zykTJXMSLZjWqG1RwYnTt80TNgEtOdP2 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-05_03,2024-10-04_01,2024-09-30_01 X-Patchwork-Delegate: bpf@iogearbox.net get_cpu_cycles() is replaced with rdtsc instruction on x86_64. Add tests to check it. Signed-off-by: Vadim Fedorenko Acked-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_cpu_cycles.c | 104 ++++++++++++++++++ 2 files changed, 106 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 75f7a2ce334b..d76cc90f4bfb 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -97,6 +97,7 @@ #include "verifier_xdp_direct_packet_access.skel.h" #include "verifier_bits_iter.skel.h" #include "verifier_lsm.skel.h" +#include "verifier_cpu_cycles.skel.h" #define MAX_ENTRIES 11 @@ -223,6 +224,7 @@ void test_verifier_xdp(void) { RUN(verifier_xdp); } void test_verifier_xdp_direct_packet_access(void) { RUN(verifier_xdp_direct_packet_access); } void test_verifier_bits_iter(void) { RUN(verifier_bits_iter); } void test_verifier_lsm(void) { RUN(verifier_lsm); } +void test_verifier_cpu_cycles(void) { RUN(verifier_cpu_cycles); } void test_verifier_mtu(void) { diff --git a/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c b/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c new file mode 100644 index 000000000000..88bfa7211858 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_cpu_cycles.c @@ -0,0 +1,104 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2022 Meta Inc. */ +#include "vmlinux.h" +#include +#include +#include "bpf_misc.h" + +extern u64 bpf_cpu_cycles_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_cycles(void) __weak __ksym; + +SEC("syscall") +__arch_x86_64 +__xlated("0: call kernel-function") +__naked int bpf_rdtsc(void) +{ + asm volatile( + "call %[bpf_get_cpu_cycles];" + "exit" + : + : __imm(bpf_get_cpu_cycles) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */ +__jited(" endbr64") +__jited(" nopl (%rax,%rax)") +__jited(" nopl (%rax)") +__jited(" pushq %rbp") +__jited(" movq %rsp, %rbp") +__jited(" endbr64") +/* save RDX in R11 as it will be overwritten */ +__jited(" movq %rdx, %r11") +/* lfence may not be executed depending on cpu features */ +__jited(" {{(lfence|)}}") +__jited(" rdtsc") +/* combine EDX:EAX into RAX */ +__jited(" shlq ${{(32|0x20)}}, %rdx") +__jited(" orq %rdx, %rax") +/* restore RDX from R11 */ +__jited(" movq %r11, %rdx") +__jited(" leave") +__naked int bpf_rdtsc_jit_x86_64(void) +{ + asm volatile( + "call %[bpf_get_cpu_cycles];" + "exit" + : + : __imm(bpf_get_cpu_cycles) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +__xlated("0: r1 = 42") +__xlated("1: call kernel-function") +__naked int bpf_cyc2ns(void) +{ + asm volatile( + "r1=0x2a;" + "call %[bpf_cpu_cycles_to_ns];" + "exit" + : + : __imm(bpf_cpu_cycles_to_ns) + : __clobber_all + ); +} + +SEC("syscall") +__arch_x86_64 +/* program entry for bpf_rdtsc_jit_x86_64(), regular function prologue */ +__jited(" endbr64") +__jited(" nopl (%rax,%rax)") +__jited(" nopl (%rax)") +__jited(" pushq %rbp") +__jited(" movq %rsp, %rbp") +__jited(" endbr64") +/* save RDX in R11 as it will be overwritten */ +__jited(" movabsq $0x2a2a2a2a2a, %rdi") +__jited(" imulq ${{.*}}, %rdi, %rax") +__jited(" shrq ${{.*}}, %rax") +__jited(" leave") +__naked int bpf_cyc2ns_jit_x86(void) +{ + asm volatile( + "r1=0x2a2a2a2a2a ll;" + "call %[bpf_cpu_cycles_to_ns];" + "exit" + : + : __imm(bpf_cpu_cycles_to_ns) + : __clobber_all + ); +} + +void rdtsc(void) +{ + bpf_get_cpu_cycles(); + bpf_cpu_cycles_to_ns(42); +} + +char _license[] SEC("license") = "GPL"; From patchwork Sat Nov 9 00:41:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vadim Fedorenko X-Patchwork-Id: 13869119 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92CE38F5C for ; Sat, 9 Nov 2024 00:42:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112964; cv=none; b=YMDRKJvu7AfKF62O/6dQgu4xefRVzPtZfW+zbSkXsP9QHYmfAggkvbFMMK5c7xCNG1zvYA1KEltWShuwp/jckfx2b2JZ/vIBnlL0q388Wj5m0Hcb5E354plcuVqPTNCziqWEPt+ASBAIqX004muRDsQZKs/gN4We6njjhO3pr10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731112964; c=relaxed/simple; bh=q2zm7XhpWXgczVLTp50flukPx5I2rnMRsgcIbA/JdIM=; h=From:To:CC:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=po7Bno5YknSSxn+yMYPy2FJOv1x623XwQ4IRm1dcbZQ8zUwTQHotTh5ikW3ZgQeRe5d1yJnBMUnu2iijIG5VrD6/v3fDkZSVfcPJY0vRjoRcFPkSZJBMc9o+J++Rqp9PR1IugoBirxdaPBjIxbeaSnPaS3WlX0PUdmUsqhCmhWI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=WBuHlVjF; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="WBuHlVjF" Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.18.1.2/8.18.1.2) with ESMTP id 4A8LFZ1d009131; Fri, 8 Nov 2024 16:42:17 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=B445bxnRd7VbM2XZKpHtwArCnEdBMewd+pcv5w8IaSQ=; b=WBuHlVjFI/n1 k56zIjmanC+2AoWO3CrUQDG8f0uYuwzAMm8DA8GQ0TRbg1mrt0xzL/EAvNCp19k7 3Z+OuK642TgY9cmopeZmAGKOOJN6cx9vYcUoUxhBQj8Bb4xC+h4OcYYb4a0tia8w 3wLWJV/0qzXv/Qun3pd/y6s8t4leQEo5sCh/oaGJOw9Ol22tQsXdcClvBW+hXS4I JKA7pN9eo+6wj+HJMXgCsKDgPxWv3jlUoqOaJykfSuxDTnehdPmcFiIAGqqS3NPl BtNCujd8rJhkeZZCtFumOGmSWe6HVAD4IGD4CYt0iFvfRKK6CV0gQIOxHuUCW14f 5v4nEsvBiw== Received: from mail.thefacebook.com ([163.114.134.16]) by m0089730.ppops.net (PPS) with ESMTPS id 42srnnsw3p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 08 Nov 2024 16:42:17 -0800 (PST) Received: from devvm4158.cln0.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c08b:78::2ac9) with Microsoft SMTP Server id 15.2.1544.11; Sat, 9 Nov 2024 00:42:13 +0000 From: Vadim Fedorenko To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Thomas Gleixner , Vadim Fedorenko , Mykola Lysenko , Jakub Kicinski CC: , , Vadim Fedorenko , Martin KaFai Lau Subject: [PATCH bpf-next v5 4/4] selftests/bpf: add usage example for cpu cycles kfuncs Date: Fri, 8 Nov 2024 16:41:57 -0800 Message-ID: <20241109004158.2259301-4-vadfed@meta.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241109004158.2259301-1-vadfed@meta.com> References: <20241109004158.2259301-1-vadfed@meta.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-GUID: FGo1yo3M2jPzlkAjGi_DRpiv2svRW0-L X-Proofpoint-ORIG-GUID: FGo1yo3M2jPzlkAjGi_DRpiv2svRW0-L X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-05_03,2024-10-04_01,2024-09-30_01 X-Patchwork-Delegate: bpf@iogearbox.net The selftest provides an example of how to measure the latency of bpf kfunc/helper call in cycles and nanoseconds. Signed-off-by: Vadim Fedorenko --- .../bpf/prog_tests/test_cpu_cycles.c | 35 +++++++++++++++++++ .../selftests/bpf/progs/test_cpu_cycles.c | 25 +++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c create mode 100644 tools/testing/selftests/bpf/progs/test_cpu_cycles.c diff --git a/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c b/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c new file mode 100644 index 000000000000..d7f3b66594b3 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_cpu_cycles.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Inc. */ + +#include +#include "test_cpu_cycles.skel.h" + +static void cpu_cycles(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct test_cpu_cycles *skel; + int err, pfd; + + skel = test_cpu_cycles__open_and_load(); + if (!ASSERT_OK_PTR(skel, "test_cpu_cycles open and load")) + return; + + pfd = bpf_program__fd(skel->progs.bpf_cpu_cycles); + if (!ASSERT_GT(pfd, 0, "test_cpu_cycles fd")) + goto fail; + + err = bpf_prog_test_run_opts(pfd, &opts); + if (!ASSERT_OK(err, "test_cpu_cycles test run")) + goto fail; + + ASSERT_NEQ(skel->bss->cycles, 0, "test_cpu_cycles 0 cycles"); + ASSERT_NEQ(skel->bss->ns, 0, "test_cpu_cycles 0 ns"); +fail: + test_cpu_cycles__destroy(skel); +} + +void test_cpu_cycles(void) +{ + if (test__start_subtest("cpu_cycles")) + cpu_cycles(); +} diff --git a/tools/testing/selftests/bpf/progs/test_cpu_cycles.c b/tools/testing/selftests/bpf/progs/test_cpu_cycles.c new file mode 100644 index 000000000000..48762f07ad7f --- /dev/null +++ b/tools/testing/selftests/bpf/progs/test_cpu_cycles.c @@ -0,0 +1,25 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Inc. */ + +#include "vmlinux.h" +#include + +extern u64 bpf_cpu_cycles_to_ns(u64 cycles) __weak __ksym; +extern u64 bpf_get_cpu_cycles(void) __weak __ksym; + +__u64 cycles, ns; + +SEC("syscall") +int bpf_cpu_cycles(void) +{ + struct bpf_pidns_info pidns; + __u64 start; + + start = bpf_get_cpu_cycles(); + bpf_get_ns_current_pid_tgid(0, 0, &pidns, sizeof(struct bpf_pidns_info)); + cycles = bpf_get_cpu_cycles() - start; + ns = bpf_cpu_cycles_to_ns(cycles); + return 0; +} + +char _license[] SEC("license") = "GPL";