From patchwork Thu Aug 22 08:41:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13773002 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-oa1-f41.google.com (mail-oa1-f41.google.com [209.85.160.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BAFE16D4EF for ; Thu, 22 Aug 2024 08:41:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316092; cv=none; b=NoSadHSzQ16JZYuwxqKTuTyvQc5xWFAuch3YW68GDQbhGcFTdZPqf0nXLfbsSD0ZfhLgW6MVoERhS8yuzRoZ4jS011TQe+5xZG77iYYd6EBfYlExjwR9uo+ylHPv+X5ZuSsxNDTuu7xp6uKgFaplLJrxsxxphSizfe7Ky5CKm8M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316092; c=relaxed/simple; bh=rS41QssJvEE6T6iwFs/Juv4ViM5z0cgFGuY7+BRm27I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ssE/CdFSR2+EoWxqNZJA0B5PsthkxVoTmH6f5vhAsFL3lEBSA1hOTawT4XXxPseFano0VxCbLmrHYJWTIFkjxgmbav2j1TSL+YyBpGE3PvHuM0WaGkRlGF+ouFIMjo+2p9/lz9hm8lSrLt2XrP0VWxAItpVPj2Wq6bybTmYR+8o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DhROKeS7; arc=none smtp.client-ip=209.85.160.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DhROKeS7" Received: by mail-oa1-f41.google.com with SMTP id 586e51a60fabf-27051f63018so331806fac.3 for ; Thu, 22 Aug 2024 01:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316089; x=1724920889; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tqyi4HV9gkoWxB0/SsFiubHnJXBJnZD+pNJohc7bIHk=; b=DhROKeS7SHGcoKK7qbjDhomyDlf1LpZLJEn84OFDuqeMLKD48kkdxvQGgustoIv1or iNaJpxmTgMuPNuog7gCsEcQX4qseaDPgY88YSvqto+T/Km6CGE7DS3er8eSKQVQbYs9c Z7JOzZgwTYdYMkoUQ5wImY3VLIhu+eg7xI5Jg5DSG23Q1Utm+NYXYnCaH9K3Z64Af9Q8 AX4KzxlZ/Oi8GTi4GiM7kHQThXUXzEqlOLp64lK7Wy5ApB2pEGxJ0pEFuaGgCGHWO0gE Rdq3Hm3kWCOvPhyjF39bk4+Yu4VB7I/yl6CLLhYm2Z8nJl+fX8u4w04JoosgunmW/Tuv dkYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316089; x=1724920889; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tqyi4HV9gkoWxB0/SsFiubHnJXBJnZD+pNJohc7bIHk=; b=LYvLN2BnMZEdhokjRuKhMz77+4GuzDl5G+9ug+46IGM8wuQtR1dsV03f+Ev82mGPvm 5ug3rzqECFrZXB50liLDKWydgm95EnpBlDORLMsOi2Kbo7Ckk8/KiOMu6lhI7OFNMHeF x1Hl8n3idkjxvTRAcY1kYEjQdOGU/SZw4T0tD6Sn5piqcnuaeMF0D9Nc2GpCI8Wl8C6X UtCBUDWtXmGvQ40dkDzmDortCLIVStkz8q6p9tCg16gkqGXzl4gODVxdBjxSywV0UjHQ HpKa6v/VHH47tEKMC+dbz3/zNmteMGHEg/xDiM3lha1i8vIkYp2SEVOj5MPWwnV7MrlH 6biw== X-Gm-Message-State: AOJu0YzFrsRG/ylEs9PcDpm2lCAKB/RGSViBTtxWpouzGAzh3pwRrp5L 6dKXP8dvK7dPKAU7VURi0hOuCzYgTbiFvQKyybZp0hiDCJHtNdnQYVhQMGyR X-Google-Smtp-Source: AGHT+IE3dZNSMQv6HTcAEvm8oQKUj84CzaS49duvxlg+bKdGwe4zXOC3szXHoET/Mqakc+G78ncDJA== X-Received: by 2002:a05:6871:7419:b0:260:ebf7:d0e7 with SMTP id 586e51a60fabf-273cfc09e56mr1546135fac.15.1724316089009; Thu, 22 Aug 2024 01:41:29 -0700 (PDT) Received: from honey-badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71434340449sm881692b3a.218.2024.08.22.01.41.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 01:41:28 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, Eduard Zingerman Subject: [PATCH bpf-next v3 1/6] bpf: rename nocsr -> bpf_fastcall in verifier Date: Thu, 22 Aug 2024 01:41:07 -0700 Message-ID: <20240822084112.3257995-2-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240822084112.3257995-1-eddyz87@gmail.com> References: <20240822084112.3257995-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Attribute used by LLVM implementation of the feature had been changed from no_caller_saved_registers to bpf_fastcall (see [1]). This commit replaces references to nocsr by references to bpf_fastcall to keep LLVM and Kernel parts in sync. [1] https://github.com/llvm/llvm-project/pull/105417 Acked-by: Yonghong Song Signed-off-by: Eduard Zingerman --- include/linux/bpf.h | 6 +- include/linux/bpf_verifier.h | 18 ++--- kernel/bpf/helpers.c | 2 +- kernel/bpf/verifier.c | 143 +++++++++++++++++------------------ 4 files changed, 84 insertions(+), 85 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f0192c173ed8..00dc4dd28cbd 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -808,12 +808,12 @@ struct bpf_func_proto { bool gpl_only; bool pkt_access; bool might_sleep; - /* set to true if helper follows contract for gcc/llvm - * attribute no_caller_saved_registers: + /* set to true if helper follows contract for llvm + * attribute bpf_fastcall: * - void functions do not scratch r0 * - functions taking N arguments scratch only registers r1-rN */ - bool allow_nocsr; + bool allow_fastcall; enum bpf_return_type ret_type; union { struct { diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 5cea15c81b8a..634a302a39e3 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -577,13 +577,13 @@ struct bpf_insn_aux_data { bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */ u8 alu_state; /* used in combination with alu_limit */ /* true if STX or LDX instruction is a part of a spill/fill - * pattern for a no_caller_saved_registers call. + * pattern for a bpf_fastcall call. */ - u8 nocsr_pattern:1; + u8 fastcall_pattern:1; /* for CALL instructions, a number of spill/fill pairs in the - * no_caller_saved_registers pattern. + * bpf_fastcall pattern. */ - u8 nocsr_spills_num:3; + u8 fastcall_spills_num:3; /* below fields are initialized once */ unsigned int orig_idx; /* original instruction index */ @@ -653,10 +653,10 @@ struct bpf_subprog_info { u32 linfo_idx; /* The idx to the main_prog->aux->linfo */ u16 stack_depth; /* max. stack depth used by this function */ u16 stack_extra; - /* offsets in range [stack_depth .. nocsr_stack_off) - * are used for no_caller_saved_registers spills and fills. + /* offsets in range [stack_depth .. fastcall_stack_off) + * are used for bpf_fastcall spills and fills. */ - s16 nocsr_stack_off; + s16 fastcall_stack_off; bool has_tail_call: 1; bool tail_call_reachable: 1; bool has_ld_abs: 1; @@ -664,8 +664,8 @@ struct bpf_subprog_info { bool is_async_cb: 1; bool is_exception_cb: 1; bool args_cached: 1; - /* true if nocsr stack region is used by functions that can't be inlined */ - bool keep_nocsr_stack: 1; + /* true if bpf_fastcall stack region is used by functions that can't be inlined */ + bool keep_fastcall_stack: 1; u8 arg_cnt; struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS]; diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 12e3aa40b180..369f3d2d1a1d 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -158,7 +158,7 @@ const struct bpf_func_proto bpf_get_smp_processor_id_proto = { .func = bpf_get_smp_processor_id, .gpl_only = false, .ret_type = RET_INTEGER, - .allow_nocsr = true, + .allow_fastcall = true, }; BPF_CALL_0(bpf_get_numa_node_id) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 920d7c5fe944..0dfd91f36417 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4579,28 +4579,28 @@ static int get_reg_width(struct bpf_reg_state *reg) return fls64(reg->umax_value); } -/* See comment for mark_nocsr_pattern_for_call() */ -static void check_nocsr_stack_contract(struct bpf_verifier_env *env, struct bpf_func_state *state, - int insn_idx, int off) +/* See comment for mark_fastcall_pattern_for_call() */ +static void check_fastcall_stack_contract(struct bpf_verifier_env *env, + struct bpf_func_state *state, int insn_idx, int off) { struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno]; struct bpf_insn_aux_data *aux = env->insn_aux_data; int i; - if (subprog->nocsr_stack_off <= off || aux[insn_idx].nocsr_pattern) + if (subprog->fastcall_stack_off <= off || aux[insn_idx].fastcall_pattern) return; - /* access to the region [max_stack_depth .. nocsr_stack_off) - * from something that is not a part of the nocsr pattern, - * disable nocsr rewrites for current subprogram by setting - * nocsr_stack_off to a value smaller than any possible offset. + /* access to the region [max_stack_depth .. fastcall_stack_off) + * from something that is not a part of the fastcall pattern, + * disable fastcall rewrites for current subprogram by setting + * fastcall_stack_off to a value smaller than any possible offset. */ - subprog->nocsr_stack_off = S16_MIN; - /* reset nocsr aux flags within subprogram, + subprog->fastcall_stack_off = S16_MIN; + /* reset fastcall aux flags within subprogram, * happens at most once per subprogram */ for (i = subprog->start; i < (subprog + 1)->start; ++i) { - aux[i].nocsr_spills_num = 0; - aux[i].nocsr_pattern = 0; + aux[i].fastcall_spills_num = 0; + aux[i].fastcall_pattern = 0; } } @@ -4652,7 +4652,7 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, if (err) return err; - check_nocsr_stack_contract(env, state, insn_idx, off); + check_fastcall_stack_contract(env, state, insn_idx, off); mark_stack_slot_scratched(env, spi); if (reg && !(off % BPF_REG_SIZE) && reg->type == SCALAR_VALUE && env->bpf_capable) { bool reg_value_fits; @@ -4787,7 +4787,7 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, return err; } - check_nocsr_stack_contract(env, state, insn_idx, min_off); + check_fastcall_stack_contract(env, state, insn_idx, min_off); /* Variable offset writes destroy any spilled pointers in range. */ for (i = min_off; i < max_off; i++) { u8 new_type, *stype; @@ -4926,7 +4926,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, reg = ®_state->stack[spi].spilled_ptr; mark_stack_slot_scratched(env, spi); - check_nocsr_stack_contract(env, state, env->insn_idx, off); + check_fastcall_stack_contract(env, state, env->insn_idx, off); if (is_spilled_reg(®_state->stack[spi])) { u8 spill_size = 1; @@ -5087,7 +5087,7 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, min_off = reg->smin_value + off; max_off = reg->smax_value + off; mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno); - check_nocsr_stack_contract(env, ptr_state, env->insn_idx, min_off); + check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off); return 0; } @@ -6804,13 +6804,13 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env, struct bpf_insn_aux_data *aux = &env->insn_aux_data[env->insn_idx]; int min_valid_off, max_bpf_stack; - /* If accessing instruction is a spill/fill from nocsr pattern, + /* If accessing instruction is a spill/fill from bpf_fastcall pattern, * add room for all caller saved registers below MAX_BPF_STACK. - * In case if nocsr rewrite won't happen maximal stack depth + * In case if bpf_fastcall rewrite won't happen maximal stack depth * would be checked by check_max_stack_depth_subprog(). */ max_bpf_stack = MAX_BPF_STACK; - if (aux->nocsr_pattern) + if (aux->fastcall_pattern) max_bpf_stack += CALLER_SAVED_REGS * BPF_REG_SIZE; if (t == BPF_WRITE || env->allow_uninit_stack) @@ -16118,12 +16118,12 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns, /* Return a bitmask specifying which caller saved registers are * clobbered by a call to a helper *as if* this helper follows - * no_caller_saved_registers contract: + * bpf_fastcall contract: * - includes R0 if function is non-void; * - includes R1-R5 if corresponding parameter has is described * in the function prototype. */ -static u32 helper_nocsr_clobber_mask(const struct bpf_func_proto *fn) +static u32 helper_fastcall_clobber_mask(const struct bpf_func_proto *fn) { u8 mask; int i; @@ -16138,8 +16138,8 @@ static u32 helper_nocsr_clobber_mask(const struct bpf_func_proto *fn) } /* True if do_misc_fixups() replaces calls to helper number 'imm', - * replacement patch is presumed to follow no_caller_saved_registers contract - * (see mark_nocsr_pattern_for_call() below). + * replacement patch is presumed to follow bpf_fastcall contract + * (see mark_fastcall_pattern_for_call() below). */ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) { @@ -16153,7 +16153,7 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } } -/* GCC and LLVM define a no_caller_saved_registers function attribute. +/* LLVM define a bpf_fastcall function attribute. * This attribute means that function scratches only some of * the caller saved registers defined by ABI. * For BPF the set of such registers could be defined as follows: @@ -16163,13 +16163,12 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) * * The contract between kernel and clang allows to simultaneously use * such functions and maintain backwards compatibility with old - * kernels that don't understand no_caller_saved_registers calls - * (nocsr for short): + * kernels that don't understand bpf_fastcall calls: * - * - for nocsr calls clang allocates registers as-if relevant r0-r5 + * - for bpf_fastcall calls clang allocates registers as-if relevant r0-r5 * registers are not scratched by the call; * - * - as a post-processing step, clang visits each nocsr call and adds + * - as a post-processing step, clang visits each bpf_fastcall call and adds * spill/fill for every live r0-r5; * * - stack offsets used for the spill/fill are allocated as lowest @@ -16177,11 +16176,11 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) * purposes; * * - when kernel loads a program, it looks for such patterns - * (nocsr function surrounded by spills/fills) and checks if - * spill/fill stack offsets are used exclusively in nocsr patterns; + * (bpf_fastcall function surrounded by spills/fills) and checks if + * spill/fill stack offsets are used exclusively in fastcall patterns; * * - if so, and if verifier or current JIT inlines the call to the - * nocsr function (e.g. a helper call), kernel removes unnecessary + * bpf_fastcall function (e.g. a helper call), kernel removes unnecessary * spill/fill pairs; * * - when old kernel loads a program, presence of spill/fill pairs @@ -16200,22 +16199,22 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) * r0 += r2; * exit; * - * The purpose of mark_nocsr_pattern_for_call is to: + * The purpose of mark_fastcall_pattern_for_call is to: * - look for such patterns; - * - mark spill and fill instructions in env->insn_aux_data[*].nocsr_pattern; - * - mark set env->insn_aux_data[*].nocsr_spills_num for call instruction; - * - update env->subprog_info[*]->nocsr_stack_off to find an offset - * at which nocsr spill/fill stack slots start; - * - update env->subprog_info[*]->keep_nocsr_stack. + * - mark spill and fill instructions in env->insn_aux_data[*].fastcall_pattern; + * - mark set env->insn_aux_data[*].fastcall_spills_num for call instruction; + * - update env->subprog_info[*]->fastcall_stack_off to find an offset + * at which bpf_fastcall spill/fill stack slots start; + * - update env->subprog_info[*]->keep_fastcall_stack. * - * The .nocsr_pattern and .nocsr_stack_off are used by - * check_nocsr_stack_contract() to check if every stack access to - * nocsr spill/fill stack slot originates from spill/fill - * instructions, members of nocsr patterns. + * The .fastcall_pattern and .fastcall_stack_off are used by + * check_fastcall_stack_contract() to check if every stack access to + * fastcall spill/fill stack slot originates from spill/fill + * instructions, members of fastcall patterns. * - * If such condition holds true for a subprogram, nocsr patterns could - * be rewritten by remove_nocsr_spills_fills(). - * Otherwise nocsr patterns are not changed in the subprogram + * If such condition holds true for a subprogram, fastcall patterns could + * be rewritten by remove_fastcall_spills_fills(). + * Otherwise bpf_fastcall patterns are not changed in the subprogram * (code, presumably, generated by an older clang version). * * For example, it is *not* safe to remove spill/fill below: @@ -16228,9 +16227,9 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) * r0 += r1; exit; * exit; */ -static void mark_nocsr_pattern_for_call(struct bpf_verifier_env *env, - struct bpf_subprog_info *subprog, - int insn_idx, s16 lowest_off) +static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env, + struct bpf_subprog_info *subprog, + int insn_idx, s16 lowest_off) { struct bpf_insn *insns = env->prog->insnsi, *stx, *ldx; struct bpf_insn *call = &env->prog->insnsi[insn_idx]; @@ -16245,8 +16244,8 @@ static void mark_nocsr_pattern_for_call(struct bpf_verifier_env *env, if (get_helper_proto(env, call->imm, &fn) < 0) /* error would be reported later */ return; - clobbered_regs_mask = helper_nocsr_clobber_mask(fn); - can_be_inlined = fn->allow_nocsr && + clobbered_regs_mask = helper_fastcall_clobber_mask(fn); + can_be_inlined = fn->allow_fastcall && (verifier_inlines_helper_call(env, call->imm) || bpf_jit_inlines_helper_call(call->imm)); } @@ -16289,36 +16288,36 @@ static void mark_nocsr_pattern_for_call(struct bpf_verifier_env *env, if (stx->off != off || ldx->off != off) break; expected_regs_mask &= ~BIT(stx->src_reg); - env->insn_aux_data[insn_idx - i].nocsr_pattern = 1; - env->insn_aux_data[insn_idx + i].nocsr_pattern = 1; + env->insn_aux_data[insn_idx - i].fastcall_pattern = 1; + env->insn_aux_data[insn_idx + i].fastcall_pattern = 1; } if (i == 1) return; - /* Conditionally set 'nocsr_spills_num' to allow forward + /* Conditionally set 'fastcall_spills_num' to allow forward * compatibility when more helper functions are marked as - * nocsr at compile time than current kernel supports, e.g: + * bpf_fastcall at compile time than current kernel supports, e.g: * * 1: *(u64 *)(r10 - 8) = r1 - * 2: call A ;; assume A is nocsr for current kernel + * 2: call A ;; assume A is bpf_fastcall for current kernel * 3: r1 = *(u64 *)(r10 - 8) * 4: *(u64 *)(r10 - 8) = r1 - * 5: call B ;; assume B is not nocsr for current kernel + * 5: call B ;; assume B is not bpf_fastcall for current kernel * 6: r1 = *(u64 *)(r10 - 8) * - * There is no need to block nocsr rewrite for such program. - * Set 'nocsr_pattern' for both calls to keep check_nocsr_stack_contract() happy, - * don't set 'nocsr_spills_num' for call B so that remove_nocsr_spills_fills() + * There is no need to block bpf_fastcall rewrite for such program. + * Set 'fastcall_pattern' for both calls to keep check_fastcall_stack_contract() happy, + * don't set 'fastcall_spills_num' for call B so that remove_fastcall_spills_fills() * does not remove spill/fill pair {4,6}. */ if (can_be_inlined) - env->insn_aux_data[insn_idx].nocsr_spills_num = i - 1; + env->insn_aux_data[insn_idx].fastcall_spills_num = i - 1; else - subprog->keep_nocsr_stack = 1; - subprog->nocsr_stack_off = min(subprog->nocsr_stack_off, off); + subprog->keep_fastcall_stack = 1; + subprog->fastcall_stack_off = min(subprog->fastcall_stack_off, off); } -static int mark_nocsr_patterns(struct bpf_verifier_env *env) +static int mark_fastcall_patterns(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprog = env->subprog_info; struct bpf_insn *insn; @@ -16335,12 +16334,12 @@ static int mark_nocsr_patterns(struct bpf_verifier_env *env) continue; lowest_off = min(lowest_off, insn->off); } - /* use this offset to find nocsr patterns */ + /* use this offset to find fastcall patterns */ for (i = subprog->start; i < (subprog + 1)->start; ++i) { insn = env->prog->insnsi + i; if (insn->code != (BPF_JMP | BPF_CALL)) continue; - mark_nocsr_pattern_for_call(env, subprog, i, lowest_off); + mark_fastcall_pattern_for_call(env, subprog, i, lowest_off); } } return 0; @@ -21244,10 +21243,10 @@ static int optimize_bpf_loop(struct bpf_verifier_env *env) return 0; } -/* Remove unnecessary spill/fill pairs, members of nocsr pattern, +/* Remove unnecessary spill/fill pairs, members of fastcall pattern, * adjust subprograms stack depth when possible. */ -static int remove_nocsr_spills_fills(struct bpf_verifier_env *env) +static int remove_fastcall_spills_fills(struct bpf_verifier_env *env) { struct bpf_subprog_info *subprog = env->subprog_info; struct bpf_insn_aux_data *aux = env->insn_aux_data; @@ -21258,8 +21257,8 @@ static int remove_nocsr_spills_fills(struct bpf_verifier_env *env) int i, j; for (i = 0; i < insn_cnt; i++, insn++) { - if (aux[i].nocsr_spills_num > 0) { - spills_num = aux[i].nocsr_spills_num; + if (aux[i].fastcall_spills_num > 0) { + spills_num = aux[i].fastcall_spills_num; /* NOPs would be removed by opt_remove_nops() */ for (j = 1; j <= spills_num; ++j) { *(insn - j) = NOP; @@ -21268,8 +21267,8 @@ static int remove_nocsr_spills_fills(struct bpf_verifier_env *env) modified = true; } if ((subprog + 1)->start == i + 1) { - if (modified && !subprog->keep_nocsr_stack) - subprog->stack_depth = -subprog->nocsr_stack_off; + if (modified && !subprog->keep_fastcall_stack) + subprog->stack_depth = -subprog->fastcall_stack_off; subprog++; modified = false; } @@ -22192,7 +22191,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 if (ret < 0) goto skip_full_check; - ret = mark_nocsr_patterns(env); + ret = mark_fastcall_patterns(env); if (ret < 0) goto skip_full_check; @@ -22209,7 +22208,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 * allocate additional slots. */ if (ret == 0) - ret = remove_nocsr_spills_fills(env); + ret = remove_fastcall_spills_fills(env); if (ret == 0) ret = check_max_stack_depth(env); From patchwork Thu Aug 22 08:41:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13773003 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24D8316F8E9 for ; Thu, 22 Aug 2024 08:41:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316092; cv=none; b=V8SDNw5Wo9TBV71KyW+33GYts7BfQCfpI0Jk/tUrvpMYwhVLSD10cVf7xX7QSTNefNLpXn8lLdMwKKmvixH23sR2Ymv+hQlvMx+UQMqaVzuyYPrbx+dQawhb/XniludzoHXq5C2dcnvWXwmvkjtfxiZaK5udlfvJO+wEaw6eF9E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316092; c=relaxed/simple; bh=hnd3Do08Llt25bMUJHeSN3tOab2iTxSrJbIUXCWGwFQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iW3JC7TZDsb/KGjWswU+3IsyH5LLa7rYzlFtplnJzOxVKnKX3b4xr0HjqHufgmHUooolb0xvvwCJ08Mw85+bZpMz388p8w0jWD90f6d34tS4S/FXnoiRmFlLac6cfCHLUq9Q8dizJ5l+C7MwcNOpGaJYlUa7BvL+bpgoRtv4OtM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gwDPO/qx; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gwDPO/qx" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-71423704ef3so466063b3a.3 for ; Thu, 22 Aug 2024 01:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316090; x=1724920890; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o8mdBEinJF/YGnCUK6Zz6XgYn6ajCQ7Sr3+ce1p+VbY=; b=gwDPO/qx5jwxkJWbFnZ6try4PG/7iFg7SAbVnG9UkjgI8k0A4KgzfTfQ6IsKUKCoep Uu6TI9IhXuLsbZv7bYpMBL/SDx1r9YIeUdpEnHS2YPfGTd+rd9QmalCGuXVJfR8TXJa/ fQSArUWgTgarZFeQBuPXl9kiF+IbBdW4+4h0Qx46PPuE+dSH5RrJ71BwU0a2v9jLjEvG y9C4De8SHkqaT+YRJUjHWSh1uMBjJa13QzBYOZO9huOdQS2koulLzvTOfpjKcF4DNIc4 1ndah0Nnj7oKc2Ca1LAUPt1/d86niaLm9XTRVx+WJBuomknkMTJnQX+9sehav1YEihaT 0Bww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316090; x=1724920890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o8mdBEinJF/YGnCUK6Zz6XgYn6ajCQ7Sr3+ce1p+VbY=; b=VrRmbvqKLAtPOiuhv8t1pFTZQ69bKG2SOEvoYRv2MYN2riasix0vZs7BSIsJHPj+l7 BZ2Z71PvE2otWKfjjwQn9wS1/75uviCK4og3HrcM+bC3Qf80suDzetLjZhxKNr11JkwM PEZzajhVxgfD48Vyi5/+wyxgub0nyrLBI7d92f1sKndCsNYND1g5Qtr0dQLr2Xe1tBcF cyjJWs3GENNLXP+mudrAYWY4J0Px2Dj48/HlEhwlQXUyuX5HvtbcgXkeXr7iiFFAof4+ TdFq+D/YXN05itl4U7TiqI0VRMrcDxUIX2P+U8t8do2AtDeHxeAWn0WA2v4N9QtI7C7Z s7Ww== X-Gm-Message-State: AOJu0YyTC5jt0eNXaDB8aFJ2Vv82k6vsnYNuYTxjbh74qT5W+/qh2VqT l7e4ckzyq14R2PC/ud542GSLfDxTbjx9HfgmNgueXVN3s15W6WEtOboL2ffP X-Google-Smtp-Source: AGHT+IGm39PM8GLIKEO/h/JnccOr+zK7FjhYVktjHCVJmgmyJSV+s+jvF7PTzErlmbGQm1jYej+v2g== X-Received: by 2002:a05:6a21:478a:b0:1c4:ba7c:741c with SMTP id adf61e73a8af0-1caeb1d90e2mr1363921637.21.1724316090020; Thu, 22 Aug 2024 01:41:30 -0700 (PDT) Received: from honey-badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71434340449sm881692b3a.218.2024.08.22.01.41.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 01:41:29 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, Eduard Zingerman Subject: [PATCH bpf-next v3 2/6] selftests/bpf: rename nocsr -> bpf_fastcall in selftests Date: Thu, 22 Aug 2024 01:41:08 -0700 Message-ID: <20240822084112.3257995-3-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240822084112.3257995-1-eddyz87@gmail.com> References: <20240822084112.3257995-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Attribute used by LLVM implementation of the feature had been changed from no_caller_saved_registers to bpf_fastcall (see [1]). This commit replaces references to nocsr by references to bpf_fastcall to keep LLVM and selftests parts in sync. [1] https://github.com/llvm/llvm-project/pull/105417 Acked-by: Yonghong Song Signed-off-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/verifier.c | 4 +-- ...rifier_nocsr.c => verifier_bpf_fastcall.c} | 26 +++++++++---------- 2 files changed, 15 insertions(+), 15 deletions(-) rename tools/testing/selftests/bpf/progs/{verifier_nocsr.c => verifier_bpf_fastcall.c} (95%) diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index cf3662dbd24f..80a90c627182 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -53,7 +53,7 @@ #include "verifier_movsx.skel.h" #include "verifier_netfilter_ctx.skel.h" #include "verifier_netfilter_retcode.skel.h" -#include "verifier_nocsr.skel.h" +#include "verifier_bpf_fastcall.skel.h" #include "verifier_or_jmp32_k.skel.h" #include "verifier_precision.skel.h" #include "verifier_prevent_map_lookup.skel.h" @@ -177,7 +177,7 @@ void test_verifier_meta_access(void) { RUN(verifier_meta_access); } void test_verifier_movsx(void) { RUN(verifier_movsx); } void test_verifier_netfilter_ctx(void) { RUN(verifier_netfilter_ctx); } void test_verifier_netfilter_retcode(void) { RUN(verifier_netfilter_retcode); } -void test_verifier_nocsr(void) { RUN(verifier_nocsr); } +void test_verifier_bpf_fastcall(void) { RUN(verifier_bpf_fastcall); } void test_verifier_or_jmp32_k(void) { RUN(verifier_or_jmp32_k); } void test_verifier_precision(void) { RUN(verifier_precision); } void test_verifier_prevent_map_lookup(void) { RUN(verifier_prevent_map_lookup); } diff --git a/tools/testing/selftests/bpf/progs/verifier_nocsr.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c similarity index 95% rename from tools/testing/selftests/bpf/progs/verifier_nocsr.c rename to tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c index 666c736d196f..e30ab9fe5096 100644 --- a/tools/testing/selftests/bpf/progs/verifier_nocsr.c +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c @@ -39,7 +39,7 @@ __naked void simple(void) : __clobber_all); } -/* The logic for detecting and verifying nocsr pattern is the same for +/* The logic for detecting and verifying bpf_fastcall pattern is the same for * any arch, however x86 differs from arm64 or riscv64 in a way * bpf_get_smp_processor_id is rewritten: * - on x86 it is done by verifier @@ -52,7 +52,7 @@ __naked void simple(void) * * It is really desirable to check instruction indexes in the xlated * patterns, so add this canary test to check that function rewrite by - * jit is correctly processed by nocsr logic, keep the rest of the + * jit is correctly processed by bpf_fastcall logic, keep the rest of the * tests as x86. */ SEC("raw_tp") @@ -463,7 +463,7 @@ __naked static void bad_write_in_subprog_aux(void) { asm volatile ( "r0 = 1;" - "*(u64 *)(r1 - 0) = r0;" /* invalidates nocsr contract for caller: */ + "*(u64 *)(r1 - 0) = r0;" /* invalidates bpf_fastcall contract for caller: */ "exit;" /* caller stack at -8 used outside of the pattern */ ::: __clobber_all); } @@ -480,7 +480,7 @@ __naked void bad_helper_write(void) { asm volatile ( "r1 = 1;" - /* nocsr pattern with stack offset -8 */ + /* bpf_fastcall pattern with stack offset -8 */ "*(u64 *)(r10 - 8) = r1;" "call %[bpf_get_smp_processor_id];" "r1 = *(u64 *)(r10 - 8);" @@ -488,7 +488,7 @@ __naked void bad_helper_write(void) "r1 += -8;" "r2 = 1;" "r3 = 42;" - /* read dst is fp[-8], thus nocsr rewrite not applied */ + /* read dst is fp[-8], thus bpf_fastcall rewrite not applied */ "call %[bpf_probe_read_kernel];" "exit;" : @@ -598,7 +598,7 @@ __arch_x86_64 __log_level(4) __msg("stack depth 8") __xlated("2: r0 = &(void __percpu *)(r0)") __success -__naked void helper_call_does_not_prevent_nocsr(void) +__naked void helper_call_does_not_prevent_bpf_fastcall(void) { asm volatile ( "r1 = 1;" @@ -689,7 +689,7 @@ __naked int bpf_loop_interaction1(void) { asm volatile ( "r1 = 1;" - /* nocsr stack region at -16, but could be removed */ + /* bpf_fastcall stack region at -16, but could be removed */ "*(u64 *)(r10 - 16) = r1;" "call %[bpf_get_smp_processor_id];" "r1 = *(u64 *)(r10 - 16);" @@ -729,7 +729,7 @@ __naked int bpf_loop_interaction2(void) { asm volatile ( "r1 = 42;" - /* nocsr stack region at -16, cannot be removed */ + /* bpf_fastcall stack region at -16, cannot be removed */ "*(u64 *)(r10 - 16) = r1;" "call %[bpf_get_smp_processor_id];" "r1 = *(u64 *)(r10 - 16);" @@ -759,8 +759,8 @@ __msg("stack depth 512+0") __xlated("r0 = &(void __percpu *)(r0)") __success /* cumulative_stack_depth() stack usage is MAX_BPF_STACK, - * called subprogram uses an additional slot for nocsr spill/fill, - * since nocsr spill/fill could be removed the program still fits + * called subprogram uses an additional slot for bpf_fastcall spill/fill, + * since bpf_fastcall spill/fill could be removed the program still fits * in MAX_BPF_STACK and should be accepted. */ __naked int cumulative_stack_depth(void) @@ -798,7 +798,7 @@ __xlated("3: r0 = &(void __percpu *)(r0)") __xlated("4: r0 = *(u32 *)(r0 +0)") __xlated("5: exit") __success -__naked int nocsr_max_stack_ok(void) +__naked int bpf_fastcall_max_stack_ok(void) { asm volatile( "r1 = 42;" @@ -820,7 +820,7 @@ __arch_x86_64 __log_level(4) __msg("stack depth 520") __failure -__naked int nocsr_max_stack_fail(void) +__naked int bpf_fastcall_max_stack_fail(void) { asm volatile( "r1 = 42;" @@ -828,7 +828,7 @@ __naked int nocsr_max_stack_fail(void) "*(u64 *)(r10 - %[max_bpf_stack_8]) = r1;" "call %[bpf_get_smp_processor_id];" "r1 = *(u64 *)(r10 - %[max_bpf_stack_8]);" - /* call to prandom blocks nocsr rewrite */ + /* call to prandom blocks bpf_fastcall rewrite */ "*(u64 *)(r10 - %[max_bpf_stack_8]) = r1;" "call %[bpf_get_prandom_u32];" "r1 = *(u64 *)(r10 - %[max_bpf_stack_8]);" From patchwork Thu Aug 22 08:41:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13773004 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B17F170A03 for ; Thu, 22 Aug 2024 08:41:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316093; cv=none; b=HLqoXYRBBSwL60FDhEBqSVdWbRb/eSOhz2uaRHFr5a5TFleZTpebM3mBveyf5NQa+P2hMiRIgAiLAUHpANonxXjxG4iXjAX6KLRNyCopEkvVESqlHfmSk02ku9Z/LCIdkD9/Wik5sRVlR4q6xbJsHmOcfsYKeqrcBpBeRTfESPs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316093; c=relaxed/simple; bh=1jNeTHyjYKd15gGzLyViANnTwl0v1Qyp38JPFULuTZU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=otnKnqi2gN69KKhGIUXEARCw3E5aBhAgSBx4TM5XYslpUDRC+RyFOX0esZMt+uV2Hp+bD0OcoDTfiAKJMjib97YS9i1GkQ9ldAfVNZy4OHAkSQpG9gJW5rEjfhCBixFNGEmDjWWqU77Fz7j3U0MTpuZLIvaoaRKEtqyu4YPupmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lYCbRjBu; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lYCbRjBu" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3de13126957so352489b6e.0 for ; Thu, 22 Aug 2024 01:41:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316091; x=1724920891; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RicDvKkSIbKYxoMdlAjvong9R0egR80C6DFshD8PuOY=; b=lYCbRjBup6YoBOt/SkFJQ0oCFEtpOERDd8iReYSCUGB3I4Y4+rwEPwdP8/iWLEoMju SjKKppa7lnm7URU4xjjAZpvAAESValV+0kZKKfEECM7UWSRD3eo6aoaDCGTRapLoehcM TJ0rHEuGxRePmVO99eX3vrhXCBj/ljilHpq0EQAHyTvWDU3NxmSMJ+pJm0RpPMOPfOgz m+NWdIe7fbJ4yOnrdsyjHCkilrBQ5TZaoZ60LK0EztdFRLlRGvaI80nr6wPvfzQqxKgC DZAK/GjzP1mOOyJEX8aKhhrGEnwXRX3qOc9g06ekTuBBqREXJQJ7ASjvX5MJHqZ4CIot ZYiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316091; x=1724920891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RicDvKkSIbKYxoMdlAjvong9R0egR80C6DFshD8PuOY=; b=EIlrIBdnxOPTYpjgyfBj4SuK0xcKKuoMLHX0ALPjRDRPx2QzneVeLcT5B6HnK1Pxu9 SU2CoP4n7Up9vGL1neQacPeWqCXQTHzXyrXa3fR0Tr0e9easNMefypjtWDUhGUcWQaAo xc7iEOFu+4ZDE4dgV/0wTDb0OAw43RObrVOgafdpQZ7ol+ihJwuzvcpW/gDI+POXtDHx LJrfUeh4FnA1Tw8sd5+Qt8KMflJFkyL2ALc3ny8IcG5BxAFRIL6EEyXU4EFK3pGiffN2 32V+wKTOKpGljgPMhbeOEhp3lXHv8Bcy5WiPSwzOVBkyAL0t0zdxS4k3IuXBtmfX24M/ sDnA== X-Gm-Message-State: AOJu0YzphyuBUWlIkQP6EQThbG4QP0CZF3tXd22wv62ob/SByj8ikXsk 9jzxsIPHgu5SJIlkHuHHyVbaMv4Wc6pCGL534O38a9je+A5zixXiWzOnpj3Q X-Google-Smtp-Source: AGHT+IElSW7pgCCXFALKJzHw1Wh6hsE4ox8qhpy4Thw+gPkUJpfYlGjMzuiTIec9heDqgOl9TjeAmw== X-Received: by 2002:a05:6808:3a10:b0:3da:e2b1:1100 with SMTP id 5614622812f47-3de19d16775mr6014955b6e.32.1724316091050; Thu, 22 Aug 2024 01:41:31 -0700 (PDT) Received: from honey-badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71434340449sm881692b3a.218.2024.08.22.01.41.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 01:41:30 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, Eduard Zingerman Subject: [PATCH bpf-next v3 3/6] bpf: support bpf_fastcall patterns for kfuncs Date: Thu, 22 Aug 2024 01:41:09 -0700 Message-ID: <20240822084112.3257995-4-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240822084112.3257995-1-eddyz87@gmail.com> References: <20240822084112.3257995-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Recognize bpf_fastcall patterns around kfunc calls. For example, suppose bpf_cast_to_kern_ctx() follows bpf_fastcall contract (which it does), in such a case allow verifier to rewrite BPF program below: r2 = 1; *(u64 *)(r10 - 32) = r2; call %[bpf_cast_to_kern_ctx]; r2 = *(u64 *)(r10 - 32); r0 = r2; By removing the spill/fill pair: r2 = 1; call %[bpf_cast_to_kern_ctx]; r0 = r2; Acked-by: Yonghong Song Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0dfd91f36417..94308cc7c503 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16125,7 +16125,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns, */ static u32 helper_fastcall_clobber_mask(const struct bpf_func_proto *fn) { - u8 mask; + u32 mask; int i; mask = 0; @@ -16153,6 +16153,26 @@ static bool verifier_inlines_helper_call(struct bpf_verifier_env *env, s32 imm) } } +/* Same as helper_fastcall_clobber_mask() but for kfuncs, see comment above */ +static u32 kfunc_fastcall_clobber_mask(struct bpf_kfunc_call_arg_meta *meta) +{ + u32 vlen, i, mask; + + vlen = btf_type_vlen(meta->func_proto); + mask = 0; + if (!btf_type_is_void(btf_type_by_id(meta->btf, meta->func_proto->type))) + mask |= BIT(BPF_REG_0); + for (i = 0; i < vlen; ++i) + mask |= BIT(BPF_REG_1 + i); + return mask; +} + +/* Same as verifier_inlines_helper_call() but for kfuncs, see comment above */ +static bool is_fastcall_kfunc_call(struct bpf_kfunc_call_arg_meta *meta) +{ + return false; +} + /* LLVM define a bpf_fastcall function attribute. * This attribute means that function scratches only some of * the caller saved registers defined by ABI. @@ -16250,6 +16270,19 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env, bpf_jit_inlines_helper_call(call->imm)); } + if (bpf_pseudo_kfunc_call(call)) { + struct bpf_kfunc_call_arg_meta meta; + int err; + + err = fetch_kfunc_meta(env, call, &meta, NULL); + if (err < 0) + /* error would be reported later */ + return; + + clobbered_regs_mask = kfunc_fastcall_clobber_mask(&meta); + can_be_inlined = is_fastcall_kfunc_call(&meta); + } + if (clobbered_regs_mask == ALL_CALLER_SAVED_REGS) return; From patchwork Thu Aug 22 08:41:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13773005 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E90217C224 for ; Thu, 22 Aug 2024 08:41:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316094; cv=none; b=CS8ggRZvnwtOmMmPzEhBiDHeZ9ixibUNwnjV+OuYK/ekJe1YMbtyBYuW0L7Iy+CpgzVDouyYqxkrziQofkY7vj8aZVf2GlBCXl/f5SVD0347L9l8LKqyNtmQtHsHxhvEFhaFtsQq7kWKJ0Oq1FOqUq22dHshS/ddS9LXEZEezHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316094; c=relaxed/simple; bh=h8BV47jBODJvK8yR8ErXbphdg6EPxoTY6p1WKqK5ISY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DOYP+9JnlNsn9k6gdt23toatke3t/qOogAEu0MI4sBszE8W+l8cr03Rvd2K4uUz3t97m2J90gsR9Kv25HpXezjiNtZj/O5NXteY/WIqNTK36pfmahKQgz9V+ZH0ylN6IKdZKGWBeZh8ewfsrHERmGuktMDyKrLikROAH3l30WK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YGg4AdeS; arc=none smtp.client-ip=209.85.210.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YGg4AdeS" Received: by mail-pf1-f171.google.com with SMTP id d2e1a72fcca58-7142014d8dfso449868b3a.3 for ; Thu, 22 Aug 2024 01:41:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316092; x=1724920892; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nsa0n2Jb5f9A4YGXDvRlV32yLFSY00XhLTjGYihf/fo=; b=YGg4AdeSKtopAvlmaIPAhxDED301O1c+kYR9N/9JKSlKM2L7JGUCnZislepFjmrv0x fuQNNzH3WjhKt0XLYprrpNMy9+nkXUc/0baG9xiMttwS0xHpyRvnq6xGEKtro+JVIPt4 z+z2U2PRtwsom7d18RQPog+V+mQUOtUSKjf+i5I55IOtMDMh5Yd1A4ogg5hWvrgR5fCo /WUyHmhf7h8JGp6uPCLy9sNRqQPphXtEoNvp8UMR7iNjYw3rYrjljM+wXJZYMKVyB6lz y9h5ZaZppra+QPEC/qM3GbSCeEZod65beN7QP8WES4YQza1TXUW8+Ujna8HB/wpszEqw MZIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316092; x=1724920892; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nsa0n2Jb5f9A4YGXDvRlV32yLFSY00XhLTjGYihf/fo=; b=w6x6k3z9j6SXOZdaICP2tTbKEIdkfov7z/pQgDdxCtxYukSCX3jD0fvCYj6K8PIwpg U81CXSAyWs2ylzxZI/uhSkfR8h2f0HZzCcRVW8flXQL8ew3djFJ5Z1uhkCLtPeThRMO6 AmZv5QJEGVs5mqOFL6iZSS6mOTnkxJGPxYzjt2ZyZDVrxAXUF5tXiqfhRHmbruthTMF8 2AgOzVWkvVLECurZAjvWsb3G2DLUyYhELbPevwZTFXLVB8LaohdkSJ0qdmby6HslULEF 0mDGEPpNBkOXbXpNYUntACLG1Tx+bun6UDTOD07mWyKYCYciJDfY+Lw6QbuZi4uyBRIk 5ezw== X-Gm-Message-State: AOJu0YysSxRgpwVWg3I/wkDWPXe3nwJqD8Mq7I1NhiB/0ay4XSj8G3jh WhPerfzHXqoGL3eBTJdsdqvBWM9f/gRUTds6cpaZIRQCSDuhROSBhAlBhd5p X-Google-Smtp-Source: AGHT+IE2Q5SC65ih+C3L5Zo2d+f4KHbG8LBMtkJFNYwkfoGk4qv5hn+0c6wRwDpniDBw4a3tx3m2QA== X-Received: by 2002:a05:6a20:d04f:b0:1be:c86e:7a4d with SMTP id adf61e73a8af0-1caeb34a613mr1204511637.53.1724316092021; Thu, 22 Aug 2024 01:41:32 -0700 (PDT) Received: from honey-badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71434340449sm881692b3a.218.2024.08.22.01.41.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 01:41:31 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, Eduard Zingerman Subject: [PATCH bpf-next v3 4/6] bpf: allow bpf_fastcall for bpf_cast_to_kern_ctx and bpf_rdonly_cast Date: Thu, 22 Aug 2024 01:41:10 -0700 Message-ID: <20240822084112.3257995-5-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240822084112.3257995-1-eddyz87@gmail.com> References: <20240822084112.3257995-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net do_misc_fixups() relaces bpf_cast_to_kern_ctx() and bpf_rdonly_cast() by a single instruction "r0 = r1". This follows bpf_fastcall contract. This commit allows bpf_fastcall pattern rewrite for these two functions in order to use them in bpf_fastcall selftests. Acked-by: Yonghong Song Signed-off-by: Eduard Zingerman --- kernel/bpf/verifier.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 94308cc7c503..543b8c9edac7 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16170,6 +16170,9 @@ static u32 kfunc_fastcall_clobber_mask(struct bpf_kfunc_call_arg_meta *meta) /* Same as verifier_inlines_helper_call() but for kfuncs, see comment above */ static bool is_fastcall_kfunc_call(struct bpf_kfunc_call_arg_meta *meta) { + if (meta->btf == btf_vmlinux) + return meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || + meta->func_id == special_kfunc_list[KF_bpf_rdonly_cast]; return false; } From patchwork Thu Aug 22 08:41:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13773006 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F20D0176FA4 for ; Thu, 22 Aug 2024 08:41:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316095; cv=none; b=I2g6ZoRm/mAGCCcY+sdLWC42nu3htez+u9ae215/QLEukaHa0F5mGsCOHM1aFp4XXsp8gY+vEO8uH4pxXfuJTHRAm3ogG4sGCNdFv9hXoN2TZFXVDtPlD9gueFp7Ntwdtx50wyDJzDTASrDZBSqiJgqetwQyHhHnzGPNU4E3KfA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316095; c=relaxed/simple; bh=TWaaiO9Y1pDoBk0qTAerg7n/u6L4FzNG2TuvIwwwBBM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=km1Adp+npZXUXtxZFMlReSBD06O8GMfkc15j5NIz+LJP3GdaLwCTGrRFbT2GP673I+5zSTzH9OV07IIFFF/W3Bh8IybHPffczQLiNnVyiDAV72MAn4d3uAR//ulkTvCusqO03EWCIThztUcffxsUedRu5ovZ/78fsnVmoqu9K6o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MKdAaWLW; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MKdAaWLW" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-7142448aaf9so381900b3a.1 for ; Thu, 22 Aug 2024 01:41:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316093; x=1724920893; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g4Q+yZxSLwa3eWpRVGP+bvgxAyKUo3mF9qAjV+QT0+I=; b=MKdAaWLWWscPukDB4iGAIpM3aXaNkQqPUWwESfNir2vQxhzCr/tHNgOtWHNV4+a3Wt gLNzQAwb4YhR52A+S3f2ngWz7SuUxiwBbLhDnViaUMUf2rHZzdUGz+M5qzcPlhdTDlJ3 rXNudADQ0zAHDRX5UYJz+3y4yQk0tFkZKxZudXiTsSUhxcfC5a83Ty7ADpjPMeiqLKTH EMhPo7Wmh78g463wCqzQO7J5UBZy5Lv3u1Sbxfg8kEk+mhFjebtXNwHP84siyco8D3Bq yfDzDzfLh8zfmWA0kQD7R91BlLbXnKw81cSgOFf5OW16smKRt076MZePeExdOsZezIDr bgcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316093; x=1724920893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g4Q+yZxSLwa3eWpRVGP+bvgxAyKUo3mF9qAjV+QT0+I=; b=ESOGUCDriBLuZ9W3oKA7zzNI6VaJ/NTtXbsR6xSV47pJeu0gNYv49rqgj97Y6YVUYv OkHZqbZLZ9C9js0cKSrFaj9ETPZ/JPMk6rOSsaoAgltSaQ4DA3/tdFLtRwctfCBEybzC uGlTfgzMGxXTbIQp5210fpOBMVR+UcyNLCbJTQylAwVZJGEOJkORJrf+c8U83AoIFNYq FaeUUX4RPwBSVpq6BYsgkHQX4l0t1JHRHJcF2zRie+6ojdURjkCts6drQGgxI19qlWRV LumaeQbtMsGlv1fHnMZIvcpnamuRAAU+tVtz7k7TUN7i0y3xJomOlQ3RLfwwI4OJEMXa smFw== X-Gm-Message-State: AOJu0YwUwtcTYj2l8gjON5PDjmpTh518JVYYxohGLBnWQCLHJJ6GKJD7 absfQNklTgA+6ghI86sOKW1m5oQXQWnlCibmQUYFxNKmpM8+/4E0RXYDIxHa X-Google-Smtp-Source: AGHT+IHc9tULavXcyw7m3jaiiVWLUAt8aHbVutnZK8meBcEXAiGlmHqZJq1YmpzR+z+SAzq1DO0nOA== X-Received: by 2002:aa7:8882:0:b0:704:151d:dcce with SMTP id d2e1a72fcca58-7143174cb47mr3885243b3a.5.1724316093022; Thu, 22 Aug 2024 01:41:33 -0700 (PDT) Received: from honey-badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71434340449sm881692b3a.218.2024.08.22.01.41.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 01:41:32 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, Eduard Zingerman Subject: [PATCH bpf-next v3 5/6] selftests/bpf: by default use arch mask allowing all archs Date: Thu, 22 Aug 2024 01:41:11 -0700 Message-ID: <20240822084112.3257995-6-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240822084112.3257995-1-eddyz87@gmail.com> References: <20240822084112.3257995-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net If test case does not specify architecture via __arch_* macro consider that it should be run for all architectures. Fixes: 7d743e4c759c ("selftests/bpf: __jited test tag to check disassembly after jit") Signed-off-by: Eduard Zingerman --- tools/testing/selftests/bpf/test_loader.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c index b229dd013355..2ca9b73e5a6b 100644 --- a/tools/testing/selftests/bpf/test_loader.c +++ b/tools/testing/selftests/bpf/test_loader.c @@ -543,7 +543,7 @@ static int parse_test_spec(struct test_loader *tester, } } - spec->arch_mask = arch_mask; + spec->arch_mask = arch_mask ?: -1; if (spec->mode_mask == 0) spec->mode_mask = PRIV; From patchwork Thu Aug 22 08:41:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eduard Zingerman X-Patchwork-Id: 13773007 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17D3517C7C3 for ; Thu, 22 Aug 2024 08:41:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316096; cv=none; b=TvP9r+UVnJB2OhKQ1WcTqZvVBoL+pusACSb5bMOvUcjQ2zlJN9MwVez9AQ1fq4KYy1DQiIS9Nt9gyc7dBc76XoTeSuVaPXMetP4lDf+8NYhm3Kpjr9fa6HQeDkgIBHOCXKcfTGga+IP6y6WmDMzq6sx/5EJYBqcYJq2oYirz4n0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724316096; c=relaxed/simple; bh=SPv7SzAnZuFx6VBDu1Us/wEVReLGH20Z+96IAklt0QU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=noo/rTmXqhsOOytf8Lnt6dg4hcKePWc6OX/vZM3byPYTtjMesxr98EDBJvHmoK0/yyqw8VRTCNLu2KYQTKc65brTaO+fsrIN+Qqn38hGN/48Shgty1uuB+wjJtfAvXOblHt2wZfMTXwtyGhQl6wtScem3tG6mz4zAGefgO3Gr9o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GZT/3qIX; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GZT/3qIX" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-71423704ef3so466097b3a.3 for ; Thu, 22 Aug 2024 01:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316094; x=1724920894; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=imjPoZCl/bEQJlJqnmv17tA/yvenbStSD5R5DrglOiI=; b=GZT/3qIX5BA4bKoLVIpRFJYi652Lc6SfJTTRgeMeFWui3605+sNhjEAcJEKOMnP09x 2Z7RgqXpMI9y4M7URQ81sEnrn2knodD1glWwB3FpElPmKxVVJ8p69shF9tHn45JQ0CSk HbVPDJc7BAzp6xG3OfvnUYs5WTVMQHkvmnzQ70Z115mrGFgeUlKczlCv1yF7b80rC6Uo YVcMAUBYaBLAOCPAqI0eeUC13380xElqfPPJ2go5dbN+K/42/aQb6Nke9AjbHVZvptC0 D2rlCeX8wB0tNaKKFgNZEmGPyma5uwtvK1qoOsZSYVgK2Qi6RBpPCchxFXZyYsZomtht bwOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316094; x=1724920894; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=imjPoZCl/bEQJlJqnmv17tA/yvenbStSD5R5DrglOiI=; b=BcjoF8fFt5Vyo6WtCfZluEEv8nmS8F+s+B/BUqo+f1tabmh4urVcByi4D/gAK4oRjb feDkkp527b1VFGSqQNNuVSEvvnwuuG95QcF5cyiw7L5khYoWAzUqzSiI0dA74EHgC1dk 49LTDdQouEJ9HkqRdQqBuQ2jFg9n6FMo9QQMInpF0dXJluYF8oBGQdh+YMT2lKKMYr2W GdpP1VupeagajEQLU7asw7ovynPCh7IH4ALCNamt2LAZCeyzQNhmGauF9xrQQ2Acuuk0 rIDYvBfeMUYg/j60l1/qChuRx1rT8XOY5Nx7rMSfoQWUdTj5+NMiyMR9grIruK6r52WG g1XQ== X-Gm-Message-State: AOJu0YzKKSQnbyiwrbDh4Dpx7W2uaeYwpPDfeJzqUBtX4W//CTq+GOeC ruoioTYHOuwdL+MIS/ycUANky7LplEW2q/lUaBiuSFmGNEsjWOBYf9j0CO8o X-Google-Smtp-Source: AGHT+IESyjjlw1PQUIY87nmi5Y7reGJjKBkLZvTJp92165UObjIKNY8tKB+xzQyUoqxUjGg5cwS8gg== X-Received: by 2002:a05:6a20:9f9b:b0:1c3:b61c:57cb with SMTP id adf61e73a8af0-1caeb360b32mr1181311637.53.1724316094033; Thu, 22 Aug 2024 01:41:34 -0700 (PDT) Received: from honey-badger.. ([38.34.87.7]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71434340449sm881692b3a.218.2024.08.22.01.41.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 01:41:33 -0700 (PDT) From: Eduard Zingerman To: bpf@vger.kernel.org, ast@kernel.org Cc: andrii@kernel.org, daniel@iogearbox.net, martin.lau@linux.dev, kernel-team@fb.com, yonghong.song@linux.dev, Eduard Zingerman Subject: [PATCH bpf-next v3 6/6] selftests/bpf: check if bpf_fastcall is recognized for kfuncs Date: Thu, 22 Aug 2024 01:41:12 -0700 Message-ID: <20240822084112.3257995-7-eddyz87@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240822084112.3257995-1-eddyz87@gmail.com> References: <20240822084112.3257995-1-eddyz87@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Use kfunc_bpf_cast_to_kern_ctx() and kfunc_bpf_rdonly_cast() to verify that bpf_fastcall pattern is recognized for kfunc calls. Acked-by: Yonghong Song Signed-off-by: Eduard Zingerman --- .../bpf/progs/verifier_bpf_fastcall.c | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c index e30ab9fe5096..9da97d2efcd9 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c @@ -2,8 +2,11 @@ #include #include +#include #include "../../../include/linux/filter.h" #include "bpf_misc.h" +#include +#include "bpf_kfuncs.h" SEC("raw_tp") __arch_x86_64 @@ -842,4 +845,56 @@ __naked int bpf_fastcall_max_stack_fail(void) ); } +SEC("cgroup/getsockname_unix") +__xlated("0: r2 = 1") +/* bpf_cast_to_kern_ctx is replaced by a single assignment */ +__xlated("1: r0 = r1") +__xlated("2: r0 = r2") +__xlated("3: exit") +__success +__naked void kfunc_bpf_cast_to_kern_ctx(void) +{ + asm volatile ( + "r2 = 1;" + "*(u64 *)(r10 - 32) = r2;" + "call %[bpf_cast_to_kern_ctx];" + "r2 = *(u64 *)(r10 - 32);" + "r0 = r2;" + "exit;" + : + : __imm(bpf_cast_to_kern_ctx) + : __clobber_all); +} + +SEC("raw_tp") +__xlated("3: r3 = 1") +/* bpf_rdonly_cast is replaced by a single assignment */ +__xlated("4: r0 = r1") +__xlated("5: r0 = r3") +void kfunc_bpf_rdonly_cast(void) +{ + asm volatile ( + "r2 = %[btf_id];" + "r3 = 1;" + "*(u64 *)(r10 - 32) = r3;" + "call %[bpf_rdonly_cast];" + "r3 = *(u64 *)(r10 - 32);" + "r0 = r3;" + : + : __imm(bpf_rdonly_cast), + [btf_id]"r"(bpf_core_type_id_kernel(union bpf_attr)) + : __clobber_common); +} + +/* BTF FUNC records are not generated for kfuncs referenced + * from inline assembly. These records are necessary for + * libbpf to link the program. The function below is a hack + * to ensure that BTF FUNC records are generated. + */ +void kfunc_root(void) +{ + bpf_cast_to_kern_ctx(0); + bpf_rdonly_cast(0, 0); +} + char _license[] SEC("license") = "GPL";