From patchwork Fri Feb 7 02:05:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E4C3C02194 for ; Fri, 7 Feb 2025 02:08:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=eQntqM+rLZWQ3ApAwlHMCjvEA5GNUgj+7EI4yYRHGrA=; b=TEafRaWKeLxOfwzjt/p5hM/jTf CuQGSt88W9RCwH34JEUKjl+Rot3xAfsnCYlb+SQmwMH4YxeyCBne3Z5zkQ+hTP+EKxdZE1TN/I/uO QVgIIbW8ryAyTrqqyQ+t/dr8aM0bvwp9qSXr0Vk7ZW/QK6fpWX7GFVqYauipx4AH5QV9BNJq1MmRJ D6b2uzJJFXmVGXL2qAe8PDLYNeLvJVqsxOwWgwRDPVWJFY9d4K2DAgz0TKMCudQwj8lBJ+yEpmueA 1Z0nzNos5m/m7YkBxwmfhuDpa7rI+deQwQIcym5+86ztNSZZIShQq/gN0DKK/Lezml7osXP+L5ZeX jjmGv7lw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDmb-0000000823V-12s6; Fri, 07 Feb 2025 02:07:53 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDk5-000000081NO-39F9 for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:05:18 +0000 Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-2f550d28f7dso3247425a91.3 for ; Thu, 06 Feb 2025 18:05:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893916; x=1739498716; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=eQntqM+rLZWQ3ApAwlHMCjvEA5GNUgj+7EI4yYRHGrA=; b=ExXInPuRacJhyT8lOhkR4jfC61nufn/u2LezvLYPsKCHprsieD/8wazbRZELjyn42G Q9Hso6aJu11sIxz91a/H+DNeKILVLIHNZpjcrSDQDzKlRB+VmKjtifKmylL2pzBywoDH ckja/IAYRDGK393mnP9P+R55MsiPPQrqaQMOk2CT1qSaM3lUK9tuACIjwjQKRoG4bVZ8 gNnjuLfkNrZO9IoEGvA1IXfY/1/fauMe/Br0hhL6Ay21NNkU4tE1rtqpC/Ai8gWfBfsD W+xTPRCOVxq2O2a4Nk7aPmEla5EHoCYmrQSdgFaLOCjypf/o5+Wn5lPz0/lfCXAdsMZS q9Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893916; x=1739498716; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eQntqM+rLZWQ3ApAwlHMCjvEA5GNUgj+7EI4yYRHGrA=; b=OJdXrhnq76ZomqQmRnKjhLiNcbSuZJv7IRXU0YCWIsVK0juognahOrgieaAYN8ITxX DRHzkBRVQBtbzMkyinQd0XLzc7QLXFCLIBWzWN2hUfmpOM6EbmxuQVSY0fs1Jk9X2GJh jO2q5PkFHRFUK9Kwh0bGN2GpmT6cTFiM5lWbCY1HgITBDffzduiaNfxtK3Ge3EEYmuGU PZcEDsd3e76RMMe7KdTviIkbZ6Gld7CyAG8FblyEjY8dgNCeZd8bZGA12xeixeDRvra3 yMB5pVhEHLTUgZaBXRhy8DiD0hmPIYvpFJhitu0bfDXB748VyeyMIiIdcJjvx8r30aUv b/YA== X-Forwarded-Encrypted: i=1; AJvYcCXYt+iNJbxQCLyczs80ndI/Esiv7h+E2d317JXR6+DpNDeijmxWQIIlQKHPHBNMdS8tJYTZfD7EoBk71nkRWi6p@lists.infradead.org X-Gm-Message-State: AOJu0YwMFH4W83nCIZlUMBnaJ6ZwNYJE95y9yji81w0W16kAzSt2kjiG pGSE8EpG7zhkY2OHqo7wIuJyhvL6iLEjNp4vTDxPHAvqv83dnBVV5YqpSOxtltPDdgPZmcj0yNP /dTWEOIzX+g== X-Google-Smtp-Source: AGHT+IG8RnoKYzqmdlNq/GNWt0Bfp60toUuFc3O205+7HMgGDaEyEjHnEEWL3iMa82zSQ8qscCoeONeApNS1Lw== X-Received: from pjg15.prod.google.com ([2002:a17:90b:3f4f:b0:2fa:1fac:2695]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4cd1:b0:2fa:157e:c78e with SMTP id 98e67ed59e1d1-2fa23f5ebb1mr2348726a91.7.1738893916574; Thu, 06 Feb 2025 18:05:16 -0800 (PST) Date: Fri, 7 Feb 2025 02:05:13 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <56a8a44753b6e12abcb61163ba97159615ca8114.1738888641.git.yepeilin@google.com> Subject: [PATCH bpf-next v2 1/9] bpf/verifier: Factor out atomic_ptr_type_ok() From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180517_797557_27A2BC6B X-CRM114-Status: GOOD ( 12.81 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Factor out atomic_ptr_type_ok() as a helper function to be used later. Signed-off-by: Peilin Ye --- kernel/bpf/verifier.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9971c03adfd5..0935b72fe716 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6116,6 +6116,26 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno) return reg->type == PTR_TO_ARENA; } +/* Return false if @regno contains a pointer whose type isn't supported for + * atomic instruction @insn. + */ +static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno, + struct bpf_insn *insn) +{ + if (is_ctx_reg(env, regno)) + return false; + if (is_pkt_reg(env, regno)) + return false; + if (is_flow_key_reg(env, regno)) + return false; + if (is_sk_reg(env, regno)) + return false; + if (is_arena_reg(env, regno)) + return bpf_jit_supports_insn(insn, true); + + return true; +} + static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { #ifdef CONFIG_NET [PTR_TO_SOCKET] = &btf_sock_ids[BTF_SOCK_TYPE_SOCK], @@ -7572,11 +7592,7 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i return -EACCES; } - if (is_ctx_reg(env, insn->dst_reg) || - is_pkt_reg(env, insn->dst_reg) || - is_flow_key_reg(env, insn->dst_reg) || - is_sk_reg(env, insn->dst_reg) || - (is_arena_reg(env, insn->dst_reg) && !bpf_jit_supports_insn(insn, true))) { + if (!atomic_ptr_type_ok(env, insn->dst_reg, insn)) { verbose(env, "BPF_ATOMIC stores into R%d %s is not allowed\n", insn->dst_reg, reg_type_str(env, reg_state(env, insn->dst_reg)->type)); From patchwork Fri Feb 7 02:05:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 22969C02194 for ; Fri, 7 Feb 2025 02:09:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=rnyfeyX5KsaoejFCT4Pvz88fBme+Wvr3R6AqYNdoRSY=; b=Qxf+8AibBBjxHnAyYGj5OGK1ru FMUZ36Ad691nDtPjBawYsLCEoRHjqnQBw1y9q2IgI9O7VGXIFP0i3sbbI9daLOVRY7WpC/xCIZhbN izqW8+5giIr0a8zSs6ol8eRDoxJ2Gluq3U6HWfNkYIIYSV/+PZHGetfARSr+MHtJcc5ll4XxCcLWC MgEMsPxjtXxsHoE3vqa374blGM+rH231mEFkIsafPRnW0WQefC8N6gPDAHL1g8RG/1W9trSMhF4bl XMggCUQX1whVA3bSGQrwVtLcj4da/ttirrxZTtrkCzYCB7IyPFs+iyybdIzl0zcqG82Iv21pLh3fY Rln1w0HA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDnv-000000082Cu-3r7i; Fri, 07 Feb 2025 02:09:15 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDkJ-000000081QE-1Azj for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:05:32 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2f9f42d98e3so3285097a91.0 for ; Thu, 06 Feb 2025 18:05:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893930; x=1739498730; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rnyfeyX5KsaoejFCT4Pvz88fBme+Wvr3R6AqYNdoRSY=; b=M6YC06d6uxJwOO5L27+4qmQQwWvYDL5WmiPHmM2cVrLH41Zcr0kytjjTXlKaKjBCgU JebpQTU8oBb98zhfar6E8POILs3g/30ous5taAw50YDQ9YBYvrwH6pU+Q31KWBy3ddVW EMe2bg2J8TjWC+Ik9Q3+luVv5OvKHUkkMFYYCoxEvqL1eGc3pI8BWD1Cguv6Ayq4kyzk +j4MSJ8ANvVe3tM3jxnhiurYiZEtKDK54E3a6fJtJXas1fUlPZMFtfcARtjVP9AWamun kcb5Gzxo1b9KNCFoo7FwukuXoTxgy0vhSRvANLbWCQJjRmsck9Cl5iclhejb1+/oYU2Z dsog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893930; x=1739498730; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rnyfeyX5KsaoejFCT4Pvz88fBme+Wvr3R6AqYNdoRSY=; b=PuqeVtrFaoktxyrxVV5IloT28VEWmIOoxt4BXkz4aM6qD1ffap+U5jZn7V4DGOmgSh td1yzmzXiJP7y00M9xbg0yl93ye98jFUqtWi0GjMu28+L/v8zIhcFwPxC/cqoM4oHt8t AEB84nh838+rauiBzGz/P62ih2/ZEksNaqvItTVDjG1qyR7CD8wXq3qnOpzr3VRp5daM 5G7WMF+dyc54v0FYBG6Q6kY/2FLxicgkeH4MO0+taqA+1Wi8ZUf2uOmBY5GCAiPU0AsU +/UbROZSNdsJU3o0135HZSAFQMaNTNo6ChHTeSjlsci1j7heIQj0WQVIdSn0tZ06bFs0 AvWA== X-Forwarded-Encrypted: i=1; AJvYcCW1kQevlrbeQjrxQhwaxs/a+Gcd7PzZcLAeNbSa6b3MmC9xY6FvsYY6wKWboM0Q/8QiON3K7DbFHnZY+O5ucnjw@lists.infradead.org X-Gm-Message-State: AOJu0YyeRwQY3Ndw2Xoge/etbnrC0ots4A0ZqSSuxM0qx/sLSRfqncIm 77v4Md9JirTzTDMkLwGGPzRbZyLCLt+/nXsAd8GEothfN++9f/JY5B8apOJOYZ0FLOzs0qyP3mS bInLf+guoJg== X-Google-Smtp-Source: AGHT+IF7v6kmdiGCxp2a0C++c/LlhPUZNvA/P6MK7nWPEghpqss2oNpv+9B3PTntoS5TGpQxBNw7FSCUTIfdEw== X-Received: from pjbsl12.prod.google.com ([2002:a17:90b:2e0c:b0:2e2:44f2:9175]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4cd1:b0:2fa:157e:c78e with SMTP id 98e67ed59e1d1-2fa23f5ebb1mr2349911a91.7.1738893930308; Thu, 06 Feb 2025 18:05:30 -0800 (PST) Date: Fri, 7 Feb 2025 02:05:23 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: Subject: [PATCH bpf-next v2 2/9] bpf/verifier: Factor out check_atomic_rmw() From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180531_318918_5DC8F279 X-CRM114-Status: GOOD ( 15.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, check_atomic() only handles atomic read-modify-write (RMW) instructions. Since we are planning to introduce other types of atomic instructions (i.e., atomic load/store), extract the existing RMW handling logic into its own function named check_atomic_rmw(). Remove the @insn_idx parameter as it is not really necessary. Use 'env->insn_idx' instead, as in other places in verifier.c. Signed-off-by: Peilin Ye --- kernel/bpf/verifier.c | 53 +++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 24 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 0935b72fe716..39eb990ec003 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7536,28 +7536,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type, bool allow_trust_mismatch); -static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_insn *insn) +static int check_atomic_rmw(struct bpf_verifier_env *env, + struct bpf_insn *insn) { int load_reg; int err; - switch (insn->imm) { - case BPF_ADD: - case BPF_ADD | BPF_FETCH: - case BPF_AND: - case BPF_AND | BPF_FETCH: - case BPF_OR: - case BPF_OR | BPF_FETCH: - case BPF_XOR: - case BPF_XOR | BPF_FETCH: - case BPF_XCHG: - case BPF_CMPXCHG: - break; - default: - verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", insn->imm); - return -EINVAL; - } - if (BPF_SIZE(insn->code) != BPF_W && BPF_SIZE(insn->code) != BPF_DW) { verbose(env, "invalid atomic operand size\n"); return -EINVAL; @@ -7619,12 +7603,12 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i /* Check whether we can read the memory, with second call for fetch * case to simulate the register fill. */ - err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off, + err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, BPF_SIZE(insn->code), BPF_READ, -1, true, false); if (!err && load_reg >= 0) - err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off, - BPF_SIZE(insn->code), BPF_READ, load_reg, - true, false); + err = check_mem_access(env, env->insn_idx, insn->dst_reg, + insn->off, BPF_SIZE(insn->code), + BPF_READ, load_reg, true, false); if (err) return err; @@ -7634,13 +7618,34 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i return err; } /* Check whether we can write into the same memory. */ - err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off, + err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, BPF_SIZE(insn->code), BPF_WRITE, -1, true, false); if (err) return err; return 0; } +static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) +{ + switch (insn->imm) { + case BPF_ADD: + case BPF_ADD | BPF_FETCH: + case BPF_AND: + case BPF_AND | BPF_FETCH: + case BPF_OR: + case BPF_OR | BPF_FETCH: + case BPF_XOR: + case BPF_XOR | BPF_FETCH: + case BPF_XCHG: + case BPF_CMPXCHG: + return check_atomic_rmw(env, insn); + default: + verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", + insn->imm); + return -EINVAL; + } +} + /* When register 'regno' is used to read the stack (either directly or through * a helper function) make sure that it's within stack boundary and, depending * on the access type and privileges, that all elements of the stack are @@ -19076,7 +19081,7 @@ static int do_check(struct bpf_verifier_env *env) enum bpf_reg_type dst_reg_type; if (BPF_MODE(insn->code) == BPF_ATOMIC) { - err = check_atomic(env, env->insn_idx, insn); + err = check_atomic(env, insn); if (err) return err; env->insn_idx++; From patchwork Fri Feb 7 02:05:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AB0BDC02194 for ; Fri, 7 Feb 2025 02:10:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=G0ksBymLsFxBv6dTHVXwRUMsPxued+iewutBsq02rrE=; b=QzKpgmj6wsceqv856vlo+vztFy d+gsX6W/AVpvPf18Ij0m6iyaRxqiKSoE1AZov2ovhSa77oX6C5v3adIQrIjNEoGVtqzWir3+cLu6h ZQiVAYTGSh5mtmoDRt1qGVor2+aWgLqugNEXYdTbQKZ1ZeEwFX1fQ8r7Jlsb3pa65LnqsX61QjGvc gvxJvet8dwqrs/fECk6BIkOaef9XhQeu47HaL+H/1jBHMQNc19p1fCoC17F5DOhCVBuWqvT/XOmb2 9XOoTmQQ+7TrZH3rWV8/7epWPxt/tcjKtqocuYaJQN1gKLowWeyAL1tHYt3i1wugnekRyDBU+bqKX yUY10g3Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDpH-000000082Kc-2JTG; Fri, 07 Feb 2025 02:10:39 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDkR-000000081Rj-3ldS for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:05:41 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2f9e1bf0a3bso5200591a91.2 for ; Thu, 06 Feb 2025 18:05:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893939; x=1739498739; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=G0ksBymLsFxBv6dTHVXwRUMsPxued+iewutBsq02rrE=; b=VSjsMtZe0PaXh/kIHI/4+26ehWi/OWT1f/AZWHXqPgHE7JLiWsXCycxyfavWqxqmqS BzklfnvxWDHYQR3Y7TOHBGDVBoewthGm+1qIfW5poke0QWf5LTk//Dc2H32VKcQ/zA32 IXH0Bqzv7olpRPt84I72dtr6pga6CI1YNcUC6QSs91XKczurZyow5OzBXASzkgcWUSbj ucDAn+vZ6y/r9DLqZTW5kdS7CKcl3mMy+s978infRYQ9sDe3v+mA2ab6z2KTOcE7473S 8sWsWc1jo88FQk3okvEM6mES1gvhZb7S5Jz/RodeKqPVLG+lIeHUNQL76pYBqqXZHD6a kArA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893939; x=1739498739; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=G0ksBymLsFxBv6dTHVXwRUMsPxued+iewutBsq02rrE=; b=HUzjNwwXjE8ZnQl2kCtbjx3ZW11xGp1HVhD0g3I986SHw3cLIaJEsLLWk97iOFRL+Q aid4RpgFi/tfK1t0nKQn8qLwmiz9gakUGUHUD4V6tpf2X1xJK0Gfc7xRB/g5jT4DeNaH JkvKL5drCRFatDEyHSQ8YFX9MvSFnvm68uKYUMTapkKZ2uSh1JFd8Ohp+rUVQsJ6Un7h nInjW7oErrkgW2zIjT3/vPipPaGzc60P4zwK+ImUbSnILMbu6tOTi327o6cKjSsclItY S78mmgT6hTbu/ltR9Dfo4QCF0B37btWR9wUlPKju77ue8PfBXgtOfvP5jLnvK9V3TKmv vuCw== X-Forwarded-Encrypted: i=1; AJvYcCXB6Mhd9OpUzzQoucBlQ/+RWrc8W/clCgfvheGnegt0mBfenzmLpAAQBqL+hICJjj1A+kNDedlbQQBTgy1kHjpK@lists.infradead.org X-Gm-Message-State: AOJu0YwYg1TfJfAFCavRWFiqdna8kILkBWDBI+ugQWMz1zg5w1Hdb1y2 KKFLLmlzDrQuxcmz9BscoB3FNInj6+AZLB0aiQnYicMqDAvkzi0EixcGOHO6XNLGIY94awhsPkP TA2hl1fpOKg== X-Google-Smtp-Source: AGHT+IEQXS8/Za22ftZ1p+Phe57YwI7GpMBkpoOV+QxJBEcmY8iFZ5iWxAf92Zvu9JrzhNYd5itkmkXtR2L03w== X-Received: from pfai16.prod.google.com ([2002:aa7:9090:0:b0:72b:ccb:c99b]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:882:b0:71e:6b8:2f4a with SMTP id d2e1a72fcca58-7305d47c9d1mr2600297b3a.12.1738893938799; Thu, 06 Feb 2025 18:05:38 -0800 (PST) Date: Fri, 7 Feb 2025 02:05:34 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <363538659933fb2e289d937d81fc8eecb284ea7b.1738888641.git.yepeilin@google.com> Subject: [PATCH bpf-next v2 3/9] bpf/verifier: Factor out check_load_mem() and check_store_reg() From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180539_938378_E13DB57C X-CRM114-Status: GOOD ( 16.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Extract BPF_LDX and most non-atomic BPF_STX instruction handling logic in do_check() into helper functions to be used later. While we are here, make that comment about "reserved fields" more specific. Suggested-by: Eduard Zingerman Signed-off-by: Peilin Ye Acked-by: Eduard Zingerman --- kernel/bpf/verifier.c | 110 +++++++++++++++++++++++++----------------- 1 file changed, 67 insertions(+), 43 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 39eb990ec003..82a5a4acf576 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7536,6 +7536,67 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type, bool allow_trust_mismatch); +static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, + bool strict_alignment_once, bool is_ldsx, + bool allow_trust_mismatch, const char *ctx) +{ + struct bpf_reg_state *regs = cur_regs(env); + enum bpf_reg_type src_reg_type; + int err; + + /* check src operand */ + err = check_reg_arg(env, insn->src_reg, SRC_OP); + if (err) + return err; + + /* check dst operand */ + err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK); + if (err) + return err; + + src_reg_type = regs[insn->src_reg].type; + + /* Check if (src_reg + off) is readable. The state of dst_reg will be + * updated by this call. + */ + err = check_mem_access(env, env->insn_idx, insn->src_reg, insn->off, + BPF_SIZE(insn->code), BPF_READ, insn->dst_reg, + strict_alignment_once, is_ldsx); + err = err ?: save_aux_ptr_type(env, src_reg_type, + allow_trust_mismatch); + err = err ?: reg_bounds_sanity_check(env, ®s[insn->dst_reg], ctx); + + return err; +} + +static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, + bool strict_alignment_once) +{ + struct bpf_reg_state *regs = cur_regs(env); + enum bpf_reg_type dst_reg_type; + int err; + + /* check src1 operand */ + err = check_reg_arg(env, insn->src_reg, SRC_OP); + if (err) + return err; + + /* check src2 operand */ + err = check_reg_arg(env, insn->dst_reg, SRC_OP); + if (err) + return err; + + dst_reg_type = regs[insn->dst_reg].type; + + /* Check if (dst_reg + off) is writeable. */ + err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, + BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg, + strict_alignment_once, false); + err = err ?: save_aux_ptr_type(env, dst_reg_type, false); + + return err; +} + static int check_atomic_rmw(struct bpf_verifier_env *env, struct bpf_insn *insn) { @@ -19051,35 +19112,16 @@ static int do_check(struct bpf_verifier_env *env) return err; } else if (class == BPF_LDX) { - enum bpf_reg_type src_reg_type; - - /* check for reserved fields is already done */ - - /* check src operand */ - err = check_reg_arg(env, insn->src_reg, SRC_OP); - if (err) - return err; + bool is_ldsx = BPF_MODE(insn->code) == BPF_MEMSX; - err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK); - if (err) - return err; - - src_reg_type = regs[insn->src_reg].type; - - /* check that memory (src_reg + off) is readable, - * the state of dst_reg will be updated by this func + /* Check for reserved fields is already done in + * resolve_pseudo_ldimm64(). */ - err = check_mem_access(env, env->insn_idx, insn->src_reg, - insn->off, BPF_SIZE(insn->code), - BPF_READ, insn->dst_reg, false, - BPF_MODE(insn->code) == BPF_MEMSX); - err = err ?: save_aux_ptr_type(env, src_reg_type, true); - err = err ?: reg_bounds_sanity_check(env, ®s[insn->dst_reg], "ldx"); + err = check_load_mem(env, insn, false, is_ldsx, true, + "ldx"); if (err) return err; } else if (class == BPF_STX) { - enum bpf_reg_type dst_reg_type; - if (BPF_MODE(insn->code) == BPF_ATOMIC) { err = check_atomic(env, insn); if (err) @@ -19093,25 +19135,7 @@ static int do_check(struct bpf_verifier_env *env) return -EINVAL; } - /* check src1 operand */ - err = check_reg_arg(env, insn->src_reg, SRC_OP); - if (err) - return err; - /* check src2 operand */ - err = check_reg_arg(env, insn->dst_reg, SRC_OP); - if (err) - return err; - - dst_reg_type = regs[insn->dst_reg].type; - - /* check that memory (dst_reg + off) is writeable */ - err = check_mem_access(env, env->insn_idx, insn->dst_reg, - insn->off, BPF_SIZE(insn->code), - BPF_WRITE, insn->src_reg, false, false); - if (err) - return err; - - err = save_aux_ptr_type(env, dst_reg_type, false); + err = check_store_reg(env, insn, false); if (err) return err; } else if (class == BPF_ST) { From patchwork Fri Feb 7 02:05:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 43541C02194 for ; Fri, 7 Feb 2025 02:12:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ddcXobZXl7/L485JhUMDf0sDIc/e0wxrN1hs3/Yc8Ck=; b=PeOy4rHgIplPNqPFR+mGtpAnA5 0ZsjopqCENjIBRL7CnLiOUN58oGcxkCGwHhs4GDLJkCaaJ65T4S6SrH1C0lREZlkUyGfUMtoaKB0R yXbLQkuTC9Nt4e7ZfTVyJnHZ6z2XK7XvZzaYY58jJQUvKY7GlMN39nwRTHH0zgICOn1QESF13oTUm LFnjgf8xVwMyYqiFwNAXre6tou3eah1eOc6LTUYcuZVJ52kapBURaF/PK9iwfXGqXIRg+tUfyT1ey 71kMhkholQ+CPdp5l7nfSqjvC6CMU+wAU7JoANNjeAQa0CATD6ayf4n521Ou85XHtKTxC72sbtigV s3lzP5qw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDqd-000000082Ta-197q; Fri, 07 Feb 2025 02:12:03 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDkk-000000081VO-0KHA for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:05:59 +0000 Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-21f5067e443so4662415ad.0 for ; Thu, 06 Feb 2025 18:05:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893956; x=1739498756; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ddcXobZXl7/L485JhUMDf0sDIc/e0wxrN1hs3/Yc8Ck=; b=IB+xptKyn8Fb7Gk72YgiSAig8KzD7Wg3JOz778ydQaHdg5tG3JK7EDcb8JDXb63hBD e4spbRM2Hw5T7YnIBqCkI05knG3JcUcdL53mj5krjtEb+cJND0eDqRAK95VkO2qSiQzh 43wopF2kgRTOd2PKuIdWQt06nMDLsn7SpofWO5NZ/37lfFq4MmD/SpZ60GRohlhr3522 U5+JXWN/t2/PzR4CHfaF0QrmJqOnSl01FDYokWvOdWSaY4BXnkfce3hnYKLtRneM4lMf XLZy8t/HcjF6NBMR7NShPL+SFU1+D7jfFi9whXv7xRBYMILrX3ZEybkIpInWncQkdB+v cgbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893956; x=1739498756; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ddcXobZXl7/L485JhUMDf0sDIc/e0wxrN1hs3/Yc8Ck=; b=PJeADJ71pGBXId98kqjXwtSrPQsSlLkDFgDQSfwVPrAaDR4xi43kGXcMdzOPwaWO+p wHvmliwU5ZVBr/6fzfllABw8Rv3SR9mubwfvwTxpplpBD6i0r6FkYL++d4oeq9hnZKS2 DL/lDHxla+2BB0BJnXWZGra7ywRKjz1wVxj2O4BKuoejIzw6y8G0WVfOrycY2xkyelXb EAMQKNmyeb/2Ym69Ouz8H4Rv0DQkM6aTvIhIR5iH910VoxLiZgn3faTi4T4Mq5GNT4Up CLoHUcfpfQw05rMUy70WQa6Uv9gDZSvfuRbd0CM4LCTzD8QbiT2Wg9gynKhcUUGZUSfj ki3Q== X-Forwarded-Encrypted: i=1; AJvYcCULE7HOAOqdusr7m4HIQL3eJijQu7msth0OKo55RTb4kJwMlVj3UtK8WelgaklS+3NyUhzsWQX/j/kvN3UQ87ew@lists.infradead.org X-Gm-Message-State: AOJu0YxTok2xmubWLNhrOoGFPyo9v/2GXQftavR9a1eDhITV9kqgMikz RXAY8/YUOQNWxXZXBXtjXtL71Nib36ZDg/5/iwc64S1j2Cwtj024sf+nmC1qHUPBINEy8hcnChe xqW6/K00sMA== X-Google-Smtp-Source: AGHT+IHaFr+HT1k7LbVBAkON0Sg6P00qfAjEH6NZPFl87AchQ8sNqQ5UN8DGIlVFWcsU20JIhNme+XM/inTD9A== X-Received: from plbkn6.prod.google.com ([2002:a17:903:786:b0:21f:544:758]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:244c:b0:215:401b:9535 with SMTP id d9443c01a7336-21f4e756726mr24529425ad.47.1738893956357; Thu, 06 Feb 2025 18:05:56 -0800 (PST) Date: Fri, 7 Feb 2025 02:05:44 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: Subject: [PATCH bpf-next v2 4/9] bpf: Introduce load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180558_124713_8A3BC415 X-CRM114-Status: GOOD ( 21.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce BPF instructions with load-acquire and store-release semantics, as discussed in [1]. The following new flags are defined: BPF_ATOMIC_LOAD 0x10 BPF_ATOMIC_STORE 0x20 BPF_ATOMIC_TYPE(imm) ((imm) & 0xf0) BPF_RELAXED 0x0 BPF_ACQUIRE 0x1 BPF_RELEASE 0x2 BPF_ACQ_REL 0x3 BPF_SEQ_CST 0x4 BPF_LOAD_ACQ (BPF_ATOMIC_LOAD | BPF_ACQUIRE) BPF_STORE_REL (BPF_ATOMIC_STORE | BPF_RELEASE) A "load-acquire" is a BPF_STX | BPF_ATOMIC instruction with the 'imm' field set to BPF_LOAD_ACQ (0x11). Similarly, a "store-release" is a BPF_STX | BPF_ATOMIC instruction with the 'imm' field set to BPF_STORE_REL (0x22). Unlike existing atomic operations that only support BPF_W (32-bit) and BPF_DW (64-bit) size modifiers, load-acquires and store-releases also support BPF_B (8-bit) and BPF_H (16-bit). An 8- or 16-bit load-acquire zero-extends the value before writing it to a 32-bit register, just like ARM64 instruction LDARH and friends. As an example, consider the following 64-bit load-acquire BPF instruction: db 10 00 00 11 00 00 00 r0 = load_acquire((u64 *)(r1 + 0x0)) opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX imm (0x00000011): BPF_LOAD_ACQ Similarly, a 16-bit BPF store-release: cb 21 00 00 22 00 00 00 store_release((u16 *)(r1 + 0x0), w2) opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX imm (0x00000022): BPF_STORE_REL In arch/{arm64,s390,x86}/net/bpf_jit_comp.c, have bpf_jit_supports_insn(..., /*in_arena=*/true) return false for the new instructions, until the corresponding JIT compiler supports them. [1] https://lore.kernel.org/all/20240729183246.4110549-1-yepeilin@google.com/ Acked-by: Eduard Zingerman Signed-off-by: Peilin Ye Acked-by: Ilya Leoshkevich --- arch/arm64/net/bpf_jit_comp.c | 4 +++ arch/s390/net/bpf_jit_comp.c | 14 +++++--- arch/x86/net/bpf_jit_comp.c | 4 +++ include/linux/bpf.h | 11 ++++++ include/linux/filter.h | 2 ++ include/uapi/linux/bpf.h | 13 +++++++ kernel/bpf/core.c | 63 ++++++++++++++++++++++++++++++---- kernel/bpf/disasm.c | 12 +++++++ kernel/bpf/verifier.c | 45 ++++++++++++++++++++++-- tools/include/uapi/linux/bpf.h | 13 +++++++ 10 files changed, 168 insertions(+), 13 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 8446848edddb..8c3b47d9e441 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2667,8 +2667,12 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) if (!in_arena) return true; switch (insn->code) { + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_H: case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: + if (bpf_atomic_is_load_store(insn)) + return false; if (!cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) return false; } diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 9d440a0b729e..0776dfde2dba 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2919,10 +2919,16 @@ bool bpf_jit_supports_arena(void) bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) { - /* - * Currently the verifier uses this function only to check which - * atomic stores to arena are supported, and they all are. - */ + if (!in_arena) + return true; + switch (insn->code) { + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_H: + case BPF_STX | BPF_ATOMIC | BPF_W: + case BPF_STX | BPF_ATOMIC | BPF_DW: + if (bpf_atomic_is_load_store(insn)) + return false; + } return true; } diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index a43fc5af973d..f0c31c940fb8 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3771,8 +3771,12 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) if (!in_arena) return true; switch (insn->code) { + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_H: case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: + if (bpf_atomic_is_load_store(insn)) + return false; if (insn->imm == (BPF_AND | BPF_FETCH) || insn->imm == (BPF_OR | BPF_FETCH) || insn->imm == (BPF_XOR | BPF_FETCH)) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index f3f50e29d639..96c936fd125f 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -990,6 +990,17 @@ static inline bool bpf_pseudo_func(const struct bpf_insn *insn) return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC; } +/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an + * atomic load or store, and false if it is a read-modify-write instruction. + */ +static inline bool +bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn) +{ + const s32 type = BPF_ATOMIC_TYPE(atomic_insn->imm); + + return type == BPF_ATOMIC_LOAD || type == BPF_ATOMIC_STORE; +} + struct bpf_prog_ops { int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr, union bpf_attr __user *uattr); diff --git a/include/linux/filter.h b/include/linux/filter.h index a3ea46281595..e36812a5b01f 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -364,6 +364,8 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn) * BPF_XOR | BPF_FETCH src_reg = atomic_fetch_xor(dst_reg + off16, src_reg); * BPF_XCHG src_reg = atomic_xchg(dst_reg + off16, src_reg) * BPF_CMPXCHG r0 = atomic_cmpxchg(dst_reg + off16, r0, src_reg) + * BPF_LOAD_ACQ dst_reg = smp_load_acquire(src_reg + off16) + * BPF_STORE_REL smp_store_release(dst_reg + off16, src_reg) */ #define BPF_ATOMIC_OP(SIZE, OP, DST, SRC, OFF) \ diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index fff6cdb8d11a..e78306e6e2be 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -51,6 +51,19 @@ #define BPF_XCHG (0xe0 | BPF_FETCH) /* atomic exchange */ #define BPF_CMPXCHG (0xf0 | BPF_FETCH) /* atomic compare-and-write */ +#define BPF_ATOMIC_LOAD 0x10 +#define BPF_ATOMIC_STORE 0x20 +#define BPF_ATOMIC_TYPE(imm) ((imm) & 0xf0) + +#define BPF_RELAXED 0x00 +#define BPF_ACQUIRE 0x01 +#define BPF_RELEASE 0x02 +#define BPF_ACQ_REL 0x03 +#define BPF_SEQ_CST 0x04 + +#define BPF_LOAD_ACQ (BPF_ATOMIC_LOAD | BPF_ACQUIRE) /* load-acquire */ +#define BPF_STORE_REL (BPF_ATOMIC_STORE | BPF_RELEASE) /* store-release */ + enum bpf_cond_pseudo_jmp { BPF_MAY_GOTO = 0, }; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index da729cbbaeb9..3f3127479a93 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1663,14 +1663,17 @@ EXPORT_SYMBOL_GPL(__bpf_call_base); INSN_3(JMP, JSET, K), \ INSN_2(JMP, JA), \ INSN_2(JMP32, JA), \ + /* Atomic operations. */ \ + INSN_3(STX, ATOMIC, B), \ + INSN_3(STX, ATOMIC, H), \ + INSN_3(STX, ATOMIC, W), \ + INSN_3(STX, ATOMIC, DW), \ /* Store instructions. */ \ /* Register based. */ \ INSN_3(STX, MEM, B), \ INSN_3(STX, MEM, H), \ INSN_3(STX, MEM, W), \ INSN_3(STX, MEM, DW), \ - INSN_3(STX, ATOMIC, W), \ - INSN_3(STX, ATOMIC, DW), \ /* Immediate based. */ \ INSN_3(ST, MEM, B), \ INSN_3(ST, MEM, H), \ @@ -2152,24 +2155,33 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) if (BPF_SIZE(insn->code) == BPF_W) \ atomic_##KOP((u32) SRC, (atomic_t *)(unsigned long) \ (DST + insn->off)); \ - else \ + else if (BPF_SIZE(insn->code) == BPF_DW) \ atomic64_##KOP((u64) SRC, (atomic64_t *)(unsigned long) \ (DST + insn->off)); \ + else \ + goto default_label; \ break; \ case BOP | BPF_FETCH: \ if (BPF_SIZE(insn->code) == BPF_W) \ SRC = (u32) atomic_fetch_##KOP( \ (u32) SRC, \ (atomic_t *)(unsigned long) (DST + insn->off)); \ - else \ + else if (BPF_SIZE(insn->code) == BPF_DW) \ SRC = (u64) atomic64_fetch_##KOP( \ (u64) SRC, \ (atomic64_t *)(unsigned long) (DST + insn->off)); \ + else \ + goto default_label; \ break; STX_ATOMIC_DW: STX_ATOMIC_W: + STX_ATOMIC_H: + STX_ATOMIC_B: switch (IMM) { + /* Atomic read-modify-write instructions support only W and DW + * size modifiers. + */ ATOMIC_ALU_OP(BPF_ADD, add) ATOMIC_ALU_OP(BPF_AND, and) ATOMIC_ALU_OP(BPF_OR, or) @@ -2181,20 +2193,59 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) SRC = (u32) atomic_xchg( (atomic_t *)(unsigned long) (DST + insn->off), (u32) SRC); - else + else if (BPF_SIZE(insn->code) == BPF_DW) SRC = (u64) atomic64_xchg( (atomic64_t *)(unsigned long) (DST + insn->off), (u64) SRC); + else + goto default_label; break; case BPF_CMPXCHG: if (BPF_SIZE(insn->code) == BPF_W) BPF_R0 = (u32) atomic_cmpxchg( (atomic_t *)(unsigned long) (DST + insn->off), (u32) BPF_R0, (u32) SRC); - else + else if (BPF_SIZE(insn->code) == BPF_DW) BPF_R0 = (u64) atomic64_cmpxchg( (atomic64_t *)(unsigned long) (DST + insn->off), (u64) BPF_R0, (u64) SRC); + else + goto default_label; + break; + /* Atomic load and store instructions support all size + * modifiers. + */ + case BPF_LOAD_ACQ: + switch (BPF_SIZE(insn->code)) { +#define LOAD_ACQUIRE(SIZEOP, SIZE) \ + case BPF_##SIZEOP: \ + DST = (SIZE)smp_load_acquire( \ + (SIZE *)(unsigned long)(SRC + insn->off)); \ + break; + LOAD_ACQUIRE(B, u8) + LOAD_ACQUIRE(H, u16) + LOAD_ACQUIRE(W, u32) + LOAD_ACQUIRE(DW, u64) +#undef LOAD_ACQUIRE + default: + goto default_label; + } + break; + case BPF_STORE_REL: + switch (BPF_SIZE(insn->code)) { +#define STORE_RELEASE(SIZEOP, SIZE) \ + case BPF_##SIZEOP: \ + smp_store_release( \ + (SIZE *)(unsigned long)(DST + insn->off), (SIZE)SRC); \ + break; + STORE_RELEASE(B, u8) + STORE_RELEASE(H, u16) + STORE_RELEASE(W, u32) + STORE_RELEASE(DW, u64) +#undef STORE_RELEASE + default: + goto default_label; + } break; default: diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 309c4aa1b026..974d172d6735 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -267,6 +267,18 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); + } else if (BPF_MODE(insn->code) == BPF_ATOMIC && + insn->imm == BPF_LOAD_ACQ) { + verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))\n", + insn->code, insn->dst_reg, + bpf_ldst_string[BPF_SIZE(insn->code) >> 3], + insn->src_reg, insn->off); + } else if (BPF_MODE(insn->code) == BPF_ATOMIC && + insn->imm == BPF_STORE_REL) { + verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)\n", + insn->code, + bpf_ldst_string[BPF_SIZE(insn->code) >> 3], + insn->dst_reg, insn->off, insn->src_reg); } else { verbose(cbs->private_data, "BUG_%02x\n", insn->code); } diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 82a5a4acf576..7ebc224bf9cb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -579,6 +579,13 @@ static bool is_cmpxchg_insn(const struct bpf_insn *insn) insn->imm == BPF_CMPXCHG; } +static bool is_atomic_load_insn(const struct bpf_insn *insn) +{ + return BPF_CLASS(insn->code) == BPF_STX && + BPF_MODE(insn->code) == BPF_ATOMIC && + BPF_ATOMIC_TYPE(insn->imm) == BPF_ATOMIC_LOAD; +} + static int __get_spi(s32 off) { return (-off - 1) / BPF_REG_SIZE; @@ -3481,7 +3488,7 @@ static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, } if (class == BPF_STX) { - /* BPF_STX (including atomic variants) has multiple source + /* BPF_STX (including atomic variants) has one or more source * operands, one of which is a ptr. Check whether the caller is * asking about it. */ @@ -4095,7 +4102,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, * dreg still needs precision before this insn */ } - } else if (class == BPF_LDX) { + } else if (class == BPF_LDX || is_atomic_load_insn(insn)) { if (!bt_is_reg_set(bt, dreg)) return 0; bt_clear_reg(bt, dreg); @@ -7686,6 +7693,32 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, return 0; } +static int check_atomic_load(struct bpf_verifier_env *env, + struct bpf_insn *insn) +{ + if (!atomic_ptr_type_ok(env, insn->src_reg, insn)) { + verbose(env, "BPF_ATOMIC loads from R%d %s is not allowed\n", + insn->src_reg, + reg_type_str(env, reg_state(env, insn->src_reg)->type)); + return -EACCES; + } + + return check_load_mem(env, insn, true, false, false, "atomic_load"); +} + +static int check_atomic_store(struct bpf_verifier_env *env, + struct bpf_insn *insn) +{ + if (!atomic_ptr_type_ok(env, insn->dst_reg, insn)) { + verbose(env, "BPF_ATOMIC stores into R%d %s is not allowed\n", + insn->dst_reg, + reg_type_str(env, reg_state(env, insn->dst_reg)->type)); + return -EACCES; + } + + return check_store_reg(env, insn, true); +} + static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) { switch (insn->imm) { @@ -7700,6 +7733,10 @@ static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) case BPF_XCHG: case BPF_CMPXCHG: return check_atomic_rmw(env, insn); + case BPF_LOAD_ACQ: + return check_atomic_load(env, insn); + case BPF_STORE_REL: + return check_atomic_store(env, insn); default: verbose(env, "BPF_ATOMIC uses invalid atomic opcode %02x\n", insn->imm); @@ -20445,7 +20482,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) insn->code == (BPF_ST | BPF_MEM | BPF_W) || insn->code == (BPF_ST | BPF_MEM | BPF_DW)) { type = BPF_WRITE; - } else if ((insn->code == (BPF_STX | BPF_ATOMIC | BPF_W) || + } else if ((insn->code == (BPF_STX | BPF_ATOMIC | BPF_B) || + insn->code == (BPF_STX | BPF_ATOMIC | BPF_H) || + insn->code == (BPF_STX | BPF_ATOMIC | BPF_W) || insn->code == (BPF_STX | BPF_ATOMIC | BPF_DW)) && env->insn_aux_data[i + delta].ptr_type == PTR_TO_ARENA) { insn->code = BPF_STX | BPF_PROBE_ATOMIC | BPF_SIZE(insn->code); diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 2acf9b336371..4a20a125eb46 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -51,6 +51,19 @@ #define BPF_XCHG (0xe0 | BPF_FETCH) /* atomic exchange */ #define BPF_CMPXCHG (0xf0 | BPF_FETCH) /* atomic compare-and-write */ +#define BPF_ATOMIC_LOAD 0x10 +#define BPF_ATOMIC_STORE 0x20 +#define BPF_ATOMIC_TYPE(imm) ((imm) & 0xf0) + +#define BPF_RELAXED 0x00 +#define BPF_ACQUIRE 0x01 +#define BPF_RELEASE 0x02 +#define BPF_ACQ_REL 0x03 +#define BPF_SEQ_CST 0x04 + +#define BPF_LOAD_ACQ (BPF_ATOMIC_LOAD | BPF_ACQUIRE) /* load-acquire */ +#define BPF_STORE_REL (BPF_ATOMIC_STORE | BPF_RELEASE) /* store-release */ + enum bpf_cond_pseudo_jmp { BPF_MAY_GOTO = 0, }; From patchwork Fri Feb 7 02:06:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2178AC02194 for ; Fri, 7 Feb 2025 02:13:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=yopLqhfx9FBzsjoQTq5VQnotD7AE589toKqt2dLQW/g=; b=BHH6CuIVRFnEjEb6a9vqqGLvZS 8OrtiHL3NHj/VN+q/R+Y7MGCcfoBvxnrmIT1hpjIvqh1ECr0yGmGefiKc/QH0yFx9zrHWBZau/otM kl5HE+aBHE0LG4ywMI3c0H64M4HCujd/oh9+ElSZ+OPnE21C2MuHcJOJK8HzHoEYInqik+EZtKzKU tT/c25jtS5TFZ8NukOLLm90cumJhje5VfcMNsy/UEP6r+V3oyDPnYJSxjyxQVwgKyeDaUtCs+lNyY XuPv8iY7AaqeaXiCtawhjiGpqZ+aMrshOD2Hi4v2HvFZDWi1N2cKmiDhGhgGDV2R+30wrmUfBZBjH 7FhcGnXw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDrx-000000082at-3ro9; Fri, 07 Feb 2025 02:13:25 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDkt-000000081YQ-3dCl for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:06:08 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2fa166cf693so3103542a91.1 for ; Thu, 06 Feb 2025 18:06:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893967; x=1739498767; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yopLqhfx9FBzsjoQTq5VQnotD7AE589toKqt2dLQW/g=; b=TG3K0QtnTMxY7eo4fSkARpQ5684ZqxU4jwwfJd4EAi5ZujllIgQ5MM0KzHMk/BXHxE SIixorYTALuffJDX8TPatirkitrJiXzf2W/t29HAP42bz72TWn2JXWxz/2NPpS7J29Pe nb350VtVnOKn3xBso0YSl7tyBqKl6J0JZM0BTzQyJc8n5otCS1gmN2SNURsVC/6KPrw/ MW21oCBsWqK8dLkjdJfrqNEkZLw5OCZvurbexjbprsgF08gOZ0qGEYAe+v3xtrN2iQ1E dP+Zd4GKMIYj8+UTpHOjXVNKV1wgFfyR/uY/az5hT1FUps/fsTrWsP1QEVbPoZETiaM/ ESmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893967; x=1739498767; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yopLqhfx9FBzsjoQTq5VQnotD7AE589toKqt2dLQW/g=; b=DgevwCx1zUlehZvAFvTAAyLoUJZh6gCAvirMZssTkEHGOVq/xwFGnI5rmvnQYiNfOr 5qUjIMGKQs9IJJ00Ajxf82z6aoCVNcSKfiz67J7Z4SniKLUnz2ryGW6ed+YIP5YG3qxJ JrTSFdZtygLk1zjsq56P+Q8J5osGdN7CuqNUL+eP6bmv5KgQFwQ8K1ORaSwYsY/VtMSP 71ks2oxmRlA3ZGJDSDmEYzmpxJJkRi0TbX7Dhn/3reloCPAxLzuGEUlZ8i4OZJQxFeay keCwcdu/ix4cAgQY45RuQGeKhN7Rj81mRwE+JUfGuVMrnlFi/aeNWiFHEgXOb78D+nKu 377w== X-Forwarded-Encrypted: i=1; AJvYcCU/CSYOKkZQ/3b91SnRLYlvrVzBpaPH0gkWLVnS7XoM8SIT8J3awj0tVSG8qNku0uIo26gapXKwezTf0hZ1PlTo@lists.infradead.org X-Gm-Message-State: AOJu0Yzg/u5fv3aV3UCepOMiIjHLnlJrLzqgRwKEtkz1MDnrWSYA5hV0 d8cA8dCNnQnxfO0WL0xPUNstfAeM8MSNpbd48DoIwyLovte12WUoGJlzA9+o6awWwXvQGT+P/Bd BU2vgB/cFAA== X-Google-Smtp-Source: AGHT+IFkXM4xOo99sOcxT31u2xnrLGGSfuSlHl8QgKpY5jPoM6In+piNqVi02IXVxH8TVZTEDBs9f2fc1XsrBg== X-Received: from pjbpt7.prod.google.com ([2002:a17:90b:3d07:b0:2fa:b84:b308]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3dc3:b0:2ee:8ea0:6b9c with SMTP id 98e67ed59e1d1-2fa24064223mr2524108a91.12.1738893966755; Thu, 06 Feb 2025 18:06:06 -0800 (PST) Date: Fri, 7 Feb 2025 02:06:00 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: Subject: [PATCH bpf-next v2 5/9] arm64: insn: Add BIT(23) to {load,store}_ex's mask From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180607_902372_A2606392 X-CRM114-Status: GOOD ( 11.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We are planning to add load-acquire (LDAR{,B,H}) and store-release (STLR{,B,H}) instructions to insn.{c,h}; add BIT(23) to mask of load_ex and store_ex to prevent aarch64_insn_is_{load,store}_ex() from returning false-positives for load-acquire and store-release instructions. Reference: Arm Architecture Reference Manual (ARM DDI 0487K.a, ID032224), * C6.2.228 LDXR * C6.2.165 LDAXR * C6.2.161 LDAR * C6.2.393 STXR * C6.2.360 STLXR * C6.2.353 STLR Signed-off-by: Peilin Ye --- arch/arm64/include/asm/insn.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index e390c432f546..2d8316b3abaf 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -351,8 +351,8 @@ __AARCH64_INSN_FUNCS(ldr_imm, 0x3FC00000, 0x39400000) __AARCH64_INSN_FUNCS(ldr_lit, 0xBF000000, 0x18000000) __AARCH64_INSN_FUNCS(ldrsw_lit, 0xFF000000, 0x98000000) __AARCH64_INSN_FUNCS(exclusive, 0x3F800000, 0x08000000) -__AARCH64_INSN_FUNCS(load_ex, 0x3F400000, 0x08400000) -__AARCH64_INSN_FUNCS(store_ex, 0x3F400000, 0x08000000) +__AARCH64_INSN_FUNCS(load_ex, 0x3FC00000, 0x08400000) +__AARCH64_INSN_FUNCS(store_ex, 0x3FC00000, 0x08000000) __AARCH64_INSN_FUNCS(mops, 0x3B200C00, 0x19000400) __AARCH64_INSN_FUNCS(stp, 0x7FC00000, 0x29000000) __AARCH64_INSN_FUNCS(ldp, 0x7FC00000, 0x29400000) From patchwork Fri Feb 7 02:06:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964242 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CBE71C02194 for ; Fri, 7 Feb 2025 02:15:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tZVryglY4SpsTL4//l81oNX0vaTjg5yUDPJMzMkBO6s=; b=TVnthru94csWSIhAszJFI2BOYV X1ewILUpDTttfZpqkM0nWvquTL36URwl3W3haNIx3By7Dvn9c9LGlyq55qsNlsFBCgBBBMh1kLmkf W9ScX5XOqDMSFGHVTAfNO2DSPa+LPI4CGGGiT+iQDMzNhVjxmOq4qj0eJ3Gl0nIKt6y1scDFpDN3D 1s88NKZRSFA/E1OIEJ0iseFBCbExWNa3L+GHzKa1scVclLoPYiarg7HqT/9L5fAYN69lD4fObQwB4 NNIQyKv24ZDtZdXU5tw5xhtmmPoNLcaM4jEQ+Or3r75uoARbwJGX4Ej0dEHTAvbSHuYAxEdsB/c1u bwOzSlNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDtJ-000000082in-2OOX; Fri, 07 Feb 2025 02:14:49 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDl2-000000081ap-1a3c for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:06:17 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2f9fdc4dc84so3252273a91.2 for ; Thu, 06 Feb 2025 18:06:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893975; x=1739498775; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tZVryglY4SpsTL4//l81oNX0vaTjg5yUDPJMzMkBO6s=; b=ti0FOSkMtDL50JnYCRM4FGrqDfOXbHd7Qg4y+RK4+REO7aP3EIg/7H5B6HSagbjMkN HN4lvOk5KtiWafe940o0MPoRrqbrPssYKJKe3TBFufVc7Q2OJa6fER90759XMv+sp9+D xu+lFP5iXp8Js396gswJUWH00nOKTfWRT8iO19JSKEd6ZyVDxelZT9iRC5ifrd3oIwgx oyvHNdJd7I/6NNWetdA0WX9yL9KgGXYCvp17jqItyvF2XjilDlF7dlZx+Z4ExF0jWhlE ae/bsHZrcxY0/i0HVzqW3zXiAupeWoBEpym1Lo6hPjh9tFx0RjRKD9zYMaUoPPv7Oi1B EfvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893975; x=1739498775; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tZVryglY4SpsTL4//l81oNX0vaTjg5yUDPJMzMkBO6s=; b=E7arc7eHLTttPra5wVHQYYal+CVahZQFMgh3U91VA7ELxTKvBMU/o4500NK1c6MKeq hhwWNRDRKZ/V9Wkx5vhXZzF89qUkffDmMFtsmkNLTR8OLmK2aeuyUe0guFurmtS3MWzB umxJ8TD64Tgf67EMjI4fhgoQXQ9EmRyEKYd7qzn5/kHSgDFMCXvniiBBgC9/rC1s20BH ZcUoPKJZ57ekxA6RN4fGLfQINmpIbieG8A9Qmry4Ae0DqaT1n7dL4D4c/nlveTaRFkNr 0GUhxeoFWxmDbHGz8i3V22DwZ/C+sXlO86fPYLN30+okC/XLn4gxHhzfOrPB0CAN6dEv q+6g== X-Forwarded-Encrypted: i=1; AJvYcCXvOrGSOnQIqmXC33Bz/1NKMY/FSRetItWbC7j3TAwhezgnsTGAIGT0tk+8qvXy3YZrU764TsoQ0qq8e4BDRjHg@lists.infradead.org X-Gm-Message-State: AOJu0Yzbn0h6I0wNmn147iOrqPwAabX/LbRHnWka539v4kb5GHOdia7L 17epJXimYZ2EIVkUwKdxMK7DqigmBBu5QrmpEm/pAtAeBiJy/33JW5g14evL2/dqelGWYEd7MBX NW8V3rSII4Q== X-Google-Smtp-Source: AGHT+IGIQlJHYn4lDP+h2wWyOKBebCVWMqyOh64SVoatr4LfPmXKB3GVTfYlGBX4UVXE3HcAasyz3VEj0/8t8w== X-Received: from pjyf6.prod.google.com ([2002:a17:90a:ec86:b0:2fa:15aa:4d1e]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:184d:b0:2ee:d824:b559 with SMTP id 98e67ed59e1d1-2fa243f058cmr2144082a91.28.1738893974955; Thu, 06 Feb 2025 18:06:14 -0800 (PST) Date: Fri, 7 Feb 2025 02:06:10 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <8d599a837068e7a526200bbe7c6355ad4c5fc028.1738888641.git.yepeilin@google.com> Subject: [PATCH bpf-next v2 6/9] arm64: insn: Add load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180616_414313_9C856CB7 X-CRM114-Status: GOOD ( 10.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add load-acquire ("load_acq", LDAR{,B,H}) and store-release ("store_rel", STLR{,B,H}) instructions. Breakdown of encoding: size L (Rs) o0 (Rt2) Rn Rt mask (0x3fdffc00): 00 111111 1 1 0 11111 1 11111 00000 00000 value, load_acq (0x08dffc00): 00 001000 1 1 0 11111 1 11111 00000 00000 value, store_rel (0x089ffc00): 00 001000 1 0 0 11111 1 11111 00000 00000 As suggested by Xu [1], include all Should-Be-One (SBO) bits ("Rs" and "Rt2" fields) in the "mask" and "value" numbers. It is worth noting that we are adding the "no offset" variant of STLR instead of the "pre-index" variant, which has a different encoding. Reference: Arm Architecture Reference Manual (ARM DDI 0487K.a, ID032224), * C6.2.161 LDAR * C6.2.353 STLR [1] https://lore.kernel.org/bpf/4e6641ce-3f1e-4251-8daf-4dd4b77d08c4@huaweicloud.com/ Signed-off-by: Peilin Ye --- arch/arm64/include/asm/insn.h | 8 ++++++++ arch/arm64/lib/insn.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h index 2d8316b3abaf..39577f1d079a 100644 --- a/arch/arm64/include/asm/insn.h +++ b/arch/arm64/include/asm/insn.h @@ -188,8 +188,10 @@ enum aarch64_insn_ldst_type { AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX, AARCH64_INSN_LDST_LOAD_PAIR_POST_INDEX, AARCH64_INSN_LDST_STORE_PAIR_POST_INDEX, + AARCH64_INSN_LDST_LOAD_ACQ, AARCH64_INSN_LDST_LOAD_EX, AARCH64_INSN_LDST_LOAD_ACQ_EX, + AARCH64_INSN_LDST_STORE_REL, AARCH64_INSN_LDST_STORE_EX, AARCH64_INSN_LDST_STORE_REL_EX, AARCH64_INSN_LDST_SIGNED_LOAD_IMM_OFFSET, @@ -351,6 +353,8 @@ __AARCH64_INSN_FUNCS(ldr_imm, 0x3FC00000, 0x39400000) __AARCH64_INSN_FUNCS(ldr_lit, 0xBF000000, 0x18000000) __AARCH64_INSN_FUNCS(ldrsw_lit, 0xFF000000, 0x98000000) __AARCH64_INSN_FUNCS(exclusive, 0x3F800000, 0x08000000) +__AARCH64_INSN_FUNCS(load_acq, 0x3FDFFC00, 0x08DFFC00) +__AARCH64_INSN_FUNCS(store_rel, 0x3FDFFC00, 0x089FFC00) __AARCH64_INSN_FUNCS(load_ex, 0x3FC00000, 0x08400000) __AARCH64_INSN_FUNCS(store_ex, 0x3FC00000, 0x08000000) __AARCH64_INSN_FUNCS(mops, 0x3B200C00, 0x19000400) @@ -602,6 +606,10 @@ u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1, int offset, enum aarch64_insn_variant variant, enum aarch64_insn_ldst_type type); +u32 aarch64_insn_gen_load_acq_store_rel(enum aarch64_insn_register reg, + enum aarch64_insn_register base, + enum aarch64_insn_size_type size, + enum aarch64_insn_ldst_type type); u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg, enum aarch64_insn_register base, enum aarch64_insn_register state, diff --git a/arch/arm64/lib/insn.c b/arch/arm64/lib/insn.c index b008a9b46a7f..9bef696e2230 100644 --- a/arch/arm64/lib/insn.c +++ b/arch/arm64/lib/insn.c @@ -540,6 +540,35 @@ u32 aarch64_insn_gen_load_store_pair(enum aarch64_insn_register reg1, offset >> shift); } +u32 aarch64_insn_gen_load_acq_store_rel(enum aarch64_insn_register reg, + enum aarch64_insn_register base, + enum aarch64_insn_size_type size, + enum aarch64_insn_ldst_type type) +{ + u32 insn; + + switch (type) { + case AARCH64_INSN_LDST_LOAD_ACQ: + insn = aarch64_insn_get_load_acq_value(); + break; + case AARCH64_INSN_LDST_STORE_REL: + insn = aarch64_insn_get_store_rel_value(); + break; + default: + pr_err("%s: unknown load-acquire/store-release encoding %d\n", + __func__, type); + return AARCH64_BREAK_FAULT; + } + + insn = aarch64_insn_encode_ldst_size(size, insn); + + insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RT, insn, + reg); + + return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, + base); +} + u32 aarch64_insn_gen_load_store_ex(enum aarch64_insn_register reg, enum aarch64_insn_register base, enum aarch64_insn_register state, From patchwork Fri Feb 7 02:06:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964243 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09023C02194 for ; Fri, 7 Feb 2025 02:16:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=3/uv2akh8MDqd9n19hYJv/xvE4Iur0KMmE3+KGKTk/Y=; b=o5B0Uo+3PpO1zEnuW+i2J0v3Lg xIaPNnPXzQ3iXhT/sKRVDlAWz9Yx2eA8h9QotNp/IVYeVrz1/XYXM8JFh6E66hpnWrzYbLps9zjxp F+fmbzJyMYJnXOny0QVBsmmk0RVhNYMAq/ocbnm/dCVw0YkDD84vCHHEg2npiPXgYOgfWJDHFqHtD gUEUFhxMtZvQiKPhIa7yKCo1Hq0hrab7vj+xaqqSmI8QHsO/Qzmez3HQvD+L9HzRH7opU6MtxHQV4 4Z4F6XZT8yXDtTypgMw2DztbwY6JxEf5KtV/ETb1l/tyyf24hEscC6PyvQmsiw1xlp+KrTFWPERpJ d33LRdmw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDuf-00000008301-0oKC; Fri, 07 Feb 2025 02:16:13 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDlD-000000081dV-3Mv7 for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:06:28 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-2f9f2bf2aaeso3265797a91.1 for ; Thu, 06 Feb 2025 18:06:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893986; x=1739498786; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3/uv2akh8MDqd9n19hYJv/xvE4Iur0KMmE3+KGKTk/Y=; b=vlCyVqCpk0p5+ObSN17Zi5W64I3/hTlYbMz9B0830q139/6ruzcKCO7S7dfCyivsDE Xy/uJDkfc+4s7kUx/2tZrjAXSbX1sJgZYe/kug00LuyM4j8Jk/04o86wQyd8duXpGSIH LSjUAGDM7/zL+F3I7vhDaHcJrJHoeaoait9DnlpFujhqKDfgYVS5TZZiFOKVGoqaC3Z6 iRtOPljBxlM88Y4DpspbEQXGQrGTBCSaKR3doH9jSqtzkXmBBGnCeTPy5BP+LwsWadNk i1ev3MhCcBchgZs7mp8fUwcURRbpxu1iC1XFlPeNWd+c/UNpGijn2Y5yrvKc4i10+GPU OSkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893986; x=1739498786; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3/uv2akh8MDqd9n19hYJv/xvE4Iur0KMmE3+KGKTk/Y=; b=m4g/CrGFmaABoKQqVZatv0KSIbTUDMCFjRVMf7oqhZ24CaY1ke+h6VrHMlDGafLbEM RD7IymptAXl8XtvBy+tetkzLSf8PP84dVkSa5WKqu32wImrSphMfZ9jo+6WhcxtTItqz sc17aMmdD6tM9T74YT/2JlUAE0L57Ks6z/oMhgcKv3XHS+2rxP3R57yqK5HV3U0jwXQY 5chGhAW0OI/76R342V4N56IJm8v+BatYl9Zsm7foKFwLbjikyDtq3ZF/pTaY51ZO2cRF tgCJr47egKS/LMMQ3MwZFgRGlXn6t8QzgWLiUkbkp7XbiZhG0vbvbHn92O3IYO58iWcU WykQ== X-Forwarded-Encrypted: i=1; AJvYcCXaG49ESDuVZfrOZLjS04u3J+OvIjx8O4C1ej3HBGBrr4qoYqCnHVEseT/MI+yUhJF5rpjdXa/pzRu6h0fnp99T@lists.infradead.org X-Gm-Message-State: AOJu0YxPHzqT98eOyXnDGQDqqktkvxK32+EpsxzzukyC3pNdbWdoKPCA 5LbLrV2EAu4ca91N4691l0C8VsaL7+BzDhQvLxK0HltiL/PslYSMAUb3VH9CQccEsAc1y+DFyzu BpoEUaCuiUQ== X-Google-Smtp-Source: AGHT+IHmd7jXzZpPbORG/2teAdCv8IQKKJx1keqS0VzPbP7gZVccyL5ZzCdiANnZVKcUIXtFTxwxZpQFooUwFw== X-Received: from pfvx11.prod.google.com ([2002:a05:6a00:270b:b0:725:dec7:dd47]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:230c:b0:725:e4b9:a600 with SMTP id d2e1a72fcca58-7305d4f0412mr2817045b3a.16.1738893986567; Thu, 06 Feb 2025 18:06:26 -0800 (PST) Date: Fri, 7 Feb 2025 02:06:19 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <73f55267778db55ba9e854f0e3bef24c4bbba2fb.1738888641.git.yepeilin@google.com> Subject: [PATCH bpf-next v2 7/9] bpf, arm64: Support load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180627_840422_1D98E476 X-CRM114-Status: GOOD ( 15.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Support BPF load-acquire (BPF_LOAD_ACQ) and store-release (BPF_STORE_REL) instructions in the arm64 JIT compiler. For example: db 10 00 00 11 00 00 00 r0 = load_acquire((u64 *)(r1 + 0x0)) 95 00 00 00 00 00 00 00 exit opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX imm (0x00000011): BPF_LOAD_ACQ The JIT compiler would emit an LDAR instruction for the above, e.g.: ldar x7, [x0] Similarly, consider the following 16-bit store-release: cb 21 00 00 22 00 00 00 store_release((u16 *)(r1 + 0x0), w2) 95 00 00 00 00 00 00 00 exit opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX imm (0x00000022): BPF_ATOMIC_STORE | BPF_RELEASE An STLRH instruction would be emitted, e.g.: stlrh w1, [x0] For a complete mapping: load-acquire 8-bit LDARB (BPF_LOAD_ACQ) 16-bit LDARH 32-bit LDAR (32-bit) 64-bit LDAR (64-bit) store-release 8-bit STLRB (BPF_STORE_REL) 16-bit STLRH 32-bit STLR (32-bit) 64-bit STLR (64-bit) Arena accesses are supported. bpf_jit_supports_insn(..., /*in_arena=*/true) always returns true for BPF_LOAD_ACQ and BPF_STORE_REL instructions, as they don't depend on ARM64_HAS_LSE_ATOMICS. Signed-off-by: Peilin Ye --- arch/arm64/net/bpf_jit.h | 20 ++++++++ arch/arm64/net/bpf_jit_comp.c | 91 ++++++++++++++++++++++++++++++++--- 2 files changed, 105 insertions(+), 6 deletions(-) diff --git a/arch/arm64/net/bpf_jit.h b/arch/arm64/net/bpf_jit.h index b22ab2f97a30..a3b0e693a125 100644 --- a/arch/arm64/net/bpf_jit.h +++ b/arch/arm64/net/bpf_jit.h @@ -119,6 +119,26 @@ aarch64_insn_gen_load_store_ex(Rt, Rn, Rs, A64_SIZE(sf), \ AARCH64_INSN_LDST_STORE_REL_EX) +/* Load-acquire & store-release */ +#define A64_LDAR(Rt, Rn, size) \ + aarch64_insn_gen_load_acq_store_rel(Rt, Rn, AARCH64_INSN_SIZE_##size, \ + AARCH64_INSN_LDST_LOAD_ACQ) +#define A64_STLR(Rt, Rn, size) \ + aarch64_insn_gen_load_acq_store_rel(Rt, Rn, AARCH64_INSN_SIZE_##size, \ + AARCH64_INSN_LDST_STORE_REL) + +/* Rt = [Rn] (load acquire) */ +#define A64_LDARB(Wt, Xn) A64_LDAR(Wt, Xn, 8) +#define A64_LDARH(Wt, Xn) A64_LDAR(Wt, Xn, 16) +#define A64_LDAR32(Wt, Xn) A64_LDAR(Wt, Xn, 32) +#define A64_LDAR64(Xt, Xn) A64_LDAR(Xt, Xn, 64) + +/* [Rn] = Rt (store release) */ +#define A64_STLRB(Wt, Xn) A64_STLR(Wt, Xn, 8) +#define A64_STLRH(Wt, Xn) A64_STLR(Wt, Xn, 16) +#define A64_STLR32(Wt, Xn) A64_STLR(Wt, Xn, 32) +#define A64_STLR64(Xt, Xn) A64_STLR(Xt, Xn, 64) + /* * LSE atomics * diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 8c3b47d9e441..c439df0233d1 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -647,6 +647,82 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) return 0; } +static int emit_atomic_load_store(const struct bpf_insn *insn, + struct jit_ctx *ctx) +{ + const s32 type = BPF_ATOMIC_TYPE(insn->imm); + const s16 off = insn->off; + const u8 code = insn->code; + const bool arena = BPF_MODE(code) == BPF_PROBE_ATOMIC; + const u8 arena_vm_base = bpf2a64[ARENA_VM_START]; + const u8 dst = bpf2a64[insn->dst_reg]; + const u8 src = bpf2a64[insn->src_reg]; + const u8 tmp = bpf2a64[TMP_REG_1]; + u8 reg; + + switch (type) { + case BPF_ATOMIC_LOAD: + reg = src; + break; + case BPF_ATOMIC_STORE: + reg = dst; + break; + default: + pr_err_once("unknown atomic load/store op type %02x\n", type); + return -EINVAL; + } + + if (off) { + emit_a64_add_i(1, tmp, reg, tmp, off, ctx); + reg = tmp; + } + if (arena) { + emit(A64_ADD(1, tmp, reg, arena_vm_base), ctx); + reg = tmp; + } + + switch (insn->imm) { + case BPF_LOAD_ACQ: + switch (BPF_SIZE(code)) { + case BPF_B: + emit(A64_LDARB(dst, reg), ctx); + break; + case BPF_H: + emit(A64_LDARH(dst, reg), ctx); + break; + case BPF_W: + emit(A64_LDAR32(dst, reg), ctx); + break; + case BPF_DW: + emit(A64_LDAR64(dst, reg), ctx); + break; + } + break; + case BPF_STORE_REL: + switch (BPF_SIZE(code)) { + case BPF_B: + emit(A64_STLRB(src, reg), ctx); + break; + case BPF_H: + emit(A64_STLRH(src, reg), ctx); + break; + case BPF_W: + emit(A64_STLR32(src, reg), ctx); + break; + case BPF_DW: + emit(A64_STLR64(src, reg), ctx); + break; + } + break; + default: + pr_err_once("unknown atomic load/store op code %02x\n", + insn->imm); + return -EINVAL; + } + + return 0; +} + #ifdef CONFIG_ARM64_LSE_ATOMICS static int emit_lse_atomic(const struct bpf_insn *insn, struct jit_ctx *ctx) { @@ -1641,11 +1717,17 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, return ret; break; + case BPF_STX | BPF_ATOMIC | BPF_B: + case BPF_STX | BPF_ATOMIC | BPF_H: case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_B: + case BPF_STX | BPF_PROBE_ATOMIC | BPF_H: case BPF_STX | BPF_PROBE_ATOMIC | BPF_W: case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW: - if (cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) + if (bpf_atomic_is_load_store(insn)) + ret = emit_atomic_load_store(insn, ctx); + else if (cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) ret = emit_lse_atomic(insn, ctx); else ret = emit_ll_sc_atomic(insn, ctx); @@ -2667,13 +2749,10 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) if (!in_arena) return true; switch (insn->code) { - case BPF_STX | BPF_ATOMIC | BPF_B: - case BPF_STX | BPF_ATOMIC | BPF_H: case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: - if (bpf_atomic_is_load_store(insn)) - return false; - if (!cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) + if (!bpf_atomic_is_load_store(insn) && + !cpus_have_cap(ARM64_HAS_LSE_ATOMICS)) return false; } return true; From patchwork Fri Feb 7 02:06:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964244 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5916C02194 for ; Fri, 7 Feb 2025 02:17:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=qDBuIJWrKAPdCSK+NN6+KD/nwtGtMidK+XAGCC08g9k=; b=eKSwufpQGdQLIrBymbX2qsFlph qS3tGNe6fRrxyIGI+NYCHndGXtjyzwnTDPgn2sozU1hbe4GlH0xuJDFwBbMSrNisWq8q8Uzm1jH2R EamSHsmncXgAnSTDBcHLD8fWY71cDMCsMO1vPkrj7wM93ek4DVHH+Wi7zjmtGyXRzsNFCbSws124k /vEzq18+hNces188XNG8INjP5OVFrI3/M0s9LvLAZPohCDzwUt7a3jhgzNk86XR3PuEs3Cw1msijz /0Du8Es0/2+2/P3NpZ7gl9hx0my/tixb9Jse6D/q6zd9eGMA2zLC4L1kOuDUschl7N0BhYHrTVDsy yKq+laog==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDw0-000000083An-3ntf; Fri, 07 Feb 2025 02:17:36 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDlK-000000081es-1cup for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:06:35 +0000 Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-2f9fbe02121so3222589a91.0 for ; Thu, 06 Feb 2025 18:06:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738893993; x=1739498793; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qDBuIJWrKAPdCSK+NN6+KD/nwtGtMidK+XAGCC08g9k=; b=Avaf4dzTHAAw30zQtVIxu7MjaQbSd+8cYQ0L76MD7WWx8r35crkqiBQPIj4BsuarAh gtn+tMcs876pVtkCnDwKrNlhlW2OydGOB38OjXGe6ocoO73SRgcCgyRZNS7wOg3Qpjpu +tBAVL9JjQLNr5rFwfF/4clF5/4nF76hMzaYt6d13Yc0JlcvFurZXv6AAMv14vmkcKcn vh1SbQqX9XE5wNUfqBawKs1lWvgNxsdNqa1mXzHsGD7ugs9hTLAwEykJURB6CJeJ82fW Q4tQ+m1EIYdrMuz7pZnWW1iJ5T9g4JFtEny4N7XzOjE1p6n+e+TMjzUypxAvcB0Rn50O I3YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738893993; x=1739498793; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qDBuIJWrKAPdCSK+NN6+KD/nwtGtMidK+XAGCC08g9k=; b=wojnJmhJ66CeYhTiODYyqYXRfATPhMKKVT1WA3SFnbyAhZSDylBqWDctpxHuUfCioE 1ce5NehBRGhUgIhK2XHFjPKs009UOszUhfKTMyoe5CTktN8wF+KVHH9Dot6WzgIvMuz4 9Bu7LlCZYPPJ63U94qsLZwNEPuodAFhKES2eEeNMOBqht41dr6B4LrJ9iOp814NGdewV HU0UN0mBXlxd1GfavpulTRztv9+QbatC0qNenueuwJlFtK5RvhotCOOXfXSt8j1qHm39 22G//EyQmkCAaM8yOjw2G2JM3Yx2mjCQYxib0S1fGSSenN80H3ZbdH+SFwYYqOsrajJV /dhg== X-Forwarded-Encrypted: i=1; AJvYcCUg0tiiMGw02i4+VpfeFoeSkf8yYcNyyDlGr7IEE0pffdvAClGngwrUy0tmpE358cSo9tqcKXMcl2F29uNl0pJ1@lists.infradead.org X-Gm-Message-State: AOJu0YwaF3sBgeh7bPyksfqy1Q+FucgqkD6PhXzEKEBn2VFbU3UHcEJw 9USrhUW8qzjbUYXT34qfVfHM10FwoVp2MG+UjVuie5vnaMUrBWhz9/z2mpFGBVJViM48QqjIOm4 CbD9BXd974g== X-Google-Smtp-Source: AGHT+IG/qi8F1L5u/c6+Y7D+xHDSNV7nJ5W3zIK50GxT1EeANRxL5xpKqmsfwfsoi2gRh9jERt4m4Z+AGaRbdQ== X-Received: from pjyf5.prod.google.com ([2002:a17:90a:ec85:b0:2fa:1481:81f5]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:23c7:b0:2ee:5bc9:75c7 with SMTP id 98e67ed59e1d1-2fa23f4262bmr1864444a91.5.1738893993086; Thu, 06 Feb 2025 18:06:33 -0800 (PST) Date: Fri, 7 Feb 2025 02:06:29 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <3ac854ac5cc62e78fadd2a7f1af9087ec3fc7a9c.1738888641.git.yepeilin@google.com> Subject: [PATCH bpf-next v2 8/9] selftests/bpf: Add selftests for load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180634_442705_51A10E55 X-CRM114-Status: GOOD ( 20.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add several ./test_progs tests: - arena_atomics/load_acquire - arena_atomics/store_release - verifier_load_acquire/* - verifier_store_release/* - verifier_precision/bpf_load_acquire - verifier_precision/bpf_store_release The last two tests are added to check if backtrack_insn() handles the new instructions correctly. Additionally, the last test also makes sure that the verifier "remembers" the value (in src_reg) we store-release into e.g. a stack slot. For example, if we take a look at the test program: #0: r1 = 8; /* store_release((u64 *)(r10 - 8), r1); */ #1: .8byte %[store_release]; #2: r1 = *(u64 *)(r10 - 8); #3: r2 = r10; #4: r2 += r1; #5: r0 = 0; #6: exit; At #1, if the verifier doesn't remember that we wrote 8 to the stack, then later at #4 we would be adding an unbounded scalar value to the stack pointer, which would cause the program to be rejected: VERIFIER LOG: ============= ... math between fp pointer and register with unbounded min value is not allowed All new tests depend on #ifdef ENABLE_ATOMICS_TESTS. Currently they only run for arm64. Signed-off-by: Peilin Ye Acked-by: Eduard Zingerman --- .../selftests/bpf/prog_tests/arena_atomics.c | 50 ++++ .../selftests/bpf/prog_tests/verifier.c | 4 + .../selftests/bpf/progs/arena_atomics.c | 88 ++++++ .../bpf/progs/verifier_load_acquire.c | 190 +++++++++++++ .../selftests/bpf/progs/verifier_precision.c | 47 ++++ .../bpf/progs/verifier_store_release.c | 262 ++++++++++++++++++ 6 files changed, 641 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_load_acquire.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_store_release.c diff --git a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c index 26e7c06c6cb4..d714af9af50c 100644 --- a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c +++ b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c @@ -162,6 +162,52 @@ static void test_uaf(struct arena_atomics *skel) ASSERT_EQ(skel->arena->uaf_recovery_fails, 0, "uaf_recovery_fails"); } +static void test_load_acquire(struct arena_atomics *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + /* No need to attach it, just run it directly */ + prog_fd = bpf_program__fd(skel->progs.load_acquire); + err = bpf_prog_test_run_opts(prog_fd, &topts); + if (!ASSERT_OK(err, "test_run_opts err")) + return; + if (!ASSERT_OK(topts.retval, "test_run_opts retval")) + return; + + ASSERT_EQ(skel->arena->load_acquire8_result, 0x12, + "load_acquire8_result"); + ASSERT_EQ(skel->arena->load_acquire16_result, 0x1234, + "load_acquire16_result"); + ASSERT_EQ(skel->arena->load_acquire32_result, 0x12345678, + "load_acquire32_result"); + ASSERT_EQ(skel->arena->load_acquire64_result, 0x1234567890abcdef, + "load_acquire64_result"); +} + +static void test_store_release(struct arena_atomics *skel) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + int err, prog_fd; + + /* No need to attach it, just run it directly */ + prog_fd = bpf_program__fd(skel->progs.store_release); + err = bpf_prog_test_run_opts(prog_fd, &topts); + if (!ASSERT_OK(err, "test_run_opts err")) + return; + if (!ASSERT_OK(topts.retval, "test_run_opts retval")) + return; + + ASSERT_EQ(skel->arena->store_release8_result, 0x12, + "store_release8_result"); + ASSERT_EQ(skel->arena->store_release16_result, 0x1234, + "store_release16_result"); + ASSERT_EQ(skel->arena->store_release32_result, 0x12345678, + "store_release32_result"); + ASSERT_EQ(skel->arena->store_release64_result, 0x1234567890abcdef, + "store_release64_result"); +} + void test_arena_atomics(void) { struct arena_atomics *skel; @@ -198,6 +244,10 @@ void test_arena_atomics(void) test_xchg(skel); if (test__start_subtest("uaf")) test_uaf(skel); + if (test__start_subtest("load_acquire")) + test_load_acquire(skel); + if (test__start_subtest("store_release")) + test_store_release(skel); cleanup: arena_atomics__destroy(skel); diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c index 8a0e1ff8a2dc..8bdad4167cf5 100644 --- a/tools/testing/selftests/bpf/prog_tests/verifier.c +++ b/tools/testing/selftests/bpf/prog_tests/verifier.c @@ -45,6 +45,7 @@ #include "verifier_ldsx.skel.h" #include "verifier_leak_ptr.skel.h" #include "verifier_linked_scalars.skel.h" +#include "verifier_load_acquire.skel.h" #include "verifier_loops1.skel.h" #include "verifier_lwt.skel.h" #include "verifier_map_in_map.skel.h" @@ -80,6 +81,7 @@ #include "verifier_spill_fill.skel.h" #include "verifier_spin_lock.skel.h" #include "verifier_stack_ptr.skel.h" +#include "verifier_store_release.skel.h" #include "verifier_subprog_precision.skel.h" #include "verifier_subreg.skel.h" #include "verifier_tailcall_jit.skel.h" @@ -173,6 +175,7 @@ void test_verifier_int_ptr(void) { RUN(verifier_int_ptr); } void test_verifier_iterating_callbacks(void) { RUN(verifier_iterating_callbacks); } void test_verifier_jeq_infer_not_null(void) { RUN(verifier_jeq_infer_not_null); } void test_verifier_jit_convergence(void) { RUN(verifier_jit_convergence); } +void test_verifier_load_acquire(void) { RUN(verifier_load_acquire); } void test_verifier_ld_ind(void) { RUN(verifier_ld_ind); } void test_verifier_ldsx(void) { RUN(verifier_ldsx); } void test_verifier_leak_ptr(void) { RUN(verifier_leak_ptr); } @@ -211,6 +214,7 @@ void test_verifier_sockmap_mutate(void) { RUN(verifier_sockmap_mutate); } void test_verifier_spill_fill(void) { RUN(verifier_spill_fill); } void test_verifier_spin_lock(void) { RUN(verifier_spin_lock); } void test_verifier_stack_ptr(void) { RUN(verifier_stack_ptr); } +void test_verifier_store_release(void) { RUN(verifier_store_release); } void test_verifier_subprog_precision(void) { RUN(verifier_subprog_precision); } void test_verifier_subreg(void) { RUN(verifier_subreg); } void test_verifier_tailcall_jit(void) { RUN(verifier_tailcall_jit); } diff --git a/tools/testing/selftests/bpf/progs/arena_atomics.c b/tools/testing/selftests/bpf/progs/arena_atomics.c index 40dd57fca5cc..62189b428fe3 100644 --- a/tools/testing/selftests/bpf/progs/arena_atomics.c +++ b/tools/testing/selftests/bpf/progs/arena_atomics.c @@ -6,6 +6,8 @@ #include #include #include "bpf_arena_common.h" +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" struct { __uint(type, BPF_MAP_TYPE_ARENA); @@ -274,4 +276,90 @@ int uaf(const void *ctx) return 0; } +__u8 __arena_global load_acquire8_value = 0x12; +__u16 __arena_global load_acquire16_value = 0x1234; +__u32 __arena_global load_acquire32_value = 0x12345678; +__u64 __arena_global load_acquire64_value = 0x1234567890abcdef; + +__u8 __arena_global load_acquire8_result = 0x12; +__u16 __arena_global load_acquire16_result = 0x1234; +__u32 __arena_global load_acquire32_result = 0x12345678; +__u64 __arena_global load_acquire64_result = 0x1234567890abcdef; + +SEC("raw_tp/sys_enter") +int load_acquire(const void *ctx) +{ +#if defined(ENABLE_ATOMICS_TESTS) && defined(__TARGET_ARCH_arm64) + load_acquire8_result = 0; + load_acquire16_result = 0; + load_acquire32_result = 0; + load_acquire64_result = 0; + +#define LOAD_ACQUIRE_ARENA(SIZEOP, SIZE, SRC, DST) \ + { asm volatile ( \ + "r1 = %[" #SRC "] ll;" \ + "r1 = addr_space_cast(r1, 0x0, 0x1);" \ + ".8byte %[load_acquire_insn];" \ + "r3 = %[" #DST "] ll;" \ + "r3 = addr_space_cast(r3, 0x0, 0x1);" \ + "*(" #SIZE " *)(r3 + 0) = r2;" \ + : \ + : __imm_addr(SRC), \ + __imm_insn(load_acquire_insn, \ + BPF_ATOMIC_OP(BPF_##SIZEOP, BPF_LOAD_ACQ, \ + BPF_REG_2, BPF_REG_1, 0)), \ + __imm_addr(DST) \ + : __clobber_all); } \ + + LOAD_ACQUIRE_ARENA(B, u8, load_acquire8_value, load_acquire8_result) + LOAD_ACQUIRE_ARENA(H, u16, load_acquire16_value, + load_acquire16_result) + LOAD_ACQUIRE_ARENA(W, u32, load_acquire32_value, + load_acquire32_result) + LOAD_ACQUIRE_ARENA(DW, u64, load_acquire64_value, + load_acquire64_result) +#undef LOAD_ACQUIRE_ARENA +#endif + + return 0; +} + +__u8 __arena_global store_release8_result = 0x12; +__u16 __arena_global store_release16_result = 0x1234; +__u32 __arena_global store_release32_result = 0x12345678; +__u64 __arena_global store_release64_result = 0x1234567890abcdef; + +SEC("raw_tp/sys_enter") +int store_release(const void *ctx) +{ +#if defined(ENABLE_ATOMICS_TESTS) && defined(__TARGET_ARCH_arm64) + store_release8_result = 0; + store_release16_result = 0; + store_release32_result = 0; + store_release64_result = 0; + +#define STORE_RELEASE_ARENA(SIZEOP, DST, VAL) \ + { asm volatile ( \ + "r1 = " VAL ";" \ + "r2 = %[" #DST "] ll;" \ + "r2 = addr_space_cast(r2, 0x0, 0x1);" \ + ".8byte %[store_release_insn];" \ + : \ + : __imm_addr(DST), \ + __imm_insn(store_release_insn, \ + BPF_ATOMIC_OP(BPF_##SIZEOP, BPF_STORE_REL, \ + BPF_REG_2, BPF_REG_1, 0)) \ + : __clobber_all); } \ + + STORE_RELEASE_ARENA(B, store_release8_result, "0x12") + STORE_RELEASE_ARENA(H, store_release16_result, "0x1234") + STORE_RELEASE_ARENA(W, store_release32_result, "0x12345678") + STORE_RELEASE_ARENA(DW, store_release64_result, + "0x1234567890abcdef ll") +#undef STORE_RELEASE_ARENA +#endif + + return 0; +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c new file mode 100644 index 000000000000..6327638f031c --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c @@ -0,0 +1,190 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#if defined(ENABLE_ATOMICS_TESTS) && defined(__TARGET_ARCH_arm64) + +SEC("socket") +__description("load-acquire, 8-bit") +__success __success_unpriv __retval(0x12) +__naked void load_acquire_8(void) +{ + asm volatile ( + "*(u8 *)(r10 - 1) = 0x12;" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u8 *)(r10 - 1)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_B, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_10, -1)) + : __clobber_all); +} + +SEC("socket") +__description("load-acquire, 16-bit") +__success __success_unpriv __retval(0x1234) +__naked void load_acquire_16(void) +{ + asm volatile ( + "*(u16 *)(r10 - 2) = 0x1234;" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u16 *)(r10 - 2)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_H, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_10, -2)) + : __clobber_all); +} + +SEC("socket") +__description("load-acquire, 32-bit") +__success __success_unpriv __retval(0x12345678) +__naked void load_acquire_32(void) +{ + asm volatile ( + "*(u32 *)(r10 - 4) = 0x12345678;" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u32 *)(r10 - 4)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_W, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_10, -4)) + : __clobber_all); +} + +SEC("socket") +__description("load-acquire, 64-bit") +__success __success_unpriv __retval(0x1234567890abcdef) +__naked void load_acquire_64(void) +{ + asm volatile ( + "*(u64 *)(r10 - 8) = 0x1234567890abcdef;" + ".8byte %[load_acquire_insn];" // r0 = load_acquire((u64 *)(r10 - 8)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_10, -8)) + : __clobber_all); +} + +SEC("socket") +__description("load-acquire with uninitialized src_reg") +__failure __failure_unpriv __msg("R2 !read_ok") +__naked void load_acquire_with_uninitialized_src_reg(void) +{ + asm volatile ( + ".8byte %[load_acquire_insn];" // r0 = load_acquire((u64 *)(r2 + 0)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_2, 0)) + : __clobber_all); +} + +SEC("socket") +__description("load-acquire with non-pointer src_reg") +__failure __failure_unpriv __msg("R1 invalid mem access 'scalar'") +__naked void load_acquire_with_non_pointer_src_reg(void) +{ + asm volatile ( + "r1 = 0;" + ".8byte %[load_acquire_insn];" // r0 = load_acquire((u64 *)(r1 + 0)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_1, 0)) + : __clobber_all); +} + +SEC("socket") +__description("misaligned load-acquire") +__failure __failure_unpriv __msg("misaligned stack access off") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void load_acquire_misaligned(void) +{ + asm volatile ( + "*(u64 *)(r10 - 8) = 0;" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u32 *)(r10 - 5)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_W, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_10, -5)) + : __clobber_all); +} + +SEC("socket") +__description("load-acquire from ctx pointer") +__failure __failure_unpriv __msg("BPF_ATOMIC loads from R1 ctx is not allowed") +__naked void load_acquire_from_ctx_pointer(void) +{ + asm volatile ( + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u8 *)(r1 + 0)); + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_B, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_1, 0)) + : __clobber_all); +} + +SEC("xdp") +__description("load-acquire from pkt pointer") +__failure __msg("BPF_ATOMIC loads from R2 pkt is not allowed") +__naked void load_acquire_from_pkt_pointer(void) +{ + asm volatile ( + "r2 = *(u32 *)(r1 + %[xdp_md_data]);" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u8 *)(r2 + 0)); + "exit;" + : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_B, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_2, 0)) + : __clobber_all); +} + +SEC("flow_dissector") +__description("load-acquire from flow_keys pointer") +__failure __msg("BPF_ATOMIC loads from R2 flow_keys is not allowed") +__naked void load_acquire_from_flow_keys_pointer(void) +{ + asm volatile ( + "r2 = *(u64 *)(r1 + %[__sk_buff_flow_keys]);" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u8 *)(r2 + 0)); + "exit;" + : + : __imm_const(__sk_buff_flow_keys, + offsetof(struct __sk_buff, flow_keys)), + __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_B, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_2, 0)) + : __clobber_all); +} + +SEC("sk_reuseport") +__description("load-acquire from sock pointer") +__failure __msg("BPF_ATOMIC loads from R2 sock is not allowed") +__naked void load_acquire_from_sock_pointer(void) +{ + asm volatile ( + "r2 = *(u64 *)(r1 + %[sk_reuseport_md_sk]);" + ".8byte %[load_acquire_insn];" // w0 = load_acquire((u8 *)(r2 + 0)); + "exit;" + : + : __imm_const(sk_reuseport_md_sk, offsetof(struct sk_reuseport_md, sk)), + __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_B, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_2, 0)) + : __clobber_all); +} + +#else + +SEC("socket") +__description("load-acquire is not supported by compiler or jit, use a dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c index 6b564d4c0986..3bc4f50015e6 100644 --- a/tools/testing/selftests/bpf/progs/verifier_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_precision.c @@ -2,6 +2,7 @@ /* Copyright (C) 2023 SUSE LLC */ #include #include +#include "../../../include/linux/filter.h" #include "bpf_misc.h" SEC("?raw_tp") @@ -92,6 +93,52 @@ __naked int bpf_end_bswap(void) #endif /* v4 instruction */ +#if defined(ENABLE_ATOMICS_TESTS) && defined(__TARGET_ARCH_arm64) + +SEC("?raw_tp") +__success __log_level(2) +__msg("mark_precise: frame0: regs=r1 stack= before 2: (bf) r2 = r10") +__msg("mark_precise: frame0: regs=r1 stack= before 1: (db) r1 = load_acquire((u64 *)(r10 -8))") +__msg("mark_precise: frame0: regs= stack=-8 before 0: (7a) *(u64 *)(r10 -8) = 8") +__naked int bpf_load_acquire(void) +{ + asm volatile ( + "*(u64 *)(r10 - 8) = 8;" + ".8byte %[load_acquire_insn];" /* r1 = load_acquire((u64 *)(r10 - 8)); */ + "r2 = r10;" + "r2 += r1;" /* mark_precise */ + "r0 = 0;" + "exit;" + : + : __imm_insn(load_acquire_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_1, BPF_REG_10, -8)) + : __clobber_all); +} + +SEC("?raw_tp") +__success __log_level(2) +__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r2 = r10") +__msg("mark_precise: frame0: regs=r1 stack= before 2: (79) r1 = *(u64 *)(r10 -8)") +__msg("mark_precise: frame0: regs= stack=-8 before 1: (db) store_release((u64 *)(r10 -8), r1)") +__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8") +__naked int bpf_store_release(void) +{ + asm volatile ( + "r1 = 8;" + ".8byte %[store_release_insn];" /* store_release((u64 *)(r10 - 8), r1); */ + "r1 = *(u64 *)(r10 - 8);" + "r2 = r10;" + "r2 += r1;" /* mark_precise */ + "r0 = 0;" + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +#endif /* load-acquire, store-release */ + SEC("?raw_tp") __success __log_level(2) /* diff --git a/tools/testing/selftests/bpf/progs/verifier_store_release.c b/tools/testing/selftests/bpf/progs/verifier_store_release.c new file mode 100644 index 000000000000..f6e7cc6538c4 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/verifier_store_release.c @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include "../../../include/linux/filter.h" +#include "bpf_misc.h" + +#if defined(ENABLE_ATOMICS_TESTS) && defined(__TARGET_ARCH_arm64) + +SEC("socket") +__description("store-release, 8-bit") +__success __success_unpriv __retval(0x12) +__naked void store_release_8(void) +{ + asm volatile ( + "w1 = 0x12;" + ".8byte %[store_release_insn];" // store_release((u8 *)(r10 - 1), w1); + "w0 = *(u8 *)(r10 - 1);" + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_B, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -1)) + : __clobber_all); +} + +SEC("socket") +__description("store-release, 16-bit") +__success __success_unpriv __retval(0x1234) +__naked void store_release_16(void) +{ + asm volatile ( + "w1 = 0x1234;" + ".8byte %[store_release_insn];" // store_release((u16 *)(r10 - 2), w1); + "w0 = *(u16 *)(r10 - 2);" + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_H, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -2)) + : __clobber_all); +} + +SEC("socket") +__description("store-release, 32-bit") +__success __success_unpriv __retval(0x12345678) +__naked void store_release_32(void) +{ + asm volatile ( + "w1 = 0x12345678;" + ".8byte %[store_release_insn];" // store_release((u32 *)(r10 - 4), w1); + "w0 = *(u32 *)(r10 - 4);" + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_W, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -4)) + : __clobber_all); +} + +SEC("socket") +__description("store-release, 64-bit") +__success __success_unpriv __retval(0x1234567890abcdef) +__naked void store_release_64(void) +{ + asm volatile ( + "r1 = 0x1234567890abcdef ll;" + ".8byte %[store_release_insn];" // store_release((u64 *)(r10 - 8), r1); + "r0 = *(u64 *)(r10 - 8);" + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +SEC("socket") +__description("store-release with uninitialized src_reg") +__failure __failure_unpriv __msg("R2 !read_ok") +__naked void store_release_with_uninitialized_src_reg(void) +{ + asm volatile ( + ".8byte %[store_release_insn];" // store_release((u64 *)(r10 - 8), r2); + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_2, -8)) + : __clobber_all); +} + +SEC("socket") +__description("store-release with uninitialized dst_reg") +__failure __failure_unpriv __msg("R2 !read_ok") +__naked void store_release_with_uninitialized_dst_reg(void) +{ + asm volatile ( + "r1 = 0;" + ".8byte %[store_release_insn];" // store_release((u64 *)(r2 - 8), r1); + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_2, BPF_REG_1, -8)) + : __clobber_all); +} + +SEC("socket") +__description("store-release with non-pointer dst_reg") +__failure __failure_unpriv __msg("R1 invalid mem access 'scalar'") +__naked void store_release_with_non_pointer_dst_reg(void) +{ + asm volatile ( + "r1 = 0;" + ".8byte %[store_release_insn];" // store_release((u64 *)(r1 + 0), r1); + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_1, BPF_REG_1, 0)) + : __clobber_all); +} + +SEC("socket") +__description("misaligned store-release") +__failure __failure_unpriv __msg("misaligned stack access off") +__flag(BPF_F_ANY_ALIGNMENT) +__naked void store_release_misaligned(void) +{ + asm volatile ( + "w0 = 0;" + ".8byte %[store_release_insn];" // store_release((u32 *)(r10 - 5), w0); + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_W, BPF_STORE_REL, BPF_REG_10, BPF_REG_0, -5)) + : __clobber_all); +} + +SEC("socket") +__description("store-release to ctx pointer") +__failure __failure_unpriv __msg("BPF_ATOMIC stores into R1 ctx is not allowed") +__naked void store_release_to_ctx_pointer(void) +{ + asm volatile ( + "w0 = 0;" + ".8byte %[store_release_insn];" // store_release((u8 *)(r1 + 0), w0); + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_B, BPF_STORE_REL, BPF_REG_1, BPF_REG_0, 0)) + : __clobber_all); +} + +SEC("xdp") +__description("store-release to pkt pointer") +__failure __msg("BPF_ATOMIC stores into R2 pkt is not allowed") +__naked void store_release_to_pkt_pointer(void) +{ + asm volatile ( + "w0 = 0;" + "r2 = *(u32 *)(r1 + %[xdp_md_data]);" + ".8byte %[store_release_insn];" // store_release((u8 *)(r2 + 0), w0); + "exit;" + : + : __imm_const(xdp_md_data, offsetof(struct xdp_md, data)), + __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_B, BPF_STORE_REL, BPF_REG_2, BPF_REG_0, 0)) + : __clobber_all); +} + +SEC("flow_dissector") +__description("store-release to flow_keys pointer") +__failure __msg("BPF_ATOMIC stores into R2 flow_keys is not allowed") +__naked void store_release_to_flow_keys_pointer(void) +{ + asm volatile ( + "w0 = 0;" + "r2 = *(u64 *)(r1 + %[__sk_buff_flow_keys]);" + ".8byte %[store_release_insn];" // store_release((u8 *)(r2 + 0), w0); + "exit;" + : + : __imm_const(__sk_buff_flow_keys, + offsetof(struct __sk_buff, flow_keys)), + __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_B, BPF_STORE_REL, BPF_REG_2, BPF_REG_0, 0)) + : __clobber_all); +} + +SEC("sk_reuseport") +__description("store-release to sock pointer") +__failure __msg("BPF_ATOMIC stores into R2 sock is not allowed") +__naked void store_release_to_sock_pointer(void) +{ + asm volatile ( + "w0 = 0;" + "r2 = *(u64 *)(r1 + %[sk_reuseport_md_sk]);" + ".8byte %[store_release_insn];" // store_release((u8 *)(r2 + 0), w0); + "exit;" + : + : __imm_const(sk_reuseport_md_sk, offsetof(struct sk_reuseport_md, sk)), + __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_B, BPF_STORE_REL, BPF_REG_2, BPF_REG_0, 0)) + : __clobber_all); +} + +SEC("socket") +__description("store-release, leak pointer to stack") +__success __success_unpriv __retval(0) +__naked void store_release_leak_pointer_to_stack(void) +{ + asm volatile ( + ".8byte %[store_release_insn];" // store_release((u64 *)(r10 - 8), r1); + "r0 = 0;" + "exit;" + : + : __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_10, BPF_REG_1, -8)) + : __clobber_all); +} + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, long long); + __type(value, long long); +} map_hash_8b SEC(".maps"); + +SEC("socket") +__description("store-release, leak pointer to map") +__success __retval(0) +__failure_unpriv __msg_unpriv("R6 leaks addr into map") +__naked void store_release_leak_pointer_to_map(void) +{ + asm volatile ( + "r6 = r1;" + "r1 = %[map_hash_8b] ll;" + "r2 = 0;" + "*(u64 *)(r10 - 8) = r2;" + "r2 = r10;" + "r2 += -8;" + "call %[bpf_map_lookup_elem];" + "if r0 == 0 goto l0_%=;" + ".8byte %[store_release_insn];" // store_release((u64 *)(r0 + 0), r6); +"l0_%=:" + "r0 = 0;" + "exit;" + : + : __imm_addr(map_hash_8b), + __imm(bpf_map_lookup_elem), + __imm_insn(store_release_insn, + BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_0, BPF_REG_6, 0)) + : __clobber_all); +} + +#else + +SEC("socket") +__description("store-release is not supported by compiler or jit, use a dummy test") +__success +int dummy_test(void) +{ + return 0; +} + +#endif + +char _license[] SEC("license") = "GPL"; From patchwork Fri Feb 7 02:06:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peilin Ye X-Patchwork-Id: 13964245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 84DACC02198 for ; Fri, 7 Feb 2025 02:19:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=2ep8DalcKj7dauwNLlP5ARZOtPwUTNVJ49jqJ539nuE=; b=pcdJYP0nQJH1jeHp/r8G0KGAiM dkYMG+ORQAWMQ4UdEX/8dd1jXQHL0QejEOPzYQbOfOGCUN86tfEZhKaZ1dgDXOxrCQymoqwDbz2iY ZUvhjMHjF4y4+NlspnhUQNbCYzpXLEAqNrDVLjqA8AidoHfkpDOjIxP+e1x/Sh0Upshp+Z7qS6p+q WqOnvw2/FSws2hgMWEc8gCPiZIid9US85+9iv5lbPMXWx5iztQGsfbDwD/3AP0AYnqcJOpSlpKK7S i3D+K4NAVJz80J/8oRHWsu1mKuxqpNnrGrQEntvtFdaX4mdEE7XNyNJ0Rn2nzsdQXGpCzcfT8iYd5 woVM0d/Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tgDxM-000000083Na-0i0z; Fri, 07 Feb 2025 02:19:00 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tgDlR-000000081in-3Y8h for linux-arm-kernel@lists.infradead.org; Fri, 07 Feb 2025 02:06:42 +0000 Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-21f40f40788so25193085ad.3 for ; Thu, 06 Feb 2025 18:06:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738894001; x=1739498801; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2ep8DalcKj7dauwNLlP5ARZOtPwUTNVJ49jqJ539nuE=; b=kSdNkGdA1aLy0rad2rhzWW6rlQYlGDptpBQUqEwpnbBQ7PS3NK21CgQCGLM77hBDmZ Aom0NSF0FZqadFDisATMtzV1trtuWD1q6PXPgQEbLElwCLHIJcFyAe/1hBrgJCU3ZG3E YhPDDLFwsG27kG4aBmE8C2Lv7hIybCXLoaG236mQsLAmCR5UN5djQ3v9HMtT+7lvS6IX Dny02NZfo21SN6LaUz4tUUXCZ9RQFCXv16RIYQD3p1e33QQ2CweEeBzFvpjQ4ujiDl7u F9+c/aslpN5jFwTWWb5iTwWthcqkPCXDBPXAsHcUjLcOr+rjf1XLG3ijGvVwiz0ad75Z 0uLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738894001; x=1739498801; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2ep8DalcKj7dauwNLlP5ARZOtPwUTNVJ49jqJ539nuE=; b=J9e3adUBUvXFd4Xlad41GHK5QPRtcSTwrdmQmVy0Cf/noHPwxRmcqz4QYqyeN1Le74 1AZ2VrLapkkhlQGUI7jCYYUV381LIycWLH88m2yC8nonnhwoKA03g569E5ePjSkrCJbF 1GM2KrMJSJks43rJr1/NxE/ufGTnWiekwR0aV0XFb+PT5iUAYnyAwlnromklpN9JDLzh bkZnNcikePJiCLsXU4fsBU+P+t44dJ6qPI7V/5dC9CgTXUNtpkiIARp5KDpfMo76BFOP 0yAtx+7qZAM1uzl+zvF0Qx1xUC6/mzfNV+a64yUJFl9cVbm8s6teogM3xxqkIn6aTxlc 4pGw== X-Forwarded-Encrypted: i=1; AJvYcCWFQKChpL6nA4tj4a+17z8HSmAyX+5qPCkU9RI8PeTfhWTaC0N5kSdlHfEEXNJw8CqsShXfQy3x+21D1HsGBAGO@lists.infradead.org X-Gm-Message-State: AOJu0YyrHJwSzB/x7mDSVp9M7+d45BLqAWnJmtAu1WVuoTC6VV+ams7P WmVTp2InezE95zRMNvIOcArI8txUtyGQgOEaTi+rhbVM69D6jAkI1RxdxpbhdV7JghQOn0ivr5c M43mSB4FPHQ== X-Google-Smtp-Source: AGHT+IHO+QQmGU+1/m5wHtsMoItJdlgCTFiDvf89nZcXM296VZbNzIAooIz/Ko6Udd05tpk4RbaW4EcMXoxUsg== X-Received: from pfnv19.prod.google.com ([2002:aa7:8513:0:b0:730:5488:ee30]) (user=yepeilin job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:1394:b0:1e0:c77c:450d with SMTP id adf61e73a8af0-1ee03a242cemr2973087637.1.1738894000789; Thu, 06 Feb 2025 18:06:40 -0800 (PST) Date: Fri, 7 Feb 2025 02:06:36 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: Subject: [PATCH bpf-next v2 9/9] bpf, docs: Update instruction-set.rst for load-acquire and store-release instructions From: Peilin Ye To: bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Peilin Ye , bpf@ietf.org, Xu Kuohai , Eduard Zingerman , David Vernet , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Jonathan Corbet , "Paul E. McKenney" , Puranjay Mohan , Ilya Leoshkevich , Heiko Carstens , Vasily Gorbik , Catalin Marinas , Will Deacon , Quentin Monnet , Mykola Lysenko , Shuah Khan , Ihor Solodrai , Yingchi Long , Josh Don , Barret Rhoden , Neel Natu , Benjamin Segall , linux-kernel@vger.kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250206_180641_893016_32B35619 X-CRM114-Status: GOOD ( 14.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Update documentation for the new load-acquire and store-release instructions. Rename existing atomic operations as "atomic read-modify-write (RMW) operations". Following RFC 9669, section 7.3. "Adding Instructions", create new conformance groups "atomic32v2" and "atomic64v2", where: * atomic32v2: includes all instructions in "atomic32", plus the new 8-bit, 16-bit and 32-bit atomic load-acquire and store-release instructions * atomic64v2: includes all instructions in "atomic64" and "atomic32v2", plus the new 64-bit atomic load-acquire and store-release instructions Cc: bpf@ietf.org Signed-off-by: Peilin Ye --- .../bpf/standardization/instruction-set.rst | 114 +++++++++++++++--- 1 file changed, 98 insertions(+), 16 deletions(-) diff --git a/Documentation/bpf/standardization/instruction-set.rst b/Documentation/bpf/standardization/instruction-set.rst index ab820d565052..86917932e9ef 100644 --- a/Documentation/bpf/standardization/instruction-set.rst +++ b/Documentation/bpf/standardization/instruction-set.rst @@ -139,8 +139,14 @@ This document defines the following conformance groups: specification unless otherwise noted. * base64: includes base32, plus instructions explicitly noted as being in the base64 conformance group. -* atomic32: includes 32-bit atomic operation instructions (see `Atomic operations`_). -* atomic64: includes atomic32, plus 64-bit atomic operation instructions. +* atomic32: includes 32-bit atomic read-modify-write instructions (see + `Atomic operations`_). +* atomic32v2: includes atomic32, plus 8-bit, 16-bit and 32-bit atomic + load-acquire and store-release instructions. +* atomic64: includes atomic32, plus 64-bit atomic read-modify-write + instructions. +* atomic64v2: unifies atomic32v2 and atomic64, plus 64-bit atomic load-acquire + and store-release instructions. * divmul32: includes 32-bit division, multiplication, and modulo instructions. * divmul64: includes divmul32, plus 64-bit division, multiplication, and modulo instructions. @@ -653,20 +659,31 @@ Atomic operations are operations that operate on memory and can not be interrupted or corrupted by other access to the same memory region by other BPF programs or means outside of this specification. -All atomic operations supported by BPF are encoded as store operations -that use the ``ATOMIC`` mode modifier as follows: +All atomic operations supported by BPF are encoded as ``STX`` instructions +that use the ``ATOMIC`` mode modifier, with the 'imm' field encoding the +actual atomic operation. These operations are categorized based on the second +lowest nibble (bits 4-7) of the 'imm' field: -* ``{ATOMIC, W, STX}`` for 32-bit operations, which are +* ``ATOMIC_LOAD`` and ``ATOMIC_STORE`` indicate atomic load and store + operations, respectively (see `Atomic load and store operations`_). +* All other defined values indicate an atomic read-modify-write operation, as + described in the following section. + +Atomic read-modify-write operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The atomic read-modify-write (RMW) operations are encoded as follows: + +* ``{ATOMIC, W, STX}`` for 32-bit RMW operations, which are part of the "atomic32" conformance group. -* ``{ATOMIC, DW, STX}`` for 64-bit operations, which are +* ``{ATOMIC, DW, STX}`` for 64-bit RMW operations, which are part of the "atomic64" conformance group. -* 8-bit and 16-bit wide atomic operations are not supported. +* 8-bit and 16-bit wide atomic RMW operations are not supported. -The 'imm' field is used to encode the actual atomic operation. -Simple atomic operation use a subset of the values defined to encode -arithmetic operations in the 'imm' field to encode the atomic operation: +Simple atomic RMW operation use a subset of the values defined to encode +arithmetic operations in the 'imm' field to encode the atomic RMW operation: -.. table:: Simple atomic operations +.. table:: Simple atomic read-modify-write operations ======== ===== =========== imm value description @@ -686,10 +703,10 @@ arithmetic operations in the 'imm' field to encode the atomic operation: *(u64 *)(dst + offset) += src -In addition to the simple atomic operations, there also is a modifier and -two complex atomic operations: +In addition to the simple atomic RMW operations, there also is a modifier and +two complex atomic RMW operations: -.. table:: Complex atomic operations +.. table:: Complex atomic read-modify-write operations =========== ================ =========================== imm value description @@ -699,8 +716,8 @@ two complex atomic operations: CMPXCHG 0xf0 | FETCH atomic compare and exchange =========== ================ =========================== -The ``FETCH`` modifier is optional for simple atomic operations, and -always set for the complex atomic operations. If the ``FETCH`` flag +The ``FETCH`` modifier is optional for simple atomic RMW operations, and +always set for the complex atomic RMW operations. If the ``FETCH`` flag is set, then the operation also overwrites ``src`` with the value that was in memory before it was modified. @@ -713,6 +730,71 @@ The ``CMPXCHG`` operation atomically compares the value addressed by value that was at ``dst + offset`` before the operation is zero-extended and loaded back to ``R0``. +Atomic load and store operations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To encode an atomic load or store operation, the lowest 8 bits of the 'imm' +field are divided as follows:: + + +-+-+-+-+-+-+-+-+ + | type | order | + +-+-+-+-+-+-+-+-+ + +**type** + The operation type is one of: + +.. table:: Atomic load and store operation types + + ============ ===== ============ + type value description + ============ ===== ============ + ATOMIC_LOAD 0x1 atomic load + ATOMIC_STORE 0x2 atomic store + ============ ===== ============ + +**order** + The memory order is one of: + +.. table:: Memory orders + + ======= ===== ======================= + order value description + ======= ===== ======================= + RELAXED 0x0 relaxed + ACQUIRE 0x1 acquire + RELEASE 0x2 release + ACQ_REL 0x3 acquire and release + SEQ_CST 0x4 sequentially consistent + ======= ===== ======================= + +Currently the following combinations of ``type`` and ``order`` are allowed: + +.. table:: Atomic load and store operations + + ========= ===== ==================== + imm value description + ========= ===== ==================== + LOAD_ACQ 0x11 atomic load-acquire + STORE_REL 0x22 atomic store-release + ========= ===== ==================== + +``{ATOMIC, , STX}`` with 'imm' = LOAD_ACQ means:: + + dst = load_acquire((unsigned size *)(src + offset)) + +``{ATOMIC, , STX}`` with 'imm' = STORE_REL means:: + + store_release((unsigned size *)(dst + offset), src) + +Where '' is one of: ``B``, ``H``, ``W``, or ``DW``, and 'unsigned size' +is one of: u8, u16, u32, or u64. + +8-bit, 16-bit and 32-bit atomic load-acquire and store-release instructions +are part of the "atomic32v2" conformance group. + +64-bit atomic load-acquire and store-release instructions are part of the +"atomic64v2" conformance group. + 64-bit immediate instructions -----------------------------