From patchwork Thu Feb 1 04:20:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540615 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E50F3A8EE for ; Thu, 1 Feb 2024 04:21:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761277; cv=none; b=cVnJ8wQnrDi414+VfeJSfG3kUuuiCDqJEfxsKKl2WiGaYlnMQKjItj3Lw5Y5euMvIG67X+2/WpQQW2dwe5ghQYWm0CQYalcS/58SB+GLoUNHlMyxjbZl/vBT7y3AHBevrN7nA80/MDelHnno3dvcIlKuKyen54VHqD25lKwM5BU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761277; c=relaxed/simple; bh=qKYwvKsLrw3JsKcg64nkIme+xoRLx4arV2gWZJ4CE2w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Aj8gzgmSoTN4Yb829Tdx0lBKkSu1iPDV9zeYpW9FON5oqnQ8zYJbgc6TNfO+aDyJN6ntPfcHANeX6cIryXFwy9zs59SyvbeaEndJIhLryBoywmCkFbGdO0B+VtMhV1vzeDsho1otgZm6mXgpLQux50KMcFAb/beqzxDmai75hzg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TazvSpIi; arc=none smtp.client-ip=209.85.208.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TazvSpIi" Received: by mail-ed1-f68.google.com with SMTP id 4fb4d7f45d1cf-55f279dca99so583659a12.3 for ; Wed, 31 Jan 2024 20:21:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761273; x=1707366073; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aHXSg/hnGXQwFDv+dpmcNyKdBOSJCDlZGg07N4nhKKg=; b=TazvSpIivWeT7qIf0SjHujdI1CdgVdMrECnKJ8yqBZUdzlzLoHCqDzP0c17h5Q8miN KNdzKlY4uMe2JMpRLUjXM+PEc6katiD3Dk/iFM6oc+nRVaXFZ5SHWnOGt2YZLG0ITKdj oM4fy2dz6HH4uiGGVinOzVZfxCQnP1Tmb5JDTFxJuStWJHf3Ot2uKldAj2G13WCPgKpV xmXHmxkQyIyPKi/5wwGHavImB4sHTtdc3/z6bF9geiNopM69AewJSbPHo5eg2X9XjoQP ofmro5dJwx8Bu/3owy1YkcL/1Pf/2vLYHwTkMbA8/HRkrA/qIum5jCdW2KqKQ2bD9qXc 0Jmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761273; x=1707366073; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aHXSg/hnGXQwFDv+dpmcNyKdBOSJCDlZGg07N4nhKKg=; b=PFiz6hEgAlDPUzrjem2scK1sH7laaDjavMS7CLBS5g6jdgeiVeK9o+hSCXPvh+EVm/ Ex3H7AE+qzbfmDBtRrInN6yLE3aWg9AoTHErFxvEFZBXbdwjU4ejeVunUrxEo3OpbDHA PwxAgbsyBCOivMAjwoBATtL1YStXkOrnVDft7BhxQoBWFvwubtGLk9jt9MJbklYCNMek +B3p32zX0MYAzDuDXrSrZjCtLIvDBMpnlSBT+7eJ0iC0OvuAggNGRE1U9T3pDmxk/RNR 4Mmz+RZiMErPVzwUGBQ5r8VBOLDLwjQjjImTW5nYnbfHOtnA1G1rJF9LLWlYGfeZ59C0 8Lgw== X-Gm-Message-State: AOJu0YzE5R3h1L0o0cIV16dMTmqomOCnsyZmezyLKbf9g6s2d9U5DxGg LjvpF85WkM/o1GBbsqTHwFAttTIiYj6Ja5NK70a3RUXPogplk3ZOpBe0rqVqjr0= X-Google-Smtp-Source: AGHT+IEDfD3aXNX6mJb2XFZw4iDgZi8uDV8NVhFchr3XxFqB05aDxc2JoQB4abARwlS5FBCN/JRohQ== X-Received: by 2002:a05:6402:b23:b0:55e:ee3a:723 with SMTP id bo3-20020a0564020b2300b0055eee3a0723mr2303523edb.12.1706761272524; Wed, 31 Jan 2024 20:21:12 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id ba8-20020a0564021ac800b0055f2af9b01bsm2703825edb.17.2024.01.31.20.21.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:11 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 01/14] bpf: Mark subprogs as throw reachable before do_check pass Date: Thu, 1 Feb 2024 04:20:56 +0000 Message-Id: <20240201042109.1150490-2-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=7543; i=memxor@gmail.com; h=from:subject; bh=qKYwvKsLrw3JsKcg64nkIme+xoRLx4arV2gWZJ4CE2w=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwMtn4aVsJ7zkZzBzOj5IAVI/huSU9j6vdPw Vb+avFqUoWJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDAAKCRBM4MiGSL8R yp0TD/46YPc0jk/GpfbeYkLPxmJuxWQ2dDF3YaprVZCf67FFbh6sHrXNTAW+1Jo44MQwULTrpTo bbT/q4CGwfFcMEqWpxyjxjghGLvajN/6886mhqk36HJyMsF9X09EP0ZcFVGJ0rXEPpsq9ZopRlx kLlbj51UPksokivbtlhjJ2zSjUzh7/6aPG8ByD5hCXrmEjUq9ACXZqi7DdmyKEdFJ6Fg+DO4Tc8 0hHoTlBrBAh46CkmVYH+HtWMigRMvVBm8sMN5+reLOiP+TlAXGmcoB6NfPhh0jt0symMns8QfcJ hDLe93rJvDR90Cc+9VSnAhkgTMAhGrSvU554YZEz5fvcYvRlDElQAZPjm2OO+jS8Vgs56fR9vOQ UOwaHNxM8pw2fvdeqM5sXE4T6tLTOimaeKGZaEiGgul7TckV1uT3t70w5z7gjr4wXgfTNf8/2Vn kNBUcw24e2CAB2VOhqT6sG9gS6+RGg7WMwrXyp1Hdh6B5Egc46frk5MfDyl3tZwc7VTfoGjqskt 48Re100ITxnsQWliRYPUhPwlRXLAPthHIeHeHfNKh+2ePuaopyqv3mBSJX+w9//TQFAGTKcBywL RvfTYyX6J4m0ibLNoXyQ7QvRLLtm9w8ZdajvEbElI3FpGzcHNoeC5E1ThNWk0TJaTMBcT8ksVnu GmUTGU7C+3mrwiw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC The motivation of this patch is to figure out which subprogs participate in exception propagation. In other words, whichever subprog's execution can lead to an exception being thrown either directly or indirectly (by way of calling other subprogs). With the current exceptions support, the runtime performs stack unwinding when bpf_throw is called. For now, any resources acquired by the program cannot be released, therefore bpf_throw calls made with non-zero acquired references must be rejected during verification. However, there currently exists a loophole in this restriction due to the way the verification procedure is structured. The verifier will first walk over the main subprog's instructions, but not descend into subprog calls to ones with global linkage. These global subprogs will then be independently verified instead. Therefore, in a situation where a global subprog ends up throwing an exception (either directly by calling bpf_throw, or indirectly by way of calling another subprog that does so), the verifier will fail to notice this fact and may permit throwing BPF exceptions with non-zero acquired references. Therefore, to fix this, we add a summarization pass before the do_check stage which walks all call chains of the program and marks all of the subprogs that are reachable from a bpf_throw call which unwinds the program stack. We only do so if we actually see a bpf_throw call in the program though, since we do not want to walk all instructions unless we need to. One we analyze all possible call chains of the program, we will be able to mark them as 'is_throw_reachable' in their subprog_info. After performing this step, we need to make another change as to how subprog call verification occurs. In case of global subprog, we will need to explore an alternate program path where the call instruction processing of a global subprog's call will immediately throw an exception. We will thus simulate a normal path without any exceptions, and one where the exception is thrown and the program proceeds no further. In this way, the verifier will be able to detect the whether any acquired references or locks exist in the verifier state and thus reject the program if needed. Fixes: f18b03fabaa9 ("bpf: Implement BPF exceptions") Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf_verifier.h | 2 + kernel/bpf/verifier.c | 86 ++++++++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 0dcde339dc7e..1d666b6c21e6 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -626,6 +626,7 @@ struct bpf_subprog_info { bool is_async_cb: 1; bool is_exception_cb: 1; bool args_cached: 1; + bool is_throw_reachable: 1; u8 arg_cnt; struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS]; @@ -691,6 +692,7 @@ struct bpf_verifier_env { bool bypass_spec_v4; bool seen_direct_write; bool seen_exception; + bool seen_throw_insn; struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index cd4d780e5400..bba53c4e3a0c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2941,6 +2941,8 @@ static int check_subprogs(struct bpf_verifier_env *env) insn[i].src_reg == 0 && insn[i].imm == BPF_FUNC_tail_call) subprog[cur_subprog].has_tail_call = true; + if (!env->seen_throw_insn && is_bpf_throw_kfunc(&insn[i])) + env->seen_throw_insn = true; if (BPF_CLASS(code) == BPF_LD && (BPF_MODE(code) == BPF_ABS || BPF_MODE(code) == BPF_IND)) subprog[cur_subprog].has_ld_abs = true; @@ -5866,6 +5868,9 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx) if (!is_bpf_throw_kfunc(insn + i)) continue; + /* When this is allowed, don't forget to update logic for sync and + * async callbacks in mark_exception_reachable_subprogs. + */ if (subprog[idx].is_cb) err = true; for (int c = 0; c < frame && !err; c++) { @@ -16205,6 +16210,83 @@ static int check_btf_info(struct bpf_verifier_env *env, return 0; } +/* We walk the call graph of the program in this function, and mark everything in + * the call chain as 'is_throw_reachable'. This allows us to know which subprog + * calls may propagate an exception and generate exception frame descriptors for + * those call instructions. We already do that for bpf_throw calls made directly, + * but we need to mark the subprogs as we won't be able to see the call chains + * during symbolic execution in do_check_common due to global subprogs. + * + * Note that unlike check_max_stack_depth, we don't explore the async callbacks + * apart from main subprogs, as we don't support throwing from them for now, but + */ +static int mark_exception_reachable_subprogs(struct bpf_verifier_env *env) +{ + struct bpf_subprog_info *subprog = env->subprog_info; + struct bpf_insn *insn = env->prog->insnsi; + int idx = 0, frame = 0, i, subprog_end; + int ret_insn[MAX_CALL_FRAMES]; + int ret_prog[MAX_CALL_FRAMES]; + + /* No need if we never saw any bpf_throw() call in the program. */ + if (!env->seen_throw_insn) + return 0; + + i = subprog[idx].start; +restart: + subprog_end = subprog[idx + 1].start; + for (; i < subprog_end; i++) { + int next_insn, sidx; + + if (bpf_pseudo_kfunc_call(insn + i) && !insn[i].off) { + if (!is_bpf_throw_kfunc(insn + i)) + continue; + subprog[idx].is_throw_reachable = true; + for (int j = 0; j < frame; j++) + subprog[ret_prog[j]].is_throw_reachable = true; + } + + if (!bpf_pseudo_call(insn + i) && !bpf_pseudo_func(insn + i)) + continue; + /* remember insn and function to return to */ + ret_insn[frame] = i + 1; + ret_prog[frame] = idx; + + /* find the callee */ + next_insn = i + insn[i].imm + 1; + sidx = find_subprog(env, next_insn); + if (sidx < 0) { + WARN_ONCE(1, "verifier bug. No program starts at insn %d\n", next_insn); + return -EFAULT; + } + /* We cannot distinguish between sync or async cb, so we need to follow + * both. Async callbacks don't really propagate exceptions but calling + * bpf_throw from them is not allowed anyway, so there is no harm in + * exploring them. + * TODO: To address this properly, we will have to move is_cb, + * is_async_cb markings to the stage before do_check. + */ + i = next_insn; + idx = sidx; + + frame++; + if (frame >= MAX_CALL_FRAMES) { + verbose(env, "the call stack of %d frames is too deep !\n", frame); + return -E2BIG; + } + goto restart; + } + /* end of for() loop means the last insn of the 'subprog' + * was reached. Doesn't matter whether it was JA or EXIT + */ + if (frame == 0) + return 0; + frame--; + i = ret_insn[frame]; + idx = ret_prog[frame]; + goto restart; +} + /* check %cur's range satisfies %old's */ static bool range_within(struct bpf_reg_state *old, struct bpf_reg_state *cur) @@ -20939,6 +21021,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 if (ret < 0) goto skip_full_check; + ret = mark_exception_reachable_subprogs(env); + if (ret < 0) + goto skip_full_check; + ret = do_check_main(env); ret = ret ?: do_check_subprogs(env); From patchwork Thu Feb 1 04:20:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540616 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22C413B78D for ; Thu, 1 Feb 2024 04:21:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761278; cv=none; b=tFqVq0hE4SN8VRQS+WXMQ7bMOKLxxMvym9O2iAgPNPJeX24fXCNe4olkSbA5ZCsgHw+v7ryKWL8AMR5GawKoIjA0kjroCRdlKO8KKSHMUjN/yOpPxEyuWBEwGx7HGFMxqqc8x70YoqeP+N6Mg1HqRY8LNgFvMKJBwhBDzugyJ3Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761278; c=relaxed/simple; bh=hEYvcM4+e5mMwgHu/M6fr5pKvJ2Q6QC5ZxpvtZXZr7E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Z/54crkQbnuZMe0LIoQ5Nkn/k4GoUkZ0YSaYeXiMULf4FwKx0MxsBGZ2cBD+YSPpMKaSi3JtToQ5XyeQ970nDUu5tuRFZpsD/6k4lZ9GOWCgI9woJgtyszdm1XqYxd8dIBZkvAjeM2MN0bFy0gg0r3unebielvIqUjGnKZOFtNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=N5uAY5OJ; arc=none smtp.client-ip=209.85.208.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N5uAY5OJ" Received: by mail-ed1-f68.google.com with SMTP id 4fb4d7f45d1cf-55783b7b47aso605971a12.0 for ; Wed, 31 Jan 2024 20:21:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761274; x=1707366074; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KogAFSQ6rYjm7jb5WlDa4cAynBRb/EBeu6/yjS/jYLA=; b=N5uAY5OJwod921gzF8xTwghR6QY1wa1lcWcAyg3ozO2mg/oB06M69Yt5v7uZR0/sxd WG+snkwA7ACqopiK5jjQQKeokz9p4+VqHaBZ1WP+ZItlOuFP7t2CjqZVZmVsFP8m2jD8 ynrUc+puQwlORGbKenrW9jfbA3WpHoJHdsH5r391Pq9X2w34K/dAZix75pWO8Yuy4xOu OgckYPnfgUFfs9Z4ob0vapjVyBiWedeMQ29AGn5+4Z5hUFcJGQOFzLcGUnvmHBoHy8wW zIzSbvUzgWXC/dwDanF9axEdOD3bUyq0D9/UkDv/XeDPxVwFe17Y2RcQvzQFuVAA2cg4 zzDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761274; x=1707366074; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KogAFSQ6rYjm7jb5WlDa4cAynBRb/EBeu6/yjS/jYLA=; b=pQoGuBDRezPZAZFR/JJrIMmBdVgT0Cecpsdd3nSNdDKyEIaW28ixoIinIuwTe2uZea l3qEg8eALXXNCSA+PrewDO8LuZ8F9dVkbQsmqGhFdKADkdxHdQiDWitiz6mETwmNrIJ9 KFsYedqDbM8HGhM+ZVtwl4YXYfSmogJUHX5wG7luNcTUKBTi7KJMe2H5vBR9fhQkjoT9 fEO6frM8+USA11LvoW4uYdhEmiqnB4l8cWBOSnElP4QF9Wcn1T3sdmCGQmZjwBaxe8Ju 5N+lstbMCGGPz47KJKP+0LK3tmtxBYFFXyeHb+u3MiAt3c4NTyyI4ZWr28OowoexxLZ1 JgOw== X-Gm-Message-State: AOJu0YyadvidJDGHc1rpdBgoQKAiEpcHJL/Hve59Jn6FEto1aw7ieTvI hifDMd+gOzALqlPfEf2H6ObEI5UTRf74tnJV1EE/oRDSYRdq2SitCFFwRanJSco= X-Google-Smtp-Source: AGHT+IERPIYZuyTYYvvoTFNr8wLdOE2AsNaNHcrqIeZBCdTGsd0SGmWsLMNVRwJ6o3C/wZosteZm6Q== X-Received: by 2002:a05:6402:60a:b0:55f:4602:bf7d with SMTP id n10-20020a056402060a00b0055f4602bf7dmr2442058edv.26.1706761274066; Wed, 31 Jan 2024 20:21:14 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id b4-20020a056402138400b0055f2e703b52sm2653518edv.33.2024.01.31.20.21.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:13 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 02/14] bpf: Process global subprog's exception propagation Date: Thu, 1 Feb 2024 04:20:57 +0000 Message-Id: <20240201042109.1150490-3-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5104; i=memxor@gmail.com; h=from:subject; bh=hEYvcM4+e5mMwgHu/M6fr5pKvJ2Q6QC5ZxpvtZXZr7E=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwM/i/OmtJvvXOj8D85wSqhw344HLfCrxryb JGh92eQOcGJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDAAKCRBM4MiGSL8R yh8+D/9hiK/05rQE7XTTTHt499WIh6e2jOaSD5Me+DJY7RMHrcWusB0w+wIVNohUHwGSXS0ffkq kaYoWbuHzLZCM3nCcOxdBACtSfxapTZLyYdyGuYa3qDHW8WHmZqWY5sr36E7Q+s0H5+JbMcYwXi Z04KfQzpyo92o1v1ZMrhYD9UxWmF15GcNEq3YEnUfp/RCPvbNEP6sz0zUkLSLttpT1B0zhBQxbb 5tE0nSrPqc9UniknJLD9JvsjimF1KtP4n6e23MpfvR+4scZ5ll3ZK6fyeRAO+HOxNg/DjoMxHiH urHHKwxqjcCBhDajFEjVsxiKQskGDqbtCtlkxLjrJ35xr4vg20u70MEgsdro3R0LNGuwV1TNIAX gH4yQcecWHfMMB38zhJIWELdtlqxEoCeWjv/i5hOIcCWsG8MvWZaFqJtupZa0dV/3xSGbsFlNKC QDp5LVH1nYm5Geq8IH7LMMbDW1ct8ng8UyJxhOTegsbEwUeydpZfxKD2B63BN4v3z/CjonT6fne LRN3jdY7hC132YM4UhJk8ifZKNqC14soeLYXg3BeV+oFcI6Y6qjqdXcRweGis9IYjKIRs5zcfPh MmezqQRXpsrFtUKhmOLc+la5+doLHgyQspcbv2T2yjYZuNOTwnzS2W9f0YKyWh9LAiHEQiDPUn3 ssItqeSaaOTN39A== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Global subprogs are not descended during symbolic execution, but we summarized whether they can throw an exception (reachable from another exception throwing subprog) in mark_exception_reachable_subprogs added by the previous patch. We must now ensure that we explore the path of the program where invoking the call instruction leads to an exception being thrown, so that we can correctly reject programs where it is not permissible to throw an exception. For instance, it might be permissible to throw from a global subprog, but its caller may hold references. Without this patch, the verifier will accept such programs. To do this, we use push_stack to push a separate branch into the branch stack of the verifier, with the same current and previous insn_idx. Then, we set a bit in the verifier state of the branch to indicate that the next instruction it will process is of a global subprog call which will throw an exception. When we encounter this instruction, this bit will be cleared. Special care must be taken to update the state pruning logic, as without any changes, it is possible that we end up pruning when popping the exception throwing state for exploration. Therefore, while we can never have the 'global_subprog_call_exception' bit set in the verifier state of an explored state, we will see it in the current state, and use this to reject pruning requests and continue its exploration. Note that we process the exception after processing the call instruction, similar to how we do a process_bpf_exit_full jump in case of bpf_throw kfuncs. Fixes: f18b03fabaa9 ("bpf: Implement BPF exceptions") Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman --- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 1d666b6c21e6..5482701e6ad9 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -426,6 +426,7 @@ struct bpf_verifier_state { * while they are still in use. */ bool used_as_loop_entry; + bool global_subprog_call_exception; /* first and last insn idx of this verifier state */ u32 first_insn_idx; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bba53c4e3a0c..622c638b123b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1418,6 +1418,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state, dst_state->dfs_depth = src->dfs_depth; dst_state->callback_unroll_depth = src->callback_unroll_depth; dst_state->used_as_loop_entry = src->used_as_loop_entry; + dst_state->global_subprog_call_exception = src->global_subprog_call_exception; for (i = 0; i <= src->curframe; i++) { dst = dst_state->frame[i]; if (!dst) { @@ -9497,6 +9498,15 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, verbose(env, "Func#%d ('%s') is global and assumed valid.\n", subprog, sub_name); + if (subprog_info(env, subprog)->is_throw_reachable && !env->cur_state->global_subprog_call_exception) { + struct bpf_verifier_state *branch = push_stack(env, env->insn_idx, env->prev_insn_idx, false); + + if (!branch) { + verbose(env, "verifier internal error: cannot push branch to explore exception of global subprog\n"); + return -EFAULT; + } + branch->global_subprog_call_exception = true; + } /* mark global subprog for verifying after main prog */ subprog_aux(env, subprog)->called = true; clear_caller_saved_regs(env, caller->regs); @@ -9505,6 +9515,9 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, mark_reg_unknown(env, caller->regs, BPF_REG_0); caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; + if (env->cur_state->global_subprog_call_exception) + verbose(env, "Func#%d ('%s') may throw exception, exploring program path where exception is thrown\n", + subprog, sub_name); /* continue with next insn after call */ return 0; } @@ -16784,6 +16797,10 @@ static bool states_equal(struct bpf_verifier_env *env, if (old->active_rcu_lock != cur->active_rcu_lock) return false; + /* Prevent pruning to explore state where global subprog call throws an exception. */ + if (cur->global_subprog_call_exception) + return false; + /* for states to be equal callsites have to be the same * and all frame states need to be equivalent */ @@ -17675,6 +17692,11 @@ static int do_check(struct bpf_verifier_env *env) } if (insn->src_reg == BPF_PSEUDO_CALL) { err = check_func_call(env, insn, &env->insn_idx); + if (!err && env->cur_state->global_subprog_call_exception) { + env->cur_state->global_subprog_call_exception = false; + exception_exit = true; + goto process_bpf_exit_full; + } } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { err = check_kfunc_call(env, insn, &env->insn_idx); if (!err && is_bpf_throw_kfunc(insn)) { From patchwork Thu Feb 1 04:20:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540617 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 753EC3BB2C for ; Thu, 1 Feb 2024 04:21:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.66 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761280; cv=none; b=dPhey3xlMx70He+48eVWDvbLKvUxeOfzeKOULqxhNNNBGa4dI6u2FKPnYM9Db3MB1SehFGM43TQZWoynIWsNn8wo0yNfna/q6s3m77sSdrlNbP55OHoSw+AyhQlTIkPuLkraU14r6BtHUSl/k998pxgHwzi0LeOl0FEr44iRcJ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761280; c=relaxed/simple; bh=8/CsqgYsgnzXyV2WkXGz0peItpfG969p0XnileF4+5w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pHKNsu89jmNw8i9ZGPG2rbU/6fAaZIx1FR5bOc2SBAxUwmGP9XkXauxNu5DV8CI2LFp4ylX073uukxa4YWpxGb0+hghvGMnbuHYy4Fbss1cpZKLnB159034qPaAMu5GARG+xOpw/aZ3i5jLuMwHJkwZ3UZ08BURt3NuLtHcnDh4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cYWzs4sM; arc=none smtp.client-ip=209.85.167.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cYWzs4sM" Received: by mail-lf1-f66.google.com with SMTP id 2adb3069b0e04-5112cb7ae27so861744e87.0 for ; Wed, 31 Jan 2024 20:21:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761276; x=1707366076; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e751ZAVsIPuZft7onPZSCr8w6RHxKKY+J2i9lIT9Goc=; b=cYWzs4sMeHKhNMdoiZB4vzZcqcRaQvErFnP01PVuUKm4dPD1ZYGu9xk6TWpLnfi5E0 qO8afxZb/uXVyT6z7xWbPvHhGDwU4vTSVV0A4vvBCrQkf7VuxGZ6PZ7CbpT9f4L/sk95 vt5VdS2D328qe4f9nyMM2favCTJ+rLCrtnInpVR+it5zhexiwH+tqvxO6ATJIdPPVA2S qSq+z6MowuRZy82aH3TXDsUIlp8qIGxK71euCAOnBIEPcEjU1YbC+R3Rybn3WooDyKmX TNigng4h2PCU6w+HmOJyUHRWxVmWrU4/VK3vLYyjAEnjwYU9lLXTZVRYPErMBY6ks4me TA9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761276; x=1707366076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e751ZAVsIPuZft7onPZSCr8w6RHxKKY+J2i9lIT9Goc=; b=ABd6ZsLoUY3B0befd+kCpsGOtNERp/WrKb/6I86GYhpohNQrtnTU5zj4+hMvUYJ8an YmJS8pSFwU+/jAouRkaC6SYolkiqfO+aNRdNQ6EVXiBfQtAJ4KUcjJl6kHNIP5LDPWAw H/pG2PkuXAHIbvff0MrNrRbMBdr1G51OocglfpvCQJpye5M920VNSVw2ZySdAOLlK8vl mBGBbk1v0JwELJLe/+YcMLh33YbjRoBciQ634m83wiX5GFTzxYYdd+kjs/Tl2gTwYbLu 1mnmYnXLHH2OhTSbkPqpuZ/DTRC2PjhrZOmT3eUwLktzs7ifQ8jxkbhIxreyszK5Scep 2Lag== X-Gm-Message-State: AOJu0YxmJakn0Mv+p5+u55e+ESgBUMXzRRjWJb7bYkyePPQT/PmCZzS9 4F3/nmITOhB6ZLvSQJ08u0Z5z3fexZk2YGdihcLbyN/AMdNIWxWMXGxIoH+AQhk= X-Google-Smtp-Source: AGHT+IHIwTytGPxsK9vzSqdzPMoV2rjWER4idjtXgM7vt0AttPIMzS0XGDoQylZs9vdCIWDlzUfXpw== X-Received: by 2002:ac2:58e4:0:b0:511:1fb6:eb42 with SMTP id v4-20020ac258e4000000b005111fb6eb42mr831272lfo.61.1706761275505; Wed, 31 Jan 2024 20:21:15 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id fj18-20020a1709069c9200b00a3496fa1f7fsm6845034ejc.91.2024.01.31.20.21.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:14 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 03/14] selftests/bpf: Add test for throwing global subprog with acquired refs Date: Thu, 1 Feb 2024 04:20:58 +0000 Message-Id: <20240201042109.1150490-4-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1709; i=memxor@gmail.com; h=from:subject; bh=8/CsqgYsgnzXyV2WkXGz0peItpfG969p0XnileF4+5w=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwNY6BE6oFflfQh4oOP2N0GKs12XgnhYrBHW Fforz0M1JWJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDQAKCRBM4MiGSL8R ylDHD/kBK6uWn6e9ENv2KcjkZzHZrIT0BZ5VnCMog4vFzZYUzXQ9tdvNyJs//YMmU3DpSeyv43T rOUFgDOEm6vH9tYhbk41BlplGqzgVF3v3oCmrEZRYmzsadwID087T+D84T5jUh2p2eyWmdm6KNq JMIxmQJUzWtdbXaSQ/kPekwil97dttoFn+tL3y+gy5R0HssoW5IFy76xTMTjH9H0B/O6Na82cMS kF6aAZcqqCUBEnosp8xT7/X5CPUOsQTT0BQwX4gyI5T9SSaph6ALTzA2HsM7VR+BS3Aib9DEvxi 1YHbP1lxlyl5KzckTBJqq7I/BvHJblMg4WLhk3WjNoaXlpSdrp9KoxeupBxej14pMgyg9S16g4S fuam8Wy1vvHUog8GCyi39F10dP8xRrcZpitOTAJZTReKR+alh33Kc5xYe6sfCB/oFXBZV2zA1v6 tbNIZO1CQoqlfwDY/noaQKVRv01Jm+wzmFJK4hVJpO+IoNzmWtPtZG6cdDsGBEx/TbYUtQIeQRH KDtbtKFfLcFSZ4wVS2+LuBn78zTKVQz+mwNG5FwUQU5Dp6fVjNUAK6Qw4kdbCQgx5RlR0/x1NdY d0MxUurM18li4NG9+PIAcKAVUCNN4n/s7ZjRzgSAcBDiMoO/L0MZYuabUrL1uzLX4K1j0jvcORT 05N0dcb7xogiPZw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add a test case to exercise verifier logic where a global function that may potentially throw an exception is invoked from the main subprog, such that during exploration, the reference state is not visible when the bpf_throw instruction is explored. Without the fixes in prior commits, bpf_throw will not complain when unreleased resources are lingering in the program when a possible exception may be thrown. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman --- .../selftests/bpf/progs/exceptions_fail.c | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index 9cceb6521143..28602f905d7d 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -146,6 +146,13 @@ __noinline static int throwing_subprog(struct __sk_buff *ctx) return 0; } +__noinline int throwing_global_subprog(struct __sk_buff *ctx) +{ + if (ctx->len) + bpf_throw(0); + return 0; +} + SEC("?tc") __failure __msg("bpf_rcu_read_unlock is missing") int reject_subprog_with_rcu_read_lock(void *ctx) @@ -346,4 +353,18 @@ int reject_exception_throw_cb_diff(struct __sk_buff *ctx) return 0; } +SEC("?tc") +__failure __msg("exploring program path where exception is thrown") +int reject_exception_throw_ref_call_throwing_global(struct __sk_buff *ctx) +{ + struct { long a; } *p = bpf_obj_new(typeof(*p)); + + if (!p) + return 0; + if (ctx->protocol) + throwing_global_subprog(ctx); + bpf_obj_drop(p); + return 0; +} + char _license[] SEC("license") = "GPL"; From patchwork Thu Feb 1 04:20:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540618 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ej1-f68.google.com (mail-ej1-f68.google.com [209.85.218.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 006303BB34 for ; Thu, 1 Feb 2024 04:21:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761281; cv=none; b=s2ZVK+XO6L2Bx6dgY2lqvXn90A6TBR686lLUKq3OgMjY1Ljmd5byL0qEkzPWqWOw4mFcrIcHrKVHUhreU9kqLBsg3a/+Oipqpn6NPMkyS7k0yFtnX91J0FzwcG8OX4VTxadR9K+fxq6J5CQESHx5vkw3P5IGoNJM1vVgTILOTkI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761281; c=relaxed/simple; bh=WgtZY2YvuMhHbJ1b2SWiTNkGKSwbKaq2lGPQ6xAut+8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KtOhxFz5ikSCLfteWryMXx0xYLvxXhhYrUrbPZ0asu9dSKaqtCMYpRrQaNL2gxK8RtCqRwc/11fvyE8xZ7FWvfl8wDlqyj1c7DOIHETck4XFk9zw9+iQUYyHeIO0b/B768u+dLTjxt4OWzoZ2uHE8u8IEGFIKhtT5a1fpjd6uXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aQgxMiNm; arc=none smtp.client-ip=209.85.218.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aQgxMiNm" Received: by mail-ej1-f68.google.com with SMTP id a640c23a62f3a-a30e445602cso356762666b.0 for ; Wed, 31 Jan 2024 20:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761277; x=1707366077; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=q2tnvsDI26CUilziLHwen7viPO5j2bDTmFZaLGY88Jg=; b=aQgxMiNmAuDmHfZkfcnpyY8vEwdIzDwUyJsZv16YwUDkwNurnGhGi9AGJnHao+XStB uZMpQ6dlTVEww6X+plzYSylSpFfqTb92EW7lFkblZjakWl44riXVhM3kqXjjU5403Mj3 bLoTm0BD0IEwz0/hGrw0lPsO+KxXDUbN/xkY7T17hEENCWV8Yz7IHPTaKTmGULWCvGET JtPepAgqy4JkaWYOawpQ/nPEQHGojhcs3dnZLcuDbWOPdctPnGRqfKIEmkM9lAUnS17J nBvjUiuapMa9GLDFmc/GYOJCQYs2hYkF3sIiSrSqm2GWyJe9yEb4rTmVySLztR/NfX6a qfcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761277; x=1707366077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q2tnvsDI26CUilziLHwen7viPO5j2bDTmFZaLGY88Jg=; b=RpmLcjj4T/qiQjTlmOKMtAfS/iTBhIBZpH5N6MrZauzjZPHz3+VktsMhAM1qhoeNLG n42Jc3lnzsjar4hWqJWmXwWcnCInn4fldepikKc6d8EFwN+txGMAvCjHl6IF7CwDYhBw on39m/Eks6FU1hBU5XEF/lRbg8xjLLFZ/Zh8wLM8ogJ3T9NbZ504pPqN9Zy5KHNyYUvc RvZlenJs94zbA5SQqRzoIwERM7JplmIORt7UmnUnkn9Klp/Yt18B/rAHgM36CepLZDEI X7LO/z2lRTtRZbYP1oUI+7CNHR9FIPfkFPiLkuU9a2XQuG8HpjVXxErBNZAsQoS56FjC TKXw== X-Gm-Message-State: AOJu0YwivMBOrTyHoMlqPOtm6oRpHRlGRCkz5yMzauh+ZMdIiazEYzIV BjUmm7GtvQOu+N5ruC68Vl5UWY67T7W7vA2JsSN0vMVbLB25wKBFgSRCiT9Iz/s= X-Google-Smtp-Source: AGHT+IEOECYW6RGDIqgPeE3oD09pxICLk/zHpsv2QpRvQmbCnCWQ7h0rurM48ui4FD7kZ8zWV2EpIw== X-Received: by 2002:a17:907:a642:b0:a33:8fed:b9d5 with SMTP id vu2-20020a170907a64200b00a338fedb9d5mr5497765ejc.3.1706761277216; Wed, 31 Jan 2024 20:21:17 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id a10-20020a170906368a00b00a349d05c837sm6764829ejc.154.2024.01.31.20.21.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:16 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 04/14] bpf: Refactor check_pseudo_btf_id's BTF reference bump Date: Thu, 1 Feb 2024 04:20:59 +0000 Message-Id: <20240201042109.1150490-5-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4053; i=memxor@gmail.com; h=from:subject; bh=WgtZY2YvuMhHbJ1b2SWiTNkGKSwbKaq2lGPQ6xAut+8=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwN6qUqhyswMHWLYfdcQeWRDf3ejt/w7tEv3 N01PoKDNdmJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDQAKCRBM4MiGSL8R ynMJD/9IlIsxjBKmbPsqZNQAbZC0TX1pqXMo5ZlrS6eOhowrE+YNyMgw31NfdsbzIq8hEWYeHjH K9PowXBMkzKNZp3TCTkqEusxQHkY2ZPPX7eKay+qdwSa1eQWY/k3AV6dGZ4iVbNRQWxQt+m91Id 8HXRglnVegByaokyQmZoe7dH2sVol5zICdfJsDwwPq/hFFBPLLCxa3v+ihIIU+Au8VNaHhAqXd/ WrKn2wBNwl3M7yMb1xuEgNgyoMC9c2G5RqN7BzI8HqeWOAH3wVSWWNZnBoWOdYRG4n5mg7ed21K 1f/0rAW6hMLiFYhWnlqQMMGE2clHztEeqN6vz8a+DQ+UsdTYPe1Qyz6p0bgE4ZiSklxqKPm0Wyo E8IcWT71SBJEwKO+TNH9NkOLMurfvCCxhs3JBuE78wa2Qp7IliLJtemdh6lHvxopEnm4BCjX/aT P2dQ2EGUCaFQZYvuaeTvzlUe9qENnLttTf1zdz9YNRJnZ5Seg41MMU+jTrANn0usoXhzBVEMUVB cf2R7Nhx5RLf+QTquwG3eaFauoAkFe3xxbtVKbxKzl7HC+SSuXMw3Yn0RbSDQaZmIJjptWE7ePl vV/1+Wk7a+WKxed03aTDuQaEA9ROZRlKDySFht1MfX+crHNwjkK3mSEjh681blJQJtzFzGyWSg9 PGlwieycs9N0dTA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Refactor check_pseudo_btf_id's code which adds a new BTF reference to the used_btfs into a separate helper function called add_used_btfs. This will be later useful in exception frame generation to take BTF references with their modules, so that we can keep the modules alive whose functions may be required to unwind a given BPF program when it eventually throws an exception. While typically module references should already be held in such a case, since the program will have used a kfunc to acquire a reference that it did not clean up before throwing an exception, there are corner cases where this may not be true (e.g. one program producing the object, and another simply using bpf_kptr_xchg, and not having a kfunc call into the module). Therefore, it is more prudent to simply bump the reference whenever we encounter such cases for exception frame generation. The behaviour of add_used_btfs is to take an input BTF object with its reference count already raised, and the consume the reference count in case of successful insertion. In case of an error, the caller is responsible for releasing the reference. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman --- kernel/bpf/verifier.c | 70 ++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 622c638b123b..03ad9a9d47c9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17861,6 +17861,42 @@ static int find_btf_percpu_datasec(struct btf *btf) return -ENOENT; } +static int add_used_btf(struct bpf_verifier_env *env, struct btf *btf) +{ + struct btf_mod_pair *btf_mod; + int i, err; + + /* check whether we recorded this BTF (and maybe module) already */ + for (i = 0; i < env->used_btf_cnt; i++) { + if (env->used_btfs[i].btf == btf) { + btf_put(btf); + return 0; + } + } + + if (env->used_btf_cnt >= MAX_USED_BTFS) { + err = -E2BIG; + goto err; + } + + btf_mod = &env->used_btfs[env->used_btf_cnt]; + btf_mod->btf = btf; + btf_mod->module = NULL; + + /* if we reference variables from kernel module, bump its refcount */ + if (btf_is_module(btf)) { + btf_mod->module = btf_try_get_module(btf); + if (!btf_mod->module) { + err = -ENXIO; + goto err; + } + } + env->used_btf_cnt++; + return 0; +err: + return err; +} + /* replace pseudo btf_id with kernel symbol address */ static int check_pseudo_btf_id(struct bpf_verifier_env *env, struct bpf_insn *insn, @@ -17868,7 +17904,6 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env, { const struct btf_var_secinfo *vsi; const struct btf_type *datasec; - struct btf_mod_pair *btf_mod; const struct btf_type *t; const char *sym_name; bool percpu = false; @@ -17921,7 +17956,7 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env, if (btf_type_is_func(t)) { aux->btf_var.reg_type = PTR_TO_MEM | MEM_RDONLY; aux->btf_var.mem_size = 0; - goto check_btf; + goto add_btf; } datasec_id = find_btf_percpu_datasec(btf); @@ -17962,35 +17997,10 @@ static int check_pseudo_btf_id(struct bpf_verifier_env *env, aux->btf_var.btf = btf; aux->btf_var.btf_id = type; } -check_btf: - /* check whether we recorded this BTF (and maybe module) already */ - for (i = 0; i < env->used_btf_cnt; i++) { - if (env->used_btfs[i].btf == btf) { - btf_put(btf); - return 0; - } - } - - if (env->used_btf_cnt >= MAX_USED_BTFS) { - err = -E2BIG; +add_btf: + err = add_used_btf(env, btf); + if (err < 0) goto err_put; - } - - btf_mod = &env->used_btfs[env->used_btf_cnt]; - btf_mod->btf = btf; - btf_mod->module = NULL; - - /* if we reference variables from kernel module, bump its refcount */ - if (btf_is_module(btf)) { - btf_mod->module = btf_try_get_module(btf); - if (!btf_mod->module) { - err = -ENXIO; - goto err_put; - } - } - - env->used_btf_cnt++; - return 0; err_put: btf_put(btf); From patchwork Thu Feb 1 04:21:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540619 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lf1-f45.google.com (mail-lf1-f45.google.com [209.85.167.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E75983BB2C for ; Thu, 1 Feb 2024 04:21:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761284; cv=none; b=gz8zYoCp79MLKU71bHvntEx9BWnzEJ9CbSpnt8Ki9t2I6HAe3qgmh5UJ0sdTIkUB4TVldVPI5v60446aAxxvjYvL0NLcP2Wf17YeqfWhcEndE2q9yY8dzkIOSdsJT8KQS7PzDK2Q3ujJlKUZCFk9SHDu9LIfbN/B/sjENCamxrA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761284; c=relaxed/simple; bh=1lL77sT2um2zxl/ESTKSHiacgeljWABZA/qmoqPk6JY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TjBHDdvlhJL4kyMwJUoeP3L6mBHVBOKAusmuCgPxheWeaUKv2RtW2YKEO+q2syx8BX5yTnOPyUQmN69gfL9U59QG+f0Usyc2U5ZbCgG59+4QG4KM8veoOkq/hanV8zpmPdjipZMSQEIO097/6UIg49Iei43hi8qf2V7M+hHNo58= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RPEr8dT0; arc=none smtp.client-ip=209.85.167.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RPEr8dT0" Received: by mail-lf1-f45.google.com with SMTP id 2adb3069b0e04-51117bfd452so801781e87.3 for ; Wed, 31 Jan 2024 20:21:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761279; x=1707366079; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EkfJ/YtdzjljBnGu5bZEpyNJmG60w4t6Elqcq8KaSMs=; b=RPEr8dT0W9ihAaKDTTZ0o5H/nOsh8SE9JRb14MvOkaGqSy2QqsSnv/vTikrk+04S5L wNcYJ6wFSpi1gfd1t6wkxnznkKaQFoFnnNVnEbBWpDvI+dPdSuqTixfS6YiLEt44v+lA l7AeHtHjV9MJpnhM/n9nIZtSsNrmw32Y5WYpO41YtmHTolM3OvCw7VfdV/8z+DfXIqU6 f53guH86iztqedMLe0bMxLIf8tmTqURD92BUpKuQ9EPOwOYN36pds2I1QNW04P+UZD5b 1FHJW9OQ9pkOE6piXe2aXdvgRSSV1geIGizN7PjVy+OWHebdqQovzRXtLxQ8BaboBu+M luJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761279; x=1707366079; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EkfJ/YtdzjljBnGu5bZEpyNJmG60w4t6Elqcq8KaSMs=; b=X/3ijxYfjHf1Vv6ZhGouYSahO2zThDmHqvSPItEFk473ACnZrr1c/PYUHwu3rneY6a nSA2NFZLR0T8T71fMHvPLZXnaMjAVQchEYO9o0vlXb9Gr1kQXSs27Dx4hg6gZoBD6V4O EjusrJyl67MNMShPuLlMgIxTrgVBGYhI1KT1fcVizqGcQ9H7ZBF41d+0Xlc7gu6h8NlP 905W+b1/9uip037/iA+0VX4+g3+CGakZ5o+wf+aYGMkiaS/OLLh0/afRchqw1y4OJR1M jikoB3lBDXXHS9RlqHDbifMvdmmn9X/igGIygyJPscKm6/paD6TJrf4KDWAeKkzrnWb5 1qYQ== X-Gm-Message-State: AOJu0YxKsj1ABAVUitQ/+8Hs0BYkRec8XoAAbjyEROcMVJSA/rLg5qpr EWzN/KoKCO/8noT/jHhM7rC1/tE/Iy1mx79CMCzW953yBHW1JVJLY0OdySJc X-Google-Smtp-Source: AGHT+IGhWLjh7r68O/gLUqzbfqUsvlsm1S8N5A9SroHk3cDjQVUmZxlD5jNJ0mPIvhjaw0ID7YDs2Q== X-Received: by 2002:ac2:4147:0:b0:50f:ffc5:5cda with SMTP id c7-20020ac24147000000b0050fffc55cdamr915071lfi.57.1706761278850; Wed, 31 Jan 2024 20:21:18 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id vh9-20020a170907d38900b00a369b479982sm668576ejc.218.2024.01.31.20.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:17 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 05/14] bpf: Implement BPF exception frame descriptor generation Date: Thu, 1 Feb 2024 04:21:00 +0000 Message-Id: <20240201042109.1150490-6-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=22535; i=memxor@gmail.com; h=from:subject; bh=1lL77sT2um2zxl/ESTKSHiacgeljWABZA/qmoqPk6JY=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwNmVCEAFMr5ejxHCYWyMM+nNZPRp9gRidGs RNZ+xb0b+2JAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDQAKCRBM4MiGSL8R ysa5EAC7NhLKBYiYXalOe+qkOnrr6g3uUnK1zfnQAviKyghWMMjhBfK7LOjoIx0tEkJ0eRMrDFr HLB8t7vlbR6u/ei/OaMkALeX14mBVCUv+7LIWdMU/8kZOvOgK+Swzel5ClYYcbo9lkcxUx1jo7f XBSDnZy7Gz8uhEiWP7upTNwxynOU0lMexPczeY/CHycM+vRAdsPgiPDZSm1HA+Ll+YLCv1FoMu5 +MDuD4mEJGFbIS7QqdP7MHltBZ7gx4GgOXhVchiF5nIzRpx4Ia8D+H52VR3EDTVRp6yder/e0TI t36n4Yk1lJb0L2JPcRqyUlk+vB+Mm4kmtvhSZBlcllyOTBeb0rImDaEWZW+i5cQeQraVm3QGETd k2nZeNWvjH+He8CXWpiOdE9hURrkH7pIPGc/M59Gjy8GOUzaQuHL0n9n9HeJcXuH0xdhSqsPs7L 2b1glmNlmyehLgfExW888/s+A8hOHXsorVO1AK8SNb7ADf8Y7m33iU7nTnvlmgiXLSaC4qDe29A 7o+dD0aFPTIQXavwUQtBWZ7rydYdPv7Y7k2ofM28CuJPXdZ5vtn7DYix3CgAb03H52SmeZrgXZp /XP/WqAX1VILS09I0ZMJRmhj8xHeYOOXM2UE7WTH0ThYsnz+L6x4bB9jKtC5mvowI/Zs2q7/WUP deQbMYTHpBR068A== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Introduce support to the verifier to generate a set of descriptors for each BPF frame to describe the register and stack state, which can be used to reason about the resources acquired by the program at that particular program point, and subsequent patches can introduce support to unwind a given program when it throws an exception while holding ownership of such resources. Descriptors generated for each frame are then tied back to the subprog they belong to, and attached to the bpf_prog instance during the JIT phase, with the program counter serving as a key to find the descriptor of interest for a given subprog. Logically, during the unwinding phase, for each frame, we will use the program counter and bpf_prog object to figure out how we should release acquired resources if any in the frame. Let's study how the frame descriptor generation algorithm works. Whenever an exception throwing instruction is encountered, thus global subprog calls which are throw reachable, and bpf_throw kfunc, we call gen_exception_frame_descs. This function will start with the current frame, and explore the registers and other objects on the current stack. We consider 8-byte granularity as all registers spilled on the stack and objects like dynptr and iter are 8-byte aligned. For each such stack entry, we inspect the slot_type and figure out whether it is a spilled register or a dynptr/iter object. For any acquired resources on the stack, and insert entries representing them into a frame descriptor table for the current subprog at the current instruction index. The same steps are repeated for registers that are callee saved, as these would be possibly spilled on stack of one of the frames in the call chain and would have to be located in order to be freed. In case of registers (spilled or callee saved), we make a special provision for register_is_null scalar values, to increase the chances of merging frame descriptors where the only divergence is NULL in one state being replaced with a valid pointer in another. The next important step is the logic to merge the frame descriptors. It is possible that the verifier reaches the same instruction index in a program from multiple paths, and has to generate frame descriptors for them at that program counter. In such a case, we always ensure that after generating the frame descriptor, we attempt to "merge" it with an existing one. The merging rules are fairly simple except for a few caveats. First, if the layout and type of objects on the stack and in registers is the same, we have a successful merge. Next, in case of registers (spilled or callee saved), we have a special where if the old entry has NULL, the new type (non-NULL) replaces it, and if the new entry has NULL, it satisfies the merge rules with the old entry (can be of any type). This helps in cases where we have an optional value held in a register or stack slot in one program path, which is replaced by the actual value in the other program path. This can also be the case in case of conditionals, where the verifier may see acquired references in verifier state depending on if a condition is true (therefore, not in all of the program paths traversing the same instruction). To illustrate with an example, in the following program: struct foo *p = NULL; if (x) p = bpf_obj_new(typeof(*p)); if (y) bpf_throw(0); if (p) bpf_obj_drop(p); In such a case, bpf_throw may be reached for x == 0, y == 1 and x == 1, y == 1, with two possible values of p. As long as both can be passed into the release function (i.e. NULL or a valid pointer value), we can satisfy the merge. TODO: We need to reserve a slot for STACK_ZERO as well. TODO: Improve the error message in case we have pointer and misc instead of zero. Currently, we only consider resources which are modelled as acquired references in verifier state. In particular, this excludes resources like held spinlocks and RCU read sections. For now, both of these will not be handled, and the verifier will continue to complain when exceptions are thrown in their presence. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 27 ++ include/linux/bpf_verifier.h | 2 + kernel/bpf/core.c | 13 + kernel/bpf/verifier.c | 368 ++++++++++++++++++ .../selftests/bpf/progs/exceptions_fail.c | 4 +- 5 files changed, 412 insertions(+), 2 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 1ebbee1d648e..463c8d22ad72 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1424,6 +1424,7 @@ struct btf_mod_pair { }; struct bpf_kfunc_desc_tab; +struct bpf_exception_frame_desc_tab; struct bpf_prog_aux { atomic64_t refcnt; @@ -1518,6 +1519,7 @@ struct bpf_prog_aux { struct module *mod; u32 num_exentries; struct exception_table_entry *extable; + struct bpf_exception_frame_desc_tab *fdtab; union { struct work_struct work; struct rcu_head rcu; @@ -3367,4 +3369,29 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) return prog->aux->func_idx != 0; } +struct bpf_frame_desc_reg_entry { + u32 type; + s16 spill_type; + union { + s16 off; + u16 regno; + }; + struct btf *btf; + u32 btf_id; +}; + +struct bpf_exception_frame_desc { + u64 pc; + u32 stack_cnt; + struct bpf_frame_desc_reg_entry regs[4]; + struct bpf_frame_desc_reg_entry stack[]; +}; + +struct bpf_exception_frame_desc_tab { + u32 cnt; + struct bpf_exception_frame_desc **desc; +}; + +void bpf_exception_frame_desc_tab_free(struct bpf_exception_frame_desc_tab *fdtab); + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 5482701e6ad9..0113a3a940e2 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -631,6 +631,8 @@ struct bpf_subprog_info { u8 arg_cnt; struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS]; + + struct bpf_exception_frame_desc_tab *fdtab; }; struct bpf_verifier_env; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 71c459a51d9e..995a4dcfa970 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2734,6 +2734,14 @@ static void bpf_free_used_btfs(struct bpf_prog_aux *aux) kfree(aux->used_btfs); } +void bpf_exception_frame_desc_tab_free(struct bpf_exception_frame_desc_tab *fdtab) +{ + if (!fdtab) + return; + for (int i = 0; i < fdtab->cnt; i++) + kfree(fdtab->desc[i]); +} + static void bpf_prog_free_deferred(struct work_struct *work) { struct bpf_prog_aux *aux; @@ -2747,6 +2755,11 @@ static void bpf_prog_free_deferred(struct work_struct *work) if (aux->cgroup_atype != CGROUP_BPF_ATTACH_TYPE_INVALID) bpf_cgroup_atype_put(aux->cgroup_atype); #endif + /* Free all exception frame descriptors */ + for (int i = 0; i < aux->func_cnt; i++) { + bpf_exception_frame_desc_tab_free(aux->func[i]->aux->fdtab); + aux->func[i]->aux->fdtab = NULL; + } bpf_free_used_maps(aux); bpf_free_used_btfs(aux); if (bpf_prog_is_dev_bound(aux)) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 03ad9a9d47c9..27233c308d83 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10004,6 +10004,366 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta, return 0; } +static void print_frame_desc_reg_entry(struct bpf_verifier_env *env, struct bpf_frame_desc_reg_entry *fd, const char *pfx) +{ + const char *type = fd->off < 0 ? "stack" : "reg"; + const char *key = fd->off < 0 ? "off" : "regno"; + const char *spill_type; + + switch (fd->spill_type) { + case STACK_INVALID: + spill_type = ""; + break; + case STACK_SPILL: + spill_type = "reg"; + break; + case STACK_DYNPTR: + spill_type = "dynptr"; + break; + case STACK_ITER: + spill_type = "iter"; + break; + default: + spill_type = "???"; + break; + } + verbose(env, "frame_desc: %s%s fde: %s=%d spill_type=%s ", pfx, type, key, fd->off, spill_type); + if (fd->btf) { + const struct btf_type *t = btf_type_by_id(fd->btf, fd->btf_id); + + verbose(env, "type=%s%s btf=%s btf_id=%d\n", fd->off < 0 ? "" : "ptr_", + btf_name_by_offset(fd->btf, t->name_off), btf_get_name(fd->btf), fd->btf_id); + } else { + verbose(env, "type=%s\n", fd->spill_type == STACK_DYNPTR ? "ringbuf" : reg_type_str(env, fd->type)); + } +} + +static int merge_frame_desc(struct bpf_verifier_env *env, struct bpf_frame_desc_reg_entry *ofd, struct bpf_frame_desc_reg_entry *fd) +{ + int ofd_type, fd_type; + + /* If ofd->off/regno is 0, this is uninitialized reg entry, just merge new entry. */ + if (!ofd->off) + goto merge_new; + /* Exact merge for dynptr, iter, reg, stack. */ + if (!memcmp(ofd, fd, sizeof(*ofd))) + goto none; + /* First, for a successful merge, both spill_type should be same.*/ + if (ofd->spill_type != fd->spill_type) + goto fail; + /* Then, both should correspond to a reg or stack entry for non-exact merge. */ + if (ofd->spill_type != STACK_SPILL && ofd->spill_type != STACK_INVALID) + goto fail; + ofd_type = ofd->type; + fd_type = fd->type; + /* One of the old or new entry must be NULL, if both are not same. */ + if (ofd_type == fd_type) + goto none; + if (ofd_type != SCALAR_VALUE && fd_type != SCALAR_VALUE) + goto fail; + if (fd_type == SCALAR_VALUE) + goto none; + verbose(env, "frame_desc: merge: merging new frame desc entry into old\n"); + print_frame_desc_reg_entry(env, ofd, "old "); +merge_new: + if (!ofd->off) + verbose(env, "frame_desc: merge: creating new frame desc entry\n"); + print_frame_desc_reg_entry(env, fd, "new "); + *ofd = *fd; + return 0; +none: + verbose(env, "frame_desc: merge: no merging needed of new frame desc entry into old\n"); + print_frame_desc_reg_entry(env, ofd, "old "); + print_frame_desc_reg_entry(env, fd, "new "); + return 0; +fail: + verbose(env, "frame_desc: merge: failed to merge old and new frame desc entry\n"); + print_frame_desc_reg_entry(env, ofd, "old "); + print_frame_desc_reg_entry(env, fd, "new "); + return -EINVAL; +} + +static int find_and_merge_frame_desc(struct bpf_verifier_env *env, struct bpf_exception_frame_desc_tab *fdtab, u64 pc, struct bpf_frame_desc_reg_entry *fd) +{ + struct bpf_exception_frame_desc **descs = NULL, *desc = NULL, *p; + int ret = 0; + + for (int i = 0; i < fdtab->cnt; i++) { + if (pc != fdtab->desc[i]->pc) + continue; + descs = &fdtab->desc[i]; + desc = fdtab->desc[i]; + break; + } + + if (!desc) { + verbose(env, "frame_desc: find_and_merge: cannot find frame descriptor for pc=%llu, creating new entry\n", pc); + return -ENOENT; + } + + if (fd->off < 0) + goto stack; + /* We didn't find a match for regno or offset, fill it into the frame descriptor. */ + return merge_frame_desc(env, &desc->regs[fd->regno - BPF_REG_6], fd); + +stack: + for (int i = 0; i < desc->stack_cnt; i++) { + struct bpf_frame_desc_reg_entry *ofd = desc->stack + i; + + if (ofd->off != fd->off) + continue; + ret = merge_frame_desc(env, ofd, fd); + if (ret < 0) + return ret; + return 0; + } + p = krealloc(desc, offsetof(typeof(*desc), stack[desc->stack_cnt + 1]), GFP_USER | __GFP_ZERO); + if (!p) { + return -ENOMEM; + } + verbose(env, "frame_desc: merge: creating new frame desc entry\n"); + print_frame_desc_reg_entry(env, fd, "new "); + desc = p; + desc->stack[desc->stack_cnt] = *fd; + desc->stack_cnt++; + *descs = desc; + return 0; +} + +/* Implementation details: + * This function is responsible for pushing a prepared bpf_frame_desc_reg_entry + * into the frame descriptor array tied to each subprog. The first step is + * ensuring the array is allocated and has enough capacity. Second, we must find + * if there is an existing descriptor already for the program counter under + * consideration, and try to report an error if we see conflicting frame + * descriptor generation requests for the same instruction in the program. + * Note that by default, we let NULL registers and stack slots occupy an entry. + * This is done so that any future non-NULL registers or stack slots at the same + * regno or offset can be satisfied by changing the type of entry to a "stronger" + * pointer type. The release handler can deal with NULL or valid values, + * therefore such a logic allows handling cases where the program may only have + * a pointer in some of the program paths and NULL in others while reaching the + * same instruction that causes an exception to be thrown. + * Likewise, a NULL entry merges into the stronger pointer type entry when a + * frame descriptor already exists before pushing a new one. + */ +static int push_exception_frame_desc(struct bpf_verifier_env *env, int frameno, struct bpf_frame_desc_reg_entry *fd) +{ + struct bpf_func_state *frame = env->cur_state->frame[frameno], *curframe = cur_func(env); + struct bpf_subprog_info *si = subprog_info(env, frame->subprogno); + struct bpf_exception_frame_desc_tab *fdtab = si->fdtab; + struct bpf_exception_frame_desc **desc; + u64 pc = env->insn_idx; + int ret; + + /* If this is not the current frame, then we need to figure out the callsite + * for its callee to identify the pc. + */ + if (frameno != curframe->frameno) + pc = env->cur_state->frame[frameno + 1]->callsite; + + if (!fdtab) { + fdtab = kzalloc(sizeof(*si->fdtab), GFP_USER); + if (!fdtab) + return -ENOMEM; + fdtab->desc = kzalloc(sizeof(*fdtab->desc), GFP_USER); + if (!fdtab->desc) { + kfree(fdtab); + return -ENOMEM; + } + si->fdtab = fdtab; + } + + ret = find_and_merge_frame_desc(env, fdtab, pc, fd); + if (!ret) + return 0; + if (ret < 0 && ret != -ENOENT) + return ret; + /* We didn't find a frame descriptor for pc, grow the array and insert it. */ + desc = realloc_array(fdtab->desc, fdtab->cnt ?: 1, fdtab->cnt + 1, sizeof(*fdtab->desc)); + if (!desc) { + return -ENOMEM; + } + fdtab->desc = desc; + fdtab->desc[fdtab->cnt] = kzalloc(sizeof(*fdtab->desc[0]), GFP_USER); + if (!fdtab->desc[fdtab->cnt]) + return -ENOMEM; + fdtab->desc[fdtab->cnt]->pc = pc; + fdtab->cnt++; + return find_and_merge_frame_desc(env, fdtab, pc, fd); +} + +static int gen_exception_frame_desc_reg_entry(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off, int frameno) +{ + struct bpf_frame_desc_reg_entry fd = {}; + + if ((!reg->ref_obj_id && reg->type != NOT_INIT) || reg->type == SCALAR_VALUE) + return 0; + if (base_type(reg->type) == PTR_TO_BTF_ID) + fd.btf = reg->btf; + fd.type = reg->type & ~PTR_MAYBE_NULL; + fd.btf_id = fd.btf ? reg->btf_id : 0; + fd.spill_type = off < 0 ? STACK_SPILL : STACK_INVALID; + fd.off = off; + verbose(env, "frame_desc: frame%d: insn_idx=%d %s=%d size=%d ref_obj_id=%d type=%s\n", + frameno, env->insn_idx, off < 0 ? "off" : "regno", off, BPF_REG_SIZE, reg->ref_obj_id, reg_type_str(env, reg->type)); + return push_exception_frame_desc(env, frameno, &fd); +} + +static int gen_exception_frame_desc_dynptr_entry(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off, int frameno) +{ + struct bpf_frame_desc_reg_entry fd = {}; + int type = reg->dynptr.type; + + /* We only need to generate an entry when the dynptr is refcounted, + * otherwise it encapsulates no resource that needs to be released. + */ + if (!dynptr_type_refcounted(type)) + return 0; + switch (type) { + case BPF_DYNPTR_TYPE_RINGBUF: + fd.type = BPF_DYNPTR_TYPE_RINGBUF; + fd.spill_type = STACK_DYNPTR; + fd.off = off; + verbose(env, "frame_desc: frame%d: insn_idx=%d off=%d size=%lu dynptr_ringbuf\n", frameno, env->insn_idx, off, + BPF_DYNPTR_NR_SLOTS * BPF_REG_SIZE); + break; + default: + verbose(env, "verifier internal error: refcounted dynptr type unhandled for exception frame descriptor entry\n"); + return -EFAULT; + } + return push_exception_frame_desc(env, frameno, &fd); +} + +static int add_used_btf(struct bpf_verifier_env *env, struct btf *btf); + +static int gen_exception_frame_desc_iter_entry(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int off, int frameno) +{ + struct bpf_frame_desc_reg_entry fd = {}; + struct btf *btf = reg->iter.btf; + u32 btf_id = reg->iter.btf_id; + const struct btf_type *t; + int ret; + + fd.btf = btf; + fd.type = reg->type; + fd.btf_id = btf_id; + fd.spill_type = STACK_ITER; + fd.off = off; + t = btf_type_by_id(btf, btf_id); + verbose(env, "frame_desc: frame%d: insn_idx=%d off=%d size=%u ref_obj_id=%d iter_%s\n", + frameno, env->insn_idx, off, t->size, reg->ref_obj_id, btf_name_by_offset(btf, t->name_off)); + btf_get(btf); + ret = add_used_btf(env, btf); + if (ret < 0) { + btf_put(btf); + return ret; + } + return push_exception_frame_desc(env, frameno, &fd); +} + +static int gen_exception_frame_desc_stack_entry(struct bpf_verifier_env *env, struct bpf_func_state *frame, int stack_off) +{ + int spi = stack_off / BPF_REG_SIZE, off = -stack_off - 1; + struct bpf_reg_state *reg, not_init_reg, null_reg; + int slot_type, ret; + + __mark_reg_not_init(env, ¬_init_reg); + __mark_reg_known_zero(&null_reg); + + slot_type = frame->stack[spi].slot_type[BPF_REG_SIZE - 1]; + reg = &frame->stack[spi].spilled_ptr; + + switch (slot_type) { + case STACK_SPILL: + /* We skip all kinds of scalar registers, except NULL values, which consume a slot. */ + if (is_spilled_scalar_reg(&frame->stack[spi]) && !register_is_null(&frame->stack[spi].spilled_ptr)) + break; + ret = gen_exception_frame_desc_reg_entry(env, reg, off, frame->frameno); + if (ret < 0) + return ret; + break; + case STACK_DYNPTR: + /* Keep iterating until we find the first slot. */ + if (!reg->dynptr.first_slot) + break; + ret = gen_exception_frame_desc_dynptr_entry(env, reg, off, frame->frameno); + if (ret < 0) + return ret; + break; + case STACK_ITER: + /* Keep iterating until we find the first slot. */ + if (!reg->ref_obj_id) + break; + ret = gen_exception_frame_desc_iter_entry(env, reg, off, frame->frameno); + if (ret < 0) + return ret; + break; + case STACK_MISC: + case STACK_INVALID: + /* Create an invalid entry for MISC and INVALID */ + ret = gen_exception_frame_desc_reg_entry(env, ¬_init_reg, off, frame->frameno); + if (ret < 0) + return 0; + break; + case STACK_ZERO: + reg = &null_reg; + for (int i = BPF_REG_SIZE - 1; i >= 0; i--) { + if (frame->stack[spi].slot_type[i] != STACK_ZERO) + reg = ¬_init_reg; + } + ret = gen_exception_frame_desc_reg_entry(env, &null_reg, off, frame->frameno); + if (ret < 0) + return ret; + break; + default: + verbose(env, "verifier internal error: frame%d stack off=%d slot_type=%d missing handling for exception frame generation\n", + frame->frameno, off, slot_type); + return -EFAULT; + } + return 0; +} + +/* We generate exception descriptors for all frames at the current program + * counter. For caller frames, we use their callsite as their program counter, + * and we go on generating it until the main frame. + * + * It's necessary to detect whether the stack layout is different, in that case + * frame descriptor generation should fail and we cannot really support runtime + * unwinding in that case. + */ +static int gen_exception_frame_descs(struct bpf_verifier_env *env) +{ + struct bpf_reg_state not_init_reg; + int ret; + + __mark_reg_not_init(env, ¬_init_reg); + + for (int frameno = env->cur_state->curframe; frameno >= 0; frameno--) { + struct bpf_func_state *frame = env->cur_state->frame[frameno]; + + verbose(env, "frame_desc: frame%d: Stack:\n", frameno); + for (int i = BPF_REG_SIZE - 1; i < frame->allocated_stack; i += BPF_REG_SIZE) { + ret = gen_exception_frame_desc_stack_entry(env, frame, i); + if (ret < 0) + return ret; + } + + verbose(env, "frame_desc: frame%d: Registers:\n", frameno); + for (int i = BPF_REG_6; i < BPF_REG_FP; i++) { + struct bpf_reg_state *reg = &frame->regs[i]; + + /* Treat havoc scalars as incompatible type. */ + if (reg->type == SCALAR_VALUE && !register_is_null(reg)) + reg = ¬_init_reg; + ret = gen_exception_frame_desc_reg_entry(env, reg, i, frame->frameno); + if (ret < 0) + return ret; + } + } + return 0; +} + static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exit) { struct bpf_func_state *state = cur_func(env); @@ -17694,12 +18054,18 @@ static int do_check(struct bpf_verifier_env *env) err = check_func_call(env, insn, &env->insn_idx); if (!err && env->cur_state->global_subprog_call_exception) { env->cur_state->global_subprog_call_exception = false; + err = gen_exception_frame_descs(env); + if (err < 0) + return err; exception_exit = true; goto process_bpf_exit_full; } } else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { err = check_kfunc_call(env, insn, &env->insn_idx); if (!err && is_bpf_throw_kfunc(insn)) { + err = gen_exception_frame_descs(env); + if (err < 0) + return err; exception_exit = true; goto process_bpf_exit_full; } @@ -21184,6 +21550,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3 mutex_unlock(&bpf_verifier_lock); vfree(env->insn_aux_data); err_free_env: + for (int i = 0; i < env->subprog_cnt; i++) + bpf_exception_frame_desc_tab_free(env->subprog_info[i].fdtab); kfree(env); return ret; } diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index 28602f905d7d..5a517065b4e6 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -213,7 +213,7 @@ __noinline static int subprog_cb_ref(u32 i, void *ctx) } SEC("?tc") -__failure __msg("Unreleased reference") +__failure __msg("cannot be called from callback subprog 1") int reject_with_cb_reference(void *ctx) { struct foo *f; @@ -235,7 +235,7 @@ int reject_with_cb(void *ctx) } SEC("?tc") -__failure __msg("Unreleased reference") +__success int reject_with_subprog_reference(void *ctx) { return subprog_ref(ctx) + 1; From patchwork Thu Feb 1 04:21:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540620 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F30D73B78D for ; Thu, 1 Feb 2024 04:21:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761284; cv=none; b=isIs8Owq+3fEqmor6sWgttPAfc5fw+ga2wVLUTno65q8dLn/lHWZjA1+mzKE/g1VDOKlixCcw99YlECk06jN6H0uM09SioikYY/uJak0AazYAp/ziktdI9QDRKzzxioAw20ULBH7nxr2NVYWq0/NrzUkMVzOlqimsDtswtUDGCE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761284; c=relaxed/simple; bh=9/c2AaKHS4Ris+0Lp1KuGSpm1h7kSRvJFZgOzePFslo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EUx2CIDilLvGKO+COgJ1sAXeoe2a4yZjYDOcNQUxfCaOn5zxyI+RzKj0qF25jl5HQyXqprxpcLoS7W8e1iukQTDRX1zlC0QlIibxo8PoKAwc1DGVBcCaJGIGLCsA+1O5mz59trSvkQAmfKdz7wEzI/i2uiMtvX9mCBvCSVA/kY4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iJHxVyda; arc=none smtp.client-ip=209.85.208.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iJHxVyda" Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-2cf4fafa386so7337031fa.1 for ; Wed, 31 Jan 2024 20:21:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761280; x=1707366080; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eSDuCib4mmoNQQ1LinjjB6FAjxnk/c1B43QuLBvcuFA=; b=iJHxVydaUbEGQrmXYXvSH1mZouDL8UeYESFKBE1OJfytf+Z2kxGDk1Pmi4erd7Hy+C M0IBBH69XTcr/DNDXOCw+k08tdiJXpCAcQPzdU86cXL3U04HTh8c6tyrFU1CUMWvX0US RMAW8mgS9P5lFcUlwe5lZaQoR0jhGaZYphjDLZif9roO1IrxbBcod0F/NyaCYTJpzqUM zHRLqQ0sXN9kJ0xc7eHI+2jKVsgdKnHIz/8ROVZk9BjmtSAYL06igTA1vIRTrnvwC54S qumtravtDFYDL3FbtO9DtH272Mbzd8wOGkr1h4g/Mbj+4l9CmgX4Kz9uSwOaM09LMalK 0mHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761280; x=1707366080; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eSDuCib4mmoNQQ1LinjjB6FAjxnk/c1B43QuLBvcuFA=; b=ds0p7xTpodAJ6Hz3kSRseubq0ni4V4D65zWPCd+P4sOb2ythBmNVeR/P/mKrP0Tlkw MW47x3p8okFoIL21biS1RxwTGZz2s0bOVm0MV0+mmaPpKy4jMfoQLpCRg/JdQ2B4gmlG 4OVUq5haNFQPQTdIfkfPZW2+5RvwyjlCuO8FhwNgoOAoDaiCqwM6ZYD/a4DimPRMZAv+ hoT0t95+91WL7vY8D2oGU3QENaMeTx6e1tniX1tFmM0kkPtvXiq7eLPwXr3AWIl7dA9k Q1e2o5OFV5GxLNVITzNny+Fy+T9PHIIlGr4vXzLgmyHRkGngKFMbKAYU3uqBbeSVWarO GgaQ== X-Gm-Message-State: AOJu0YwoRg+805257EiZsVrLY5zpvIMdvQAHByyhMBVqm2Vb40y9EIVw iTxgdSQygIZYgfn/4QtpQG2w3qkxJLKlBTBkDxcl91QNzZEoyne+Ey15I5eu X-Google-Smtp-Source: AGHT+IHlgGw8qWZBKXQzHTOrU3Gi0P0x7yRVRFneUrWVMrVK8IaULW9onzfwsCwj+xwabS3MColu/g== X-Received: by 2002:a2e:b710:0:b0:2ce:d23:ec79 with SMTP id j16-20020a2eb710000000b002ce0d23ec79mr2296691ljo.40.1706761280361; Wed, 31 Jan 2024 20:21:20 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id dj10-20020a05640231aa00b0055f1717216fsm3128556edb.51.2024.01.31.20.21.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:19 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 06/14] bpf: Adjust frame descriptor pc on instruction patching Date: Thu, 1 Feb 2024 04:21:01 +0000 Message-Id: <20240201042109.1150490-7-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3514; i=memxor@gmail.com; h=from:subject; bh=9/c2AaKHS4Ris+0Lp1KuGSpm1h7kSRvJFZgOzePFslo=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwOjo/557iiZfLQt9gP7hKhrM0CW/SNrKTSf 262qaTTgfeJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDgAKCRBM4MiGSL8R yvuOD/9Koul7sAMyZpfRg6Sv1Dy3G1xBkYpiUZW4zkKtw/aTXA0HDEJTWQcJTNWSzwYEEOkeJyX zIoLruijYDJCCRu6kCiBf61lI002nh6HeKzjYvREsdZ6xDp32cny/5kYnySm/kCk/cDjWxJfvUJ 5dlYb0JRhGT1QvraGdWrm1bcZenptmD2g7Old3hREI5Jd2aNcHReEwUwX1I/puseyGaMRg5jiAK w+dzHyU5+FoH5q85N+IrT14jdDGdLsbohCBq+MuO9o7SKd+VsQi7WVDMF40nT3vVlv17vDFBln1 sQG8rqNBf2uCUcSeJUF5f0PdradLMfShB0PZJoo2AUYo2iR/ogBxAqatKPT8/PZLBbVUayccE30 vHsmyGojIS80gMUL4gYv1TQpbhiXatHWyP+1RTntbnUa8gJtVBVVNeyqBShAhryZ9nj1A/SuyX2 T4r3SBBffQ+pOrAUN2UYyrJgzLflkpz5G9FVtUC/RzDwEuF3+KFJPexVTxF09TenHZ1bA3hZaE6 i3sX+9bPM7NAOj/Sax3fOn1MCX7O2kHtEcE5NGwp6Yb4zHD9vkDyqZrVMX63blO5KHvFVwvYIRB agQ2w8sw5ebhKlWwikDY0EZHt3cCKCJ1xFaAfwltNmHe/lzQpr1QUwRUcURxMwDJptdURCEy+pv qSnK8jIe5+QcimQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC When instruction patching (addition or removal) occurs, the fdtab attached to each subprog, and the program counter in its descriptors will be out of sync wrt relative position in the program. To fix this, we need to adjust the pc, free any unneeded fdtab and descriptors, and ensure the entries correspond to the correct instruction offsets. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 50 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 27233c308d83..e5b1db1db679 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -18737,6 +18737,23 @@ static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len } } +static void adjust_subprog_frame_descs(struct bpf_verifier_env *env, u32 off, u32 len) +{ + if (len == 1) + return; + for (int i = 0; i <= env->subprog_cnt; i++) { + struct bpf_exception_frame_desc_tab *fdtab = subprog_info(env, i)->fdtab; + + if (!fdtab) + continue; + for (int j = 0; j < fdtab->cnt; j++) { + if (fdtab->desc[j]->pc <= off) + continue; + fdtab->desc[j]->pc += len - 1; + } + } +} + static void adjust_poke_descs(struct bpf_prog *prog, u32 off, u32 len) { struct bpf_jit_poke_descriptor *tab = prog->aux->poke_tab; @@ -18775,6 +18792,7 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of } adjust_insn_aux_data(env, new_data, new_prog, off, len); adjust_subprog_starts(env, off, len); + adjust_subprog_frame_descs(env, off, len); adjust_poke_descs(new_prog, off, len); return new_prog; } @@ -18805,6 +18823,10 @@ static int adjust_subprog_starts_after_remove(struct bpf_verifier_env *env, /* move fake 'exit' subprog as well */ move = env->subprog_cnt + 1 - j; + /* Free fdtab for subprog_info that we are going to destroy. */ + for (int k = i; k < j; k++) + bpf_exception_frame_desc_tab_free(env->subprog_info[k].fdtab); + memmove(env->subprog_info + i, env->subprog_info + j, sizeof(*env->subprog_info) * move); @@ -18835,6 +18857,30 @@ static int adjust_subprog_starts_after_remove(struct bpf_verifier_env *env, return 0; } +static int adjust_subprog_frame_descs_after_remove(struct bpf_verifier_env *env, u32 off, u32 cnt) +{ + for (int i = 0; i < env->subprog_cnt; i++) { + struct bpf_exception_frame_desc_tab *fdtab = subprog_info(env, i)->fdtab; + + if (!fdtab) + continue; + for (int j = 0; j < fdtab->cnt; j++) { + /* Part of a subprog_info whose instructions were removed partially, but the fdtab remained. */ + if (fdtab->desc[j]->pc >= off && fdtab->desc[j]->pc < off + cnt) { + void *p = fdtab->desc[j]; + if (j < fdtab->cnt - 1) + memmove(fdtab->desc + j, fdtab->desc + j + 1, sizeof(fdtab->desc[0]) * (fdtab->cnt - j - 1)); + kfree(p); + fdtab->cnt--; + j--; + } + if (fdtab->desc[j]->pc >= off + cnt) + fdtab->desc[j]->pc -= cnt; + } + } + return 0; +} + static int bpf_adj_linfo_after_remove(struct bpf_verifier_env *env, u32 off, u32 cnt) { @@ -18916,6 +18962,10 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt) if (err) return err; + err = adjust_subprog_frame_descs_after_remove(env, off, cnt); + if (err) + return err; + err = bpf_adj_linfo_after_remove(env, off, cnt); if (err) return err; From patchwork Thu Feb 1 04:21:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540621 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7689B1EF15 for ; Thu, 1 Feb 2024 04:21:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761286; cv=none; b=ZtIYRiniGAQZw+WAriwzKXvkxaDFtZAn9bRLOtHG6sMK86pT5UEXxu7TWpV0QJ67IL1v/fvWuRfil9suarC4TtT6ULl1nRR+wqJKaljUGEj9hJdGG+ZUQEnsP1heuUDLeNTadx104H7aHygrZUXjh5LkpMp9B7aW2GRsqmACjW8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761286; c=relaxed/simple; bh=W6f6MmNaMQ4brkjN81MYPDjeQ4XbWi6k+9bKkOT9ytQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lJD2fqdWgjnEmdyoNXoKaBcT++SFTWqDysXJ2MxVu2dR2I39WWMS9ZW1/Pr4fm9BVEOskmDo44NvuLRPFK9iKL45TDY94gRyt+NfId3a0ijXL9Zu0hTs+kFI/7zeAX2bOg/e7jfQ9IZy54ncz1+hbwoSnoX3YiaDPqKOygHzwtI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Z89KRO/G; arc=none smtp.client-ip=209.85.208.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Z89KRO/G" Received: by mail-lj1-f178.google.com with SMTP id 38308e7fff4ca-2d073b54359so4412701fa.0 for ; Wed, 31 Jan 2024 20:21:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761282; x=1707366082; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7cCmbC7PpeJvrJ88sWzeovZ1efioD+wJX0ZxsMhoJQs=; b=Z89KRO/GyBTwFzRPGBWRP2Q1bN9ZxydC0bVPr/ekZ3WaOkKWjKh/JzYlZsZctnMZy2 WisG1yNBvmEBXKJfw9WE1QoHDgBHWEfMApJ6hiXoJuuvIMdmog85TlZ9mD8SAwOCf3ZX d34L9+kvZLlIFvmcKG138PfqOJWHyROWada/ZUWPD8iJBp3kIIc8WPvHm9SB+OFWMqKn EslVsbZSvRDn6bDQi34nxdrcXrBNoeOTHW4Yz7dmngEW7paVFx+0P9nNuK/vmFr9VJwK y3PHyAQ8Al5CawU4M8GTazYOC5of7nKQw1un1RG+vALj5W99rQpUkT9M7Br3UNRmMPop plXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761282; x=1707366082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7cCmbC7PpeJvrJ88sWzeovZ1efioD+wJX0ZxsMhoJQs=; b=g3ocYfCVlgmZaQg/uxS0hizgJNukfnUEaV2JkQikJ87gYZjRN4TzAvhU39+ymCvaGv C8apyXPTX1KFiXXdKVDeEB++inNJGztR6L0HCbBUWc5PauZfNHADNHRzLboIHE9U86XC M6MvtRTgbrx/oW4mdDRdxL/26lXeCx10ZZhwxLAuJQRVOazUVlMkeyLIZ+TtUValNzAS TkuHETcSbiFYR+IJFmr1CQel/Uic7Qf9dHL8e8JQq5nOnHac5HS2M2c+9a/EiTcSPm6O 4c8sWggkuvYtB6rDVTI9dnPs+GTX887TRILptQ/qQSr0gjwTxSV5+GNaUbzFKYkrQm7l BVyQ== X-Gm-Message-State: AOJu0YyQxVxp013ZSW4nbZ61EyGdk+1TLaEL2zM9EFVi6DCneM1HXC0M TocdzTfaAvxe8u1unQha9fDCUSHVYtRWCUrpJ9I17b29QJda9sjhg9A+D7VA X-Google-Smtp-Source: AGHT+IFLl36dXTnVDQe5feu1j4iZS9M1vZZAVY1YPH5qr3iXrwO5hLb+0XKRV1143rPliHDEXatcAw== X-Received: by 2002:a05:651c:1242:b0:2cf:4e53:7c38 with SMTP id h2-20020a05651c124200b002cf4e537c38mr2266863ljh.22.1706761281756; Wed, 31 Jan 2024 20:21:21 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id a20-20020aa7d754000000b0055efeee7722sm3658680eds.79.2024.01.31.20.21.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:21 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 07/14] bpf: Use hidden subprog trampoline for bpf_throw Date: Thu, 1 Feb 2024 04:21:02 +0000 Message-Id: <20240201042109.1150490-8-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9267; i=memxor@gmail.com; h=from:subject; bh=W6f6MmNaMQ4brkjN81MYPDjeQ4XbWi6k+9bKkOT9ytQ=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwOPIXJJ5Pvyg5Rn414zDGYd6lVBAbcLB0aI 9PO5o0QNw2JAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDgAKCRBM4MiGSL8R ygVQD/9P5T42bULV9be2trljRF/kueo/0S6Vx4zW0CjjkYUk28qkIg9FfnzXf92i0M35vfeuke0 b3tnhPNddx8aSM3DPE6eIczVrg/HMsVQevA3C6XUmQPr8v173lD01O4geZqSWbNsHG9Tz9721NV wpgrDkpAsetoXAVD8+eYFVv1YueZ46LBZI1T8M/CcG9s8Wm3zJiUhKifTgRNAOepmlA2U+lPaZF kQxMgTid2Wn9YcSSwYF3dtf5YTly7HJTWwC19RYXZ2PdEaGJonrPV8I5b8EEEyI081iAgfjaaqu Tz5BaG7jc65TNJc/krwMBRP4d99LYHIQ7fnqYaZzF2oLAHLrJYARVpMfryPgPDnyG2oNgwMyw/3 m2hFoUdhuNhUezwv7X1VLWtJ4M/MQiYaaAD7aavRkcvVQftHShe1sowHzlxzEOBW7Zryq43epTo fJpbLYwcXcmrb3Tv0AvLSVcnGXCSvt6usbqSonggNA1UvvsUZI8/51guCka+yNJnpHJ8UKpEoXg 3HVMxUSi41ROtbOQX+sMi3s9WYBPJnL+diAVyrOFjpDAFHNbLy1Q+3tPDgXVNJBL570SNDt/8pX 0fPVvMXNxBUCwaW8xPycAmiXfs5TPZdnn5XJTqKSqp4gqLQvGM0KjUF7Cc9jrPREDERsa6nXxdJ LuWt89G1cXTe/lA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC When we perform a bpf_throw kfunc call, callee saved registers in BPF calling convention (R6-R9) may end up getting saved and clobbered by bpf_throw. Typically, the kernel will restore the registers before returning back to the BPF program, but in case of bpf_throw, the function will never return. Therefore, any acquired resources sitting in these registers will end up getting destroyed if not saved on the stack, without any cleanup happening for them. Also, when in a BPF call chain, caller frames may have held acquired resources in R6-R9 and called their subprogs, which may have spilled these on their stack frame to reuse these registers before entering the bpf_throw kfunc. Thus, we also need to locate and free these callee saved registers for each frame. It is thus necessary to save these registers somewhere before we call into the bpf_throw kfunc. Instead of adding spills everywhere bpf_throw is called, we can use a new hidden subprog that saves R6-R9 on the stack and then calls into bpf_throw. This way, all of the bpf_throw call sites can be turned into call instructions for this subprog, and the hidden subprog in turn will save the callee-saved registers before calling into the bpf_throw kfunc. In this way, when unwinding the stack, we can locate the callee saved registers on the hidden subprog stack frame and perform their cleanup. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 24 ++++++++-------- include/linux/bpf.h | 5 ++++ include/linux/bpf_verifier.h | 3 +- kernel/bpf/verifier.c | 55 ++++++++++++++++++++++++++++++++++-- 4 files changed, 71 insertions(+), 16 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index e1390d1e331b..87692d983ffd 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -640,9 +640,10 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog, offset = ctx->tail_call_indirect_label - (prog + 2 - start); EMIT2(X86_JE, offset); /* je out */ - if (bpf_prog->aux->exception_boundary) { + if (bpf_prog->aux->exception_boundary || bpf_prog->aux->bpf_throw_tramp) { pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + if (bpf_prog->aux->exception_boundary) + pop_r12(&prog); } else { pop_callee_regs(&prog, callee_regs_used); } @@ -699,9 +700,10 @@ static void emit_bpf_tail_call_direct(struct bpf_prog *bpf_prog, emit_jump(&prog, (u8 *)poke->tailcall_target + X86_PATCH_SIZE, poke->tailcall_bypass); - if (bpf_prog->aux->exception_boundary) { + if (bpf_prog->aux->exception_boundary || bpf_prog->aux->bpf_throw_tramp) { pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + if (bpf_prog->aux->exception_boundary) + pop_r12(&prog); } else { pop_callee_regs(&prog, callee_regs_used); } @@ -1164,12 +1166,9 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image /* Exception callback will clobber callee regs for its own use, and * restore the original callee regs from main prog's stack frame. */ - if (bpf_prog->aux->exception_boundary) { - /* We also need to save r12, which is not mapped to any BPF - * register, as we throw after entry into the kernel, which may - * overwrite r12. - */ - push_r12(&prog); + if (bpf_prog->aux->exception_boundary || bpf_prog->aux->bpf_throw_tramp) { + if (bpf_prog->aux->exception_boundary) + push_r12(&prog); push_callee_regs(&prog, all_callee_regs_used); } else { push_callee_regs(&prog, callee_regs_used); @@ -2031,9 +2030,10 @@ st: if (is_imm8(insn->off)) seen_exit = true; /* Update cleanup_addr */ ctx->cleanup_addr = proglen; - if (bpf_prog->aux->exception_boundary) { + if (bpf_prog->aux->exception_boundary || bpf_prog->aux->bpf_throw_tramp) { pop_callee_regs(&prog, all_callee_regs_used); - pop_r12(&prog); + if (bpf_prog->aux->exception_boundary) + pop_r12(&prog); } else { pop_callee_regs(&prog, callee_regs_used); } diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 463c8d22ad72..83cff18a1b66 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3369,6 +3369,11 @@ static inline bool bpf_is_subprog(const struct bpf_prog *prog) return prog->aux->func_idx != 0; } +static inline bool bpf_is_hidden_subprog(const struct bpf_prog *prog) +{ + return prog->aux->func_idx >= prog->aux->func_cnt; +} + struct bpf_frame_desc_reg_entry { u32 type; s16 spill_type; diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 0113a3a940e2..04e27fce33d6 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -683,6 +683,7 @@ struct bpf_verifier_env { u32 id_gen; /* used to generate unique reg IDs */ u32 hidden_subprog_cnt; /* number of hidden subprogs */ int exception_callback_subprog; + int bpf_throw_tramp_subprog; bool explore_alu_limits; bool allow_ptr_leaks; /* Allow access to uninitialized stack memory. Writes with fixed offset are @@ -699,7 +700,7 @@ struct bpf_verifier_env { struct bpf_insn_aux_data *insn_aux_data; /* array of per-insn state */ const struct bpf_line_info *prev_linfo; struct bpf_verifier_log log; - struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 2]; /* max + 2 for the fake and exception subprogs */ + struct bpf_subprog_info subprog_info[BPF_MAX_SUBPROGS + 3]; /* max + 3 for the fake, exception, and bpf_throw_tramp subprogs */ union { struct bpf_idmap idmap_scratch; struct bpf_idset idset_scratch; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e5b1db1db679..942243cba9f1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19836,9 +19836,9 @@ static int add_hidden_subprog(struct bpf_verifier_env *env, struct bpf_insn *pat int cnt = env->subprog_cnt; struct bpf_prog *prog; - /* We only reserve one slot for hidden subprogs in subprog_info. */ - if (env->hidden_subprog_cnt) { - verbose(env, "verifier internal error: only one hidden subprog supported\n"); + /* We only reserve two slots for hidden subprogs in subprog_info. */ + if (env->hidden_subprog_cnt == 2) { + verbose(env, "verifier internal error: only two hidden subprogs supported\n"); return -EFAULT; } /* We're not patching any existing instruction, just appending the new @@ -19892,6 +19892,42 @@ static int do_misc_fixups(struct bpf_verifier_env *env) mark_subprog_exc_cb(env, env->exception_callback_subprog); } + if (env->seen_exception) { + struct bpf_insn patch[] = { + /* Use the correct insn_cnt here, as we want to append past the hidden subprog above. */ + env->prog->insnsi[env->prog->len - 1], + /* Scratch R6-R9 so that the JIT spills them to the stack on entry. */ + BPF_MOV64_IMM(BPF_REG_6, 0), + BPF_MOV64_IMM(BPF_REG_7, 0), + BPF_MOV64_IMM(BPF_REG_8, 0), + BPF_MOV64_IMM(BPF_REG_9, 0), + BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, BPF_PSEUDO_KFUNC_CALL, 0, special_kfunc_list[KF_bpf_throw]), + }; + const bool all_callee_regs_used[4] = {true, true, true, true}; + + ret = add_hidden_subprog(env, patch, ARRAY_SIZE(patch)); + if (ret < 0) + return ret; + prog = env->prog; + insn = prog->insnsi; + + env->bpf_throw_tramp_subprog = env->subprog_cnt - 1; + /* Ensure to mark callee_regs_used, so that we can collect any saved_regs if necessary. */ + memcpy(env->subprog_info[env->bpf_throw_tramp_subprog].callee_regs_used, all_callee_regs_used, sizeof(all_callee_regs_used)); + /* Certainly, we have seen a bpf_throw call in this program, as + * seen_exception is true, therefore the bpf_kfunc_desc entry for it must + * be populated and found here. We need to do the fixup now, otherwise + * the loop over insn_cnt below won't see this kfunc call. + */ + ret = fixup_kfunc_call(env, &prog->insnsi[prog->len - 1], insn_buf, prog->len - 1, &cnt); + if (ret < 0) + return ret; + if (cnt != 0) { + verbose(env, "verifier internal error: unhandled patching for bpf_throw fixup in bpf_throw_tramp subprog\n"); + return -EFAULT; + } + } + for (i = 0; i < insn_cnt; i++, insn++) { /* Make divide-by-zero exceptions impossible. */ if (insn->code == (BPF_ALU64 | BPF_MOD | BPF_X) || @@ -20012,6 +20048,19 @@ static int do_misc_fixups(struct bpf_verifier_env *env) if (insn->src_reg == BPF_PSEUDO_CALL) continue; if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) { + /* All bpf_throw calls in this program must be patched to call the + * bpf_throw_tramp subprog instead. This ensures we correctly save + * the R6-R9 before entry into kernel, and can clean them up if + * needed. + * Note: seen_exception must be set, otherwise no bpf_throw_tramp is + * generated. + */ + if (env->seen_exception && is_bpf_throw_kfunc(insn)) { + *insn = BPF_CALL_REL(0); + insn->imm = (int)env->subprog_info[env->bpf_throw_tramp_subprog].start - (i + delta) - 1; + continue; + } + ret = fixup_kfunc_call(env, insn, insn_buf, i + delta, &cnt); if (ret) return ret; From patchwork Thu Feb 1 04:21:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540622 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 814CD3B78D for ; Thu, 1 Feb 2024 04:21:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761287; cv=none; b=qefCVDZUeDIeJwglEYYj2HBAFddpYMRURfIajVr/PZjztNJzyfMSGiZH+0n9LYERL0Yhtx2kCNfbK3jaFCB5huCz4LJSDRXIlJHXpWtfpla5lCqQ1Fe1uRQh1HOB81pZNzE59UvWmCwgJGIc2VOMWDX8AEFk8riRsEjf8uMUZjk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761287; c=relaxed/simple; bh=Rvduhhs9SqJSr3lnxyZEUmCv0lcQNLMp4Px4fRQyQD8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=HCM6QMUljZrnUwlfSuVoOFCIMTtivZ4b0Svgmm8pqY/NaiGA6wV0jB3xAXE4PmZeM/9IRi0hcPPQ1FEH21hDnQULyLnMZKTQPH1SVJdmTtqE82cgXpPxS4E8aiRvejNDA38hCc4Obc/Sg/hP8zvVVbuhYohHzq7AYoGflePF+vk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DMj4H35I; arc=none smtp.client-ip=209.85.208.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DMj4H35I" Received: by mail-lj1-f180.google.com with SMTP id 38308e7fff4ca-2d043160cd1so6555291fa.1 for ; Wed, 31 Jan 2024 20:21:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761283; x=1707366083; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=U8fvce7p2wAAkT/EkHD52Iuw7K3tvalS7pMOB11qeDI=; b=DMj4H35IR+rRMIo5K1vjrTrYiNSuDP/ujTsy9xUneq7pNvk+eFOtaj0YLPN/lvTv6e sDj0qTI2yVwoIumDh5urWZbRPb/2sOVhEZp0uwCsndEl0pMaUZioM1DIB1HyOKUgvzRb ExN1PN/OAwPX7tBUYS1xTrz4jonVdhXjGd1qN7gR2mg8Zonf9MGt8tZPZFiZRSD3kN4Y LhzgPC+1m8mCS1AvmRo8X0sNxaiUWJVSgc3S7ItlDv/Jr66R9AAUEgd9WKsMSuzVFCfs 0baGcpjdBi1XnqQOUQadNI4HykWqBJy8eI34AB1bHU0dYa6vyjqDmeQZ93f1duomJGS3 9YvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761283; x=1707366083; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U8fvce7p2wAAkT/EkHD52Iuw7K3tvalS7pMOB11qeDI=; b=DvL9ZcA3na+JNg8xVXIH9FhD0pLbKOGge+TXoUXbWv6ax1FWQXc8/PJHiFANru/ld6 T3R5QRcBA2k2pXQuC5r0epXenb8xjxtZDjUbd3D0gS797sIcKpynk3wAwDSa1frPIUNi nCynunmHsyaVYj4/x9roBOgSL74HMdeJJ9LYY1nLDjnA7HTXUKONE/w/2kznDfnmgCGw +zzrU4hRtlF6C1iMyBDDgUaim4ssVWLJK1HVjWnuJUE2XGaexiLOgZtQoegBlqE9z5Ta 7mzbhgvAIsrmG6O7gcEH207YbYB7rQ9ZekhjtjksR5uRPT1r543I8g7GTrNbjoxKYf/g gLwg== X-Gm-Message-State: AOJu0YxHmfEjWoUL+hZMZQEW5Voc7dlOHWJuuzza5gf1x4crKURsecHI Jg09+qxSOZf0wSRtMdBdw2/XddHKv7dD8GML8DvWlOBQLuG8j9HL5GNrvZDv X-Google-Smtp-Source: AGHT+IG/ELi2M+tAqxNjM45pJTMr2zKRZ1nxfU9IC+lDjq/dpIC/i/YP1D0XgMOJwiHxPuRlY8NpGQ== X-Received: by 2002:a2e:2c06:0:b0:2cf:1aa1:beaf with SMTP id s6-20020a2e2c06000000b002cf1aa1beafmr2351314ljs.42.1706761283238; Wed, 31 Jan 2024 20:21:23 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id c8-20020a509f88000000b0055f43384da2sm2252707edf.56.2024.01.31.20.21.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:22 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 08/14] bpf: Compute used callee saved registers for subprogs Date: Thu, 1 Feb 2024 04:21:03 +0000 Message-Id: <20240201042109.1150490-9-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4012; i=memxor@gmail.com; h=from:subject; bh=Rvduhhs9SqJSr3lnxyZEUmCv0lcQNLMp4Px4fRQyQD8=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwOx+WhL0UXhC7OxmrLqpP+SYybZBX63Bc+T Ili6dTdPbeJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDgAKCRBM4MiGSL8R yiWXEACeROP15jWMVl4+a2zSzs36yger9vW/bybWdk1acxSJQua6modZZytpylahFRUgit+A8nw IShoG4MF85djBbhEBL7ZniL6r081tBujc8b3+k6Vu/6ootzDdU8quzQXYS69gXoNAVTcA9pQ3Ed ACzaypCBx8nZ12T66W2V+AO5QnPJjvTZPTBt8ANcfnEaUXBmNazNBHBvPWKLOXanNWkqfnJnp8Z j+EGR7X0BXGjD9d8t1cbudXNYXJz2miCP9N7+wZ95s+AIjFCNfkx0qAqeXxbu1CyIvL4SCwaDG3 4qrJwc2rKO0ncUP53RAIKDjdnRd62BviPRC8whGJsRHlJ7JQXErw+m8JKleXJG94IvtHuzgXjr6 SDAJaT4Sav4ktIbdfqG7GdcthXve9tNbirPR8SLZpvZathC5PHhMWMPJSCpziqWL0DvpzFln/F2 ICBPdmHkECofqQ6oScBJH2yX0w8DTpmJp1PYVoaFypeXq1elVq2hp4tMiK8+JPCAl0d4K0dQrqm uWu1TJJkg61EQ/Q30dKXn6TR3fjAr9TlBrHb0Zm24iFJP+kAJm1ATEszJzE75pEsmMtL/c5nsab 7Y+rXe+OCvk2S13Mz8gQoy51FucdzM3ym5igWInQ2hNzYHH9+VXk/HX0rwfcFmMRrF3xoXnutX6 i2PYhpOsYBxqGTA== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC During runtime unwinding and cleanup, we will need to figure out where the callee saved registers are stored on the stack, so that when a bpf_thtrow call is made, all frames can release their callee saved registers by finding their saved copies on the stack of callee frames. While the previous patch ensured any BPF callee saved registers are saved on a hidden subprog stack frame before entry into kernel (where we would not know their location if spilled), there are cases where a subprog's R6-R9 are not spilled into its immediate callee stack frame, but much later in the call chain in some later callee stack frame. As such, we would need to figure out while walking down the stack which frames have spilled their incoming callee saved regs, and thus keep track of where the latest spill would have happened with respect to a given frame in the stack trace. To perform this, we would need to know which callee saved registers are saved by a given subprog at runtime during the unwinding phase. Right now, there is a convenient way the x86 JIT figures this out in detect_reg_usage. Utilize such logic in verifier core, and copy this information to bpf_prog_aux struct before the JIT step to preserve this information at runtime, through bpf_prog_aux. Signed-off-by: Kumar Kartikeya Dwivedi --- include/linux/bpf.h | 1 + include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 10 ++++++++++ 3 files changed, 12 insertions(+) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 83cff18a1b66..4ac6add0cec8 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1460,6 +1460,7 @@ struct bpf_prog_aux { bool xdp_has_frags; bool exception_cb; bool exception_boundary; + bool callee_regs_used[4]; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 04e27fce33d6..e08ff540ec44 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -620,6 +620,7 @@ struct bpf_subprog_info { u32 start; /* insn idx of function entry point */ u32 linfo_idx; /* The idx to the main_prog->aux->linfo */ u16 stack_depth; /* max. stack depth used by this function */ + bool callee_regs_used[4]; bool has_tail_call: 1; bool tail_call_reachable: 1; bool has_ld_abs: 1; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 942243cba9f1..aeaf97b0a749 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -2942,6 +2942,15 @@ static int check_subprogs(struct bpf_verifier_env *env) insn[i].src_reg == 0 && insn[i].imm == BPF_FUNC_tail_call) subprog[cur_subprog].has_tail_call = true; + /* Collect callee regs used in the subprog. */ + if (insn[i].dst_reg == BPF_REG_6 || insn[i].src_reg == BPF_REG_6) + subprog[cur_subprog].callee_regs_used[0] = true; + if (insn[i].dst_reg == BPF_REG_7 || insn[i].src_reg == BPF_REG_7) + subprog[cur_subprog].callee_regs_used[1] = true; + if (insn[i].dst_reg == BPF_REG_8 || insn[i].src_reg == BPF_REG_8) + subprog[cur_subprog].callee_regs_used[2] = true; + if (insn[i].dst_reg == BPF_REG_9 || insn[i].src_reg == BPF_REG_9) + subprog[cur_subprog].callee_regs_used[3] = true; if (!env->seen_throw_insn && is_bpf_throw_kfunc(&insn[i])) env->seen_throw_insn = true; if (BPF_CLASS(code) == BPF_LD && @@ -19501,6 +19510,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) } func[i]->aux->num_exentries = num_exentries; func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable; + memcpy(&func[i]->aux->callee_regs_used, env->subprog_info[i].callee_regs_used, sizeof(func[i]->aux->callee_regs_used)); func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb; if (!i) func[i]->aux->exception_boundary = env->seen_exception; From patchwork Thu Feb 1 04:21:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540623 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lf1-f65.google.com (mail-lf1-f65.google.com [209.85.167.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 352D01EF15 for ; Thu, 1 Feb 2024 04:21:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.65 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761289; cv=none; b=EjSGrntQFJXF7euEaDsIWRlx+2bHKdYJpm2vgtHMFWGvACpxbQZWgJeJ1OBdOVaPnvRsNCvXlKKoxQRSQOU5e9gH+F9KuaPhRclTg5GZUQtI7jPRH3XXVyjHq3YDShvvh+53k3ZEKKhvXB6gi930vQ7u0djMc2Avuls4FJh4JK8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761289; c=relaxed/simple; bh=CXqh0gEVtnrZu0eS4CU8N1pndmxwXNh+IddwyXYewsc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tFxAtoZN2IiRrQqGKrkXXXVCs+FNka3W8dedX3NcCaXwU3Q3p6tv8LhhS7QA3JmSrbRDnw3P6elxpT3vkiu2VzHXXPUUmiQ3XzPpioAdrkP6rouXyrsRsgB8GAqP3XyYvt1ImHSIkkmmjq3MBpT6eQJQQcCJ+UMksA45Z/k+NNM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Vi9cBRVK; arc=none smtp.client-ip=209.85.167.65 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Vi9cBRVK" Received: by mail-lf1-f65.google.com with SMTP id 2adb3069b0e04-5111c7d40deso763483e87.1 for ; Wed, 31 Jan 2024 20:21:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761285; x=1707366085; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NzuiAPjE4eXQhj+Wjpkpb2ptqxxi7MZnbXgTNeQLKsM=; b=Vi9cBRVKYeUWYuY9WZNt8unEXz3iEtvMVoMv3o8Je+RxueqoQFdV60CU+JiNFMoF5a Q+AEw1rPNJDu//+hRzuJ6EkjCQpqb5EOpnwiVMEvPmzt3Hg4dm4UKqi8p0U8VWor8uUn DOQ2U+sjAv/E/shIj2s8b3/Q4GAkFWu+VKQZQpTOtPwZx2asjdgFJOT4BCCnrW/frknN H5xdBG3FIuGbxJg0QtfJpSLoT5gi9bAgxJFqnDIxm/edfSFTuYH4TJ7xa14Zel9mGq90 S9g90n3PySqsBg2+mYejfl8J6I8jdi74nseiVpvjbWiQg5+zC/UBVAoQVf5enwrFPhZn HxfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761285; x=1707366085; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NzuiAPjE4eXQhj+Wjpkpb2ptqxxi7MZnbXgTNeQLKsM=; b=tAwoSSoU01xYdyJBl8NP2HkHIBq8uTS7tOGkCwCYP2mS3BlwEJLB2yBdC2Lhc3WJU3 KNzXDInLUw4phIx7GY9q1Ho4+36p+B39GCvV4A1hUxb4WKVWrBUEHv/8rpzfyazEIkpU 4mwSvvJvOBWIHcZx+VvqIk55S2bDLx+xDrBrPr/93TA7HPbgr5QQG/rmJXisvTDgWivq /QUGtGovPzxF9g2ASIA2dwkOexr/DXxC+EDYFIrgaPtxk/gTpjLc9fy+7RTyhH55eKeM OAwYi1U7XTAQ2zKTWZHh2sZhiBxLI53Bq6L5KnEbfyyb6WD1XEBZiNCPj+N3evr5vKh/ 5PZA== X-Gm-Message-State: AOJu0YzWU7eWe5ScVZSZVwl8uWm7Iq0bqB/sk7lfuB01fcUxogprruj/ oB+MQ9IefcmPr8Yb3q1LqehaEJWeN2V/+Z/F2zGCuB9M9XlvveuYoRhTJ/5XN0g= X-Google-Smtp-Source: AGHT+IE619EEUEIhqoOTJWExPSF+oh9lbR08Myw+g2n8auaExEUazvNbPTR2XG5X6xgzp1Wg9y+BvA== X-Received: by 2002:ac2:485b:0:b0:511:169c:8f59 with SMTP id 27-20020ac2485b000000b00511169c8f59mr878828lfy.56.1706761284715; Wed, 31 Jan 2024 20:21:24 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id ll9-20020a170907190900b00a3664259b81sm1436220ejc.134.2024.01.31.20.21.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:24 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 09/14] bpf, x86: Fix up pc offsets for frame descriptor entries Date: Thu, 1 Feb 2024 04:21:04 +0000 Message-Id: <20240201042109.1150490-10-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4340; i=memxor@gmail.com; h=from:subject; bh=CXqh0gEVtnrZu0eS4CU8N1pndmxwXNh+IddwyXYewsc=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwP/lJs/ypdoqKR5uKdM6KC4gz0KqVZ0tX7w F26E0WlVIiJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDwAKCRBM4MiGSL8R ykxXD/9Jnp+wShaVfQo2xiXK5SudSK7DqpxcezSLY2JTdVpJ1EBwdTotSf9pU3e2K6nxxh9Doky mg9H+uCGbbHtV47EH1KzemcN5KeHXAdHgDnslzPMdrlZyVXftTKv1wNIvjF4ExvGTzcqr/n0WHP WyHvtDn6iM4YNzGUbDG5kVE5fdirJab0RsBxcjFHRcPYptHGjT7rwXsRFNLAwqTKe2KLvc6MaNe q6BfvbCuVEE78raB2/IMEoBbutfjtt5cfB3rvCP8gXTw99gCIkOCy35Fy0K/4rzHOKGvvd/GP3k XVktcOnAkem160qLgXx4o4ZVoya/yB/19gWC9OxfPnt/cJ+TQeYvDwJQz6x0ii8xAhjRzYPaO8S WZbT489aX4tIuT42xvzNGOjKME8xZojJ/3WB+o8PnGkgb+O0A/hNBrFsR843Xo6o7cK0Jo3PWpa 7DyBds+XAOr5jdJ1XTH3yDBkyXvT5cAlPO67dFlobUXm/sb0fzPxN6B+yxJYKzoBbZJoR3pbPcV FYMXKzzJc6ugAQAkGci4QsuN9uwxO1Kt/Q6fk1HSZHyIrAS7BQuIuUxsvs1ejQ356QFcMtXSSpM re2lWyW3nyBrVNoD69YQIZS1MbU23e93Vn5FAilPpS5WOO7XOlzbfICTXTrkny3AjRGhCzPU6ag Md6+RMINVEnWB9Q== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Until now, the program counter value stored in frame descriptor entries was the instruction index of the BPF program's insn and callsites when going down the frames in a call chain. However, at runtime, the program counter will be the pointer to the next instruction, and thus needs to be computed in a position independent way to tally it at runtime to find the frame descriptor when unwinding. To do this, we first convert the global instruction index into an instruction index relative to the start of a subprog, and add 1 to it (to reflect that at runtime, the program counter points to the next instruction). Then, we modify the JIT (for now, x86) to convert them to instruction offsets relative to the start of the JIT image, which is the prog->bpf_func of the subprog in question at runtime. Later, subtracting the prog->bpf_func pointer from runtime program counter will yield the same offset, and allow us to figure out the corresponding frame descriptor entry. Note that we have to mark a frame descriptor entry as 'final' because bpf_int_jit_compile can be called multiple times, and we would try to convert our already converted pc values again, therefore once we do the conversion remember it and do not repeat it. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 11 +++++++++++ include/linux/bpf.h | 2 ++ kernel/bpf/verifier.c | 15 +++++++++++++++ 3 files changed, 28 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 87692d983ffd..0dd0791c6ee0 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3112,6 +3112,17 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = orig_prog; } + if (prog->aux->fdtab && !prog->aux->fdtab->final && image) { + struct bpf_exception_frame_desc_tab *fdtab = prog->aux->fdtab; + + for (int i = 0; i < fdtab->cnt; i++) { + struct bpf_exception_frame_desc *desc = fdtab->desc[i]; + + desc->pc = addrs[desc->pc]; + } + prog->aux->fdtab->final = true; + } + if (!image || !prog->is_func || extra_pass) { if (image) bpf_prog_fill_jited_linfo(prog, addrs + 1); diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 4ac6add0cec8..e310d3ceb14e 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1460,6 +1460,7 @@ struct bpf_prog_aux { bool xdp_has_frags; bool exception_cb; bool exception_boundary; + bool bpf_throw_tramp; bool callee_regs_used[4]; /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; @@ -3395,6 +3396,7 @@ struct bpf_exception_frame_desc { struct bpf_exception_frame_desc_tab { u32 cnt; + bool final; struct bpf_exception_frame_desc **desc; }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index aeaf97b0a749..ec9acadc9ea8 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -19514,6 +19514,20 @@ static int jit_subprogs(struct bpf_verifier_env *env) func[i]->aux->exception_cb = env->subprog_info[i].is_exception_cb; if (!i) func[i]->aux->exception_boundary = env->seen_exception; + if (i == env->bpf_throw_tramp_subprog) + func[i]->aux->bpf_throw_tramp = true; + /* Fix up pc of fdtab entries to be relative to subprog start before JIT. */ + if (env->subprog_info[i].fdtab) { + for (int k = 0; k < env->subprog_info[i].fdtab->cnt; k++) { + struct bpf_exception_frame_desc *desc = env->subprog_info[i].fdtab->desc[k]; + /* Add 1 to point to the next instruction, which will be the PC at runtime. */ + desc->pc = desc->pc - subprog_start + 1; + } + } + /* Transfer fdtab to subprog->aux */ + func[i]->aux->fdtab = env->subprog_info[i].fdtab; + env->subprog_info[i].fdtab = NULL; + func[i] = bpf_int_jit_compile(func[i]); if (!func[i]->jited) { err = -ENOTSUPP; @@ -19604,6 +19618,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) prog->aux->real_func_cnt = env->subprog_cnt; prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; prog->aux->exception_boundary = func[0]->aux->exception_boundary; + prog->aux->fdtab = func[0]->aux->fdtab; bpf_prog_jit_attempt_done(prog); return 0; out_free: From patchwork Thu Feb 1 04:21:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540624 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-lj1-f195.google.com (mail-lj1-f195.google.com [209.85.208.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 743D23B795 for ; Thu, 1 Feb 2024 04:21:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761290; cv=none; b=t6YTCxT2FNwu1zV6t9uYkntEGU4QVGMZoiXAFLIEbDtmMNHfD7RnDsceGX6tuPwgfisVKAAoEmCIzUYRoz54qz7FS37YpsWOuLqPHLFTUelPV6ImVUZFsVu5sMIMUXGR0g8caD5WbZt1ciKAYpNphYNrKWig1VqMKoJpR3wYgOc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761290; c=relaxed/simple; bh=sBIV6Wl9RHkDMRnh1vsABLcYDEGo4g1ambDgfDE4lIQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=c3vOswEnnKHLI1GQNS52hUq/9kjhpuOSfdpVVCoTNFAvzEjUfN1B/6vTnbD/vGIo6Uz8/c+YGey26IOb1M1YCoCOikQKPnKUV1xkUKz8eXYKELQNu35Jlq8Cr7RWKo6GyCERwawd4r2uUZI4p6O5JTM2JaSLtCo1aQBQnmQQta8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QhOLFU2o; arc=none smtp.client-ip=209.85.208.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QhOLFU2o" Received: by mail-lj1-f195.google.com with SMTP id 38308e7fff4ca-2d0600551e8so6389881fa.0 for ; Wed, 31 Jan 2024 20:21:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761286; x=1707366086; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T7MaNjw8a51AIAgFfdQM8x4xHcs+fnZB56LSdgMMeYA=; b=QhOLFU2oL0wKp29OrwwctT7uqHKB4qxfGwY1n0z1Qi68lWQ9J8lwM7ektTocTpzwXq BKwAhDpxk2ZFdHD4KTkWO3LjhM4hYycs50jMCGc5KAa4UcxA6e4slilClcPwUfEdEkzn nlMdtep+xIZriuPBdrwwNeiZ0w08MTzIfuXcaW+HIy48wgMFd5jLzpSISkJLb2PHpNZl g2JhYCf16QV5RbjsgCl26ThLd5rzJDqQx7VyosXSfaRmU+TgRKC0bQ9Gq3ZSA9XwyZRL M2whsJ8/i6Ies+HgrAUM0JRWFtfwGDVnNvAcsywpd0zz1zZxhxZyA4qzYKHQKQcZzDAD 64zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761286; x=1707366086; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T7MaNjw8a51AIAgFfdQM8x4xHcs+fnZB56LSdgMMeYA=; b=Ce5xALZNyY18MmfeTYVwgKEwHJGvvMQ5eiqEWIxICnJ5IW1XoL9n5kr50oba8I3Wfq pIk+hJRXzMTH+Wz+W/CzHDIHIh/SMpGwxVeVS6F2QlcoDuioWlsuJ8YaatGXT98MYRxO JBwKzGvqYmu/J2J071rOamvaCNAwZYc3np6cFDfS4TzbJf0GFTS/j7SqCvIrnjUjJirj 0UmeRiNz6dU6Hi0FQCYEHTxPLvdKoUUm3f1WgIgfbCI0kLgM8vIuxXx8Z3+nhiLHXvUp c01OgjpQzrtq3XLxVbBIojfv6RiVOG3TGYx/FfwuxyhdWk+phWrplzJxL6GDPgBH/Vw2 OdEg== X-Gm-Message-State: AOJu0YztRCikvuvVqQz/knFlRv3K2b3y7mh8KeSS0yBFMyTRIIQsemHL OBc9aASmlnraizweFE5aJ9pMVhQg8q4wo1T6JayLwo5+IKflgnxUmc8HMoZnTgU= X-Google-Smtp-Source: AGHT+IFbGPrnUo22ypTMw92PfcfgDUjBRQ/t9g3Fme4tG4rCZzAgOKtaDslMBhEDpBGWF+rAbypz4w== X-Received: by 2002:a05:6512:1046:b0:510:1b51:e431 with SMTP id c6-20020a056512104600b005101b51e431mr922204lfb.4.1706761286153; Wed, 31 Jan 2024 20:21:26 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id pv13-20020a170907208d00b00a35b708185esm3979717ejb.71.2024.01.31.20.21.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:25 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 10/14] bpf, x86: Implement runtime resource cleanup for exceptions Date: Thu, 1 Feb 2024 04:21:05 +0000 Message-Id: <20240201042109.1150490-11-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=14238; i=memxor@gmail.com; h=from:subject; bh=sBIV6Wl9RHkDMRnh1vsABLcYDEGo4g1ambDgfDE4lIQ=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwPokv+N84diq7OXurgPSnkvSchzZAM/sp5F vhd0B3JVJWJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDwAKCRBM4MiGSL8R yiqaEACDHlBTBlzBgKWah8mU7e8A6YGpl75GZDhxf29s3PDNn+tnbpMQq0cON2jG/I41VqgxZVz l0iDMvCPC2Oor3nlTMVG6dFD3ikJ/yGTh0uvf6mZe/Uwp6MDR0hW235IjAc5jpwSjmttZ91Caic mxIAXHa4aXHkFqIhtHOsU6O98jcGD+k0puiVSrjIdf335jX/vuH+kk18ktWqMlBPBacBw6bJDsO gA8Ylz7oOA2ckJA3qlt+nw3DBFMOz/g/huoliThvyjQYYG/En4iiaDvPXP1mWXTEC6aMMZ12Zq3 G+4T+djDN5wSlYE9dMm4IA3+YRB+GNz18DYo8ZfhjODI0B3MVndJl7R5cSBkluF4gPYsfwrZcvq psh9yW/jieH4TPOmWRTgGwd9XkR0t2aGuO/1r46CklhOi887QYTcwR3WUnfZZZKvmATeSFGNa/P ZhYkgGIfI1WaJErYAKAwkCdeq3zJNSoI+dPBUuteo5RbmkyNTsmyvt5xS6TpLMVVP/tbPW4SrEB 34m344y72sF6cyzMIZ1eBziOZxoh+pRDFrgVDIXwIV/iNgPWMfPWWIxIMZVuofy59PnLcGDQ1Ts imLWzx4AZ0MSZr/xl5LY1tWeOtt5GV55AGrcSoxA5bLQXq47TIgCYo8zZYTr/qmvtkP0adyypCN PLGVDDDjRWtnuBQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Finally, tie all ends together and implement functionality to process a frame descriptor at runtime for each frame when bpf_throw is called, and release resources present on the program's stack frames. For each frame, we do bpf_cleanup_frame_resource, which will use the instruction pointer at runtime to figure out the right frame descriptor entry. After this, we explore all stack and registers and call their respective cleanup procedures. Next, if the frame corresponds to a subprog, we all save the location of where it has spilled its callers R6-R9 registers. If so, we record their value in the unwinding context. Only doing this when each frame has scratched the register in question allows us to arrive at the set of values actually needed during the freeing step for registers, regardless of how many callees existed and the varying locations of spilled callee saved registers. These registers can also lie across different frames, but will collected top down when arriving at a frame. Finally, after doing the cleanup, we go on to execute the exception callback and finish unwinding the stack. Signed-off-by: Kumar Kartikeya Dwivedi --- arch/x86/net/bpf_jit_comp.c | 81 ++++++++++++++++++++++++ include/linux/bpf.h | 22 +++++++ include/linux/filter.h | 3 + kernel/bpf/core.c | 5 ++ kernel/bpf/helpers.c | 121 +++++++++++++++++++++++++++++++++--- kernel/bpf/verifier.c | 20 ++++++ net/core/filter.c | 5 ++ 7 files changed, 247 insertions(+), 10 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 0dd0791c6ee0..26a96fee2f4e 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -3191,6 +3191,11 @@ bool bpf_jit_supports_exceptions(void) return IS_ENABLED(CONFIG_UNWINDER_ORC); } +bool bpf_jit_supports_exceptions_cleanup(void) +{ + return bpf_jit_supports_exceptions(); +} + void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) { #if defined(CONFIG_UNWINDER_ORC) @@ -3208,6 +3213,82 @@ void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp WARN(1, "verification of programs using bpf_throw should have failed\n"); } +static int bpf_frame_spilled_caller_reg_off(struct bpf_prog *prog, int regno) +{ + int off = 0; + + for (int i = BPF_REG_9; i >= BPF_REG_6; i--) { + if (regno == i) + return off; + if (prog->aux->callee_regs_used[i - BPF_REG_6]) + off += sizeof(u64); + } + WARN_ON_ONCE(1); + return 0; +} + +void arch_bpf_cleanup_frame_resource(struct bpf_prog *prog, struct bpf_throw_ctx *ctx, u64 ip, u64 sp, u64 bp) { + struct bpf_exception_frame_desc_tab *fdtab = prog->aux->fdtab; + struct bpf_exception_frame_desc *fd = NULL; + u64 ip_off = ip - (u64)prog->bpf_func; + + /* Hidden subprogs and subprogs without fdtab do not need cleanup. */ + if (bpf_is_hidden_subprog(prog) || !fdtab) + goto end; + + for (int i = 0; i < fdtab->cnt; i++) { + if (ip_off != fdtab->desc[i]->pc) + continue; + fd = fdtab->desc[i]; + break; + } + /* This should always be found, but let's bail if we cannot find it. */ + if (WARN_ON_ONCE(!fd)) + return; + + for (int i = 0; i < fd->stack_cnt; i++) { + void *ptr = (void *)((s64)bp + fd->stack[i].off); + + bpf_cleanup_resource(fd->stack + i, ptr); + } + + for (int i = 0; i < ARRAY_SIZE(fd->regs); i++) { + void *ptr; + + if (!fd->regs[i].regno || fd->regs[i].type == NOT_INIT || fd->regs[i].type == SCALAR_VALUE) + continue; + /* Our sp will be bp of new frame before caller regs are spilled, so offset is relative to our sp. */ + WARN_ON_ONCE(!ctx->saved_reg[i]); + ptr = (void *)&ctx->saved_reg[i]; + bpf_cleanup_resource(fd->regs + i, ptr); + ctx->saved_reg[i] = 0; + } +end: + /* There could be a case where we have something in main R6, R7, R8, R9 that + * needs releasing, and the callchain is as follows: + * main -> subprog1 -> subprog2 -> bpf_throw_tramp -> bpf_throw + * In such a case, subprog1 may use only R6, R7 and subprog2 may use R8, R9 being unscratched until + * subprog2 calls bpf_throw. In that case, subprog2 will spill R6-R9. The + * loop below when we are called for each subprog in order will ensure we have the correct saved_reg + * from the PoV of the current bpf_prog corresponding to a frame. + * E.g. in the chain main -> s1 -> s2 -> bpf_throw_tramp -> bpf_throw + * Let's say R6-R9 have values A, B, C, D in main when calling subprog1. + * Below, we show the computed saved_regs values as we walk the stack: + * For bpf_throw_tramp, saved_regs = { 0, 0, 0, 0 } + * For s2, saved_regs = { 0, 0, 0, D } // D loaded from bpf_throw_tramp frame + * For s1, saved_regs = { 0, 0, C, D } // C loaded from subprog2 frame + * For main, saved_regs = { A, B, C, D } // A, B loaded from subprog1 frame + * Thus, for main, we have the correct saved_regs values even though they + * were spilled in multiple callee stack frames down the call chain. + */ + if (bpf_is_subprog(prog)) { + for (int i = 0; i < ARRAY_SIZE(prog->aux->callee_regs_used); i++) { + if (prog->aux->callee_regs_used[i]) + ctx->saved_reg[i] = *(u64 *)((s64)sp + bpf_frame_spilled_caller_reg_off(prog, BPF_REG_6 + i)); + } + } +} + void bpf_arch_poke_desc_update(struct bpf_jit_poke_descriptor *poke, struct bpf_prog *new, struct bpf_prog *old) { diff --git a/include/linux/bpf.h b/include/linux/bpf.h index e310d3ceb14e..a7c8c118c534 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -3402,4 +3402,26 @@ struct bpf_exception_frame_desc_tab { void bpf_exception_frame_desc_tab_free(struct bpf_exception_frame_desc_tab *fdtab); +struct bpf_throw_ctx { + struct bpf_prog_aux *aux; + u64 sp; + u64 bp; + union { + struct { + u64 saved_r6; + u64 saved_r7; + u64 saved_r8; + u64 saved_r9; + }; + u64 saved_reg[4]; + }; + int cnt; +}; + +void arch_bpf_cleanup_frame_resource(struct bpf_prog *prog, struct bpf_throw_ctx *ctx, u64 ip, u64 sp, u64 bp); +void bpf_cleanup_resource(struct bpf_frame_desc_reg_entry *fd, void *ptr); +int bpf_cleanup_resource_reg(struct bpf_frame_desc_reg_entry *fd, void *ptr); +int bpf_cleanup_resource_dynptr(struct bpf_frame_desc_reg_entry *fd, void *ptr); +int bpf_cleanup_resource_iter(struct bpf_frame_desc_reg_entry *fd, void *ptr); + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/filter.h b/include/linux/filter.h index fee070b9826e..9779d8281a59 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -955,6 +955,7 @@ bool bpf_jit_supports_subprog_tailcalls(void); bool bpf_jit_supports_kfunc_call(void); bool bpf_jit_supports_far_kfunc_call(void); bool bpf_jit_supports_exceptions(void); +bool bpf_jit_supports_exceptions_cleanup(void); bool bpf_jit_supports_ptr_xchg(void); void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie); bool bpf_helper_changes_pkt_data(void *func); @@ -1624,4 +1625,6 @@ static inline void bpf_xdp_copy_buf(struct xdp_buff *xdp, unsigned long off, voi } #endif /* CONFIG_NET */ +void bpf_sk_release_dtor(void *ptr); + #endif /* __LINUX_FILTER_H__ */ diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 995a4dcfa970..6e6260c1e926 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2979,6 +2979,11 @@ bool __weak bpf_jit_supports_exceptions(void) return false; } +bool __weak bpf_jit_supports_exceptions_cleanup(void) +{ + return false; +} + void __weak arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie) { } diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 4db1c658254c..304fe26cba65 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2,6 +2,7 @@ /* Copyright (c) 2011-2014 PLUMgrid, http://plumgrid.com */ #include +#include #include #include #include @@ -2499,12 +2500,113 @@ __bpf_kfunc void bpf_rcu_read_unlock(void) rcu_read_unlock(); } -struct bpf_throw_ctx { - struct bpf_prog_aux *aux; - u64 sp; - u64 bp; - int cnt; -}; +int bpf_cleanup_resource_reg(struct bpf_frame_desc_reg_entry *fd, void *ptr) +{ + u64 reg_value = ptr ? *(u64 *)ptr : 0; + struct btf_struct_meta *meta; + const struct btf_type *t; + u32 dtor_id; + + switch (fd->type) { + case PTR_TO_SOCKET: + case PTR_TO_TCP_SOCK: + case PTR_TO_SOCK_COMMON: + if (reg_value) + bpf_sk_release_dtor((void *)reg_value); + return 0; + case PTR_TO_MEM | MEM_RINGBUF: + if (reg_value) + bpf_ringbuf_discard_proto.func(reg_value, 0, 0, 0, 0); + return 0; + case PTR_TO_BTF_ID | MEM_ALLOC: + case PTR_TO_BTF_ID | MEM_ALLOC | MEM_PERCPU: + if (!reg_value) + return 0; + meta = btf_find_struct_meta(fd->btf, fd->btf_id); + if (fd->type & MEM_PERCPU) + bpf_percpu_obj_drop_impl((void *)reg_value, meta); + else + bpf_obj_drop_impl((void *)reg_value, meta); + return 0; + case PTR_TO_BTF_ID: +#ifdef CONFIG_NET + if (bsearch(&fd->btf_id, btf_sock_ids, MAX_BTF_SOCK_TYPE, sizeof(btf_sock_ids[0]), btf_id_cmp_func)) { + if (reg_value) + bpf_sk_release_dtor((void *)reg_value); + return 0; + } +#endif + dtor_id = btf_find_dtor_kfunc(fd->btf, fd->btf_id, BPF_DTOR_KPTR | BPF_DTOR_CLEANUP); + if (dtor_id < 0) + return -EINVAL; + t = btf_type_by_id(fd->btf, dtor_id); + if (!t) + return -EINVAL; + if (reg_value) { + void (*dtor)(void *) = (void *)kallsyms_lookup_name(btf_name_by_offset(fd->btf, t->name_off)); + dtor((void *)reg_value); + } + return 0; + case SCALAR_VALUE: + case NOT_INIT: + return 0; + default: + break; + } + return -EINVAL; +} + +int bpf_cleanup_resource_dynptr(struct bpf_frame_desc_reg_entry *fd, void *ptr) +{ + switch (fd->type) { + case BPF_DYNPTR_TYPE_RINGBUF: + if (ptr) + bpf_ringbuf_discard_dynptr_proto.func((u64)ptr, 0, 0, 0, 0); + return 0; + default: + break; + } + return -EINVAL; +} + +int bpf_cleanup_resource_iter(struct bpf_frame_desc_reg_entry *fd, void *ptr) +{ + const struct btf_type *t; + void (*dtor)(void *); + u32 dtor_id; + + dtor_id = btf_find_dtor_kfunc(fd->btf, fd->btf_id, BPF_DTOR_CLEANUP); + if (dtor_id < 0) + return -EINVAL; + t = btf_type_by_id(fd->btf, dtor_id); + if (!t) + return -EINVAL; + dtor = (void *)kallsyms_lookup_name(btf_name_by_offset(fd->btf, t->name_off)); + if (ptr) + dtor(ptr); + return 0; +} + +void bpf_cleanup_resource(struct bpf_frame_desc_reg_entry *fd, void *ptr) +{ + if (!ptr) + return; + switch (fd->spill_type) { + case STACK_DYNPTR: + bpf_cleanup_resource_dynptr(fd, ptr); + break; + case STACK_ITER: + bpf_cleanup_resource_iter(fd, ptr); + break; + case STACK_SPILL: + case STACK_INVALID: + bpf_cleanup_resource_reg(fd, ptr); + break; + default: + WARN_ON_ONCE(1); + break; + } +} static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp) { @@ -2514,13 +2616,12 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp) if (!is_bpf_text_address(ip)) return !ctx->cnt; prog = bpf_prog_ksym_find(ip); - ctx->cnt++; - if (bpf_is_subprog(prog)) - return true; ctx->aux = prog->aux; ctx->sp = sp; ctx->bp = bp; - return false; + arch_bpf_cleanup_frame_resource(prog, ctx, ip, sp, bp); + ctx->cnt++; + return bpf_is_subprog(prog); } __bpf_kfunc void bpf_throw(u64 cookie) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index ec9acadc9ea8..3e3b8a20451c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10216,6 +10216,11 @@ static int gen_exception_frame_desc_reg_entry(struct bpf_verifier_env *env, stru fd.off = off; verbose(env, "frame_desc: frame%d: insn_idx=%d %s=%d size=%d ref_obj_id=%d type=%s\n", frameno, env->insn_idx, off < 0 ? "off" : "regno", off, BPF_REG_SIZE, reg->ref_obj_id, reg_type_str(env, reg->type)); + + if (bpf_cleanup_resource_reg(&fd, NULL)) { + verbose(env, "frame_desc: frame%d: failed to simulate cleanup for frame desc entry\n", frameno); + return -EFAULT; + } return push_exception_frame_desc(env, frameno, &fd); } @@ -10241,6 +10246,11 @@ static int gen_exception_frame_desc_dynptr_entry(struct bpf_verifier_env *env, s verbose(env, "verifier internal error: refcounted dynptr type unhandled for exception frame descriptor entry\n"); return -EFAULT; } + + if (bpf_cleanup_resource_dynptr(&fd, NULL)) { + verbose(env, "frame_desc: frame%d: failed to simulate cleanup for frame desc entry\n", frameno); + return -EFAULT; + } return push_exception_frame_desc(env, frameno, &fd); } @@ -10268,6 +10278,11 @@ static int gen_exception_frame_desc_iter_entry(struct bpf_verifier_env *env, str btf_put(btf); return ret; } + + if (bpf_cleanup_resource_iter(&fd, NULL)) { + verbose(env, "frame_desc: frame%d: failed to simulate cleanup for frame desc entry\n", frameno); + return -EFAULT; + } return push_exception_frame_desc(env, frameno, &fd); } @@ -10348,6 +10363,11 @@ static int gen_exception_frame_descs(struct bpf_verifier_env *env) __mark_reg_not_init(env, ¬_init_reg); + if (!bpf_jit_supports_exceptions_cleanup()) { + verbose(env, "JIT does not support cleanup of resources when throwing exceptions\n"); + return -ENOTSUPP; + } + for (int frameno = env->cur_state->curframe; frameno >= 0; frameno--) { struct bpf_func_state *frame = env->cur_state->frame[frameno]; diff --git a/net/core/filter.c b/net/core/filter.c index 524adf1fa6d0..789e36f79f4c 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -6912,6 +6912,11 @@ static const struct bpf_func_proto bpf_sk_release_proto = { .arg1_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON | OBJ_RELEASE, }; +void bpf_sk_release_dtor(void *ptr) +{ + bpf_sk_release((u64)ptr, 0, 0, 0, 0); +} + BPF_CALL_5(bpf_xdp_sk_lookup_udp, struct xdp_buff *, ctx, struct bpf_sock_tuple *, tuple, u32, len, u32, netns_id, u64, flags) { From patchwork Thu Feb 1 04:21:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540625 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ej1-f68.google.com (mail-ej1-f68.google.com [209.85.218.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 773AA3C465 for ; Thu, 1 Feb 2024 04:21:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761291; cv=none; b=PT3D1RrHdV0VINulO5Czje+7nqDf620j3ALsSsA3GVT3anNmvdtseCm/Zkybo3JUP4qe4ASqz/lc5mRpt/oM8IduCGk7SNUp2poKA6pMpNV6qO4SG71q0LDnKSSqzrrZ6KHcWLRYi/5OWpsF/eXO7aPModE52o7pP3xU4pRbwUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761291; c=relaxed/simple; bh=Nohh56u+465Lt6chQfJ/XP9DJ0vNzIUQddkXi+4GSAM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iPZX5Uq9i0lS6QUTPV4yLsftqH7Odtc4/xVrTUuGjveT8Roa8RFsZ5Z+qAcrp2S3PZB0LTzuebNEtk6cG0QIzZ7Lxugqsk96kp2GW7kSWnfEri1UFzzfnnv4btHiG05s5ELyVKQ8/eBb90B3BV/+E6iAC/rt9wKAg9g2oWko+38= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ndk6VAAr; arc=none smtp.client-ip=209.85.218.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ndk6VAAr" Received: by mail-ej1-f68.google.com with SMTP id a640c23a62f3a-a350bfcc621so51354566b.0 for ; Wed, 31 Jan 2024 20:21:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761287; x=1707366087; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=skBcnJ7cz4044hjwmIqAd593TGS+mKC7y1F5stomKew=; b=Ndk6VAArhJj7dKGiNlVHmHJzsA/tNET3WUa3vMI06/xgGXs0l911aYokCbfIxpBOdN sOjgIDWsa9tl3AsJQEL8plN2DfhRVMyXrkNokdBEuuiKP0zltjM9oGJiGNoayZT48guS VNeBvaTM/6x+zUniXKFFnCIOHlU5ylRvf9u/ff+PYPdbY3hB504V9wpJjluHlFpdkor0 yRdJxcJShy75vHVseOMInowg5h5t2agGHO3Ibm8DzSXGc2kPuaQffJdLFF2tfW/u1HWs gJoOEbio9ao8T+ifrv5oiMz9S25ZhnKB6zzqg3p8t4sJ0Kmh2QcL1P7xeewY5mbHNb/j fZPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761287; x=1707366087; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=skBcnJ7cz4044hjwmIqAd593TGS+mKC7y1F5stomKew=; b=qoWvFAzTO9+yS4KpmRZ1PtTlpgR9GMTIi1+ZOJmwvxKcP4Cwmlell0l5slsjapQfJJ +YNPlJ7Y+g6+J61wqqjXVqhqN4GjEVn8c7eUWa1fYgjjAHBx0IfS24RyI/LjoWLrzLTL idzdWYiTfF4J6gRT9rQxHZlL+xJ/SDXmHYUfi1R6nwdrrVbbte9D5hLDImFwWDebBdHY 1HtB5M/1MiLER3MhbHEE4+aOMsCKeCP4oXiKCzhui6CiCfApOBkZx/wLcnVDJDbHE6i3 OFoz/d8o5CVdoh4VRLIfWIGg9wajN7qBBO+3Yif6WeVj58fe39wKOdYUtlW9AGBmICdy KkXA== X-Gm-Message-State: AOJu0YwRETs3OVRVi2H0NExuMheXwxck9Xvg7pYiqB4dOUjwSJUjLCju N0yoyzRmBAJ+nnhrq9ZpgD6OsIM9UCKaRkBFxJdsazQvjBNain3USavzQaFOXr0= X-Google-Smtp-Source: AGHT+IHVOkWtXVUsCPMcDhaFkGaQQ9bEaa7mOE57s5PBMbNFYv6GPUBkZapD0XwDpzNY/ircsZyMVw== X-Received: by 2002:a17:906:5fca:b0:a35:fe4c:e76b with SMTP id k10-20020a1709065fca00b00a35fe4ce76bmr2472891ejv.66.1706761287458; Wed, 31 Jan 2024 20:21:27 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id vv9-20020a170907a68900b00a354a5d2c39sm5501189ejc.31.2024.01.31.20.21.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:26 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 11/14] bpf: Release references in verifier state when throwing exceptions Date: Thu, 1 Feb 2024 04:21:06 +0000 Message-Id: <20240201042109.1150490-12-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4957; i=memxor@gmail.com; h=from:subject; bh=Nohh56u+465Lt6chQfJ/XP9DJ0vNzIUQddkXi+4GSAM=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwPsiKlloCSbpR7rik/dNtDX+6A8hr6oc7GV M0SCVqepNGJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscDwAKCRBM4MiGSL8R yiZyD/9Bd4fZYPWh2vicTpzbCpV15itcFK8Q2Y1R3CYF7WlMCv5mGpknNN1gp6CI7T5DpNDlN07 AGYhlyp9HL4avvEK/SCVt7lY6MpBWzsYfGZorVs76I5gwOSl1FEoaeg4zMOxQn285A0oJAcpWxD nooIpE+FAbBo0df+J3dZpF2q8mlhR8q4nvGt4L8YfCuA7PPrEHceTLfEh58+uFRilDpwfqqtTOs 3PdWGCv+anez1hCX4wUhXivubj1/k0iCb3694L99k7ARGXOCcZIGjnlUs6VrOawGA8qfmEOKEb8 8EHvmDfFLbPIanmBqsDZz4i5HUzFI/ltTJy96zJI2b66xLnOwus3HVeE/pFdSlLZ4xwN4JDCdJ/ NSbC4kGI1Q74bs5zh7wneS8ndQeuYkpJGECJZlGcTTBTns9+6td0o3Jf2g/adRFIL/U1Ld2WTtn Lhi5hJnA8xcVQaTRzNwCtMBDRKm1ILVjqdVm34hyjG7wV9iLmby7Lgprnas39U9fRewcO4TixFx Go1SKgdlj/U00P9aoG4gDNxeU4cJNol1cH57M4rPQWhAgQLq8+8QHurNC9ODPsC6+J/fyQzvdq3 AYBc2s03I40JDlKnJjHncpLDcUEfyNKFWpOW0TH+9Tqx3eQvsmAukoTpJOLPBecSMv2p/1ZJbZ+ 0NMuWgcNXRsQAXw== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Reflect in the verifier state that references would be released whenever we throw a BPF exception. Now that we support generating frame descriptors, and performing the runtime cleanup, whenever processing an entry corresponding to an acquired reference, make sure we release its reference state. Note that we only release this state for the current frame, as the acquired refs are only checked against that when processing an exceptional exit. This would ensure that for acquired resources apart from locks and RCU read sections, BPF programs never fail in case of lingering resources during verification. While at it, we can tweak check_reference_leak to drop the exception_exit parameter, and fix selftests that will fail due to the changed behaviour. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman --- kernel/bpf/verifier.c | 18 ++++++++++++------ .../selftests/bpf/progs/exceptions_fail.c | 2 +- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3e3b8a20451c..8edefcd999ea 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10221,6 +10221,8 @@ static int gen_exception_frame_desc_reg_entry(struct bpf_verifier_env *env, stru verbose(env, "frame_desc: frame%d: failed to simulate cleanup for frame desc entry\n", frameno); return -EFAULT; } + if (reg->ref_obj_id && frameno == cur_func(env)->frameno) + WARN_ON_ONCE(release_reference(env, reg->ref_obj_id)); return push_exception_frame_desc(env, frameno, &fd); } @@ -10251,6 +10253,8 @@ static int gen_exception_frame_desc_dynptr_entry(struct bpf_verifier_env *env, s verbose(env, "frame_desc: frame%d: failed to simulate cleanup for frame desc entry\n", frameno); return -EFAULT; } + if (frameno == cur_func(env)->frameno) + WARN_ON_ONCE(release_reference(env, reg->ref_obj_id)); return push_exception_frame_desc(env, frameno, &fd); } @@ -10283,6 +10287,8 @@ static int gen_exception_frame_desc_iter_entry(struct bpf_verifier_env *env, str verbose(env, "frame_desc: frame%d: failed to simulate cleanup for frame desc entry\n", frameno); return -EFAULT; } + if (frameno == cur_func(env)->frameno) + WARN_ON_ONCE(release_reference(env, reg->ref_obj_id)); return push_exception_frame_desc(env, frameno, &fd); } @@ -10393,17 +10399,17 @@ static int gen_exception_frame_descs(struct bpf_verifier_env *env) return 0; } -static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exit) +static int check_reference_leak(struct bpf_verifier_env *env) { struct bpf_func_state *state = cur_func(env); bool refs_lingering = false; int i; - if (!exception_exit && state->frameno && !state->in_callback_fn) + if (state->frameno && !state->in_callback_fn) return 0; for (i = 0; i < state->acquired_refs; i++) { - if (!exception_exit && state->in_callback_fn && state->refs[i].callback_ref != state->frameno) + if (state->in_callback_fn && state->refs[i].callback_ref != state->frameno) continue; verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", state->refs[i].id, state->refs[i].insn_idx); @@ -10658,7 +10664,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn switch (func_id) { case BPF_FUNC_tail_call: - err = check_reference_leak(env, false); + err = check_reference_leak(env); if (err) { verbose(env, "tail_call would lead to reference leak\n"); return err; @@ -15593,7 +15599,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * gen_ld_abs() may terminate the program at runtime, leading to * reference leak. */ - err = check_reference_leak(env, false); + err = check_reference_leak(env); if (err) { verbose(env, "BPF_LD_[ABS|IND] cannot be mixed with socket references\n"); return err; @@ -18149,7 +18155,7 @@ static int do_check(struct bpf_verifier_env *env) * function, for which reference_state must * match caller reference state when it exits. */ - err = check_reference_leak(env, exception_exit); + err = check_reference_leak(env); if (err) return err; diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index 5a517065b4e6..dfd164a7a261 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -354,7 +354,7 @@ int reject_exception_throw_cb_diff(struct __sk_buff *ctx) } SEC("?tc") -__failure __msg("exploring program path where exception is thrown") +__success __log_level(2) __msg("exploring program path where exception is thrown") int reject_exception_throw_ref_call_throwing_global(struct __sk_buff *ctx) { struct { long a; } *p = bpf_obj_new(typeof(*p)); From patchwork Thu Feb 1 04:21:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540626 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ej1-f66.google.com (mail-ej1-f66.google.com [209.85.218.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 416993C46C for ; Thu, 1 Feb 2024 04:21:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.66 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761293; cv=none; b=nvFoOvPutrivAPptY58VE0BMiCIQKcP+eGal684Yxdwk9+8fCgDmC1bcAj1kcplevQWqgjWpTF3dVDDwUGzsbbbhjzhs3KxJY9GwV9FOsIF1kcZIMoBV3Cjvzo4OoK688gC2lXXjisJcMOM+S8WmcdyMaVyT+EtG7QfsMVOT74E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761293; c=relaxed/simple; bh=ZSPFWnHn9tpYuXlwy/W39jwhB+YgpL3iD+6Fxtkp4Uw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MeK5ukFceKsV6WpVEDYKBhAy76NDRtpRMjZDcTA+Vibn5jUP7cXeYHB4zt5qYusJ4UYtHbavi14v4iB3b8Xmvc62k2huax64Ei27syq3NuWkh31Hr5Ys1X6MPpDc3xmkcwk72XBjUbZqq71VwO1wm9UqWvdfvbafvn6lAALV05I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=V+xzgyRw; arc=none smtp.client-ip=209.85.218.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="V+xzgyRw" Received: by mail-ej1-f66.google.com with SMTP id a640c23a62f3a-a36126ee41eso51946366b.2 for ; Wed, 31 Jan 2024 20:21:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761289; x=1707366089; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8jHG+muzkbV4bFwdgTxkuBTkEK2OJnBsTWFCEVngZrY=; b=V+xzgyRwpqfWE39ZkvcTDrj5mOVARDfc9f0+Fndw5D2vLpCsdx1SnERmMB6TK4FCaX jv2uY1mvoP0+K0dCBM7iERa4t4PN4jowcBz7oQF3/BvPvQnmg+oFvt4dcFHmEBNE/aF4 xfkrI+rEYgGJUXCrM1JIczrJ9a2tMmOJ9NEetxSLmuJBd2qxqvJ0e5h5nQMSasR2TD7k 9yInWPW/Q0C4SiJptr4AJm2vIm1oldrFmu/BY9aZK4Unj+k0EaCNpgfDTbUynJGAvmFv WTEXpOB8eOir5xmvnE72MA+FP/dADfYWKledhXN3N9HbinFv63Hi9ZUdy4kMJP41rvGn eIpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761289; x=1707366089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8jHG+muzkbV4bFwdgTxkuBTkEK2OJnBsTWFCEVngZrY=; b=lHzW0mJJ9Ku/8Xcw+s9QrrwrT/ZzJknQT4CmrOto6kvbeML+2qNA9e8fqPoYZsqtm6 I1F22rcOFb4+4EmgYEGCACkTYM7Z0NNM/e6i34fIqxQXby7IRVdOuNTYAHQhUtaogARS 9gLdqJ5WdCGWmmWVNZljkydjDUqc+2CFJmam9KFJ6rEmdzWjc/4oqxFaUt6KvEnWmJuv T4nRtddANHZA7Hh+fPVnNe8nr4SoMJqd3P8/s9JA4ynXCMfY8R8BNMppkvzn7zaG1RcR QpcJ/AwyaDH3I7CiseVEAdYcRGLIPFJlQhy13UXHhkO8GN+LnAxx7nxy8e+FTwsTANa1 AkwA== X-Gm-Message-State: AOJu0Yy4Td0fySZ2LVM/Rst0OMM0C8F7sXkUTokmPHKOrQbnQu8NRwdi KVefThsfGtJZNVz6YpewYYcSHwnPvx+r8BYsy1u0Aa7WRFergwxpfymFumZYr9U= X-Google-Smtp-Source: AGHT+IHlYYcn/V+BMe7I6gj7t48GKk5F6UbXn+94t2YzfPGcvwWjPTecTx+mx0ewnFdNcNs9b0fUFA== X-Received: by 2002:a17:906:a399:b0:a36:6a6e:8104 with SMTP id k25-20020a170906a39900b00a366a6e8104mr2296026ejz.39.1706761288755; Wed, 31 Jan 2024 20:21:28 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id og16-20020a1709071dd000b00a28aba1f56fsm6759608ejc.210.2024.01.31.20.21.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:28 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Jiri Kosina , Benjamin Tissoires , Pablo Neira Ayuso , Florian Westphal , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 12/14] bpf: Register cleanup dtors for runtime unwinding Date: Thu, 1 Feb 2024 04:21:07 +0000 Message-Id: <20240201042109.1150490-13-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=11863; i=memxor@gmail.com; h=from:subject; bh=ZSPFWnHn9tpYuXlwy/W39jwhB+YgpL3iD+6Fxtkp4Uw=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwQinYX8KREDrGuREgIcoEHWemmoetEzUhP6 PwyDf6PWz2JAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscEAAKCRBM4MiGSL8R yuBtD/0X/de92rEhn/m5jC6lF9h5mYMDS/L7StiRpBLcDz9T9ycm9Y2jrH3JSqF/VwEElFktUS/ eGn44iQuaVMcTg/TllQc1gV+wKPWBk9dV4mmUrHUcBLRLyNwci9Fnxn9RWQgZwqZ46ks54odSEy TMjtJWuKpEz38VlFR+rW+TITqtwoVt8qcgajXO0wJZ6qQOu9ReRAvCbE/pgoP5T5++Cgn1PU5q+ ayvJXpMPi3EobaxBkIvVfs9i75rQSgNF4ykTNDZu5iuir0fLFYQdoQNBrR0/hxf4dMv8gHMcXVT s0Mir2SmW+uRXv9YYPf+z5AlKU48Nap7J3lVkOgA0NWjF7uJ3nRRiBOfuyP1oNURQQzbnbwV8bN PgJNr1IVy6w9ey6odaZAjx0SXidXJdrOGXrAJ/G0m3d+YfIW7oFwO3C+IhNtQmIQc/KAKAu4UpN NzgYaZkGwql8em4YAU4VsN3nBFZAW5ucQRt7il6GOrXebfXXpHCk6zaH4SUdsVlBrbanvRkX9+H zFCMxHA66TghSBW/WCCFbqH8sSS8xYNN83xegaqkMFxnNX338bwcju9V1mg7+KWuUbfKuDQ38A2 tcSnbEDK7FlgVMHhOWzKdvNRgd4LDvjqAZmlvz1zb0y3WI6cE417N9Q7nzXeh3Tt45gLwTPKDGC Jm15cB77423yuQg== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Reuse exist BTF dtor infrastructure to also include dtor kfuncs that can be used to release PTR_TO_BTF_ID pointers and other BTF objects (iterators). For this purpose, we extend btf_id_dtor_kfunc object with a flags field, and ensure that entries that cannot work as kptrs are not allowed to be embedded in map values. Prior to this change, btf_id_dtor_kfunc served a dual role of allow list of kptrs and finding their dtors. To separate this role, we must now explicitly pass only BPF_DTOR_KPTR to ensure we don't look up other cleanup kfuncs in the dtor table. Finally, set up iterator and other objects that can be acquired to be released by adding their cleanup kfunc dtor entries and registering them with the BTF. Cc: Jiri Kosina Cc: Benjamin Tissoires Cc: Pablo Neira Ayuso Cc: Florian Westphal Signed-off-by: Kumar Kartikeya Dwivedi --- drivers/hid/bpf/hid_bpf_dispatch.c | 17 ++++++++++++ include/linux/btf.h | 10 +++++-- kernel/bpf/btf.c | 11 +++++--- kernel/bpf/cpumask.c | 3 ++- kernel/bpf/helpers.c | 43 +++++++++++++++++++++++++++--- kernel/trace/bpf_trace.c | 16 +++++++++++ net/bpf/test_run.c | 4 ++- net/netfilter/nf_conntrack_bpf.c | 14 +++++++++- net/xfrm/xfrm_state_bpf.c | 16 +++++++++++ 9 files changed, 123 insertions(+), 11 deletions(-) diff --git a/drivers/hid/bpf/hid_bpf_dispatch.c b/drivers/hid/bpf/hid_bpf_dispatch.c index 02c441aaa217..eea1699b91cc 100644 --- a/drivers/hid/bpf/hid_bpf_dispatch.c +++ b/drivers/hid/bpf/hid_bpf_dispatch.c @@ -452,6 +452,10 @@ static const struct btf_kfunc_id_set hid_bpf_syscall_kfunc_set = { .set = &hid_bpf_syscall_kfunc_ids, }; +BTF_ID_LIST(hid_bpf_dtor_id_list) +BTF_ID(struct, hid_bpf_ctx) +BTF_ID(func, hid_bpf_release_context) + int hid_bpf_connect_device(struct hid_device *hdev) { struct hid_bpf_prog_list *prog_list; @@ -496,6 +500,13 @@ EXPORT_SYMBOL_GPL(hid_bpf_device_init); static int __init hid_bpf_init(void) { + const struct btf_id_dtor_kfunc dtors[] = { + { + .btf_id = hid_bpf_dtor_id_list[0], + .kfunc_btf_id = hid_bpf_dtor_id_list[1], + .flags = BPF_DTOR_CLEANUP, + }, + }; int err; /* Note: if we exit with an error any time here, we would entirely break HID, which @@ -505,6 +516,12 @@ static int __init hid_bpf_init(void) * will not be available, so nobody will be able to use the functionality. */ + err = register_btf_id_dtor_kfuncs(dtors, ARRAY_SIZE(dtors), THIS_MODULE); + if (err) { + pr_warn("error while registering hid_bpf cleanup dtors: %d", err); + return 0; + } + err = register_btf_fmodret_id_set(&hid_bpf_fmodret_set); if (err) { pr_warn("error while registering fmodret entrypoints: %d", err); diff --git a/include/linux/btf.h b/include/linux/btf.h index 1ee8977b8c95..219cc4a5d22d 100644 --- a/include/linux/btf.h +++ b/include/linux/btf.h @@ -120,9 +120,15 @@ struct btf_kfunc_id_set { btf_kfunc_filter_t filter; }; +enum { + BPF_DTOR_KPTR = (1 << 0), + BPF_DTOR_CLEANUP = (1 << 1), +}; + struct btf_id_dtor_kfunc { u32 btf_id; u32 kfunc_btf_id; + u32 flags; }; struct btf_struct_meta { @@ -521,7 +527,7 @@ u32 *btf_kfunc_is_modify_return(const struct btf *btf, u32 kfunc_btf_id, int register_btf_kfunc_id_set(enum bpf_prog_type prog_type, const struct btf_kfunc_id_set *s); int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset); -s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id); +s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id, u32 flags); int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt, struct module *owner); struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id); @@ -555,7 +561,7 @@ static inline int register_btf_kfunc_id_set(enum bpf_prog_type prog_type, { return 0; } -static inline s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id) +static inline s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id, u32 flags) { return -ENOENT; } diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index ef380e546952..17b9c04a71dd 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3657,7 +3657,7 @@ static int btf_parse_kptr(const struct btf *btf, struct btf_field *field, * can be used as a referenced pointer and be stored in a map at * the same time. */ - dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id); + dtor_btf_id = btf_find_dtor_kfunc(kptr_btf, id, BPF_DTOR_KPTR); if (dtor_btf_id < 0) { ret = dtor_btf_id; goto end_btf; @@ -8144,7 +8144,7 @@ int register_btf_fmodret_id_set(const struct btf_kfunc_id_set *kset) } EXPORT_SYMBOL_GPL(register_btf_fmodret_id_set); -s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id) +s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id, u32 flags) { struct btf_id_dtor_kfunc_tab *tab = btf->dtor_kfunc_tab; struct btf_id_dtor_kfunc *dtor; @@ -8156,7 +8156,7 @@ s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id) */ BUILD_BUG_ON(offsetof(struct btf_id_dtor_kfunc, btf_id) != 0); dtor = bsearch(&btf_id, tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func); - if (!dtor) + if (!dtor || !(dtor->flags & flags)) return -ENOENT; return dtor->kfunc_btf_id; } @@ -8171,6 +8171,11 @@ static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc for (i = 0; i < cnt; i++) { dtor_btf_id = dtors[i].kfunc_btf_id; + if (!dtors[i].flags) { + pr_err("missing flag for btf_id_dtor_kfunc entry\n"); + return -EINVAL; + } + dtor_func = btf_type_by_id(btf, dtor_btf_id); if (!dtor_func || !btf_type_is_func(dtor_func)) return -EINVAL; diff --git a/kernel/bpf/cpumask.c b/kernel/bpf/cpumask.c index dad0fb1c8e87..7209adc1af6b 100644 --- a/kernel/bpf/cpumask.c +++ b/kernel/bpf/cpumask.c @@ -467,7 +467,8 @@ static int __init cpumask_kfunc_init(void) const struct btf_id_dtor_kfunc cpumask_dtors[] = { { .btf_id = cpumask_dtor_ids[0], - .kfunc_btf_id = cpumask_dtor_ids[1] + .kfunc_btf_id = cpumask_dtor_ids[1], + .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP, }, }; diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 304fe26cba65..e1dfc4053f45 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2685,9 +2685,19 @@ static const struct btf_kfunc_id_set generic_kfunc_set = { BTF_ID_LIST(generic_dtor_ids) BTF_ID(struct, task_struct) BTF_ID(func, bpf_task_release_dtor) +BTF_ID(struct, bpf_iter_num) +BTF_ID(func, bpf_iter_num_destroy) +BTF_ID(struct, bpf_iter_task) +BTF_ID(func, bpf_iter_task_destroy) +BTF_ID(struct, bpf_iter_task_vma) +BTF_ID(func, bpf_iter_task_vma_destroy) #ifdef CONFIG_CGROUPS BTF_ID(struct, cgroup) BTF_ID(func, bpf_cgroup_release_dtor) +BTF_ID(struct, bpf_iter_css) +BTF_ID(func, bpf_iter_css_destroy) +BTF_ID(struct, bpf_iter_css_task) +BTF_ID(func, bpf_iter_css_task_destroy) #endif BTF_KFUNCS_START(common_btf_ids) @@ -2732,12 +2742,39 @@ static int __init kfunc_init(void) const struct btf_id_dtor_kfunc generic_dtors[] = { { .btf_id = generic_dtor_ids[0], - .kfunc_btf_id = generic_dtor_ids[1] + .kfunc_btf_id = generic_dtor_ids[1], + .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP, }, -#ifdef CONFIG_CGROUPS { .btf_id = generic_dtor_ids[2], - .kfunc_btf_id = generic_dtor_ids[3] + .kfunc_btf_id = generic_dtor_ids[3], + .flags = BPF_DTOR_CLEANUP, + }, + { + .btf_id = generic_dtor_ids[4], + .kfunc_btf_id = generic_dtor_ids[5], + .flags = BPF_DTOR_CLEANUP, + }, + { + .btf_id = generic_dtor_ids[6], + .kfunc_btf_id = generic_dtor_ids[7], + .flags = BPF_DTOR_CLEANUP, + }, +#ifdef CONFIG_CGROUPS + { + .btf_id = generic_dtor_ids[8], + .kfunc_btf_id = generic_dtor_ids[9], + .flags = BPF_DTOR_KPTR | BPF_DTOR_CLEANUP, + }, + { + .btf_id = generic_dtor_ids[10], + .kfunc_btf_id = generic_dtor_ids[11], + .flags = BPF_DTOR_CLEANUP, + }, + { + .btf_id = generic_dtor_ids[12], + .kfunc_btf_id = generic_dtor_ids[13], + .flags = BPF_DTOR_CLEANUP, }, #endif }; diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 241ddf5e3895..7a4bab3e698c 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -1426,8 +1426,24 @@ static const struct btf_kfunc_id_set bpf_key_sig_kfunc_set = { .set = &key_sig_kfunc_set, }; +BTF_ID_LIST(bpf_key_dtor_id_list) +BTF_ID(struct, bpf_key) +BTF_ID(func, bpf_key_put) + static int __init bpf_key_sig_kfuncs_init(void) { + const struct btf_id_dtor_kfunc dtors[] = { + { + .btf_id = bpf_key_dtor_id_list[0], + .kfunc_btf_id = bpf_key_dtor_id_list[1], + .flags = BPF_DTOR_CLEANUP, + }, + }; + int ret; + + ret = register_btf_id_dtor_kfuncs(dtors, ARRAY_SIZE(dtors), THIS_MODULE); + if (ret < 0) + return 0; return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_key_sig_kfunc_set); } diff --git a/net/bpf/test_run.c b/net/bpf/test_run.c index 5535f9adc658..4f506b27bb13 100644 --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -1691,11 +1691,13 @@ static int __init bpf_prog_test_run_init(void) const struct btf_id_dtor_kfunc bpf_prog_test_dtor_kfunc[] = { { .btf_id = bpf_prog_test_dtor_kfunc_ids[0], - .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[1] + .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[1], + .flags = BPF_DTOR_KPTR, }, { .btf_id = bpf_prog_test_dtor_kfunc_ids[2], .kfunc_btf_id = bpf_prog_test_dtor_kfunc_ids[3], + .flags = BPF_DTOR_KPTR, }, }; int ret; diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c index d2492d050fe6..00eb111d9c1a 100644 --- a/net/netfilter/nf_conntrack_bpf.c +++ b/net/netfilter/nf_conntrack_bpf.c @@ -485,11 +485,23 @@ static const struct btf_kfunc_id_set nf_conntrack_kfunc_set = { .set = &nf_ct_kfunc_set, }; +BTF_ID_LIST(nf_dtor_id_list) +BTF_ID(struct, nf_conn) +BTF_ID(func, bpf_ct_release) + int register_nf_conntrack_bpf(void) { + const struct btf_id_dtor_kfunc dtors[] = { + { + .btf_id = nf_dtor_id_list[0], + .kfunc_btf_id = nf_dtor_id_list[1], + .flags = BPF_DTOR_CLEANUP, + }, + }; int ret; - ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &nf_conntrack_kfunc_set); + ret = register_btf_id_dtor_kfuncs(dtors, ARRAY_SIZE(dtors), THIS_MODULE); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &nf_conntrack_kfunc_set); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &nf_conntrack_kfunc_set); if (!ret) { mutex_lock(&nf_conn_btf_access_lock); diff --git a/net/xfrm/xfrm_state_bpf.c b/net/xfrm/xfrm_state_bpf.c index 2248eda741f8..fdf6c22d145f 100644 --- a/net/xfrm/xfrm_state_bpf.c +++ b/net/xfrm/xfrm_state_bpf.c @@ -127,8 +127,24 @@ static const struct btf_kfunc_id_set xfrm_state_xdp_kfunc_set = { .set = &xfrm_state_kfunc_set, }; +BTF_ID_LIST(dtor_id_list) +BTF_ID(struct, xfrm_state) +BTF_ID(func, bpf_xdp_xfrm_state_release) + int __init register_xfrm_state_bpf(void) { + const struct btf_id_dtor_kfunc dtors[] = { + { + .btf_id = dtor_id_list[0], + .kfunc_btf_id = dtor_id_list[1], + .flags = BPF_DTOR_CLEANUP, + }, + }; + int ret; + + ret = register_btf_id_dtor_kfuncs(dtors, ARRAY_SIZE(dtors), THIS_MODULE); + if (ret < 0) + return ret; return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &xfrm_state_xdp_kfunc_set); } From patchwork Thu Feb 1 04:21:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540627 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-ed1-f66.google.com (mail-ed1-f66.google.com [209.85.208.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 74CC33B78D for ; Thu, 1 Feb 2024 04:21:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.66 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761294; cv=none; b=hERjYmSVc7guyhYX9qNhXYMF/YwNKeyyGJjtLd+Ix9whQOtzIBSnKduZEcSV8YJldmUEgUUbdRdBf276UoJPXbKLwYKjE7GweeHQLSc2Cnv2ly+HgDbyUgUfyVR46+BkCv+IlJwaNHFliIre0QFg6VHxOfZs+Wj5rN7h/5fnHqk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761294; c=relaxed/simple; bh=XU5PGSxoKSfgmoxEeAMnJYVKcZEFNE5w1yXreQ4Wi4c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YvVlinklfjwsivZII8aQnponONk4gkq3J47tkcElxwH9HzSoUrIi27XbyaR7CJs3VEo3E13yyRHnUoSsiECVNnnw5LX91+IDboxZUay5iGBZFiWfdY8RLIROJKwdevcoLfCOrtOhmdUoh/SlBkCRtS12bLESfWnN/uTpRIZYsj8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WP8fgFma; arc=none smtp.client-ip=209.85.208.66 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WP8fgFma" Received: by mail-ed1-f66.google.com with SMTP id 4fb4d7f45d1cf-55efbaca48bso557809a12.2 for ; Wed, 31 Jan 2024 20:21:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761290; x=1707366090; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GyEdEXxYMF4SNG+EY6o6CSJuRU5QFMRjr7Kwq62OyVE=; b=WP8fgFmas55n7ba6rYbhYsUAAnRoRfsI7vHMrgYiZ1tBkjTjeqbiGlfoYZ6aPzRr5H H8FCB0WHWzXEBO5yTpwPArqGpi9mpZjTsI8UGL5buDiO7JiEE9gQZcphzuuqF/GJ0NHu a/RpGJ/LUXz9FYa8/DUeJSDLmMW+Z1LqbqDYpI6JP1Vcxsb8N6wxD6uhWmADpNTlyll/ UgA7HldKrNV5ggFv7pUNztyq2fTNAOzBR84qyfuIaLdFCj6XUrwg9GtXd4yuVRq2qCp2 mLsGAeG9IO1apu11sx9Wj4atG9QXrdjP/uDps6ubfFr82x0+xfpGhR/qoK0Q9Pj73r96 3ueg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761290; x=1707366090; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GyEdEXxYMF4SNG+EY6o6CSJuRU5QFMRjr7Kwq62OyVE=; b=RBtTDloYvc7EnfUsvhA9Al+BGmfsY71Lv1/0urpIprrayLfWrzVq4LdV0SuxF4l3PV BHymz4VT2K0d+p1D6QFnraZX/Y1Iy4NCjDXD8ocwRZOtf9JjjBtdDcki9Zh88wDuV3wU flJ12xwQb0sYf+qshz3zbs7fBhacxEVsCunWjOonuZ681WwrtAltRgBz/1OUF7Y3j/1C hMAxFfFYrsZfuzjvtdFzZbH6vv3LrRoGMFy5mniV4gBZd2DmtIpaqBuTLmyASjpSt/0C TMf3bVlAVjOxoF1/UznnEoQd2XhUa46ILxtBbgfqFRBRySCG7C9Uxkrfn496W9H0UzBy TzaA== X-Gm-Message-State: AOJu0YzEq8yNzeuHRjsGu9CSb9wwC544VuZ+cBcHUhnZeGuTkykys6E1 L2dTTKtRFDK2tMSMrmXiZfwrLL6XdusH+AQE1stZMgk3e5J/nISmXvEBo0IUqr8= X-Google-Smtp-Source: AGHT+IH24qSOuEDBYxTjv2EVFOtnaCEH2b1/dkbUjlEhjt9LN8fC2YRrObbswCvZpe/u1cNg3+NenQ== X-Received: by 2002:a05:6402:160d:b0:55f:3fe8:3178 with SMTP id f13-20020a056402160d00b0055f3fe83178mr2610094edv.16.1706761290100; Wed, 31 Jan 2024 20:21:30 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id p14-20020a056402500e00b0055c67e6454asm6501337eda.70.2024.01.31.20.21.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:29 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 13/14] bpf: Make bpf_throw available to all program types Date: Thu, 1 Feb 2024 04:21:08 +0000 Message-Id: <20240201042109.1150490-14-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=810; i=memxor@gmail.com; h=from:subject; bh=XU5PGSxoKSfgmoxEeAMnJYVKcZEFNE5w1yXreQ4Wi4c=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwQ7rRmTvy4QqZ8xAsYFZUQkUlXvyM5/l/zf lNXNoDQWguJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscEAAKCRBM4MiGSL8R yq0yEACtwdNhCCsjd1yLbef5wHJaU1k2VtAfB+I7WXlz/cb72xbf4kzvtBxhwMVuomFkmXuMMIT zbarRJAYGhG6DxiThT/BfAZ95y5E5ODJXCU7593QvvQz6IJR6zJkxh3T/MSnrBQ28am+GLi+GJp WakBCKzlV97MFhhrZaH/dj5F+E+Sq6cB0KGpC65ETyG7fpVPFNY3cL8BERh5DZpnatOGLpN6+j8 iAn9g7IrYZWgA38UPjC9sG2mNXiGJntFRCqtJQLUwJFIEB6krxbzCk0EI7eoLUMtarrY43V0ySG 76j78CZI5Kc/k1dA1dWBZNdUwcagqTmuveZ/28gG0F6NMcxGHmFw+2/9G499zPmlLsAtEfb+e/H RkfAh+fDq+tEJoi9AHPq1pa/PIW8Kxku5xiHvAI7ZNdFOdWcihKGim/ufTgaNBoXF6tSlhCxDb+ L1AUnus8dRSbynfc6d3NCo6ylEXjP2LFolcXMqvjrBU2T3MO3R3VuEhMBkAZxWJAHsR07fw+t8q +W9kih5n80LPcEA9uPbopqBJSxsfCRPy2YQ1oAnB1377j3u904cW35LW7/WGoSeW58NpwdcYZUn sQQZuhdhWYK+pAjK0lgiwCJjNZz9KxASE5a7RAJBQyvbnZTDrWl8Juba/1nX7swmEShVQmtQPyu 6912xNoFbza2c4g== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Move bpf_throw kfunc to common_kfunc_set so that any program type can utilize it to throw exceptions. This will also be useful to test a wider variety of programs to test the cleanup logic properly. Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/helpers.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index e1dfc4053f45..388174f34a9b 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2729,6 +2729,7 @@ BTF_ID_FLAGS(func, bpf_dynptr_is_null) BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) BTF_ID_FLAGS(func, bpf_dynptr_size) BTF_ID_FLAGS(func, bpf_dynptr_clone) +BTF_ID_FLAGS(func, bpf_throw) BTF_KFUNCS_END(common_btf_ids) static const struct btf_kfunc_id_set common_kfunc_set = { From patchwork Thu Feb 1 04:21:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kumar Kartikeya Dwivedi X-Patchwork-Id: 13540628 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA1153C485 for ; Thu, 1 Feb 2024 04:21:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.68 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761296; cv=none; b=Jui7d/sVwdqxhj+CLKwL4VSJLWKbNnwdV9Qf9sj++VT7YkQaShSbtz0SQa5z4ORK+V2v4Rtt2Aj4vItKHVX47qKqj46u8G6sGd7sKQwlr+kVun9mnt/k5V1XpUJFNS5IrXYF99yDIbULbmqBkw2s2M57c6lCfKiFFtYe55M8iyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706761296; c=relaxed/simple; bh=IjjNLd6pt8ZtD/1nEkgQngCD7nEFkiMi90W9nzsZLrI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=exyFPgtPB1ua0y1m43HuFdEBoKaibRLZfbbpwFA9Y30pWqb5SDvUhZ6UAG15HQVWxP63CKsy2KFZMlfkBQ6vEUFj2PZegs97GvoR4M5dw/sbcxIbG2TZbpU8V6hTh3hvy4ZbRd+MDhPIz2cJoHbBteaj9hcB6bXMK6IJ/a8sfbs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lVCt3Z9s; arc=none smtp.client-ip=209.85.221.68 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lVCt3Z9s" Received: by mail-wr1-f68.google.com with SMTP id ffacd0b85a97d-339289fead2so274944f8f.3 for ; Wed, 31 Jan 2024 20:21:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706761291; x=1707366091; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/JOMGc1rzEyjwLiXBTAoSIAQfvOpFkVoDLvtHwP8uT0=; b=lVCt3Z9slauxxXvKRsPhJzrDh3QJd6F6phI0ycbkK0Sse16Udy6JFamhNEAI2TAFpu Zpk00H5QJDkvQ2WmKgURcLS7e7mY4Z31/IKD5Nts1Cl1sknBVzUpkRmzzlCbgHuYDu+Z LPzbiO18xZE9gWtG2BW+gx10U7LqIy5dJVkVxHAgQh7+t5AbIdAA/LqNQM167+cc4elt 3cgHPYm/6ZJ9HtvWQFn6ZcHe6jCXzLsWzAeJWOGcqPL9dPWyAG/sKaB/nd3THyv+YwIm HF16Ec/yZzxTRn2Pfyp5Ba4yoXWaFicL0I8juY0ud6d+DRxu9vuAXFi2wuOm6cQaGKLs iI0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706761291; x=1707366091; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/JOMGc1rzEyjwLiXBTAoSIAQfvOpFkVoDLvtHwP8uT0=; b=tyINyP/bGPNrAHIEHgtY/U84OtplmD2XxtGLGX8ZED1aLKAzUUNKtKay159DuxFe0s tAC5x90lA5ewISPUbDFVAJJ/eaoyLs4fbd+wA8u32e5KsbeoYbGYC9v1AeQd92dkD6Dk UoVbIsOiag7Vr+sVvV+0jRlLg5PFCtUZ/d/bf0QNAYJbSGvi+HftkAMzZ6hwfoMZ2jDJ tHvCKJifL1DaNgz9rIp/vdCiqsjgBpjqHVl4iW2CYxSGG/zTr294TX3jIbnI5qh3mvpM DYNkoaLRGuy/EM9MtN1ph9LDqD7zo6AOhBRJE3YukdvDrASneycRQt8scnzkknNfVCKA P23A== X-Gm-Message-State: AOJu0YzcXcHbqp5l3xumWsIvqFgF2LqT7mp5fpjmLkUrAxqOsafAReOs PZzhsp70+V7p6k9cadXFjHpPXnX4OoIYU/gzEAyYqh5chWZkO5woh667ZCD3O/0= X-Google-Smtp-Source: AGHT+IEKD4vyudTXdd/TD8RhrqLda40h5gy4A6G7YCQDSdKnB0iZS/CyfDg5USjcSgV9JrZXXMjyRg== X-Received: by 2002:a05:6000:1367:b0:337:c4c1:a3af with SMTP id q7-20020a056000136700b00337c4c1a3afmr2647073wrz.35.1706761291354; Wed, 31 Jan 2024 20:21:31 -0800 (PST) Received: from localhost (nat-icclus-192-26-29-3.epfl.ch. [192.26.29.3]) by smtp.gmail.com with ESMTPSA id s14-20020aa7d78e000000b0055f4fbc32ccsm1858259edq.89.2024.01.31.20.21.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 31 Jan 2024 20:21:30 -0800 (PST) From: Kumar Kartikeya Dwivedi To: bpf@vger.kernel.org Cc: Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Martin KaFai Lau , David Vernet , Tejun Heo , Raj Sahu , Dan Williams , Rishabh Iyer , Sanidhya Kashyap Subject: [RFC PATCH v1 14/14] selftests/bpf: Add tests for exceptions runtime cleanup Date: Thu, 1 Feb 2024 04:21:09 +0000 Message-Id: <20240201042109.1150490-15-memxor@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201042109.1150490-1-memxor@gmail.com> References: <20240201042109.1150490-1-memxor@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=23097; i=memxor@gmail.com; h=from:subject; bh=IjjNLd6pt8ZtD/1nEkgQngCD7nEFkiMi90W9nzsZLrI=; b=owEBbQKS/ZANAwAKAUzgyIZIvxHKAcsmYgBluxwR3tG5ZIPb7mhOQwEN++30fxmR/SoowDPEO MiXGpNpDmyJAjMEAAEKAB0WIQRLvip+Buz51YI8YRFM4MiGSL8RygUCZbscEQAKCRBM4MiGSL8R yrm+D/4sMA78GaqmprwEIEGPKOgELAOicP89bcelEQTM9jv13UugOpLudvO/XjtuthTPB8sVlav bCOGg4bZL3rq3NWFzl4u6D/CpWoOA/bDU4bc2indC2TrMZXujnP6Ok4KU/lpuCS11uXeHjmRTmJ HGjL633U5A6FjSCw7Lfz5KJrbZRJhWyW7dS/v1jK3FwOi9pJsI9oz5EyKXUfbribkKOW9CNcLJo CAwWmOE6Jo0zlFkBvdouvIEfo5pjagEMDHC1uwqmCA2AcsopQvFjLD18sRDH5ZJUv0DshNx5A3r IkWVppMV/HVdFsYc4jgPMAwWlKcxervEVDIq3I56U+bKorPHUkRaAvAlMM7Ce2g1/hubyEgg8Lw FfEpzKSk+YPPvuUlSy6Qala1fpOLuF68Idnjnl2vv6JuyEKDHIjd7BnWbcGaqfPF1lvpYACxA2F GOmcOpLNmYKy+zKo0MBbtGD9IONIp8tKjnfmj/GX07tBSPZiyOpWTgQD7GIlUdPqzWeP6EMDD23 XcXSomHtvw7xUX0DudaqPbYi5esZCT0kxlPHaCUe+6C7bLoYpfD/lqsJia5wXYONdNr0+NhdRPb Bf3vf3fkqzbh4Y3qZ3ol3vztfvWknkM38Z1YrPP4WAS/lJZTBJ66aJVGAUOcdin2ZxTbZlTUGkW Fvp0ofnSLz6/UAQ== X-Developer-Key: i=memxor@gmail.com; a=openpgp; fpr=4BBE2A7E06ECF9D5823C61114CE0C88648BF11CA X-Patchwork-Delegate: bpf@iogearbox.net X-Patchwork-State: RFC Add tests for the runtime cleanup support for exceptions, ensuring that resources are correctly identified and released when an exception is thrown. Also, we add negative tests to exercise corner cases the verifier should reject. Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 + tools/testing/selftests/bpf/DENYLIST.s390x | 1 + .../bpf/prog_tests/exceptions_cleanup.c | 65 +++ .../selftests/bpf/progs/exceptions_cleanup.c | 468 ++++++++++++++++++ .../bpf/progs/exceptions_cleanup_fail.c | 154 ++++++ .../selftests/bpf/progs/exceptions_fail.c | 13 - 6 files changed, 689 insertions(+), 13 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup.c create mode 100644 tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 5c2cc7e8c5d0..6fc79727cd14 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -1,6 +1,7 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 exceptions # JIT does not support calling kfunc bpf_throw: -524 +exceptions_unwind # JIT does not support calling kfunc bpf_throw: -524 fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # needs CONFIG_FPROBE kprobe_multi_test # needs CONFIG_FPROBE diff --git a/tools/testing/selftests/bpf/DENYLIST.s390x b/tools/testing/selftests/bpf/DENYLIST.s390x index 1a63996c0304..f09a73dee72c 100644 --- a/tools/testing/selftests/bpf/DENYLIST.s390x +++ b/tools/testing/selftests/bpf/DENYLIST.s390x @@ -1,5 +1,6 @@ # TEMPORARY # Alphabetical order exceptions # JIT does not support calling kfunc bpf_throw (exceptions) +exceptions_unwind # JIT does not support calling kfunc bpf_throw (exceptions) get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace) stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?) diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c new file mode 100644 index 000000000000..78df037b60ea --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/exceptions_cleanup.c @@ -0,0 +1,65 @@ +#include "bpf/bpf.h" +#include "exceptions.skel.h" +#include +#include + +#include "exceptions_cleanup.skel.h" +#include "exceptions_cleanup_fail.skel.h" + +static void test_exceptions_cleanup_fail(void) +{ + RUN_TESTS(exceptions_cleanup_fail); +} + +void test_exceptions_cleanup(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, ropts, + .data_in = &pkt_v4, + .data_size_in = sizeof(pkt_v4), + .repeat = 1, + ); + struct exceptions_cleanup *skel; + int ret; + + if (test__start_subtest("exceptions_cleanup_fail")) + test_exceptions_cleanup_fail(); + + skel = exceptions_cleanup__open_and_load(); + if (!ASSERT_OK_PTR(skel, "exceptions_cleanup__open_and_load")) + return; + + ret = exceptions_cleanup__attach(skel); + if (!ASSERT_OK(ret, "exceptions_cleanup__attach")) + return; + +#define RUN_EXC_CLEANUP_TEST(name) \ + ret = bpf_prog_test_run_opts(bpf_program__fd(skel->progs.name), \ + &ropts); \ + if (!ASSERT_OK(ret, #name ": return value")) \ + return; \ + if (!ASSERT_EQ(ropts.retval, 0xeB9F, #name ": opts.retval")) \ + return; \ + ret = bpf_prog_test_run_opts( \ + bpf_program__fd(skel->progs.exceptions_cleanup_check), \ + &ropts); \ + if (!ASSERT_OK(ret, #name " CHECK: return value")) \ + return; \ + if (!ASSERT_EQ(ropts.retval, 0, #name " CHECK: opts.retval")) \ + return; \ + skel->bss->only_count = 0; + + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_prog_num_iter); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_prog_num_iter_mult); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_prog_dynptr_iter); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_obj); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_percpu_obj); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_ringbuf); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_reg); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_null_or_ptr_do_ptr); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_null_or_ptr_do_null); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_callee_saved); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_frame); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_loop_iterations); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_dead_code_elim); + RUN_EXC_CLEANUP_TEST(exceptions_cleanup_frame_dce); +} diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c new file mode 100644 index 000000000000..ccf14fe6bd1b --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup.c @@ -0,0 +1,468 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include "bpf_misc.h" +#include "bpf_kfuncs.h" +#include "bpf_experimental.h" + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 8); +} ringbuf SEC(".maps"); + +enum { + RES_DYNPTR, + RES_ITER, + RES_REG, + RES_SPILL, + __RES_MAX, +}; + +struct bpf_resource { + int type; +}; + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1024); + __type(key, int); + __type(value, struct bpf_resource); +} hashmap SEC(".maps"); + +const volatile bool always_false = false; +bool only_count = false; +int res_count = 0; + +#define MARK_RESOURCE(ptr, type) ({ res_count++; bpf_map_update_elem(&hashmap, &(void *){ptr}, &(struct bpf_resource){type}, 0); }); +#define FIND_RESOURCE(ptr) ((struct bpf_resource *)bpf_map_lookup_elem(&hashmap, &(void *){ptr}) ?: &(struct bpf_resource){__RES_MAX}) +#define FREE_RESOURCE(ptr) bpf_map_delete_elem(&hashmap, &(void *){ptr}) +#define VAL 0xeB9F + +SEC("fentry/bpf_cleanup_resource") +int BPF_PROG(exception_cleanup_mark_free, struct bpf_frame_desc_reg_entry *fd, void *ptr) +{ + if (fd->spill_type == STACK_INVALID) + bpf_probe_read_kernel(&ptr, sizeof(ptr), ptr); + if (only_count) { + res_count--; + return 0; + } + switch (fd->spill_type) { + case STACK_SPILL: + if (FIND_RESOURCE(ptr)->type == RES_SPILL) + FREE_RESOURCE(ptr); + break; + case STACK_INVALID: + if (FIND_RESOURCE(ptr)->type == RES_REG) + FREE_RESOURCE(ptr); + break; + case STACK_ITER: + if (FIND_RESOURCE(ptr)->type == RES_ITER) + FREE_RESOURCE(ptr); + break; + case STACK_DYNPTR: + if (FIND_RESOURCE(ptr)->type == RES_DYNPTR) + FREE_RESOURCE(ptr); + break; + } + return 0; +} + +static long map_cb(struct bpf_map *map, void *key, void *value, void *ctx) +{ + int *cnt = ctx; + + (*cnt)++; + return 0; +} + +SEC("tc") +int exceptions_cleanup_check(struct __sk_buff *ctx) +{ + int cnt = 0; + + if (only_count) + return res_count; + bpf_for_each_map_elem(&hashmap, map_cb, &cnt, 0); + return cnt; +} + +SEC("tc") +int exceptions_cleanup_prog_num_iter(struct __sk_buff *ctx) +{ + int i; + + bpf_for(i, 0, 10) { + MARK_RESOURCE(&___it, RES_ITER); + bpf_throw(VAL); + } + return 0; +} + +SEC("tc") +int exceptions_cleanup_prog_num_iter_mult(struct __sk_buff *ctx) +{ + int i, j, k; + + bpf_for(i, 0, 10) { + MARK_RESOURCE(&___it, RES_ITER); + bpf_for(j, 0, 10) { + MARK_RESOURCE(&___it, RES_ITER); + bpf_for(k, 0, 10) { + MARK_RESOURCE(&___it, RES_ITER); + bpf_throw(VAL); + } + } + } + return 0; +} + +__noinline +static int exceptions_cleanup_subprog(struct __sk_buff *ctx) +{ + int i; + + bpf_for(i, 0, 10) { + MARK_RESOURCE(&___it, RES_ITER); + bpf_throw(VAL); + } + return ctx->len; +} + +SEC("tc") +int exceptions_cleanup_prog_dynptr_iter(struct __sk_buff *ctx) +{ + struct bpf_dynptr rbuf; + int ret = 0; + + bpf_ringbuf_reserve_dynptr(&ringbuf, 8, 0, &rbuf); + MARK_RESOURCE(&rbuf, RES_DYNPTR); + if (ctx->protocol) + ret = exceptions_cleanup_subprog(ctx); + bpf_ringbuf_discard_dynptr(&rbuf, 0); + return ret; +} + +SEC("tc") +int exceptions_cleanup_obj(struct __sk_buff *ctx) +{ + struct { int i; } *p; + + p = bpf_obj_new(typeof(*p)); + MARK_RESOURCE(&p, RES_SPILL); + bpf_throw(VAL); + return p->i; +} + +SEC("tc") +int exceptions_cleanup_percpu_obj(struct __sk_buff *ctx) +{ + struct { int i; } *p; + + p = bpf_percpu_obj_new(typeof(*p)); + MARK_RESOURCE(&p, RES_SPILL); + bpf_throw(VAL); + return !p; +} + +SEC("tc") +int exceptions_cleanup_ringbuf(struct __sk_buff *ctx) +{ + void *p; + + p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + MARK_RESOURCE(&p, RES_SPILL); + bpf_throw(VAL); + return 0; +} + +SEC("tc") +int exceptions_cleanup_reg(struct __sk_buff *ctx) +{ + void *p; + + p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + MARK_RESOURCE(p, RES_REG); + bpf_throw(VAL); + if (p) + bpf_ringbuf_discard(p, 0); + return 0; +} + +SEC("tc") +int exceptions_cleanup_null_or_ptr_do_ptr(struct __sk_buff *ctx) +{ + union { + void *p; + char buf[8]; + } volatile p; + u64 z = 0; + + __builtin_memcpy((void *)&p.p, &z, sizeof(z)); + MARK_RESOURCE((void *)&p.p, RES_SPILL); + if (ctx->len) + p.p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + bpf_throw(VAL); + return 0; +} + +SEC("tc") +int exceptions_cleanup_null_or_ptr_do_null(struct __sk_buff *ctx) +{ + union { + void *p; + char buf[8]; + } volatile p; + + p.p = 0; + MARK_RESOURCE((void *)p.buf, RES_SPILL); + if (!ctx->len) + p.p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + bpf_throw(VAL); + return 0; +} + +__noinline static int mark_resource_subprog(u64 a, u64 b, u64 c, u64 d) +{ + MARK_RESOURCE((void *)a, RES_REG); + MARK_RESOURCE((void *)b, RES_REG); + MARK_RESOURCE((void *)c, RES_REG); + MARK_RESOURCE((void *)d, RES_REG); + return 0; +} + +SEC("tc") +int exceptions_cleanup_callee_saved(struct __sk_buff *ctx) +{ + asm volatile ( + "r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + r6 = r0; \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + r7 = r0; \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + r8 = r0; \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + r9 = r0; \ + r1 = r6; \ + r2 = r7; \ + r3 = r8; \ + r4 = r9; \ + call mark_resource_subprog; \ + r1 = 0xeB9F; \ + call bpf_throw; \ + " : : __imm(bpf_ringbuf_reserve), + __imm_addr(ringbuf) + : __clobber_all); + mark_resource_subprog(0, 0, 0, 0); + return 0; +} + +SEC("tc") +int exceptions_cleanup_callee_saved_noopt(struct __sk_buff *ctx) +{ + mark_resource_subprog(1, 2, 3, 4); + return 0; +} + +__noinline int global_subprog_throw(struct __sk_buff *ctx) +{ + u64 *p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + bpf_throw(VAL); + return p ? *p : 0 + ctx->len; +} + +__noinline int global_subprog(struct __sk_buff *ctx) +{ + u64 *p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + if (!p) + return ctx->len; + global_subprog_throw(ctx); + bpf_ringbuf_discard(p, 0); + return !!p + ctx->len; +} + +__noinline static int static_subprog(struct __sk_buff *ctx) +{ + struct bpf_dynptr rbuf; + u64 *p, r = 0; + + bpf_ringbuf_reserve_dynptr(&ringbuf, 8, 0, &rbuf); + p = bpf_dynptr_data(&rbuf, 0, 8); + if (!p) + goto end; + *p = global_subprog(ctx); + r += *p; +end: + bpf_ringbuf_discard_dynptr(&rbuf, 0); + return r + ctx->len; +} + +SEC("tc") +int exceptions_cleanup_frame(struct __sk_buff *ctx) +{ + struct foo { int i; } *p = bpf_obj_new(typeof(*p)); + int i; + only_count = 1; + res_count = 4; + if (!p) + return 1; + p->i = static_subprog(ctx); + i = p->i; + bpf_obj_drop(p); + return i + ctx->len; +} + +SEC("tc") +__success +int exceptions_cleanup_loop_iterations(struct __sk_buff *ctx) +{ + struct { int i; } *f[50] = {}; + int i; + + only_count = true; + + for (i = 0; i < 50; i++) { + f[i] = bpf_obj_new(typeof(*f[0])); + if (!f[i]) + goto end; + res_count++; + if (i == 49) { + bpf_throw(VAL); + } + } +end: + for (i = 0; i < 50; i++) { + if (!f[i]) + continue; + bpf_obj_drop(f[i]); + } + return 0; +} + +SEC("tc") +int exceptions_cleanup_dead_code_elim(struct __sk_buff *ctx) +{ + void *p; + + p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + if (!p) + return 0; + asm volatile ( + "r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + " ::: "r0"); + bpf_throw(VAL); + bpf_ringbuf_discard(p, 0); + return 0; +} + +__noinline int global_subprog_throw_dce(struct __sk_buff *ctx) +{ + u64 *p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + bpf_throw(VAL); + return p ? *p : 0 + ctx->len; +} + +__noinline int global_subprog_dce(struct __sk_buff *ctx) +{ + u64 *p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + if (!p) + return ctx->len; + asm volatile ( + "r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + " ::: "r0"); + global_subprog_throw_dce(ctx); + bpf_ringbuf_discard(p, 0); + return !!p + ctx->len; +} + +__noinline static int static_subprog_dce(struct __sk_buff *ctx) +{ + struct bpf_dynptr rbuf; + u64 *p, r = 0; + + bpf_ringbuf_reserve_dynptr(&ringbuf, 8, 0, &rbuf); + p = bpf_dynptr_data(&rbuf, 0, 8); + if (!p) + goto end; + asm volatile ( + "r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + r0 = r0; \ + " ::: "r0"); + *p = global_subprog_dce(ctx); + r += *p; +end: + bpf_ringbuf_discard_dynptr(&rbuf, 0); + return r + ctx->len; +} + +SEC("tc") +int exceptions_cleanup_frame_dce(struct __sk_buff *ctx) +{ + struct foo { int i; } *p = bpf_obj_new(typeof(*p)); + int i; + only_count = 1; + res_count = 4; + if (!p) + return 1; + p->i = static_subprog_dce(ctx); + i = p->i; + bpf_obj_drop(p); + return i + ctx->len; +} + +SEC("tc") +int reject_slot_with_zero_vs_ptr_ok(struct __sk_buff *ctx) +{ + asm volatile ( + "r7 = *(u32 *)(r1 + 0); \ + r0 = 0; \ + *(u64 *)(r10 - 8) = r0; \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + if r7 != 0 goto jump4; \ + call %[bpf_ringbuf_reserve]; \ + *(u64 *)(r10 - 8) = r0; \ + jump4: \ + r0 = 0; \ + r1 = 0; \ + call bpf_throw; \ + " : : __imm(bpf_ringbuf_reserve), + __imm_addr(ringbuf) + : "memory"); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c new file mode 100644 index 000000000000..b3c70f92b35f --- /dev/null +++ b/tools/testing/selftests/bpf/progs/exceptions_cleanup_fail.c @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include + +#include "bpf_misc.h" +#include "bpf_experimental.h" + +struct { + __uint(type, BPF_MAP_TYPE_RINGBUF); + __uint(max_entries, 8); +} ringbuf SEC(".maps"); + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_with_reference(void *ctx) +{ + struct { int i; } *f; + + f = bpf_obj_new(typeof(*f)); + if (!f) + return 0; + bpf_throw(0); + return 0; +} + +SEC("?tc") +__failure __msg("frame_desc: merge: failed to merge old and new frame desc entry") +int reject_slot_with_distinct_ptr(struct __sk_buff *ctx) +{ + void *p; + + if (ctx->len) { + p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + } else { + p = bpf_obj_new(typeof(struct { int i; })); + } + bpf_throw(0); + return !p; +} + +SEC("?tc") +__failure __msg("frame_desc: merge: failed to merge old and new frame desc entry") +int reject_slot_with_distinct_ptr_old(struct __sk_buff *ctx) +{ + void *p; + + if (ctx->len) { + p = bpf_obj_new(typeof(struct { int i; })); + } else { + p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + } + bpf_throw(0); + return !p; +} + +SEC("?tc") +__failure __msg("frame_desc: merge: failed to merge old and new frame desc entry") +int reject_slot_with_misc_vs_ptr(struct __sk_buff *ctx) +{ + void *p = (void *)bpf_ktime_get_ns(); + + if (ctx->protocol) + p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + bpf_throw(0); + return !p; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_slot_with_misc_vs_ptr_old(struct __sk_buff *ctx) +{ + void *p = bpf_ringbuf_reserve(&ringbuf, 8, 0); + + if (ctx->protocol) + p = (void *)bpf_ktime_get_ns(); + bpf_throw(0); + return !p; +} + +SEC("?tc") +__failure __msg("frame_desc: merge: failed to merge old and new frame desc entry") +int reject_slot_with_invalid_vs_ptr(struct __sk_buff *ctx) +{ + asm volatile ( + "r7 = r1; \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + r4 = *(u32 *)(r7 + 0); \ + r6 = *(u64 *)(r10 - 8); \ + if r4 == 0 goto jump; \ + call %[bpf_ringbuf_reserve]; \ + r6 = r0; \ + jump: \ + r0 = 0; \ + r1 = 0; \ + call bpf_throw; \ + " : : __imm(bpf_ringbuf_reserve), + __imm_addr(ringbuf) + : "memory"); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_slot_with_invalid_vs_ptr_old(struct __sk_buff *ctx) +{ + asm volatile ( + "r7 = r1; \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + r6 = r0; \ + r4 = *(u32 *)(r7 + 0); \ + if r4 == 0 goto jump2; \ + r6 = *(u64 *)(r10 - 8); \ + jump2: \ + r0 = 0; \ + r1 = 0; \ + call bpf_throw; \ + " : : __imm(bpf_ringbuf_reserve), + __imm_addr(ringbuf) + : "memory"); + return 0; +} + +SEC("?tc") +__failure __msg("Unreleased reference") +int reject_slot_with_zero_vs_ptr(struct __sk_buff *ctx) +{ + asm volatile ( + "r7 = *(u32 *)(r1 + 0); \ + r1 = %[ringbuf] ll; \ + r2 = 8; \ + r3 = 0; \ + call %[bpf_ringbuf_reserve]; \ + *(u64 *)(r10 - 8) = r0; \ + r0 = 0; \ + if r7 != 0 goto jump3; \ + *(u64 *)(r10 - 8) = r0; \ + jump3: \ + r0 = 0; \ + r1 = 0; \ + call bpf_throw; \ + " : : __imm(bpf_ringbuf_reserve), + __imm_addr(ringbuf) + : "memory"); + return 0; +} + +char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c index dfd164a7a261..1e73200c6276 100644 --- a/tools/testing/selftests/bpf/progs/exceptions_fail.c +++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c @@ -182,19 +182,6 @@ int reject_with_rbtree_add_throw(void *ctx) return 0; } -SEC("?tc") -__failure __msg("Unreleased reference") -int reject_with_reference(void *ctx) -{ - struct foo *f; - - f = bpf_obj_new(typeof(*f)); - if (!f) - return 0; - bpf_throw(0); - return 0; -} - __noinline static int subprog_ref(struct __sk_buff *ctx) { struct foo *f;