From patchwork Thu Feb 1 12:52:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13541044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 88AB4C4828C for ; Thu, 1 Feb 2024 12:53:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uZ85kyGZzCeVW3ax4pB1aw2eqng8kgGpCECGV/nwcso=; b=Ex3/EiTXeoH0et A7lP2RJf8e7Kx53hz82o07/CGfBASFwDXB6EDi856d8yAkSsGX5oPelPTI1dMzTKbpBAdBUra0d4e pOK6/W2T0y16+czHM2JcBFXgcPpIbB277+RgXu09KrUO6PKU8DUQOaKCz8NvdPhlTsyfhRQxiL9vA +sVluhuzSHHstT/Uaj8LX4vVpR+U2Oh0eUDgIk6KaZnVsbQrn8curdfTrXqarnFci/rTNAoa7crtQ tM3q20ob+Vh9REVB/1PDuVssxQKEQ8HUBu9OSuYky/7mtDq2Dhf3I4OnmhvQoghXzd3akXd1n/2Bb scNU9C9R8EexjanOCgCw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVWYn-00000007q4g-1fj8; Thu, 01 Feb 2024 12:52:53 +0000 Received: from mail-wm1-x32e.google.com ([2a00:1450:4864:20::32e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVWYk-00000007q3A-3OJZ for linux-arm-kernel@lists.infradead.org; Thu, 01 Feb 2024 12:52:51 +0000 Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-40e72a567eeso8187215e9.0 for ; Thu, 01 Feb 2024 04:52:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706791968; x=1707396768; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9Z+EJFZpEuuwaeylDdMVXwrB+zfZxw9ouTIGewC5rpI=; b=KN9Z4kf85Uaou24XyEhdk0QcJ4Egdig5hPEh72qe8hf65rlwgvUOvHfNyUW7D/UGzG pj2wt/0m+La2/u50tHAAboy+CPYYr6ob1EU5wIheg6AnuRwiPoZd3frG9qY5dCZNipWG deIIQeRiTFcyogwX4pZt26J6kglS7smfdjnMt0LD8GDxJgr6Oi8wkPUi+DTtg1/cv5XP EfNTrHucUzrE4n+OwUQ9b5pu+rdcPFZ6R70fGflBYBCTm0tq0KP8lD/C55U9INm40NcU J1mjqyX6YoSuEk3DFhQbQfuoU72L+PMOzb4QPIceoaWHQZVxB30oovnDO0F3OnsynvE6 P3Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706791968; x=1707396768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9Z+EJFZpEuuwaeylDdMVXwrB+zfZxw9ouTIGewC5rpI=; b=U7410GFTe72DFs0EOEhN2Mkaq0dDUUOfmo8EJNzxAJLnnczmxPXWKTTWP/Ugx66D8g mIVQsXOEYep/rHT0THjq4gqoZifXPzONfEhugPIBxbp45Ia30gowlSNBFFJJUSju/A9c kiuEbwd9y9D0O6vW+0XnGgO8TBnXlEDuSG64Ftz2DnAjMFK91FwZz2Wh8brpot4+1DSx POsyHKEMQIfobKSqfquuA2zeUfR1GBX08HvTZj4ygtwjMlir+QaFdxqjYiHnraD642LN hI7xYUWLQQvXX6G5SsBjbafy5VmvnEna32G57CW5Zjj1LutFEWCuRuYE3hen17Xv2XM8 EFJQ== X-Gm-Message-State: AOJu0YxWiTPY6s0C0fmmNxO9ahuhSxdK2OppqL6q2is5bm0ySQtrSYUr JDGA1ymLoNIFbZzjYr3OpWpo4n/WiecWCRwXpoQDIIpHjPSGwc8t X-Google-Smtp-Source: AGHT+IGTMK7XQp7iF4IDr0TpDZGBRyCM3MhuXPCpoD1p/qdWnwKIbPEdTe/VC1xYZeppYuIxQN0xxA== X-Received: by 2002:a05:600c:4f82:b0:40e:f536:cff2 with SMTP id n2-20020a05600c4f8200b0040ef536cff2mr4074854wmq.2.1706791968330; Thu, 01 Feb 2024 04:52:48 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWHgD5rN5YetrM1Nc56KSl9RzlcGotJ0aNpjd7oRytX1ETCxj/luETj2mKRMRT/6tOjFLhV3KzLjku6oKWsr3sdZjR/lRNVZbxeH37ztTwfRsYQFai/FawAturIJcS4cPjGhksMK2oNZ/aAeECcX41YR6iM3lWfAFcrLn71AiZZOX2NnDUTaKlQJ6F9pTYjiHXSM86scoeNUwgEXCjSgMRM/l80oET8sroXz0XXtl2A4i0RnClVo0f7BoiNXZBx1E675M1YTBlPMAxmUM0rBYndbriX3jvAQsvodBdIDMw9xSA0UNv4V9RGT6mJCM9VczcBxzY/1hO+nNZvnqXUdlU6DSChxfoDizqLYF2DNhe77khIkkOKpod9LXiXWSEV9CU6OpGtEotiTC+rLmr0EJAdHxcwxGIxoTODtR7RKURAup2AX2covHHMF8AYVE6w8a8jDvvZVffkcWF4kGy5lHm0ZsQD8paU1OThlaVI3DfVy/Q95QwkracQrdKgiW3M7V1PXtuzYqLonwEx+MjkzGwOp6ptbnFJeP5ix3an7u5e8vJuKA== Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id j13-20020a05600c190d00b0040e88d1422esm4397968wmq.31.2024.02.01.04.52.48 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Feb 2024 04:52:48 -0800 (PST) From: Puranjay Mohan To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Catalin Marinas , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Kumar Kartikeya Dwivedi Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v3 1/2] arm64: stacktrace: Implement arch_bpf_stack_walk() for the BPF JIT Date: Thu, 1 Feb 2024 12:52:24 +0000 Message-Id: <20240201125225.72796-2-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201125225.72796-1-puranjay12@gmail.com> References: <20240201125225.72796-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240201_045250_881138_EBBA70DD X-CRM114-Status: GOOD ( 13.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org This will be used by bpf_throw() to unwind till the program marked as exception boundary and run the callback with the stack of the main program. This is required for supporting BPF exceptions on ARM64. Signed-off-by: Puranjay Mohan Acked-by: Catalin Marinas --- arch/arm64/kernel/stacktrace.c | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c index 7f88028a00c0..66cffc5fc0be 100644 --- a/arch/arm64/kernel/stacktrace.c +++ b/arch/arm64/kernel/stacktrace.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -266,6 +267,31 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry, kunwind_stack_walk(arch_kunwind_consume_entry, &data, task, regs); } +struct bpf_unwind_consume_entry_data { + bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp); + void *cookie; +}; + +static bool +arch_bpf_unwind_consume_entry(const struct kunwind_state *state, void *cookie) +{ + struct bpf_unwind_consume_entry_data *data = cookie; + + return data->consume_entry(data->cookie, state->common.pc, 0, + state->common.fp); +} + +noinline noinstr void arch_bpf_stack_walk(bool (*consume_entry)(void *cookie, u64 ip, u64 sp, + u64 fp), void *cookie) +{ + struct bpf_unwind_consume_entry_data data = { + .consume_entry = consume_entry, + .cookie = cookie, + }; + + kunwind_stack_walk(arch_bpf_unwind_consume_entry, &data, current, NULL); +} + static bool dump_backtrace_entry(void *arg, unsigned long where) { char *loglvl = arg; From patchwork Thu Feb 1 12:52:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Puranjay Mohan X-Patchwork-Id: 13541045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 037D6C47DDB for ; Thu, 1 Feb 2024 12:53:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=x6uV+GskrLo/Fw67uvPeBviaKstFRH/K+G6m8rBDgz4=; b=5F6BNYuPEK4VVZ PNTNYfiDop39sbhFQWm7QSD7iTDxDAxjY2R3GHMffNqzvl0uQLwAkZ8AfmTvQL/dpnfMh73nq1pCD I+jQSCjdZVKWN/oq/pVOQhnvPKSGxCgaVGsVZ6K6dAi489fmCzoiPGLbMP0JO657vvpRcXM/XgQGt F62eC7h4fdetkAC6BZzjXH+KlTS5zxSHdn6xZ1GA/wt0utW40J5K0ssWVrSFpyrgqcN563+j1Wo1x 62gz2pPbR5EqNMb0bJTKyUxvBP1RalLwZLa8I1EOcUioAyjDwcXLLRo4konwqabDIp8JmGaGzVcgg lRyO5oPz2dGdtahfnueA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVWYq-00000007q5q-0pQB; Thu, 01 Feb 2024 12:52:56 +0000 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVWYm-00000007q41-3dWg for linux-arm-kernel@lists.infradead.org; Thu, 01 Feb 2024 12:52:54 +0000 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-40ef6da20feso4879315e9.0 for ; Thu, 01 Feb 2024 04:52:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706791970; x=1707396770; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TLwU1mAjRk6tVu2Bx5aUYeeeXQpRVOBvWkTmhqcFn0M=; b=il0+3/S2tlg9ogSruPQT3oKfNeJre55hKB087ADRbhesDq6bKqp8dqz4Z+x2bjgPQp La8bh9XCso20bCQJ/8LVauGuMjaTK3NlVlMOlYvqmOjF9uuzkbbMmCWpNjSD5Qu6U4HD xtuD0IhbRJ2043w8dS9+MQNH4X5SDSydXvLmQ2/yixyoOTylcgWvaTGn5zS38q/LE491 CQdYfM155oJuP2fGag/JRwGwdU6ENIxU/K33z/e97Gvsct/FAz7ZYlJizuBkCRYEMyI1 gtGyk8rmZGHeWaxoF92LQVtBnEMAX/QdRF7CTDLw9ePLZs98adpgE1NS/EzkAUVSUvcj 5JQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706791970; x=1707396770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TLwU1mAjRk6tVu2Bx5aUYeeeXQpRVOBvWkTmhqcFn0M=; b=k57mqcQ6WXv7x4iuVdtjPf3nHq8xBm3kCZMBYFm/EG6jqLj906/5xjK/8TCOEX/Yqb 43x8jr9nJBETcBUf9pOsc8gMZPOQbB3cOWBIflGufc3IDT89ti28m5pfOtcUMa0l4KeG nq+7QJIY8jzLWo/YmA0vDZr6ydIsuHMpRS14q48flXmR/JAJkq2o4tnfz1XaQODEgwbi 0OPBHeyJoidMubwj9n13Z8dmAwZRIjhp6C/MDGyNIYVkXRS19PjczvIsZI1RgtAqMtae uWQ+tPYNdvqqf6CXQwhLrf0GP3+V5ImfPwwj7dn4rbZZn8y6fyrqjmTCBk8LHNrvfl3y Sz3A== X-Gm-Message-State: AOJu0YzNZ1K7RgXzxJ/AVhMwy6KO4/ZgrGZgPobCvl0AOY8rvsFAP5kT I5M6L/UYo9PLMdzAr2F5guvDxbsU7pInsvIcW7Rx8/2cXLgMNHDk X-Google-Smtp-Source: AGHT+IFGz6afCU/24Gt4i46Ecnb88g6ZlJ+iaxfz/q3qWmZcs6vTMg4kJ8EW+MqV8A34TQ9ij7WoSg== X-Received: by 2002:a05:600c:3b28:b0:40f:b76e:e261 with SMTP id m40-20020a05600c3b2800b0040fb76ee261mr2619886wms.9.1706791970527; Thu, 01 Feb 2024 04:52:50 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCXC1Lt6JUZ+hdm9QD3AOAZ1hsC5ynhzSr/lhQzOhIQijFN6d734bRAmjeMLNUrAVXYs1z5PP8kzglCZRLX3mKwuFhQ9qsVE6M7p1F28XM8kDJYlmk8+tsAoJkCh22zzSanTqWFSLlAl2lfGbUd2oQg9EOVeYHDNxQR8xpj6T3sMaShhmd9G03GV9eNPjhNUzEYWkf3ppm+ExzdWxGkfrynxNSP6gyOkzY4z1YUNa+JVwYGNIDpV6KwBEbRKDd23Y77ICCRk2DcYYbw75RUr7e35DeuO9HbZATYZ6QfshRClTPwY9+hwQbc0J2b3DQEvS64oDWDfwzmvWKinmHYQ0v+tWjvQisZU1hKHZ+Hvs5zSAfKCFtveFCOWe0QAs0uNlYpxek7qtYnLnDRlL5afyvz7VmycJwu/CF7xTdntTdUe4J5kphK0F45fq1+HEiJSj8E9hhOv6XV91GgpGwrtRXJD0oo2kDUwm8SeZiIWlrwNOeJRjPcFvX2PDP43h/eFzjoX0X4RNl70wWyqwnp/RRpp95w2dGNr9RBQph2ndBqEcKVvAA== Received: from localhost (54-240-197-231.amazon.com. [54.240.197.231]) by smtp.gmail.com with ESMTPSA id e1-20020a05600c4e4100b0040ee4f38968sm4425715wmq.2.2024.02.01.04.52.50 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Feb 2024 04:52:50 -0800 (PST) From: Puranjay Mohan To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Zi Shen Lim , Catalin Marinas , Will Deacon , bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Kumar Kartikeya Dwivedi Cc: puranjay12@gmail.com Subject: [PATCH bpf-next v3 2/2] bpf, arm64: support exceptions Date: Thu, 1 Feb 2024 12:52:25 +0000 Message-Id: <20240201125225.72796-3-puranjay12@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20240201125225.72796-1-puranjay12@gmail.com> References: <20240201125225.72796-1-puranjay12@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240201_045252_939919_B5A7E91D X-CRM114-Status: GOOD ( 22.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The prologue generation code has been modified to make the callback program use the stack of the program marked as exception boundary where callee-saved registers are already pushed. As the bpf_throw function never returns, if it clobbers any callee-saved registers, they would remain clobbered. So, the prologue of the exception-boundary program is modified to push R23 and R24 as well, which the callback will then recover in its epilogue. The Procedure Call Standard for the Arm 64-bit Architecture[1] states that registers r19 to r28 should be saved by the callee. BPF programs on ARM64 already save all callee-saved registers except r23 and r24. This patch adds an instruction in prologue of the program to save these two registers and another instruction in the epilogue to recover them. These extra instructions are only added if bpf_throw() is used. Otherwise the emitted prologue/epilogue remains unchanged. [1] https://github.com/ARM-software/abi-aa/blob/main/aapcs64/aapcs64.rst Signed-off-by: Puranjay Mohan --- arch/arm64/net/bpf_jit_comp.c | 87 +++++++++++++++----- tools/testing/selftests/bpf/DENYLIST.aarch64 | 1 - 2 files changed, 68 insertions(+), 20 deletions(-) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index cfd5434de483..20720ec346b8 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -285,7 +285,8 @@ static bool is_lsi_offset(int offset, int scale) /* Tail call offset to jump into */ #define PROLOGUE_OFFSET (BTI_INSNS + 2 + PAC_INSNS + 8) -static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) +static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf, + bool is_exception_cb) { const struct bpf_prog *prog = ctx->prog; const bool is_main_prog = !bpf_is_subprog(prog); @@ -333,19 +334,34 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) emit(A64_MOV(1, A64_R(9), A64_LR), ctx); emit(A64_NOP, ctx); - /* Sign lr */ - if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) - emit(A64_PACIASP, ctx); - - /* Save FP and LR registers to stay align with ARM64 AAPCS */ - emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); - emit(A64_MOV(1, A64_FP, A64_SP), ctx); - - /* Save callee-saved registers */ - emit(A64_PUSH(r6, r7, A64_SP), ctx); - emit(A64_PUSH(r8, r9, A64_SP), ctx); - emit(A64_PUSH(fp, tcc, A64_SP), ctx); - emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx); + if (!is_exception_cb) { + /* Sign lr */ + if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL)) + emit(A64_PACIASP, ctx); + /* Save FP and LR registers to stay align with ARM64 AAPCS */ + emit(A64_PUSH(A64_FP, A64_LR, A64_SP), ctx); + emit(A64_MOV(1, A64_FP, A64_SP), ctx); + + /* Save callee-saved registers */ + emit(A64_PUSH(r6, r7, A64_SP), ctx); + emit(A64_PUSH(r8, r9, A64_SP), ctx); + emit(A64_PUSH(fp, tcc, A64_SP), ctx); + emit(A64_PUSH(fpb, A64_R(28), A64_SP), ctx); + } else { + /* + * Exception callback receives FP of Main Program as third + * parameter + */ + emit(A64_MOV(1, A64_FP, A64_R(2)), ctx); + /* + * Main Program already pushed the frame record and the + * callee-saved registers. The exception callback will not push + * anything and re-use the main program's stack. + * + * 10 registers are on the stack + */ + emit(A64_SUB_I(1, A64_SP, A64_FP, 80), ctx); + } /* Set up BPF prog stack base register */ emit(A64_MOV(1, fp, A64_SP), ctx); @@ -365,6 +381,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) emit_bti(A64_BTI_J, ctx); } + /* + * Program acting as exception boundary should save all ARM64 + * Callee-saved registers as the exception callback needs to recover + * all ARM64 Callee-saved registers in its epilogue. + */ + if (prog->aux->exception_boundary) { + /* + * As we are pushing two more registers, BPF_FP should be moved + * 16 bytes + */ + emit(A64_SUB_I(1, fp, fp, 16), ctx); + emit(A64_PUSH(A64_R(23), A64_R(24), A64_SP), ctx); + } + emit(A64_SUB_I(1, fpb, fp, ctx->fpb_offset), ctx); /* Stack must be multiples of 16B */ @@ -653,7 +683,7 @@ static void build_plt(struct jit_ctx *ctx) plt->target = (u64)&dummy_tramp; } -static void build_epilogue(struct jit_ctx *ctx) +static void build_epilogue(struct jit_ctx *ctx, bool is_exception_cb) { const u8 r0 = bpf2a64[BPF_REG_0]; const u8 r6 = bpf2a64[BPF_REG_6]; @@ -666,6 +696,15 @@ static void build_epilogue(struct jit_ctx *ctx) /* We're done with BPF stack */ emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); + /* + * Program acting as exception boundary pushes R23 and R24 in addition + * to BPF callee-saved registers. Exception callback uses the boundary + * program's stack frame, so recover these extra registers in the above + * two cases. + */ + if (ctx->prog->aux->exception_boundary || is_exception_cb) + emit(A64_POP(A64_R(23), A64_R(24), A64_SP), ctx); + /* Restore x27 and x28 */ emit(A64_POP(fpb, A64_R(28), A64_SP), ctx); /* Restore fs (x25) and x26 */ @@ -1575,7 +1614,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) * BPF line info needs ctx->offset[i] to be the offset of * instruction[i] in jited image, so build prologue first. */ - if (build_prologue(&ctx, was_classic)) { + if (build_prologue(&ctx, was_classic, prog->aux->exception_cb)) { prog = orig_prog; goto out_off; } @@ -1586,7 +1625,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) } ctx.epilogue_offset = ctx.idx; - build_epilogue(&ctx); + build_epilogue(&ctx, prog->aux->exception_cb); build_plt(&ctx); extable_align = __alignof__(struct exception_table_entry); @@ -1614,7 +1653,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) ctx.idx = 0; ctx.exentry_idx = 0; - build_prologue(&ctx, was_classic); + build_prologue(&ctx, was_classic, prog->aux->exception_cb); if (build_body(&ctx, extra_pass)) { bpf_jit_binary_free(header); @@ -1622,7 +1661,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto out_off; } - build_epilogue(&ctx); + build_epilogue(&ctx, prog->aux->exception_cb); build_plt(&ctx); /* 3. Extra pass to validate JITed code. */ @@ -2310,3 +2349,13 @@ bool bpf_jit_supports_ptr_xchg(void) { return true; } + +bool bpf_jit_supports_exceptions(void) +{ + /* We unwind through both kernel frames starting from within bpf_throw + * call and BPF frames. Therefore we require FP unwinder to be enabled + * to walk kernel frames and reach BPF frames in the stack trace. + * ARM64 kernel is aways compiled with CONFIG_FRAME_POINTER=y + */ + return true; +} diff --git a/tools/testing/selftests/bpf/DENYLIST.aarch64 b/tools/testing/selftests/bpf/DENYLIST.aarch64 index 5c2cc7e8c5d0..0445ac38bc07 100644 --- a/tools/testing/selftests/bpf/DENYLIST.aarch64 +++ b/tools/testing/selftests/bpf/DENYLIST.aarch64 @@ -1,6 +1,5 @@ bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3 -exceptions # JIT does not support calling kfunc bpf_throw: -524 fexit_sleep # The test never returns. The remaining tests cannot start. kprobe_multi_bench_attach # needs CONFIG_FPROBE kprobe_multi_test # needs CONFIG_FPROBE