From patchwork Mon Mar 3 13:28:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13998858 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D609212FBD; Mon, 3 Mar 2025 13:30:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.196 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008639; cv=none; b=AWSbMRKrTqaYwk1ZZeN4fRlGNZExpuz31IBj34SUxAl2k/gfCAG7ABDA0IlrgmRudQPC/E2tfqnftQM2GBLvMU24mHGzAtMTUuxRsZGVuBxEs7d6xVik2zUyqv5XEvFYn8t/Bo6RBb7JdS88Ec8gA+AYMaFDV5WPwypmFoAtWT0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008639; c=relaxed/simple; bh=Jrm+7746S7K9yWA40dUMBKTAN0ud8hVu4FIcLZVjvE0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d0Ds2jrVdx7L4s4xMmai1nt9ue5+X3d3E1LHowYJcThCCfw5RAm03/hKZuEiwuo/SJJfMEuTU91onreXP0ezh2ZthIZXx4uUR4jUoxn2S/VDjXYzyUnO5Tx9HloaoyNZjdabmSH70BztHKNYA6mBo407gboZd36EocwVOZlmq0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SEfylcch; arc=none smtp.client-ip=209.85.214.196 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SEfylcch" Received: by mail-pl1-f196.google.com with SMTP id d9443c01a7336-22382657540so32195485ad.2; Mon, 03 Mar 2025 05:30:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741008637; x=1741613437; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4HoYAW6D8hO+aaE9DpE+f5d9KWUfqlQeSUmUL0RBP80=; b=SEfylcchyw/AsFqZ/ZmtJOBYtRr/sV0ONWVuUVJBVI/pgrL2ixQJhqZnpZCCkmf+4T gvWRtbCsLcRH7cVRshFHmkDyxGDJyfGg42kAlri0S4Sk2NfVTOUMNu4qKaLG58f12xr5 LfUQwMdFNyIKo7rnCglKXOwqD60X9Jp7cFvlhMEIMxtvDM2txe9GVbMEPr4TDnhy+jn0 ZHfoKKyyp7AgoIo/i1ALr9iUJgD1hgczPiq6aihl0rSmsVXCkwwuxTNw1TA2Z+cV/tjI CueFCepWMLyEqGa2VH0doLyJkAbK2IgS3Cl5f9yiKbj0W+RbeTHxNPlJe8frFU4v7Hu9 XLEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741008637; x=1741613437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4HoYAW6D8hO+aaE9DpE+f5d9KWUfqlQeSUmUL0RBP80=; b=sudlHx5ktu+ipwUF/WPODvPa6CaoPDRUJNVyRO3EwL6858EM7IHdmMeeh61MJnTTA9 GZdv4sxLwnT5l7FtjZApusG2MbXgxiC9Tltvjm4LlX2MJG1V36nug6Rtolj+dHTtnvG1 hBf86auTXDhPpzAPRnao0B2RPN6DL5H41APueC0fe/NRXe3qB5Hjx3D43IvnN6VprKJB K/F4QOoJ9wyiDV48EkW/ugw4/42HtcUiXH8VIIuyX45n19wQ/zp8E+FAzH6SPp3SZKTz +ZMb1Kai9wJlG9ZFJQXWCMQPKZRjW0xrwRx19ukuely4EaGLKsrOQtuAz6m/xnpqfTmk g2Tw== X-Forwarded-Encrypted: i=1; AJvYcCU/z4xUdFZi9xKRQRV4oiWwD6sUW1RSMcIrYxVNiT9M5pc9rlB/OPIpOg22QIZWUe+5BBDBcAOa1xfGGhMaowP7BCZc@vger.kernel.org, AJvYcCVLVqflwQEpe24hWK1lbOmzbh1qJK7+tn4ra0yBvwpjWbFX7EyOl5ynzl6UPBrAawqK9GMtUztef0g1kx8I@vger.kernel.org, AJvYcCVsRfQMiMuPj7KNaiIGVq3t7asm1TmRHYTEedHUAAi0edCg1HZAuJvtuQ1Vpf047klUzZ30MarC@vger.kernel.org, AJvYcCWYVf1VzNkN1SodqqE0quIJXy7DFDAobGTvniBZz4kWncoxU9h9r46aDx84rqZLGdKRF20=@vger.kernel.org X-Gm-Message-State: AOJu0YwIkq7Q2dZsz1ZoJ1dJ+LfoBlRXM0EKfcrSr/chvP0q501E7ay6 G6VhabWKOJq7N2ozW8UigveyxXFaJCOoQxFuna5XacYRbsdSzFm8 X-Gm-Gg: ASbGncvY0ivwP4Hr6FZqHcg4npdetYn4hjMk3V7cU+1dO/LYFUW2+hz+Qd1w1/B3jEb z1y4/8USATTXocQNLZ4aqrLrxwczmpVRfDANqMbBR3k3sSSuzF1ewdaTMbLyGUDN4ykfbzM5Z8M 8kRoxnKglk/IlJ9LzPEB624zbASbnhiRsQs3zlRJYIgJNJ/4Af3V0PD6q5Xcq4ybQNJFF5RHfAh OiNgTuBuTD5y5mHamwQwzgM9hOHQGjEPJp/MLREJVSoFyjYlOoT1MB2NvWGaQJl5RzuFtvP29Fb +8KXSSRXStCUKHyCEGep/R+CZiNgIXBxYjVFwZh2HFuO0xTk62FjAhjoIfFKUQ== X-Google-Smtp-Source: AGHT+IGA2mWJTfvekJWlfyFWg7JZnBJRMEmVPtYCpiX/IyOQcjy5rGPgwuYYTPg+xM28+Vd56S613A== X-Received: by 2002:a17:903:230c:b0:223:7006:4db2 with SMTP id d9443c01a7336-22370064ea3mr156649035ad.31.1741008636600; Mon, 03 Mar 2025 05:30:36 -0800 (PST) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223505359b8sm77297035ad.253.2025.03.03.05.30.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 05:30:36 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: peterz@infradead.org, rostedt@goodmis.org, mark.rutland@arm.com, alexei.starovoitov@gmail.com Cc: catalin.marinas@arm.com, will@kernel.org, mhiramat@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, jolsa@kernel.org, davem@davemloft.net, dsahern@kernel.org, mathieu.desnoyers@efficios.com, nathan@kernel.org, nick.desaulniers+lkml@gmail.com, morbo@google.com, samitolvanen@google.com, kees@kernel.org, dongml2@chinatelecom.cn, akpm@linux-foundation.org, riel@surriel.com, rppt@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, netdev@vger.kernel.org, llvm@lists.linux.dev Subject: [PATCH v4 1/4] x86/ibt: factor out cfi and fineibt offset Date: Mon, 3 Mar 2025 21:28:34 +0800 Message-Id: <20250303132837.498938-2-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303132837.498938-1-dongml2@chinatelecom.cn> References: <20250303132837.498938-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For now, the layout of cfi and fineibt is hard coded, and the padding is fixed on 16 bytes. Factor out FINEIBT_INSN_OFFSET and CFI_INSN_OFFSET. CFI_INSN_OFFSET is the offset of cfi, which is the same as FUNCTION_ALIGNMENT when CALL_PADDING is enabled. And FINEIBT_INSN_OFFSET is the offset where we put the fineibt preamble on, which is 16 for now. When the FUNCTION_ALIGNMENT is bigger than 16, we place the fineibt preamble on the last 16 bytes of the padding for better performance, which means the fineibt preamble don't use the space that cfi uses. The FINEIBT_INSN_OFFSET is not used in fineibt_caller_start and fineibt_paranoid_start, as it is always "0x10". Note that we need to update the offset in fineibt_caller_start and fineibt_paranoid_start if FINEIBT_INSN_OFFSET changes. Signed-off-by: Menglong Dong --- v4: - rebase to the newest tip/x86/core, the fineibt has some updating --- arch/x86/include/asm/cfi.h | 13 +++++++++---- arch/x86/kernel/alternative.c | 18 +++++++++++------- arch/x86/net/bpf_jit_comp.c | 22 +++++++++++----------- 3 files changed, 31 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/cfi.h b/arch/x86/include/asm/cfi.h index 2f6a01f098b5..04525f2f6bf2 100644 --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -108,6 +108,14 @@ extern bhi_thunk __bhi_args_end[]; struct pt_regs; +#ifdef CONFIG_CALL_PADDING +#define FINEIBT_INSN_OFFSET 16 +#define CFI_INSN_OFFSET CONFIG_FUNCTION_ALIGNMENT +#else +#define FINEIBT_INSN_OFFSET 0 +#define CFI_INSN_OFFSET 5 +#endif + #ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); #define __bpfcall @@ -118,11 +126,8 @@ static inline int cfi_get_offset(void) { switch (cfi_mode) { case CFI_FINEIBT: - return 16; case CFI_KCFI: - if (IS_ENABLED(CONFIG_CALL_PADDING)) - return 16; - return 5; + return CFI_INSN_OFFSET; default: return 0; } diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index 32e4b801db99..0088d2313f33 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -917,7 +917,7 @@ void __init_or_module noinline apply_seal_endbr(s32 *start, s32 *end) poison_endbr(addr); if (IS_ENABLED(CONFIG_FINEIBT)) - poison_cfi(addr - 16); + poison_cfi(addr); } } @@ -980,12 +980,13 @@ u32 cfi_get_func_hash(void *func) { u32 hash; - func -= cfi_get_offset(); switch (cfi_mode) { case CFI_FINEIBT: + func -= FINEIBT_INSN_OFFSET; func += 7; break; case CFI_KCFI: + func -= CFI_INSN_OFFSET; func += 1; break; default: @@ -1372,7 +1373,7 @@ static int cfi_rewrite_preamble(s32 *start, s32 *end) * have determined there are no indirect calls to it and we * don't need no CFI either. */ - if (!is_endbr(addr + 16)) + if (!is_endbr(addr + CFI_INSN_OFFSET)) continue; hash = decode_preamble_hash(addr, &arity); @@ -1380,6 +1381,7 @@ static int cfi_rewrite_preamble(s32 *start, s32 *end) addr, addr, 5, addr)) return -EINVAL; + addr += (CFI_INSN_OFFSET - FINEIBT_INSN_OFFSET); text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size); WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) != 0x12345678); text_poke_early(addr + fineibt_preamble_hash, &hash, 4); @@ -1402,10 +1404,10 @@ static void cfi_rewrite_endbr(s32 *start, s32 *end) for (s = start; s < end; s++) { void *addr = (void *)s + *s; - if (!exact_endbr(addr + 16)) + if (!exact_endbr(addr + CFI_INSN_OFFSET)) continue; - poison_endbr(addr + 16); + poison_endbr(addr + CFI_INSN_OFFSET); } } @@ -1543,12 +1545,12 @@ static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, return; case CFI_FINEIBT: - /* place the FineIBT preamble at func()-16 */ + /* place the FineIBT preamble at func()-FINEIBT_INSN_OFFSET */ ret = cfi_rewrite_preamble(start_cfi, end_cfi); if (ret) goto err; - /* rewrite the callers to target func()-16 */ + /* rewrite the callers to target func()-FINEIBT_INSN_OFFSET */ ret = cfi_rewrite_callers(start_retpoline, end_retpoline); if (ret) goto err; @@ -1588,6 +1590,7 @@ static void poison_cfi(void *addr) */ switch (cfi_mode) { case CFI_FINEIBT: + addr -= FINEIBT_INSN_OFFSET; /* * FineIBT prefix should start with an ENDBR. */ @@ -1607,6 +1610,7 @@ static void poison_cfi(void *addr) break; case CFI_KCFI: + addr -= CFI_INSN_OFFSET; /* * kCFI prefix should start with a valid hash. */ diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 72776dcb75aa..ee86a5df5ffb 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -415,6 +415,12 @@ static int emit_call(u8 **prog, void *func, void *ip); static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int arity) { u8 *prog = *pprog; +#ifdef CONFIG_CALL_PADDING + int i; + + for (i = 0; i < CFI_INSN_OFFSET - 16; i++) + EMIT1(0x90); +#endif EMIT_ENDBR(); EMIT3_off32(0x41, 0x81, 0xea, hash); /* subl $hash, %r10d */ @@ -432,20 +438,14 @@ static void emit_fineibt(u8 **pprog, u8 *ip, u32 hash, int arity) static void emit_kcfi(u8 **pprog, u32 hash) { u8 *prog = *pprog; +#ifdef CONFIG_CALL_PADDING + int i; +#endif EMIT1_off32(0xb8, hash); /* movl $hash, %eax */ #ifdef CONFIG_CALL_PADDING - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); - EMIT1(0x90); + for (i = 0; i < CFI_INSN_OFFSET - 5; i++) + EMIT1(0x90); #endif EMIT_ENDBR(); From patchwork Mon Mar 3 13:28:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13998859 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59E01212B2B; Mon, 3 Mar 2025 13:30:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.194 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008647; cv=none; b=T97vw6JuAocKkwWXmuWXIyxaTgMz1sLz140V7zIovaniFlU791JgSJuDkRs1tGkgSSx69OGVCjV5T6PpNANvmoqCv8lqRZXdQqWCh+eHQwJkjsctzlGDOXwigxVRLiArffyT+CV6RucO4FSRvSnoNz+br2cZ4vzOLt4KjuoXKuk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008647; c=relaxed/simple; bh=/mOskEeuSOoI6Ik4GZTrnIyvVE91D9OQfL2ePFfqnyY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=A5zAkd5twkwqi+gpWK25a63kJbrbq1XDUp/qJV0rzcdFA1kmXX+wZmeyovdJoInYDyJX06JFrTvraqUclai4tIaczBaTPAKce1kRIzLH8mmd4j9o47ANiq95bACV1guHluiLU0o1Df19E+9MHzNVzQYDL7ssmX6J+MlsFhuLOQ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dfMOQrl/; arc=none smtp.client-ip=209.85.214.194 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dfMOQrl/" Received: by mail-pl1-f194.google.com with SMTP id d9443c01a7336-2233622fdffso84011485ad.2; Mon, 03 Mar 2025 05:30:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741008645; x=1741613445; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g+TPvgOYa0/i6hWJUb51m5UVXTU1tbJahC/+FJNQh8Y=; b=dfMOQrl/JvemPuk2oWyYMQvAKpK6dxpNu5BRSFqcV7fdVglYdgD4GGZ4bxFc6O849v y8dWrzb+FbOj4RQS2VJnr4LGBESOQPkffmOnXwsGMtgL6c6VWVppPi/VXW/2wxn30vbC Y0zdQ63fRE1GeURAk627IP6C2euWM7qz5SM3A6pjRvNLpilnEztoVrUkkE8YWJfA4Pln whS5Qmn9OMv+aXJlfAIMXefKIXi7Qty4XYLjmI+RjCUjeInLWspTNGe/0Zj9Ik9hayLW /2vyrYiYoVnt6mGBJ9r44SGpidhwbO7jnuXCrMJCC1y0tQBJC6YmiZfaUlytQAjSY4nV Z9Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741008645; x=1741613445; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g+TPvgOYa0/i6hWJUb51m5UVXTU1tbJahC/+FJNQh8Y=; b=od82cK8P9gaHJjUl42iQ5rUPRXJ+Ms+o6d5FLrNIRD2DhJ1FLNmFIqLvMPYeXWZt8S vHcEg6FOkFA9xii9SzJrK+mxlLOEMkroPaKedJWvzsBFDryZIoOgtb7foaRzkDz5uUEk iEPvO5hKSKmllQxQoINXW6l5Xko1r2+kfp3AouxCD1A4DqM0dUP1/gzh7HHJ21KbMxqm lcGSpJ33qu9ASCrc6JBjx8RA2oLzIaCBXgxkjRK/vEdK6EKUOC8+wS01IhVQoMz7ee1y erZvtcqBE1TA5wppXhg0ywvBKAsjMYCLU5Iq5RfAH+6Z3DkCjr1f+IHhUu88kzvdrfFu WMYA== X-Forwarded-Encrypted: i=1; AJvYcCU0xds5uT6/9WAuUCDgtGCGFB7JlEyr4b+ONVV9D5jcQGClbyXcERDnhWNz6Gc7Ef6+GVua2wFc@vger.kernel.org, AJvYcCVgZhcwvFs2GUtKBnZW9xeZkqnzdiISWPKg1R4vpr6GMCNGbKDQQ+6AfAvolY13GEAuV8g3sAfeNobWWRYX@vger.kernel.org, AJvYcCXdh1UQgHqeE9WtcR2E4BYnelLfH0xRBpuli8RVRZ78msuveSCW/i74A7UZOcfeSP9KfLg=@vger.kernel.org, AJvYcCXkcPeIGo/Gd+uMm6VvmvcCp9lG3IhjvuZGX58u4DfyuAEkguUv7vHSnvniPCeYsPpBUrEEg3zeT5mSMXWI4Nh8Ajgq@vger.kernel.org X-Gm-Message-State: AOJu0Yy9ffyq/1r3zJxzoJv5RQNIj3v17M3RVd7nPxenXpRp3q+3CO9u hPPikIK3bMur2msbpaDzKDE4gxHefL4lmOqdJTwWmyiL7i7m/OqK X-Gm-Gg: ASbGncvK/0xw3urFQSjAp/o5K9+/dWh2/2vZe0NlI84T6dJBH1amI6uQt4H6DwlJI7I AUt3QiV7t4N9I63u/E+uLj6uyw1HBT0oq4wtfjJazm0+eBgBVQmexks+XdJYnolHQEr6erySTNk yDwxSVluGAmTFYsEgt2w4IrRao8prfDCt71OP/rtE4xUqODJcB7XrGOOKpMMQyZEIljgidLfvwV sbIcMYEha0wsIRTa9AyMbXCHf5OGOR7xpJN38VbiVLlYBZMc/PFNMJquglBqEHfC3HLvOrqnxV9 mM57q7YkwZTo/fWiE2gsTKrxxb6Sx9GOfYKL3ABYAa2e+LdGp+exxBWrkR3Nbw== X-Google-Smtp-Source: AGHT+IFlbT3UDNEQKtNvTWGWrAZWwF7WOjWCx8Sdq18QPhAIVe0wmrIF4oYZBBWwy52qUNo/JhCf0Q== X-Received: by 2002:a17:903:1790:b0:21f:71b4:d2aa with SMTP id d9443c01a7336-22368fa54b4mr247215185ad.5.1741008644544; Mon, 03 Mar 2025 05:30:44 -0800 (PST) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223505359b8sm77297035ad.253.2025.03.03.05.30.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 05:30:44 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: peterz@infradead.org, rostedt@goodmis.org, mark.rutland@arm.com, alexei.starovoitov@gmail.com Cc: catalin.marinas@arm.com, will@kernel.org, mhiramat@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, jolsa@kernel.org, davem@davemloft.net, dsahern@kernel.org, mathieu.desnoyers@efficios.com, nathan@kernel.org, nick.desaulniers+lkml@gmail.com, morbo@google.com, samitolvanen@google.com, kees@kernel.org, dongml2@chinatelecom.cn, akpm@linux-foundation.org, riel@surriel.com, rppt@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, netdev@vger.kernel.org, llvm@lists.linux.dev Subject: [PATCH v4 2/4] add per-function metadata storage support Date: Mon, 3 Mar 2025 21:28:35 +0800 Message-Id: <20250303132837.498938-3-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303132837.498938-1-dongml2@chinatelecom.cn> References: <20250303132837.498938-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For now, there isn't a way to set and get per-function metadata with a low overhead, which is not convenient for some situations. Take BPF trampoline for example, we need to create a trampoline for each kernel function, as we have to store some information of the function to the trampoline, such as BPF progs, function arg count, etc. The performance overhead and memory consumption can be higher to create these trampolines. With the supporting of per-function metadata storage, we can store these information to the metadata, and create a global BPF trampoline for all the kernel functions. In the global trampoline, we get the information that we need from the function metadata through the ip (function address) with almost no overhead. Another beneficiary can be ftrace. For now, all the kernel functions that are enabled by dynamic ftrace will be added to a filter hash if there are more than one callbacks. And hash lookup will happen when the traced functions are called, which has an impact on the performance, see __ftrace_ops_list_func() -> ftrace_ops_test(). With the per-function metadata supporting, we can store the information that if the callback is enabled on the kernel function to the metadata. Support per-function metadata storage in the function padding, and previous discussion can be found in [1]. Generally speaking, we have two way to implement this feature: 1. Create a function metadata array, and prepend a insn which can hold the index of the function metadata in the array. And store the insn to the function padding. 2. Allocate the function metadata with kmalloc(), and prepend a insn which hold the pointer of the metadata. And store the insn to the function padding. Compared with way 2, way 1 consume less space, but we need to do more work on the global function metadata array. And we implement this function in the way 1. Link: https://lore.kernel.org/bpf/CADxym3anLzM6cAkn_z71GDd_VeKiqqk1ts=xuiP7pr4PO6USPA@mail.gmail.com/ [1] Signed-off-by: Menglong Dong --- v2: - add supporting for arm64 - split out arch relevant code - refactor the commit log --- include/linux/kfunc_md.h | 25 ++++ kernel/Makefile | 1 + kernel/trace/Makefile | 1 + kernel/trace/kfunc_md.c | 239 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 266 insertions(+) create mode 100644 include/linux/kfunc_md.h create mode 100644 kernel/trace/kfunc_md.c diff --git a/include/linux/kfunc_md.h b/include/linux/kfunc_md.h new file mode 100644 index 000000000000..df616f0fcb36 --- /dev/null +++ b/include/linux/kfunc_md.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KFUNC_MD_H +#define _LINUX_KFUNC_MD_H + +#include + +struct kfunc_md { + int users; + /* we can use this field later, make sure it is 8-bytes aligned + * for now. + */ + int pad0; + void *func; +}; + +extern struct kfunc_md *kfunc_mds; + +struct kfunc_md *kfunc_md_find(void *ip); +struct kfunc_md *kfunc_md_get(void *ip); +void kfunc_md_put(struct kfunc_md *meta); +void kfunc_md_put_by_ip(void *ip); +void kfunc_md_lock(void); +void kfunc_md_unlock(void); + +#endif diff --git a/kernel/Makefile b/kernel/Makefile index 87866b037fbe..7435674d5da3 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -108,6 +108,7 @@ obj-$(CONFIG_TRACE_CLOCK) += trace/ obj-$(CONFIG_RING_BUFFER) += trace/ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_RETHOOK) += trace/ +obj-$(CONFIG_FUNCTION_METADATA) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile index 057cd975d014..9780ee3f8d8d 100644 --- a/kernel/trace/Makefile +++ b/kernel/trace/Makefile @@ -106,6 +106,7 @@ obj-$(CONFIG_FTRACE_RECORD_RECURSION) += trace_recursion_record.o obj-$(CONFIG_FPROBE) += fprobe.o obj-$(CONFIG_RETHOOK) += rethook.o obj-$(CONFIG_FPROBE_EVENTS) += trace_fprobe.o +obj-$(CONFIG_FUNCTION_METADATA) += kfunc_md.o obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o obj-$(CONFIG_RV) += rv/ diff --git a/kernel/trace/kfunc_md.c b/kernel/trace/kfunc_md.c new file mode 100644 index 000000000000..7ec25bcf778d --- /dev/null +++ b/kernel/trace/kfunc_md.c @@ -0,0 +1,239 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include + +#define ENTRIES_PER_PAGE (PAGE_SIZE / sizeof(struct kfunc_md)) + +static u32 kfunc_md_count = ENTRIES_PER_PAGE, kfunc_md_used; +struct kfunc_md __rcu *kfunc_mds; +EXPORT_SYMBOL_GPL(kfunc_mds); + +static DEFINE_MUTEX(kfunc_md_mutex); + + +void kfunc_md_unlock(void) +{ + mutex_unlock(&kfunc_md_mutex); +} +EXPORT_SYMBOL_GPL(kfunc_md_unlock); + +void kfunc_md_lock(void) +{ + mutex_lock(&kfunc_md_mutex); +} +EXPORT_SYMBOL_GPL(kfunc_md_lock); + +static u32 kfunc_md_get_index(void *ip) +{ + return *(u32 *)(ip - KFUNC_MD_DATA_OFFSET); +} + +static void kfunc_md_init(struct kfunc_md *mds, u32 start, u32 end) +{ + u32 i; + + for (i = start; i < end; i++) + mds[i].users = 0; +} + +static int kfunc_md_page_order(void) +{ + return fls(DIV_ROUND_UP(kfunc_md_count, ENTRIES_PER_PAGE)) - 1; +} + +/* Get next usable function metadata. On success, return the usable + * kfunc_md and store the index of it to *index. If no usable kfunc_md is + * found in kfunc_mds, a larger array will be allocated. + */ +static struct kfunc_md *kfunc_md_get_next(u32 *index) +{ + struct kfunc_md *new_mds, *mds; + u32 i, order; + + mds = rcu_dereference(kfunc_mds); + if (mds == NULL) { + order = kfunc_md_page_order(); + new_mds = (void *)__get_free_pages(GFP_KERNEL, order); + if (!new_mds) + return NULL; + kfunc_md_init(new_mds, 0, kfunc_md_count); + /* The first time to initialize kfunc_mds, so it is not + * used anywhere yet, and we can update it directly. + */ + rcu_assign_pointer(kfunc_mds, new_mds); + mds = new_mds; + } + + if (likely(kfunc_md_used < kfunc_md_count)) { + /* maybe we can manage the used function metadata entry + * with a bit map ? + */ + for (i = 0; i < kfunc_md_count; i++) { + if (!mds[i].users) { + kfunc_md_used++; + *index = i; + mds[i].users++; + return mds + i; + } + } + } + + order = kfunc_md_page_order(); + /* no available function metadata, so allocate a bigger function + * metadata array. + */ + new_mds = (void *)__get_free_pages(GFP_KERNEL, order + 1); + if (!new_mds) + return NULL; + + memcpy(new_mds, mds, kfunc_md_count * sizeof(*new_mds)); + kfunc_md_init(new_mds, kfunc_md_count, kfunc_md_count * 2); + + rcu_assign_pointer(kfunc_mds, new_mds); + synchronize_rcu(); + free_pages((u64)mds, order); + + mds = new_mds + kfunc_md_count; + *index = kfunc_md_count; + kfunc_md_count <<= 1; + kfunc_md_used++; + mds->users++; + + return mds; +} + +static int kfunc_md_text_poke(void *ip, void *insn, void *nop) +{ + void *target; + int ret = 0; + u8 *prog; + + target = ip - KFUNC_MD_INSN_OFFSET; + mutex_lock(&text_mutex); + if (insn) { + if (!memcmp(target, insn, KFUNC_MD_INSN_SIZE)) + goto out; + + if (memcmp(target, nop, KFUNC_MD_INSN_SIZE)) { + ret = -EBUSY; + goto out; + } + prog = insn; + } else { + if (!memcmp(target, nop, KFUNC_MD_INSN_SIZE)) + goto out; + prog = nop; + } + + ret = kfunc_md_arch_poke(target, prog); +out: + mutex_unlock(&text_mutex); + return ret; +} + +static bool __kfunc_md_put(struct kfunc_md *md) +{ + u8 nop_insn[KFUNC_MD_INSN_SIZE]; + + if (WARN_ON_ONCE(md->users <= 0)) + return false; + + md->users--; + if (md->users > 0) + return false; + + if (!kfunc_md_arch_exist(md->func)) + return false; + + kfunc_md_arch_nops(nop_insn); + /* release the metadata by recovering the function padding to NOPS */ + kfunc_md_text_poke(md->func, NULL, nop_insn); + /* TODO: we need a way to shrink the array "kfunc_mds" */ + kfunc_md_used--; + + return true; +} + +/* Decrease the reference of the md, release it if "md->users <= 0" */ +void kfunc_md_put(struct kfunc_md *md) +{ + mutex_lock(&kfunc_md_mutex); + __kfunc_md_put(md); + mutex_unlock(&kfunc_md_mutex); +} +EXPORT_SYMBOL_GPL(kfunc_md_put); + +/* Get a exist metadata by the function address, and NULL will be returned + * if not exist. + * + * NOTE: rcu lock should be held during reading the metadata, and + * kfunc_md_lock should be held if writing happens. + */ +struct kfunc_md *kfunc_md_find(void *ip) +{ + struct kfunc_md *md; + u32 index; + + if (kfunc_md_arch_exist(ip)) { + index = kfunc_md_get_index(ip); + if (WARN_ON_ONCE(index >= kfunc_md_count)) + return NULL; + + md = rcu_dereference(kfunc_mds) + index; + return md; + } + return NULL; +} +EXPORT_SYMBOL_GPL(kfunc_md_find); + +void kfunc_md_put_by_ip(void *ip) +{ + struct kfunc_md *md; + + mutex_lock(&kfunc_md_mutex); + md = kfunc_md_find(ip); + if (md) + __kfunc_md_put(md); + mutex_unlock(&kfunc_md_mutex); +} +EXPORT_SYMBOL_GPL(kfunc_md_put_by_ip); + +/* Get a exist metadata by the function address, and create one if not + * exist. Reference of the metadata will increase 1. + * + * NOTE: always call this function with kfunc_md_lock held, and all + * updating to metadata should also hold the kfunc_md_lock. + */ +struct kfunc_md *kfunc_md_get(void *ip) +{ + u8 nop_insn[KFUNC_MD_INSN_SIZE], insn[KFUNC_MD_INSN_SIZE]; + struct kfunc_md *md; + u32 index; + + md = kfunc_md_find(ip); + if (md) { + md->users++; + return md; + } + + md = kfunc_md_get_next(&index); + if (!md) + return NULL; + + kfunc_md_arch_pretend(insn, index); + kfunc_md_arch_nops(nop_insn); + + if (kfunc_md_text_poke(ip, insn, nop_insn)) { + kfunc_md_used--; + md->users = 0; + return NULL; + } + md->func = ip; + + return md; +} +EXPORT_SYMBOL_GPL(kfunc_md_get); From patchwork Mon Mar 3 13:28:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13998860 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78DE8212D8A; Mon, 3 Mar 2025 13:30:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008655; cv=none; b=Huj8dsdJsZXACvKPnNF1gAgt15rlOXC6hB0rpumcJwf03ugfq9T5wFfhe6WkiUbyjEKHuT54HAaZ8GG8/tNJGWVVvXa9H3LNEkqCYXxX+PwOxUJLaGN+sy7uKbizvsWrMwdvpNfG/XhuGfVYNgsOvB7i6wzY9M18XXH9M83kUu4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008655; c=relaxed/simple; bh=0opDRmJgFcne9MVvIB9mWH4N0ry/ocl2wUXCnUeZZWU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=unuuv/WuzFJterWZ2+DY/X6rChPXNqFX/No/3wdqdXKMPGQmTEzXLonW0lJRsss0cxQFuS7S8oHIaXsUMeUX9a5GvWaES1mPrAQIROOId0WKsBKcAVscNuz9O3cB9L4eDRHP3WUPk1TxWg3zUroOCxabcpj3G/07lAQ7Sjl8vwc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yo3VuO1y; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yo3VuO1y" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-2232b12cd36so58594305ad.0; Mon, 03 Mar 2025 05:30:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741008653; x=1741613453; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e0whn35vPm8J+vjBTbfA5u1AHwSTrw7swcARc6I++Hc=; b=Yo3VuO1yECF1+kFHLH0euRtLVZw/zHR0aag+qhLUnLNwo/6dTKDcoBq8r8qUoXxqa8 J9iCfesx+8hqAUOMfjaO2WyG7BX56E7wcaEOFI4b+T4ZcX9Ku2AAECuPF9u79fI+Og95 dy0YKP8msNzZ8ZO8xVGYZ48aFCjJrmEkiQpbfii4sErkFXiNBwdG8bNGMDLx+TnviruW Kku4DV2CPXCKF2ccZznsQSFhqAgppiMiqe96pAgV1vKSnIgc05G98cLBMJPcCfNSAY5+ APN5IppKTHfs6QWNv3mOAOsRf8URxHmn8aVPfW3ZsUcYMeCM8maYfcadS4Wksy/LgWRM MF3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741008653; x=1741613453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e0whn35vPm8J+vjBTbfA5u1AHwSTrw7swcARc6I++Hc=; b=cN3Pqa66nf8KP6JEEOetEfup+PSwbH+BYCH+svOO+eR6iqlU8Y6r2pAN2CB/vfNS8k fCoeuEKfxJ/mr1JDz3eFEE4QlUa3BirpvIP9jiM6JHJTeXWdw2QDjbRy5X6FKtNCGC81 QrzUyKha1km3dLlXBYwA+y53ZZCx2cp9Ye7NE7NUzMoMTQhCe+TA+1Q4kRLyJPPB6T/q ZaIaqg4a5YsyN1vv5t7a5BQKcQuuM3uhiCxI3TKcH9hKTz/n4mdiZgays7fhX7hws5m2 FFn1wE0LRB8sNAgZNqoVjCk++OA/ZvwNshl1uSyG3pILFVEnUMAgYciXhdPpQJ4AbHsX XHcA== X-Forwarded-Encrypted: i=1; AJvYcCU03EUyFuF1B+ZnixmX/WdPawaMl34Cjvo3VHKhyx2cNKbKSOEWugI8uv98t1QBYRQXFmcXPSLZ3Vsphk1d@vger.kernel.org, AJvYcCUNbdOEd77zxjPXxRw2bEAM1GUyJ2EQOEpEu9Bs85OBDRrJv0aeKYtvkiCvtgMDBQ58g1jgv/cZ81knnF2JT7n03VY8@vger.kernel.org, AJvYcCUdMVQtvYCuO4KhTKzGQuE0sqzGPbtGUurIVNp3xGXH/qU9n5aDCFUwtLZd71pun0G6Vto=@vger.kernel.org, AJvYcCVUOPK1YNrnzcjtLMxrOX2i5yAbWNf/tv6DItHENLEX6LBh6Ccb0Ykt/E5RLkolFMD4OkvDqeSE@vger.kernel.org X-Gm-Message-State: AOJu0Yy4WKIBGMWziMcEmuWKe084oXNkzrvUDIMDkvGgI/jBjMJ5kBK5 jMh+73nuOBmzqUJCyOYmY6P37mgmeKESD6FLMs4JF+rxVx1QWZrx X-Gm-Gg: ASbGncu1G/WMyjicL0VxJOVwDQo1z95VWLVGEoMOybuv5uhxW1Rs3s+/ptj9mKl/az+ 1DBk6noW1zZqLgK4HnOp7hIwgn+XQu6HOPi299H9cyZzut7t6qdD9PGY5qgTSFwuo21Qop2jgO+ koJafqg0UzbfZTqw7MN24X0MImuTLPCmshhevbFr63l8WyMqnCDBTaMwtNilEOmjAscpRhZBSw5 UfFGKHAn1C5AVapXIfuy0TGpD+2Czfid7rkq+rU9roMnYDoFJSMWzqKgyVHIaIsw7dh/BRDR230 NAXphmvJwqhDec6NvFT7Oh0ZHQxKk8+yjgE2vvSd4f45xz1j8p6/4wQpRtXyWQ== X-Google-Smtp-Source: AGHT+IECVEaRvzz7jFgM9Xs6Qsz7AN1fGJ1Lrd2ZRMORcgHaHlhwWOyshFl83O3butQP+uNtOa6yXg== X-Received: by 2002:a17:902:e5c5:b0:223:4bd6:3863 with SMTP id d9443c01a7336-22368f6a174mr217254605ad.10.1741008652539; Mon, 03 Mar 2025 05:30:52 -0800 (PST) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223505359b8sm77297035ad.253.2025.03.03.05.30.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 05:30:52 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: peterz@infradead.org, rostedt@goodmis.org, mark.rutland@arm.com, alexei.starovoitov@gmail.com Cc: catalin.marinas@arm.com, will@kernel.org, mhiramat@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, jolsa@kernel.org, davem@davemloft.net, dsahern@kernel.org, mathieu.desnoyers@efficios.com, nathan@kernel.org, nick.desaulniers+lkml@gmail.com, morbo@google.com, samitolvanen@google.com, kees@kernel.org, dongml2@chinatelecom.cn, akpm@linux-foundation.org, riel@surriel.com, rppt@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, netdev@vger.kernel.org, llvm@lists.linux.dev Subject: [PATCH v4 3/4] x86: implement per-function metadata storage for x86 Date: Mon, 3 Mar 2025 21:28:36 +0800 Message-Id: <20250303132837.498938-4-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303132837.498938-1-dongml2@chinatelecom.cn> References: <20250303132837.498938-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 With CONFIG_CALL_PADDING enabled, there will be 16-bytes padding space before all the kernel functions. And some kernel features can use it, such as MITIGATION_CALL_DEPTH_TRACKING, CFI_CLANG, FINEIBT, etc. In my research, MITIGATION_CALL_DEPTH_TRACKING will consume the tail 9-bytes in the function padding, CFI_CLANG will consume the head 5-bytes, and FINEIBT will consume all the 16 bytes if it is enabled. So there will be no space for us if MITIGATION_CALL_DEPTH_TRACKING and CFI_CLANG are both enabled, or FINEIBT is enabled. In x86, we need 5-bytes to prepend a "mov %eax xxx" insn, which can hold a 4-bytes index. So we have following logic: 1. use the head 5-bytes if CFI_CLANG is not enabled 2. use the tail 5-bytes if MITIGATION_CALL_DEPTH_TRACKING and FINEIBT are not enabled 3. compile the kernel with FUNCTION_ALIGNMENT_32B otherwise In the third case, we make the kernel function 32 bytes aligned, and there will be 32 bytes padding before the functions. According to my testing, the text size didn't increase on this case, which is weird. With 16-bytes padding: -rwxr-xr-x 1 401190688 x86-dev/vmlinux* -rw-r--r-- 1 251068 x86-dev/vmlinux.a -rw-r--r-- 1 851892992 x86-dev/vmlinux.o -rw-r--r-- 1 12395008 x86-dev/arch/x86/boot/bzImage With 32-bytes padding: -rwxr-xr-x 1 401318128 x86-dev/vmlinux* -rw-r--r-- 1 251154 x86-dev/vmlinux.a -rw-r--r-- 1 853636704 x86-dev/vmlinux.o -rw-r--r-- 1 12509696 x86-dev/arch/x86/boot/bzImage The way I tested should be right, and this is a good news for us. On the third case, the layout of the padding space will be like this if fineibt is enabled: __cfi_func: mov -- 5 -- cfi, not used anymore nop nop nop mov -- 5 -- function metadata nop nop nop fineibt -- 16 -- fineibt func: nopw -- 4 ...... I tested the fineibt with "cfi=fineibt" cmdline, and it works well together with FUNCTION_METADATA enabled. And I also tested the performance of this function by setting metadata for all the kernel function, and it consumes 0.7s for 70k+ functions, not bad :/ I can't find a machine that support IBT, so I didn't test the IBT. I'd appreciate it if someone can do this testing for me :/ Signed-off-by: Menglong Dong --- v3: - select FUNCTION_ALIGNMENT_32B on case3, instead of extra 5-bytes --- arch/x86/Kconfig | 18 ++++++++++++ arch/x86/include/asm/ftrace.h | 54 +++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5c277261507e..b0614188c80b 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2518,6 +2518,24 @@ config PREFIX_SYMBOLS def_bool y depends on CALL_PADDING && !CFI_CLANG +config FUNCTION_METADATA + bool "Per-function metadata storage support" + default y + depends on CC_HAS_ENTRY_PADDING && OBJTOOL + select CALL_PADDING + select FUNCTION_ALIGNMENT_32B if ((CFI_CLANG && CALL_THUNKS) || FINEIBT) + help + Support per-function metadata storage for kernel functions, and + get the metadata of the function by its address with almost no + overhead. + + The index of the metadata will be stored in the function padding + and consumes 5-bytes. FUNCTION_ALIGNMENT_32B will be selected if + "(CFI_CLANG && CALL_THUNKS) || FINEIBT" to make sure there is + enough available padding space for this function. However, it + seems that the text size almost don't change, compare with + FUNCTION_ALIGNMENT_16B. + menuconfig CPU_MITIGATIONS bool "Mitigations for CPU vulnerabilities" default y diff --git a/arch/x86/include/asm/ftrace.h b/arch/x86/include/asm/ftrace.h index f2265246249a..700bb729e949 100644 --- a/arch/x86/include/asm/ftrace.h +++ b/arch/x86/include/asm/ftrace.h @@ -4,6 +4,28 @@ #include +#ifdef CONFIG_FUNCTION_METADATA +#if (defined(CONFIG_CFI_CLANG) && defined(CONFIG_CALL_THUNKS)) || (defined(CONFIG_FINEIBT)) + /* the CONFIG_FUNCTION_PADDING_BYTES is 32 in this case, use the + * range: [align + 8, align + 13]. + */ + #define KFUNC_MD_INSN_OFFSET (CONFIG_FUNCTION_PADDING_BYTES - 8) + #define KFUNC_MD_DATA_OFFSET (CONFIG_FUNCTION_PADDING_BYTES - 9) +#else + #ifdef CONFIG_CFI_CLANG + /* use the space that CALL_THUNKS suppose to use */ + #define KFUNC_MD_INSN_OFFSET (5) + #define KFUNC_MD_DATA_OFFSET (4) + #else + /* use the space that CFI_CLANG suppose to use */ + #define KFUNC_MD_INSN_OFFSET (CONFIG_FUNCTION_PADDING_BYTES) + #define KFUNC_MD_DATA_OFFSET (CONFIG_FUNCTION_PADDING_BYTES - 1) + #endif +#endif + +#define KFUNC_MD_INSN_SIZE (5) +#endif + #ifdef CONFIG_FUNCTION_TRACER #ifndef CC_USING_FENTRY # error Compiler does not support fentry? @@ -156,4 +178,36 @@ static inline bool arch_trace_is_compat_syscall(struct pt_regs *regs) #endif /* !COMPILE_OFFSETS */ #endif /* !__ASSEMBLY__ */ +#if !defined(__ASSEMBLY__) && defined(CONFIG_FUNCTION_METADATA) +#include + +static inline bool kfunc_md_arch_exist(void *ip) +{ + return *(u8 *)(ip - KFUNC_MD_INSN_OFFSET) == 0xB8; +} + +static inline void kfunc_md_arch_pretend(u8 *insn, u32 index) +{ + *insn = 0xB8; + *(u32 *)(insn + 1) = index; +} + +static inline void kfunc_md_arch_nops(u8 *insn) +{ + *(insn++) = BYTES_NOP1; + *(insn++) = BYTES_NOP1; + *(insn++) = BYTES_NOP1; + *(insn++) = BYTES_NOP1; + *(insn++) = BYTES_NOP1; +} + +static inline int kfunc_md_arch_poke(void *ip, u8 *insn) +{ + text_poke(ip, insn, KFUNC_MD_INSN_SIZE); + text_poke_sync(); + return 0; +} + +#endif + #endif /* _ASM_X86_FTRACE_H */ From patchwork Mon Mar 3 13:28:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Menglong Dong X-Patchwork-Id: 13998861 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47BB0214801; Mon, 3 Mar 2025 13:31:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.195 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008662; cv=none; b=aAyAwZ3lzEPd2DG6Ceky8vIOw+wLGDye9qGq+pLSvLp1QXgjLqEe0FGP4USrPNLnFnPFlW+Pg4H+M1ISFk+CFR4lmuHGkH2Mf9HVinkHnr7RIpPRMfcyclqOnPS4xMtnSgeAH+XlAGz00cudKm4arPqSQUgmOMtJmO0TVEpNSFQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741008662; c=relaxed/simple; bh=8+jQ2LOFLvD8yNjmgsq7xpDv7f8F3syKUpRlLU8kRU0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CFobIGKvsmG2kIsDo0m1WCyULLXTXsxPxL/A4pJoIJxlNdiQc+J8PKQ01krb73bJQth05n2u29QEyR33I/YGz4FXc/WyYwsohgXVOPzuH6B9PJe/pQbF/g5LiNzckjxhtfn+HR+es0ulCC63wxdWqIr06oFplTef55zq/UfV/pU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EhNBr+/Q; arc=none smtp.client-ip=209.85.214.195 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EhNBr+/Q" Received: by mail-pl1-f195.google.com with SMTP id d9443c01a7336-22328dca22fso65779915ad.1; Mon, 03 Mar 2025 05:31:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741008660; x=1741613460; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KvSQXTXm2WJtvrSSulKAAYVeipdazZIwWA04KeDmP3s=; b=EhNBr+/QzSGBpX2IY6/QL72F/ryYFo3emfmsyWpE8E74EfQLripm/UkS61WwAgQDOJ UTSXWnORJ5ODi8AxNJ/8PqjipoFwxVXRFOc1RVkHifYzyTlCVg6WFwJidptYBCBMEpEg klrZxOIOO6EtVzbtjP0+2D0uwUlgkJtE1L0GdF8JsvaUtU4z8+4KU4KxM8Xmj2swKKuA xbspuk0zO1MNlLAXJjkm0IHV2QVIqBXNSYkS8LlCfK19euWcYE9zzyM20ZyYwjz12xIJ O3INi/L20xmiIx+52wVWhGdT2aAN3/AuT89PFtpISQnor56BcXWnqkJuyWLl14mROx+8 zsEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741008660; x=1741613460; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KvSQXTXm2WJtvrSSulKAAYVeipdazZIwWA04KeDmP3s=; b=to+9kOilBa9wU8igexWm+qGrkwPL6yAeH0UT5XcCc1e02kvYV8T29XBq350C0vSujJ Tr9uY+LjQbRU3oNrNjBcd+5RFwele/ARymRkZKQEy9TLf7z1tYyb4Bzcg4+V0utmsyti 7WRG/YvtWeFqDZayr4sucrEILkOvRAvwdeGtauwELcdOc6187hB4Q8jObSwRAPvwe3eA JosRnHN/n9IofmJ9UoZJo6w4EPhMrAOy8/qf5k8SRuCzxoa0TukIppdbP0p6oLuIEtXH FdMQXRtdSv3KbgvM/rw29NgooD5YqhIyqYncKOFWBfEA5KMEbF8jzzSJR4jFJ904C8xb r55w== X-Forwarded-Encrypted: i=1; AJvYcCU4dQqLEyvOSpXgw8iDU0Jso81lFC2+wtD+pxR1345bnGtgy1lTFRCw5X+eF4ncIh+TBxI=@vger.kernel.org, AJvYcCUEuO4i4XfXKcm3StLOzj6CTy2nMW3NNvd3vRcX5e/LatmBk0e7jSofdxdrg16JA2eiNEGmyPpK@vger.kernel.org, AJvYcCW2wG32040KXJ5c5Wj1r9br+0qDb1pkhcRdZ7XGDzyxQLU+OM06g29ENNssTvYmeq4RI40pQjDeuBo76JTGcML3CBT+@vger.kernel.org, AJvYcCXm4s3rLxlvBL1SU3X7MsS6iI+WESyqIDaWHakO55KjAB3TYHLDoOrpl/fS45TTSjkl3QAZRPBZihlF+nCk@vger.kernel.org X-Gm-Message-State: AOJu0YwNOXlGkbfg1kC62IS7wI9ow2G8jQz1OD5iDJiFFI1sgIQeydu2 A8i3mVUN6UMktuqNgCOSIRARICoMPVwv8pt6hjxDspQJqPVzfc7M X-Gm-Gg: ASbGncuUj4ZUis7Fta8z4/LYyMJlMgXPGRyib1QYIMaWnIy2k7sf8rBgcSihWyjWFFs QXlRxv7EZ1XkY0DaDiE2i9g6joDFZdlG1rq1kehwl4vwGHp6myJAeW9XsFWR9pjgky8z1NZdH03 4fgui6UUKytP0NX5tAOk6+TTJ8u5vL1tnDaPtb70gc68onAAs7sZX95fKXOspJD0g69S80LfQVk Du4zFwvWeQc/3tpnuS0SXVQnpnJsm8IPbl7XrwwMeO1X0WjBuaY9ESxdA6Ub0Iyf68gYhSIk3ZR 3Gd5Wu7JHB0nb5A7EXFjdUmllNv/DPk3JeTl0s/wuYUemVg2JS8PmiLL1Ch2Ow== X-Google-Smtp-Source: AGHT+IFrx9yunejrhIIBax0i6zoJK5cErggOeRBXOxeJLSAEu6Vgq/DyW9livSLEsya9w6YWkG4/Ug== X-Received: by 2002:a17:902:fc8d:b0:223:44dc:3f36 with SMTP id d9443c01a7336-2236925eef4mr220020495ad.43.1741008660442; Mon, 03 Mar 2025 05:31:00 -0800 (PST) Received: from localhost.localdomain ([43.129.244.20]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-223505359b8sm77297035ad.253.2025.03.03.05.30.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 05:31:00 -0800 (PST) From: Menglong Dong X-Google-Original-From: Menglong Dong To: peterz@infradead.org, rostedt@goodmis.org, mark.rutland@arm.com, alexei.starovoitov@gmail.com Cc: catalin.marinas@arm.com, will@kernel.org, mhiramat@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, jolsa@kernel.org, davem@davemloft.net, dsahern@kernel.org, mathieu.desnoyers@efficios.com, nathan@kernel.org, nick.desaulniers+lkml@gmail.com, morbo@google.com, samitolvanen@google.com, kees@kernel.org, dongml2@chinatelecom.cn, akpm@linux-foundation.org, riel@surriel.com, rppt@kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, netdev@vger.kernel.org, llvm@lists.linux.dev Subject: [PATCH v4 4/4] arm64: implement per-function metadata storage for arm64 Date: Mon, 3 Mar 2025 21:28:37 +0800 Message-Id: <20250303132837.498938-5-dongml2@chinatelecom.cn> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250303132837.498938-1-dongml2@chinatelecom.cn> References: <20250303132837.498938-1-dongml2@chinatelecom.cn> Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The per-function metadata storage is already used by ftrace if CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS is enabled, and it store the pointer of the callback directly to the function padding, which consume 8-bytes, in the commit baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS"). So we can directly store the index to the function padding too, without a prepending. With CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS enabled, the function is 8-bytes aligned, and we will compile the kernel with extra 8-bytes (2 NOPS) padding space. Otherwise, the function is 4-bytes aligned, and only extra 4-bytes (1 NOPS) is needed. However, we have the same problem with Mark in the commit above: we can't use the function padding together with CFI_CLANG, which can make the clang compiles a wrong offset to the pre-function type hash. He said that he was working with others on this problem 2 years ago. Hi Mark, is there any progress on this problem? Signed-off-by: Menglong Dong --- arch/arm64/Kconfig | 15 +++++++++++++++ arch/arm64/Makefile | 23 ++++++++++++++++++++-- arch/arm64/include/asm/ftrace.h | 34 +++++++++++++++++++++++++++++++++ arch/arm64/kernel/ftrace.c | 13 +++++++++++-- 4 files changed, 81 insertions(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 940343beb3d4..7ed80f5eb267 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1536,6 +1536,21 @@ config NODES_SHIFT Specify the maximum number of NUMA Nodes available on the target system. Increases memory reserved to accommodate various tables. +config FUNCTION_METADATA + bool "Per-function metadata storage support" + default y + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE if !FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY + depends on !CFI_CLANG + help + Support per-function metadata storage for kernel functions, and + get the metadata of the function by its address with almost no + overhead. + + The index of the metadata will be stored in the function padding, + which will consume 4-bytes. If FUNCTION_ALIGNMENT_8B is enabled, + extra 8-bytes function padding will be reserved during compiling. + Otherwise, only extra 4-bytes function padding is needed. + source "kernel/Kconfig.hz" config ARCH_SPARSEMEM_ENABLE diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 2b25d671365f..2df2b0f4dd90 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -144,12 +144,31 @@ endif CHECKFLAGS += -D__aarch64__ +ifeq ($(CONFIG_FUNCTION_METADATA),y) + ifeq ($(CONFIG_FUNCTION_ALIGNMENT_8B),y) + __padding_nops := 2 + else + __padding_nops := 1 + endif +else + __padding_nops := 0 +endif + ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS),y) + __padding_nops := $(shell echo $(__padding_nops) + 2 | bc) KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY - CC_FLAGS_FTRACE := -fpatchable-function-entry=4,2 + CC_FLAGS_FTRACE := -fpatchable-function-entry=$(shell echo $(__padding_nops) + 2 | bc),$(__padding_nops) else ifeq ($(CONFIG_DYNAMIC_FTRACE_WITH_ARGS),y) + CC_FLAGS_FTRACE := -fpatchable-function-entry=$(shell echo $(__padding_nops) + 2 | bc),$(__padding_nops) KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY - CC_FLAGS_FTRACE := -fpatchable-function-entry=2 +else ifeq ($(CONFIG_FUNCTION_METADATA),y) + CC_FLAGS_FTRACE += -fpatchable-function-entry=$(__padding_nops),$(__padding_nops) + ifneq ($(CONFIG_FUNCTION_TRACER),y) + KBUILD_CFLAGS += $(CC_FLAGS_FTRACE) + # some file need to remove this cflag when CONFIG_FUNCTION_TRACER + # is not enabled, so we need to export it here + export CC_FLAGS_FTRACE + endif endif ifeq ($(CONFIG_KASAN_SW_TAGS), y) diff --git a/arch/arm64/include/asm/ftrace.h b/arch/arm64/include/asm/ftrace.h index bfe3ce9df197..aa3eaa91bf82 100644 --- a/arch/arm64/include/asm/ftrace.h +++ b/arch/arm64/include/asm/ftrace.h @@ -24,6 +24,16 @@ #define FTRACE_PLT_IDX 0 #define NR_FTRACE_PLTS 1 +#ifdef CONFIG_FUNCTION_METADATA +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS +#define KFUNC_MD_DATA_OFFSET (AARCH64_INSN_SIZE * 3) +#else +#define KFUNC_MD_DATA_OFFSET AARCH64_INSN_SIZE +#endif +#define KFUNC_MD_INSN_SIZE AARCH64_INSN_SIZE +#define KFUNC_MD_INSN_OFFSET KFUNC_MD_DATA_OFFSET +#endif + /* * Currently, gcc tends to save the link register after the local variables * on the stack. This causes the max stack tracer to report the function @@ -216,6 +226,30 @@ static inline bool arch_syscall_match_sym_name(const char *sym, */ return !strcmp(sym + 8, name); } + +#ifdef CONFIG_FUNCTION_METADATA +#include + +static inline bool kfunc_md_arch_exist(void *ip) +{ + return !aarch64_insn_is_nop(*(u32 *)(ip - KFUNC_MD_INSN_OFFSET)); +} + +static inline void kfunc_md_arch_pretend(u8 *insn, u32 index) +{ + *(u32 *)insn = index; +} + +static inline void kfunc_md_arch_nops(u8 *insn) +{ + *(u32 *)insn = aarch64_insn_gen_nop(); +} + +static inline int kfunc_md_arch_poke(void *ip, u8 *insn) +{ + return aarch64_insn_patch_text_nosync(ip, *(u32 *)insn); +} +#endif #endif /* ifndef __ASSEMBLY__ */ #ifndef __ASSEMBLY__ diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index d7c0d023dfe5..4191ff0037f5 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -88,8 +88,10 @@ unsigned long ftrace_call_adjust(unsigned long addr) * to `BL `, which is at `addr + 4` bytes in either case. * */ - if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) - return addr + AARCH64_INSN_SIZE; + if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS)) { + addr += AARCH64_INSN_SIZE; + goto out; + } /* * When using patchable-function-entry with pre-function NOPs, addr is @@ -139,6 +141,13 @@ unsigned long ftrace_call_adjust(unsigned long addr) /* Skip the first NOP after function entry */ addr += AARCH64_INSN_SIZE; +out: + if (IS_ENABLED(CONFIG_FUNCTION_METADATA)) { + if (IS_ENABLED(CONFIG_FUNCTION_ALIGNMENT_8B)) + addr += 2 * AARCH64_INSN_SIZE; + else + addr += AARCH64_INSN_SIZE; + } return addr; }