From patchwork Wed Jun 19 22:48:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13704693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFB5BC27C53 for ; Wed, 19 Jun 2024 22:49:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E7F66B00BB; Wed, 19 Jun 2024 18:49:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D8A58D0091; Wed, 19 Jun 2024 18:49:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6C408D008F; Wed, 19 Jun 2024 18:49:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AC5C56B00BC for ; Wed, 19 Jun 2024 18:49:47 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 326281615A0 for ; Wed, 19 Jun 2024 22:49:47 +0000 (UTC) X-FDA: 82249132014.07.8CFD6D6 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf16.hostedemail.com (Postfix) with ESMTP id E98CF18000F for ; Wed, 19 Jun 2024 22:49:44 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718837378; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rh8eO3kA3ELL8Nh5M7QOXpKcI9PqUg1Ec5K8z4cuJgc=; b=7Z18B4WQi5/vVLoDKTuSW+b+kibvvgC8jcnwAPzzxu4bRiyQ4r+WE5dQeAQMBEqvUUBFEJ 2AESl210at+qo8NOBhbjyozJTCFLGlCKL2iefug3YRctLyejNlvDGnhTpAGZf7eHCMY0kn GGTNewutOVJRMpuMOYjPQxbs+xw1wIM= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718837379; a=rsa-sha256; cv=none; b=i5C0pxuB4I17ZiTWyfg0JdD9EUdKNyPLIijqoBROTfIWhx2y2/pSwXi4cNRX1UByj2tfAk lx2F6INCGekkZiiljrGdntHWV7SP5y+uzo2SjOyszBoBAGm/XO7UH5oj0IuajEV42y+Xo2 ZUXlGGhoIWZkrW48yxwzPqIxFu6zWwc= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 78F4B21A5F; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 593BA13ABD; Wed, 19 Jun 2024 22:49:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id QO+DFYdgc2aFIAAAD6G6ig (envelope-from ); Wed, 19 Jun 2024 22:49:43 +0000 From: Vlastimil Babka Date: Thu, 20 Jun 2024 00:48:58 +0200 Subject: [PATCH v2 4/7] bpf: support error injection static keys for multi_link attached progs MIME-Version: 1.0 Message-Id: <20240620-fault-injection-statickeys-v2-4-e23947d3d84b@suse.cz> References: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> In-Reply-To: <20240620-fault-injection-statickeys-v2-0-e23947d3d84b@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=4495; i=vbabka@suse.cz; h=from:subject:message-id; bh=3qciDcxlabL8RPKApoikpRFG6GtFoTLNySS+QbqQz9Q=; b=owEBbQGS/pANAwAIAbvgsHXSRYiaAcsmYgBmc2B7A+FYtqvio9GTWv3xme47MHxydtFu6JcXT U9aKA8HdJyJATMEAAEIAB0WIQR7u8hBFZkjSJZITfG74LB10kWImgUCZnNgewAKCRC74LB10kWI mtXfB/9Wf8ufmNZ+KgSqZhasEn0ZqY1pTjQK/1k4ql5arsN4WjQN5LONf7zFbsRueo1MccTXI8j /fH+X3PTcoSG6fRMjz3+vuHCOkrsWH3yUtKIWVCjSGbCnoTDSPalLr/doZP8HAWLrmhpFni/wZC 6VZHYbuRigc7AohRXcOu7R4Px5zhFhdbb+6z1EZkeKJzmzriFXhSIczEoPYa6g6RmKDL/HsZiEY Iwcv1O2H9gYU+430HbPN6am111EVHOeM6rY2MCwZaJjWJjd3at7bfhl/tIQWnAM0gal51sJCbR6 2uksKs0VJHgNrhMdsLGxDLtpwTXG5HrhbVle/JopZQNha/Sy X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Stat-Signature: zni4yen6gwroj93wcc8j5j3yz3wxymey X-Rspam-User: X-Rspamd-Queue-Id: E98CF18000F X-Rspamd-Server: rspam02 X-HE-Tag: 1718837384-800182 X-HE-Meta: U2FsdGVkX19oyBd+wiHZMwYTu2PDmXI+RdwdJuzkDkQLXmdinc3zDI5Gi4hxsIfuplWzusP5tKFqwJRpJmUkV/V6EJHPzJzdCbe314GTj0ilQmN+4QOSfH08SRduaP0fenGsZ+/sqiEp+Sejo7m0h6ajdEPYxuYIzOpx+jTHTNc/so6GvO79JC6naBHjR+7NnJwLmGwQa4okgn+xt6XOZ/ObjAArjckbv5WtcDouyKYgerXJBdilV5YdCZ+iQRZduWYGqrIR7sFKb8RauA+ZEuYSnf3ep5rIryeRKhc3BzLTSzxX60kdT0MJaicdTmaeEXQNXTJV6FV6oP5zsgYSlBn6TwrWi+qGea96oM5oTS1iwStCMMXozL7fQd2EmuEx+iinCanJGS3p4YPnmk5dp4ZT5oLZczRUKj0JgPkJ+C0giVL66Sdr+ufg6hmPa8THFrfVVKZj2Nqco+hJ+u8uvHMW8RZ62xOwJQ+GQUKwXWcSbykt05RkZLqO2CR5vk0N4JbTGRQuR+/le4447P2jPWe0gXw2VpciTd56kJfSonEEpaG0NXuGoZ2CrKdcTgNjdcz4xB5G8HOtCPRnqHedEV0NfzmVVtVRS27SvkGngFRZlQuEUhexCIof30xNK0wG65x6eU2EBICa1Sv1Q1RIBg4mcFvcqdHfaoUVRIhozg1oj2j84gHXRch7Qqh7+Q0yv2gwfHl3L/FB7IUrEf2L2jIhYG2nTfqLm2YjRkhsUJQX8NipFRvzbNXF5F0hPxXHckrHXOt9bBL9Z4pU0hHVAk4CpfLa9ZGMe2KVcy8V3gh3uUaokEaxFhdmRPN0n+F+KMn1zK83M9MRaRl+LVpC+OqJSWBEFGBAscmA+K00RKq4oW3wNNwyRSwcn7qpwnMQtPoXtcpFY9Cnwe4FZgFiI9ChJUKZelU6Ng1pO6/zuE8DLGrULRc6moWfwT/BVMy4t1B/0TZDfsKoLVcINnd HNEtepuc 7uAXMx5J2ze8KhrnnHhxOaZdDphTA2HUhxZaCfT/wmeYWHO+MaFRNpCglpjX1cbJoinqlIcm+N3JMnzYCVIDcatE2ycUztOKBfJ+/ilIBpoM5SOmh42B/x1P5Zixciy2EWvtvJv0X8KjRDabZYwPHv/BSf/k1/gyrSbq7Nt4EMx9EiIZgEypZxYMTJGVOVbifEcfQTciyxQEP+iBfAio3oQL64hhFj+cE0+TJ3f1mdXEZkxM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Functions marked for error injection can have an associated static key that guards the callsite(s) to avoid overhead of calling an empty function when no error injection is in progress. Outside of the error injection framework itself, bpf programs can be atteched to kprobes and override results of error-injectable functions. To make sure these functions are actually called, attaching such bpf programs should control the static key accordingly. Therefore, add an array of static keys to struct bpf_kprobe_multi_link and fill it in addrs_check_error_injection_list() for programs with kprobe_override enabled, using get_injection_key() instead of within_error_injection_list(). Introduce bpf_kprobe_ei_keys_control() to control the static keys and call the control function when doing multi_link_attach and release. Signed-off-by: Vlastimil Babka --- kernel/trace/bpf_trace.c | 59 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 6 deletions(-) diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 944de1c41209..ef0fadb76bfa 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -2613,6 +2613,7 @@ struct bpf_kprobe_multi_link { struct bpf_link link; struct fprobe fp; unsigned long *addrs; + struct static_key **ei_keys; u64 *cookies; u32 cnt; u32 mods_cnt; @@ -2687,11 +2688,30 @@ static void free_user_syms(struct user_syms *us) kvfree(us->buf); } +static void bpf_kprobe_ei_keys_control(struct bpf_kprobe_multi_link *link, bool enable) +{ + u32 i; + + for (i = 0; i < link->cnt; i++) { + if (!link->ei_keys[i]) + break; + + if (enable) + static_key_slow_inc(link->ei_keys[i]); + else + static_key_slow_dec(link->ei_keys[i]); + } +} + static void bpf_kprobe_multi_link_release(struct bpf_link *link) { struct bpf_kprobe_multi_link *kmulti_link; kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link); + + if (kmulti_link->ei_keys) + bpf_kprobe_ei_keys_control(kmulti_link, false); + unregister_fprobe(&kmulti_link->fp); kprobe_multi_put_modules(kmulti_link->mods, kmulti_link->mods_cnt); } @@ -2703,6 +2723,7 @@ static void bpf_kprobe_multi_link_dealloc(struct bpf_link *link) kmulti_link = container_of(link, struct bpf_kprobe_multi_link, link); kvfree(kmulti_link->addrs); kvfree(kmulti_link->cookies); + kvfree(kmulti_link->ei_keys); kfree(kmulti_link->mods); kfree(kmulti_link); } @@ -2985,13 +3006,19 @@ static int get_modules_for_addrs(struct module ***mods, unsigned long *addrs, u3 return arr.mods_cnt; } -static int addrs_check_error_injection_list(unsigned long *addrs, u32 cnt) +static int addrs_check_error_injection_list(unsigned long *addrs, struct static_key **ei_keys, + u32 cnt) { - u32 i; + struct static_key *ei_key; + u32 i, j = 0; for (i = 0; i < cnt; i++) { - if (!within_error_injection_list(addrs[i])) + ei_key = get_injection_key(addrs[i]); + if (IS_ERR(ei_key)) return -EINVAL; + + if (ei_key) + ei_keys[j++] = ei_key; } return 0; } @@ -3000,6 +3027,7 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr { struct bpf_kprobe_multi_link *link = NULL; struct bpf_link_primer link_primer; + struct static_key **ei_keys = NULL; void __user *ucookies; unsigned long *addrs; u32 flags, cnt, size; @@ -3075,9 +3103,24 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr goto error; } - if (prog->kprobe_override && addrs_check_error_injection_list(addrs, cnt)) { - err = -EINVAL; - goto error; + if (prog->kprobe_override) { + ei_keys = kvcalloc(cnt, sizeof(*ei_keys), GFP_KERNEL); + if (!ei_keys) { + err = -ENOMEM; + goto error; + } + + if (addrs_check_error_injection_list(addrs, ei_keys, cnt)) { + err = -EINVAL; + goto error; + } + + if (ei_keys[0]) { + link->ei_keys = ei_keys; + } else { + kvfree(ei_keys); + ei_keys = NULL; + } } link = kzalloc(sizeof(*link), GFP_KERNEL); @@ -3132,10 +3175,14 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr return err; } + if (link->ei_keys) + bpf_kprobe_ei_keys_control(link, true); + return bpf_link_settle(&link_primer); error: kfree(link); + kvfree(ei_keys); kvfree(addrs); kvfree(cookies); return err;