From patchwork Wed Feb 21 22:59:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13566578 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAE673C470; Wed, 21 Feb 2024 22:59:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708556358; cv=none; b=a6qTvUG3YdXjPetVgKAPGYVhgBtx5N+RozWkrUK/rpKKmoRa2XzLPIlWAuI0xEHJ4LTk95DDt5rgYzbz9TCC1OXYA02keFaVQgDgYkABvlflOWPN2TxpkzRfgWrG5n9NMcEPyALja51pi3pid+mdqOQz83VXCd5ErQ5zpKKAEos= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708556358; c=relaxed/simple; bh=8DszHd8Yh2iLgRH5yS1Ubvp/IpbprkrxBS0YFmimpR4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=tXr6NcTICQPasPtmw6DGS2hGpiRHpPwVMH1/aERYbYwtwpEtihwQgFTvYTzed6/06az/lgEa2UTMA/KPzVzfOliZum32fpqutimdjlw297gO232EQ+RJDE9CDJCg3jDovcE8WdQCf/RLNmk3Y5OAQd3GEGSK8Yt2DYBgB79L9/0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=edL3vLAr; arc=none smtp.client-ip=209.85.128.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="edL3vLAr" Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-6079d44b02bso52642427b3.3; Wed, 21 Feb 2024 14:59:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708556355; x=1709161155; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ApHR3wqPEgShTz8EdjIgnsj0XrqO8wckCeRoLIO1Hlc=; b=edL3vLAro4/eES0SeMSbVmFS91KP/naGBIBwiR8hRlxHTHpZgLsL3Wp3MppxXMPjiW muMklmL+0Hc+vskwsmhK9lvIBQDbqSfTILUj/xdlj7qO4VYaFej1G6w0rouIJyluMBH1 fbuPrOpk0WL5/Zy9WAczb+t1tSrcJiYyN/DHdK6mgqlO2vyJJO2ySI/AthfYCkrd9hpS ueZARtzugdyv2NpJUYYSdI4ewARRadLmUW0Uh4aYAKx/yVelETVhuAEmhnxJ7d6yNzcE i2Lqe9yf1iiAtlmP4zcjAb01StcizkXlDaljJ2bBOCdSOIDCN5muEm4ivQ+7R/0FylP+ 6Vwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708556355; x=1709161155; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ApHR3wqPEgShTz8EdjIgnsj0XrqO8wckCeRoLIO1Hlc=; b=mzympbhhHs2rts8arMSRCa7rVUAAWr1jDf4vIbBfvRKp/XZ2bYdF4DgV50Vt1yU8dB X6ufoDkkvVT0zdSrv3qoTHRvWtI0+aczucJ1V0ZrBMoXKVUZKKxOE3tDhHGlB5952Ey3 Re81lGDmns6GfUEiKaZdjzyqZ+L2t1xIh4Jr1EE6f4i3mzO5kd+OWJSyNef4K9PTszYm NZVUfL306sF11y35ktE4C9SPI2bytcp1NuMyLiyaVxldQucI0xD4hp/UR/QAIuFTwp1h Gg4m6gKFvyYDA1ikUdCOjIbCFkQIAuzwt1t/Qe3f+cd86yvmh8nQkviOAMYQ7mTbdZGE Kzew== X-Forwarded-Encrypted: i=1; AJvYcCXLFSYyd/ovPtEdRMGfdVO+jUMPe8LDyginIaVQ+oN8+0rl9OWrdvSlyMolDUOhUuupvBCrlIhn1odDjhIoG8KsTa75TZRU X-Gm-Message-State: AOJu0YwjzLRT5tHQpeQI9pTOcQ5WitDsPiO2znML0OVZKIEYy7hVH1WH vAAvZayJb2yg3eh3w/NBtgEG5qSPweywztDvoC+4JYnp/hr86UZgmwRCknxR X-Google-Smtp-Source: AGHT+IHiy7PkPnbN35acFNftsrHydVmFdo29ylKaVbWiUS2vDhj86RpTdukMThDBIg7AedqsjXPr3Q== X-Received: by 2002:a81:ae13:0:b0:604:7b9e:f622 with SMTP id m19-20020a81ae13000000b006047b9ef622mr15760158ywh.30.1708556355445; Wed, 21 Feb 2024 14:59:15 -0800 (PST) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:bc3b:b762:a625:955f]) by smtp.gmail.com with ESMTPSA id s67-20020a0de946000000b006078c48a265sm2820090ywe.6.2024.02.21.14.59.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 14:59:15 -0800 (PST) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee , netdev@vger.kernel.org Subject: [PATCH bpf-next v2 1/3] bpf, net: validate struct_ops when updating value. Date: Wed, 21 Feb 2024 14:59:09 -0800 Message-Id: <20240221225911.757861-2-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240221225911.757861-1-thinker.li@gmail.com> References: <20240221225911.757861-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Perform all validations when updating values of struct_ops maps. Doing validation in st_ops->reg() and st_ops->update() is not necessary anymore. However, tcp_register_congestion_control() has been called in various places. It still needs to do validations. Cc: netdev@vger.kernel.org Signed-off-by: Kui-Feng Lee --- kernel/bpf/bpf_struct_ops.c | 11 ++++++----- net/ipv4/tcp_cong.c | 6 +----- 2 files changed, 7 insertions(+), 10 deletions(-) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index 0d7be97a2411..c244ed5114fd 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -667,13 +667,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, *(unsigned long *)(udata + moff) = prog->aux->id; } + if (st_ops->validate) { + err = st_ops->validate(kdata); + if (err) + goto reset_unlock; + } + if (st_map->map.map_flags & BPF_F_LINK) { err = 0; - if (st_ops->validate) { - err = st_ops->validate(kdata); - if (err) - goto reset_unlock; - } arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); /* Let bpf_link handle registration & unregistration. * diff --git a/net/ipv4/tcp_cong.c b/net/ipv4/tcp_cong.c index 1b34050a7538..28ffcfbeef14 100644 --- a/net/ipv4/tcp_cong.c +++ b/net/ipv4/tcp_cong.c @@ -146,11 +146,7 @@ EXPORT_SYMBOL_GPL(tcp_unregister_congestion_control); int tcp_update_congestion_control(struct tcp_congestion_ops *ca, struct tcp_congestion_ops *old_ca) { struct tcp_congestion_ops *existing; - int ret; - - ret = tcp_validate_congestion_control(ca); - if (ret) - return ret; + int ret = 0; ca->key = jhash(ca->name, sizeof(ca->name), strlen(ca->name)); From patchwork Wed Feb 21 22:59:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13566579 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14C7D5232 for ; Wed, 21 Feb 2024 22:59:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708556359; cv=none; b=bM6WuyG8PHzNOPh+KJqM3H5Ls4W3eMHWeFiIi2U5A1wJixHcz9ULqz3cku7jZUxDVr+ffcRjdTWJNzMM8MdR0kRvlsge11PRcBMTft1oOhdTuCuEMx3hALaG9ZZM3gOUtwS/LF3zma3C6wIIEC0/ob9aubGX7ZZ2wz0eFNAEFHM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708556359; c=relaxed/simple; bh=AQuGZabXM0QUiKy2406UmkHLUrpu8r0Sm5l/hiHl4aI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JeLc3Uxh4AVXNpGSnQW/Y8gEYHf778HoQ/wY+PFbFhfQZnOBD4tL2kakM35LXsliX6JRRLZ10UjjCfJ0/7nbTUh4UkFcZRaBnFIT83WO6M8D3jK2gjTlXDzCoctc3+aAfthNtSDE1ycVX+Mx/M4wjqml5JP4KfPoDj+AzN82ocQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FXyO2h2c; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FXyO2h2c" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-6083dc087b3so38079667b3.0 for ; Wed, 21 Feb 2024 14:59:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708556356; x=1709161156; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l3Tgcz2FmHfN5mrra5ju37RF75F3ETRnI/0SVQD+JbQ=; b=FXyO2h2ckXhCo3eby6bGLTsWxMgIfrW/7Hgz+EuDrwQN3B3dZs7rUTHWNZiuA2bAeX y1cfulXZP41Z161XyK0dHmZ4UMWbVDi+zoVCcSk72x0/1HhyXDT8Pmj74QSP9EmB0IuT aEfhlKGRKtVFOSPrR+9wIyh03mQAn4BxYokQmrp7x3Rpv7IwGf78P3/MSnzxg94dsVqd 7aoouIpHRoEsQ1uGDHzYdqlv7w77YG7bopp9+hU9sqWUkurgEoNS0+hSbmZq7RbRqgD9 +LkZyy0P+kQJio+JkViCFXZMUA4J0RxzwNq6X1a+KAzS1w27nP+TOCvO5Kpe7hh9cbt0 xEuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708556356; x=1709161156; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l3Tgcz2FmHfN5mrra5ju37RF75F3ETRnI/0SVQD+JbQ=; b=A3PSP8EbNVZaylNK2bNJiJf06clCXMmtvrsuIV89v0YjPAKcUoNGkCagDPEZKSptS3 tGvDa2NCMqz1NWdny4YVodZvAYRxfbE3h57auooCqgZUSlWBUT1x9sMOWxokUP2Mf1xE G7VNEe8MS8mk8TI0F6t5JpfQMqS33CSdy4IEZLz1FTJfxXPEYm4ycFJ5vgSa+8u+rlLL gCvCptHEHSW5phDhUnGwdUajwEMuY7busO2ZsanlBfQFuADbv1hnY1PxFP1Mk9ocAaHu R9SEl8Aq5X7ZY0sIa9EQtL97itE+Z0McgnVVBu6/CVgYumwp0qd/Av1X+4+V6/E6xfmS V8Nw== X-Gm-Message-State: AOJu0YykbWOH9ntPQvIcGO6MZOgCNqdi0TqzWd6H67wU/aBZ0FJR7Mqx Gg9iO/jClFB3SvFBicHYn30kzbpphZuSaAZ4rEFfgT2UMt+V0CRr85Nvh9uh X-Google-Smtp-Source: AGHT+IEJ4nxmlpmFBEK05u8rouAQklDf/2g9F83LhQnSRZoyrXdAEnWEKs7CQBYQe39dML2Q5zkzYw== X-Received: by 2002:a81:bc45:0:b0:608:7686:21b5 with SMTP id b5-20020a81bc45000000b00608768621b5mr4628170ywl.35.1708556356556; Wed, 21 Feb 2024 14:59:16 -0800 (PST) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:bc3b:b762:a625:955f]) by smtp.gmail.com with ESMTPSA id s67-20020a0de946000000b006078c48a265sm2820090ywe.6.2024.02.21.14.59.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 14:59:16 -0800 (PST) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v2 2/3] bpf: struct_ops supports more than one page for trampolines. Date: Wed, 21 Feb 2024 14:59:10 -0800 Message-Id: <20240221225911.757861-3-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240221225911.757861-1-thinker.li@gmail.com> References: <20240221225911.757861-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee The BPF struct_ops previously only allowed for one page to be used for the trampolines of all links in a map. However, we have recently run out of space due to the large number of BPF program links. By allocating additional pages when we exhaust an existing page, we can accommodate more links in a single map. The variable st_map->image has been changed to st_map->image_pages, and its type has been changed to an array of pointers to buffers of PAGE_SIZE. Every struct_ops map can have MAX_IMAGE_PAGES (8) pages for trampolines at most. Signed-off-by: Kui-Feng Lee --- kernel/bpf/bpf_struct_ops.c | 112 +++++++++++++++++++++++++----------- 1 file changed, 79 insertions(+), 33 deletions(-) diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c index c244ed5114fd..15bf8058161b 100644 --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -18,6 +18,8 @@ struct bpf_struct_ops_value { char data[] ____cacheline_aligned_in_smp; }; +#define MAX_TRAMP_IMAGE_PAGES 8 + struct bpf_struct_ops_map { struct bpf_map map; struct rcu_head rcu; @@ -30,12 +32,11 @@ struct bpf_struct_ops_map { */ struct bpf_link **links; u32 links_cnt; - /* image is a page that has all the trampolines + u32 image_pages_cnt; + /* image_pages is an array of pages that has all the trampolines * that stores the func args before calling the bpf_prog. - * A PAGE_SIZE "image" is enough to store all trampoline for - * "links[]". */ - void *image; + void *image_pages[MAX_TRAMP_IMAGE_PAGES]; /* The owner moduler's btf. */ struct btf *btf; /* uvalue->data stores the kernel struct @@ -456,6 +457,41 @@ static void bpf_struct_ops_map_put_progs(struct bpf_struct_ops_map *st_map) } } +static void bpf_struct_ops_map_free_image(struct bpf_struct_ops_map *st_map) +{ + int i; + void *image; + + bpf_jit_uncharge_modmem(PAGE_SIZE * st_map->image_pages_cnt); + for (i = 0; i < st_map->image_pages_cnt; i++) { + image = st_map->image_pages[i]; + arch_free_bpf_trampoline(image, PAGE_SIZE); + } + st_map->image_pages_cnt = 0; +} + +static void *bpf_struct_ops_map_inc_image(struct bpf_struct_ops_map *st_map) +{ + void *image; + int err; + + if (st_map->image_pages_cnt >= MAX_TRAMP_IMAGE_PAGES) + return ERR_PTR(-E2BIG); + + err = bpf_jit_charge_modmem(PAGE_SIZE); + if (err) + return ERR_PTR(err); + + image = arch_alloc_bpf_trampoline(PAGE_SIZE); + if (!image) { + bpf_jit_uncharge_modmem(PAGE_SIZE); + return ERR_PTR(-ENOMEM); + } + st_map->image_pages[st_map->image_pages_cnt++] = image; + + return image; +} + static int check_zero_holes(const struct btf *btf, const struct btf_type *t, void *data) { const struct btf_member *member; @@ -531,10 +567,10 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, const struct btf_type *module_type; const struct btf_member *member; const struct btf_type *t = st_ops_desc->type; + void *image = NULL, *image_end = NULL; struct bpf_tramp_links *tlinks; void *udata, *kdata; int prog_fd, err; - void *image, *image_end; u32 i; if (flags) @@ -573,15 +609,14 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, udata = &uvalue->data; kdata = &kvalue->data; - image = st_map->image; - image_end = st_map->image + PAGE_SIZE; module_type = btf_type_by_id(btf_vmlinux, st_ops_ids[IDX_MODULE_ID]); for_each_member(i, t, member) { const struct btf_type *mtype, *ptype; struct bpf_prog *prog; struct bpf_tramp_link *link; - u32 moff; + u32 moff, tflags; + int tsize; moff = __btf_member_bit_offset(t, member) / 8; ptype = btf_type_resolve_ptr(st_map->btf, member->type, NULL); @@ -653,10 +688,38 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, &bpf_struct_ops_link_lops, prog); st_map->links[i] = &link->link; - err = bpf_struct_ops_prepare_trampoline(tlinks, link, - &st_ops->func_models[i], - *(void **)(st_ops->cfi_stubs + moff), - image, image_end); + tflags = BPF_TRAMP_F_INDIRECT; + if (st_ops->func_models[i].ret_size > 0) + tflags |= BPF_TRAMP_F_RET_FENTRY_RET; + + /* Compute the size of the trampoline */ + tlinks[BPF_TRAMP_FENTRY].links[0] = link; + tlinks[BPF_TRAMP_FENTRY].nr_links = 1; + tsize = arch_bpf_trampoline_size(&st_ops->func_models[i], + tflags, tlinks, NULL); + if (tsize < 0) { + err = tsize; + goto reset_unlock; + } + + /* Allocate pages */ + if (tsize > (unsigned long)image_end - (unsigned long)image) { + if (tsize > PAGE_SIZE) { + err = -E2BIG; + goto reset_unlock; + } + image = bpf_struct_ops_map_inc_image(st_map); + if (IS_ERR(image)) { + err = PTR_ERR(image); + goto reset_unlock; + } + image_end = image + PAGE_SIZE; + } + + err = arch_prepare_bpf_trampoline(NULL, image, image_end, + &st_ops->func_models[i], + tflags, tlinks, + *(void **)(st_ops->cfi_stubs + moff)); if (err < 0) goto reset_unlock; @@ -672,10 +735,11 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, if (err) goto reset_unlock; } + for (i = 0; i < st_map->image_pages_cnt; i++) + arch_protect_bpf_trampoline(st_map->image_pages[i], PAGE_SIZE); if (st_map->map.map_flags & BPF_F_LINK) { err = 0; - arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); /* Let bpf_link handle registration & unregistration. * * Pair with smp_load_acquire() during lookup_elem(). @@ -684,7 +748,6 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, goto unlock; } - arch_protect_bpf_trampoline(st_map->image, PAGE_SIZE); err = st_ops->reg(kdata); if (likely(!err)) { /* This refcnt increment on the map here after @@ -707,9 +770,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, * there was a race in registering the struct_ops (under the same name) to * a sub-system through different struct_ops's maps. */ - arch_unprotect_bpf_trampoline(st_map->image, PAGE_SIZE); reset_unlock: + bpf_struct_ops_map_free_image(st_map); bpf_struct_ops_map_put_progs(st_map); memset(uvalue, 0, map->value_size); memset(kvalue, 0, map->value_size); @@ -776,10 +839,7 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map) if (st_map->links) bpf_struct_ops_map_put_progs(st_map); bpf_map_area_free(st_map->links); - if (st_map->image) { - arch_free_bpf_trampoline(st_map->image, PAGE_SIZE); - bpf_jit_uncharge_modmem(PAGE_SIZE); - } + bpf_struct_ops_map_free_image(st_map); bpf_map_area_free(st_map->uvalue); bpf_map_area_free(st_map); } @@ -889,20 +949,6 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr) st_map->st_ops_desc = st_ops_desc; map = &st_map->map; - ret = bpf_jit_charge_modmem(PAGE_SIZE); - if (ret) - goto errout_free; - - st_map->image = arch_alloc_bpf_trampoline(PAGE_SIZE); - if (!st_map->image) { - /* __bpf_struct_ops_map_free() uses st_map->image as flag - * for "charged or not". In this case, we need to unchange - * here. - */ - bpf_jit_uncharge_modmem(PAGE_SIZE); - ret = -ENOMEM; - goto errout_free; - } st_map->uvalue = bpf_map_area_alloc(vt->size, NUMA_NO_NODE); st_map->links_cnt = btf_type_vlen(t); st_map->links = From patchwork Wed Feb 21 22:59:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kui-Feng Lee X-Patchwork-Id: 13566580 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60E9E126F0B for ; Wed, 21 Feb 2024 22:59:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708556361; cv=none; b=eLeerIauakKHJsSs+BoTfu8SjiAR5JT96L5sAhgiWVa3AGdimNidLOpQKJxcoq4IpwiW8YDYqwpaRU/qGz+/hw0ur1otZ5E57TwBYDciqnlXOwyuKE4qCEPfTWRlCdI48L2BCSPy0BBAT6izrES8qCmlB95AB9pwA9WqWShC+9c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708556361; c=relaxed/simple; bh=eI9M7LeZIbVf4PAKpdJzSqozqw81fFyHCfaAf8GvGos=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JFpwC8RFQ/TVQpdKNbTJqexLMUCFURvM3NInAN+5Lroio3QulVI3ynk96z6vU5PoAaP0/aSDr1C7Vmw6dZDafokO1x1PFKfMtHaIA737uUgT7eFQCW8Nu+OoqcpmdZwgsFIZ6SqAgm2yLngbXnJsYfjDn/msdMeQJoDVBtdrIRk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=d1TbxbK8; arc=none smtp.client-ip=209.85.128.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="d1TbxbK8" Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-60869c68926so24903267b3.3 for ; Wed, 21 Feb 2024 14:59:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1708556358; x=1709161158; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hpHAZChXrmDGEf/hZq9KDg2E3R4kWtgxbD/4zGjA7lY=; b=d1TbxbK8yRk18LOvLsuDd9a5dymry4A5CVN6qCCl3KqOqesDtuhBaFp7/m5L17YAgn cfLZ/t5GxDyuHv3I50oEOXkJhR0S7jgxEuVzJTcztR49Xrg/D1TPCJiFlK0Xv7OLsYiO 5IlSO02JOjv5AApOdUerAKoSJUQZ1budquQTlURkoSi80fRiEb7OeHRryrbuvmr79MB8 LF5AF2yZ5i3EMMRrQF9iQnuKwZ4n5LIes5R5fAvqrjgIFx3k3q1G//HdlBU7c3In42OX R17rVdjJcdUTdba+vKy9kaAWcpzm21oQZztwBhBfi9AR/NXGgPx6+v7qkDKGOU+m6NTF L6wQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708556358; x=1709161158; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hpHAZChXrmDGEf/hZq9KDg2E3R4kWtgxbD/4zGjA7lY=; b=iyaG2ZhnuqaotzTGKA8QO16CXnWs0TDiyq4CiCvadmUn8pGMFkPLS+D2cJRs/R3Sy1 6dOTohVJBRK+/W/K8+LCuXGSh4cDtiuyFvdLLAieSQz60AHjvaHMoM2Q9TrzNe19pOLj VOkz5oHjjjl4FlE0Mj+C5+TKdmKZOOzgtwrzfxoyJ/0GdRkbUe4WM/sdSscJHqUwfBFz N7rGyi6DHUbDlPZf+Y9KE2mu8YmRnaFOo/TrhtdYigVpy9F6Li0pxmNjkiws1z2Whac0 3vcTiMT2fasf18MLQ1UwOCSsijmD2Ht68U+/lVaHfrUoMHVnGpYqXs5W7ryVF2kdmWj7 YLsA== X-Gm-Message-State: AOJu0YxGt0POt966A/QaU/fP/G/dLsrIRimFPno09wSWtiJvNFga7JH7 ZrFH0JPYNc3AnZ3aIgABAJYdAeKOBkeqaAdtqRe6/BRA0qZBwBV7kO3RDno6 X-Google-Smtp-Source: AGHT+IH36POtfhaTUFVrMB4akrqnwhqFmYNRLIWDg2ktuwCWlMyq84LXV1hqq03Tc6i3aTft/0q3Vg== X-Received: by 2002:a05:690c:d07:b0:608:2f27:5c0e with SMTP id cn7-20020a05690c0d0700b006082f275c0emr12258553ywb.8.1708556357761; Wed, 21 Feb 2024 14:59:17 -0800 (PST) Received: from kickker.attlocal.net ([2600:1700:6cf8:1240:bc3b:b762:a625:955f]) by smtp.gmail.com with ESMTPSA id s67-20020a0de946000000b006078c48a265sm2820090ywe.6.2024.02.21.14.59.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Feb 2024 14:59:17 -0800 (PST) From: thinker.li@gmail.com To: bpf@vger.kernel.org, ast@kernel.org, martin.lau@linux.dev, song@kernel.org, kernel-team@meta.com, andrii@kernel.org Cc: sinquersw@gmail.com, kuifeng@meta.com, Kui-Feng Lee Subject: [PATCH bpf-next v2 3/3] selftests/bpf: Test struct_ops maps with a large number of program links. Date: Wed, 21 Feb 2024 14:59:11 -0800 Message-Id: <20240221225911.757861-4-thinker.li@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240221225911.757861-1-thinker.li@gmail.com> References: <20240221225911.757861-1-thinker.li@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net From: Kui-Feng Lee Create and load a struct_ops map with a large number of programs to generate trampolines taking a size over multiple pages. The map includes 40 programs. Their trampolines takes 6.6k+, more than 1.5 pages, on x86. Signed-off-by: Kui-Feng Lee --- .../selftests/bpf/bpf_testmod/bpf_testmod.h | 44 ++++++++ .../prog_tests/test_struct_ops_multi_pages.c | 31 ++++++ .../bpf/progs/struct_ops_multi_pages.c | 102 ++++++++++++++++++ 3 files changed, 177 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c create mode 100644 tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h index c3b0cf788f9f..3e6481b2a50a 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.h @@ -35,6 +35,50 @@ struct bpf_testmod_ops { void (*test_2)(int a, int b); /* Used to test nullable arguments. */ int (*test_maybe_null)(int dummy, struct task_struct *task); + + /* The following pointers are used to test the maps having multiple + * pages of trampolines. + */ + int (*tramp_1)(int value); + int (*tramp_2)(int value); + int (*tramp_3)(int value); + int (*tramp_4)(int value); + int (*tramp_5)(int value); + int (*tramp_6)(int value); + int (*tramp_7)(int value); + int (*tramp_8)(int value); + int (*tramp_9)(int value); + int (*tramp_10)(int value); + int (*tramp_11)(int value); + int (*tramp_12)(int value); + int (*tramp_13)(int value); + int (*tramp_14)(int value); + int (*tramp_15)(int value); + int (*tramp_16)(int value); + int (*tramp_17)(int value); + int (*tramp_18)(int value); + int (*tramp_19)(int value); + int (*tramp_20)(int value); + int (*tramp_21)(int value); + int (*tramp_22)(int value); + int (*tramp_23)(int value); + int (*tramp_24)(int value); + int (*tramp_25)(int value); + int (*tramp_26)(int value); + int (*tramp_27)(int value); + int (*tramp_28)(int value); + int (*tramp_29)(int value); + int (*tramp_30)(int value); + int (*tramp_31)(int value); + int (*tramp_32)(int value); + int (*tramp_33)(int value); + int (*tramp_34)(int value); + int (*tramp_35)(int value); + int (*tramp_36)(int value); + int (*tramp_37)(int value); + int (*tramp_38)(int value); + int (*tramp_39)(int value); + int (*tramp_40)(int value); }; #endif /* _BPF_TESTMOD_H */ diff --git a/tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c new file mode 100644 index 000000000000..9edb2985bc1b --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/test_struct_ops_multi_pages.c @@ -0,0 +1,31 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include + +#include "struct_ops_multi_pages.skel.h" + +static void do_struct_ops_multi_pages(void) +{ + struct struct_ops_multi_pages *skel; + struct bpf_link *link; + + /* The size of all trampolines of skel->maps.multi_pages should be + * over 1 page (at least for x86). + */ + skel = struct_ops_multi_pages__open_and_load(); + if (!ASSERT_OK_PTR(skel, "struct_ops_multi_pages_open_and_load")) + return; + + link = bpf_map__attach_struct_ops(skel->maps.multi_pages); + ASSERT_OK_PTR(link, "attach_multi_pages"); + + bpf_link__destroy(link); + struct_ops_multi_pages__destroy(skel); +} + +void test_struct_ops_multi_pages(void) +{ + + if (test__start_subtest("multi_pages")) + do_struct_ops_multi_pages(); +} diff --git a/tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c b/tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c new file mode 100644 index 000000000000..9efcc6e4d356 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/struct_ops_multi_pages.c @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */ +#include +#include +#include +#include "../bpf_testmod/bpf_testmod.h" + +char _license[] SEC("license") = "GPL"; + +#define TRAMP(x) \ + SEC("struct_ops/tramp_" #x) \ + int BPF_PROG(tramp_ ## x, int a) \ + { \ + return a; \ + } + +TRAMP(1) +TRAMP(2) +TRAMP(3) +TRAMP(4) +TRAMP(5) +TRAMP(6) +TRAMP(7) +TRAMP(8) +TRAMP(9) +TRAMP(10) +TRAMP(11) +TRAMP(12) +TRAMP(13) +TRAMP(14) +TRAMP(15) +TRAMP(16) +TRAMP(17) +TRAMP(18) +TRAMP(19) +TRAMP(20) +TRAMP(21) +TRAMP(22) +TRAMP(23) +TRAMP(24) +TRAMP(25) +TRAMP(26) +TRAMP(27) +TRAMP(28) +TRAMP(29) +TRAMP(30) +TRAMP(31) +TRAMP(32) +TRAMP(33) +TRAMP(34) +TRAMP(35) +TRAMP(36) +TRAMP(37) +TRAMP(38) +TRAMP(39) +TRAMP(40) + +#define F_TRAMP(x) .tramp_ ## x = (void *)tramp_ ## x + +SEC(".struct_ops.link") +struct bpf_testmod_ops multi_pages = { + F_TRAMP(1), + F_TRAMP(2), + F_TRAMP(3), + F_TRAMP(4), + F_TRAMP(5), + F_TRAMP(6), + F_TRAMP(7), + F_TRAMP(8), + F_TRAMP(9), + F_TRAMP(10), + F_TRAMP(11), + F_TRAMP(12), + F_TRAMP(13), + F_TRAMP(14), + F_TRAMP(15), + F_TRAMP(16), + F_TRAMP(17), + F_TRAMP(18), + F_TRAMP(19), + F_TRAMP(20), + F_TRAMP(21), + F_TRAMP(22), + F_TRAMP(23), + F_TRAMP(24), + F_TRAMP(25), + F_TRAMP(26), + F_TRAMP(27), + F_TRAMP(28), + F_TRAMP(29), + F_TRAMP(30), + F_TRAMP(31), + F_TRAMP(32), + F_TRAMP(33), + F_TRAMP(34), + F_TRAMP(35), + F_TRAMP(36), + F_TRAMP(37), + F_TRAMP(38), + F_TRAMP(39), + F_TRAMP(40), +};