From patchwork Sat Apr 19 15:57:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Rife X-Patchwork-Id: 14057911 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFA0C1D514E for ; Sat, 19 Apr 2025 15:58:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078291; cv=none; b=Xf6sPKN/gxcEdgoSZb/j3uL+WBD+SyoCUFuwGdzUVKoyC8J0HQ2Y4jiTnPW/Hi5IUn0x4BEMtxN/LmFFjc/I/hWtUACXwMENsvVyiICL72ZRBCg8uj3xbsK+pjic1kCKLyPLf6u3xb3bLON9bKXAMiMrkXKiBHqWkJ4yyKxqpuk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078291; c=relaxed/simple; bh=aaNcjAggGzajn421WtwlUUhGoU4T/w9h9mWzPOqksEg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rOqsI7lUdjIQy/qAFkTm3Zy3+qxjRg+7VkP3Sv3HxzxGsLMlIrSOSSFj+wHZefGJXIlqfagjJPUZ5moM1KTuzVokVF05x4AGNMrKQNUDmHvUNsNEh1ZKF3IUF/7l8kRE41+IsCRzMPRfnuzqs8CsYj3nkHRaPTu05XA0p4h2vmQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io; spf=none smtp.mailfrom=jrife.io; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b=bFlqG/bF; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=jrife.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b="bFlqG/bF" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2ff5544af03so654669a91.1 for ; Sat, 19 Apr 2025 08:58:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jrife-io.20230601.gappssmtp.com; s=20230601; t=1745078289; x=1745683089; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rcR3qNY5OnJnRoRte86S6UevB3XwzFNbcqfVprWZqms=; b=bFlqG/bFlpkY/ykt2Aeh270yxBfDq5qCbmKwkbmoOUGpvZy7mc0WMcqr2nLBj0Pz6+ nCy4krfEUWj8zDOMjQyvCjLPFfNI46yHsE8pvBCcG8dTyy7mw5U7J8Ne3XQLZiciuLBc 44AOX8Y6Kmw9qhW8RY1cQQ9ea12xpE/8mQztxLcAmGU8JWI6gVbqjoEqn54rnIbQuyms 8hc0TIi+lIHbNMtz90D1v1UIDeFEZXR8lOHh3VgEckSk4TDVYdBqyi7fOEF17bdLVzfv tApQbodlnfDlpvqWZv9uaY7iq5PzhrZuygqFO2p1L2TfL5nqbo/0crL9BcKzpAupmoiV SrXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745078289; x=1745683089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rcR3qNY5OnJnRoRte86S6UevB3XwzFNbcqfVprWZqms=; b=SIfTFS/9qv+D7n4OZVSezQT88SHmcBx/mB4lsKheUNRrrbk49d1MCgWti4v0VI2FAJ LPCp3eBFOszGNs+OO9gAW37mwFBDy+GiG0oETpFN6GmphL2rbvjfbzlGoZmvrnwNseCr Rdpe0+osqciOSNGFTVa2zjK9S0iRC5V+yt83mwN8SvoY9cAg4SzWnWrFXZNu5gw/0EsJ NFHZ1WpDlD7waye5u2PW6d5wmHjvVt4Sd+ZaPlw4OSn6tkX1YhteLyiFtorl/gKLIddy DeN+a4ZhrAti+dli10WH85X7GQSWKwDuM6FaLoafYTetW1c7LPjvFwE1Gt8ulOmnHOqV 8jUw== X-Forwarded-Encrypted: i=1; AJvYcCWP+1WYIpf2XnbzTW0Y0ShY2xJ1cmn/TdPfsuTHHHKtUFRU3oaJb1dkZ6aWfJInz7+ibgI=@vger.kernel.org X-Gm-Message-State: AOJu0Yy+wTBtW4CkTzvoYG1l/EBenYiEkWVgI9qvBLSal4G/63yZO17g uTlTogBN7WxttPJvPB0IpCPM1uipoWXfdjE7jf6q2m1zcdZOfY2Y5hbYhS+WjLo= X-Gm-Gg: ASbGncvbAz7RecGzJfsHhAbnPMM/xsy1Xz7sWI4lMMwByp+GeUCxbDoh2BiLWxcgas7 QAwRjF7I+zVXjG6kVOvGymQr76jJeracRTSMv+BMCgDkQb93gEvYSbyB4YGN0EkVcaPAhQoXa4i BDUlOngWdfujKHiIgxBw+CP2+Z07XFksXIvsDU5yTFkBYZXa0HO0c9roVkljaIfoMP7urdzwXVM l3pYMwLn+CTEBHbuhtyVVTaobTwoNySpgH3cx7KVKWrKqB9AyW8DHm5FKKB2u6AqpOoiOWKzbmL 5RpITkDs9AYw6tZqh971kuO/pLQP8g== X-Google-Smtp-Source: AGHT+IHpSG1Yw8p+U7M76MoE3JR+c3kgUiqxOUfo8o+bN/9pt5LExG/gfMiODUgLiOCcO6yzjdCIXQ== X-Received: by 2002:a05:6a00:a95:b0:736:4d90:f9c0 with SMTP id d2e1a72fcca58-73dc144d908mr3018415b3a.1.1745078289029; Sat, 19 Apr 2025 08:58:09 -0700 (PDT) Received: from t14.. ([2001:5a8:4528:b100:1195:fa96:2874:6b2c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73dbf8be876sm3464157b3a.36.2025.04.19.08.58.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Apr 2025 08:58:08 -0700 (PDT) From: Jordan Rife To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: Jordan Rife , Aditi Ghag , Daniel Borkmann , Martin KaFai Lau , Willem de Bruijn , Kuniyuki Iwashima Subject: [PATCH v4 bpf-next 1/6] bpf: udp: Make mem flags configurable through bpf_iter_udp_realloc_batch Date: Sat, 19 Apr 2025 08:57:58 -0700 Message-ID: <20250419155804.2337261-2-jordan@jrife.io> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250419155804.2337261-1-jordan@jrife.io> References: <20250419155804.2337261-1-jordan@jrife.io> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Prepare for the next two patches which need to be able to choose either GFP_USER or GFP_ATOMIC for calls to bpf_iter_udp_realloc_batch by making memory flags configurable. Signed-off-by: Jordan Rife Reviewed-by: Kuniyuki Iwashima --- net/ipv4/udp.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index d0bffcfa56d8..0ac31dec339a 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -3395,7 +3395,7 @@ struct bpf_udp_iter_state { }; static int bpf_iter_udp_realloc_batch(struct bpf_udp_iter_state *iter, - unsigned int new_batch_sz); + unsigned int new_batch_sz, int flags); static struct sock *bpf_iter_udp_batch(struct seq_file *seq) { struct bpf_udp_iter_state *iter = seq->private; @@ -3471,7 +3471,8 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) iter->st_bucket_done = true; goto done; } - if (!resized && !bpf_iter_udp_realloc_batch(iter, batch_sks * 3 / 2)) { + if (!resized && !bpf_iter_udp_realloc_batch(iter, batch_sks * 3 / 2, + GFP_USER)) { resized = true; /* After allocating a larger batch, retry one more time to grab * the whole bucket. @@ -3825,12 +3826,12 @@ DEFINE_BPF_ITER_FUNC(udp, struct bpf_iter_meta *meta, struct udp_sock *udp_sk, uid_t uid, int bucket) static int bpf_iter_udp_realloc_batch(struct bpf_udp_iter_state *iter, - unsigned int new_batch_sz) + unsigned int new_batch_sz, int flags) { struct sock **new_batch; new_batch = kvmalloc_array(new_batch_sz, sizeof(*new_batch), - GFP_USER | __GFP_NOWARN); + flags | __GFP_NOWARN); if (!new_batch) return -ENOMEM; @@ -3853,7 +3854,7 @@ static int bpf_iter_init_udp(void *priv_data, struct bpf_iter_aux_info *aux) if (ret) return ret; - ret = bpf_iter_udp_realloc_batch(iter, INIT_BATCH_SZ); + ret = bpf_iter_udp_realloc_batch(iter, INIT_BATCH_SZ, GFP_USER); if (ret) bpf_iter_fini_seq_net(priv_data); From patchwork Sat Apr 19 15:57:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Rife X-Patchwork-Id: 14057912 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB9D71DDC0F for ; Sat, 19 Apr 2025 15:58:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078292; cv=none; b=nFIauWxi09SRcrnayj8foyWBFFs5364iXwgZFe7Vgdj4VrMHiq+wCqxkIge/ek+OeLhisXBwI5YS68y27UrMYMqSWVur8B0KuMJHUzDJgsN2aegOE9P8G2yA8jKTUiXm1TflfRjwrgOLamqHoPFtvW74/SYxihv4uBnLDvRexpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078292; c=relaxed/simple; bh=IB/qvuVBkNoELVVm0IKpr+V9WQcltfeYsfdKQqo2BWw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FUfFjuLjbGKTyAO/UQUyehzDAVpW5HOhTyhw4nHysqckAu0MQX09NPMy8uxY1roHZ3FG5ZCY/umCk3PiyyNbvWZrD4WnpSILE49SWsUvv8HUBSOke5gBN/VbYXxd49fUgRBOFno85nsFbtpiBgDHdf62/HYIi7WZJRDAk3VYBCE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io; spf=none smtp.mailfrom=jrife.io; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b=vPTYydIS; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=jrife.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b="vPTYydIS" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22403c99457so4066265ad.3 for ; Sat, 19 Apr 2025 08:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jrife-io.20230601.gappssmtp.com; s=20230601; t=1745078290; x=1745683090; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w9gbCNiQ+y8HUaE+W6y8YEShtYnd+/dcegXASkNVABE=; b=vPTYydISfigVT9R7po9w0XI+Wxt802he2Nq17TnQ8bvySm5Vju6EhzSLCGyo5wkUrG M7CseeczBuwPT59rwkm3gp7EJA/7ntdWIidQQcEekxuPZ9Kw5P4vocwKfuW04TO+MNcB DbqZADaS9Su/z8Eb0reMrgkK1v6DM8YzO0AbuF6LSb+Sydajd81CWfIx59scBA+2oS+k iaP1wMmbPaCjM1SsWqxGYWRoesf/499JHPlrGpl2K3r+ProTYHdy4yI74qbuMhqP9GDu J+xgPzP/4eqRUvO1/qvymk4dv0q2Tn4XR+RUQXs/wate378T5czTGlZkK/PT0LP4tCsg fyHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745078290; x=1745683090; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w9gbCNiQ+y8HUaE+W6y8YEShtYnd+/dcegXASkNVABE=; b=MpX8/KBCb57T5K33JcjgQzFvZd7jEwTZfxWEMxV70FER/e73rcJS/Uow8DYkRF1Dsx OdcfKGn9JqJCRsNIoCgqLF5K7m5CxKRWluUzc/OYz1EgfJQL8ZrQCz/zE+6iw9dD/qDU 4Byo413fyS87dRklXrVFN792Ekiuw27AwQ8cUJDAAHoEwTnfwVQV2Avc7hJL7ig4YXTK GO62ZeWDORLZQkWBAiGjpE1z2cghoXWO9Am/m1nTRqdDhjJmrSf04IUIjJPPqwX2zsT9 0yGp74p3cUuy0C2ryT9gieVSucmJmaF8dWyPydUP8Sq8n4Wzn5FRCs8sC8OoG9mUXs7x cbHw== X-Forwarded-Encrypted: i=1; AJvYcCWnNDExX6Phm7crflOECBrJuFOSvrsyrRWxFXGrYqcT8e2tYN354MCIaVpurJwqirWcAM0=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+IZc4qc+SZVRIIghhRtrD3k61wmalv628cZU7OgAfHZ3rEc5M 764qAj9RINK5G2uaYI9BK3i+6ztnl9gZ7Jg8f2TAJ1myjNOwbXSukaI2DHoKmQs= X-Gm-Gg: ASbGncv36DPkDgTHzQsbHHBzdalUcp/TLPbdFzWUKissloGD3MPnWIdqEbgvNVflcTh 4peZOyw1B2WL1c+MNyaSgQvMw7UWVyKjBPe6Bx8AI3A7H+2Bx3+5OPhaHO0Jod93n6xOYbtj3h4 +GOhK9gamuypxkJz0r/ZaiwmLKU4FsYGSsiaY05YnISNBXg3cJLeqE/O7c3wqNLCuJOgIzZqe+q Ai3rOG8jvhhmrvGmGB69WBSiULaZ+hJpl0sLGlda2zgHGViHNgLX7XlCwjPX1DghGC7fHtEQ4bk wORU/kzoCTrk1H/SvIsY2uJTCZcGgQ== X-Google-Smtp-Source: AGHT+IFWNTa+2uuLbzUYIrOvLAxNoctrrfBqbHbCZvuFKhisyD/38/3byOIohM77r1Wb3z5R1yt4rw== X-Received: by 2002:a17:902:e54a:b0:223:5696:44d6 with SMTP id d9443c01a7336-22c5361b458mr37007615ad.12.1745078290007; Sat, 19 Apr 2025 08:58:10 -0700 (PDT) Received: from t14.. ([2001:5a8:4528:b100:1195:fa96:2874:6b2c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73dbf8be876sm3464157b3a.36.2025.04.19.08.58.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Apr 2025 08:58:09 -0700 (PDT) From: Jordan Rife To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: Jordan Rife , Aditi Ghag , Daniel Borkmann , Martin KaFai Lau , Willem de Bruijn , Kuniyuki Iwashima Subject: [PATCH v4 bpf-next 2/6] bpf: udp: Make sure iter->batch always contains a full bucket snapshot Date: Sat, 19 Apr 2025 08:57:59 -0700 Message-ID: <20250419155804.2337261-3-jordan@jrife.io> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250419155804.2337261-1-jordan@jrife.io> References: <20250419155804.2337261-1-jordan@jrife.io> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Require that iter->batch always contains a full bucket snapshot. This invariant is important to avoid skipping or repeating sockets during iteration when combined with the next few patches. Before, there were two cases where a call to bpf_iter_udp_batch may only capture part of a bucket: 1. When bpf_iter_udp_realloc_batch() returns -ENOMEM [1]. 2. When more sockets are added to the bucket while calling bpf_iter_udp_realloc_batch(), making the updated batch size insufficient [2]. In cases where the batch size only covers part of a bucket, it is possible to forget which sockets were already visited, especially if we have to process a bucket in more than two batches. This forces us to choose between repeating or skipping sockets, so don't allow this: 1. Stop iteration and propagate -ENOMEM up to userspace if reallocation fails instead of continuing with a partial batch. 2. Retry bpf_iter_udp_realloc_batch() two times without holding onto the bucket lock (hslot2->lock) so that we can use GFP_USER and maximize the chances that memory allocation succeeds. On the third attempt, if we still haven't been able to capture a full bucket snapshot, hold onto the bucket lock through bpf_iter_udp_realloc_batch() to guarantee that the bucket size doesn't change while we allocate more memory and fill the batch. On the last pass, we must use GFP_ATOMIC since we hold onto the spin lock. Testing all scenarios directly is a bit difficult, but I did some manual testing to exercise the code paths where GFP_ATOMIC is used and where where ERR_PTR(err) is returned to make sure there are no deadlocks. I used the realloc test case included later in this series to trigger a scenario where a realloc happens inside bpf_iter_udp_realloc_batch and made a small code tweak to force the first two realloc attempts to allocate a too-small buffer, thus requiring another attempt until the GFP_ATOMIC case is hit. Some printks showed three reallocs with the tests passing: Apr 16 00:08:32 crow kernel: go again (mem_flags=GFP_USER) Apr 16 00:08:32 crow kernel: go again (mem_flags=GFP_USER) Apr 16 00:08:32 crow kernel: go again (mem_flags=GFP_ATOMIC) With this setup, I also forced bpf_iter_udp_realloc_batch to return -ENOMEM on one of the retries to ensure that iteration ends and that the read() in userspace fails, forced the hlist_empty condition to be true on the GFP_ATOMIC pass to test the first WARN_ON_ONCE condition code path, and walked back iter->end_sk on the GFP_ATOMIC pass to test the second WARN_ON_ONCE condition code path. In each case, locks were released and the loop terminated. [1]: https://lore.kernel.org/bpf/CABi4-ogUtMrH8-NVB6W8Xg_F_KDLq=yy-yu-tKr2udXE2Mu1Lg@mail.gmail.com/ [2]: https://lore.kernel.org/bpf/7ed28273-a716-4638-912d-f86f965e54bb@linux.dev/ Signed-off-by: Jordan Rife Suggested-by: Martin KaFai Lau --- net/ipv4/udp.c | 61 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 14 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 0ac31dec339a..d3128261e4cb 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -3377,6 +3377,7 @@ int udp4_seq_show(struct seq_file *seq, void *v) } #ifdef CONFIG_BPF_SYSCALL +#define MAX_REALLOC_ATTEMPTS 3 struct bpf_iter__udp { __bpf_md_ptr(struct bpf_iter_meta *, meta); __bpf_md_ptr(struct udp_sock *, udp_sk); @@ -3401,11 +3402,13 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) struct bpf_udp_iter_state *iter = seq->private; struct udp_iter_state *state = &iter->state; struct net *net = seq_file_net(seq); + int resizes = MAX_REALLOC_ATTEMPTS; int resume_bucket, resume_offset; struct udp_table *udptable; unsigned int batch_sks = 0; - bool resized = false; + spinlock_t *lock = NULL; struct sock *sk; + int err = 0; resume_bucket = state->bucket; resume_offset = iter->offset; @@ -3433,10 +3436,13 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) struct udp_hslot *hslot2 = &udptable->hash2[state->bucket].hslot; if (hlist_empty(&hslot2->head)) - continue; + goto next_bucket; iter->offset = 0; - spin_lock_bh(&hslot2->lock); + if (!lock) { + lock = &hslot2->lock; + spin_lock_bh(lock); + } udp_portaddr_for_each_entry(sk, &hslot2->head) { if (seq_sk_match(seq, sk)) { /* Resume from the last iterated socket at the @@ -3454,15 +3460,28 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) batch_sks++; } } - spin_unlock_bh(&hslot2->lock); if (iter->end_sk) break; +next_bucket: + /* Somehow the bucket was emptied or all matching sockets were + * removed while we held onto its lock. This should not happen. + */ + if (WARN_ON_ONCE(!resizes)) + /* Best effort; reset the resize budget and move on. */ + resizes = MAX_REALLOC_ATTEMPTS; + if (lock) + spin_unlock_bh(lock); + lock = NULL; } /* All done: no batch made. */ - if (!iter->end_sk) - return NULL; + if (!iter->end_sk) { + sk = NULL; + goto done; + } + + sk = iter->batch[0]; if (iter->end_sk == batch_sks) { /* Batching is done for the current bucket; return the first @@ -3471,16 +3490,30 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) iter->st_bucket_done = true; goto done; } - if (!resized && !bpf_iter_udp_realloc_batch(iter, batch_sks * 3 / 2, - GFP_USER)) { - resized = true; - /* After allocating a larger batch, retry one more time to grab - * the whole bucket. - */ - goto again; + + /* Somehow the batch size still wasn't big enough even though we held + * a lock on the bucket. This should not happen. + */ + if (WARN_ON_ONCE(!resizes)) + goto done; + + resizes--; + if (resizes) { + spin_unlock_bh(lock); + lock = NULL; } + err = bpf_iter_udp_realloc_batch(iter, batch_sks * 3 / 2, + resizes ? GFP_USER : GFP_ATOMIC); + if (err) { + sk = ERR_PTR(err); + goto done; + } + + goto again; done: - return iter->batch[0]; + if (lock) + spin_unlock_bh(lock); + return sk; } static void *bpf_iter_udp_seq_next(struct seq_file *seq, void *v, loff_t *pos) From patchwork Sat Apr 19 15:58:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Rife X-Patchwork-Id: 14057913 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EAE851DF244 for ; Sat, 19 Apr 2025 15:58:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078293; cv=none; b=T0XCGshC324PgP5XIDCOaTvyFbGYa3uSbRgS4K5Md/aqHBjFH1cB380oxOe92oLEi193MZgQzl1ob/wEKiRks+O4wKLKDN2CqxB4IjDlsXkjrrH1hDsoh8ALe9LVD37ywTj2Q59/FVpbQ2QrkT8HtuYJKI4VRTtSzdxvrHNz6g4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078293; c=relaxed/simple; bh=ftgQcqMQpwQf6jHLXFaZm/cI/dHMSA8F/nMjTREAVsI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CBxqOkwJvESEV4+OeRolzWG/G1hmd6s/5h3IHYU8MUoKBwSuFN2vFjLyAhAj81JxlxmCjau700i7GC9ijfRn0dm9OVPPxg3WI4EqW5yrYJcxcPmVVHgyl1saWvgfrZCu+mNg145M52GtVKXNAB4ZjHF33Q2GS9rbAgfd977GaZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io; spf=none smtp.mailfrom=jrife.io; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b=mVOMDgQj; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=jrife.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b="mVOMDgQj" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-af9a6958a08so263719a12.1 for ; Sat, 19 Apr 2025 08:58:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jrife-io.20230601.gappssmtp.com; s=20230601; t=1745078291; x=1745683091; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CJ9tF5XVwy2MCSOmuzi3dTz+kjigAZ4e/XhnhYr6xsg=; b=mVOMDgQjnsAvf9DB8EWCxL2fwY84JiEd5xDD6krVMSOpnVKKgwcPfc3C6ah/J39GM4 hB7FUKCjA46ri5qJqWAlh4uJf259couSUw03xRPWnpggmyeWB4KGvuyRGS/1gazszI8q MqM6M7TuLS6lYhTOeFHzaQ4oe7N0o3Mk4t6wE+dZHIc7h4HsRfCUFqgcIIeI07Zq1wKj 8/GCcxFz6JV2veEtbwSpuaPO+ZQlffWlJpsl8kPEgmKNRFO3gox97wSm7kHBHVBmQSC9 9dCwI9SYD4vopSY2LGD7WvvTDAArVQVV16lhil2FNrCMpE+3an+ki76FnEiL6aPcCPNL 5gUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745078291; x=1745683091; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CJ9tF5XVwy2MCSOmuzi3dTz+kjigAZ4e/XhnhYr6xsg=; b=SmGTeJl1f0ktIzyTiT5gBWXMm48wQvOlvOnTlgs7n9E3nBF1LziEhtefO4aNSKvop2 ZMK5gHJd9Ch/n4DWUsxuVan456PNtIcYYZgka0v3SKD05z72YuwU0BdHfWI6PteM2LXd vqx/5mkU6WqUyFNMi+DRm2+aDYkEFOWfqbb7BLd85/HjxVWwZUOedL6OMe9Wgip05JhP c8EU+9AxaK3qeDBuTRxBB7dP6eWolQdpdhdyvf/WTGEvLRTwS9xvrXHfncKPEhgCVzGZ j0QIY44gSwPdw+IN2BHv4c3ea1lhU3kIELZzbIXFsr099j2Pl/7YasT6OnwWaFtZbH7m Wahg== X-Forwarded-Encrypted: i=1; AJvYcCUOTx+QZcFYCDRDQqpkXBW9ErMS/GgR6I/B7J+/pWVh7nxlOJVRlw643hgcnm1XoD2RV9M=@vger.kernel.org X-Gm-Message-State: AOJu0Yx4r0pHgNWUA5Vp5RRZScBkZLyJ+4cmcidBdI4oL9do5xm3/tOO 0YteMv4HTSAKmhuv9onm/5n7zKFgNuEn6CTLe1f+UMj4S7fc0Tpi3R3YLymkfxM= X-Gm-Gg: ASbGnctmzIO2FsqCnUzqn+7d4pjh/9a3o6YLOLQWgJPYJnAKMHzbZtbOcUg0YVwK+bT HDpi9M3saRN6q9AXXpkXOzQ80yncE+pxVEYrmSapGDFh7BRHa3pBnPSVwyek1O3pA/o7SGuY988 f++Q3DX2SvfNMCz7gTHHSqT3RULD8gTKz3M9Khf+iGo1G3+m3aFwydMS7ovNRpAnuIQ8XyJVmkX WQ/qaFfs5vRDfZb6YNKRa0d3sql6KmyjcrvNg/ku0OiYCITyPgqdImibgt9MnaNDUdxcl5iCbDW J801BIWXzF5rNG7d9AoKB3kKtDwKlg== X-Google-Smtp-Source: AGHT+IEEEVWf/c9Yuxr07De8kexelueNYcfCBdKeMc0n3ueM9/r0r59EqjCCUxNoBhp8qHnC00UNYw== X-Received: by 2002:a05:6a00:3a12:b0:730:915c:b70 with SMTP id d2e1a72fcca58-73dc119a591mr3090239b3a.0.1745078290962; Sat, 19 Apr 2025 08:58:10 -0700 (PDT) Received: from t14.. ([2001:5a8:4528:b100:1195:fa96:2874:6b2c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73dbf8be876sm3464157b3a.36.2025.04.19.08.58.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Apr 2025 08:58:10 -0700 (PDT) From: Jordan Rife To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: Jordan Rife , Aditi Ghag , Daniel Borkmann , Martin KaFai Lau , Willem de Bruijn , Kuniyuki Iwashima Subject: [PATCH v4 bpf-next 3/6] bpf: udp: Use bpf_udp_iter_batch_item for bpf_udp_iter_state batch items Date: Sat, 19 Apr 2025 08:58:00 -0700 Message-ID: <20250419155804.2337261-4-jordan@jrife.io> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250419155804.2337261-1-jordan@jrife.io> References: <20250419155804.2337261-1-jordan@jrife.io> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Prepare for the next patch that tracks cookies between iterations by converting struct sock **batch to union bpf_udp_iter_batch_item *batch inside struct bpf_udp_iter_state. Signed-off-by: Jordan Rife Reviewed-by: Kuniyuki Iwashima --- net/ipv4/udp.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index d3128261e4cb..261dd7f508dd 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -3385,13 +3385,17 @@ struct bpf_iter__udp { int bucket __aligned(8); }; +union bpf_udp_iter_batch_item { + struct sock *sock; +}; + struct bpf_udp_iter_state { struct udp_iter_state state; unsigned int cur_sk; unsigned int end_sk; unsigned int max_sk; int offset; - struct sock **batch; + union bpf_udp_iter_batch_item *batch; bool st_bucket_done; }; @@ -3455,7 +3459,7 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) } if (iter->end_sk < iter->max_sk) { sock_hold(sk); - iter->batch[iter->end_sk++] = sk; + iter->batch[iter->end_sk++].sock = sk; } batch_sks++; } @@ -3481,7 +3485,7 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) goto done; } - sk = iter->batch[0]; + sk = iter->batch[0].sock; if (iter->end_sk == batch_sks) { /* Batching is done for the current bucket; return the first @@ -3525,7 +3529,7 @@ static void *bpf_iter_udp_seq_next(struct seq_file *seq, void *v, loff_t *pos) * done with seq_show(), so unref the iter->cur_sk. */ if (iter->cur_sk < iter->end_sk) { - sock_put(iter->batch[iter->cur_sk++]); + sock_put(iter->batch[iter->cur_sk++].sock); ++iter->offset; } @@ -3533,7 +3537,7 @@ static void *bpf_iter_udp_seq_next(struct seq_file *seq, void *v, loff_t *pos) * available in the current bucket batch. */ if (iter->cur_sk < iter->end_sk) - sk = iter->batch[iter->cur_sk]; + sk = iter->batch[iter->cur_sk].sock; else /* Prepare a new batch. */ sk = bpf_iter_udp_batch(seq); @@ -3598,7 +3602,7 @@ static int bpf_iter_udp_seq_show(struct seq_file *seq, void *v) static void bpf_iter_udp_put_batch(struct bpf_udp_iter_state *iter) { while (iter->cur_sk < iter->end_sk) - sock_put(iter->batch[iter->cur_sk++]); + sock_put(iter->batch[iter->cur_sk++].sock); } static void bpf_iter_udp_seq_stop(struct seq_file *seq, void *v) @@ -3861,7 +3865,7 @@ DEFINE_BPF_ITER_FUNC(udp, struct bpf_iter_meta *meta, static int bpf_iter_udp_realloc_batch(struct bpf_udp_iter_state *iter, unsigned int new_batch_sz, int flags) { - struct sock **new_batch; + union bpf_udp_iter_batch_item *new_batch; new_batch = kvmalloc_array(new_batch_sz, sizeof(*new_batch), flags | __GFP_NOWARN); From patchwork Sat Apr 19 15:58:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Rife X-Patchwork-Id: 14057914 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBCAB1DF73C for ; Sat, 19 Apr 2025 15:58:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078294; cv=none; b=PKVwQfHWJ9ollM0PKeEiXACcw/lnhZPDICSAgeLOWm1MCAoLWKNe5/qR2ijxwevL/dehEESge7THqaRdDK791ALnAxalliHTDKwkVtlWr9mQUUjL4BFdzXyjIjQRVk3jl0MJmG3rUGIZXmh8WeBHwX0ARFvfreh1v0BVixWXijM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078294; c=relaxed/simple; bh=wRxeAltoZzfEN6xOTYcrjYnKQrVosbflDTlg9dI9CuA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=r5YAolDqsX7zE3vk2cPRtMSNA45Z/jmhgKSgVE/eOYi09pW2K3i/iIWnj10b1KrBlm3aKFwL4VuFcnHQjDIM1zh4/QphjtwqgO93mglzoczePVtT9Bs1HdNbsv1BwwmAMgvIWUJFTySi+S1H1Tf2tgvra6GW+/QBm6zXrWrPl1g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io; spf=none smtp.mailfrom=jrife.io; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b=eDhKN8te; arc=none smtp.client-ip=209.85.210.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=jrife.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b="eDhKN8te" Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-73de145c639so2099b3a.1 for ; Sat, 19 Apr 2025 08:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jrife-io.20230601.gappssmtp.com; s=20230601; t=1745078292; x=1745683092; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TJz/DKd+SyfwJrSN217KEOOrpQawXIPq6hbCTcWU3lE=; b=eDhKN8teMm4WLGb6wEP3A3/DFi/HvyFUN6LWN+9l8bDD3yO1N9LOYhsOJYPjfEx/2y 9G6qR1BnAyuFCvWu/U63QS2VPETD3j2lhIS8dfSCRpcWYKEu9ZQtLVMOG8dDDj5DYEwq hN60mEzIqz3cAl0EN47haTHeX0Ma2v8SwwY4uSYIb0Xjb+ikzVp6hfaBtOLUUZFVJ/ie TA2YQZWqmqfgvXxY5UftmnNj+mshCnhRhrTEFRA18eTZAD+eyddT6lYk4s1Qd80mnQDf mB+LSC765aGUtVV+Lgx5oqJxTVPYMR+6XytoX93LhASJUjDn8cyNYxaFz3vIhaW6Hzw2 BMDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745078292; x=1745683092; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TJz/DKd+SyfwJrSN217KEOOrpQawXIPq6hbCTcWU3lE=; b=ULipbwaMfGKLP0kzxeim2aOWlr6IzD7GS1OrC4FHeJQTm+LJQbcQFB376I/sLx7EAg oJPRQ84oB5L5gf3ETk+7pfFQ+Jj8uE0JRBcBsj85xphtiTrUablg5kXUI8FrJ9Liz2cx Gn7UcKFEfN37PzDNwjQ0+I4tZslQUVeos46ceUUvK0gbRamGqx4lz6PIXvgr6atUtMzQ VT0rmT2H9RkrijvvayoiIZTzUCmuFUfiqg4RZ7flCF17H8YyZtPoazL+yr/W8BvIy0XG VoLiD7BlwXJ11W55jvJQs4ar+hPrsP8rydl7upR5p8Ts2RlGE+dDc/1QEHKifTtpcAwV U3yw== X-Forwarded-Encrypted: i=1; AJvYcCVDdzbrsOhDSE4dnxN8WmVfKaTMui5/1ZMh1yVP7P64R7dC/lpDC0y7LB4fDiGHVKYXl4E=@vger.kernel.org X-Gm-Message-State: AOJu0Ywckp9obd08LxfaR3kWcERaNzMmPSYOTO5XSuS6GyuSkLfeLSM3 nFhQXKIxfsdSYdG4GmfhwpepWiaYfrL9lgCTt+ow9213Ms6as2u9WK6D9Kl2a58= X-Gm-Gg: ASbGncs8GgFCePhOkI+6c20ObJjU9rwAQeYC/aTOj6Jp5tLn0zvqil4nXbfa0UKy4IR YWqNdswG7VqwfdO4FqiBvl0Uc+MuEiqKjyhjfR5qaEXq9DUFNzAFqAPeqjROlMKMs/ZwgfX/Dk9 0sDMoYIVlHucVUmsJvEa9StLMVy9/umUD/RbF6Z3JtXt2iTYOJpERkMmm1Y2Q2R+Tc6z6qHfNIz 5v/P07YgnWHUZ2driNRWd+H/MOEActL+kg5jSBwWxojTUn85Uns80Y8uqh+MhBtlRm8csB6NDiy BcUVZ9i7yVR0G4wOUou6LhOy3am29w== X-Google-Smtp-Source: AGHT+IEpXoH4m6ZY7mfRwZUxap2KHQQVIMA9kzePqu3uqomtmo5c7vtiKDri2mZHerNe+707od48LQ== X-Received: by 2002:a05:6a00:1312:b0:739:b1df:2bf1 with SMTP id d2e1a72fcca58-73dc15d3551mr2826023b3a.5.1745078292012; Sat, 19 Apr 2025 08:58:12 -0700 (PDT) Received: from t14.. ([2001:5a8:4528:b100:1195:fa96:2874:6b2c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73dbf8be876sm3464157b3a.36.2025.04.19.08.58.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Apr 2025 08:58:11 -0700 (PDT) From: Jordan Rife To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: Jordan Rife , Aditi Ghag , Daniel Borkmann , Martin KaFai Lau , Willem de Bruijn , Kuniyuki Iwashima Subject: [PATCH v4 bpf-next 4/6] bpf: udp: Avoid socket skips and repeats during iteration Date: Sat, 19 Apr 2025 08:58:01 -0700 Message-ID: <20250419155804.2337261-5-jordan@jrife.io> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250419155804.2337261-1-jordan@jrife.io> References: <20250419155804.2337261-1-jordan@jrife.io> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Replace the offset-based approach for tracking progress through a bucket in the UDP table with one based on socket cookies. Remember the cookies of unprocessed sockets from the last batch and use this list to pick up where we left off or, in the case that the next socket disappears between reads, find the first socket after that point that still exists in the bucket and resume from there. In order to make the control flow a bit easier to follow inside bpf_iter_udp_batch, introduce the udp_portaddr_for_each_entry_from macro and use this to split bucket processing into two stages: finding the starting point and adding items to the next batch. Originally, I implemented this patch inside a single udp_portaddr_for_each_entry loop, as it was before, but I found the resulting logic a bit messy. Overall, this version seems more readable. Signed-off-by: Jordan Rife --- include/linux/udp.h | 3 ++ net/ipv4/udp.c | 77 ++++++++++++++++++++++++++++++++++----------- 2 files changed, 62 insertions(+), 18 deletions(-) diff --git a/include/linux/udp.h b/include/linux/udp.h index 0807e21cfec9..a69da9c4c1c5 100644 --- a/include/linux/udp.h +++ b/include/linux/udp.h @@ -209,6 +209,9 @@ static inline void udp_allow_gso(struct sock *sk) #define udp_portaddr_for_each_entry(__sk, list) \ hlist_for_each_entry(__sk, list, __sk_common.skc_portaddr_node) +#define udp_portaddr_for_each_entry_from(__sk) \ + hlist_for_each_entry_from(__sk, __sk_common.skc_portaddr_node) + #define udp_portaddr_for_each_entry_rcu(__sk, list) \ hlist_for_each_entry_rcu(__sk, list, __sk_common.skc_portaddr_node) diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 261dd7f508dd..701f99e30244 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -93,6 +93,7 @@ #include #include #include +#include #include #include #include @@ -3387,6 +3388,7 @@ struct bpf_iter__udp { union bpf_udp_iter_batch_item { struct sock *sock; + __u64 cookie; }; struct bpf_udp_iter_state { @@ -3394,28 +3396,44 @@ struct bpf_udp_iter_state { unsigned int cur_sk; unsigned int end_sk; unsigned int max_sk; - int offset; union bpf_udp_iter_batch_item *batch; bool st_bucket_done; }; static int bpf_iter_udp_realloc_batch(struct bpf_udp_iter_state *iter, unsigned int new_batch_sz, int flags); +static struct sock *bpf_iter_udp_resume(struct sock *first_sk, + union bpf_udp_iter_batch_item *cookies, + int n_cookies) +{ + struct sock *sk = NULL; + int i = 0; + + for (; i < n_cookies; i++) { + sk = first_sk; + udp_portaddr_for_each_entry_from(sk) + if (cookies[i].cookie == atomic64_read(&sk->sk_cookie)) + goto done; + } +done: + return sk; +} + static struct sock *bpf_iter_udp_batch(struct seq_file *seq) { struct bpf_udp_iter_state *iter = seq->private; struct udp_iter_state *state = &iter->state; + unsigned int find_cookie, end_cookie = 0; struct net *net = seq_file_net(seq); int resizes = MAX_REALLOC_ATTEMPTS; - int resume_bucket, resume_offset; struct udp_table *udptable; unsigned int batch_sks = 0; spinlock_t *lock = NULL; + int resume_bucket; struct sock *sk; int err = 0; resume_bucket = state->bucket; - resume_offset = iter->offset; /* The current batch is done, so advance the bucket. */ if (iter->st_bucket_done) @@ -3431,6 +3449,8 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) * before releasing the bucket lock. This allows BPF programs that are * called in seq_show to acquire the bucket lock if needed. */ + find_cookie = iter->cur_sk; + end_cookie = iter->end_sk; iter->cur_sk = 0; iter->end_sk = 0; iter->st_bucket_done = false; @@ -3442,21 +3462,29 @@ static struct sock *bpf_iter_udp_batch(struct seq_file *seq) if (hlist_empty(&hslot2->head)) goto next_bucket; - iter->offset = 0; if (!lock) { lock = &hslot2->lock; spin_lock_bh(lock); } - udp_portaddr_for_each_entry(sk, &hslot2->head) { + /* Initialize sk to the first socket in hslot2. */ + sk = hlist_entry_safe(hslot2->head.first, struct sock, + __sk_common.skc_portaddr_node); + /* Resume from the first (in iteration order) unseen socket from + * the last batch that still exists in resume_bucket. Most of + * the time this will just be where the last iteration left off + * in resume_bucket unless that socket disappeared between + * reads. + * + * Skip this if end_cookie isn't set; this is the first + * batch, we're on bucket zero, and we want to start from the + * beginning. + */ + if (state->bucket == resume_bucket && end_cookie) + sk = bpf_iter_udp_resume(sk, + &iter->batch[find_cookie], + end_cookie - find_cookie); + udp_portaddr_for_each_entry_from(sk) { if (seq_sk_match(seq, sk)) { - /* Resume from the last iterated socket at the - * offset in the bucket before iterator was stopped. - */ - if (state->bucket == resume_bucket && - iter->offset < resume_offset) { - ++iter->offset; - continue; - } if (iter->end_sk < iter->max_sk) { sock_hold(sk); iter->batch[iter->end_sk++].sock = sk; @@ -3528,10 +3556,8 @@ static void *bpf_iter_udp_seq_next(struct seq_file *seq, void *v, loff_t *pos) /* Whenever seq_next() is called, the iter->cur_sk is * done with seq_show(), so unref the iter->cur_sk. */ - if (iter->cur_sk < iter->end_sk) { + if (iter->cur_sk < iter->end_sk) sock_put(iter->batch[iter->cur_sk++].sock); - ++iter->offset; - } /* After updating iter->cur_sk, check if there are more sockets * available in the current bucket batch. @@ -3601,8 +3627,19 @@ static int bpf_iter_udp_seq_show(struct seq_file *seq, void *v) static void bpf_iter_udp_put_batch(struct bpf_udp_iter_state *iter) { - while (iter->cur_sk < iter->end_sk) - sock_put(iter->batch[iter->cur_sk++].sock); + union bpf_udp_iter_batch_item *item; + unsigned int cur_sk = iter->cur_sk; + __u64 cookie; + + /* Remember the cookies of the sockets we haven't seen yet, so we can + * pick up where we left off next time around. + */ + while (cur_sk < iter->end_sk) { + item = &iter->batch[cur_sk++]; + cookie = __sock_gen_cookie(item->sock); + sock_put(item->sock); + item->cookie = cookie; + } } static void bpf_iter_udp_seq_stop(struct seq_file *seq, void *v) @@ -3873,6 +3910,10 @@ static int bpf_iter_udp_realloc_batch(struct bpf_udp_iter_state *iter, return -ENOMEM; bpf_iter_udp_put_batch(iter); + /* Make sure the new batch has the cookies of the sockets we haven't + * visited yet. + */ + memcpy(new_batch, iter->batch, sizeof(*iter->batch) * iter->end_sk); kvfree(iter->batch); iter->batch = new_batch; iter->max_sk = new_batch_sz; From patchwork Sat Apr 19 15:58:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Rife X-Patchwork-Id: 14057915 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBF0D1DFE0A for ; Sat, 19 Apr 2025 15:58:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078295; cv=none; b=pN2ljf5gwjBYUAvXRZXftjPYZbZl2/q1XeOxfelgpP8POhTyNSze+hfB93ubNYffWpVEI1nWhPyOjOyOaIVmHzp8Er9q46mmvdImA9dGiFqwTd+MsIXNct3OhYttjaoZmiVv5uOdczazCLlT9dwlklSz0VgeQW63XmV2Mkfc5RM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078295; c=relaxed/simple; bh=8D3K8jEJFy2ljx8/dNSQp4nr5+k3ju7eECDWQX6lssg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OQEiq/wFleQCR0T+8aqO2mGIT9BHZQ6Lk02CufhWAkalI26F8MVFND4LgnaIhHk6Y6xdOk4Ad7dqx3k1HeDdPV/H3TmVWPSdQ9N3EzrEz4qj3hCnuwYErzFwmiHww9c1wP1xkcqZEMSdHIfxBznGDQawPAgk5J7KYbCN5cOrSBM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io; spf=none smtp.mailfrom=jrife.io; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b=gH1UmjI0; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=jrife.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b="gH1UmjI0" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-2ff53b26af2so389003a91.0 for ; Sat, 19 Apr 2025 08:58:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jrife-io.20230601.gappssmtp.com; s=20230601; t=1745078293; x=1745683093; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4N8moraxYLNX0qhNglQq0LQLGcDgagzwGmqz/6Y4IFY=; b=gH1UmjI0EHDLFeK+GirBxHFTbur0MeU2LRL/3sm6IiojmEwwf1UDwbv8QVcnvIyoP3 mkE6AScOevvfnnmqWpgusvoSdtMbGr7vW4CaEz9YljR+6vRTfaTmdGntsMRpmqZn5yeT 6zEUd9eLuhibLmXPJqf99S61pUbG8I6pkES/ae8Y6EcMWnXm0dx7poHg6wxfsSm6cHht C+foV+1vRXkSYuqBZLxJVBq36f66eGcnNgfnaYc/8RjFMGteDHw9QuJETKcSqxc2Hssq sN1t12MXN3mQMDz+QHees9fASD0OQTtLcB3LwfA8SbAMI/VsBYeGqI9sPWez3Xk+qio7 NqOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745078293; x=1745683093; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4N8moraxYLNX0qhNglQq0LQLGcDgagzwGmqz/6Y4IFY=; b=F1KP6z49CAGiGEOoYXwGGlBVR/CkcUPk8iocFct9Oliygl7k/HZrexXtgXIcLWN/1e Zbzj8sX5tXEQhnUGoQyd42XZkMPHg/cCZcpGLelRMPI+BJcUt7Bh0ti0rrxZhCavIB1G 8rsMblymTAGhwHk1za7SY6wEzIGQArnL/0UAyH0Bf6Nuu5A7GnQalhcfXjO2kyzpV0el j5sIsGbK9RWZdrePbvHvlD9WmH6mV4mab8trXVIlriob48EUFzh9HTTnAjQ5Op157ANu JPXulafabbFgspsi1KI5ctedWzleHmJoiTGUY/Blnhfd8Ew4+4aUB4O3BTmpDaic5lDP n9+Q== X-Forwarded-Encrypted: i=1; AJvYcCWGjIwYWXEk+HvsEueDmLFAAZNs8Fl0yU6vNM3DChzOwXNiS6pi9g8elHKQyi2Rc5LKvZ8=@vger.kernel.org X-Gm-Message-State: AOJu0YxcVuKNZfWInb9l0ShhsXiFzSCKrUbYTR/uCEORkUEJJa3Mdg3B XZqxwt+wSCILpR0R7iFM7+zBrUTWTnerspNJM7qnnE3MYOE6H7rEZddhHI30dEg= X-Gm-Gg: ASbGncvroN5b9GU7KHyPLHRXTyL0LTfWQUEcPEVILCMiVuOOyKpzlBZ5wnVWhTsZFpp 1Njpq6nMdVeCLnfiZCKOsPud8niJcHgEfXdF89o6yc4MiCwwMpa1fMnyJ3uoJVlN69utxhPNHgD L4tIneyehVcjVDWt4heygGNgSvvSEIQQMb/Z1Mx3tW/3iK7ZTnft7EcTnCqsWtD5rRKuZXebGfA xNxEnyYV7a02PUGKTqlUZYfY+3fbPG+TLBQZouT4MBqLrdOChIiA726gsiU/hPtge8Q+LnXVv8g kP7ZtHK+7Sb+piLhGjvQ+wymehlZhNTVCG9clgLl X-Google-Smtp-Source: AGHT+IGwr1IoVc4T9jwUrTdcFgxHzIfi4vUqD/KzsCN1sRmqeSTCogZygbF0wjg5Ag7ap8jFKeKlnw== X-Received: by 2002:a05:6a00:3984:b0:730:8c9d:5842 with SMTP id d2e1a72fcca58-73dc1612e3dmr2724601b3a.5.1745078292998; Sat, 19 Apr 2025 08:58:12 -0700 (PDT) Received: from t14.. ([2001:5a8:4528:b100:1195:fa96:2874:6b2c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73dbf8be876sm3464157b3a.36.2025.04.19.08.58.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Apr 2025 08:58:12 -0700 (PDT) From: Jordan Rife To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: Jordan Rife , Aditi Ghag , Daniel Borkmann , Martin KaFai Lau , Willem de Bruijn , Kuniyuki Iwashima Subject: [PATCH v4 bpf-next 5/6] selftests/bpf: Return socket cookies from sock_iter_batch progs Date: Sat, 19 Apr 2025 08:58:02 -0700 Message-ID: <20250419155804.2337261-6-jordan@jrife.io> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250419155804.2337261-1-jordan@jrife.io> References: <20250419155804.2337261-1-jordan@jrife.io> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Extend the iter_udp_soreuse and iter_tcp_soreuse programs to write the cookie of the current socket, so that we can track the identity of the sockets that the iterator has seen so far. Update the existing do_test function to account for this change to the iterator program output. At the same time, teach both programs to work with AF_INET as well. Signed-off-by: Jordan Rife --- .../bpf/prog_tests/sock_iter_batch.c | 33 +++++++++++-------- .../selftests/bpf/progs/bpf_tracing_net.h | 1 + .../selftests/bpf/progs/sock_iter_batch.c | 24 +++++++++++--- 3 files changed, 41 insertions(+), 17 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c b/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c index d56e18b25528..74dbe91806a0 100644 --- a/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c +++ b/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c @@ -9,12 +9,18 @@ static const int nr_soreuse = 4; +struct iter_out { + int idx; + __u64 cookie; +} __packed; + static void do_test(int sock_type, bool onebyone) { int err, i, nread, to_read, total_read, iter_fd = -1; - int first_idx, second_idx, indices[nr_soreuse]; + struct iter_out outputs[nr_soreuse]; struct bpf_link *link = NULL; struct sock_iter_batch *skel; + int first_idx, second_idx; int *fds[2] = {}; skel = sock_iter_batch__open(); @@ -34,6 +40,7 @@ static void do_test(int sock_type, bool onebyone) goto done; skel->rodata->ports[i] = ntohs(local_port); } + skel->rodata->sf = AF_INET6; err = sock_iter_batch__load(skel); if (!ASSERT_OK(err, "sock_iter_batch__load")) @@ -55,38 +62,38 @@ static void do_test(int sock_type, bool onebyone) * from a bucket and leave one socket out from * that bucket on purpose. */ - to_read = (nr_soreuse - 1) * sizeof(*indices); + to_read = (nr_soreuse - 1) * sizeof(*outputs); total_read = 0; first_idx = -1; do { - nread = read(iter_fd, indices, onebyone ? sizeof(*indices) : to_read); - if (nread <= 0 || nread % sizeof(*indices)) + nread = read(iter_fd, outputs, onebyone ? sizeof(*outputs) : to_read); + if (nread <= 0 || nread % sizeof(*outputs)) break; total_read += nread; if (first_idx == -1) - first_idx = indices[0]; - for (i = 0; i < nread / sizeof(*indices); i++) - ASSERT_EQ(indices[i], first_idx, "first_idx"); + first_idx = outputs[0].idx; + for (i = 0; i < nread / sizeof(*outputs); i++) + ASSERT_EQ(outputs[i].idx, first_idx, "first_idx"); } while (total_read < to_read); - ASSERT_EQ(nread, onebyone ? sizeof(*indices) : to_read, "nread"); + ASSERT_EQ(nread, onebyone ? sizeof(*outputs) : to_read, "nread"); ASSERT_EQ(total_read, to_read, "total_read"); free_fds(fds[first_idx], nr_soreuse); fds[first_idx] = NULL; /* Read the "whole" second bucket */ - to_read = nr_soreuse * sizeof(*indices); + to_read = nr_soreuse * sizeof(*outputs); total_read = 0; second_idx = !first_idx; do { - nread = read(iter_fd, indices, onebyone ? sizeof(*indices) : to_read); - if (nread <= 0 || nread % sizeof(*indices)) + nread = read(iter_fd, outputs, onebyone ? sizeof(*outputs) : to_read); + if (nread <= 0 || nread % sizeof(*outputs)) break; total_read += nread; - for (i = 0; i < nread / sizeof(*indices); i++) - ASSERT_EQ(indices[i], second_idx, "second_idx"); + for (i = 0; i < nread / sizeof(*outputs); i++) + ASSERT_EQ(outputs[i].idx, second_idx, "second_idx"); } while (total_read <= to_read); ASSERT_EQ(nread, 0, "nread"); /* Both so_reuseport ports should be in different buckets, so diff --git a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h index 659694162739..17db400f0e0d 100644 --- a/tools/testing/selftests/bpf/progs/bpf_tracing_net.h +++ b/tools/testing/selftests/bpf/progs/bpf_tracing_net.h @@ -128,6 +128,7 @@ #define sk_refcnt __sk_common.skc_refcnt #define sk_state __sk_common.skc_state #define sk_net __sk_common.skc_net +#define sk_rcv_saddr __sk_common.skc_rcv_saddr #define sk_v6_daddr __sk_common.skc_v6_daddr #define sk_v6_rcv_saddr __sk_common.skc_v6_rcv_saddr #define sk_flags __sk_common.skc_flags diff --git a/tools/testing/selftests/bpf/progs/sock_iter_batch.c b/tools/testing/selftests/bpf/progs/sock_iter_batch.c index 96531b0d9d55..8f483337e103 100644 --- a/tools/testing/selftests/bpf/progs/sock_iter_batch.c +++ b/tools/testing/selftests/bpf/progs/sock_iter_batch.c @@ -17,6 +17,12 @@ static bool ipv6_addr_loopback(const struct in6_addr *a) a->s6_addr32[2] | (a->s6_addr32[3] ^ bpf_htonl(1))) == 0; } +static bool ipv4_addr_loopback(__be32 a) +{ + return a == bpf_ntohl(0x7f000001); +} + +volatile const unsigned int sf; volatile const __u16 ports[2]; unsigned int bucket[2]; @@ -26,16 +32,20 @@ int iter_tcp_soreuse(struct bpf_iter__tcp *ctx) struct sock *sk = (struct sock *)ctx->sk_common; struct inet_hashinfo *hinfo; unsigned int hash; + __u64 sock_cookie; struct net *net; int idx; if (!sk) return 0; + sock_cookie = bpf_get_socket_cookie(sk); sk = bpf_core_cast(sk, struct sock); - if (sk->sk_family != AF_INET6 || + if (sk->sk_family != sf || sk->sk_state != TCP_LISTEN || - !ipv6_addr_loopback(&sk->sk_v6_rcv_saddr)) + sk->sk_family == AF_INET6 ? + !ipv6_addr_loopback(&sk->sk_v6_rcv_saddr) : + !ipv4_addr_loopback(sk->sk_rcv_saddr)) return 0; if (sk->sk_num == ports[0]) @@ -52,6 +62,7 @@ int iter_tcp_soreuse(struct bpf_iter__tcp *ctx) hinfo = net->ipv4.tcp_death_row.hashinfo; bucket[idx] = hash & hinfo->lhash2_mask; bpf_seq_write(ctx->meta->seq, &idx, sizeof(idx)); + bpf_seq_write(ctx->meta->seq, &sock_cookie, sizeof(sock_cookie)); return 0; } @@ -63,14 +74,18 @@ int iter_udp_soreuse(struct bpf_iter__udp *ctx) { struct sock *sk = (struct sock *)ctx->udp_sk; struct udp_table *udptable; + __u64 sock_cookie; int idx; if (!sk) return 0; + sock_cookie = bpf_get_socket_cookie(sk); sk = bpf_core_cast(sk, struct sock); - if (sk->sk_family != AF_INET6 || - !ipv6_addr_loopback(&sk->sk_v6_rcv_saddr)) + if (sk->sk_family != sf || + sk->sk_family == AF_INET6 ? + !ipv6_addr_loopback(&sk->sk_v6_rcv_saddr) : + !ipv4_addr_loopback(sk->sk_rcv_saddr)) return 0; if (sk->sk_num == ports[0]) @@ -84,6 +99,7 @@ int iter_udp_soreuse(struct bpf_iter__udp *ctx) udptable = sk->sk_net.net->ipv4.udp_table; bucket[idx] = udp_sk(sk)->udp_portaddr_hash & udptable->mask; bpf_seq_write(ctx->meta->seq, &idx, sizeof(idx)); + bpf_seq_write(ctx->meta->seq, &sock_cookie, sizeof(sock_cookie)); return 0; } From patchwork Sat Apr 19 15:58:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jordan Rife X-Patchwork-Id: 14057916 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C9211E0DF5 for ; Sat, 19 Apr 2025 15:58:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078296; cv=none; b=d5L92m8TckJ99HQ21u5saaoisVmFMCMMPHfaZQWqF3YAnXnmPCj4lIfqu7v/Wf87EO/tQZBwzWPOMWS5eikR2/8R7Evfdiiib7buFKBPUPfeW5s43C5G6EQJji3QZnditEAdIcLDKPFtdxNaOzMG8mJRGRc85/cEmZ96EAYaYfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745078296; c=relaxed/simple; bh=cNXWBHt6J7e3H+76Mh3BMIenbMHu29MKu1h73tRrASw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oW37hk2OIodzSx4h0xr6mwbkPk549//SwsJ3Uwqw46x1K+Qv4V/LsPY5qrW1HHOentcoPhlYWYGVkj63Dt+Xus/NCSkHuRNAVUu1sLQxr6y255+Ya4iMcoEF/9kIp2S4nr0JKLrptENT9g0bEVI5MhcFINMgsr0XFQuGJ2mjB9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io; spf=none smtp.mailfrom=jrife.io; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b=Oflm9Q4o; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=jrife.io Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=jrife.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=jrife-io.20230601.gappssmtp.com header.i=@jrife-io.20230601.gappssmtp.com header.b="Oflm9Q4o" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2ff5544af03so654691a91.1 for ; Sat, 19 Apr 2025 08:58:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jrife-io.20230601.gappssmtp.com; s=20230601; t=1745078294; x=1745683094; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KaKbW65uYaVDYGMsoIHJ/fjnTr1nbMz9f/dqAmx8e2I=; b=Oflm9Q4o5I1Ro7+fInK7L4Szma4qF2P7kCH+/JE0dSz01qgQacEdzoBYsCn0QtVPeI d+Ji1ZTMmOso6dLhTBiee0dwVWSJ4dWdMw3uLQuPsE8pIWptdfb/KMVtKEO3Ccs8cP2x DYnyEqHjK/K58+xokapmP2RBZKh3+0P7GCCBy+8M7hx1KW5foJpdaeKaZm76T1NUEWLS hqXgrsZAOd+G2iVKSoQ+xRPUw5yEgO+/JiSO3U39fTEibLLhxiGGJRyLd6Lzra25I644 AQsJoWko2KLxOETGRONahd9k6tHEQnwB+HclHwu8mQpn49VAyglzydC0zvLSHhniB0CD CoFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745078294; x=1745683094; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KaKbW65uYaVDYGMsoIHJ/fjnTr1nbMz9f/dqAmx8e2I=; b=UgSKFn/eZ8U21Rcyb2bj6kDHZgqCX/UD9JuUdIGC9RJfGbKhgPXR4XT2X6spME053S lPAx+rK85xrgdIeJqnVtKF3XppIZuEqzCxG9X8XyKEg4QM1OOcRjnHp+Iux1NDFJd1WI PDa6H5NJmK73de18G7tp6pnwDgZyygnyP43fQgOkST6kemb39SG/4jv7augP6d0usPL/ l/QslFW7Aj3jdO2t90WehnHO1iib7e84vKqGL0pwN8WCtdDPYp6EQab5oY+AXcOgf/eM HaZ2HLsyR74er2b+9Ofh7AC0CKilJHcGSdOJQuEf9O1cErgR8quGn4BsvIuQKQlSgee2 65pw== X-Forwarded-Encrypted: i=1; AJvYcCX8k618ANWo2JM6o0vYRxcLnuWCobO88y5cdoy4wTon1SkzmSmjbhzGQy2szGUeN4oInxM=@vger.kernel.org X-Gm-Message-State: AOJu0YxFhIh8gKxyINnfE/DH1Cx7wJtSvt5EIuFFE6023AhAFWWQpSw9 zEgc6p9ZI0OogTGc6+oe+Rx+RBNC3MJfHTZwSPPMq8VGOShYQH/LkllMXNkHReRZTjdin6ubKfa jl3E= X-Gm-Gg: ASbGncunC6SuQ+kuZopuodfeH8gWtQoZgtwu2JUSECAh13KTu7Sw4f0oR5NuRpM0UNm l2kDvJkGhB62BjTQ7I5zjtwLnaulEgTW4jceeTqNEIAtldsGlgY4BIcJQYjhjKJ+2S5rrm9LnBv s0Aejm/PBHCmIgaNog30uagAJaMEwM3oJ+2dZvKjz7ohYTPmNPJsFJFYDxlhw+Q3LFlzysyXzTm 0ytDOYH/NKlU2J0agmx2IBQBYpJ6jpqJCEggdhKYPq/N7lIvpwcBRorwzkw+bnlHfOtvLd31ULv wlBcFFxisyHkulX3rfIJdb+1StJ7uQ== X-Google-Smtp-Source: AGHT+IEPQIMy5EbSGJmUQXVK90krjzGlsnceK+mu2vOGI0wgPy7fMclnx0vWOlLmoLhVcwcGsmgFHA== X-Received: by 2002:a05:6a00:3a12:b0:730:915c:b70 with SMTP id d2e1a72fcca58-73dc119a591mr3090281b3a.0.1745078294042; Sat, 19 Apr 2025 08:58:14 -0700 (PDT) Received: from t14.. ([2001:5a8:4528:b100:1195:fa96:2874:6b2c]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73dbf8be876sm3464157b3a.36.2025.04.19.08.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 19 Apr 2025 08:58:13 -0700 (PDT) From: Jordan Rife To: netdev@vger.kernel.org, bpf@vger.kernel.org Cc: Jordan Rife , Aditi Ghag , Daniel Borkmann , Martin KaFai Lau , Willem de Bruijn , Kuniyuki Iwashima Subject: [PATCH v4 bpf-next 6/6] selftests/bpf: Add tests for bucket resume logic in UDP socket iterators Date: Sat, 19 Apr 2025 08:58:03 -0700 Message-ID: <20250419155804.2337261-7-jordan@jrife.io> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250419155804.2337261-1-jordan@jrife.io> References: <20250419155804.2337261-1-jordan@jrife.io> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Introduce a set of tests that exercise various bucket resume scenarios: * remove_seen resumes iteration after removing a socket from the bucket that we've already processed. Before, with the offset-based approach, this test would have skipped an unseen socket after resuming iteration. With the cookie-based approach, we now see all sockets exactly once. * remove_unseen exercises the condition where the next socket that we would have seen is removed from the bucket before we resume iteration. This tests the scenario where we need to scan past the first cookie in our remembered cookies list to find the socket from which to resume iteration. * remove_all exercises the condition where all sockets we remembered were removed from the bucket to make sure iteration terminates and returns no more results. * add_some exercises the condition where a few, but not enough to trigger a realloc, sockets are added to the head of the current bucket between reads. Before, with the offset-based approach, this test would have repeated sockets we've already seen. With the cookie-based approach, we now see all sockets exactly once. * force_realloc exercises the condition that we need to realloc the batch on a subsequent read, since more sockets than can be held in the current batch array were added to the current bucket. This exercies the logic inside bpf_iter_udp_realloc_batch that copies cookies into the new batch to make sure nothing is skipped or repeated. Signed-off-by: Jordan Rife --- .../bpf/prog_tests/sock_iter_batch.c | 414 ++++++++++++++++++ 1 file changed, 414 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c b/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c index 74dbe91806a0..218c7258c0e0 100644 --- a/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c +++ b/tools/testing/selftests/bpf/prog_tests/sock_iter_batch.c @@ -7,6 +7,7 @@ #define TEST_NS "sock_iter_batch_netns" +static const int init_batch_size = 16; static const int nr_soreuse = 4; struct iter_out { @@ -14,6 +15,418 @@ struct iter_out { __u64 cookie; } __packed; +struct sock_count { + __u64 cookie; + int count; +}; + +static int insert(__u64 cookie, struct sock_count counts[], int counts_len) +{ + int insert = -1; + int i = 0; + + for (; i < counts_len; i++) { + if (!counts[i].cookie) { + insert = i; + } else if (counts[i].cookie == cookie) { + insert = i; + break; + } + } + if (insert < 0) + return insert; + + counts[insert].cookie = cookie; + counts[insert].count++; + + return counts[insert].count; +} + +static int read_n(int iter_fd, int n, struct sock_count counts[], + int counts_len) +{ + struct iter_out out; + int nread = 1; + int i = 0; + + for (; nread > 0 && (n < 0 || i < n); i++) { + nread = read(iter_fd, &out, sizeof(out)); + if (!nread || !ASSERT_EQ(nread, sizeof(out), "nread")) + break; + ASSERT_GE(insert(out.cookie, counts, counts_len), 0, "insert"); + } + + ASSERT_TRUE(n < 0 || i == n, "n < 0 || i == n"); + + return i; +} + +static __u64 socket_cookie(int fd) +{ + __u64 cookie; + socklen_t cookie_len = sizeof(cookie); + + if (!ASSERT_OK(getsockopt(fd, SOL_SOCKET, SO_COOKIE, &cookie, + &cookie_len), "getsockopt(SO_COOKIE)")) + return 0; + return cookie; +} + +static bool was_seen(int fd, struct sock_count counts[], int counts_len) +{ + __u64 cookie = socket_cookie(fd); + int i = 0; + + for (; cookie && i < counts_len; i++) + if (cookie == counts[i].cookie) + return true; + + return false; +} + +static int get_seen_socket(int *fds, struct sock_count counts[], int n) +{ + int i = 0; + + for (; i < n; i++) + if (was_seen(fds[i], counts, n)) + return i; + return -1; +} + +static int get_nth_socket(int *fds, int fds_len, struct bpf_link *link, int n) +{ + int i, nread, iter_fd; + int nth_sock_idx = -1; + struct iter_out out; + + iter_fd = bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_OK_FD(iter_fd, "bpf_iter_create")) + return -1; + + for (; n >= 0; n--) { + nread = read(iter_fd, &out, sizeof(out)); + if (!nread || !ASSERT_GE(nread, 1, "nread")) + goto done; + } + + for (i = 0; i < fds_len && nth_sock_idx < 0; i++) + if (fds[i] >= 0 && socket_cookie(fds[i]) == out.cookie) + nth_sock_idx = i; +done: + close(iter_fd); + return nth_sock_idx; +} + +static int get_seen_count(int fd, struct sock_count counts[], int n) +{ + __u64 cookie = socket_cookie(fd); + int count = 0; + int i = 0; + + for (; cookie && !count && i < n; i++) + if (cookie == counts[i].cookie) + count = counts[i].count; + + return count; +} + +static void check_n_were_seen_once(int *fds, int fds_len, int n, + struct sock_count counts[], int counts_len) +{ + int seen_once = 0; + int seen_cnt; + int i = 0; + + for (; i < fds_len; i++) { + /* Skip any sockets that were closed or that weren't seen + * exactly once. + */ + if (fds[i] < 0) + continue; + seen_cnt = get_seen_count(fds[i], counts, counts_len); + if (seen_cnt && ASSERT_EQ(seen_cnt, 1, "seen_cnt")) + seen_once++; + } + + ASSERT_EQ(seen_once, n, "seen_once"); +} + +static void remove_seen(int family, int sock_type, const char *addr, __u16 port, + int *socks, int socks_len, struct sock_count *counts, + int counts_len, struct bpf_link *link, int iter_fd) +{ + int close_idx; + + /* Iterate through the first socks_len - 1 sockets. */ + read_n(iter_fd, socks_len - 1, counts, counts_len); + + /* Make sure we saw socks_len - 1 sockets exactly once. */ + check_n_were_seen_once(socks, socks_len, socks_len - 1, counts, + counts_len); + + /* Close a socket we've already seen to remove it from the bucket. */ + close_idx = get_seen_socket(socks, counts, counts_len); + if (!ASSERT_GE(close_idx, 0, "close_idx")) + return; + close(socks[close_idx]); + socks[close_idx] = -1; + + /* Iterate through the rest of the sockets. */ + read_n(iter_fd, -1, counts, counts_len); + + /* Make sure the last socket wasn't skipped and that there were no + * repeats. + */ + check_n_were_seen_once(socks, socks_len, socks_len - 1, counts, + counts_len); +} + +static void remove_unseen(int family, int sock_type, const char *addr, + __u16 port, int *socks, int socks_len, + struct sock_count *counts, int counts_len, + struct bpf_link *link, int iter_fd) +{ + int close_idx; + + /* Iterate through the first socket. */ + read_n(iter_fd, 1, counts, counts_len); + + /* Make sure we saw a socket from fds. */ + check_n_were_seen_once(socks, socks_len, 1, counts, counts_len); + + /* Close what would be the next socket in the bucket to exercise the + * condition where we need to skip past the first cookie we remembered. + */ + close_idx = get_nth_socket(socks, socks_len, link, 1); + if (!ASSERT_GE(close_idx, 0, "close_idx")) + return; + close(socks[close_idx]); + socks[close_idx] = -1; + + /* Iterate through the rest of the sockets. */ + read_n(iter_fd, -1, counts, counts_len); + + /* Make sure the remaining sockets were seen exactly once and that we + * didn't repeat the socket that was already seen. + */ + check_n_were_seen_once(socks, socks_len, socks_len - 1, counts, + counts_len); +} + +static void remove_all(int family, int sock_type, const char *addr, + __u16 port, int *socks, int socks_len, + struct sock_count *counts, int counts_len, + struct bpf_link *link, int iter_fd) +{ + int close_idx, i; + + /* Iterate through the first socket. */ + read_n(iter_fd, 1, counts, counts_len); + + /* Make sure we saw a socket from fds. */ + check_n_were_seen_once(socks, socks_len, 1, counts, counts_len); + + /* Close all remaining sockets to exhaust the list of saved cookies and + * exit without putting any sockets into the batch on the next read. + */ + for (i = 0; i < socks_len - 1; i++) { + close_idx = get_nth_socket(socks, socks_len, link, 1); + if (!ASSERT_GE(close_idx, 0, "close_idx")) + return; + close(socks[close_idx]); + socks[close_idx] = -1; + } + + /* Make sure there are no more sockets returned */ + ASSERT_EQ(read_n(iter_fd, -1, counts, counts_len), 0, "read_n"); +} + +static void add_some(int family, int sock_type, const char *addr, __u16 port, + int *socks, int socks_len, struct sock_count *counts, + int counts_len, struct bpf_link *link, int iter_fd) +{ + int *new_socks = NULL; + + /* Iterate through the first socks_len - 1 sockets. */ + read_n(iter_fd, socks_len - 1, counts, counts_len); + + /* Make sure we saw socks_len - 1 sockets exactly once. */ + check_n_were_seen_once(socks, socks_len, socks_len - 1, counts, + counts_len); + + /* Double the number of sockets in the bucket. */ + new_socks = start_reuseport_server(family, sock_type, addr, port, 0, + socks_len); + if (!ASSERT_OK_PTR(new_socks, "start_reuseport_server")) + goto done; + + /* Iterate through the rest of the sockets. */ + read_n(iter_fd, -1, counts, counts_len); + + /* Make sure each of the original sockets was seen exactly once. */ + check_n_were_seen_once(socks, socks_len, socks_len, counts, + counts_len); +done: + free_fds(new_socks, socks_len); +} + +static void force_realloc(int family, int sock_type, const char *addr, + __u16 port, int *socks, int socks_len, + struct sock_count *counts, int counts_len, + struct bpf_link *link, int iter_fd) +{ + int *new_socks = NULL; + + /* Iterate through the first socket just to initialize the batch. */ + read_n(iter_fd, 1, counts, counts_len); + + /* Double the number of sockets in the bucket to force a realloc on the + * next read. + */ + new_socks = start_reuseport_server(family, sock_type, addr, port, 0, + socks_len); + if (!ASSERT_OK_PTR(new_socks, "start_reuseport_server")) + goto done; + + /* Iterate through the rest of the sockets. */ + read_n(iter_fd, -1, counts, counts_len); + + /* Make sure each socket from the first set was seen exactly once. */ + check_n_were_seen_once(socks, socks_len, socks_len, counts, + counts_len); +done: + free_fds(new_socks, socks_len); +} + +struct test_case { + void (*test)(int family, int sock_type, const char *addr, __u16 port, + int *socks, int socks_len, struct sock_count *counts, + int counts_len, struct bpf_link *link, int iter_fd); + const char *description; + int init_socks; + int max_socks; + int sock_type; + int family; +}; + +static struct test_case resume_tests[] = { + { + .description = "udp: resume after removing a seen socket", + .init_socks = nr_soreuse, + .max_socks = nr_soreuse, + .sock_type = SOCK_DGRAM, + .family = AF_INET6, + .test = remove_seen, + }, + { + .description = "udp: resume after removing one unseen socket", + .init_socks = nr_soreuse, + .max_socks = nr_soreuse, + .sock_type = SOCK_DGRAM, + .family = AF_INET6, + .test = remove_unseen, + }, + { + .description = "udp: resume after removing all unseen sockets", + .init_socks = nr_soreuse, + .max_socks = nr_soreuse, + .sock_type = SOCK_DGRAM, + .family = AF_INET6, + .test = remove_all, + }, + { + .description = "udp: resume after adding a few sockets", + .init_socks = nr_soreuse, + .max_socks = nr_soreuse, + .sock_type = SOCK_DGRAM, + /* Use AF_INET so that new sockets are added to the head of the + * bucket's list. + */ + .family = AF_INET, + .test = add_some, + }, + { + .description = "udp: force a realloc to occur", + .init_socks = init_batch_size, + .max_socks = init_batch_size * 2, + .sock_type = SOCK_DGRAM, + /* Use AF_INET6 so that new sockets are added to the tail of the + * bucket's list, needing to be added to the next batch to force + * a realloc. + */ + .family = AF_INET6, + .test = force_realloc, + }, +}; + +static void do_resume_test(struct test_case *tc) +{ + static const __u16 port = 10001; + struct bpf_link *link = NULL; + struct sock_iter_batch *skel; + struct sock_count *counts; + int err, iter_fd = -1; + const char *addr; + int local_port; + int *fds; + + counts = calloc(tc->max_socks, sizeof(*counts)); + if (!counts) + return; + skel = sock_iter_batch__open(); + if (!ASSERT_OK_PTR(skel, "sock_iter_batch__open")) + return; + + /* Prepare a bucket of sockets in the kernel hashtable */ + addr = tc->family == AF_INET6 ? "::1" : "127.0.0.1"; + fds = start_reuseport_server(tc->family, tc->sock_type, addr, port, 0, + tc->init_socks); + if (!ASSERT_OK_PTR(fds, "start_reuseport_server")) + goto done; + local_port = get_socket_local_port(*fds); + if (!ASSERT_GE(local_port, 0, "get_socket_local_port")) + goto done; + skel->rodata->ports[0] = ntohs(local_port); + skel->rodata->sf = tc->family; + + err = sock_iter_batch__load(skel); + if (!ASSERT_OK(err, "sock_iter_batch__load")) + goto done; + + link = bpf_program__attach_iter(tc->sock_type == SOCK_STREAM ? + skel->progs.iter_tcp_soreuse : + skel->progs.iter_udp_soreuse, + NULL); + if (!ASSERT_OK_PTR(link, "bpf_program__attach_iter")) + goto done; + + iter_fd = bpf_iter_create(bpf_link__fd(link)); + if (!ASSERT_OK_FD(iter_fd, "bpf_iter_create")) + goto done; + + tc->test(tc->family, tc->sock_type, addr, port, fds, tc->init_socks, + counts, tc->max_socks, link, iter_fd); +done: + free(counts); + free_fds(fds, tc->init_socks); + if (iter_fd >= 0) + close(iter_fd); + bpf_link__destroy(link); + sock_iter_batch__destroy(skel); +} + +static void do_resume_tests(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(resume_tests); i++) { + if (test__start_subtest(resume_tests[i].description)) { + do_resume_test(&resume_tests[i]); + } + } +} + static void do_test(int sock_type, bool onebyone) { int err, i, nread, to_read, total_read, iter_fd = -1; @@ -135,6 +548,7 @@ void test_sock_iter_batch(void) do_test(SOCK_DGRAM, true); do_test(SOCK_DGRAM, false); } + do_resume_tests(); close_netns(nstoken); done: