From patchwork Thu Feb 20 07:29:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983421 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 385CF25760; Thu, 20 Feb 2025 07:29:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036597; cv=none; b=Pt/yDDTODH5cfudi2L/hNPIZvl88WA41esTKrAgEjkmbTXIXvNYRJxqaHe57X8bPsg/8We6eD2hSOkdiSxPI3pcAyNXXc1LVrARpMZ5Dxfspr2dK7tEuC9qz92eJp4j6THht0pWrHo+eySC6IpF52OA8el8nM91AGwaquzYqn2A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036597; c=relaxed/simple; bh=6jOBXQ0/BeF1R0G4OuKYN32U0jVvUvMDDDlTPGH/BHI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Pk8bAtuv7gTmU0NiqdqOu75L+KsN+wlr+PNVPHE6rg1Dlshf6bExsLxOQBUN/Xx3xgDVJhttKwC3ktUquFSDpyUCC+TKNWRHqF0oQ9YIclQQS9DjTFwGk6Y7SjBgwJZQyknYJYa4iCoN5m8o+E4iEmPIxpCceChMFdDnbJoC2YE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nHmy20p+; arc=none smtp.client-ip=209.85.216.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nHmy20p+" Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2fbfe16cc39so1221759a91.3; Wed, 19 Feb 2025 23:29:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036595; x=1740641395; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7SIpMNJaEln02czHucseSyDjX69GDtQ/wBexwQPO1aI=; b=nHmy20p+y3a216RAkPTfMwyxlx6Me8TncFj8s9V0nWpRhacEl2wHXVFA4rbY5QLxI7 uNNH2ARoMu0Y9icVfV5fLG8s51ri61SycS14aeRuen1HBAXbCwXbNnmFAExg31ZvLynG +s7nWCVcIMnqOgMa1do2qSbwp7dszhKs3ifWh7oyYE2NeE6HVR/nbD75lg5fyDtiXJKm 6buBPE3s+uHqBiBDcZEkDxSZRekEqS/O1ZT4RACaO61n+j+Iv2AQHGrkQFHjV1Cp+v4A UO1z03+VQYp3OBoUHqLdZuqP0M8mf1nXcYENnPtDGrokG9Ide2O3yaBw8sGXRVOMdZ3t t6ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036595; x=1740641395; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7SIpMNJaEln02czHucseSyDjX69GDtQ/wBexwQPO1aI=; b=PdewIbnysLrX3dZ6lot5AgNt7qdHcwjKYI8gxYP4Qg7wqcE5M6PHuEm62HimH4+LU7 KYIeO+XBdGDs0JMlLngGSnPxl4pMRAP03C9fp4pVoInaoHF8knDhIYOKvhb3TfjsxuIe 5pC7CqMxqM9/GB2m85mmNuHeFeICbmw+r8zKnd6LTSpjubSlDCY7t6pz75WiBLmXdp6B XBgEZZ8/62WUuxPzp0y78QWi9+Ayl/qJczZnk4L2SbbZpr67Fy9E5suQ2T3BZCLrSvoV ihDQABmQdcDu+BhhHYTCkshTIVqKJZ7PKvc2NiGoKtoOQHq26HcZCuSTmggVjW7ja7kv xWuQ== X-Forwarded-Encrypted: i=1; AJvYcCUxQ/rdwgdAhZjMvqz8m+G9ki48KpTbxNNexZyoBn/3LVY1lIzwFqBdDXeAc8cO+DbhQHiT3pI=@vger.kernel.org X-Gm-Message-State: AOJu0YwKUcGnIK40DaTKAgZOnQFUUzaG1Kh/9YDgoLjapYFQkpAe0LX+ kQb9/j0PwhxW+R/lVyLEz9lruLww5hJxGR7Jago9BQgRhLRAwn5A X-Gm-Gg: ASbGncsjta55kYqvRWh8mj5Nvd5dTqrhhdAwySt0tPFb3uFPQ1voucSZmU1MAHIV1xu rDNk9ZT/l3xZiz+RMexiSAeoLR5b5zWa3M2KefQn35Jqf6ZtA9cldtWgjajnavqZ8cRb2cBzXoD b6xIO+0Mn8Yjp6nE2N1H7aOYn5IuaU9QeMDilRBMCGOOEOtzlpT2OIO1NXTni3GE/emvLVO9Azp nROfkYI9VZIubunLPQPHOXaMLgsePPoDRbJRqzAFwjBxz7XptoOrJY1wbLswyWESO/lYIKhXSWe iVdNyVnJFCkxAC3e1gaK3q1cXv9CTpHeBlTknzfn8MGoUei6ESRTLt3InGd+3EM= X-Google-Smtp-Source: AGHT+IGXIaOuiQLLXQ5vWEmsXYHmUFb741WLr/m3jayVFh7DZEI8i6JghMWXxCo5yP1kWdv3k3cMWw== X-Received: by 2002:a17:90b:2f8f:b0:2ef:2d9f:8e58 with SMTP id 98e67ed59e1d1-2fc411508famr28198550a91.34.1740036595430; Wed, 19 Feb 2025 23:29:55 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.29.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:29:55 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 01/12] bpf: add networking timestamping support to bpf_get/setsockopt() Date: Thu, 20 Feb 2025 15:29:29 +0800 Message-Id: <20250220072940.99994-2-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net The new SK_BPF_CB_FLAGS and new SK_BPF_CB_TX_TIMESTAMPING are added to bpf_get/setsockopt. The later patches will implement the BPF networking timestamping. The BPF program will use bpf_setsockopt(SK_BPF_CB_FLAGS, SK_BPF_CB_TX_TIMESTAMPING) to enable the BPF networking timestamping on a socket. Signed-off-by: Jason Xing --- include/net/sock.h | 3 +++ include/uapi/linux/bpf.h | 8 ++++++++ net/core/filter.c | 23 +++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 8 ++++++++ 4 files changed, 42 insertions(+) diff --git a/include/net/sock.h b/include/net/sock.h index 8036b3b79cd8..870c3672d9af 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -303,6 +303,7 @@ struct sk_filter; * @sk_stamp: time stamp of last packet received * @sk_stamp_seq: lock for accessing sk_stamp on 32 bit architectures only * @sk_tsflags: SO_TIMESTAMPING flags + * @sk_bpf_cb_flags: used in bpf_setsockopt() * @sk_use_task_frag: allow sk_page_frag() to use current->task_frag. * Sockets that can be used under memory reclaim should * set this to false. @@ -525,6 +526,8 @@ struct sock { u8 sk_txtime_deadline_mode : 1, sk_txtime_report_errors : 1, sk_txtime_unused : 6; +#define SK_BPF_CB_FLAG_TEST(SK, FLAG) ((SK)->sk_bpf_cb_flags & (FLAG)) + u8 sk_bpf_cb_flags; void *sk_user_data; #ifdef CONFIG_SECURITY diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index fff6cdb8d11a..3a2af105fff0 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -6916,6 +6916,13 @@ enum { BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F, }; +/* Definitions for sk_bpf_cb_flags */ +enum { + SK_BPF_CB_TX_TIMESTAMPING = 1<<0, + SK_BPF_CB_MASK = (SK_BPF_CB_TX_TIMESTAMPING - 1) | + SK_BPF_CB_TX_TIMESTAMPING +}; + /* List of known BPF sock_ops operators. * New entries can only be added at the end */ @@ -7094,6 +7101,7 @@ enum { TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */ TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */ TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */ + SK_BPF_CB_FLAGS = 1009, /* Used to set socket bpf flags */ }; enum { diff --git a/net/core/filter.c b/net/core/filter.c index 2ec162dd83c4..1c6c07507a78 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -5222,6 +5222,25 @@ static const struct bpf_func_proto bpf_get_socket_uid_proto = { .arg1_type = ARG_PTR_TO_CTX, }; +static int sk_bpf_set_get_cb_flags(struct sock *sk, char *optval, bool getopt) +{ + u32 sk_bpf_cb_flags; + + if (getopt) { + *(u32 *)optval = sk->sk_bpf_cb_flags; + return 0; + } + + sk_bpf_cb_flags = *(u32 *)optval; + + if (sk_bpf_cb_flags & ~SK_BPF_CB_MASK) + return -EINVAL; + + sk->sk_bpf_cb_flags = sk_bpf_cb_flags; + + return 0; +} + static int sol_socket_sockopt(struct sock *sk, int optname, char *optval, int *optlen, bool getopt) @@ -5238,6 +5257,7 @@ static int sol_socket_sockopt(struct sock *sk, int optname, case SO_MAX_PACING_RATE: case SO_BINDTOIFINDEX: case SO_TXREHASH: + case SK_BPF_CB_FLAGS: if (*optlen != sizeof(int)) return -EINVAL; break; @@ -5247,6 +5267,9 @@ static int sol_socket_sockopt(struct sock *sk, int optname, return -EINVAL; } + if (optname == SK_BPF_CB_FLAGS) + return sk_bpf_set_get_cb_flags(sk, optval, getopt); + if (getopt) { if (optname == SO_BINDTODEVICE) return -EINVAL; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 2acf9b336371..0d4c348e42de 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -6913,6 +6913,13 @@ enum { BPF_SOCK_OPS_ALL_CB_FLAGS = 0x7F, }; +/* Definitions for sk_bpf_cb_flags */ +enum { + SK_BPF_CB_TX_TIMESTAMPING = 1<<0, + SK_BPF_CB_MASK = (SK_BPF_CB_TX_TIMESTAMPING - 1) | + SK_BPF_CB_TX_TIMESTAMPING +}; + /* List of known BPF sock_ops operators. * New entries can only be added at the end */ @@ -7091,6 +7098,7 @@ enum { TCP_BPF_SYN_IP = 1006, /* Copy the IP[46] and TCP header */ TCP_BPF_SYN_MAC = 1007, /* Copy the MAC, IP[46], and TCP header */ TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */ + SK_BPF_CB_FLAGS = 1009, /* Used to set socket bpf flags */ }; enum { From patchwork Thu Feb 20 07:29:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983422 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25ACA1E7660; Thu, 20 Feb 2025 07:30:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036603; cv=none; b=s4lcQqvEZojcMk3oXrqYyXSuh3Q9+w4QiWLEefzr+DmwpixzZuPv7z9VMukgQC76HbPrZKVa6A86KUzfAeoUpIurpRAuBdRID3ZXMG4SWhgaT14keE/aOhSq1aGKG8FcPrf8ywQWirbO08DhPSmncN1lr6ybfx9fA0rblky8+Jk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036603; c=relaxed/simple; bh=fdGpchZoGSlb5vKPY6i549xOdFj4Bj/Z/DVhj+lTEDU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=K67PBgfrqNuXc7yb7igwPZIyoswgH9iGlROXBkdkqU2fBrjqjgxPWTAj9fC3AScUWqoMQjVruSNTcRiE5LlC7uO5wk3dHuCx9fSeYaFjJqpriqECewFcl+lEeMIhCpYG8vX/Qng76/ZmwpfQaFeZxP7qBo+5I5WS8KXTyEb3GMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SucWRgmb; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SucWRgmb" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2211acda7f6so11120605ad.3; Wed, 19 Feb 2025 23:30:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036601; x=1740641401; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pssKYgW/7sA/8gWm1fZ533hOVjuNqc6I9BqF6wMFUD4=; b=SucWRgmbfeQdsnd3SCp/9+0pI8z5ktltxbrYXd5gnUT1tlWUpmdaf+fJPd6Ypc3JuB XRUsAZTdIQZqOKUH7xRSCBtdt5PvjyQJyafAI31pwFeRizpWu+2TiJ1zazIO6onU2ycv KxA60mcV8VE8GYFfIE5f3Z/3Yt7AP5owwZINBqMN/lWufHEF/t2nK0y85cLQXlQ8m0Rr qGnMzpouJWdatRIsyHrT6TAMc86JAHzyD1S6v8fo8R0BrOEIsStpbTcLw+L5clLBMuKz axeL8k/LuwxKi7TMx9GECBIHydAEUy5XlJZ8DzVU3LPg2ckVm8gkJA3U8U8gk44p6KFW BfqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036601; x=1740641401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pssKYgW/7sA/8gWm1fZ533hOVjuNqc6I9BqF6wMFUD4=; b=m5CUYpQAg9I37myKemPOadwl73XtK/+LRBVa27usIVLQidkE4XZSY9LpXK43f51o/q m+g6RaSScCV1QNE2cTvXGbsYuffxPO116M2zASy6hxLXrc3sk9lbd4HrBuGGtoX1bolL /sRdsaVjJRW271dNnXmXDrsnOUZcTBDEiMiw+VhnwqfPRoS7JmRkDjr5KtShMNTpeYdH LQDGCwWiEswuJVqMBBqD/w97nYagpPEEKOWEgH7/vFIUrucbvbcrg7F05ypiwvTUKqmk o3pdKCYqerHriI411jwI4ckkak+rDr6RZlSCASG2f1zMNpMLVr/fyZMYTocMCLNtFH4u 3BLA== X-Forwarded-Encrypted: i=1; AJvYcCVVR7+ZFvCb4saM/1/SyO+KV95KSnB1wWEb5B/qD43frxP3Ph1DgM7NFP925ySYraPvI+fkoDY=@vger.kernel.org X-Gm-Message-State: AOJu0Ywgy6UYnR5nFM/SIEYsqYS77FC00SmTy+N6opXhBbNKrbuQjDtU AgPLKgzgzehB2vYgiauSdIlH9/bEtzpZAn21QAaueXiJhoCLvrdk X-Gm-Gg: ASbGnctci/T4PFwaVxZXy0UeqdJ9ugJiRVcuY3PEWZtzQ9Al5u4gSoApUEIK99TGH6N ALmPLdvGY3bNTqmkmHDFO3cFZqu7h2AV03cmSVverX34ygH7+Ow7Kfjn1OdYJhkx0Lfwpt+XAvd hfvLQZljodcGIzHJLcT2VwgvC3AOZ5W9jgJb7ioQsl9lxbbJgKYXyPzVwWz1riB3bsjCFYn6fhx OveL0ftVfBWWx3dhTu3OJq0L/dJ18Hxc7pxYXZIc/7KTC+sbhrd2/Br8XK2s/ztcQeLpSgKswij eOvo9uv5nbW14V2mqeJITDetyulOQufamwRNtB9w1CIKZH9VdAUOhOjuDSQOs1A= X-Google-Smtp-Source: AGHT+IEoFsdrY68TSvtv3OsJixPWIjk8QRDN5vMaNZ7SM5Pew0tDYfINcWqqksvjl1YGVMAv/Ywaww== X-Received: by 2002:a17:903:2346:b0:220:d81a:bebf with SMTP id d9443c01a7336-2216f43f09bmr106926875ad.0.1740036601390; Wed, 19 Feb 2025 23:30:01 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.29.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:01 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 02/12] bpf: prepare the sock_ops ctx and call bpf prog for TX timestamping Date: Thu, 20 Feb 2025 15:29:30 +0800 Message-Id: <20250220072940.99994-3-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net This patch introduces a new bpf_skops_tx_timestamping() function that prepares the "struct bpf_sock_ops" ctx and then executes the sockops BPF program. The subsequent patch will utilize bpf_skops_tx_timestamping() at the existing TX timestamping kernel callbacks (__sk_tstamp_tx specifically) to call the sockops BPF program. Later, four callback points to report information to user space based on this patch will be introduced. Signed-off-by: Jason Xing --- include/net/sock.h | 7 +++++++ net/core/sock.c | 14 ++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/include/net/sock.h b/include/net/sock.h index 870c3672d9af..9fa27693fb02 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -2923,6 +2923,13 @@ int sock_set_timestamping(struct sock *sk, int optname, struct so_timestamping timestamping); void sock_enable_timestamps(struct sock *sk); +#if defined(CONFIG_CGROUP_BPF) +void bpf_skops_tx_timestamping(struct sock *sk, struct sk_buff *skb, int op); +#else +static inline void bpf_skops_tx_timestamping(struct sock *sk, struct sk_buff *skb, int op) +{ +} +#endif void sock_no_linger(struct sock *sk); void sock_set_keepalive(struct sock *sk); void sock_set_priority(struct sock *sk, u32 priority); diff --git a/net/core/sock.c b/net/core/sock.c index eae2ae70a2e0..bde45569d4da 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -948,6 +948,20 @@ int sock_set_timestamping(struct sock *sk, int optname, return 0; } +#if defined(CONFIG_CGROUP_BPF) +void bpf_skops_tx_timestamping(struct sock *sk, struct sk_buff *skb, int op) +{ + struct bpf_sock_ops_kern sock_ops; + + memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp)); + sock_ops.op = op; + sock_ops.is_fullsock = 1; + sock_ops.sk = sk; + bpf_skops_init_skb(&sock_ops, skb, 0); + __cgroup_bpf_run_filter_sock_ops(sk, &sock_ops, CGROUP_SOCK_OPS); +} +#endif + void sock_set_keepalive(struct sock *sk) { lock_sock(sk); From patchwork Thu Feb 20 07:29:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983423 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 919E21E284C; Thu, 20 Feb 2025 07:30:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036610; cv=none; b=Jb4K4tPW3eTbkzLSZfEiBzGVsUF2Dta4WqDkI85FY8eCuMOeIjcLEdvTT7x3bNuCdsLHrJkTKzjWqlv0ppcGQO835aR71wYdXXjw85dbGVCdwkbDmTaEFeVTOXimfr+sua6XKwvF1KI454J/e3XTN2OuybzEd22+kBjoVjtf8XY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036610; c=relaxed/simple; bh=bKrm1hiTC/nINgvY+o1Fo1gLAQbsMuznKHX0Uw3gAzY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O2nNP0uWZ5JGYmpm+t3iMBhEQdqXJgGT7P0osvI0T+GrXIAFtHsZad55M0ufWIFf4HdzlZyriQVKCpTRpUA4wBq54sKFJoXAmL9lB+kSL48u26vWkl/9mjWIj7dTR+0gLbBbt4bMynW3Mfq3yeTq8/E4b+c66N0eeAv/bMgYVhk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Fh2xQ3mD; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Fh2xQ3mD" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22114b800f7so9743895ad.2; Wed, 19 Feb 2025 23:30:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036608; x=1740641408; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7Alr0Cy5SCB+CgKRZoduXfthZLgwnrJQ/3rmN5eLpq4=; b=Fh2xQ3mDa/MMZjI5EDxwzKddTE4uiGhBf9HqP/5GsPE0dP/ulQszc4sGdSceihTNRf SI+pIzWmsSY60afyTABInnOdaN1yiGlr3d9a9JTX8m4dHcDoFKRUMITrDokrRvWeiRnY LZUztoH6ew97jmQJXVeMlmj8//pP9VSxZK4h6Po28o5K8+HUVXQ3+fsvGsO8BySeokRT N+vkdbIAJwTQDBTyhkQeN9tFUJ84c2ErLQZ3K3LUoXGPVwZWu4eHFj0nb9Z87WWAKApA W2Wo9aHzr3Y1LtBb/44q/3vHPOkbhpAn+346doI7WtaCO7lU3nX66w0RcLWWLhpaXa0M Fw3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036608; x=1740641408; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7Alr0Cy5SCB+CgKRZoduXfthZLgwnrJQ/3rmN5eLpq4=; b=ibUXuKFLoa2Bx/DI7aC1lPqeOtf5u+eWNmEAu9LdNax0KdY4QtlQVx1x7zfutn39g5 TWw1kMNUytzWpvD03kwYFe0EE8g0Y/KT8G5KvSBwaKUhTQjalDP6U6hItelNV50Ryt32 zeqdGpwN/E3pnpKEx5Yu+i5VqiTyvaNYyxTTn1MaTHszi7E7hBrMmeqZNa028b+krtDv HR/5NiFw9lf8jN3d4cF47wY4rRh6qpfnB6E4pL2Iy2hpC8vCBqpiNHuGgK7mnXoctgm3 C18MyDurkG+ui34RMY3M1ubM/KGW7t9vmvqrivadgkekmm4ZTSq0+BhxW5LZ+va8eCJf CtMA== X-Forwarded-Encrypted: i=1; AJvYcCXk2QRqXtLyHqPDEpHt/umJ6FCFnXAX2ce05txbUPUgAQzkBm/fCmraGXdS7kWPaOgtfskgMPU=@vger.kernel.org X-Gm-Message-State: AOJu0YyHUY45l2ON3QPXaUfPpYxdvVLkixu0xH2xPKOIxvFawzwJlgDV FvIrfQkAAIuyc+wS+36B4EUDKtWn9sNIShWjWLnEcKVJzUfRA793 X-Gm-Gg: ASbGnctthoFjMHlN+JjbnFkJ6AAmzzhlN4SzmdYgFFm6/V3wZDmkhNGX1jluq6RaZYw 8spxp6ULB9cVZYQoiaj4UmvTXeUTUqw6SGTrHrO44pcVjVvD1rpCP1BsBpimOTglqVCOMZSjgEJ xwSj+mJ1UVOrpSz+ps7kvLcmqOtPK/eKVbvrRsSNOsuuXx+cdcPkViu14mEfq6Xg+3dzWx9XVpn 7ra0+k7ewrtz22TCjoSYLk9bEds/YG39C/iMx5Nc6/CTIX1tQ6SoC96mQPoU/b0f7rhLUI9lf1X Bqi8Z3XxOfGbVxMef1QFl0GRyjQGCFkACzya4LJ4NY0+7OKe+14KpJWfuh3auyk= X-Google-Smtp-Source: AGHT+IGeuTfgjjKCKKNs1GfIXAWrCK1qBRWUji+A3MTCHLcWC12oGHFwZubGg5tJU0HfOCPFTvgQCw== X-Received: by 2002:a17:902:d2c7:b0:21f:1ae1:dd26 with SMTP id d9443c01a7336-221711d2158mr96571615ad.52.1740036607689; Wed, 19 Feb 2025 23:30:07 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:07 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 03/12] bpf: prevent unsafe access to the sock fields in the BPF timestamping callback Date: Thu, 20 Feb 2025 15:29:31 +0800 Message-Id: <20250220072940.99994-4-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net The subsequent patch will implement BPF TX timestamping. It will call the sockops BPF program without holding the sock lock. This breaks the current assumption that all sock ops programs will hold the sock lock. The sock's fields of the uapi's bpf_sock_ops requires this assumption. To address this, a new "u8 is_locked_tcp_sock;" field is added. This patch sets it in the current sock_ops callbacks. The "is_fullsock" test is then replaced by the "is_locked_tcp_sock" test during sock_ops_convert_ctx_access(). The new TX timestamping callbacks added in the subsequent patch will not have this set. This will prevent unsafe access from the new timestamping callbacks. Potentially, we could allow read-only access. However, this would require identifying which callback is read-safe-only and also requires additional BPF instruction rewrites in the covert_ctx. Since the BPF program can always read everything from a socket (e.g., by using bpf_core_cast), this patch keeps it simple and disables all read and write access to any socket fields through the bpf_sock_ops UAPI from the new TX timestamping callback. Moreover, note that some of the fields in bpf_sock_ops are specific to tcp_sock, and sock_ops currently only supports tcp_sock. In the future, UDP timestamping will be added, which will also break this assumption. The same idea used in this patch will be reused. Considering that the current sock_ops only supports tcp_sock, the variable is named is_locked_"tcp"_sock. Signed-off-by: Jason Xing --- include/linux/filter.h | 1 + include/net/tcp.h | 1 + net/core/filter.c | 8 ++++---- net/ipv4/tcp_input.c | 2 ++ net/ipv4/tcp_output.c | 2 ++ 5 files changed, 10 insertions(+), 4 deletions(-) diff --git a/include/linux/filter.h b/include/linux/filter.h index a3ea46281595..d36d5d5180b1 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1508,6 +1508,7 @@ struct bpf_sock_ops_kern { void *skb_data_end; u8 op; u8 is_fullsock; + u8 is_locked_tcp_sock; u8 remaining_opt_len; u64 temp; /* temp and everything after is not * initialized to 0 before calling diff --git a/include/net/tcp.h b/include/net/tcp.h index 5b2b04835688..4c4dca59352b 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -2649,6 +2649,7 @@ static inline int tcp_call_bpf(struct sock *sk, int op, u32 nargs, u32 *args) memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp)); if (sk_fullsock(sk)) { sock_ops.is_fullsock = 1; + sock_ops.is_locked_tcp_sock = 1; sock_owned_by_me(sk); } diff --git a/net/core/filter.c b/net/core/filter.c index 1c6c07507a78..8631036f6b64 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -10381,10 +10381,10 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type, } \ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF( \ struct bpf_sock_ops_kern, \ - is_fullsock), \ + is_locked_tcp_sock), \ fullsock_reg, si->src_reg, \ offsetof(struct bpf_sock_ops_kern, \ - is_fullsock)); \ + is_locked_tcp_sock)); \ *insn++ = BPF_JMP_IMM(BPF_JEQ, fullsock_reg, 0, jmp); \ if (si->dst_reg == si->src_reg) \ *insn++ = BPF_LDX_MEM(BPF_DW, reg, si->src_reg, \ @@ -10469,10 +10469,10 @@ static u32 sock_ops_convert_ctx_access(enum bpf_access_type type, temp)); \ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF( \ struct bpf_sock_ops_kern, \ - is_fullsock), \ + is_locked_tcp_sock), \ reg, si->dst_reg, \ offsetof(struct bpf_sock_ops_kern, \ - is_fullsock)); \ + is_locked_tcp_sock)); \ *insn++ = BPF_JMP_IMM(BPF_JEQ, reg, 0, 2); \ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF( \ struct bpf_sock_ops_kern, sk),\ diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index eb82e01da911..95733dcdfb4b 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -169,6 +169,7 @@ static void bpf_skops_parse_hdr(struct sock *sk, struct sk_buff *skb) memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp)); sock_ops.op = BPF_SOCK_OPS_PARSE_HDR_OPT_CB; sock_ops.is_fullsock = 1; + sock_ops.is_locked_tcp_sock = 1; sock_ops.sk = sk; bpf_skops_init_skb(&sock_ops, skb, tcp_hdrlen(skb)); @@ -185,6 +186,7 @@ static void bpf_skops_established(struct sock *sk, int bpf_op, memset(&sock_ops, 0, offsetof(struct bpf_sock_ops_kern, temp)); sock_ops.op = bpf_op; sock_ops.is_fullsock = 1; + sock_ops.is_locked_tcp_sock = 1; sock_ops.sk = sk; /* sk with TCP_REPAIR_ON does not have skb in tcp_finish_connect */ if (skb) diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index bc95d2a5924f..a0e779bdbc6b 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -525,6 +525,7 @@ static void bpf_skops_hdr_opt_len(struct sock *sk, struct sk_buff *skb, sock_owned_by_me(sk); sock_ops.is_fullsock = 1; + sock_ops.is_locked_tcp_sock = 1; sock_ops.sk = sk; } @@ -570,6 +571,7 @@ static void bpf_skops_write_hdr_opt(struct sock *sk, struct sk_buff *skb, sock_owned_by_me(sk); sock_ops.is_fullsock = 1; + sock_ops.is_locked_tcp_sock = 1; sock_ops.sk = sk; } From patchwork Thu Feb 20 07:29:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983424 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7185F25760; Thu, 20 Feb 2025 07:30:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036615; cv=none; b=l0mxE02+zSW10/Mj2cG1W9rp5k7+lBTW48Gpwp7ye5zmoFCVZU49NrtcMCvB0GJhcInR0KN+865B2aAFQIdAJr5HqGi5zmsse5UEIMvLYThRCkT3lZmikjHCrWFq2cdazkVtrl/M+MhrB0i6TQoBRVsM7zCXp9+tiaOicBTtRO0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036615; c=relaxed/simple; bh=3f4fr3wEVqOkIsOc+1STd0Ns5iBgLGFkFca2uJbMweM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gYDkAVhww8oX+jNyDep5mh80F2UJFpKcrx5h/oq0aXmhEcydr+KFU8mPXUJ4Ku2zYWRnQX5XPjaSlIszd4BAa39h0cwDioAsI2pJ7Z+ujKRSWkFoLz/PIgJ9Yp/LdPNT4MYfC7CcCd6IkuMpj8juprWe1mizXfgpgYaskOXq0ro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=h8a/h7OZ; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="h8a/h7OZ" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-22100006bc8so9491545ad.0; Wed, 19 Feb 2025 23:30:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036614; x=1740641414; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/jcgdad4TkftRTh1q65WudnE3mEROlme3+Pzdel+ZVs=; b=h8a/h7OZcGbbsOdNJaWxOu952EsyqUasjHkvW2uifUC+CBDslDUd9J6mDzmKd3b1cX iQY39gjo4IT+ITQoNkNfjxvJTYgN9HRqvBiEIj4BNudDXkP0hfDn+/1ke2mRikRYwuVQ t2OZiJlxSqyuhWXNPy5bnyZuYC/czZFnzbGOEk8Zfh1qpzP2JXO2NP6YK3vHUtvpa+ZH D7L5eotoqSgyLCZ9wIR3dVImy3KtkDR+hGmLXXWKK+yFnh2E+hhY6HF0t/kF9Nhhmsjs VH9wxSYExyMn7uFtGmL2QnRnGh58Uy1vGTpQfDF1POq2Vfizl9xIFc4rSwgCp/1LhAtw EOYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036614; x=1740641414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/jcgdad4TkftRTh1q65WudnE3mEROlme3+Pzdel+ZVs=; b=JbDy4PptxyGecn84CERtHDqJah/tBgY0UZNhnZWXAwQQcAK269wTFFzXU0i3GxDrZQ kB5ZWrfWccK9AirZR1Mtpo6IFmUq876upchXVArEwrDz+MDay9t4BlobPqW1tgamMJBS 5v+x5l+E25T5hvUE+bUl5PehAH0/tzJsGW03QMX3DfpQKDAG7oGitc0et0E95OqZ/l1f xmhutFM6plbbZ2vdo9hMsEk1ArB0LigLDbdTv0wUaPrs5WgBnTn7caif7KgqVYDrjLUS 0Z5Np2Cf37k1Au3J+0R4q7T5xstl7CEVJumSQFCT8rvoHiPonk/KfWHh6hycfubLuiOl Hy2A== X-Forwarded-Encrypted: i=1; AJvYcCVOBF3HicEpDn1WSkGz0SzfMLb0oXo2hzGhjyYy0evsrpxE7dRTJV/CZ9zM/471pI6aXtnoVNk=@vger.kernel.org X-Gm-Message-State: AOJu0YyDQBJNG8cRQOJ5Fv26k52G2+1JdJ7kqOgNxeLeMx549CROZlOS XDWWY3a72vOpiRb7ZPp+ck7GP8yKW7/PpyOFj9SvIUrcM8KbkE4O X-Gm-Gg: ASbGncvLM6MxoDpYVmPXZTiLM/K8lQCd7xCKbqKEzsE6+yibxHb8XfLJKLAjmwnHyuv wtHz/WXpzs+e4Z1+NizgSlobnoQ9i60W6Eh6oQNiOQPGi18m1WrcAsO5VuG2qqVFCLZInqCaXPZ 2pAu2vYAJYaOhGcurqBY8QFtLV1GhQqC5W2Y88RCIINvJs17nnu1moLsjXn6G96v0A1rdDguGXl zoD9eEqIU83HPud2JZAecNKd9BPinTfejtcM9Gz41d2D8iNbcVQmdBxhbXrk9VCNq57sFLPShZw /I3w86JFiAe8UhNYPv8JV6e1BsOfxk6s2aP8d8U+KbQEr/fhsB5xi1Vlb4elAhE= X-Google-Smtp-Source: AGHT+IEvDjZK4i9Zp7obJtYIKJ0SIe5pUWNSGsNurKhmAeicUEQHZGzXF7RW+UNqOFBG4ScLVrodww== X-Received: by 2002:a17:903:41c3:b0:21f:7821:55b6 with SMTP id d9443c01a7336-22104030832mr316654115ad.13.1740036613697; Wed, 19 Feb 2025 23:30:13 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:13 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 04/12] bpf: disable unsafe helpers in TX timestamping callbacks Date: Thu, 20 Feb 2025 15:29:32 +0800 Message-Id: <20250220072940.99994-5-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net New TX timestamping sock_ops callbacks will be added in the subsequent patch. Some of the existing BPF helpers will not be safe to be used in the TX timestamping callbacks. The bpf_sock_ops_setsockopt, bpf_sock_ops_getsockopt, and bpf_sock_ops_cb_flags_set require owning the sock lock. TX timestamping callbacks will not own the lock. The bpf_sock_ops_load_hdr_opt needs the skb->data pointing to the TCP header. This will not be true in the TX timestamping callbacks. At the beginning of these helpers, this patch checks the bpf_sock->op to ensure these helpers are used by the existing sock_ops callbacks only. Signed-off-by: Jason Xing --- net/core/filter.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/net/core/filter.c b/net/core/filter.c index 8631036f6b64..7f56d0bbeb00 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -5523,6 +5523,11 @@ static int __bpf_setsockopt(struct sock *sk, int level, int optname, return -EINVAL; } +static bool is_locked_tcp_sock_ops(struct bpf_sock_ops_kern *bpf_sock) +{ + return bpf_sock->op <= BPF_SOCK_OPS_WRITE_HDR_OPT_CB; +} + static int _bpf_setsockopt(struct sock *sk, int level, int optname, char *optval, int optlen) { @@ -5673,6 +5678,9 @@ static const struct bpf_func_proto bpf_sock_addr_getsockopt_proto = { BPF_CALL_5(bpf_sock_ops_setsockopt, struct bpf_sock_ops_kern *, bpf_sock, int, level, int, optname, char *, optval, int, optlen) { + if (!is_locked_tcp_sock_ops(bpf_sock)) + return -EOPNOTSUPP; + return _bpf_setsockopt(bpf_sock->sk, level, optname, optval, optlen); } @@ -5758,6 +5766,9 @@ static int bpf_sock_ops_get_syn(struct bpf_sock_ops_kern *bpf_sock, BPF_CALL_5(bpf_sock_ops_getsockopt, struct bpf_sock_ops_kern *, bpf_sock, int, level, int, optname, char *, optval, int, optlen) { + if (!is_locked_tcp_sock_ops(bpf_sock)) + return -EOPNOTSUPP; + if (IS_ENABLED(CONFIG_INET) && level == SOL_TCP && optname >= TCP_BPF_SYN && optname <= TCP_BPF_SYN_MAC) { int ret, copy_len = 0; @@ -5800,6 +5811,9 @@ BPF_CALL_2(bpf_sock_ops_cb_flags_set, struct bpf_sock_ops_kern *, bpf_sock, struct sock *sk = bpf_sock->sk; int val = argval & BPF_SOCK_OPS_ALL_CB_FLAGS; + if (!is_locked_tcp_sock_ops(bpf_sock)) + return -EOPNOTSUPP; + if (!IS_ENABLED(CONFIG_INET) || !sk_fullsock(sk)) return -EINVAL; @@ -7609,6 +7623,9 @@ BPF_CALL_4(bpf_sock_ops_load_hdr_opt, struct bpf_sock_ops_kern *, bpf_sock, u8 search_kind, search_len, copy_len, magic_len; int ret; + if (!is_locked_tcp_sock_ops(bpf_sock)) + return -EOPNOTSUPP; + /* 2 byte is the minimal option len except TCPOPT_NOP and * TCPOPT_EOL which are useless for the bpf prog to learn * and this helper disallow loading them also. From patchwork Thu Feb 20 07:29:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983425 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88093198E81; Thu, 20 Feb 2025 07:30:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036622; cv=none; b=Zm3ItEZqwHZ0wpJlRH4hQ9lBVaSfnfDsfblEsP3V14Io/BtXu49qDWBF1KHgs0ByXON5KXyRoeySaAJXr6MRW14bXCrfceGDPxlrcKHsVbjNpQbzs3owlPznG53gelkV0mTZkywR6LuJs+l0pukt7gVBPRsqnJxKCAxqa4gJfNw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036622; c=relaxed/simple; bh=QnkYLGR0o0oh4JHZ1jUcf0VUu0sG49ZtxlYMSdcro+0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RKfkZ/BnMtLoMsTJv1epNgiNOPbqQie65OZM0THNM1X308DiQuAhYjBNWXtuU4Da+krnVBQf7ai2wJiy4CgYAqu6YdZXxR8iBXUeKOycjOXL1A/tm7YzDeKIK5LYsFjq9Bu7ZCF6spHdksrkW0uU4F9f81/VWntbH9HaVnIMx2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WY+Mr/+J; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WY+Mr/+J" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-220d601886fso8203935ad.1; Wed, 19 Feb 2025 23:30:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036620; x=1740641420; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=B65KFV6AD6yU5phhSrRrO9qdMMHj6DYghKCH3Cqv1DY=; b=WY+Mr/+J2/n3g1xMB040xzEtOgtQU10hb0918fwSRSCmHPgNkIvL4yJUnOs6a+CqWc 0HdZTuLHboKjfK7cbb4M2P46Qh5xhmSCPbKo+vX4mTWx9hDJhMWhsdo+yIEjKA8BBSPQ KWAW17ihB/k7HAst21noT+sEox8dkWFsWlqYJcejOP+IpnGgMDrKqVLUSCiEKnFb6+UO x8RywcfGaKebgvirX45Px2f/1YveKk0wl0ulYTxF6mHTFoAOen84odU1Vqzalcb+nV6t IwPNJ5dn5PX3MUekEbNfj42CwDsNDWI5YK9mJ4lluJtcJ/7v4vJj3rlTwYWle9nuzuPP F++g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036620; x=1740641420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=B65KFV6AD6yU5phhSrRrO9qdMMHj6DYghKCH3Cqv1DY=; b=vmRWBYfkqlq8FkVvmsAZVnG67aIXXD6PrUTtPdROyePVN0UOvZcw4b5lE+02xch0Vb 8c+cEPg/PsBco5JkI8LvrBhdp7flFpNUr7TLJLMuaLjDiyF6MAvLO3nfMAl5fLDXvGXd 7xF4w0lRB/oasIECpb2yBd0k3fv63sZcwdtt/Wh2/k6jNoaRgvE69mhoQwD843xlDxTy 68b0ZXNSmUR2m6EyX0nwVM6Z0ybxKo/2YABaX5cTqkQxcTV34aWq78wVYROMendQ+lzr AoTBkCsI+LqSYjZ2cYOLAVoMI9y/0Fbib0qKfwgbCnfNHAfE4F2YWwLE66ItMOZnxwz8 kMzA== X-Forwarded-Encrypted: i=1; AJvYcCV9/byWs/cydGlHNEs5giwPBnveeiHO00Ibwdzuk7/X/xSVpEA83fsUgKNz30JOzj2EVWTr6qM=@vger.kernel.org X-Gm-Message-State: AOJu0YwW3xTub4CPWHQMjWsPiC/qasFFpcb0VgzjBE1l1m3MBFCWixGg NBVnez4eIGXerVISpVAWs+cnPeg72Wj8WqAxgnw1/Zv69wk8YO2L X-Gm-Gg: ASbGncvuDw/IDYmr1ZxpgWRST1GcOMl6fnYhnIvel2k5YTsmy8+gYGJ+Dufrz1HjZaH VQZx/lFqxtPmTaqI8jK0CxZl7RAFLZtMg6X68wb8Amvr5NB/GDXXB2OWr426HyfbaDbW+yYBajt +pZPjAf3UWH+eZw0cpX5uA8yI5ACeDLgqqT7QJVV4jsxdcLdVmQYWOpe52WfpJ8OjtH5cyIDcCy TNapWjrGalBcK53HmFDMdbFMPpuBMqYoa4C7O4OEbgazbDXrHdTmKMgO3Y6csDm8QwWmp2xWiQt 4y/5vztw6OY86Cv+uIS7Zuw2c//9Lb28GZy3aVNVfWbAvFeqJCoDB3uOZM00hds= X-Google-Smtp-Source: AGHT+IG+B3MQ90cGIGbgwYLS6Kz77P4yW6P4p/ivaBXxM71emFDscyQWaa2GEeYItFHL/1zKxnshgg== X-Received: by 2002:a17:902:ea11:b0:220:ea7a:8e63 with SMTP id d9443c01a7336-22103f19fc8mr324133045ad.17.1740036619688; Wed, 19 Feb 2025 23:30:19 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:19 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 05/12] net-timestamp: prepare for isolating two modes of SO_TIMESTAMPING Date: Thu, 20 Feb 2025 15:29:33 +0800 Message-Id: <20250220072940.99994-6-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net No functional changes here. Only add test to see if the orig_skb matches the usage of application SO_TIMESTAMPING. In this series, bpf timestamping and previous socket timestamping are implemented in the same function __skb_tstamp_tx(). To test the socket enables socket timestamping feature, this function skb_tstamp_tx_report_so_timestamping() is added. In the next patch, another check for bpf timestamping feature will be introduced just like the above report function, namely, skb_tstamp_tx_report_bpf_timestamping(). Then users will be able to know the socket enables either or both of features. Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- net/core/skbuff.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index a441613a1e6c..341a3290e898 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5539,6 +5539,23 @@ void skb_complete_tx_timestamp(struct sk_buff *skb, } EXPORT_SYMBOL_GPL(skb_complete_tx_timestamp); +static bool skb_tstamp_tx_report_so_timestamping(struct sk_buff *skb, + struct skb_shared_hwtstamps *hwtstamps, + int tstype) +{ + switch (tstype) { + case SCM_TSTAMP_SCHED: + return skb_shinfo(skb)->tx_flags & SKBTX_SCHED_TSTAMP; + case SCM_TSTAMP_SND: + return skb_shinfo(skb)->tx_flags & (hwtstamps ? SKBTX_HW_TSTAMP : + SKBTX_SW_TSTAMP); + case SCM_TSTAMP_ACK: + return TCP_SKB_CB(skb)->txstamp_ack; + } + + return false; +} + void __skb_tstamp_tx(struct sk_buff *orig_skb, const struct sk_buff *ack_skb, struct skb_shared_hwtstamps *hwtstamps, @@ -5551,6 +5568,9 @@ void __skb_tstamp_tx(struct sk_buff *orig_skb, if (!sk) return; + if (!skb_tstamp_tx_report_so_timestamping(orig_skb, hwtstamps, tstype)) + return; + tsflags = READ_ONCE(sk->sk_tsflags); if (!hwtstamps && !(tsflags & SOF_TIMESTAMPING_OPT_TX_SWHW) && skb_shinfo(orig_skb)->tx_flags & SKBTX_IN_PROGRESS) From patchwork Thu Feb 20 07:29:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983426 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B14CF1B4236; Thu, 20 Feb 2025 07:30:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036628; cv=none; b=ULpJoIv2INryiE4Yf+jbUG5H17b0GZphZ5tOrM6vexTgCWan6hzwHtq+BbOqyZivN0a89/g6jwVmTTjFL4slQQnzUvvFdR2h7ufwdZKkOTHbc/SAJd1zjflEjNyIShxeq6BHDOA2NoPzaW7Hz9JvnjW/XDUzJ/coG1GrUO2NnLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036628; c=relaxed/simple; bh=6uYVTz2gJSY8H3PtNKoONElW/xXfMWaFm2pm9IGwdRo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ewdceeSjl1a4PRogJrX312flBh12usloABzgFEhmBJVtxa0fHvQpxwfSomql6zOqv7Xdj7IgTNP6vtu56ChoK2ioyIl8tonlHvctCGI/W7ck894FxqbMD5QyOnpFOsToWLPW+rSbTN8F4O8OFGWEkeuM8M5XRPit2rBRVizzON8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=c/m98moO; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c/m98moO" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21c2f1b610dso15824825ad.0; Wed, 19 Feb 2025 23:30:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036626; x=1740641426; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FUQLHRT/pqNOpAj8Sq4vgFRFXpsqaM61vdZmkVHjc3E=; b=c/m98moO9SmG9otDf7osDP/dMifqv3JDePea2RwT99QcSfAUFAS4S3HSHyTn1j2rf7 y761RI7JegcyEqbrOA3SJr7ltvCWT+PGMi5RXOC0mLFCu8zTXmIrtifQgQHAxyKTALgw kLbSAWsjvVrQV9v88KysPwcbs9RsQsKq5xDMR62EnKCoQ1VXR8uF8hfo/wGhsmGvDwNS HXktLhJXRZlH07Z1gh9q637j7JvpyRgMj3xK/jhyYVenW3jomz3nhw9n6wwKzaJC1vB3 jtDqZSF2LFq8IGaP66nI5Qjc01Xmz+aTJNGRpq+S/RZjR+PWb/emgECPB4KNCsvexcGL 6Dqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036626; x=1740641426; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FUQLHRT/pqNOpAj8Sq4vgFRFXpsqaM61vdZmkVHjc3E=; b=UvyusxRQiHfxYcstCdMIUTrGNriLqMSxEHS+RTQ4DPiArD4J6xHX1BRqSwpqFuuewh io70WZEmSve5pGppjAP5ybF5zgoxouU/cOpq5JPZ+UVCzfZpb0/WnsFBssrOhem1QzlP x6Ws3sCCIXsxFqayjXsxzKB0rBvOKtTv4ua/Xi5QfILbxQIcui/CnphoMk0lV7f0z/gd VNsReujhh2FK/gJCCsih5fRU530u/wRW6yt/GyycZTXR4tY7ogMjqKLs7f/guf26VN0B dlY6HJvk0VdbnBo59r51QcWAnHzvl7TbI7I2IDX+Xjzl7IDZYJgqq3UAY6IoUmnq0X1g nKTQ== X-Forwarded-Encrypted: i=1; AJvYcCUazrYc+GnlIwGVCusTXljrmMSwfo4XfZ9CcLXRRucqyO1IpaGN0bjvbamBAL+bXkIJa35R8J8=@vger.kernel.org X-Gm-Message-State: AOJu0YzfLDLJTRGQwCG7nSUExcS4oTRzHmjmXleo1X1GExRlZDrvKdei DJOiOVCYI4yS8tid91xRuaLpTJ1+riizcoGhRX+FW80K3Mr+yKW0 X-Gm-Gg: ASbGncsCJKBqIv2IEg+aJw5QcA+8m5npx1HhxSE2w8sQnSRIXPVnXJuCIvXuo214tRz 9qQ2jA0KbKMcS7BnTwKDdRDvFVcCPLIfvWdUpskxIx4tfj9z2+c33b9dNBGkjBkN4FLQxLmat93 sYKAOVP3MJF31MWHW5Mv/roBH6ZwEbhLZtdatxSjaIkFq8z8hcYc/kI4wkKofttWs+SF/PrMZH8 pV7Ngvqob9Jt82LZqKNggFHgpPY1afcO3Ksr1dPdA6nlxhEHIkVPjuU0+pAv8yPECRAJgkw35V8 gfJA8rikFHYGjiISQcbYpJgqZK4wjTcujDKqNUocmvvE250R9hVSZwnHxhi3pzA= X-Google-Smtp-Source: AGHT+IFUMB9Gshc/8cToe6h+rCcQ4EFwQ8+HzBBlcaiokstUia1FqFJRwZ+BSP14Xju5P2BqSj4lkw== X-Received: by 2002:a17:903:1cc:b0:21f:3e2d:7d36 with SMTP id d9443c01a7336-22103efebf9mr370657385ad.1.1740036625809; Wed, 19 Feb 2025 23:30:25 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:25 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 06/12] bpf: add BPF_SOCK_OPS_TSTAMP_SCHED_CB callback Date: Thu, 20 Feb 2025 15:29:34 +0800 Message-Id: <20250220072940.99994-7-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Support SCM_TSTAMP_SCHED case for bpf timestamping. Add a new sock_ops callback, BPF_SOCK_OPS_TSTAMP_SCHED_CB. This callback will occur at the same timestamping point as the user space's SCM_TSTAMP_SCHED. The BPF program can use it to get the same SCM_TSTAMP_SCHED timestamp without modifying the user-space application. A new SKBTX_BPF flag is added to mark skb_shinfo(skb)->tx_flags, ensuring that the new BPF timestamping and the current user space's SO_TIMESTAMPING do not interfere with each other. Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- include/linux/skbuff.h | 6 +++++- include/uapi/linux/bpf.h | 5 +++++ net/core/dev.c | 3 ++- net/core/skbuff.c | 20 ++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 5 +++++ 5 files changed, 37 insertions(+), 2 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index bb2b751d274a..52f6e033e704 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -489,10 +489,14 @@ enum { /* generate software time stamp when entering packet scheduling */ SKBTX_SCHED_TSTAMP = 1 << 6, + + /* used for bpf extension when a bpf program is loaded */ + SKBTX_BPF = 1 << 7, }; #define SKBTX_ANY_SW_TSTAMP (SKBTX_SW_TSTAMP | \ - SKBTX_SCHED_TSTAMP) + SKBTX_SCHED_TSTAMP | \ + SKBTX_BPF) #define SKBTX_ANY_TSTAMP (SKBTX_HW_TSTAMP | \ SKBTX_HW_TSTAMP_USE_CYCLES | \ SKBTX_ANY_SW_TSTAMP) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 3a2af105fff0..7bcd8b955598 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7035,6 +7035,11 @@ enum { * by the kernel or the * earlier bpf-progs. */ + BPF_SOCK_OPS_TSTAMP_SCHED_CB, /* Called when skb is passing + * through dev layer when + * SK_BPF_CB_TX_TIMESTAMPING + * feature is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect diff --git a/net/core/dev.c b/net/core/dev.c index c0021cbd28fc..cbbde68c17cb 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -4500,7 +4500,8 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) skb_reset_mac_header(skb); skb_assert_len(skb); - if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_SCHED_TSTAMP)) + if (unlikely(skb_shinfo(skb)->tx_flags & + (SKBTX_SCHED_TSTAMP | SKBTX_BPF))) __skb_tstamp_tx(skb, NULL, NULL, skb->sk, SCM_TSTAMP_SCHED); /* Disable soft irqs for various locks below. Also diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 341a3290e898..3206f7708974 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5556,6 +5556,23 @@ static bool skb_tstamp_tx_report_so_timestamping(struct sk_buff *skb, return false; } +static void skb_tstamp_tx_report_bpf_timestamping(struct sk_buff *skb, + struct sock *sk, + int tstype) +{ + int op; + + switch (tstype) { + case SCM_TSTAMP_SCHED: + op = BPF_SOCK_OPS_TSTAMP_SCHED_CB; + break; + default: + return; + } + + bpf_skops_tx_timestamping(sk, skb, op); +} + void __skb_tstamp_tx(struct sk_buff *orig_skb, const struct sk_buff *ack_skb, struct skb_shared_hwtstamps *hwtstamps, @@ -5568,6 +5585,9 @@ void __skb_tstamp_tx(struct sk_buff *orig_skb, if (!sk) return; + if (skb_shinfo(orig_skb)->tx_flags & SKBTX_BPF) + skb_tstamp_tx_report_bpf_timestamping(orig_skb, sk, tstype); + if (!skb_tstamp_tx_report_so_timestamping(orig_skb, hwtstamps, tstype)) return; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 0d4c348e42de..c3b950325846 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7032,6 +7032,11 @@ enum { * by the kernel or the * earlier bpf-progs. */ + BPF_SOCK_OPS_TSTAMP_SCHED_CB, /* Called when skb is passing + * through dev layer when + * SK_BPF_CB_TX_TIMESTAMPING + * feature is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect From patchwork Thu Feb 20 07:29:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983427 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 988FB198E81; Thu, 20 Feb 2025 07:30:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036634; cv=none; b=vBWZ5W1BkGPVN8P8PLvJIyBlB0ogK0QXKtZsjeYsdau87FXioxUsl9d4P78Dv6Y0RvBwnADvl3566z6HAheSMvCDMpRryosuF/dvAJaEluF01L4ILQdfj9K0t9WqQrFkRgHZcCSzOR9Js7dzBsQ+H30poKgNaXSHG1jsS0u+jiY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036634; c=relaxed/simple; bh=hroW5I6Dr6LnEAbVDw2T0BlqrZU5dRCIZeqE11C5wm4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LJeMalp4YXQGPHapb7h89CYHMpvaokHUkaozLzU7QMzixqIeTaFNQSZa26MKGFvb6jR0PAJfDxqzIohufRuaxa4jluDFOrMpwi+pVHTqqQIlgDPSwJ7U2VhtPfFDYJMhIxDzv4LYOxo48gHe2+ZFr4WzUaz91kvoGhuzV/su83w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hmOvdyZK; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hmOvdyZK" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2fc33aef343so1290476a91.1; Wed, 19 Feb 2025 23:30:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036632; x=1740641432; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hVcwCmBIPrNKOU3UyL7A8k6pR2axRK5QWFDs8fBPHiE=; b=hmOvdyZKM1oMvqXFWj6tD3oF9Dgq1lWuBuhPtnIafmcMJbZPsF7oX5V9pG/o1UTIHw 7mfjzH5/dGXSx2MYwPm3BSv706rTP2DUyP1QNQheN66XE259lTJUL5huJS5B2qydBl7h szX2y771rDcfc0WrHDX9/wQNWyJOM2WSu3meKLIvJyXWd3JTRleeq5jk+KsePmX1nL0r N8ch9FWxvb0YyKFpe0rUgSbWy5t0AIwhTUawxlApkfP6QzZTZSJyz8c1p0qa1H5K7myH bFs1HfIuQnVXUlDr7HCa0WPltLpfbyUO6gzGLs9HfRj47AJbKxAn4jXDGi8pOWGEepjh /U0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036632; x=1740641432; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hVcwCmBIPrNKOU3UyL7A8k6pR2axRK5QWFDs8fBPHiE=; b=wYOsU2F8fEum33JpDAeBV9RGnRscwhJ3QNeayNSvm6Usctx68ayoJi1+7zntahJwad eCn7F9miCMh7ZqDOGnARHp4fGYk1BIwaGyG6m5AZKb5YhRP0Zcwl3Sb85foaaqoCiIAm iJu/Lz/8oLhfMX3TDswqJx/Y6BUfh47VTjiRh+eTFNysl+8fb3tp5E9HiYe5jqdpDVkR 64Sqr73dK1XMIxfiaRZsnqyhSAu0SGlZetqRyaMdgheMWi4RwanS4z4arAm7YVCr1/4f f0gkYpzy0WmaZG1f+W4fQvLEhnqXzI+d0yPowoJUyKGsiWvKD8ZQGcNgWo8D1egDD31Z SNAA== X-Forwarded-Encrypted: i=1; AJvYcCXk14YKoIZHMh8RYLudwrEVzPRua1Jgcbqa+g1ZtpkHNDYkI0htwU8C0ooumrLrQqsSV8alN4k=@vger.kernel.org X-Gm-Message-State: AOJu0Yz0ZOQFmpDEoVElVpAt4pVDZqrKNL6/N+GuW1zUaXtd9JoNYtGO 73rpI9iZtGcd9L4/Pococ32YOktjV5+5EjUVoBret0GJfbNiNh0d X-Gm-Gg: ASbGnctlgmMhCzfjH0e5PJy4o3Z5lALa1Aq82jUXLJ/B3Igq6qcXqs7Di44yJSCgEhg Kzmf7UHyZmtj4dT2HTyGJE1O10u3BYEzQfWA9KzfiPwQy2ixPOGF89cmtjEdcBXbtW+xLzvf5qR XyNSMZOeZnsF59K8cxmseUuP7DjpYRLoNVlMC2JsjzNNSFwdQOfQU/8YJc29hwL/BOr6A7U8oVY NoYDWLq2qXYfhLzkjCRwMZQk3QVx2WRmardbvHz6K4csm+2FRs0oOTC4dDaQj3Q9zLX773LB7Dg kNTUJUwIm2rK4RjyD7JeFoV1m8VfC1c/Vl+WTQy1ue5Y/60o7itmNkhchNsHaHE= X-Google-Smtp-Source: AGHT+IE9FIIr2DXQYA8MCCXBY5uVqKIwLAGMfkf7tCP5wSoArZGznxJQd6PiBbROGMYNyAm/5h8KVA== X-Received: by 2002:a17:90b:278b:b0:2f2:8bdd:cd8b with SMTP id 98e67ed59e1d1-2fcb5ac05f1mr10591700a91.29.1740036631957; Wed, 19 Feb 2025 23:30:31 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:31 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 07/12] bpf: add BPF_SOCK_OPS_TSTAMP_SND_SW_CB callback Date: Thu, 20 Feb 2025 15:29:35 +0800 Message-Id: <20250220072940.99994-8-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Support sw SCM_TSTAMP_SND case for bpf timestamping. Add a new sock_ops callback, BPF_SOCK_OPS_TSTAMP_SND_SW_CB. This callback will occur at the same timestamping point as the user space's software SCM_TSTAMP_SND. The BPF program can use it to get the same SCM_TSTAMP_SND timestamp without modifying the user-space application. Based on this patch, BPF program will get the software timestamp when the driver is ready to send the skb. In the sebsequent patch, the hardware timestamp will be supported. Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- include/linux/skbuff.h | 2 +- include/uapi/linux/bpf.h | 4 ++++ net/core/skbuff.c | 9 ++++++++- tools/include/uapi/linux/bpf.h | 4 ++++ 4 files changed, 17 insertions(+), 2 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 52f6e033e704..76582500c5ea 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -4568,7 +4568,7 @@ void skb_tstamp_tx(struct sk_buff *orig_skb, static inline void skb_tx_timestamp(struct sk_buff *skb) { skb_clone_tx_timestamp(skb); - if (skb_shinfo(skb)->tx_flags & SKBTX_SW_TSTAMP) + if (skb_shinfo(skb)->tx_flags & (SKBTX_SW_TSTAMP | SKBTX_BPF)) skb_tstamp_tx(skb, NULL); } diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 7bcd8b955598..59adcef3326a 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7040,6 +7040,10 @@ enum { * SK_BPF_CB_TX_TIMESTAMPING * feature is on. */ + BPF_SOCK_OPS_TSTAMP_SND_SW_CB, /* Called when skb is about to send + * to the nic when SK_BPF_CB_TX_TIMESTAMPING + * feature is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 3206f7708974..308db7dae1ab 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5557,6 +5557,7 @@ static bool skb_tstamp_tx_report_so_timestamping(struct sk_buff *skb, } static void skb_tstamp_tx_report_bpf_timestamping(struct sk_buff *skb, + struct skb_shared_hwtstamps *hwtstamps, struct sock *sk, int tstype) { @@ -5566,6 +5567,11 @@ static void skb_tstamp_tx_report_bpf_timestamping(struct sk_buff *skb, case SCM_TSTAMP_SCHED: op = BPF_SOCK_OPS_TSTAMP_SCHED_CB; break; + case SCM_TSTAMP_SND: + if (hwtstamps) + return; + op = BPF_SOCK_OPS_TSTAMP_SND_SW_CB; + break; default: return; } @@ -5586,7 +5592,8 @@ void __skb_tstamp_tx(struct sk_buff *orig_skb, return; if (skb_shinfo(orig_skb)->tx_flags & SKBTX_BPF) - skb_tstamp_tx_report_bpf_timestamping(orig_skb, sk, tstype); + skb_tstamp_tx_report_bpf_timestamping(orig_skb, hwtstamps, + sk, tstype); if (!skb_tstamp_tx_report_so_timestamping(orig_skb, hwtstamps, tstype)) return; diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index c3b950325846..1ead1e9d31be 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7037,6 +7037,10 @@ enum { * SK_BPF_CB_TX_TIMESTAMPING * feature is on. */ + BPF_SOCK_OPS_TSTAMP_SND_SW_CB, /* Called when skb is about to send + * to the nic when SK_BPF_CB_TX_TIMESTAMPING + * feature is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect From patchwork Thu Feb 20 07:29:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983428 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA1911E7C08; Thu, 20 Feb 2025 07:30:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036640; cv=none; b=nFRSrdyaOGgPXOmW4CXxkjrZ4vL415SXS/qosyCGa6nxF0ptJFHS1V7SVGvkeiTDjQ05hMQSLSScdArV9iOoWkN9T0mzBNvbkiCO8vCRPzbiSupsxY0fvrQNiFiDRDpDgglWv1rDyofO9OgUxvyoHrMWVIJvQC51qV9tAfNSJ2w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036640; c=relaxed/simple; bh=BL6jk5+WNhi/FA/CAS87bIAkls1+bzgDfR+0WSaUfuw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nONl7OtgjoH6b9vzy5MLKZiX+E3oevFrYTQq/N5CknUnj12HRe+ac3dYqhMQeAmvUEEBxkaND7VRNf9cpupG8lRxK3oMku/6ogRcy6zzi7eAngEogLqWrNrGCkybcHa9Xo949xhTY9PXSHBhF8Qg7LETAUoP9fy2j7I8gFg9kGg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dxge8Mi1; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dxge8Mi1" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2211cd4463cso10482455ad.2; Wed, 19 Feb 2025 23:30:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036638; x=1740641438; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g4qUx08LGWRnLuJ7bCXGD0v8j0A90dZsifhXe0rQHuM=; b=dxge8Mi1y8l8xbSBF6HeHs/n1Bil7EN6FgL1LPNjk5XTA7d3L6yldksRNcpoLfJoLU 38bXIvAm7LskQV79x5UymQFspsrPFZqhERQYdAMcMwojJTRNj9kg0VkGVsF6A2TlD9fF +lsI0u5JnJXHEjIjAPdVcyFEzLLk8XM+PnkfLcKKL3uWln+GdK5biZkbxevpUwWrrNzC pG5A+4B5ctKJb3zCxxbkYDY7F24zEjTDkevNMy4aEaUZvXBj3gaN/nDUIfwz9D5xMaPL rrqO2fejkErRfuFJKzsEGLMfNxeMXq8LLjBOw4475ImFJWzXbImo4Ny877fBlZArOZiH ws0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036638; x=1740641438; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g4qUx08LGWRnLuJ7bCXGD0v8j0A90dZsifhXe0rQHuM=; b=Cn+3rcp1/8GUpdaJFG1QUGp9keZdvJq6tm+bBszc5ZFqfBNfj1tIVTMnzIhejTRTlx VSZctmuUet7VmTlKctfGA3xu3zeDtkm1Rctg952NoTZldK62ya0NyiSlKGcqVF0cYx1v 1V4dtdYBX0pisl/Ruu3mjXmstp90husDngfzjcUBYDT11fCCbZoCGjRWtHTx9c0OlSrq nF5qoBehILhWvi7x+OQ/YO1w2oxkp8OmcQGSHnQ08yIODc3q+sBqL+mjFjUho4DEU19N 0nMHnkv4zbhSVuWSYk4bA2y82yCO4V51uyAki9Mbx6nT20C1mRaDnfILvXyscAuJxCtc Qr7g== X-Forwarded-Encrypted: i=1; AJvYcCUbxB2dZmC6j3UfvhzBVsoPGlnHMiL2//Tikgvf1WLunR4oFVnx2qYJg2NkPlxWh+mpLmgBLsU=@vger.kernel.org X-Gm-Message-State: AOJu0Yzz5IfxaJKVG1XcyJPNNv71gaz/FVpvdpPm81qQR6nOrJ70mizS qsrS5Lmjd+TcYbE/7iI144Prm/T457em2ssMnv0VMq1GnMKyM7ay X-Gm-Gg: ASbGnctKKTidQTAHVCgC6RhSJaKYllwud5tS9bvBOSixWkU2YWJ73ZEz/bPOVc4au9H OLkSc7GVr7niNPODFKJV1ACzqqG3u+oOZUt0eaSUXH2bj6zvQw/XdzxQ2Xt2XS4BzlooHonCLr6 rehFQTHC63W+vKwQ0/iEaNKbF5mI5k1IvHkiZiGoLoxdYYt9eCsemsKE6jACUAlX95J1cmYCd5T otsU3abNoA8e6N8Yq04ECiIYNvf4gyhEss/RnyZIqXT4rBDej3Z6D2CgdYc2gMzW3g7UBWoU+t4 pHoA8uVwSI2wyZWDd1ULfAv2+VgzbU6/1/SNOeFh8n/jJlG6HOU4v14Y+qeckl8= X-Google-Smtp-Source: AGHT+IEzXxOBkzwYOrCDoAlQ/aLNhemU3u6aSEHyKmgBoYUsVxkrFP63VkWpzmibJsAhL1E26OA6QQ== X-Received: by 2002:a17:903:94e:b0:21c:fb6:7c3c with SMTP id d9443c01a7336-22104034f86mr352074285ad.17.1740036638131; Wed, 19 Feb 2025 23:30:38 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:37 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 08/12] bpf: add BPF_SOCK_OPS_TSTAMP_SND_HW_CB callback Date: Thu, 20 Feb 2025 15:29:36 +0800 Message-Id: <20250220072940.99994-9-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Support hw SCM_TSTAMP_SND case for bpf timestamping. Add a new sock_ops callback, BPF_SOCK_OPS_TSTAMP_SND_HW_CB. This callback will occur at the same timestamping point as the user space's hardware SCM_TSTAMP_SND. The BPF program can use it to get the same SCM_TSTAMP_SND timestamp without modifying the user-space application. To avoid increasing the code complexity, replace SKBTX_HW_TSTAMP with SKBTX_HW_TSTAMP_NOBPF instead of changing numerous callers from driver side using SKBTX_HW_TSTAMP. The new definition of SKBTX_HW_TSTAMP means the combination tests of socket timestamping and bpf timestamping. After this patch, drivers can work under the bpf timestamping. Considering some drivers don't assign the skb with hardware timestamp, this patch does the assignment and then BPF program can acquire the hwstamp from skb directly. Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- include/linux/skbuff.h | 4 +++- include/uapi/linux/bpf.h | 4 ++++ net/core/skbuff.c | 11 +++++++---- net/dsa/user.c | 2 +- net/socket.c | 2 +- tools/include/uapi/linux/bpf.h | 4 ++++ 6 files changed, 20 insertions(+), 7 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 76582500c5ea..0b4f1889500d 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -470,7 +470,7 @@ struct skb_shared_hwtstamps { /* Definitions for tx_flags in struct skb_shared_info */ enum { /* generate hardware time stamp */ - SKBTX_HW_TSTAMP = 1 << 0, + SKBTX_HW_TSTAMP_NOBPF = 1 << 0, /* generate software time stamp when queueing packet to NIC */ SKBTX_SW_TSTAMP = 1 << 1, @@ -494,6 +494,8 @@ enum { SKBTX_BPF = 1 << 7, }; +#define SKBTX_HW_TSTAMP (SKBTX_HW_TSTAMP_NOBPF | SKBTX_BPF) + #define SKBTX_ANY_SW_TSTAMP (SKBTX_SW_TSTAMP | \ SKBTX_SCHED_TSTAMP | \ SKBTX_BPF) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 59adcef3326a..4ec1a86288ef 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7044,6 +7044,10 @@ enum { * to the nic when SK_BPF_CB_TX_TIMESTAMPING * feature is on. */ + BPF_SOCK_OPS_TSTAMP_SND_HW_CB, /* Called in hardware phase when + * SK_BPF_CB_TX_TIMESTAMPING feature + * is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 308db7dae1ab..77b8866f94c5 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5547,7 +5547,7 @@ static bool skb_tstamp_tx_report_so_timestamping(struct sk_buff *skb, case SCM_TSTAMP_SCHED: return skb_shinfo(skb)->tx_flags & SKBTX_SCHED_TSTAMP; case SCM_TSTAMP_SND: - return skb_shinfo(skb)->tx_flags & (hwtstamps ? SKBTX_HW_TSTAMP : + return skb_shinfo(skb)->tx_flags & (hwtstamps ? SKBTX_HW_TSTAMP_NOBPF : SKBTX_SW_TSTAMP); case SCM_TSTAMP_ACK: return TCP_SKB_CB(skb)->txstamp_ack; @@ -5568,9 +5568,12 @@ static void skb_tstamp_tx_report_bpf_timestamping(struct sk_buff *skb, op = BPF_SOCK_OPS_TSTAMP_SCHED_CB; break; case SCM_TSTAMP_SND: - if (hwtstamps) - return; - op = BPF_SOCK_OPS_TSTAMP_SND_SW_CB; + if (hwtstamps) { + op = BPF_SOCK_OPS_TSTAMP_SND_HW_CB; + *skb_hwtstamps(skb) = *hwtstamps; + } else { + op = BPF_SOCK_OPS_TSTAMP_SND_SW_CB; + } break; default: return; diff --git a/net/dsa/user.c b/net/dsa/user.c index 291ab1b4acc4..794fe553dd77 100644 --- a/net/dsa/user.c +++ b/net/dsa/user.c @@ -897,7 +897,7 @@ static void dsa_skb_tx_timestamp(struct dsa_user_priv *p, { struct dsa_switch *ds = p->dp->ds; - if (!(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP)) + if (!(skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP_NOBPF)) return; if (!ds->ops->port_txtstamp) diff --git a/net/socket.c b/net/socket.c index 262a28b59c7f..517de433d4bb 100644 --- a/net/socket.c +++ b/net/socket.c @@ -676,7 +676,7 @@ void __sock_tx_timestamp(__u32 tsflags, __u8 *tx_flags) u8 flags = *tx_flags; if (tsflags & SOF_TIMESTAMPING_TX_HARDWARE) { - flags |= SKBTX_HW_TSTAMP; + flags |= SKBTX_HW_TSTAMP_NOBPF; /* PTP hardware clocks can provide a free running cycle counter * as a time base for virtual clocks. Tell driver to use the diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 1ead1e9d31be..0a7db1440653 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7041,6 +7041,10 @@ enum { * to the nic when SK_BPF_CB_TX_TIMESTAMPING * feature is on. */ + BPF_SOCK_OPS_TSTAMP_SND_HW_CB, /* Called in hardware phase when + * SK_BPF_CB_TX_TIMESTAMPING feature + * is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect From patchwork Thu Feb 20 07:29:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983431 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 014A91C5D5E; Thu, 20 Feb 2025 07:30:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036646; cv=none; b=XVKHN0lJ3o31CuuqSrv/3ZJJRF8o1ZlqfU2h9yshLHid0aZwFnsWtkQsNpGfcsbBZxQ5INiMPFraXnjni7Wn7YcMKCZ/A/P2+g74YcvdgiZQbvHC++A1jwfo7vGOOD7H7cRQpWd41zB6Q2TufwXRsV5hTIB05CSx20nDDcZ4ycY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036646; c=relaxed/simple; bh=pR9zQpfGxWDpMnOrXN6Jpazai6FhriaUKtjsEv8XPGc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mV7DHpQjzVWejmmhIoJcinYIWvuwFu3Yh1uA/hto9jf5ox8Gjm2NEehZKYEXzXkimr4vVKc3ZR32K0MiIKfS8AOtzLYoJm4e+c64ZM9rg6g3AlAaVoJKscedKyRTk1uyuPX1SLuuBN2gOxMrBRdvtlcuga3pi80ZrIJTIMOKZNg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XJUtXSEF; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XJUtXSEF" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-219f8263ae0so11821525ad.0; Wed, 19 Feb 2025 23:30:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036644; x=1740641444; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JVedz6vK3X/jikH9HldxENXdxMpSZlQIWb+EwsIWA6E=; b=XJUtXSEF36o3sxV5eTu8vnMNIxt3SE/1Y4taptl2Zk201bYmdgoYnP9ONgQIuL9Zu2 HrYfsK4a7YmMm9ttd1gIo+ETYNXseqabAtio3Cn/MuDPaZYqd1RwzdDP/85AOTLlLpU6 M7nTKlt3zaSpkGM29gz+7kVGeuTha4wZt8nrv6dyGF5MkMMqut++0wY2IHt76vWM/9R3 KPkSTOn2iQPKY2rC50PeYcOYmY9M5IHU+d9/esw9IOuiy4KdUfkWahUWJZu5yHRHRePh 58TLZd8MJTblBHpygXuibMLhfOiTvyqY9m2roqFdqxXx1fMPqjPqKXLcNDp2uhjS9SDj 6O5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036644; x=1740641444; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JVedz6vK3X/jikH9HldxENXdxMpSZlQIWb+EwsIWA6E=; b=qIvMyYu05VjDdCHeFCBFF3aZNjlVXak+HY4Na6s0cZF/0sheh0qrb5H9E3xPmbdYMR Cs8c5O7aLjPLiVuz+njBgzBLQ4WbwL41ni/wy3nuFnWgEuEoQKOCaHPQ+3i2yE935FeF JpNwsN+QxRreC72+zLC1u5wgSJoFDwc0tSMmVWdJTrtJ2tAPDZWryCKehOpyRT8bT5l1 PC0GlB1MUnJac7HPK2PT6lSp4aWrAG59iuqbLQMJTK5L4hvH2p2CzrP4DMkplwZdlu4f ljwhYorsrUal4gBk/3d34YJuTNHckMa1Re7vG1nHyBkOUrKkPqVVzigb5VlpMwv15nU+ p5iw== X-Forwarded-Encrypted: i=1; AJvYcCUbgjEhkI5+/DcYzdgbYlJtcpZhv45uTyMyQNdMIgHtfy49G9MUEklVZfcaU4iXhmtCkfYJNgo=@vger.kernel.org X-Gm-Message-State: AOJu0Yzqj8XyHbP2WhAKODUKSmTlRQp0phEwdtNwi0neNpmN1lpeAsoI QDGj/TAtIEQYtaBs2vuGybda9oq3HZbVSSIMS5l2m4BjivbcWfJp X-Gm-Gg: ASbGncuriLYFgfvWHyDQCdG+4pNEXJPW2tujtwuTUgFpTKcYTXoVZjEYJt0vH7q2x4I 0czEHi9iiuGrHKz2MnxfiPJPyw5eEnOCELRBhnGfAtnZg9E55gGB7eQqd4SQCjU1DjtHMPYAnfa IJdGRpv8Y23TGlLS3DEcALGr4kyUOdmazW5nslfzSdcF4WkQXUAV+evxJ9p6coLrcakRC8T0Wae 7zLZ9A1hS0/iqo/ZV5cqT2G8W5ooJZLj2X/KOwg7TKd8wp9jwtfSYcfJ10vDbz8Ak6ALseKZk3x CgkZzTcxghM6AeUsGFj4HM4BEVSqKq3KBOJ3ykkvqe4XeGd4aS05gC/AZ3ncsvw= X-Google-Smtp-Source: AGHT+IE85mZ3dsJgMoTFUrjjjQilbTC8gfmR0ATRmv6p81XdmnXHuAlK43mN25l9Fclqy9joFWNcdg== X-Received: by 2002:a17:903:32c3:b0:220:ce37:e31f with SMTP id d9443c01a7336-2210403dcc7mr298966345ad.17.1740036644174; Wed, 19 Feb 2025 23:30:44 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:43 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 09/12] bpf: add BPF_SOCK_OPS_TSTAMP_ACK_CB callback Date: Thu, 20 Feb 2025 15:29:37 +0800 Message-Id: <20250220072940.99994-10-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Support the ACK case for bpf timestamping. Add a new sock_ops callback, BPF_SOCK_OPS_TSTAMP_ACK_CB. This callback will occur at the same timestamping point as the user space's SCM_TSTAMP_ACK. The BPF program can use it to get the same SCM_TSTAMP_ACK timestamp without modifying the user-space application. This patch extends txstamp_ack to two bits: 1 stands for SO_TIMESTAMPING mode, 2 bpf extension. Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- include/net/tcp.h | 6 ++++-- include/uapi/linux/bpf.h | 5 +++++ net/core/skbuff.c | 5 ++++- net/ipv4/tcp.c | 2 +- tools/include/uapi/linux/bpf.h | 5 +++++ 5 files changed, 19 insertions(+), 4 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index 4c4dca59352b..2e2fc72e115b 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -958,10 +958,12 @@ struct tcp_skb_cb { __u8 sacked; /* State flags for SACK. */ __u8 ip_dsfield; /* IPv4 tos or IPv6 dsfield */ - __u8 txstamp_ack:1, /* Record TX timestamp for ack? */ +#define TSTAMP_ACK_SK 0x1 +#define TSTAMP_ACK_BPF 0x2 + __u8 txstamp_ack:2, /* Record TX timestamp for ack? */ eor:1, /* Is skb MSG_EOR marked? */ has_rxtstamp:1, /* SKB has a RX timestamp */ - unused:5; + unused:4; __u32 ack_seq; /* Sequence number ACK'd */ union { struct { diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 4ec1a86288ef..6f728342fabc 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7048,6 +7048,11 @@ enum { * SK_BPF_CB_TX_TIMESTAMPING feature * is on. */ + BPF_SOCK_OPS_TSTAMP_ACK_CB, /* Called when all the skbs in the + * same sendmsg call are acked + * when SK_BPF_CB_TX_TIMESTAMPING + * feature is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 77b8866f94c5..dd33c12f00ca 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5550,7 +5550,7 @@ static bool skb_tstamp_tx_report_so_timestamping(struct sk_buff *skb, return skb_shinfo(skb)->tx_flags & (hwtstamps ? SKBTX_HW_TSTAMP_NOBPF : SKBTX_SW_TSTAMP); case SCM_TSTAMP_ACK: - return TCP_SKB_CB(skb)->txstamp_ack; + return TCP_SKB_CB(skb)->txstamp_ack & TSTAMP_ACK_SK; } return false; @@ -5575,6 +5575,9 @@ static void skb_tstamp_tx_report_bpf_timestamping(struct sk_buff *skb, op = BPF_SOCK_OPS_TSTAMP_SND_SW_CB; } break; + case SCM_TSTAMP_ACK: + op = BPF_SOCK_OPS_TSTAMP_ACK_CB; + break; default: return; } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 0d704bda6c41..12b9c4f9c151 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -488,7 +488,7 @@ static void tcp_tx_timestamp(struct sock *sk, struct sockcm_cookie *sockc) sock_tx_timestamp(sk, sockc, &shinfo->tx_flags); if (tsflags & SOF_TIMESTAMPING_TX_ACK) - tcb->txstamp_ack = 1; + tcb->txstamp_ack |= TSTAMP_ACK_SK; if (tsflags & SOF_TIMESTAMPING_TX_RECORD_MASK) shinfo->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1; } diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 0a7db1440653..11d9fc3e3434 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7045,6 +7045,11 @@ enum { * SK_BPF_CB_TX_TIMESTAMPING feature * is on. */ + BPF_SOCK_OPS_TSTAMP_ACK_CB, /* Called when all the skbs in the + * same sendmsg call are acked + * when SK_BPF_CB_TX_TIMESTAMPING + * feature is on. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect From patchwork Thu Feb 20 07:29:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983432 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDAB01C5D5E; Thu, 20 Feb 2025 07:30:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036652; cv=none; b=YO9XzvoFeKoU1OePH17fgYEFzQAExxDO44KAqgIw+nMtweqjLhFk6MzdbJwsRpGDrbK2dq1k/WppZO4xyaT0I/vP5npc207cPisiPEgpL3x3kOTJ7PrUJQvup2tAXrGfz/dJ6pwfcrppoQLUh0YApXXdHfGSZIuflmYBhoRUKdQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036652; c=relaxed/simple; bh=QI63TEo7OBUA2R+M2ML5uZeeala2bdwqtRWUt0H/DSk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RoGI31BYt7H8Svdpnuev20N613GD1C0+BAoCkHVt3zLLVqLE4FGlDyZw+bJR0Gw0lEy8EdZZYJQWq9bimEXZyy6hJGUomSq20x3q/LF1lTT9ez+nVmmKvxJvgUh/mGhNxq2F4bqT9dWKrkpVAE4I08L0MkoqB29P6Rb7Cw7lSJI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aRL70anV; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aRL70anV" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-220e83d65e5so10052795ad.1; Wed, 19 Feb 2025 23:30:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036650; x=1740641450; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/p8GGgo+9N78a/jndiD1gK8PO1CY8jXwXhYb2d1ooT4=; b=aRL70anVJr0N2pW0iDWT05uZuk3xy/UMTLRjMVlK6GuvM2jYJ5wcXEls93oNXb5aqX F5eE5a+dlu+Y8Z/fW86owVAy8PsXFwiGn4tF0m97FgWQmKWylZuJUT3VeShh2B+55s6M sSOZkhWsIl5laZn+cmz2IaE2H2S7V9AdPMX6UNYe1WwoWvoxe+qN7GRcQloDv0sk2hEn dKSOXnC9J1jEsgCTf6ZmjBZ3fP23ZX4R/e4SrvfC5zTro7FlKjoUfn8v2z84QMVi0BOg QW871/jFIsja9nWQh5GEVN36MGYEpTtaFi0/BGNFrOkgzJ0h0zJXgzfzFaSWZwsQxJeR xizQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036650; x=1740641450; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/p8GGgo+9N78a/jndiD1gK8PO1CY8jXwXhYb2d1ooT4=; b=NpNFkQmfDCKuGIAmwX4gM6cvGXfYlzKiChI/uvQ4Jnbze/zZ2eQR3LT1/jrFU7G6nu fBdWokD2Chi4QD82SmGRAh90yPNMsiL6Cn+8Zvrqiz29IjgrxbxDHJBea/t16xSU5VTg 1JbB8gtBmXrsoG7mXUhMhH5r/LlTY3eHWVlQ3oKw4VT3lhDYTamf0OmMEVewHLqUbqsM Sj7mtQWAM/NPJiaOP2FyZEA4f4lILjaEprDMyaCWX+KhJMKPmusAO7T06gawt+SNi5Is 2FCZa2hJmyWYVSm1idxzLiIP0tA9ljuNBwN1k3T0kUpIgZxWSjEVUBdi535rTAnpPSQy E3kw== X-Forwarded-Encrypted: i=1; AJvYcCXRxqb5lMdZt7hSZCt6PMl3rYJrJ7FJWb5cpPsYgDFtJpAmeZ6RkKzcTfV6DdPJC5F7DkM6UmQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxrSn1V/QwKCaMznPBLTdiqfTag+ck+UWxqilQNoaZtsdoQ/4on wqh/wgRi5iErdpGg6nzFUrml7XujhPE7gP1Y5IGRSlZjCAX3dzrH X-Gm-Gg: ASbGnctAsTy82YEvcqtH/H2ZrFoe+rrNt6hYeSBgzhURTzXjkUTl4N3bc+uuJu0KSXB OnGTlEF3SY8permo3/9n8kj8OtbkLrvBUH0djcCfEBtK++R8ottxf1LGhWqwYcuV+6Ci4sGihs3 e3ksc5HSiEjJS8ybNJPwbfpa7aFxpyrhDvNWGulUPvuhmD1JTlynPWYn9YAvGjxCWJ2i4Bf+I9j QkO/8fXlpuA2az0YQ20p63Xbc9/A8cN/zqNgh/sp5Ulg0bQD8hJM7pkY9wPfGR9TAPd1PpFHKgU TPUy2Fa8KAxgWDHugscsXNsBWvVVWwvCpgldze6jY4/mIJAtOkqZ9+2ftGxhoFU= X-Google-Smtp-Source: AGHT+IEL4GrZJ70BNJH9vlmqwATRjyw9TS+BS7dI03WRf+qwCYRZSswIz2hiKgw7fUMS4d7y8tXJ7A== X-Received: by 2002:a17:902:eccf:b0:220:f140:f7be with SMTP id d9443c01a7336-221711a6986mr129567965ad.41.1740036650187; Wed, 19 Feb 2025 23:30:50 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:49 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 10/12] bpf: add BPF_SOCK_OPS_TSTAMP_SENDMSG_CB callback Date: Thu, 20 Feb 2025 15:29:38 +0800 Message-Id: <20250220072940.99994-11-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net This patch introduces a new callback in tcp_tx_timestamp() to correlate tcp_sendmsg timestamp with timestamps from other tx timestamping callbacks (e.g., SND/SW/ACK). Without this patch, BPF program wouldn't know which timestamps belong to which flow because of no socket lock protection. This new callback is inserted in tcp_tx_timestamp() to address this issue because tcp_tx_timestamp() still owns the same socket lock with tcp_sendmsg_locked() in the meanwhile tcp_tx_timestamp() initializes the timestamping related fields for the skb, especially tskey. The tskey is the bridge to do the correlation. For TCP, BPF program hooks the beginning of tcp_sendmsg_locked() and then stores the sendmsg timestamp at the bpf_sk_storage, correlating this timestamp with its tskey that are later used in other sending timestamping callbacks. Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- include/uapi/linux/bpf.h | 5 +++++ net/ipv4/tcp.c | 4 ++++ tools/include/uapi/linux/bpf.h | 5 +++++ 3 files changed, 14 insertions(+) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index 6f728342fabc..a46f6c35b232 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -7053,6 +7053,11 @@ enum { * when SK_BPF_CB_TX_TIMESTAMPING * feature is on. */ + BPF_SOCK_OPS_TSTAMP_SENDMSG_CB, /* Called when every sendmsg syscall + * is triggered. It's used to correlate + * sendmsg timestamp with corresponding + * tskey. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 12b9c4f9c151..2b4791a6791c 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -492,6 +492,10 @@ static void tcp_tx_timestamp(struct sock *sk, struct sockcm_cookie *sockc) if (tsflags & SOF_TIMESTAMPING_TX_RECORD_MASK) shinfo->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1; } + + if (cgroup_bpf_enabled(CGROUP_SOCK_OPS) && + SK_BPF_CB_FLAG_TEST(sk, SK_BPF_CB_TX_TIMESTAMPING) && skb) + bpf_skops_tx_timestamping(sk, skb, BPF_SOCK_OPS_TSTAMP_SENDMSG_CB); } static bool tcp_stream_is_readable(struct sock *sk, int target) diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index 11d9fc3e3434..8429c01573cd 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -7050,6 +7050,11 @@ enum { * when SK_BPF_CB_TX_TIMESTAMPING * feature is on. */ + BPF_SOCK_OPS_TSTAMP_SENDMSG_CB, /* Called when every sendmsg syscall + * is triggered. It's used to correlate + * sendmsg timestamp with corresponding + * tskey. + */ }; /* List of TCP states. There is a build check in net/ipv4/tcp.c to detect From patchwork Thu Feb 20 07:29:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983433 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1A461E7C28; Thu, 20 Feb 2025 07:30:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036658; cv=none; b=YovXmJZtNlPn9FwtJ+UeakM1rG3pJSsG+jwdtsBOdxJ7wz+sHamMCEE6idaiY3IzX84rBINGkPVzAo3CTZgqsDI2CmgdnKVxrv5PQjc/5Yzrb1W1CYhC0QyvVMXOl6ggYjbo1/peTH4osd54/xNM4zu247jdQBRtnELwQW6r2zo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036658; c=relaxed/simple; bh=DuwBsmdyCTo+cinAC8/Ea5sGzPUwGddRI8TSZBF9dcw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Hsnn52ImJvoMlA0eM4PUnXwH39houn4pRK5KguvxZaOSBwqaURIItr3sdh7oXIhEG9VAds65k7v/CiNtx1Kdo7mjKnEXbjsCPBZaZLjKuhF7sNK9Aq4i5ILYuc4nik3aKW5qEtx9Oy/XDhCJm5T2coDXa8R1RaC726en7X6Fj10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=J/g4QyyP; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J/g4QyyP" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2fc3db58932so899604a91.2; Wed, 19 Feb 2025 23:30:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036656; x=1740641456; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+r05UJr0uzwoUXWnrUBpOZC+aSB1ejZmiLmZcOtCjpk=; b=J/g4QyyPgjAqBOOguwc45u5aivkdKoQn3zYUXFcWc6NXxZMF4lUQP9m2aEz45MSzyK AKkAWuRk/i+uyvdf7ShiDM2x8KE/uXTIpOHzQXZ+Ieg9wY3QLubuvpe4sbQcOEicyM41 JeZmlNXQncYEoN7KfcP5OczkFRneT54jmHGzZWv2NDRzvV0uzTiJOg/a/awXNPxdS0ca 5TAtfb6XYLVggi9tG286zTJ2wsd1ej36FKpvv7NO5ODej9XTTyJKHcGnscxtVO68R2ar ndVovg3pZHI1+9fE4TcV/tUEeNTojiJlIdw8VSIf9PWGxKkGKk3yxJ38+uXWYSDunfKm iJtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036656; x=1740641456; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+r05UJr0uzwoUXWnrUBpOZC+aSB1ejZmiLmZcOtCjpk=; b=HTNhR2r+hQIzvThmuohUOXN9Ccvl9CoZG893ybXbhXVUx55Y3iE8dPknRfDtuuD5sX KpytOLrtsAmuCluavdwhwLwfY79NBSCshc+VtCoRh7yohLTO8ctULo0irXet3LGunjti jY6muVHSTfazjzhiNGvneqJonXFmBYyWH/kqh0aRdBvb7vcMt78sQ1iNNBQGeRI7viNs Kv+irLnHeXYOalga6T8ENVXNTz4GdX6XcmCHgo/4+wEnfpR9abZEbEnh/fznVQYPTIGT tOgd40ZcLHjXSB48W2wh19aZ/yB4J4zpNWqShA6khBpHmcEcHupGPs2U5vklVe+sAtKx RYDw== X-Forwarded-Encrypted: i=1; AJvYcCXUq7J+4IYygSLvwqobPJJQPZ6IyiOAS/lw68jIYtOCjdR/sRTayUTcFjf+PMOWDxwYHtglw0w=@vger.kernel.org X-Gm-Message-State: AOJu0YyISu6p0yF1wvDuNfPF7f6OWx4gHI3zOSefSZWadj2FBU7nNOJQ +z9d9DoQYLPmcQkRA29SJEHGjZItjMAx0BENUL8MsWRqYhQ3PaVa X-Gm-Gg: ASbGncuPixYhUDgtpe9jiWahdqmj26d1uTfMPg4lLpfjf9/bSWNProlYYMw7a6SNcz/ Qgr2LF503a6iU+H3e8MnaOHMyKdwgkiZ/kjqw+yttHv90UF/7EqvQdZMloL3cveJXjl+laSMK3R jne2zKA4HOFY51NDy7S/qjCmwEzioQoqXjv3N3J7nyaHbKim36Ee05ZXDkk2anmE2JodPVNBYDX 7LiJmETlcG+5Xj5heWRZz3WYybYkV5OkREAVTg7Pw2g2+TCQjXnkhpu/n1qaqAUOi6ca4U7juK/ HAfF5tefGZJw+BUseYRNmIupexU29tZ+eyhUEDccKfBlXmyop9MPooi82UhWiX8= X-Google-Smtp-Source: AGHT+IHrRY697fD2ruGz3EKE+JU0HNs92QA5D6yvh2Ya7Q3CBnY4g4s3LJ3kF+mFpaQaDHTPiMeWSA== X-Received: by 2002:a17:90b:2b8e:b0:2ea:7fd8:9dc1 with SMTP id 98e67ed59e1d1-2fcd0ca3b0bmr2848901a91.18.1740036656254; Wed, 19 Feb 2025 23:30:56 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:30:55 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 11/12] bpf: support selective sampling for bpf timestamping Date: Thu, 20 Feb 2025 15:29:39 +0800 Message-Id: <20250220072940.99994-12-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net Add the bpf_sock_ops_enable_tx_tstamp kfunc to allow BPF programs to selectively enable TX timestamping on a skb during tcp_sendmsg(). For example, BPF program will limit tracking X numbers of packets and then will stop there instead of tracing all the sendmsgs of matched flow all along. It would be helpful for users who cannot afford to calculate latencies from every sendmsg call probably due to the performance or storage space consideration. Signed-off-by: Jason Xing --- kernel/bpf/btf.c | 1 + net/core/filter.c | 31 ++++++++++++++++++++++++++++++- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 9433b6467bbe..740210f883dc 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -8522,6 +8522,7 @@ static int bpf_prog_type_to_kfunc_hook(enum bpf_prog_type prog_type) case BPF_PROG_TYPE_CGROUP_SOCK_ADDR: case BPF_PROG_TYPE_CGROUP_SOCKOPT: case BPF_PROG_TYPE_CGROUP_SYSCTL: + case BPF_PROG_TYPE_SOCK_OPS: return BTF_KFUNC_HOOK_CGROUP; case BPF_PROG_TYPE_SCHED_ACT: return BTF_KFUNC_HOOK_SCHED_ACT; diff --git a/net/core/filter.c b/net/core/filter.c index 7f56d0bbeb00..2c8e986bceb3 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -12102,6 +12102,25 @@ __bpf_kfunc int bpf_sk_assign_tcp_reqsk(struct __sk_buff *s, struct sock *sk, #endif } +__bpf_kfunc int bpf_sock_ops_enable_tx_tstamp(struct bpf_sock_ops_kern *skops, + u64 flags) +{ + struct sk_buff *skb; + + if (skops->op != BPF_SOCK_OPS_TSTAMP_SENDMSG_CB) + return -EOPNOTSUPP; + + if (flags) + return -EINVAL; + + skb = skops->skb; + skb_shinfo(skb)->tx_flags |= SKBTX_BPF; + TCP_SKB_CB(skb)->txstamp_ack |= TSTAMP_ACK_BPF; + skb_shinfo(skb)->tskey = TCP_SKB_CB(skb)->seq + skb->len - 1; + + return 0; +} + __bpf_kfunc_end_defs(); int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags, @@ -12135,6 +12154,10 @@ BTF_KFUNCS_START(bpf_kfunc_check_set_tcp_reqsk) BTF_ID_FLAGS(func, bpf_sk_assign_tcp_reqsk, KF_TRUSTED_ARGS) BTF_KFUNCS_END(bpf_kfunc_check_set_tcp_reqsk) +BTF_KFUNCS_START(bpf_kfunc_check_set_sock_ops) +BTF_ID_FLAGS(func, bpf_sock_ops_enable_tx_tstamp, KF_TRUSTED_ARGS) +BTF_KFUNCS_END(bpf_kfunc_check_set_sock_ops) + static const struct btf_kfunc_id_set bpf_kfunc_set_skb = { .owner = THIS_MODULE, .set = &bpf_kfunc_check_set_skb, @@ -12155,6 +12178,11 @@ static const struct btf_kfunc_id_set bpf_kfunc_set_tcp_reqsk = { .set = &bpf_kfunc_check_set_tcp_reqsk, }; +static const struct btf_kfunc_id_set bpf_kfunc_set_sock_ops = { + .owner = THIS_MODULE, + .set = &bpf_kfunc_check_set_sock_ops, +}; + static int __init bpf_kfunc_init(void) { int ret; @@ -12173,7 +12201,8 @@ static int __init bpf_kfunc_init(void) ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP, &bpf_kfunc_set_xdp); ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_CGROUP_SOCK_ADDR, &bpf_kfunc_set_sock_addr); - return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_kfunc_set_tcp_reqsk); + ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_kfunc_set_tcp_reqsk); + return ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SOCK_OPS, &bpf_kfunc_set_sock_ops); } late_initcall(bpf_kfunc_init); From patchwork Thu Feb 20 07:29:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Xing X-Patchwork-Id: 13983434 X-Patchwork-Delegate: bpf@iogearbox.net Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F03B1E883A; Thu, 20 Feb 2025 07:31:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036664; cv=none; b=VDxmDgjYRlP765is+K0oZcWkfsii3j10vIaP/Rply8AYX9RcA9cmONHvDCRyRqDG6rXKH3zpuECPNQJKrsTlck0oQLr4OjiQP4QlQPl77SB9tPKa3tjl0zgQqXs6GMJmVqERLpfGHaWJCfkNqXDY0r2LVpGsYvYq0f6XxR/SdHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740036664; c=relaxed/simple; bh=uao7UsH2a4mpXrvqkTaxAUgNf4TWgDFGK3ujwGFB5e8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=A3v7eYSCSmfEx/VU4Gr6/zVvkFXAv2lyZg0jzWEI1J0fqSB81LqP5YLG9aP4bIN0LYYL9yDeYmoC034zXNhunQm80iJY6byKzeohqUTceGHuIUt+gijqpEY8rYRjhRlEefU0ZymGQMxBFvYsfsL+ve/xlFBhY/rqQmo7J0x8AMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f0f99BPR; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f0f99BPR" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-220c8cf98bbso12154855ad.1; Wed, 19 Feb 2025 23:31:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740036662; x=1740641462; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=86eIJXq0RgP79LF85nM5ZFL0YR6jEATstaLLqp057sY=; b=f0f99BPRAk/TPDub3XV98HykJR93S5uwLuifPATeAfGqenM6yPu/ls+IYuixh3XCL/ zkHYr8vKMhVbdRM6zR9En54J+xZjj353A/236sd+BfTnuXMsN1IOJ+oZGX7UsXV5S/rW RKc5xi1nVGBwE2ezqWDMqvTeTc3o2daFbYowRXoTNkytJo50diQGkmpViu+Qj+8Yx3A2 2ixt8OugicgTAIAEKp5qtMJywXv1k4QzyXQSmOFgKZFkz0uyCyzqHyiJxbhhuf7jVah8 JzbBD703y7NdgQ/GVU3ENww9IDnBGEDC8ni5YChmUujJZBQfkQtntaacRpWUur2VRAKy KEXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740036662; x=1740641462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=86eIJXq0RgP79LF85nM5ZFL0YR6jEATstaLLqp057sY=; b=lCL1jAz5HJPJxS+Fewj8p45z059JFXQfhX/Y3MFkYFQFHbUglBTIEXVbFYiGI1larC YUsKAdaUhNyVmqRQ/v9PsYjHQ69KIv0t+1x217TyBJK8tSYh4NALujWnkLxPYhQxSRq5 ti6+C+UVXC7vY7AOm6UXPysqvwtkuhBPufMIX0561qn+mreBWxHRWI527BUC5th18PY1 1VSjwSX4CtWnJitDGplJaV6OU3uvyRt6f+XoxqcO5dEsBew1twM5FwcJntnbA5cZHeJ0 xDIovr53bCaMP9MuOUaJtG3+sO+vj2jn4v/yb8l28xQhTx7uXRI4+MQqq/i6wMu1u1az MjSw== X-Forwarded-Encrypted: i=1; AJvYcCVuIixtzLhncyjcQBbQYjmNyb263DWAwK7tsa/ZuvybrDU2MJ0xgAGIp/0CQB3H7ibMKe8biqI=@vger.kernel.org X-Gm-Message-State: AOJu0Yy9fLA8/cqoTKAckp214hjtQfXzZfJ05h2u4z46IsFBGEhcTXev Fz3zfw3lH4Ky9z1GJOPUD0BUxZvjgabprefcT+9ozVMs5Nq91uso X-Gm-Gg: ASbGncv13QdXkzHZvwpA6sQIHIi/POU6JHKTbAK+F88ADBMM3f+6UEBbhXVrfRrMV9F 79eM7kBZB4oLlZ0LU58L94PmzrCEOA+f5C/4eVXmtO4180QyTQ5HWQU+Z815vbY0MtdattP657z zbZ02eDUPUYdbrgoUCRAfWp4O9b2FLAnkepHTRf1CWUcwjpt6i5eeCoTS7Wt4Wn7Opi7ROPl3wC eDVWQXWs5V3V84euLn9rBlCoQmXLOOJWvs/vn3yMUUcypntdrOXvQJkwfbqDIj6oTBOlLQ4mUnj ibW/HPvLVNPQEGbXFMj+zTadG2ITGhSdD/uCfJbWPjB/LtqKHH+f9jiDQJD62mU= X-Google-Smtp-Source: AGHT+IHEoYnDgMYtDeZGpgRJXyMwOqAnuEsBukwk8bq2ztTnJwewXYj8LiFIKTs1nETOvT4tso6s1A== X-Received: by 2002:a17:902:ec85:b0:21f:35fd:1b6c with SMTP id d9443c01a7336-221040c0706mr308430425ad.45.1740036662425; Wed, 19 Feb 2025 23:31:02 -0800 (PST) Received: from KERNELXING-MB0.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-220d545d643sm114048205ad.126.2025.02.19.23.30.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Feb 2025 23:31:02 -0800 (PST) From: Jason Xing To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, dsahern@kernel.org, willemdebruijn.kernel@gmail.com, willemb@google.com, ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, eddyz87@gmail.com, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@fomichev.me, haoluo@google.com, jolsa@kernel.org, shuah@kernel.org, ykolal@fb.com Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, Jason Xing Subject: [PATCH bpf-next v13 12/12] selftests/bpf: add simple bpf tests in the tx path for timestamping feature Date: Thu, 20 Feb 2025 15:29:40 +0800 Message-Id: <20250220072940.99994-13-kerneljasonxing@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20250220072940.99994-1-kerneljasonxing@gmail.com> References: <20250220072940.99994-1-kerneljasonxing@gmail.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: bpf@iogearbox.net BPF program calculates a couple of latency deltas between each tx timestamping callbacks. It can be used in the real world to diagnose the kernel behaviour in the tx path. Check the safety issues by accessing a few bpf calls in bpf_test_access_bpf_calls() which are implemented in the patch 3 and 4. Check if the bpf timestamping can co-exist with socket timestamping. There remains a few realistic things[1][2] to highlight: 1. in general a packet may pass through multiple qdiscs. For instance with bonding or tunnel virtual devices in the egress path. 2. packets may be resent, in which case an ACK might precede a repeat SCHED and SND. 3. erroneous or malicious peers may also just never send an ACK. [1]: https://lore.kernel.org/all/67a389af981b0_14e0832949d@willemb.c.googlers.com.notmuch/ [2]: https://lore.kernel.org/all/c329a0c1-239b-4ca1-91f2-cb30b8dd2f6a@linux.dev/ Reviewed-by: Willem de Bruijn Signed-off-by: Jason Xing --- .../bpf/prog_tests/net_timestamping.c | 239 +++++++++++++++++ .../selftests/bpf/progs/net_timestamping.c | 248 ++++++++++++++++++ 2 files changed, 487 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/net_timestamping.c create mode 100644 tools/testing/selftests/bpf/progs/net_timestamping.c diff --git a/tools/testing/selftests/bpf/prog_tests/net_timestamping.c b/tools/testing/selftests/bpf/prog_tests/net_timestamping.c new file mode 100644 index 000000000000..dbfd87499b6b --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/net_timestamping.c @@ -0,0 +1,239 @@ +#include +#include +#include +#include "test_progs.h" +#include "network_helpers.h" +#include "net_timestamping.skel.h" + +#define CG_NAME "/net-timestamping-test" +#define NSEC_PER_SEC 1000000000LL + +static const char addr4_str[] = "127.0.0.1"; +static const char addr6_str[] = "::1"; +static struct net_timestamping *skel; +static const int cfg_payload_len = 30; +static struct timespec usr_ts; +static u64 delay_tolerance_nsec = 10000000000; /* 10 seconds */ +int SK_TS_SCHED; +int SK_TS_TXSW; +int SK_TS_ACK; + +static int64_t timespec_to_ns64(struct timespec *ts) +{ + return ts->tv_sec * NSEC_PER_SEC + ts->tv_nsec; +} + +static void validate_key(int tskey, int tstype) +{ + static int expected_tskey = -1; + + if (tstype == SCM_TSTAMP_SCHED) + expected_tskey = cfg_payload_len - 1; + + ASSERT_EQ(expected_tskey, tskey, "tskey mismatch"); + + expected_tskey = tskey; +} + +static void validate_timestamp(struct timespec *cur, struct timespec *prev) +{ + int64_t cur_ns, prev_ns; + + cur_ns = timespec_to_ns64(cur); + prev_ns = timespec_to_ns64(prev); + + ASSERT_LT(cur_ns - prev_ns, delay_tolerance_nsec, "latency"); +} + +static void test_socket_timestamp(struct scm_timestamping *tss, int tstype, + int tskey) +{ + static struct timespec prev_ts; + + validate_key(tskey, tstype); + + switch (tstype) { + case SCM_TSTAMP_SCHED: + validate_timestamp(&tss->ts[0], &usr_ts); + SK_TS_SCHED += 1; + break; + case SCM_TSTAMP_SND: + validate_timestamp(&tss->ts[0], &prev_ts); + SK_TS_TXSW += 1; + break; + case SCM_TSTAMP_ACK: + validate_timestamp(&tss->ts[0], &prev_ts); + SK_TS_ACK += 1; + break; + } + + prev_ts = tss->ts[0]; +} + +static void test_recv_errmsg_cmsg(struct msghdr *msg) +{ + struct sock_extended_err *serr = NULL; + struct scm_timestamping *tss = NULL; + struct cmsghdr *cm; + + for (cm = CMSG_FIRSTHDR(msg); + cm && cm->cmsg_len; + cm = CMSG_NXTHDR(msg, cm)) { + if (cm->cmsg_level == SOL_SOCKET && + cm->cmsg_type == SCM_TIMESTAMPING) { + tss = (void *)CMSG_DATA(cm); + } else if ((cm->cmsg_level == SOL_IP && + cm->cmsg_type == IP_RECVERR) || + (cm->cmsg_level == SOL_IPV6 && + cm->cmsg_type == IPV6_RECVERR) || + (cm->cmsg_level == SOL_PACKET && + cm->cmsg_type == PACKET_TX_TIMESTAMP)) { + serr = (void *)CMSG_DATA(cm); + ASSERT_EQ(serr->ee_origin, SO_EE_ORIGIN_TIMESTAMPING, + "cmsg type"); + } + + if (serr && tss) + test_socket_timestamp(tss, serr->ee_info, + serr->ee_data); + } +} + +static bool socket_recv_errmsg(int fd) +{ + static char ctrl[1024 /* overprovision*/]; + char data[cfg_payload_len]; + static struct msghdr msg; + struct iovec entry; + int n = 0; + + memset(&msg, 0, sizeof(msg)); + memset(&entry, 0, sizeof(entry)); + memset(ctrl, 0, sizeof(ctrl)); + + entry.iov_base = data; + entry.iov_len = cfg_payload_len; + msg.msg_iov = &entry; + msg.msg_iovlen = 1; + msg.msg_name = NULL; + msg.msg_namelen = 0; + msg.msg_control = ctrl; + msg.msg_controllen = sizeof(ctrl); + + n = recvmsg(fd, &msg, MSG_ERRQUEUE); + if (n == -1) + ASSERT_EQ(errno, EAGAIN, "recvmsg MSG_ERRQUEUE"); + + if (n >= 0) + test_recv_errmsg_cmsg(&msg); + + return n == -1; +} + +static void test_socket_timestamping(int fd) +{ + while (!socket_recv_errmsg(fd)); + + ASSERT_EQ(SK_TS_SCHED, 1, "SCM_TSTAMP_SCHED"); + ASSERT_EQ(SK_TS_TXSW, 1, "SCM_TSTAMP_SND"); + ASSERT_EQ(SK_TS_ACK, 1, "SCM_TSTAMP_ACK"); + + SK_TS_SCHED = 0; + SK_TS_TXSW = 0; + SK_TS_ACK = 0; +} + +static void test_tcp(int family, bool enable_socket_timestamping) +{ + struct net_timestamping__bss *bss; + char buf[cfg_payload_len]; + int sfd = -1, cfd = -1; + unsigned int sock_opt; + struct netns_obj *ns; + int cg_fd; + int ret; + + cg_fd = test__join_cgroup(CG_NAME); + if (!ASSERT_OK_FD(cg_fd, "join cgroup")) + return; + + ns = netns_new("net_timestamping_ns", true); + if (!ASSERT_OK_PTR(ns, "create ns")) + goto out; + + skel = net_timestamping__open_and_load(); + if (!ASSERT_OK_PTR(skel, "open and load skel")) + goto out; + + if (!ASSERT_OK(net_timestamping__attach(skel), "attach skel")) + goto out; + + skel->links.skops_sockopt = + bpf_program__attach_cgroup(skel->progs.skops_sockopt, cg_fd); + if (!ASSERT_OK_PTR(skel->links.skops_sockopt, "attach cgroup")) + goto out; + + bss = skel->bss; + memset(bss, 0, sizeof(*bss)); + + skel->bss->monitored_pid = getpid(); + + sfd = start_server(family, SOCK_STREAM, + family == AF_INET6 ? addr6_str : addr4_str, 0, 0); + if (!ASSERT_OK_FD(sfd, "start_server")) + goto out; + + cfd = connect_to_fd(sfd, 0); + if (!ASSERT_OK_FD(cfd, "connect_to_fd_server")) + goto out; + + if (enable_socket_timestamping) { + sock_opt = SOF_TIMESTAMPING_SOFTWARE | + SOF_TIMESTAMPING_OPT_ID | + SOF_TIMESTAMPING_TX_SCHED | + SOF_TIMESTAMPING_TX_SOFTWARE | + SOF_TIMESTAMPING_TX_ACK; + ret = setsockopt(cfd, SOL_SOCKET, SO_TIMESTAMPING, + (char *) &sock_opt, sizeof(sock_opt)); + if (!ASSERT_OK(ret, "setsockopt SO_TIMESTAMPING")) + goto out; + + ret = clock_gettime(CLOCK_REALTIME, &usr_ts); + if (!ASSERT_OK(ret, "get user time")) + goto out; + } + + ret = write(cfd, buf, sizeof(buf)); + if (!ASSERT_EQ(ret, sizeof(buf), "send to server")) + goto out; + + if (enable_socket_timestamping) + test_socket_timestamping(cfd); + + ASSERT_EQ(bss->nr_active, 1, "nr_active"); + ASSERT_EQ(bss->nr_snd, 2, "nr_snd"); + ASSERT_EQ(bss->nr_sched, 1, "nr_sched"); + ASSERT_EQ(bss->nr_txsw, 1, "nr_txsw"); + ASSERT_EQ(bss->nr_ack, 1, "nr_ack"); + +out: + if (sfd >= 0) + close(sfd); + if (cfd >= 0) + close(cfd); + net_timestamping__destroy(skel); + netns_free(ns); + close(cg_fd); +} + +void test_net_timestamping(void) +{ + if (test__start_subtest("INET4: bpf timestamping")) + test_tcp(AF_INET, false); + if (test__start_subtest("INET4: bpf and socket timestamping")) + test_tcp(AF_INET, true); + if (test__start_subtest("INET6: bpf timestamping")) + test_tcp(AF_INET6, false); + if (test__start_subtest("INET6: bpf and socket timestamping")) + test_tcp(AF_INET6, true); +} diff --git a/tools/testing/selftests/bpf/progs/net_timestamping.c b/tools/testing/selftests/bpf/progs/net_timestamping.c new file mode 100644 index 000000000000..b4c2f0f2be11 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/net_timestamping.c @@ -0,0 +1,248 @@ +#include "vmlinux.h" +#include "bpf_tracing_net.h" +#include +#include +#include "bpf_misc.h" +#include "bpf_kfuncs.h" +#include + +__u32 monitored_pid = 0; + +int nr_active; +int nr_snd; +int nr_passive; +int nr_sched; +int nr_txsw; +int nr_ack; + +struct sk_stg { + __u64 sendmsg_ns; /* record ts when sendmsg is called */ +}; + +struct sk_tskey { + u64 cookie; + u32 tskey; +}; + +struct delay_info { + u64 sendmsg_ns; /* record ts when sendmsg is called */ + u32 sched_delay; /* SCHED_CB - sendmsg_ns */ + u32 snd_sw_delay; /* SND_SW_CB - SCHED_CB */ + u32 ack_delay; /* ACK_CB - SND_SW_CB */ +}; + +struct { + __uint(type, BPF_MAP_TYPE_SK_STORAGE); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, int); + __type(value, struct sk_stg); +} sk_stg_map SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __type(key, struct sk_tskey); + __type(value, struct delay_info); + __uint(max_entries, 1024); +} time_map SEC(".maps"); + +static u64 delay_tolerance_nsec = 10000000000; /* 10 second as an example */ + +extern int bpf_sock_ops_enable_tx_tstamp(struct bpf_sock_ops_kern *skops, u64 flags) __ksym; + +static int bpf_test_sockopt(void *ctx, const struct sock *sk, int expected) +{ + int tmp, new = SK_BPF_CB_TX_TIMESTAMPING; + int opt = SK_BPF_CB_FLAGS; + int level = SOL_SOCKET; + + if (bpf_setsockopt(ctx, level, opt, &new, sizeof(new)) != expected) + return 1; + + if (bpf_getsockopt(ctx, level, opt, &tmp, sizeof(tmp)) != expected || + (!expected && tmp != new)) + return 1; + + return 0; +} + +static bool bpf_test_access_sockopt(void *ctx, const struct sock *sk) +{ + if (bpf_test_sockopt(ctx, sk, -EOPNOTSUPP)) + return true; + return false; +} + +static bool bpf_test_access_load_hdr_opt(struct bpf_sock_ops *skops) +{ + u8 opt[3] = {0}; + int load_flags = 0; + int ret; + + ret = bpf_load_hdr_opt(skops, opt, sizeof(opt), load_flags); + if (ret != -EOPNOTSUPP) + return true; + + return false; +} + +static bool bpf_test_access_cb_flags_set(struct bpf_sock_ops *skops) +{ + int ret; + + ret = bpf_sock_ops_cb_flags_set(skops, 0); + if (ret != -EOPNOTSUPP) + return true; + + return false; +} + +/* In the timestamping callbacks, we're not allowed to call the following + * BPF CALLs for the safety concern. Return false if expected. + */ +static bool bpf_test_access_bpf_calls(struct bpf_sock_ops *skops, + const struct sock *sk) +{ + if (bpf_test_access_sockopt(skops, sk)) + return true; + + if (bpf_test_access_load_hdr_opt(skops)) + return true; + + if (bpf_test_access_cb_flags_set(skops)) + return true; + + return false; +} + +static bool bpf_test_delay(struct bpf_sock_ops *skops, const struct sock *sk) +{ + struct bpf_sock_ops_kern *skops_kern; + u64 timestamp = bpf_ktime_get_ns(); + struct skb_shared_info *shinfo; + struct delay_info dinfo = {0}; + struct sk_tskey key = {0}; + struct delay_info *val; + struct sk_buff *skb; + struct sk_stg *stg; + u64 prior_ts, delay; + + if (bpf_test_access_bpf_calls(skops, sk)) + return false; + + skops_kern = bpf_cast_to_kern_ctx(skops); + skb = skops_kern->skb; + shinfo = bpf_core_cast(skb->head + skb->end, struct skb_shared_info); + + key.cookie = bpf_get_socket_cookie(skops); + if (!key.cookie) + return false; + + if (skops->op == BPF_SOCK_OPS_TSTAMP_SENDMSG_CB) { + stg = bpf_sk_storage_get(&sk_stg_map, (void *)sk, 0, 0); + if (!stg) + return false; + dinfo.sendmsg_ns = stg->sendmsg_ns; + bpf_sock_ops_enable_tx_tstamp(skops_kern, 0); + key.tskey = shinfo->tskey; + if (!key.tskey) + return false; + bpf_map_update_elem(&time_map, &key, &dinfo, BPF_ANY); + return true; + } + + key.tskey = shinfo->tskey; + if (!key.tskey) + return false; + + val = bpf_map_lookup_elem(&time_map, &key); + if (!val) + return false; + + switch (skops->op) { + case BPF_SOCK_OPS_TSTAMP_SCHED_CB: + val->sched_delay = timestamp - val->sendmsg_ns; + delay = val->sched_delay; + break; + case BPF_SOCK_OPS_TSTAMP_SND_SW_CB: + prior_ts = val->sched_delay + val->sendmsg_ns; + val->snd_sw_delay = timestamp - prior_ts; + delay = val->snd_sw_delay; + break; + case BPF_SOCK_OPS_TSTAMP_ACK_CB: + prior_ts = val->snd_sw_delay + val->sched_delay + val->sendmsg_ns; + val->ack_delay = timestamp - prior_ts; + delay = val->ack_delay; + break; + } + + if (delay >= delay_tolerance_nsec) + return false; + + /* Since it's the last one, remove from the map after latency check */ + if (skops->op == BPF_SOCK_OPS_TSTAMP_ACK_CB) + bpf_map_delete_elem(&time_map, &key); + + return true; +} + +SEC("fentry/tcp_sendmsg_locked") +int BPF_PROG(trace_tcp_sendmsg_locked, struct sock *sk, struct msghdr *msg, + size_t size) +{ + __u32 pid = bpf_get_current_pid_tgid() >> 32; + u64 timestamp = bpf_ktime_get_ns(); + u32 flag = sk->sk_bpf_cb_flags; + struct sk_stg *stg; + + if (pid != monitored_pid || !flag) + return 0; + + stg = bpf_sk_storage_get(&sk_stg_map, sk, 0, + BPF_SK_STORAGE_GET_F_CREATE); + if (!stg) + return 0; + + stg->sendmsg_ns = timestamp; + nr_snd += 1; + return 0; +} + +SEC("sockops") +int skops_sockopt(struct bpf_sock_ops *skops) +{ + struct bpf_sock *bpf_sk = skops->sk; + const struct sock *sk; + + if (!bpf_sk) + return 1; + + sk = (struct sock *)bpf_skc_to_tcp_sock(bpf_sk); + if (!sk) + return 1; + + switch (skops->op) { + case BPF_SOCK_OPS_ACTIVE_ESTABLISHED_CB: + nr_active += !bpf_test_sockopt(skops, sk, 0); + break; + case BPF_SOCK_OPS_TSTAMP_SENDMSG_CB: + if (bpf_test_delay(skops, sk)) + nr_snd += 1; + break; + case BPF_SOCK_OPS_TSTAMP_SCHED_CB: + if (bpf_test_delay(skops, sk)) + nr_sched += 1; + break; + case BPF_SOCK_OPS_TSTAMP_SND_SW_CB: + if (bpf_test_delay(skops, sk)) + nr_txsw += 1; + break; + case BPF_SOCK_OPS_TSTAMP_ACK_CB: + if (bpf_test_delay(skops, sk)) + nr_ack += 1; + break; + } + + return 1; +} + +char _license[] SEC("license") = "GPL";