From patchwork Thu Jun 13 23:31:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13697589 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5A7814BF91 for ; Thu, 13 Jun 2024 23:31:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321502; cv=none; b=pZVKvpLSK3+b5V162HwYnEHayBjjcGSjbcJrDpqe/GVjyzXmAaDtqIqnNUHVNtPcbZ+hOO1pJmzx3Cxfg3kE0N4QHILFS5i9o301yw3BMse1w2adXCbdMfnn9sGRhmN5c2dkKZfB8dB2YkP8f9V4DIXv391FlkA8ZfXlClHZiBQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321502; c=relaxed/simple; bh=UNwIgy0eaP1U/PjG+dxmb5r6nXFEuXCnnd8Z+VVWbpM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kimDbv3L3cfZeNMPkNsGFTMwXV+oyB6mGS5bFjHfV3vkYtAi6uZanIZX15yzTvj9HEj4zChitzADrgLRed8kzsOybcLZVRr0afwFsuyLZMP3gXyJQ20e1KwHb+T7IGz9xWBrETIYeUu3eTF7QuvVMtRUL1M7c23/wRVbE6s1aVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=h+HUEF3d; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="h+HUEF3d" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7955f3d4516so243421185a.1 for ; Thu, 13 Jun 2024 16:31:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1718321499; x=1718926299; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pq1qBmFA90YYxxuCBnsZZeZJpNA0DQPz8KGklT3/y7Q=; b=h+HUEF3d2Ou0ZHwffYUgUGDaWPFRy059EymS74uU1dgVpSr8Vs+xE0VyaxytUtAm80 ske8tL5JSMfmAXw8Uts66nHgkwiob3MgCqyo/gp7aXQvdbsQDj0ujpWSp9skqBEolh9r RP1aHwZWX/xj5EiHTjcPc8MJClU3HYxvkfhVQi6f/adDxPKJ2pgKIefiZjO8P//7m93S QiVmSv1VAaeDaor8qWxXzPMwHL+glcwO9wC/OXgXl8LBebEFqeru47nuKmSNChtP64t+ ICCSjQ1GOFy5rHFakYFIsMGwrMK1/a0qCdBuh8+bdyAVGNxN2J3OCxBsI1FL9HKETQpi rtlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718321499; x=1718926299; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pq1qBmFA90YYxxuCBnsZZeZJpNA0DQPz8KGklT3/y7Q=; b=h8YORaPHE9/20R7hEC/M0DoRGMXMUND/g7QXyJpcV1ces42A3qe6Gd7v6l4Uk/chZD H66G8RXzn8O2FLm6TATtg558kWb6T2hbRwfuSLC80FVcy8jIDNoyE+hAXsBEInCzBbA7 LxiB7FUApX7R4iNOfk2PZUm+n9xo51NQ0qYD9MTz03FdVZh56fI1VpGZKlpY5woEnDHB y1FFYIv77M6VlZ6fwnB1fZjpoJorjNey8EL0J53H1NjWEnWOz0km/pE1BDbKgqkaKnnc 237fWY3Sn3jnywvlfR3l/PewHUwUJGuc/Xg+j1Jsp7jXSZ9lqculy1LgjHpEZ+luRTeN GB9w== X-Gm-Message-State: AOJu0Yyk99DNJWnnCA8ZPn3gPONQz6AM7u52gf3/sYTi5rCoV5HqP+SM D81UAEIyP7jJWGebN0xeMwYzGZkH2Yu3tEE8UR23zV3aAaJDmq95X4+8leTJnbQAMeTg2phdWcZ HrdU= X-Google-Smtp-Source: AGHT+IF2Z9qJ5IKUBtGIpcZWjdqOzhvmlSlgpDZMLHuDH+UyyedWvo8Nt+dekh1GlCitqxhvaPS6Qg== X-Received: by 2002:a05:620a:468b:b0:795:532f:3a82 with SMTP id af79cd13be357-798101b4089mr682823985a.32.1718321498715; Thu, 13 Jun 2024 16:31:38 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-441ef3d8b62sm10586731cf.11.2024.06.13.16.31.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jun 2024 16:31:38 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v5 1/4] selftests: fix OOM problem in msg_zerocopy selftest Date: Thu, 13 Jun 2024 23:31:30 +0000 Message-Id: <20240613233133.2463193-2-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240613233133.2463193-1-zijianzhang@bytedance.com> References: <20240613233133.2463193-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang In selftests/net/msg_zerocopy.c, it has a while loop keeps calling sendmsg on a socket with MSG_ZEROCOPY flag, and it will recv the notifications until the socket is not writable. Typically, it will start the receiving process after around 30+ sendmsgs. However, because of the commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale") the sender is always writable and does not get any chance to run recv notifications. The selftest always exits with OUT_OF_MEMORY because the memory used by opt_skb exceeds the core.sysctl_optmem_max. According to our experiments, this problem can be mitigated by open the DEBUG_LOCKDEP configuration for the kernel. But it will makes the notifications disordered even in good commits before commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"). We introduce "cfg_notification_limit" to force sender to receive notifications after some number of sendmsgs. And, notifications may not come in order, because of the reason we present above. We have order checking code managed by cfg_verbose. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index bdc03a2097e8..7ea5fb28c93d 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -85,6 +85,7 @@ static bool cfg_rx; static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; +static int cfg_notification_limit = 32; static bool cfg_zerocopy; static socklen_t cfg_alen; @@ -95,6 +96,7 @@ static char payload[IP_MAXPACKET]; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; +static uint32_t sends_since_notify; static unsigned long gettimeofday_ms(void) { @@ -208,6 +210,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) error(1, errno, "send"); if (cfg_verbose && ret != len) fprintf(stderr, "send: ret=%u != %u\n", ret, len); + sends_since_notify++; if (len) { packets++; @@ -435,7 +438,7 @@ static bool do_recv_completion(int fd, int domain) /* Detect notification gaps. These should not happen often, if at all. * Gaps can occur due to drops, reordering and retransmissions. */ - if (lo != next_completion) + if (cfg_verbose && lo != next_completion) fprintf(stderr, "gap: %u..%u does not append to %u\n", lo, hi, next_completion); next_completion = hi + 1; @@ -460,6 +463,7 @@ static bool do_recv_completion(int fd, int domain) static void do_recv_completions(int fd, int domain) { while (do_recv_completion(fd, domain)) {} + sends_since_notify = 0; } /* Wait for all remaining completions on the errqueue */ @@ -549,6 +553,9 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); + if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + do_recv_completions(fd, domain); + while (!do_poll(fd, POLLOUT)) { if (cfg_zerocopy) do_recv_completions(fd, domain); @@ -708,7 +715,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -736,6 +743,9 @@ static void parse_opts(int argc, char **argv) if (cfg_ifindex == 0) error(1, errno, "invalid iface: %s", optarg); break; + case 'l': + cfg_notification_limit = strtoul(optarg, NULL, 0); + break; case 'm': cfg_cork_mixed = true; break; From patchwork Thu Jun 13 23:31:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13697590 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA3B514F106 for ; Thu, 13 Jun 2024 23:31:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321503; cv=none; b=cXzzsjljAdTjX9GembygwyKEUzWkHN47URWClLVTrTVjMi8ed851NT6VFIX/gaBfffLuywx5PmmvAFIOI7/Fvb1SF4mwygJMuc12kUNGlZ/BHL31sXBjJzdCNbxMYdmQSOgiL7wPg1MxkrC6JzHOoGHuSpAQVRn0AtsJVC8EIVQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321503; c=relaxed/simple; bh=f+ZE2jdRVU4Jef4nDk8OnP+i+LGYjL2HF3ZUt3YHpk8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WXj3ki0vKo7GBaAm+PPjv8AMVeoMAF2FmqlflfA7nGHUQ1Kyaa3LR++wH54+nFoJ+mkXiduVaBCxI5zT3IVOaADbMWC7iCkA/ZZLzVN1tEz+yRr0LJpNwMb89BjCp9O1J4aYYTBycIO1Y9xPp9KBdVUTloLu/oHF35y108I9u4g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Df9bi79x; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Df9bi79x" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-63267c30eaaso813097b3.1 for ; Thu, 13 Jun 2024 16:31:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1718321500; x=1718926300; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=N3MAm54IjrMNFmR87QYhCBvb1F7SoQWln+56mCe2/YE=; b=Df9bi79xMJKg75qa1Fwm2WYu130nAt9mSjRxIPoo5JcUFWxfiY8mj8zcdvX2/iWRt4 E1Yaz97f9VMFw6yKMWhl796EQftGkM0+tvvuiANkMd56urABnRi4qsH+YYHLxU7C0k6d t60IBZV3P9pBvYQWXpJmL8t1BilqX22QKPcdkPop0u4wfOG7OKLxfMHKfNRUobdxunto FUfnu2BMdTZfFc48lIAJA4XgOu1yxSjfsu9Iml91UNXKxSGQmjD3O9E5cOQOtmykYdSx q9jp1ikX++RJJjbioGrCzYKsj6hZ4CfgYIi3FjIiuZuCIOFhNAx8fC6xIMMg3EYTQ7U2 slmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718321500; x=1718926300; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=N3MAm54IjrMNFmR87QYhCBvb1F7SoQWln+56mCe2/YE=; b=AjvkUIHZFSy2YPl1a8TsKC5uLm4K/FkAaeoxBQRxxI+YcGmBG14fMPPcjqY2fXCRdL Y0660mW3MUIjpLRFbdmYOWqzshtjRqnsXNq7kGOsf1Y+SCOp1Hq733RqamMmW86CsM9o jPam/pJYy5pCWDEMcmz4gD0DsIMPJfeA4Ty9e4jLRFX+aGJj6EMHTXyqmvR7ltY4H4P0 AoyyPSnHAQQlwlS5dWdG6b+Y6WuEBJnRHZfnyuxDlkHjeY62BhdEpGDGxlXBjFfoyT1L y3B898L8wq1G+kEDJABpNCBlTkahNuoVNh/8D4jBnS1aFEN5lcWMK3KcUghA6AeTsu8E xyWg== X-Gm-Message-State: AOJu0Yyr5PhDClsy8s3ewV84mkoAIq0JpG02lNm/CrVYMXzLbsJRU+eS KkgZFKj5zs94a8znYTlRcHbhBRNfuFGa+s1XLteMRYwI2NOmgEOZ3gnlWiX6KbWA4fRb+XXd+m1 h0fs= X-Google-Smtp-Source: AGHT+IGGEWTVn7zGcCjZIpDHbXLwahqM1ktZfo1E63xoFF1noNGdJF7AnRbX58OmiCIUyB9oCeLnVQ== X-Received: by 2002:a0d:d752:0:b0:630:8c74:eab8 with SMTP id 00721157ae682-63224a0a50amr8451007b3.42.1718321500344; Thu, 13 Jun 2024 16:31:40 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-441ef3d8b62sm10586731cf.11.2024.06.13.16.31.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jun 2024 16:31:39 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v5 2/4] sock: support put_cmsg to userspace in TX path Date: Thu, 13 Jun 2024 23:31:31 +0000 Message-Id: <20240613233133.2463193-3-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240613233133.2463193-1-zijianzhang@bytedance.com> References: <20240613233133.2463193-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang Since ____sys_sendmsg creates a kernel copy of msg_control and passes that to the callees, put_cmsg will write into this kernel buffer. If people want to piggyback some information like timestamps upon returning of sendmsg. ____sys_sendmsg will have to copy_to_user to the original buf, which is not supported. As a result, users typically have to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system call overhead. This commit supports put_cmsg to userspace in TX path by storing user msg_control address in a new field in struct msghdr, and adding a new bit flag use_msg_control_user_tx to toggle the behavior of put_cmsg. Thus, it's possible to piggyback information in the msg_control of sendmsg. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- include/linux/socket.h | 4 ++++ net/compat.c | 33 +++++++++++++++++++++++++-------- net/core/scm.c | 42 ++++++++++++++++++++++++++++++++---------- net/socket.c | 2 ++ 4 files changed, 63 insertions(+), 18 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index 89d16b90370b..8d3db04f4a39 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -71,9 +71,12 @@ struct msghdr { void __user *msg_control_user; }; bool msg_control_is_user : 1; + bool use_msg_control_user_tx : 1; bool msg_get_inq : 1;/* return INQ after receive */ unsigned int msg_flags; /* flags on received message */ + void __user *msg_control_user_tx; /* msg_control_user in TX piggyback path */ __kernel_size_t msg_controllen; /* ancillary data buffer length */ + __kernel_size_t msg_controllen_user_tx; /* msg_controllen in TX piggyback path */ struct kiocb *msg_iocb; /* ptr to iocb for async requests */ struct ubuf_info *msg_ubuf; int (*sg_from_iter)(struct sock *sk, struct sk_buff *skb, @@ -391,6 +394,7 @@ struct ucred { extern int move_addr_to_kernel(void __user *uaddr, int ulen, struct sockaddr_storage *kaddr); extern int put_cmsg(struct msghdr*, int level, int type, int len, void *data); +extern int put_cmsg_user_tx(struct msghdr *msg, int level, int type, int len, void *data); struct timespec64; struct __kernel_timespec; diff --git a/net/compat.c b/net/compat.c index 485db8ee9b28..ae9d78b1c18b 100644 --- a/net/compat.c +++ b/net/compat.c @@ -211,6 +211,8 @@ int cmsghdr_from_user_compat_to_kern(struct msghdr *kmsg, struct sock *sk, goto Einval; /* Ok, looks like we made it. Hook it up and return success. */ + kmsg->msg_control_user_tx = kmsg->msg_control_user; + kmsg->msg_controllen_user_tx = kcmlen; kmsg->msg_control_is_user = false; kmsg->msg_control = kcmsg_base; kmsg->msg_controllen = kcmlen; @@ -226,13 +228,22 @@ int cmsghdr_from_user_compat_to_kern(struct msghdr *kmsg, struct sock *sk, int put_cmsg_compat(struct msghdr *kmsg, int level, int type, int len, void *data) { - struct compat_cmsghdr __user *cm = (struct compat_cmsghdr __user *) kmsg->msg_control_user; + struct compat_cmsghdr __user *cm; struct compat_cmsghdr cmhdr; struct old_timeval32 ctv; struct old_timespec32 cts[3]; + compat_size_t msg_controllen; int cmlen; - if (cm == NULL || kmsg->msg_controllen < sizeof(*cm)) { + if (kmsg->use_msg_control_user_tx) { + cm = (struct compat_cmsghdr __user *)kmsg->msg_control_user_tx; + msg_controllen = kmsg->msg_controllen_user_tx; + } else { + cm = (struct compat_cmsghdr __user *)kmsg->msg_control_user; + msg_controllen = kmsg->msg_controllen; + } + + if (!cm || msg_controllen < sizeof(*cm)) { kmsg->msg_flags |= MSG_CTRUNC; return 0; /* XXX: return error? check spec. */ } @@ -260,9 +271,9 @@ int put_cmsg_compat(struct msghdr *kmsg, int level, int type, int len, void *dat } cmlen = CMSG_COMPAT_LEN(len); - if (kmsg->msg_controllen < cmlen) { + if (msg_controllen < cmlen) { kmsg->msg_flags |= MSG_CTRUNC; - cmlen = kmsg->msg_controllen; + cmlen = msg_controllen; } cmhdr.cmsg_level = level; cmhdr.cmsg_type = type; @@ -273,10 +284,16 @@ int put_cmsg_compat(struct msghdr *kmsg, int level, int type, int len, void *dat if (copy_to_user(CMSG_COMPAT_DATA(cm), data, cmlen - sizeof(struct compat_cmsghdr))) return -EFAULT; cmlen = CMSG_COMPAT_SPACE(len); - if (kmsg->msg_controllen < cmlen) - cmlen = kmsg->msg_controllen; - kmsg->msg_control_user += cmlen; - kmsg->msg_controllen -= cmlen; + if (msg_controllen < cmlen) + cmlen = msg_controllen; + + if (kmsg->use_msg_control_user_tx) { + kmsg->msg_control_user_tx += cmlen; + kmsg->msg_controllen_user_tx -= cmlen; + } else { + kmsg->msg_control_user += cmlen; + kmsg->msg_controllen -= cmlen; + } return 0; } diff --git a/net/core/scm.c b/net/core/scm.c index 4f6a14babe5a..de70ff1981a1 100644 --- a/net/core/scm.c +++ b/net/core/scm.c @@ -228,25 +228,29 @@ int __scm_send(struct socket *sock, struct msghdr *msg, struct scm_cookie *p) } EXPORT_SYMBOL(__scm_send); -int put_cmsg(struct msghdr * msg, int level, int type, int len, void *data) +static int __put_cmsg(struct msghdr *msg, int level, int type, int len, void *data) { int cmlen = CMSG_LEN(len); + __kernel_size_t msg_controllen; + msg_controllen = msg->use_msg_control_user_tx ? + msg->msg_controllen_user_tx : msg->msg_controllen; if (msg->msg_flags & MSG_CMSG_COMPAT) return put_cmsg_compat(msg, level, type, len, data); - if (!msg->msg_control || msg->msg_controllen < sizeof(struct cmsghdr)) { + if (!msg->msg_control || msg_controllen < sizeof(struct cmsghdr)) { msg->msg_flags |= MSG_CTRUNC; return 0; /* XXX: return error? check spec. */ } - if (msg->msg_controllen < cmlen) { + if (msg_controllen < cmlen) { msg->msg_flags |= MSG_CTRUNC; - cmlen = msg->msg_controllen; + cmlen = msg_controllen; } - if (msg->msg_control_is_user) { - struct cmsghdr __user *cm = msg->msg_control_user; + if (msg->use_msg_control_user_tx || msg->msg_control_is_user) { + struct cmsghdr __user *cm; + cm = msg->msg_control_is_user ? msg->msg_control_user : msg->msg_control_user_tx; check_object_size(data, cmlen - sizeof(*cm), true); if (!user_write_access_begin(cm, cmlen)) @@ -267,12 +271,17 @@ int put_cmsg(struct msghdr * msg, int level, int type, int len, void *data) memcpy(CMSG_DATA(cm), data, cmlen - sizeof(*cm)); } - cmlen = min(CMSG_SPACE(len), msg->msg_controllen); - if (msg->msg_control_is_user) + cmlen = min(CMSG_SPACE(len), msg_controllen); + if (msg->msg_control_is_user) { msg->msg_control_user += cmlen; - else + msg->msg_controllen -= cmlen; + } else if (msg->use_msg_control_user_tx) { + msg->msg_control_user_tx += cmlen; + msg->msg_controllen_user_tx -= cmlen; + } else { msg->msg_control += cmlen; - msg->msg_controllen -= cmlen; + msg->msg_controllen -= cmlen; + } return 0; efault_end: @@ -280,8 +289,21 @@ int put_cmsg(struct msghdr * msg, int level, int type, int len, void *data) efault: return -EFAULT; } + +int put_cmsg(struct msghdr *msg, int level, int type, int len, void *data) +{ + msg->use_msg_control_user_tx = false; + return __put_cmsg(msg, level, type, len, data); +} EXPORT_SYMBOL(put_cmsg); +int put_cmsg_user_tx(struct msghdr *msg, int level, int type, int len, void *data) +{ + msg->use_msg_control_user_tx = true; + return __put_cmsg(msg, level, type, len, data); +} +EXPORT_SYMBOL(put_cmsg_user_tx); + void put_cmsg_scm_timestamping64(struct msghdr *msg, struct scm_timestamping_internal *tss_internal) { struct scm_timestamping64 tss; diff --git a/net/socket.c b/net/socket.c index e416920e9399..2755bc7bef9c 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2561,6 +2561,8 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, err = -EFAULT; if (copy_from_user(ctl_buf, msg_sys->msg_control_user, ctl_len)) goto out_freectl; + msg_sys->msg_control_user_tx = msg_sys->msg_control_user; + msg_sys->msg_controllen_user_tx = msg_sys->msg_controllen; msg_sys->msg_control = ctl_buf; msg_sys->msg_control_is_user = false; } From patchwork Thu Jun 13 23:31:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13697591 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8DFC815539A for ; Thu, 13 Jun 2024 23:31:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321505; cv=none; b=LWrjzUhq9+uzGH2PloL3Diow1YtetXryDDh49WrVRaHrs+Onw+tsUVoBEDoxYhvQYTTHcTyVkr/AogthfAzlVt/w3rSAHL4uYO+EuopoTClaTvlolmk+35T9cGeQgM9x0nj2EhsdkcX839bSc02Tc6pN6Y5BLGxE7wbnvdp9rBM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321505; c=relaxed/simple; bh=WzlwFsEv9w7LqbUcSSdQX4ToDtf6xWdluWXCCxpla5c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rnlRflsvUxcMQq7wU3hvqhUNNQ4y77CpyucAoJgzIxd5WXXnws3oam1m5VGS8XBYcaL9HgbDn3g6oxBe0HRzExKJHIWf7I6J5h3COlfou8P2J4zGv7SMNjptA1x1OhPVMuSPhL990VzwZ0bNlMwDSv+LnIW9T7XKJ6XzV4i2/0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=T/p00Dgp; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="T/p00Dgp" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-62a08092c4dso16809817b3.0 for ; Thu, 13 Jun 2024 16:31:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1718321502; x=1718926302; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xaVZORN1j0nvMl4WsYSKuVepgrGQPkpPHhBQuRfTtcQ=; b=T/p00Dgpwzo3juyIKh34jQHGfjBKdg8lG6jf2VskWLYHLBu8hvQUzS+mU2NV/rklDf 6+/2EatHyUIsnM9U7hXlfyQcg+nio9RH61fZKKVBkuWYv+/CWUagCthwJMKHmlGQMkim GK/wxgeqSk4DHtSjsNiPnDpDlPrnJnI0PBGA84cA2AZ75pFKs7o0r4X9sePMCrkVV5tT lowqjkerNHoG3hrlkmHKpNm15hOUd/8Kqq3ds7//VlHcDe6RRz6g7ZbCmdoqMglxV3ua wEgnhDCbrCLZBArJkIIDzAvewo1hcxP9bSoHCFGYp282BfiYvjDFTqHa3q8A/2HD9sAk /bbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718321502; x=1718926302; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xaVZORN1j0nvMl4WsYSKuVepgrGQPkpPHhBQuRfTtcQ=; b=efh3u3P2IVDLTfUzYq7vPHHKMJfUk9lX9I4pkIWypsJBZ4di7J/Vigpdb9QW/3XKz9 GTqRbQot1MplBCQid3gLTGbhcguctpLWmI46f5Hvgs1VTlehdhvx9Y/Gi5m4E/xwjavj on5lT0kX6jaAqBECuGt6yNA6ePpE57uzz6HogAOSTuRfpSAZhvS+D90EG/tAF7d3rTJ3 /rvniHSGuGUmKuPCedo2ekKbF35Is3Dp577ExzCum4NlJUYpGWAIICs+ylt+GqzEgWd+ dcDCqvw+YepbdhnvIalVn4yHZIkwkxd6u54Lhokv8CgnWs1CvVf1JC3inIHnX58b++5B 1tQw== X-Gm-Message-State: AOJu0Yxb1HKrlYsXU8lDcVG5BwjvHPZCJpJPju7RAiMC/EUKFsL4QWTe wrQCBujqGh66rgAeI9cpOqD7jecrd/nmVBZtWcadHhDPizqVHabAbFI79PpkleuJ0onw1KMjgV9 v414= X-Google-Smtp-Source: AGHT+IG78J8xgX2rJKMezh3c7CSxiQaKoHZ+wjp20/3COi96NO9K2DHe7xnuYyU7iTp9b+u7e3jS5w== X-Received: by 2002:a81:f90d:0:b0:627:de70:f2f8 with SMTP id 00721157ae682-63222956691mr8997837b3.14.1718321501736; Thu, 13 Jun 2024 16:31:41 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-441ef3d8b62sm10586731cf.11.2024.06.13.16.31.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jun 2024 16:31:41 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v5 3/4] sock: add MSG_ZEROCOPY notification mechanism based on msg_control Date: Thu, 13 Jun 2024 23:31:32 +0000 Message-Id: <20240613233133.2463193-4-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240613233133.2463193-1-zijianzhang@bytedance.com> References: <20240613233133.2463193-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang The MSG_ZEROCOPY flag enables copy avoidance for socket send calls. However, zerocopy is not a free lunch. Apart from the management of user pages, the combination of poll + recvmsg to receive notifications incurs unignorable overhead in the applications. The overhead of such sometimes might be more than the CPU savings from zerocopy. We try to solve this problem with a new notification mechanism based on msgcontrol. This new mechanism aims to reduce the overhead associated with receiving notifications by embedding them directly into user arguments passed with each sendmsg control message. By doing so, we can significantly reduce the complexity and overhead for managing notifications. In an ideal pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION msg_control, and the notification will be delivered as soon as possible. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- arch/alpha/include/uapi/asm/socket.h | 2 + arch/mips/include/uapi/asm/socket.h | 2 + arch/parisc/include/uapi/asm/socket.h | 2 + arch/sparc/include/uapi/asm/socket.h | 2 + include/net/sock.h | 2 +- include/uapi/asm-generic/socket.h | 2 + include/uapi/linux/socket.h | 10 +++++ net/core/sock.c | 63 ++++++++++++++++++++++++++- net/ipv4/ip_sockglue.c | 2 +- net/ipv6/datagram.c | 2 +- 10 files changed, 84 insertions(+), 5 deletions(-) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h index e94f621903fe..7761a4e0ea2c 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -140,6 +140,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h index 60ebaed28a4c..89edc51380f0 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -151,6 +151,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h index be264c2b1a11..2911b43e6a9d 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -132,6 +132,8 @@ #define SO_PASSPIDFD 0x404A #define SO_PEERPIDFD 0x404B +#define SCM_ZC_NOTIFICATION 0x404C + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h index 682da3714686..dc045e87cc8e 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -133,6 +133,8 @@ #define SO_PASSPIDFD 0x0055 #define SO_PEERPIDFD 0x0056 +#define SCM_ZC_NOTIFICATION 0x0057 + #if !defined(__KERNEL__) diff --git a/include/net/sock.h b/include/net/sock.h index b30ea0c342a6..f31c67d15c35 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -1799,7 +1799,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc, }; } -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg, struct sockcm_cookie *sockc); int sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct sockcm_cookie *sockc); diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 8ce8a39a1e5f..7474c8a244bc 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h index d3fcd3b5ec53..26bee6291c6c 100644 --- a/include/uapi/linux/socket.h +++ b/include/uapi/linux/socket.h @@ -2,6 +2,8 @@ #ifndef _UAPI_LINUX_SOCKET_H #define _UAPI_LINUX_SOCKET_H +#include + /* * Desired design of maximum size and alignment (see RFC2553) */ @@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage { #define SOCK_TXREHASH_DISABLED 0 #define SOCK_TXREHASH_ENABLED 1 +#define SOCK_ZC_INFO_MAX 16 + +struct zc_info_elem { + __u32 lo; + __u32 hi; + __u8 zerocopy; +}; + #endif /* _UAPI_LINUX_SOCKET_H */ diff --git a/net/core/sock.c b/net/core/sock.c index 69baddcfbd8c..d7d029f9aa0a 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, } EXPORT_SYMBOL(sock_alloc_send_pskb); -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg, struct sockcm_cookie *sockc) { u32 tsflags; @@ -2863,6 +2863,65 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, case SCM_RIGHTS: case SCM_CREDENTIALS: break; + case SCM_ZC_NOTIFICATION: { + struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX]; + int cmsg_data_len, zc_info_elem_num; + struct sk_buff_head *q, local_q; + struct sock_exterr_skb *serr; + unsigned long flags; + struct sk_buff *skb; + int ret, sz, i = 0; + + if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS) + return -EINVAL; + + cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr); + if (cmsg_data_len % sizeof(struct zc_info_elem)) + return -EINVAL; + + zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem); + if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX) + return -EINVAL; + + q = &sk->sk_error_queue; + skb_queue_head_init(&local_q); + spin_lock_irqsave(&q->lock, flags); + skb = skb_peek(q); + while (skb && i < zc_info_elem_num) { + struct sk_buff *skb_next = skb_peek_next(skb, q); + + serr = SKB_EXT_ERR(skb); + if (serr->ee.ee_errno == 0 && + serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) { + zc_info_kern[i].hi = serr->ee.ee_data; + zc_info_kern[i].lo = serr->ee.ee_info; + zc_info_kern[i].zerocopy = !(serr->ee.ee_code + & SO_EE_CODE_ZEROCOPY_COPIED); + __skb_unlink(skb, q); + __skb_queue_tail(&local_q, skb); + i++; + } + skb = skb_next; + } + spin_unlock_irqrestore(&q->lock, flags); + + if (i == 0) + break; + + sz = i * sizeof(struct zc_info_elem); + ret = put_cmsg_user_tx(msg, SOL_SOCKET, SCM_ZC_NOTIFICATION, sz, zc_info_kern); + + if (unlikely(ret)) { + spin_lock_irqsave(&q->lock, flags); + skb_queue_splice_init(&local_q, q); + spin_unlock_irqrestore(&q->lock, flags); + return -ret; + } + + while ((skb = __skb_dequeue(&local_q))) + consume_skb(skb); + break; + } default: return -EINVAL; } @@ -2881,7 +2940,7 @@ int sock_cmsg_send(struct sock *sk, struct msghdr *msg, return -EINVAL; if (cmsg->cmsg_level != SOL_SOCKET) continue; - ret = __sock_cmsg_send(sk, cmsg, sockc); + ret = __sock_cmsg_send(sk, msg, cmsg, sockc); if (ret) return ret; } diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c index cf377377b52d..6360b8ba9c84 100644 --- a/net/ipv4/ip_sockglue.c +++ b/net/ipv4/ip_sockglue.c @@ -267,7 +267,7 @@ int ip_cmsg_send(struct sock *sk, struct msghdr *msg, struct ipcm_cookie *ipc, } #endif if (cmsg->cmsg_level == SOL_SOCKET) { - err = __sock_cmsg_send(sk, cmsg, &ipc->sockc); + err = __sock_cmsg_send(sk, msg, cmsg, &ipc->sockc); if (err) return err; continue; diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c index fff78496803d..c9ae30acf895 100644 --- a/net/ipv6/datagram.c +++ b/net/ipv6/datagram.c @@ -777,7 +777,7 @@ int ip6_datagram_send_ctl(struct net *net, struct sock *sk, } if (cmsg->cmsg_level == SOL_SOCKET) { - err = __sock_cmsg_send(sk, cmsg, &ipc6->sockc); + err = __sock_cmsg_send(sk, msg, cmsg, &ipc6->sockc); if (err) return err; continue; From patchwork Thu Jun 13 23:31:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13697592 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 234E1156668 for ; Thu, 13 Jun 2024 23:31:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321507; cv=none; b=uMdKQwFjo5/VfI2RDcYmyiWnl2IIepp1cYx+GbzYyFCz6pFAYyZ3tKA+BPnSoz7XZDSdNMLaLkDN2acYMSi3zUxNKNBJ45byE/MkNXC8R0yrRCoEVTZ2pvm5ECv6oGfILr/odIaN3htbXeefG5w9Y3pNF6r6ihCGRQdD6QDrfjw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1718321507; c=relaxed/simple; bh=l759AUoZn+Lkd8NE19BHlW8jwuV/0b0Hmv2kt2g0v6E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AuEDQ4jMgCQMPtVvb1mMvX6DL1cSttdzdwPoWUIMF19eCGaHr8RAnhWZ56OtN+3UFYSPYTL40kXmb9mgJ9/X+184yGSv2i3k1i4ztpTdimrJjr//pZesJ9nE85Zl2udgInS3HutnTlh7uqL8hgVe5dvwsT31hXm8KZuIaMFO+/U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Nbxk1FHj; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Nbxk1FHj" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3c9cc681e4fso742630b6e.0 for ; Thu, 13 Jun 2024 16:31:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1718321503; x=1718926303; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0xJmOuJxFnxzNhLDPuw3aSaK2S5jGXnhr5CJos9rvB4=; b=Nbxk1FHjX83xK3y9hBcq6UjzpuBUDFglMwBlkDdRiOYP1SDaKMDgGM1NA8suzIkzro gO0EFPPgHXRaXSKUWBtewCAmTrht1Oym2ybR8ImlqjoVoxfA3b0aqaeOkwRjQHx5zf9o ci9Nzr/OlaV86H0mzP4+eBIK/FWOYkF7THexZQZ6Krg0x6vBU/UKTiN6N+3Q/U56FIbI 2srczLZZTdH5Y3yYiyTjnOztlKKRUNa2kfK0GXepkXaEaxNVn4jZ4/1RftvNrLzZezS4 eFXn3l2e9ZBfhPAtdMXXrS3dUSPJ8IK19x4W5+ExgPGpQ8ZRkEs48n3Rosts7ckuWSLp PPSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718321503; x=1718926303; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0xJmOuJxFnxzNhLDPuw3aSaK2S5jGXnhr5CJos9rvB4=; b=LQGkm65G7E7uyBstCxlQ9ioB0GIJRqMFRX4jUeWTQwINSRgLMtcJG3oLJ2H4AcvnNV 9T/b9NpELKJPJ7yyrGx1+LB5NqtkLlLW8mFxDdyixjHV69HBp8UhhvDFdZxIlGzQS/ac s9Ch8vbvgwLaR+QEmO80T3i7v5EkpesdXB6WXz6gId1LYWMZS5Elh09g8pL1nvCUV5nF iCGKF8JJ1dXlrVQmzI28u45oEbvMSUYQ7k8PNSXaYcF8hEgv9usW2gK6N8NPwU9oM6Tv H7vHHcbvHxPbl6QEA5PPPQNpauxALuxTtqjgAROjgBRH0FFrS7nRoNJ8ItJ0R8EZYRyt zZyQ== X-Gm-Message-State: AOJu0YzDt9ogSd004DwrXMrArUCyFAvcbUgoCVJpqK5yshupt6lEnWRI OM8ufIMzd8fIt3bdr9FZYwF+Nj6A++6r8Ovtlmzjwm2NwSEw/8CFSeaLAm816f66/LxvcKQKxLY 47Ms= X-Google-Smtp-Source: AGHT+IETgikJHQVfvmW3cFSO8NWheg/2pj6QojSAq7ARV/z9a3BESLBWb9SHAYdIKxCJYnno1RavJg== X-Received: by 2002:a05:6808:1884:b0:3d2:1b8a:be4b with SMTP id 5614622812f47-3d24e980c71mr1275046b6e.46.1718321503274; Thu, 13 Jun 2024 16:31:43 -0700 (PDT) Received: from n191-036-066.byted.org ([139.177.233.173]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-441ef3d8b62sm10586731cf.11.2024.06.13.16.31.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Jun 2024 16:31:42 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v5 4/4] selftests: add MSG_ZEROCOPY msg_control notification test Date: Thu, 13 Jun 2024 23:31:33 +0000 Message-Id: <20240613233133.2463193-5-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240613233133.2463193-1-zijianzhang@bytedance.com> References: <20240613233133.2463193-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang We update selftests/net/msg_zerocopy.c to accommodate the new mechanism. Test result from selftests/net/msg_zerocopy.c, cfg_notification_limit = 1, in this case the original method approximately aligns with the semantics of new one. In this case, the new flag has around 13% cpu savings in TCP and 18% cpu savings in UDP. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 5147 | 4885 | 7489 | 7854 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 5859 | 5505 | 9053 | 9236 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 113.83% | 112.69% | 120.88% | 117.59% | +---------------------+---------+---------+---------+---------+ cfg_notification_limit = 32, it means less poll + recvmsg overhead, the new mechanism performs 8% better in TCP. For UDP, no obvious performance gain is observed and sometimes may lead to degradation. Thus, if users don't need to retrieve the notification ASAP in UDP, the original mechanism is preferred. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 6272 | 6138 | 12138 | 10055 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 6774 | 6620 | 11504 | 10355 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 108.00% | 107.85% | 94.78% | 102.98% | +---------------------+---------+---------+---------+---------+ Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 109 ++++++++++++++++++-- tools/testing/selftests/net/msg_zerocopy.sh | 1 + 2 files changed, 102 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index 7ea5fb28c93d..b8a1002aa6ae 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* Evaluate MSG_ZEROCOPY * * Send traffic between two processes over one of the supported @@ -66,6 +67,10 @@ #define SO_ZEROCOPY 60 #endif +#ifndef SCM_ZC_NOTIFICATION +#define SCM_ZC_NOTIFICATION 78 +#endif + #ifndef SO_EE_CODE_ZEROCOPY_COPIED #define SO_EE_CODE_ZEROCOPY_COPIED 1 #endif @@ -74,6 +79,15 @@ #define MSG_ZEROCOPY 0x4000000 #endif +enum notification_type { + MSG_ZEROCOPY_NOTIFY_ERRQUEUE = 1, + MSG_ZEROCOPY_NOTIFY_SENDMSG = 2, +}; + +#define INVALID_ZEROCOPY_VAL 2 + +#define ZC_NOTIF_ARR_SZ (sizeof(struct zc_info_elem) * SOCK_ZC_INFO_MAX) + static int cfg_cork; static bool cfg_cork_mixed; static int cfg_cpu = -1; /* default: pin to last cpu */ @@ -85,14 +99,16 @@ static bool cfg_rx; static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; -static int cfg_notification_limit = 32; -static bool cfg_zerocopy; +static int cfg_notification_limit = 16; +static enum notification_type cfg_zerocopy; static socklen_t cfg_alen; static struct sockaddr_storage cfg_dst_addr; static struct sockaddr_storage cfg_src_addr; static char payload[IP_MAXPACKET]; +static char zc_ckbuf[CMSG_SPACE(ZC_NOTIF_ARR_SZ)]; +static bool added_zcopy_info; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; @@ -169,6 +185,25 @@ static int do_accept(int fd) return fd; } +static void add_zcopy_info(struct msghdr *msg) +{ + int i; + struct cmsghdr *cm; + struct zc_info_elem *zc_info; + + if (!msg->msg_control) + error(1, errno, "NULL user arg"); + cm = (struct cmsghdr *)msg->msg_control; + zc_info = (struct zc_info_elem *)CMSG_DATA(cm); + + cm->cmsg_len = CMSG_LEN(ZC_NOTIF_ARR_SZ); + cm->cmsg_level = SOL_SOCKET; + cm->cmsg_type = SCM_ZC_NOTIFICATION; + for (i = 0; i < SOCK_ZC_INFO_MAX; i++) + zc_info[i].zerocopy = INVALID_ZEROCOPY_VAL; + added_zcopy_info = true; +} + static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) { struct cmsghdr *cm; @@ -182,7 +217,8 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie)); } -static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) +static bool do_sendmsg(int fd, struct msghdr *msg, + enum notification_type do_zerocopy, int domain) { int ret, len, i, flags; static uint32_t cookie; @@ -200,6 +236,12 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) msg->msg_controllen = CMSG_SPACE(sizeof(cookie)); msg->msg_control = (struct cmsghdr *)ckbuf; add_zcopy_cookie(msg, ++cookie); + } else if (do_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG && + sends_since_notify >= cfg_notification_limit) { + memset(&msg->msg_control, 0, sizeof(msg->msg_control)); + msg->msg_controllen = sizeof(zc_ckbuf); + msg->msg_control = (struct cmsghdr *)zc_ckbuf; + add_zcopy_info(msg); } } @@ -218,7 +260,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) if (do_zerocopy && ret) expected_completions++; } - if (do_zerocopy && domain == PF_RDS) { + if (msg->msg_control) { msg->msg_control = NULL; msg->msg_controllen = 0; } @@ -392,6 +434,48 @@ static bool do_recvmsg_completion(int fd) return ret; } +static void do_recv_completions2(void) +{ + int i; + __u32 hi, lo, range; + __u8 zerocopy; + struct cmsghdr *cm = (struct cmsghdr *)zc_ckbuf; + struct zc_info_elem *zc_info = (struct zc_info_elem *)CMSG_DATA(cm); + + if (!added_zcopy_info) + return; + + added_zcopy_info = false; + for (i = 0; i < SOCK_ZC_INFO_MAX && zc_info[i].zerocopy != INVALID_ZEROCOPY_VAL; i++) { + struct zc_info_elem elem = zc_info[i]; + + hi = elem.hi; + lo = elem.lo; + zerocopy = elem.zerocopy; + range = hi - lo + 1; + + if (cfg_verbose && lo != next_completion) + fprintf(stderr, "gap: %u..%u does not append to %u\n", + lo, hi, next_completion); + next_completion = hi + 1; + + if (zerocopied == -1) + zerocopied = zerocopy; + else if (zerocopied != zerocopy) { + fprintf(stderr, "serr: inconsistent\n"); + zerocopied = zerocopy; + } + + completions += range; + + if (cfg_verbose >= 2) + fprintf(stderr, "completed: %u (h=%u l=%u)\n", + range, hi, lo); + } + + sends_since_notify -= i; +} + static bool do_recv_completion(int fd, int domain) { struct sock_extended_err *serr; @@ -553,11 +637,15 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); - if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE && + sends_since_notify >= cfg_notification_limit) do_recv_completions(fd, domain); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + do_recv_completions2(); + while (!do_poll(fd, POLLOUT)) { - if (cfg_zerocopy) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE) do_recv_completions(fd, domain); } @@ -715,7 +803,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -749,6 +837,9 @@ static void parse_opts(int argc, char **argv) case 'm': cfg_cork_mixed = true; break; + case 'n': + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_SENDMSG; + break; case 'p': cfg_port = strtoul(optarg, NULL, 0); break; @@ -768,7 +859,7 @@ static void parse_opts(int argc, char **argv) cfg_verbose++; break; case 'z': - cfg_zerocopy = true; + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_ERRQUEUE; break; } } @@ -779,6 +870,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-D required for PF_RDS\n"); if (!cfg_rx && !saddr) error(1, 0, "-S required for PF_RDS\n"); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + error(1, 0, "PF_RDS does not support MSG_ZEROCOPY_NOTIFY_SENDMSG"); } setup_sockaddr(cfg_family, daddr, &cfg_dst_addr); setup_sockaddr(cfg_family, saddr, &cfg_src_addr); diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh index 89c22f5320e0..022a6936d86f 100755 --- a/tools/testing/selftests/net/msg_zerocopy.sh +++ b/tools/testing/selftests/net/msg_zerocopy.sh @@ -118,4 +118,5 @@ do_test() { do_test "${EXTRA_ARGS}" do_test "-z ${EXTRA_ARGS}" +do_test "-n ${EXTRA_ARGS}" echo ok