From patchwork Mon May 13 21:17:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13663687 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDA068248B for ; Mon, 13 May 2024 21:18:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715635086; cv=none; b=G7IojAEw7bMboG5sxyJPbc9eRVlnDrPdnvN/C7wcnY0JNT2YZbBv4y3tnXa6Ncm2llNdiTZv3PmzZ3lTRtjb6Gxas1jAk21xrTIU6IJH0NMbv0R4XZOrg3dmrOSuAxl6wLCb9PJKbd47DWtOqjr8TulmKpgQ+LsVBll0BRt5lec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715635086; c=relaxed/simple; bh=UNwIgy0eaP1U/PjG+dxmb5r6nXFEuXCnnd8Z+VVWbpM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=R/cd5ltTVZ0FRGuoEGZvqIgr/+2JM/0LN/O5ruCJyBIyVEjnFFrw5RNHyeZcfAWRZtZu3LD+AGxGcGgl7odBv/zBlGXem4YKDAeej6jSCdEbkzhrI6cNfxAiYmdQPhuEpUclmol3bK4jdIe3/l3nOXi+qzU4j6ZjViVJMN1yrA0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=HR2nsJQi; arc=none smtp.client-ip=209.85.160.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="HR2nsJQi" Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-43df23e034cso40998341cf.0 for ; Mon, 13 May 2024 14:18:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1715635083; x=1716239883; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pq1qBmFA90YYxxuCBnsZZeZJpNA0DQPz8KGklT3/y7Q=; b=HR2nsJQiLtinzf3J3ACiElH+2mY1IcmKO2LyGoyLttmZfJhM2W33vnEHqkwy0xZWbi 4P37rbSsOKGJuVRcsRxZiGn+5Lgp5kd0pkOPGM+423eqvNz/k83WQpE6VL47Y8i4yvIv vAuws7aUd8Ek4WZdSm6tJ/Yg8Z9BMjJvuWsCM1MKboG+yFH1hGedC79f4lEONdmCBgDc hfVmBGqmBuq3seQVo3XuhvVc6+bFN+dnGOACVnjJP/0V65x0tbu+qflqVC6diqMy+4ri WeEF33EN6E2WdTWSDI2Fnl2SD6E38MQ+tyjVgndsDRLjlXHPEYeD1tl4WumsfTTitqoo UhBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715635083; x=1716239883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pq1qBmFA90YYxxuCBnsZZeZJpNA0DQPz8KGklT3/y7Q=; b=buNSRnJFJ3nSihivazq86PeKL1AZwTScg1ak+szfmxihaMCq2mvv6d8oGjRrQ9dQut XIF2TEAaG2UvfpSHXaF3ykqspsE0CkMSkjy3hddaBzJbvS44uKqKx73mo0xKH46+gfW3 EJpcfal4uafFFYl3GIoxCXLHVJtkaLoQ75gc6RLafmE26zIVhkcTHrSKGTG1w88F3s5s P+KwuTQXRoaMk7i5khSx09VED/fOqmmVgu/fHyA3yj8Cb9yvL/Kb6om0GsDDP7s46ZWD Gw6R2wh74u7EZu7if28fowqwBvo0/aZicqCpW4iV8iatfHVTj2xhie3oHlKmJJ3CKSRe pxrg== X-Gm-Message-State: AOJu0Yya2pKmAGbmtGjjQv40yNmA2Ud80KzZ6c9Wrub2MtVLeplOs+qL mA+J3MAHzDIpuJV0vFrIqltlwYzoHaZMDxR/HQ3MCT0rdWpA66nXZbH8jRptRQP2w0pHTG8BxbA V X-Google-Smtp-Source: AGHT+IEP1He6oNspKxV17oRk3zESfdd4koiSViplxue55eM+8LGMRWiOKIEEFPkAEhuigu85ynf9ng== X-Received: by 2002:a05:622a:109:b0:43a:3502:8446 with SMTP id d75a77b69052e-43dec39be29mr249913861cf.28.1715635082834; Mon, 13 May 2024 14:18:02 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e184af783sm18340811cf.17.2024.05.13.14.18.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 May 2024 14:18:02 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v4 1/3] selftests: fix OOM problem in msg_zerocopy selftest Date: Mon, 13 May 2024 21:17:53 +0000 Message-Id: <20240513211755.2751955-2-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240513211755.2751955-1-zijianzhang@bytedance.com> References: <20240513211755.2751955-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang In selftests/net/msg_zerocopy.c, it has a while loop keeps calling sendmsg on a socket with MSG_ZEROCOPY flag, and it will recv the notifications until the socket is not writable. Typically, it will start the receiving process after around 30+ sendmsgs. However, because of the commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale") the sender is always writable and does not get any chance to run recv notifications. The selftest always exits with OUT_OF_MEMORY because the memory used by opt_skb exceeds the core.sysctl_optmem_max. According to our experiments, this problem can be mitigated by open the DEBUG_LOCKDEP configuration for the kernel. But it will makes the notifications disordered even in good commits before commit dfa2f0483360 ("tcp: get rid of sysctl_tcp_adv_win_scale"). We introduce "cfg_notification_limit" to force sender to receive notifications after some number of sendmsgs. And, notifications may not come in order, because of the reason we present above. We have order checking code managed by cfg_verbose. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index bdc03a2097e8..7ea5fb28c93d 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -85,6 +85,7 @@ static bool cfg_rx; static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; +static int cfg_notification_limit = 32; static bool cfg_zerocopy; static socklen_t cfg_alen; @@ -95,6 +96,7 @@ static char payload[IP_MAXPACKET]; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; +static uint32_t sends_since_notify; static unsigned long gettimeofday_ms(void) { @@ -208,6 +210,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) error(1, errno, "send"); if (cfg_verbose && ret != len) fprintf(stderr, "send: ret=%u != %u\n", ret, len); + sends_since_notify++; if (len) { packets++; @@ -435,7 +438,7 @@ static bool do_recv_completion(int fd, int domain) /* Detect notification gaps. These should not happen often, if at all. * Gaps can occur due to drops, reordering and retransmissions. */ - if (lo != next_completion) + if (cfg_verbose && lo != next_completion) fprintf(stderr, "gap: %u..%u does not append to %u\n", lo, hi, next_completion); next_completion = hi + 1; @@ -460,6 +463,7 @@ static bool do_recv_completion(int fd, int domain) static void do_recv_completions(int fd, int domain) { while (do_recv_completion(fd, domain)) {} + sends_since_notify = 0; } /* Wait for all remaining completions on the errqueue */ @@ -549,6 +553,9 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); + if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + do_recv_completions(fd, domain); + while (!do_poll(fd, POLLOUT)) { if (cfg_zerocopy) do_recv_completions(fd, domain); @@ -708,7 +715,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -736,6 +743,9 @@ static void parse_opts(int argc, char **argv) if (cfg_ifindex == 0) error(1, errno, "invalid iface: %s", optarg); break; + case 'l': + cfg_notification_limit = strtoul(optarg, NULL, 0); + break; case 'm': cfg_cork_mixed = true; break; From patchwork Mon May 13 21:17:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13663688 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7106F8287E for ; Mon, 13 May 2024 21:18:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715635088; cv=none; b=DZpznBWQWc8X3mzQEoAdkW4eB8UFZR3q1NcbBO71S1CC5N1J5ptX5c/Kg4OhYpXVqtk8AcBQ0vkchpRQ0gh97LjEWqpd+KIU8MGLiiIw8eFfpSqbr58cGktERtCrXteHBUI7BCvmkNPz1KxbkybcV/4cYlYXHc6yQYYIGimQ3aA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715635088; c=relaxed/simple; bh=D5CwM1zLihZr1ncMejfbrURfB0Wr3nGe8hLOMwDf2F8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Kelvje/fOztzdsxsnWMSi9kkhJio8KEYlN2Q1zy7Ohqi3E7BBeKidhs4rJBTaVXccAF8Z9U5twB7Lp973mjB4H0zIhm/bPLwBbDS0NR/FwsonkcDWL+L56avLG+S+ZurG4+o1X6y0w+N5tJH3pi0+YZ8BylyaT5WneiWyZ2JIwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=CkR3ZXeu; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="CkR3ZXeu" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-61bed738438so49546657b3.2 for ; Mon, 13 May 2024 14:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1715635084; x=1716239884; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+xrJ0Hjkib/Aqyz9aStozV3dkl0zmuhlShNv7pOvhE0=; b=CkR3ZXeuUTRd7B5HXr4wrpoZZ/WqOXZiMhGG2SA5znVWpp2cSw32Zdianxju6ozurE LSDi+wNe9oPCrptKhicZ+eJsIUyz2/gH4CrT1YlXDte3TjYtBwwaQ4gVqpt1W4USygBK zoLgAYa1qFnQwZ86kP4Fk4X7nLILPJwO0BP7ICPd2W/xEMD5Y07Ly4bYnczyEceXaG1m 3xSzs/daNiJ89OzPgUj70pJa021MxQCcYZMV+LsWy2bcS9J1p+NGqPSUfgE+vwuhRm0p y91WeF0x/7dVQM+oyJuRspKQAahn7qy+E4AVUz8/D0pfuqjqTzsf5T52kEOGXrtItcWB jMGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715635084; x=1716239884; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+xrJ0Hjkib/Aqyz9aStozV3dkl0zmuhlShNv7pOvhE0=; b=lGw6oq4gyh4UjWBff31Me4KNSCFSUtqmRmFICrIxp4zHE1qXByWf/oOMM51z1NZBnj LJTbQq0Np3hegze75hw8zM9rVnhGL2MohsHkMSk/EBl3wNbKrPFyPY9t8ZPzqauRXYhD gxuudMfzBkrHLkMgiYyM/g5rUsnfVr0d7c12fwh8Z361Gt2G5OskKDp1dOuu9zZcftpY C4hSSovdAY8mRqE5v3vz9ELLpevc64SIKZyfhXCuYK1jlDnL99DoTvuEcpFbQ9MyrgTN ORDWD2TMCIb+FeVptaPTN6hmSM51CKWMybwWMTlwQlcdfbTRYMuihOOJA5xceDgcb8pN 74dA== X-Gm-Message-State: AOJu0Yya2VRVlvq5SzRe9Iwrn45XQnnVbJLnwFswXpZZLuh+tFyORDsW jMLdRyBDke/CNet2rVUaL8nbiKYJsODzYcaTHconCPqmiOLFTmTU8TE7z3VFWQKPREapXPV2xrg / X-Google-Smtp-Source: AGHT+IHkaJdGtVBzdPjvRQSowjlaIJy1cUAS9MjWidrEesguJV0ywL3x//oO9s67GnH8pUb9CnvdsA== X-Received: by 2002:a05:690c:b18:b0:614:8185:2946 with SMTP id 00721157ae682-622cf345ef1mr63555977b3.42.1715635084518; Mon, 13 May 2024 14:18:04 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e184af783sm18340811cf.17.2024.05.13.14.18.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 May 2024 14:18:04 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v4 2/3] sock: add MSG_ZEROCOPY notification mechanism based on msg_control Date: Mon, 13 May 2024 21:17:54 +0000 Message-Id: <20240513211755.2751955-3-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240513211755.2751955-1-zijianzhang@bytedance.com> References: <20240513211755.2751955-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang The MSG_ZEROCOPY flag enables copy avoidance for socket send calls. However, zerocopy is not a free lunch. Apart from the management of user pages, the combination of poll + recvmsg to receive notifications incurs unignorable overhead in the applications. The overhead of such sometimes might be more than the CPU savings from zerocopy. We try to solve this problem with a new notification mechanism based on msgcontrol. This new mechanism aims to reduce the overhead associated with receiving notifications by embedding them directly into user arguments passed with each sendmsg control message. By doing so, we can significantly reduce the complexity and overhead for managing notifications. In an ideal pattern, the user will keep calling sendmsg with SCM_ZC_NOTIFICATION msg_control, and the notification will be delivered as soon as possible. Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- arch/alpha/include/uapi/asm/socket.h | 2 + arch/mips/include/uapi/asm/socket.h | 2 + arch/parisc/include/uapi/asm/socket.h | 2 + arch/sparc/include/uapi/asm/socket.h | 2 + include/uapi/asm-generic/socket.h | 2 + include/uapi/linux/socket.h | 10 +++++ net/core/sock.c | 65 +++++++++++++++++++++++++++ 7 files changed, 85 insertions(+) diff --git a/arch/alpha/include/uapi/asm/socket.h b/arch/alpha/include/uapi/asm/socket.h index e94f621903fe..7761a4e0ea2c 100644 --- a/arch/alpha/include/uapi/asm/socket.h +++ b/arch/alpha/include/uapi/asm/socket.h @@ -140,6 +140,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/mips/include/uapi/asm/socket.h b/arch/mips/include/uapi/asm/socket.h index 60ebaed28a4c..89edc51380f0 100644 --- a/arch/mips/include/uapi/asm/socket.h +++ b/arch/mips/include/uapi/asm/socket.h @@ -151,6 +151,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/parisc/include/uapi/asm/socket.h b/arch/parisc/include/uapi/asm/socket.h index be264c2b1a11..2911b43e6a9d 100644 --- a/arch/parisc/include/uapi/asm/socket.h +++ b/arch/parisc/include/uapi/asm/socket.h @@ -132,6 +132,8 @@ #define SO_PASSPIDFD 0x404A #define SO_PEERPIDFD 0x404B +#define SCM_ZC_NOTIFICATION 0x404C + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 diff --git a/arch/sparc/include/uapi/asm/socket.h b/arch/sparc/include/uapi/asm/socket.h index 682da3714686..dc045e87cc8e 100644 --- a/arch/sparc/include/uapi/asm/socket.h +++ b/arch/sparc/include/uapi/asm/socket.h @@ -133,6 +133,8 @@ #define SO_PASSPIDFD 0x0055 #define SO_PEERPIDFD 0x0056 +#define SCM_ZC_NOTIFICATION 0x0057 + #if !defined(__KERNEL__) diff --git a/include/uapi/asm-generic/socket.h b/include/uapi/asm-generic/socket.h index 8ce8a39a1e5f..7474c8a244bc 100644 --- a/include/uapi/asm-generic/socket.h +++ b/include/uapi/asm-generic/socket.h @@ -135,6 +135,8 @@ #define SO_PASSPIDFD 76 #define SO_PEERPIDFD 77 +#define SCM_ZC_NOTIFICATION 78 + #if !defined(__KERNEL__) #if __BITS_PER_LONG == 64 || (defined(__x86_64__) && defined(__ILP32__)) diff --git a/include/uapi/linux/socket.h b/include/uapi/linux/socket.h index d3fcd3b5ec53..16911bca45f3 100644 --- a/include/uapi/linux/socket.h +++ b/include/uapi/linux/socket.h @@ -2,6 +2,8 @@ #ifndef _UAPI_LINUX_SOCKET_H #define _UAPI_LINUX_SOCKET_H +#include + /* * Desired design of maximum size and alignment (see RFC2553) */ @@ -35,4 +37,12 @@ struct __kernel_sockaddr_storage { #define SOCK_TXREHASH_DISABLED 0 #define SOCK_TXREHASH_ENABLED 1 +#define SOCK_ZC_INFO_MAX 64 + +struct zc_info_elem { + __u32 lo; + __u32 hi; + __u8 zerocopy; +}; + #endif /* _UAPI_LINUX_SOCKET_H */ diff --git a/net/core/sock.c b/net/core/sock.c index 8d6e638b5426..eafa98c70d9a 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2842,6 +2842,71 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg, case SCM_RIGHTS: case SCM_CREDENTIALS: break; + case SCM_ZC_NOTIFICATION: { + struct zc_info_elem zc_info_kern[SOCK_ZC_INFO_MAX]; + int cmsg_data_len, zc_info_elem_num; + struct sk_buff_head *q, local_q; + struct sock_exterr_skb *serr; + struct sk_buff *skb, *tmp; + void __user *usr_addr; + unsigned long flags; + int ret, i = 0; + + if (!sock_flag(sk, SOCK_ZEROCOPY) || sk->sk_family == PF_RDS) + return -EINVAL; + + cmsg_data_len = cmsg->cmsg_len - sizeof(struct cmsghdr); + if (cmsg_data_len % sizeof(struct zc_info_elem)) + return -EINVAL; + + zc_info_elem_num = cmsg_data_len / sizeof(struct zc_info_elem); + if (!zc_info_elem_num || zc_info_elem_num > SOCK_ZC_INFO_MAX) + return -EINVAL; + + if (in_compat_syscall()) + usr_addr = compat_ptr(*(compat_uptr_t *)CMSG_DATA(cmsg)); + else + usr_addr = (void __user *)*(void **)CMSG_DATA(cmsg); + if (!access_ok(usr_addr, cmsg_data_len)) + return -EFAULT; + + q = &sk->sk_error_queue; + skb_queue_head_init(&local_q); + spin_lock_irqsave(&q->lock, flags); + skb = skb_peek(q); + while (skb && i < zc_info_elem_num) { + struct sk_buff *skb_next = skb_peek_next(skb, q); + + serr = SKB_EXT_ERR(skb); + if (serr->ee.ee_errno == 0 && + serr->ee.ee_origin == SO_EE_ORIGIN_ZEROCOPY) { + zc_info_kern[i].hi = serr->ee.ee_data; + zc_info_kern[i].lo = serr->ee.ee_info; + zc_info_kern[i].zerocopy = !(serr->ee.ee_code + & SO_EE_CODE_ZEROCOPY_COPIED); + __skb_unlink(skb, q); + __skb_queue_tail(&local_q, skb); + i++; + } + skb = skb_next; + } + spin_unlock_irqrestore(&q->lock, flags); + + ret = copy_to_user(usr_addr, + zc_info_kern, + i * sizeof(struct zc_info_elem)); + + if (unlikely(ret)) { + spin_lock_irqsave(&q->lock, flags); + skb_queue_splice_init(&local_q, q); + spin_unlock_irqrestore(&q->lock, flags); + return -EFAULT; + } + + while ((skb = __skb_dequeue(&local_q))) + consume_skb(skb); + break; + } default: return -EINVAL; } From patchwork Mon May 13 21:17:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zijian Zhang X-Patchwork-Id: 13663689 X-Patchwork-Delegate: kuba@kernel.org Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC3B382D94 for ; Mon, 13 May 2024 21:18:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715635089; cv=none; b=PtyQKPrHEP5G9TZ5qRLbqzfC9ETLi46bgGILWhv3LO1K3AW5aXYzKfSCENLsdWLZK2BkMpxXWWiWQwwkbGvUVbIa0FnaQAx2TrONCNweZZauQ5xRIaab+6X89FldYXmKqxUQoJttG0j+DrOVVHtSmjYl9qjspWfaMV45RylwVGI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715635089; c=relaxed/simple; bh=GgHXzikhDNfXD8YMRl174qa2Asj3e6hEBjHELFLc7jA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ABb5CmKXdjX1d4xgLhpT1lTx9N8zvdB7DmvmaKXtVir4CA7gOVTYkWr53vSL0DWgETR5ihmVG2oNg/vg2gtTckjK1WFY0JrNcwgqBfHEvLSGuCNDy19L27aOYEHUPIGsWB18Pa+rPcIZUrSmwwHQ2RpeudS9kI71wnVRAcklwu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=d7hZ4pXn; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="d7hZ4pXn" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-792c7704e09so324558185a.1 for ; Mon, 13 May 2024 14:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1715635086; x=1716239886; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T0DZW+CrHnVmpn7+LqCUe4u416/Fr3mRFM0ji+8pX/g=; b=d7hZ4pXnuamyvb1jHlMnZFzE/95RwmLb6o/8FZ8ubRp6CPfZWEbM7j9Zwfa6ZuQYnk Z3m7mMKZM8ajnJ5dPldDqZC8jNMuCSrjIyyiBvXvVZqI/3fmSc9QKIg7Ry188XbhGqXh HoJxDedN5Mb6FBHgQqeRwO5TDhT/jHq5kDRfTuTlhUZZjq19eMjUzQr2nqOP7DfXGvBF tYupoTH9nRhKoegccJ0kSrWjoDsrWvFw/IIhcDtfTyTHWswbZkP1VneMY+M3E+WAmZbT O6PogUxUtr/x/8/dzAePc0YJ1Lt5JNkp5K+YFfDMA5Pu8olMdfGWkJjI+I8RQ1d7Ie0j NfZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715635086; x=1716239886; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T0DZW+CrHnVmpn7+LqCUe4u416/Fr3mRFM0ji+8pX/g=; b=CrAg1mtp60d+xXA/3mb4oDAQMPn+ZZLQ9RqJ3xrdGXvze7ZvL2rwtgF6ZzbLNkwDFN 1d2yH2baAbNU/TU6bjl4nukDM+NpyWWLK58awnAnQnmwmjIHeCMyjHPpkc01dWAvSEot htYpbsU+344JoaBN2DT/pkuFIf6Ah19gwvg9+H+Obkd9+om2vugbpzwfjif2Bjq77cAn LQLYXPWYSZOG19DwqAW5RLppoZwSAAnHm4stzz5TrwW2Q0bzWOYF8T64MsXvW7ZbsNd8 AuYlDKeCxJ5/a2ZNLCtz/mtHP86rRxIIzyoMqrEB1a4O4iweZLYETu1uPg8V2wE+kvde eGtg== X-Gm-Message-State: AOJu0Yxs7ixm9FHCd4VLs4oxhg9Zq20yWhRSTggQCi3ERHp31JYsvDk/ IyGDRL35X8J6xwSBYsmIsTBN8OPNIcbNRWfAY+1BQ+LSBUIsZiYvXbn/7wuLZfGFbOq7LFG9PXJ l X-Google-Smtp-Source: AGHT+IE1U5NnYiDu6EAkp/mQpbyyd4BouJYJQJF6pbjZRG6+pMBHGnXS/hMpWdIUSyGvRAfG4sIpsg== X-Received: by 2002:a0c:eec6:0:b0:6a0:e8ed:49a6 with SMTP id 6a1803df08f44-6a15cc96cedmr249121086d6.27.1715635085812; Mon, 13 May 2024 14:18:05 -0700 (PDT) Received: from n191-036-066.byted.org ([130.44.212.94]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e184af783sm18340811cf.17.2024.05.13.14.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 May 2024 14:18:05 -0700 (PDT) From: zijianzhang@bytedance.com To: netdev@vger.kernel.org Cc: edumazet@google.com, willemdebruijn.kernel@gmail.com, cong.wang@bytedance.com, xiaochun.lu@bytedance.com, Zijian Zhang Subject: [PATCH net-next v4 3/3] selftests: add MSG_ZEROCOPY msg_control notification test Date: Mon, 13 May 2024 21:17:55 +0000 Message-Id: <20240513211755.2751955-4-zijianzhang@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20240513211755.2751955-1-zijianzhang@bytedance.com> References: <20240513211755.2751955-1-zijianzhang@bytedance.com> Precedence: bulk X-Mailing-List: netdev@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Patchwork-Delegate: kuba@kernel.org From: Zijian Zhang We update selftests/net/msg_zerocopy.c to accommodate the new mechanism. Test result from selftests/net/msg_zerocopy.c, cfg_notification_limit = 1, in this case MSG_ZEROCOPY approximately aligns with the semantics of MSG_ZEROCOPY_UARG. In this case, the new flag has around 13% cpu savings in TCP and 18% cpu savings in UDP. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 5147 | 4885 | 7489 | 7854 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 5859 | 5505 | 9053 | 9236 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 113.83% | 112.69% | 120.88% | 117.59% | +---------------------+---------+---------+---------+---------+ cfg_notification_limit = 32, it means less poll + recvmsg overhead, the new mechanism performs 8% better in TCP. For UDP, no obvious performance gain is observed and sometimes may lead to degradation. Thus, if users don't need to retrieve the notification ASAP in UDP, the original mechanism is preferred. +---------------------+---------+---------+---------+---------+ | Test Type / Protocol| TCP v4 | TCP v6 | UDP v4 | UDP v6 | +---------------------+---------+---------+---------+---------+ | ZCopy (MB) | 6272 | 6138 | 12138 | 10055 | +---------------------+---------+---------+---------+---------+ | New ZCopy (MB) | 6774 | 6620 | 11504 | 10355 | +---------------------+---------+---------+---------+---------+ | New ZCopy / ZCopy | 108.00% | 107.85% | 94.78% | 102.98% | +---------------------+---------+---------+---------+---------+ Signed-off-by: Zijian Zhang Signed-off-by: Xiaochun Lu --- tools/testing/selftests/net/msg_zerocopy.c | 106 ++++++++++++++++++-- tools/testing/selftests/net/msg_zerocopy.sh | 1 + 2 files changed, 100 insertions(+), 7 deletions(-) diff --git a/tools/testing/selftests/net/msg_zerocopy.c b/tools/testing/selftests/net/msg_zerocopy.c index 7ea5fb28c93d..d71477ee4d60 100644 --- a/tools/testing/selftests/net/msg_zerocopy.c +++ b/tools/testing/selftests/net/msg_zerocopy.c @@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* Evaluate MSG_ZEROCOPY * * Send traffic between two processes over one of the supported @@ -66,6 +67,10 @@ #define SO_ZEROCOPY 60 #endif +#ifndef SCM_ZC_NOTIFICATION +#define SCM_ZC_NOTIFICATION 78 +#endif + #ifndef SO_EE_CODE_ZEROCOPY_COPIED #define SO_EE_CODE_ZEROCOPY_COPIED 1 #endif @@ -74,6 +79,15 @@ #define MSG_ZEROCOPY 0x4000000 #endif +enum notification_type { + MSG_ZEROCOPY_NOTIFY_ERRQUEUE = 1, + MSG_ZEROCOPY_NOTIFY_SENDMSG = 2, +}; + +#define SOCK_ZC_INFO_NUM 8 + +#define INVALID_ZEROCOPY_VAL 2 + static int cfg_cork; static bool cfg_cork_mixed; static int cfg_cpu = -1; /* default: pin to last cpu */ @@ -86,13 +100,16 @@ static int cfg_runtime_ms = 4200; static int cfg_verbose; static int cfg_waittime_ms = 500; static int cfg_notification_limit = 32; -static bool cfg_zerocopy; +static enum notification_type cfg_zerocopy; static socklen_t cfg_alen; static struct sockaddr_storage cfg_dst_addr; static struct sockaddr_storage cfg_src_addr; static char payload[IP_MAXPACKET]; +static char zc_ckbuf[CMSG_SPACE(sizeof(void *))]; +static struct zc_info_elem zc_info[SOCK_ZC_INFO_NUM]; +static struct zc_info_elem *zc_info_ptr = zc_info; static long packets, bytes, completions, expected_completions; static int zerocopied = -1; static uint32_t next_completion; @@ -169,6 +186,26 @@ static int do_accept(int fd) return fd; } +static void add_zcopy_info(struct msghdr *msg) +{ + int i; + struct cmsghdr *cm; + + if (!msg->msg_control) + error(1, errno, "NULL user arg"); + cm = (void *)msg->msg_control; + /* Although only the address of the array will be written into the + * zc_ckbuf, we assign cmsg_len to CMSG_LEN(sizeof(zc_info)) to specify + * the length of the array. + */ + cm->cmsg_len = CMSG_LEN(sizeof(zc_info)); + cm->cmsg_level = SOL_SOCKET; + cm->cmsg_type = SCM_ZC_NOTIFICATION; + memcpy(CMSG_DATA(cm), &zc_info_ptr, sizeof(zc_info_ptr)); + for (i = 0; i < SOCK_ZC_INFO_NUM; i++) + zc_info[i].zerocopy = INVALID_ZEROCOPY_VAL; +} + static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) { struct cmsghdr *cm; @@ -182,7 +219,8 @@ static void add_zcopy_cookie(struct msghdr *msg, uint32_t cookie) memcpy(CMSG_DATA(cm), &cookie, sizeof(cookie)); } -static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) +static bool do_sendmsg(int fd, struct msghdr *msg, + enum notification_type do_zerocopy, int domain) { int ret, len, i, flags; static uint32_t cookie; @@ -200,6 +238,15 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) msg->msg_controllen = CMSG_SPACE(sizeof(cookie)); msg->msg_control = (struct cmsghdr *)ckbuf; add_zcopy_cookie(msg, ++cookie); + } else if (do_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) { + memset(&msg->msg_control, 0, sizeof(msg->msg_control)); + /* Although only the address of the array will be written into the + * zc_ckbuf, msg_controllen must be larger or equal than any cmsg_len + * in it. Otherwise, we will get -EINVAL. + */ + msg->msg_controllen = CMSG_SPACE(sizeof(zc_info)); + msg->msg_control = (struct cmsghdr *)zc_ckbuf; + add_zcopy_info(msg); } } @@ -218,7 +265,7 @@ static bool do_sendmsg(int fd, struct msghdr *msg, bool do_zerocopy, int domain) if (do_zerocopy && ret) expected_completions++; } - if (do_zerocopy && domain == PF_RDS) { + if (msg->msg_control) { msg->msg_control = NULL; msg->msg_controllen = 0; } @@ -392,6 +439,42 @@ static bool do_recvmsg_completion(int fd) return ret; } +static void do_recv_completions2(void) +{ + int i; + __u32 hi, lo, range; + __u8 zerocopy; + + for (i = 0; zc_info[i].zerocopy != INVALID_ZEROCOPY_VAL; i++) { + struct zc_info_elem elem = zc_info[i]; + + hi = elem.hi; + lo = elem.lo; + zerocopy = elem.zerocopy; + range = hi - lo + 1; + + if (cfg_verbose && lo != next_completion) + fprintf(stderr, "gap: %u..%u does not append to %u\n", + lo, hi, next_completion); + next_completion = hi + 1; + + if (zerocopied == -1) + zerocopied = zerocopy; + else if (zerocopied != zerocopy) { + fprintf(stderr, "serr: inconsistent\n"); + zerocopied = zerocopy; + } + + completions += range; + + if (cfg_verbose >= 2) + fprintf(stderr, "completed: %u (h=%u l=%u)\n", + range, hi, lo); + } + + sends_since_notify -= i; +} + static bool do_recv_completion(int fd, int domain) { struct sock_extended_err *serr; @@ -553,11 +636,15 @@ static void do_tx(int domain, int type, int protocol) else do_sendmsg(fd, &msg, cfg_zerocopy, domain); - if (cfg_zerocopy && sends_since_notify >= cfg_notification_limit) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE && + sends_since_notify >= cfg_notification_limit) do_recv_completions(fd, domain); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + do_recv_completions2(); + while (!do_poll(fd, POLLOUT)) { - if (cfg_zerocopy) + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_ERRQUEUE) do_recv_completions(fd, domain); } @@ -715,7 +802,7 @@ static void parse_opts(int argc, char **argv) cfg_payload_len = max_payload_len; - while ((c = getopt(argc, argv, "46c:C:D:i:l:mp:rs:S:t:vz")) != -1) { + while ((c = getopt(argc, argv, "46c:C:D:i:l:mnp:rs:S:t:vz")) != -1) { switch (c) { case '4': if (cfg_family != PF_UNSPEC) @@ -749,6 +836,9 @@ static void parse_opts(int argc, char **argv) case 'm': cfg_cork_mixed = true; break; + case 'n': + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_SENDMSG; + break; case 'p': cfg_port = strtoul(optarg, NULL, 0); break; @@ -768,7 +858,7 @@ static void parse_opts(int argc, char **argv) cfg_verbose++; break; case 'z': - cfg_zerocopy = true; + cfg_zerocopy = MSG_ZEROCOPY_NOTIFY_ERRQUEUE; break; } } @@ -779,6 +869,8 @@ static void parse_opts(int argc, char **argv) error(1, 0, "-D required for PF_RDS\n"); if (!cfg_rx && !saddr) error(1, 0, "-S required for PF_RDS\n"); + if (cfg_zerocopy == MSG_ZEROCOPY_NOTIFY_SENDMSG) + error(1, 0, "PF_RDS does not support MSG_ZEROCOPY_NOTIFY_SENDMSG"); } setup_sockaddr(cfg_family, daddr, &cfg_dst_addr); setup_sockaddr(cfg_family, saddr, &cfg_src_addr); diff --git a/tools/testing/selftests/net/msg_zerocopy.sh b/tools/testing/selftests/net/msg_zerocopy.sh index 89c22f5320e0..022a6936d86f 100755 --- a/tools/testing/selftests/net/msg_zerocopy.sh +++ b/tools/testing/selftests/net/msg_zerocopy.sh @@ -118,4 +118,5 @@ do_test() { do_test "${EXTRA_ARGS}" do_test "-z ${EXTRA_ARGS}" +do_test "-n ${EXTRA_ARGS}" echo ok