From patchwork Mon Feb 26 19:21:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572739 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B996C12F39C for ; Mon, 26 Feb 2024 19:25:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975509; cv=none; b=SeHkWg+NOqIhvhaKQWKC9DiTo4C3nkCUg9dPefseZiB/mNkBLwgzaiXjD/4F0WvEtz8ucFrTp/oLILdHqT1Zn+l7fQmVGKe4NpgwsjzgnyZp/CodryD6RFhpiskYc38AhHLS7bJ1iGCQfHMRS+uXDdZfXmF4qEnjzJcYB/Cnegg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975509; c=relaxed/simple; bh=2ayUOfaEYiRowwWtv7K6yBwCX1wE9nF3iDBASOdcNs0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IPW7Z3bXNIqhxLbDUEuXbV1odZpGVDNL78ImPb+ofiT6IY2DMJFykhXYVQuUvu+iEOLQLrUSlFrPNQ0FU+s0gd0IaP6RJBrQX8MdD7lPPIeik09iJq3T6NqoUZAm+nGR7DPdKFCXEnI4YbdSnHQH+0ETg1ZnTPljHXLWpw+EsM0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=CWXIvkz8; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="CWXIvkz8" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-7c7bd118546so12300339f.1 for ; Mon, 26 Feb 2024 11:25:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975506; x=1709580306; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jw0Je0tw5r/73HhCk/nGFBZB/5kheAEr7X7sZwheg6c=; b=CWXIvkz8mIobmAuHg+LSl1TqGViGhEvf+qXGAl6BdSNp4xvEFySV6rfsmsQo5AhOf2 Rik9GrF4lCou2LfGZYt8FjjuzxVvN2v19QZmFtDbO67NmLJC+tF3JW8Kqd3D2jcB/wjL TwsmyWLSRjV/Ft8KovykkfVSiqge3fBBgw5LgjZ5HvTuCFn+3R1PFjJ0N6f5CaW+7mIZ aK3DvB+IU0VGe2SiDMvJzmKb075C7M/6XB72UpJIa19nnRek7Xt3ZL4RdMgn0HEuUCEh GIGUV7Y5DukOV0gjfFQERfjZTvrRkLKHm3XJQrhV6NJ6pDUMWx7QeaDK7pGUezGQ1Zig VEfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975506; x=1709580306; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jw0Je0tw5r/73HhCk/nGFBZB/5kheAEr7X7sZwheg6c=; b=Xh37E/Q2CbakpPz1yxXlZQQ1UIY9j3148S1evtfz+25i3QMXQ1Yb/FFIT63OqNDChw JCVeMuNH0py6SmcDio7zoEvul3HonwY7gIs9Dv7y6X5pXQMWRUihjglW83RdkgLikryi Id86mzHGf5QTnc/3ZKIKpz3U4zF2mifDByeu4eIgHZHYklWeDxGDFBUyIyK2EATCmDQi O1K5+KwfcVKL8kvtOedmTY8j6AnnqLHpEh+LikKLcy51K0ouP/qbIO/N82WImE70Hkds twsSFTie5GMzGvLxiHkGwV2q+zwHRyZOCDY/n26ui1dbD1HBWba15RUB11/yLIbgIxM+ DoLg== X-Gm-Message-State: AOJu0YwNnbu+GAvxGGnO0zB9/AU94187jZpFCehJiJSESoeE1GR16Rd3 hKRz9iu0zBzWQGBmfUBw5T3l7xPTMCHWVi/hgltgKo21Tea7MofnB9HKbDKTGTvBZe0kiwxJOeU W X-Google-Smtp-Source: AGHT+IHcaLFu8QbwdEtmuhQUKbkvfE27L0G0aQH8G3GIJ/M79XPZmCO9sEYYJzmavAjAq14aZmUsfg== X-Received: by 2002:a6b:5d0b:0:b0:7c7:b8fb:8922 with SMTP id r11-20020a6b5d0b000000b007c7b8fb8922mr4560752iob.0.1708975506517; Mon, 26 Feb 2024 11:25:06 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:05 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 1/9] net: remove {revc,send}msg_copy_msghdr() from exports Date: Mon, 26 Feb 2024 12:21:13 -0700 Message-ID: <20240226192458.396832-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The only user of these was io_uring, and it's not using them anymore. Make them static and remove them from the socket header file. Signed-off-by: Jens Axboe --- include/linux/socket.h | 7 ------- net/socket.c | 14 +++++++------- tools/perf/trace/beauty/include/linux/socket.h | 7 ------- 3 files changed, 7 insertions(+), 21 deletions(-) diff --git a/include/linux/socket.h b/include/linux/socket.h index cfcb7e2c3813..139c330ccf2c 100644 --- a/include/linux/socket.h +++ b/include/linux/socket.h @@ -422,13 +422,6 @@ extern long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, struct user_msghdr __user *umsg, struct sockaddr __user *uaddr, unsigned int flags); -extern int sendmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct iovec **iov); -extern int recvmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct sockaddr __user **uaddr, - struct iovec **iov); extern int __copy_msghdr(struct msghdr *kmsg, struct user_msghdr *umsg, struct sockaddr __user **save_addr); diff --git a/net/socket.c b/net/socket.c index ed3df2f749bf..0f5d5079fd91 100644 --- a/net/socket.c +++ b/net/socket.c @@ -2600,9 +2600,9 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys, return err; } -int sendmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct iovec **iov) +static int sendmsg_copy_msghdr(struct msghdr *msg, + struct user_msghdr __user *umsg, unsigned flags, + struct iovec **iov) { int err; @@ -2753,10 +2753,10 @@ SYSCALL_DEFINE4(sendmmsg, int, fd, struct mmsghdr __user *, mmsg, return __sys_sendmmsg(fd, mmsg, vlen, flags, true); } -int recvmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct sockaddr __user **uaddr, - struct iovec **iov) +static int recvmsg_copy_msghdr(struct msghdr *msg, + struct user_msghdr __user *umsg, unsigned flags, + struct sockaddr __user **uaddr, + struct iovec **iov) { ssize_t err; diff --git a/tools/perf/trace/beauty/include/linux/socket.h b/tools/perf/trace/beauty/include/linux/socket.h index cfcb7e2c3813..139c330ccf2c 100644 --- a/tools/perf/trace/beauty/include/linux/socket.h +++ b/tools/perf/trace/beauty/include/linux/socket.h @@ -422,13 +422,6 @@ extern long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, struct user_msghdr __user *umsg, struct sockaddr __user *uaddr, unsigned int flags); -extern int sendmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct iovec **iov); -extern int recvmsg_copy_msghdr(struct msghdr *msg, - struct user_msghdr __user *umsg, unsigned flags, - struct sockaddr __user **uaddr, - struct iovec **iov); extern int __copy_msghdr(struct msghdr *kmsg, struct user_msghdr *umsg, struct sockaddr __user **save_addr); From patchwork Mon Feb 26 19:21:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572741 Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B20E812F361 for ; Mon, 26 Feb 2024 19:25:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975512; cv=none; b=eetdpDCnJ8yFW0KBAkha/7il26RBMSHAHcX1V4mrfmoMK7stkJt7qFnUpYbO8fo8YXDvx+xYTWyOOxA29UwYicAzZwFm0P1ZHqrVGsamPdt93Fhf+ua1YayteyDexFGBtbKXOE4BjPUnOmf+lFVJY70Ktvp9xbqlxO9E5X+x2hk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975512; c=relaxed/simple; bh=bkMmyZKh51k8y2VEtqkwLoik1cfgZqlRd2aMwSyglPQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nW+ukIdQuzy5LmHFxA+UBH5P5boi0/wMRjrkJDa3luElGQfc4AGGPLOOJLe1+9XpZo6qyOe/lswoxB03kdkTDnZf9dizEluUakmVNEyhJkS/q/262v4Bmc7oAA4igaTuTE9pBH3itxrX1ANexOh95lSJ7EU0Y5w7CmmIuj7uT7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=aR3etyve; arc=none smtp.client-ip=209.85.166.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="aR3etyve" Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-7c7b076562cso26095039f.0 for ; Mon, 26 Feb 2024 11:25:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975509; x=1709580309; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W5S26c1cI50UkIDvPXxOEeXnJ8Zl5Pckl0/AFyGZTKQ=; b=aR3etyveoxSzr/7+BwrZAsPbyz+TMNnAv4s8Kx7pJ1pAd+YJZ+HgQbY10XPzigG96X 5Y7dY4SYCXhBtJ7Zs9hGZqJex3t5PUgLmHKPBMzk7vXLloA4bukOGecVdFu+rC4Y98V7 2Uysv9qd9iuUc3t5Bus+zfJvUj487TJv6UsUWDZr+Bgt7vW9vtYbLSpdIZWVIcS3EypH r466wG+4OUvs2crJI2RTBf4ILk/EYgAejUwjTXjCfsHe3NQW7Vw14AHQDaLqaiqr2PvS Yszz1DLwEJpoaTIe2COyc5Y1z7ZpYdALoRtR6cr1mWEVSnl2YlU0wRHyBHhZicjjrYOa 8kMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975509; x=1709580309; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W5S26c1cI50UkIDvPXxOEeXnJ8Zl5Pckl0/AFyGZTKQ=; b=I4jZLIto+95O7yd+9cdEdu/1xIol/aydQfemD9sEcNtY4ZSAjm/MQ9cqGvgtVHw8zy z/c2+bO4SCi5RJwKKl0InTAEZYg92QjA4lvBUHKE5WJOsWkbE34xX+0BTRdKENRskr3e a0xsBNgpJqEcSdot+nHebuJtSaCA6YCMCnC9EjIownyWiPIvFVSTASA/Cggp+9HbZn8s TAS5tIgbw0YVP9PVvMzlIlOPRcOj9UhT09Y1mo7rxA9QrK7F2vP8XyZI30cTgEfwtdzl KGgIEC7S3WcfgnpQwoM9onEMNgS/i5rhiSScNJxVp0NgHuU9RY4tIyel3ALWrU88XoS3 /+GQ== X-Gm-Message-State: AOJu0YwqbVK3tCV8jKMfIWf8e/Ia/fbKDVpYiXhf98fskjzcsq7Qt/PA 0RONvEkZ7nfKOTvu2CfpmI5n5eJqB1O55QCuWJwW+dHeaYbwh/71k9h6WvabeNx/g+yhtf9QYof h X-Google-Smtp-Source: AGHT+IGxvMeVlV2CmRnHXyz8nPjhmH4mwdCD52CeyRjtcF8B48MqWKbJo3InLMQovC2ml4/0mkqwAQ== X-Received: by 2002:a6b:6d08:0:b0:7c7:ba40:4ba8 with SMTP id a8-20020a6b6d08000000b007c7ba404ba8mr4323277iod.1.1708975508353; Mon, 26 Feb 2024 11:25:08 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:06 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 2/9] io_uring/net: add provided buffer support for IORING_OP_SEND Date: Mon, 26 Feb 2024 12:21:14 -0700 Message-ID: <20240226192458.396832-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's pretty trivial to wire up provided buffer support for the send side, just like we do on the receive side. This enables setting up a buffer ring that an application can use to push pending sends to, and then have a send pick a buffer from that ring. One of the challenges with async IO and networking sends is that you can get into reordering conditions if you have more than one inflight at the same time. Consider the following scenario where everything is fine: 1) App queues sendA for socket1 2) App queues sendB for socket1 3) App does io_uring_submit() 4) sendA is issued, completes successfully, posts CQE 5) sendB is issued, completes successfully, posts CQE All is fine. Requests are always issued in-order, and both complete inline as most sends do. However, if we're flooding socket1 with sends, the following could also result from the same sequence: 1) App queues sendA for socket1 2) App queues sendB for socket1 3) App does io_uring_submit() 4) sendA is issued, socket1 is full, poll is armed for retry 5) Space frees up in socket1, this triggers sendA retry via task_work 6) sendB is issued, completes successfully, posts CQE 7) sendA is retried, completes successfully, posts CQE Now we've sent sendB before sendA, which can make things unhappy. If both sendA and sendB had been using provided buffers, then it would look as follows instead: 1) App queues dataA for sendA, queues sendA for socket1 2) App queues dataB for sendB queues sendB for socket1 3) App does io_uring_submit() 4) sendA is issued, socket1 is full, poll is armed for retry 5) Space frees up in socket1, this triggers sendA retry via task_work 6) sendB is issued, picks first buffer (dataA), completes successfully, posts CQE (which says "I sent dataA") 7) sendA is retried, picks first buffer (dataB), completes successfully, posts CQE (which says "I sent dataB") Now we've sent the data in order, and everybody is happy. It's worth noting that this also opens the door for supporting multishot sends, as provided buffers would be a prerequisite for that. Those can trigger either when new buffers are added to the outgoing ring, or (if stalled due to lack of space) when space frees up in the socket. Adds IORING_FEAT_SEND_BUF_SELECT as a feature flag. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 3 ++- io_uring/net.c | 19 ++++++++++++++++--- io_uring/opdef.c | 1 + 4 files changed, 20 insertions(+), 4 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 7bd10201a02b..061147cdfbb0 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -522,6 +522,7 @@ struct io_uring_params { #define IORING_FEAT_CQE_SKIP (1U << 11) #define IORING_FEAT_LINKED_FILE (1U << 12) #define IORING_FEAT_REG_REG_RING (1U << 13) +#define IORING_FEAT_SEND_BUF_SELECT (1U << 14) /* * io_uring_register(2) opcodes and arguments diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cf2f514b7cc0..edd7cded1a80 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3962,7 +3962,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, IORING_FEAT_POLL_32BITS | IORING_FEAT_SQPOLL_NONFIXED | IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS | IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP | - IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING; + IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING | + IORING_FEAT_SEND_BUF_SELECT; if (copy_to_user(params, p, sizeof(*p))) { ret = -EFAULT; diff --git a/io_uring/net.c b/io_uring/net.c index fcbaeb7cc045..c6a24973352e 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -436,6 +436,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr iomsg, *kmsg; struct socket *sock; + unsigned int cflags; unsigned flags; int min_ret = 0; int ret; @@ -489,7 +490,8 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - io_req_set_res(req, ret, 0); + cflags = io_put_kbuf(req, issue_flags); + io_req_set_res(req, ret, cflags); return IOU_OK; } @@ -497,8 +499,9 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) { struct sockaddr_storage __address; struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct msghdr msg; + size_t len = sr->len; struct socket *sock; + struct msghdr msg; unsigned flags; int min_ret = 0; int ret; @@ -531,7 +534,17 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(!sock)) return -ENOTSOCK; - ret = import_ubuf(ITER_SOURCE, sr->buf, sr->len, &msg.msg_iter); + if (io_do_buffer_select(req)) { + void __user *buf; + + buf = io_buffer_select(req, &len, issue_flags); + if (!buf) + return -ENOBUFS; + sr->buf = buf; + sr->len = len; + } + + ret = import_ubuf(ITER_SOURCE, sr->buf, len, &msg.msg_iter); if (unlikely(ret)) return ret; diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 9c080aadc5a6..88fbe5cfd379 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -273,6 +273,7 @@ const struct io_issue_def io_issue_defs[] = { .audit_skip = 1, .ioprio = 1, .manual_alloc = 1, + .buffer_select = 1, #if defined(CONFIG_NET) .prep = io_sendmsg_prep, .issue = io_send, From patchwork Mon Feb 26 19:21:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572740 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B93E212FF64 for ; Mon, 26 Feb 2024 19:25:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975512; cv=none; b=MKWn/GVLlgNcuay4jELm6K2YgwNWJgplrjXJ377PE8JYXynAfyLVOf7is7Ib7CyVnRTqqgXpsG2pFcorddLlDco/X0m2ufLOmdb0o0eDeEejy7Hte/LG/4+dlYP2iVTZYutnv9U/d9GydDzLxU/KNOm8CNl4lem0b1olCuNIDrI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975512; c=relaxed/simple; bh=lGoC2BMu61GnabMZrBdyCwtqhshjhK1Ekf9h9Qs2UU8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Z6ZBubqSP2mKQsPNgGFQSy45vlfrDuBD27OnhpZKY5gndiSpiU2Ap8FMrl3djxyza1LofsLyxc/P/n6rwOXogfiO2e8/DxIK6pFk24qj4rfg4MmdptgCQS+7KyhioJmAk2Qa7iPJRmUC9k2V9J8VdwL0efHl5YHcSstg1J7zDD4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=oD+BZzYK; arc=none smtp.client-ip=209.85.166.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="oD+BZzYK" Received: by mail-io1-f47.google.com with SMTP id ca18e2360f4ac-7c787eee137so39011839f.0 for ; Mon, 26 Feb 2024 11:25:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975509; x=1709580309; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I0ptQPO8d+OTLLsakqRHLt1Ahb+P2jcy8KvUJBo3TIw=; b=oD+BZzYKtxANNrC8YUvPJrxmjDMkhKecyo83t+7SQeQnYs0vwpfrrDjyuEjtGc6wuR KGJJeiu+onwcfpZxAtBbpSrTN/GP7+UOetz8L4AJJpruwf3hrCv64iafOxSFq2kPsX4y 8jdVHAU2PZoLEqnhiRTm45yf60yz/VYa9K6Cj0iRs/b/GjKDY1nK3EX+XHWcwKRbfYmy iUGF6P4QSCqQpZsRYDWyYiAOfmrpjDO4xOycVHHic9G9z6iSUpoWFJcM+OHm/mIQ6XBJ +Pfn0h86ymGyYkKv/sUrXUUF1ZAzQ6rwk9xAWOoBB/5uSNZBBRv12KY7khyyLkxqFQYi +ZCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975509; x=1709580309; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I0ptQPO8d+OTLLsakqRHLt1Ahb+P2jcy8KvUJBo3TIw=; b=YqdZEBIUxaJBtGjYmIDdeevSQsDyeR/q2NMCMg6LY7DOokljFVV2mLQ3LifRvoMTra d5y3N222pwclcfnHgbT118jqboygt+484W1oIILi7BbNdm6Qg5wfD9/3SqwTtEcIY5Wu 2JTsCYPknJ8zYTw/ZbRAcqAJ3NAaTBiePWPfuhSj3cl2ESKoibr2EXeYdHcQd7ltFUbu wd5Njv1mDzuczIM+yAU2fc5rDAPujqPCjjErvs5TRn87Q0XCJbeVDJPfYRAbEbT9ImDy 2aoVNPAA1cPwC5zWHNYdSBUrl1N/qclXfDSWROcSvQdY7qa5j5rGGQONZF+5jneKBJpN 2O4w== X-Gm-Message-State: AOJu0YxM49b6QZ40prxgL/AP6Lg3hLeFt5FycgmUOoJmoaiQ4bCESTH0 OGJbtHSZb5EZYsLT1XuP1DTqW8wcuUKImKNe4GLXJ9fcVMNrud/Og2Wt8mGJjYtFXxDuH1mbzql n X-Google-Smtp-Source: AGHT+IGxlaAZ9ISe685f8FnDQWCVcngONtKINNK3PvcvH3bAVYi3X8+a3PP2ctV4skgTsPIHNIYj9g== X-Received: by 2002:a6b:da17:0:b0:7c7:a02d:f102 with SMTP id x23-20020a6bda17000000b007c7a02df102mr7465761iob.0.1708975509227; Mon, 26 Feb 2024 11:25:09 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:08 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 3/9] io_uring/net: add provided buffer support for IORING_OP_SENDMSG Date: Mon, 26 Feb 2024 12:21:15 -0700 Message-ID: <20240226192458.396832-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Adds provided buffer support for sendmsg as well, see the previous commit that added it to IORING_OP_SEND for a longer explanation of why this makes sense. Signed-off-by: Jens Axboe --- io_uring/net.c | 15 ++++++++++++++- io_uring/opdef.c | 1 + 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index c6a24973352e..679eefcd11c5 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -459,6 +459,17 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_msg(req, kmsg, issue_flags); + if (io_do_buffer_select(req)) { + void __user *buf; + size_t len = sr->len; + + buf = io_buffer_select(req, &len, issue_flags); + if (!buf) + return -ENOBUFS; + + iov_iter_ubuf(&kmsg->msg.msg_iter, ITER_SOURCE, buf, len); + } + flags = sr->msg_flags; if (issue_flags & IO_URING_F_NONBLOCK) flags |= MSG_DONTWAIT; @@ -502,6 +513,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) size_t len = sr->len; struct socket *sock; struct msghdr msg; + unsigned int cflags; unsigned flags; int min_ret = 0; int ret; @@ -576,7 +588,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - io_req_set_res(req, ret, 0); + cflags = io_put_kbuf(req, issue_flags); + io_req_set_res(req, ret, cflags); return IOU_OK; } diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 88fbe5cfd379..1f6b09e61ef8 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -139,6 +139,7 @@ const struct io_issue_def io_issue_defs[] = { .pollout = 1, .ioprio = 1, .manual_alloc = 1, + .buffer_select = 1, #if defined(CONFIG_NET) .prep = io_sendmsg_prep, .issue = io_sendmsg, From patchwork Mon Feb 26 19:21:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572742 Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7EB112FF76 for ; Mon, 26 Feb 2024 19:25:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975514; cv=none; b=UV9MUPJ20gNun5xQkIVDlDob3r6T0IiSZ+pks+pxR8n62G0V6howR6K3ozOy5OCjOrq7665EGbQnfWR5byIc+P6wXU5rR2CdFYXBC0fRpmOnJquldVK1rTJKuNx+fgVdvODdVgsu4akym4V68L54UwlT8HnpBIFyeeXOsSDJcdw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975514; c=relaxed/simple; bh=Bwz1XCGG8sAa4hB/q27XYGr3ZK5UY47pzg7bfBwJ80o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Oo2OaV+oJq0dWHVDNQzmQmlGNts7TSbL8oXQE08sq9Nde45ze/KPzeR+EM8esEI9mAjaGe1VYSXEseRpuLKwINA555luMMcaJURWrOdgPk+VnOzBqTWrwkq+RCarg8ReDuIhhgfWEUsjToc3CP36x3K7MSIztYim9fINJo1Kh44= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=CuNcvDXC; arc=none smtp.client-ip=209.85.166.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="CuNcvDXC" Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-7c495be1924so52266539f.1 for ; Mon, 26 Feb 2024 11:25:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975511; x=1709580311; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qke7VBOcxahFVWc17Wbuzh6Bf+gTJGXU/ImftE7zCJs=; b=CuNcvDXCKi6YDJ7fWyVLJnD/YPoZjsatLC5o3g/tu2c/BQh7z7NUrvCpj3UkIGF5Ey 2tpGWM9DtmCZKD5BKtHxu1PYI2AFKj4jOfT+8rRfOYgTWa9GNm6x/4XPLfpjXTUJL+2B 0Dio8H5cmCVbvKv9dYpjCLv4pQwXoi9U8dZBa2cvcbETfqQowdS69Wo5GjODJtueokYa MZA7FM/NMhj/6iM1N+3qIA/QnB5HoPjHWd3407zqKN1/yxjv/hG2NMexfY/CN3V2NEjt bCfsxWzxzJr67oddBvY26rPUB4/sigr5Ns/kZbO6XVvkX4O0EWw63NW0xTp9+iBCFMta HH4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975511; x=1709580311; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qke7VBOcxahFVWc17Wbuzh6Bf+gTJGXU/ImftE7zCJs=; b=WbHlSSeymdfQL/yK/Jd/u+Z5Xo7FpgDTfIDXAQ7JXnY5hzMqlcHZsMpdh6otObzOsK HU8Prt7ZlKz2skeyuUYesXGFy8H1HhpkyXlTxUT9bIopiMuodPSrlvYxuiP+Jr+FX/j+ FbCAAflITo7whSWyodZcC02JEGfbl0uDsarm5rGNghWKcMLx1qAzOfuwd2bYgpCGPJKO MrjfRnj/OlSG1B3lo4xU2xIYkUJDLzhcgAODwWETGJWekLe3EoZ34VAURYLeVNBWiTgE 0LeYYNaafrucQviPU17B1uc5uTkMB3l1FDIhCjXiIY9wdCYOKfzU6FQ+5VufCNQq8a5y We3w== X-Gm-Message-State: AOJu0Yy0ptdkZsUSgkS16Tmbl5gSSOeJ+ZhVoo2DChouAL94Z9iL/4Gz aehp7oyGVZXWICvNsPC3Cipk2ed1eNTI81FrhsVxqgU5hIP3KtjA6wH+Uf4tnjb7BMiY/Nbzj/9 Q X-Google-Smtp-Source: AGHT+IGnWeMxC92Q2Q7Ud/prQ8XL43ODqu5UF/85S7l3lE/fZbIzShkZyoWNZQOcDUDY/sMe+DkimA== X-Received: by 2002:a05:6602:38d:b0:7c7:8933:2fec with SMTP id f13-20020a056602038d00b007c789332fecmr7386660iov.2.1708975511642; Mon, 26 Feb 2024 11:25:11 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:09 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 4/9] io_uring/kbuf: flag request if buffer pool is empty after buffer pick Date: Mon, 26 Feb 2024 12:21:16 -0700 Message-ID: <20240226192458.396832-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Normally we do an extra roundtrip for retries even if the buffer pool has depleted, as we don't check that upfront. Rather than add this check, have the buffer selection methods mark the request with REQ_F_BL_EMPTY if the used buffer group is out of buffers after this selection. This is very cheap to do once we're all the way inside there anyway, and it gives the caller a chance to make better decisions on how to proceed. For example, recv/recvmsg multishot could check this flag when it decides whether to keep receiving or not. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/kbuf.c | 10 ++++++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index bd7071aeec5d..d8111d64812b 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -480,6 +480,7 @@ enum { REQ_F_POLL_NO_LAZY_BIT, REQ_F_CANCEL_SEQ_BIT, REQ_F_CAN_POLL_BIT, + REQ_F_BL_EMPTY_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -556,6 +557,8 @@ enum { REQ_F_CANCEL_SEQ = IO_REQ_FLAG(REQ_F_CANCEL_SEQ_BIT), /* file is pollable */ REQ_F_CAN_POLL = IO_REQ_FLAG(REQ_F_CAN_POLL_BIT), + /* buffer list was empty after selection of buffer */ + REQ_F_BL_EMPTY = IO_REQ_FLAG(REQ_F_BL_EMPTY_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index ee866d646997..3d257ed9031b 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -139,6 +139,8 @@ static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, list_del(&kbuf->list); if (*len == 0 || *len > kbuf->len) *len = kbuf->len; + if (list_empty(&bl->buf_list)) + req->flags |= REQ_F_BL_EMPTY; req->flags |= REQ_F_BUFFER_SELECTED; req->kbuf = kbuf; req->buf_index = kbuf->bid; @@ -152,12 +154,16 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len, unsigned int issue_flags) { struct io_uring_buf_ring *br = bl->buf_ring; + __u16 tail, head = bl->head; struct io_uring_buf *buf; - __u16 head = bl->head; - if (unlikely(smp_load_acquire(&br->tail) == head)) + tail = smp_load_acquire(&br->tail); + if (unlikely(tail == head)) return NULL; + if (head + 1 == tail) + req->flags |= REQ_F_BL_EMPTY; + head &= bl->mask; /* mmaped buffers are always contig */ if (bl->is_mmap || head < IO_BUFFER_LIST_BUF_PER_PAGE) { From patchwork Mon Feb 26 19:21:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572743 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 962A312F39C for ; Mon, 26 Feb 2024 19:25:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975516; cv=none; b=RP56chRrVuwVZ5KNmCAkMvAAtghI8sbJxtp9e0VIyhdCD9l5V5ZuezNCXPRVqYTy0/RKcOq/khxAStFo2LmW5nIMIuLF3lxIJd98VRcf0Zo9A54/SSsxKIEYZBhNkgjtQzrpkNcJ52ClhestpQ2raZMNsewWZe055+yH6M4QB9c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975516; c=relaxed/simple; bh=9b/DCj7tK0fcxeLK2v+Pwa51LsBoMsxTPtfHZXS+g7k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nHlY6DqY5+rwSoJ9cMU8uR3gM9h19qN8oOwJBsi233BBVCE6WnkDMhUDdn3ajDE6FfR3s9nF+y90a2nD8m6CnveDZTrStZRWZGsp4i1bAac4+cwwTCNEM8VSoF48rLto9ybrHEMwSPSqicikjELRK9xnqjNVsx2rUPLgGZp8Ed8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Uzv8qwQG; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Uzv8qwQG" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so31753939f.0 for ; Mon, 26 Feb 2024 11:25:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975513; x=1709580313; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lGlHQ7gE2+xdFImai+G8EFaBONPIZKbvS7mj7TDXAzc=; b=Uzv8qwQG4KDa236r4yxNrTn4odlpke6P8Y9fdydeSU9eF2gLnNA7iTLOz/WMJQQmtT uzAuKbAhFWLVZKoh51DVHxk+hEYMkGypEi5OJTykLT8Hy0zylNXJupkF+KjpYhSUL6XP pdMJgUvkTlOf0T0ewbv0rcUetmf9ePoj6t/RJdSk23kpMmR6rS5D4sjyA/mNrGFg7qRX XyMqR+ThiiFL3O3/Ze43InIK3fWc//y6HO37mwDGzrEjTV7MSDndjf2cWlWeSoevmPN5 l6WlvMwdjHzlZJasKyARDeQstVaLALJIDjFvQ1T5apvpqSWU2+beKyRWSi17bgRyB45T 03nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975513; x=1709580313; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lGlHQ7gE2+xdFImai+G8EFaBONPIZKbvS7mj7TDXAzc=; b=pE+0C3IEC+Yqc6W92PgmtQtTe6tW9FXGikjGcVIkJcZG33CeTRGqF3nEg2hfFVwjmd xYrxAsaP4TMVrHYZbpAr6sLF0R49EJKVMEItCvYc/mhcRj3VMXGRVGodrh3HQug7vOvo 6AeblrbayHXbW91WQoHOVLjPHfCS+XM6APwx5X6HDxymrztphz5a8Oz68DQeGrJZ0kb6 Rtprc+u+P9VFsD4RTQVBoNQZG+eV8ATZ9KphOxdF5KTh2ZgK9L3JBE5JU0Q3Ve6vat4Z rMTgwQQ7+JCHLJ6thjOlxQ1LiwjXQBx4ntxZNv7ee3JcvuUtzSvAs5wPVp7Wk0qYpiS3 dcEg== X-Gm-Message-State: AOJu0Yykigl9ZTzscfiE/OIysDm0g767pbDSeqwIxkaOTNzkrhLOgUbX rWmLdLRrOXVelrYgwiKvQK5MMp1Mw5w5pEhSNxA4e7bbsBHtRsBG5YitGGrzNtK0EtEPIwDG7sx Q X-Google-Smtp-Source: AGHT+IFuVSw1pG6AMEcXJYuKAOuyXuV5NIhtrObDRCk9ADxU49DPsfuo7XLNLTk2/4pp2NIbmdQ8qg== X-Received: by 2002:a05:6602:2195:b0:7c7:ce93:f532 with SMTP id b21-20020a056602219500b007c7ce93f532mr2750719iob.1.1708975513360; Mon, 26 Feb 2024 11:25:13 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:12 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 5/9] io_uring/net: avoid redundant -ENOBUFS on recv multishot retry Date: Mon, 26 Feb 2024 12:21:17 -0700 Message-ID: <20240226192458.396832-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Now that we know if the buffer list is empty upfront, there's no point doing a retry for that case. This can help avoid a redundant -ENOBUFS which would terminate the multishot receive, requiring the app to re-arm it. Signed-off-by: Jens Axboe --- io_uring/net.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index 679eefcd11c5..aaab4f121b7f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -698,6 +698,11 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; io_recv_prep_retry(req); + + /* buffer list now empty, no point trying again */ + if (req->flags & REQ_F_BL_EMPTY) + goto enobufs; + /* Known not-empty or unknown state, retry */ if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq == -1) { if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY) @@ -706,6 +711,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, sr->nr_multishot_loops = 0; mshot_retry_ret = IOU_REQUEUE; } +enobufs: if (issue_flags & IO_URING_F_MULTISHOT) *ret = mshot_retry_ret; else From patchwork Mon Feb 26 19:21:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572744 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E08312F5A4 for ; Mon, 26 Feb 2024 19:25:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975518; cv=none; b=iXsEPvk0wDc3fdgO7tZxz/sIP7he6zPtDZJdKJxK0KxyLJJCnK0pAHoQ6XPoJRzSTqO3lQWELT11Lh5RZJ/Lp7vWY+yjCoSS3n0Nc58DaqB4jaR1ffw3qTNLus9AqOw9O9QIqQrSL9tLfcW/DC574lBZmwpOUNaYi9eZynpYIZ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975518; c=relaxed/simple; bh=SJj4iMt8345K336hGQW61NIJ2oZAfuUExnbC6B2sJG4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LmWvCSuOkcjc+qd+FopuJJvQ8ixDP7tuNQ/JlklHuMex3KzYaSGZoLxQg2Fz/WmNvU8qv0nOqxjLqXxFI3U8v9iXSBD/8mAL+1k7FKO8ahNVFl8lGwWlZbJWoJ+imV4ISTFhgRnisw/6LGp83f3KHNu2GeVrW0zf9H4sR2jdXLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=f7N/KZAP; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="f7N/KZAP" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-7c495be1924so52267339f.1 for ; Mon, 26 Feb 2024 11:25:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975515; x=1709580315; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y0eR1eNEDBGH7MxRpFv9waoCL0rfS8tfLNM5reazrhE=; b=f7N/KZAPyvcERa8cwUzETLZcfViSZ5ByMrmo1fmNSOtnjf8k8Pcrayc2aG62nv+vm+ 8HW6szZue1qLAhPlc8wqSHFk52vyMNuJuXyL0W90xil1kQjbrMi3I6ZavLQbaAuaOOUx H+mCfIvBzt9oblm2Zv1fXgAXrOkHYZ6LRiYKVUjekqARXReWVnHfKfbT65Zn3DoWXZr4 Kff1WTUOd15Uqfpz1/nuQ5U6dHbnCJVbkoxZ9WSaZSaH9meDJdjfDBIGLgPCtrrLezlt JyXx4ubOJ2W8dtX8/v/sX3DLp+Tr8pBzry2F7Y4uJ4snSl18VR345nDWB0ML/zX3m6oL DVSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975515; x=1709580315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y0eR1eNEDBGH7MxRpFv9waoCL0rfS8tfLNM5reazrhE=; b=vSgHMLbbHS/XHbwWDHrt2coVl3whodmmVzjqiJlzO6PaO15H2YCrZ/J+E7VfuA70vf Vwqal3xAxfcz6RULgcCisjy8xvPKK5EERqPJGThQvegdF4j0YFU0UENCV+acKHoGfteJ 6Ji+NI4bymDcH4uWi9Wb+2FtxHrj/eiF87CiFGyKpU1pR6ra5eHOMD2qIZfrwW7zp+Or zfaL0yGXRF3ycvKGPRp563KBHVydmzJNvF+OMzyTR1cZr2G1DDJthf+KB8VKZzRSBzmX NoJ58Emc3w/NvKzNj69vUXALP5sIBB7u0vNT0EKWgzuw626AnAukibL7bL9FwEB+ogM/ pTVw== X-Gm-Message-State: AOJu0YwQPZFhstPfPsmWIhNV6pGOHyoE46opBEhhPzmianzQa28SLBWG dM/l7ZYhRf3QUCbBcUz1ZozvA++yOWGDvHB1frqE/RVygu/Fasgf78vloJ4VMswm74935gUG0sP t X-Google-Smtp-Source: AGHT+IEV1FCkkOdVKqHo5zik2qi6cvoMB+zTODXhLgxQoigGsL/LkvlgKRbDZcorSLRG5CJJyKIEYg== X-Received: by 2002:a05:6602:38d:b0:7c7:8933:2fec with SMTP id f13-20020a056602038d00b007c789332fecmr7386801iov.2.1708975515391; Mon, 26 Feb 2024 11:25:15 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:13 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 6/9] io_uring/net: move recv/recvmsg flags out of retry loop Date: Mon, 26 Feb 2024 12:21:18 -0700 Message-ID: <20240226192458.396832-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The flags don't change, just intialize them once rather than every loop for multishot. Signed-off-by: Jens Axboe --- io_uring/net.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index aaab4f121b7f..c73e4cd246ab 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -843,6 +843,10 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (!io_check_multishot(req, issue_flags)) return io_setup_async_msg(req, kmsg, issue_flags); + flags = sr->msg_flags; + if (force_nonblock) + flags |= MSG_DONTWAIT; + retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; @@ -863,10 +867,6 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) iov_iter_ubuf(&kmsg->msg.msg_iter, ITER_DEST, buf, len); } - flags = sr->msg_flags; - if (force_nonblock) - flags |= MSG_DONTWAIT; - kmsg->msg.msg_get_inq = 1; kmsg->msg.msg_inq = -1; if (req->flags & REQ_F_APOLL_MULTISHOT) { @@ -952,6 +952,10 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) msg.msg_iocb = NULL; msg.msg_ubuf = NULL; + flags = sr->msg_flags; + if (force_nonblock) + flags |= MSG_DONTWAIT; + retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; @@ -970,9 +974,6 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) msg.msg_inq = -1; msg.msg_flags = 0; - flags = sr->msg_flags; - if (force_nonblock) - flags |= MSG_DONTWAIT; if (flags & MSG_WAITALL) min_ret = iov_iter_count(&msg.msg_iter); From patchwork Mon Feb 26 19:21:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572745 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3346712FF7F for ; Mon, 26 Feb 2024 19:25:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975519; cv=none; b=en+rmlgsVzeoYUn2fNSRQMTcMBztec76Wz6LZETsP7XJwLNJls7Nnmq0P04g4JzTBZIvW7LNbXSPzTBRAJFdi0j201SpleWBFySYV7YvtwRrMD9pbBqh5d9QBTtnYpy2DmPTidAawPVmejFWeP7GjqgSmmnaUSAtL54lP8qy0Uc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975519; c=relaxed/simple; bh=skXCAd8lJL3x9PKrWYWyooanNaZVeAY76tfivaB8mfw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RybfkTgSvDFEOeetX1R1ib96k/qePv4zh1KrgOZ9dwsAcbU/6uQn9OZc12HilXtcq7FiNgxruGe1D4I3TOBOemoYQ6kuL4jPStOH+BwkYuBBRnZSYJHvzKGjyGL8wIM+GzWU/lcDH4TyOdSQk8ADFn7wcQX1kUJaBKIPS0T0nDo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=zAN/KWRW; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="zAN/KWRW" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-7c495be1924so52267839f.1 for ; Mon, 26 Feb 2024 11:25:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975517; x=1709580317; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HX0uTWcUaFWCJT9AIroI5EDz4rYPvNDwzbOev/pFU/I=; b=zAN/KWRWWnSJcO7QNjIyGkKieKiJNZvkPFZeUskOAFXIeY5eBXMekHjpNzz97UioIq Cj1pSrwqpgWKhUXlOzPd2g1bP4dcd0VtyRl+b281CHDPLshwG8Bdyh3T16gf+RQIw+/N wYbSgMA6FUvL2qNtj5NA3fmUFH41+IkcNTcdP1Do7sFPZYewl9qkjTQYQAuPbIPxIKeU PIZcNgV2oth5oy/TDpcpIF9Auh+x9e+85UysRFpsEjXgU7uSDZPBCMBMHRFF1AjJkEGW sfk/cQ6YahMYtTqWWeLrs9Li3+J0oVLvZ78kWD1rgMhd3LJOsOTgd355Rap3cl59PGlX nLzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975517; x=1709580317; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HX0uTWcUaFWCJT9AIroI5EDz4rYPvNDwzbOev/pFU/I=; b=WEQzPw0UUGgpUd50Tt4m0+JJar/9bVk4w+orTx7IxZhuYrpAjmvv2G4OEXHktfFJHQ 5v2hHFqmitFZZJNpgbMFfeRFR4xWUbFalJuZedTHx+GF7olGq67Wah9pSraNq+LtFHA/ EiwiWjq1boe0tI9ioNb4VncboI7iziZYXd77NyKcA86IlQ59LexZebAbJLzsUU6BWlnw IQHOSWD+ZwWJHlf2DKBXua1c+DFXPqRxiOXqZ+P9dM+xwQ9KE5IG0NiyNW2kUeA3nhMf yG2fEUW7I4njShSBufU8n3f1JzhkVliXZiUcvJiNwfZpgWyNtzOkx5Qm+2PAaD1QO+oc bMQg== X-Gm-Message-State: AOJu0Yz2dGm7T+CcvD+5tZHcDsWCCh7ig/WasYOpyPu3bGw4qQO5ejXu 4PTeEMtb+kmq34lDhJU29TU3Y7jWZJg0IKG98bA20cQTBaTGlG8rFnBZip1cBDPEIWnrZv6lAAB f X-Google-Smtp-Source: AGHT+IETePzwVxcvpQCi4d4uM5Dd7N2z3raKofeec8FHrChu/Qy6VWPPdjtpaDyVP9GI3NJ4MYodcg== X-Received: by 2002:a6b:6d08:0:b0:7c7:ba40:4ba8 with SMTP id a8-20020a6b6d08000000b007c7ba404ba8mr4323569iod.1.1708975517006; Mon, 26 Feb 2024 11:25:17 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:15 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 7/9] io_uring/net: add generic multishot retry helper Date: Mon, 26 Feb 2024 12:21:19 -0700 Message-ID: <20240226192458.396832-8-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This is just moving io_recv_prep_retry() higher up so we can use it for sends as well, and renaming it to be generically useful for both sends and receives. Signed-off-by: Jens Axboe --- io_uring/net.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index c73e4cd246ab..9ef11883a34a 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -204,6 +204,16 @@ static int io_setup_async_msg(struct io_kiocb *req, return -EAGAIN; } +static inline void io_mshot_prep_retry(struct io_kiocb *req) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + + req->flags &= ~REQ_F_BL_EMPTY; + sr->done_io = 0; + sr->len = 0; /* get from the provided buffer */ + req->buf_index = sr->buf_group; +} + static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) { int hdr; @@ -654,15 +664,6 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } -static inline void io_recv_prep_retry(struct io_kiocb *req) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - - sr->done_io = 0; - sr->len = 0; /* get from the provided buffer */ - req->buf_index = sr->buf_group; -} - /* * Finishes io_recv and io_recvmsg. * @@ -697,7 +698,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; - io_recv_prep_retry(req); + io_mshot_prep_retry(req); /* buffer list now empty, no point trying again */ if (req->flags & REQ_F_BL_EMPTY) From patchwork Mon Feb 26 19:21:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572746 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7614E12FF7F for ; Mon, 26 Feb 2024 19:25:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975522; cv=none; b=eE7YijM1hw4U1Eh2i3U85NL7qS6SWefgC5ro65yPO4v58bH7VzJNYWkCDpVISiqFyTiM2DSoR+jsGDJK/1jp6llhsaL3z1DVwcaKh14SIHiZur+O2ejeDZxCx0kt7mem9ZNFi0nnVT5Le2G54g6aBXBbtAT30PUcU9g3L4UQtqA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975522; c=relaxed/simple; bh=RSZ7OAR/cdx1MKHXbpc9WXDFWd/9DySwAtqKqqNeQ2E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hiDj0y6mHpGP3Q1OZmadbKP8lMPGiMuIW12SB2ez7oXUcCnaG4LbN6ZvJqjTL/CLr+ou8zRCkOSzmyqS/mF4W/n2FVEAbOsd7HU3hvT71O0UrjYnL9kYDZHZnwR1SlmAfspl33z8gL34O74abeTA6jOcFh+qwJWQcwg8KU/oDjA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=VlC81yeZ; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="VlC81yeZ" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso63531839f.0 for ; Mon, 26 Feb 2024 11:25:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975519; x=1709580319; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vxaakl5xXGXZYuGHWx27xL1pKmhVx1dRDWD4WvDRfjo=; b=VlC81yeZXWCQ65/RFtEUFkYZ5U3+p0Gm8xt7DTTY56VX1q+5q8NlrFmhsBv0t3kfbZ 3gqfFLQb0Pp4KFNTeq/RBXLLmZyQzUyWVS52WYweVHeQsbrqayaDER1qvkWWE0/LJ4h4 eIyjDRL6gelINmKSbTZKmJ/7cbBB5bcSwS5dK/jymgvLb3yzD9/PXC51dll10wTFqZ5N yyucTqNvNf/O5I2/TS1UZsCwU/9GHKgDR3bCNoIN1mgRQMYRHATLRK2Y0Nw7SXxiZGu7 Ieozq7LPN4MSQNO4EomQ0Lzp5xeKw1dGqiubbsRoEO5S6yx0g8CbS64iEdu5SYYdKFRH mG/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975519; x=1709580319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vxaakl5xXGXZYuGHWx27xL1pKmhVx1dRDWD4WvDRfjo=; b=PMD8hNGXhhBLSWq2Nhjc1mdRz67TfNfAZKUK85A9mwxPboxjuKoauGG1MX+hnoKG4U KKE/mbXrLuBz0Fw7CgWeoDK8FwlXqPmwS++GZbZbnzzmBiNJjsFX/xKsiIi1uYW9DEwi GYTaGY7EzZevJh6MFDIvF1zEJihXLWWiQwcsnkY7VtZb9zOoOb1Ytn50l5K5slDq8JLl knRs3dOY64NhUoD+ZYz1LwzqnR2OoeVsr3kJ0kmPNlRYjBakMKlpPpVD4ZgNxHGZ4bl7 nYOBz+Huc98uL8NsRb529IslXsYMWhH4z3eyEJed5+/kjx3++q+DkkZDSoL3QAQt/07k khKw== X-Gm-Message-State: AOJu0YxybpR5q7c6QRgeJ4ITevIrm38pkLBHkyd2xsLXbMZQcYRMJ7AB yWY8nntsBO6RZgSmIyce50/zhLLBeCl0+iBWsVpl6cl8In2EjpSp20QDZhmmajF5eoRlnfC/KXW q X-Google-Smtp-Source: AGHT+IGTXLBIvrXy1bqI1legQ2mLeJvlOi2FUFMsleF1RxJPOF2QDzO9iPpTzT9G2e8/k1/O7rqTUw== X-Received: by 2002:a6b:6501:0:b0:7c7:7f73:d1a with SMTP id z1-20020a6b6501000000b007c77f730d1amr7442256iob.1.1708975519210; Mon, 26 Feb 2024 11:25:19 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:17 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 8/9] io_uring/net: support multishot for send Date: Mon, 26 Feb 2024 12:21:20 -0700 Message-ID: <20240226192458.396832-9-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This works very much like the receive side, except for sends. The idea is that an application can fill outgoing buffers in a provided buffer group, and then arm a single send that will service them all. For now this variant just terminates when we are out of buffers to send, and hence the application needs to re-arm it if IORING_CQE_F_MORE isn't set, as per usual for multishot requests. This only enables it for IORING_OP_SEND, IORING_OP_SENDMSG is coming in a separate patch. However, this patch does do a lot of the prep work that makes wiring up the sendmsg variant pretty trivial. They share the prep side. Enabling multishot for sends is, again, identical to the receive side. The app sets IORING_SEND_MULTISHOT in sqe->ioprio. This flag is also the same as IORING_RECV_MULTISHOT. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 8 ++++ io_uring/net.c | 90 ++++++++++++++++++++++++++++++----- 2 files changed, 86 insertions(+), 12 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 061147cdfbb0..feacc64c90a0 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -351,9 +351,17 @@ enum io_uring_op { * 0 is reported if zerocopy was actually possible. * IORING_NOTIF_USAGE_ZC_COPIED if data was copied * (at least partially). + * + * IORING_SEND_MULTISHOT Multishot send. Like the recv equivalent, must + * be used with provided buffers. Keeps sending + * from the given buffer group ID until it is + * empty. Sets IORING_CQE_F_MORE if more + * completions should be expected on behalf of + * the same SQE. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) #define IORING_RECV_MULTISHOT (1U << 1) +#define IORING_SEND_MULTISHOT IORING_RECV_MULTISHOT #define IORING_RECVSEND_FIXED_BUF (1U << 2) #define IORING_SEND_ZC_REPORT_USAGE (1U << 3) diff --git a/io_uring/net.c b/io_uring/net.c index 9ef11883a34a..0b990df04ac7 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -411,6 +411,8 @@ void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req) kfree(io->free_iov); } +#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_SEND_MULTISHOT) + int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -427,11 +429,17 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); sr->len = READ_ONCE(sqe->len); sr->flags = READ_ONCE(sqe->ioprio); - if (sr->flags & ~IORING_RECVSEND_POLL_FIRST) + if (sr->flags & ~SENDMSG_FLAGS) return -EINVAL; sr->msg_flags = READ_ONCE(sqe->msg_flags) | MSG_NOSIGNAL; if (sr->msg_flags & MSG_DONTWAIT) req->flags |= REQ_F_NOWAIT; + if (sr->flags & IORING_SEND_MULTISHOT) { + if (!(req->flags & REQ_F_BUFFER_SELECT)) + return -EINVAL; + req->flags |= REQ_F_APOLL_MULTISHOT; + sr->buf_group = req->buf_index; + } #ifdef CONFIG_COMPAT if (req->ctx->compat) @@ -441,6 +449,44 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static inline bool io_send_finish(struct io_kiocb *req, int *ret, + struct msghdr *msg, unsigned issue_flags) +{ + bool mshot_finished = *ret <= 0; + unsigned int cflags; + + cflags = io_put_kbuf(req, issue_flags); + + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { + io_req_set_res(req, *ret, cflags); + *ret = IOU_OK; + return true; + } + + if (mshot_finished || req->flags & REQ_F_BL_EMPTY) + goto finish; + + /* + * Fill CQE for this receive and see if we should keep trying to + * receive from this socket. + */ + if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, + *ret, cflags | IORING_CQE_F_MORE)) { + io_mshot_prep_retry(req); + *ret = IOU_ISSUE_SKIP_COMPLETE; + return false; + } + + /* Otherwise stop multishot but use the current result. */ +finish: + io_req_set_res(req, *ret, cflags); + if (issue_flags & IO_URING_F_MULTISHOT) + *ret = IOU_STOP_MULTISHOT; + else + *ret = IOU_OK; + return true; +} + int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -523,7 +569,6 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) size_t len = sr->len; struct socket *sock; struct msghdr msg; - unsigned int cflags; unsigned flags; int min_ret = 0; int ret; @@ -552,10 +597,18 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_addr(req, &__address, issue_flags); + if (!io_check_multishot(req, issue_flags)) + return -EAGAIN; + sock = sock_from_file(req->file); if (unlikely(!sock)) return -ENOTSOCK; + flags = sr->msg_flags; + if (issue_flags & IO_URING_F_NONBLOCK) + flags |= MSG_DONTWAIT; + +retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; @@ -570,19 +623,28 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret)) return ret; - flags = sr->msg_flags; - if (issue_flags & IO_URING_F_NONBLOCK) - flags |= MSG_DONTWAIT; - if (flags & MSG_WAITALL) + /* + * If MSG_WAITALL is set, or this is a multishot send, then we need + * the full amount. If just multishot is set, if we do a short send + * then we complete the multishot sequence rather than continue on. + */ + if (flags & MSG_WAITALL || req->flags & REQ_F_APOLL_MULTISHOT) min_ret = iov_iter_count(&msg.msg_iter); flags &= ~MSG_INTERNAL_SENDMSG_FLAGS; msg.msg_flags = flags; ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { - if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_addr(req, &__address, issue_flags); - + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) { + ret = io_setup_async_addr(req, &__address, issue_flags); + if (ret != -EAGAIN) + return ret; + if (issue_flags & IO_URING_F_MULTISHOT) { + io_kbuf_recycle(req, issue_flags); + return IOU_ISSUE_SKIP_COMPLETE; + } + return -EAGAIN; + } if (ret > 0 && io_net_retry(sock, flags)) { sr->len -= ret; sr->buf += ret; @@ -598,9 +660,13 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret += sr->done_io; else if (sr->done_io) ret = sr->done_io; - cflags = io_put_kbuf(req, issue_flags); - io_req_set_res(req, ret, cflags); - return IOU_OK; + else + io_kbuf_recycle(req, issue_flags); + + if (!io_send_finish(req, &ret, &msg, issue_flags)) + goto retry_multishot; + + return ret; } int io_recvmsg_prep_async(struct io_kiocb *req) From patchwork Mon Feb 26 19:21:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13572747 Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 639BA12FF9C for ; Mon, 26 Feb 2024 19:25:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975524; cv=none; b=i2pTiY7C28Vthc4fBvSfHBYd3NmosFXo9iYb4m1VsTR/WtE+v5+ZSA/lXDxmrx+rLtSNxmrjc62BsMUw9rYLovsJRK/VoeCFjNwUMd4cSMFh53nFI7ZJixSdUIgKMPNSUJ8PYHixtsLkvP+TzAK8O/PqfixKz+pFvm0fTMOWbBY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708975524; c=relaxed/simple; bh=X5JjzQ1BeSKFM9aA0ELmTfsofFkg93zHq7aQu3afbI0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VXWBheUOSMfrphqt7BnLQaiXWeluQeexS+hwfIohKlMVVN1GjWzm0bfJiwVAB4u4zYM7O9phzXFmrLfqbtHDVB0hcY3dKKfBGfeIovGvUOYi86oOLJUk62DvGQOYbIfbJFAGDpAF4HgUIBRNdKO1QCXyaDV+AjREhQaojqnwUEU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=tSUbtsDA; arc=none smtp.client-ip=209.85.166.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="tSUbtsDA" Received: by mail-io1-f48.google.com with SMTP id ca18e2360f4ac-7c45829f3b6so27408039f.1 for ; Mon, 26 Feb 2024 11:25:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708975521; x=1709580321; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A+1BT2SbKbFe4m/gssB3A8RiJ133+OKSUxeeTv8Ok8s=; b=tSUbtsDA2xk1SCX1FOnFqHB2TA6WiPgwVywPxwogVTcVO7TP398V5DUHGQauyXDMR7 NlyefhwMk1gjfc6Bmb/pw8q2cORk09d+Mwv7ApBRrojREww7wevMrE2h98w14eCSyYVS 42WhmN3eQzzg+U+qumHqF/4TyjZYif1khV4fpqEJ/xHr1oM5yWLq5uZsXSkZG9xLB3Jl Hji2LwS+IjJxy8xYxJhF41Gr6o23ysL/lLL2kTkTuomMLX1aketVDWC/nJvUSZ3sHuv/ BaS8X0h9QePGgq5oQcKRFL1NGQ1HGoq+abEtHq28bErvu/T5xuns+nAyj3i4Ic8e1hzd CbTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708975521; x=1709580321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A+1BT2SbKbFe4m/gssB3A8RiJ133+OKSUxeeTv8Ok8s=; b=BZLYfgHT6kRBDhtHkUJYaFwmAMbcdqtT86QnI0igb0jebFzyjEwCXTeaPwhM6eket3 opMyyeGnfgrurlQe7umT5S086zSBC7cktSq4rzk98I4uuUEnqYhqgeumKW532lKQldgF mB0omlOwQ5E4IRO6umfmzuPhC2XUC6thmxo84M8SyznDQL1QL0ysuxUwZZA27eeWI4oL Awy862aNbGTP625EzgertxO9jot81ki/BnecHQScW7TlC35xIgciqG2QAD1ZeY8TITA+ F98iodvZZO7irfY44Axu3JKefXW4nj3sF8JXsNpxZp1TUnAC0VDokx9Vu7Y/ovngTm4M 5o7w== X-Gm-Message-State: AOJu0YxW/OMzBQfY41m0oLJ2W/vIhPOs15Y4iTKQrIWHEDyhNQOKRJ4x yQjfhGFITKD8P656uGuFpxnIsbhl4M/pM1nD/35RqtYPsLUVG3E6H+zu2Vr1ClLxTlET8LsCUq7 C X-Google-Smtp-Source: AGHT+IGcGo0MYMOKGLfQ8uEkK7Vnm59uuAOtKzYjrWeQIPQkCMiiRPv6yth/BfLUB5zxQaRiXBuzvg== X-Received: by 2002:a6b:da17:0:b0:7c7:a02d:f102 with SMTP id x23-20020a6bda17000000b007c7a02df102mr7466195iob.0.1708975521016; Mon, 26 Feb 2024 11:25:21 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id eh3-20020a056638298300b0047466fd3b1dsm1370484jab.22.2024.02.26.11.25.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 11:25:19 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, dyudaken@gmail.com, Jens Axboe Subject: [PATCH 9/9] io_uring/net: support multishot for sendmsg Date: Mon, 26 Feb 2024 12:21:21 -0700 Message-ID: <20240226192458.396832-10-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240226192458.396832-1-axboe@kernel.dk> References: <20240226192458.396832-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Same as the IORING_OP_SEND multishot mode. Needs further work, but it's functional and can be tested. Signed-off-by: Jens Axboe --- io_uring/net.c | 47 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 33 insertions(+), 14 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 0b990df04ac7..c9d9dc611087 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -492,7 +492,6 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr iomsg, *kmsg; struct socket *sock; - unsigned int cflags; unsigned flags; int min_ret = 0; int ret; @@ -515,6 +514,14 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) (sr->flags & IORING_RECVSEND_POLL_FIRST)) return io_setup_async_msg(req, kmsg, issue_flags); + if (!io_check_multishot(req, issue_flags)) + return io_setup_async_msg(req, kmsg, issue_flags); + + flags = sr->msg_flags; + if (issue_flags & IO_URING_F_NONBLOCK) + flags |= MSG_DONTWAIT; + +retry_multishot: if (io_do_buffer_select(req)) { void __user *buf; size_t len = sr->len; @@ -526,17 +533,25 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) iov_iter_ubuf(&kmsg->msg.msg_iter, ITER_SOURCE, buf, len); } - flags = sr->msg_flags; - if (issue_flags & IO_URING_F_NONBLOCK) - flags |= MSG_DONTWAIT; - if (flags & MSG_WAITALL) + /* + * If MSG_WAITALL is set, or this is a multishot send, then we need + * the full amount. If just multishot is set, if we do a short send + * then we complete the multishot sequence rather than continue on. + */ + if (flags & MSG_WAITALL || req->flags & REQ_F_APOLL_MULTISHOT) min_ret = iov_iter_count(&kmsg->msg.msg_iter); ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); if (ret < min_ret) { - if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_msg(req, kmsg, issue_flags); + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) { + ret = io_setup_async_msg(req, kmsg, issue_flags); + if (ret == -EAGAIN && (issue_flags & IO_URING_F_MULTISHOT)) { + io_kbuf_recycle(req, issue_flags); + return IOU_ISSUE_SKIP_COMPLETE; + } + return ret; + } if (ret > 0 && io_net_retry(sock, flags)) { kmsg->msg.msg_controllen = 0; kmsg->msg.msg_control = NULL; @@ -548,18 +563,22 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret = -EINTR; req_set_fail(req); } + if (ret >= 0) + ret += sr->done_io; + else if (sr->done_io) + ret = sr->done_io; + else + io_kbuf_recycle(req, issue_flags); + + if (!io_send_finish(req, &ret, &kmsg->msg, issue_flags)) + goto retry_multishot; + /* fast path, check for non-NULL to avoid function call */ if (kmsg->free_iov) kfree(kmsg->free_iov); req->flags &= ~REQ_F_NEED_CLEANUP; io_netmsg_recycle(req, issue_flags); - if (ret >= 0) - ret += sr->done_io; - else if (sr->done_io) - ret = sr->done_io; - cflags = io_put_kbuf(req, issue_flags); - io_req_set_res(req, ret, cflags); - return IOU_OK; + return ret; } int io_send(struct io_kiocb *req, unsigned int issue_flags)