From patchwork Tue Feb 27 18:46:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13574290 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3A0D3C490 for ; Tue, 27 Feb 2024 18:48:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709059734; cv=none; b=LheNAWgq2B56n5kQ+1EOBbIvSR7BKkEdd5NJbA46ZqKTc83Hoe/QHdeWO+2nAwlgt4yYsT+lG+XuXLh5gUoaobaGqMbO0SfTUZgn+41K7DzJqYrgxPwzxmSSJdmPG9rK4wwTLLnAhaGcWZ7P3LPKrtKN4A4qsSqtwUtuGM83TU4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709059734; c=relaxed/simple; bh=v/MOVxL3a54yVwITQAeEijTWUodaSkKZDkBo3SyukPk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T1dV1Ow5GcNgV0dNlmKRhl0mnrrSXrki/mt6UgRch+/YCYliQExkRuoO+y0xU4Z5VjL+RsmA5Oy7NkAQ6Uv2iT917o4KhgxN5umCbsT/lU+Wl8GsrGBv94Whs0o4s2/fK7sGnlVyfP/LwY2ra+3M1s9T6y/IFnbaFVDwTXmfnv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=HqVuyGwu; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="HqVuyGwu" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-5cdbc42f5efso1867159a12.0 for ; Tue, 27 Feb 2024 10:48:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709059730; x=1709664530; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+8eN5RWaGQ5r+5Y9kYSI9PfD3TwopXl/tJtVRs3uPa8=; b=HqVuyGwubFYjdXUIz4roJaZTg3/u6/5bCIysTQjun4YIgz2SFqYucKPBgrVIYDL5Fv a/URMq1sBQbdpwV6hIoded8SOJXqjUFhagQb33zagzj4Wkl5PEn2/3cEfqv0ivO12EPx iwGoGe5186E1K80QN3xRt4m5AGjLzAaWktn3TMvvVSXPCzCJwE61hQdjUVv1QS+X2EQ3 kMgyzki6rSF8ddD4CAIKKhShANTnjV/xwe5ou0IRRatqia7wpT2GYkPlDdtw59CAcf8H /i0YVEeM1VoaguIW7pyrjH01RhmGpNlD+y8nYkqh9cg0+KpoaCPxEAvxJHBLZDSSvzz0 dDXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709059730; x=1709664530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+8eN5RWaGQ5r+5Y9kYSI9PfD3TwopXl/tJtVRs3uPa8=; b=BiQue5aVXSqPbSRl5HvqPNGPo/L/SZmTQ9mqGY3N+4DGgebL2ccMtV89F0M30hFGNy GLW1/hitZ9rBhjPlAfBo+U/veplOJkJEri2XLynL8y5wJptft5gPMrE+LGX94o6aRY1o RgMqTpcMkT+x7xcDcxjLAXccKIalvdb2xnAJR3NRCq6Bhz4+1xOd8n+Z/7XFlpKNWrAQ yXjLrRgKVjAf6XJbMNUEtZDI5NjgGNZWTyjCDMFyywUvALqnLLEoOEam/fsKLnQo0i+T tOaO+rqnCBKBHR77WUNXiUzWyq/GdKK0URUWlGlwPAxf5eavsuzmuYBS6oy7+rQe2k07 Tksg== X-Gm-Message-State: AOJu0Yyh4EYy246apWMgzSyN+U5hS2s8qu/nUO/PXHh7h5HR7pT6zDV/ yC6QKvRBxCfLePrLHKXBGyJ1w/LSRfvISuuctWAzKuLVxdJo2ls3W5hGErTzdwck06d93bqpeuw C X-Google-Smtp-Source: AGHT+IG03UqI2nf0DIZ6j5a0gdnkJerGiiMA7QTo0LLc+vWBRqHHC56epnBAxEw0qK9SjhOSrsPo/A== X-Received: by 2002:a05:6a00:9394:b0:6e5:382e:1890 with SMTP id ka20-20020a056a00939400b006e5382e1890mr5118594pfb.1.1709059730518; Tue, 27 Feb 2024 10:48:50 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id it2-20020a056a00458200b006e543b59587sm2282119pfb.126.2024.02.27.10.48.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 10:48:49 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] io_uring/net: unify how recvmsg and sendmsg copy in the msghdr Date: Tue, 27 Feb 2024 11:46:30 -0700 Message-ID: <20240227184845.985268-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227184845.985268-1-axboe@kernel.dk> References: <20240227184845.985268-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For recvmsg, we roll our own since we support buffer selections. This isn't the case for sendmsg right now, but in preparation for doing so, make the recvmsg copy helpers generic so we can call them from the sendmsg side as well. Signed-off-by: Jens Axboe --- io_uring/net.c | 271 ++++++++++++++++++++++++++----------------------- 1 file changed, 142 insertions(+), 129 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 43bc9a5f96f9..8c64e552a00f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -204,16 +204,150 @@ static int io_setup_async_msg(struct io_kiocb *req, return -EAGAIN; } +static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) +{ + int hdr; + + if (iomsg->namelen < 0) + return true; + if (check_add_overflow((int)sizeof(struct io_uring_recvmsg_out), + iomsg->namelen, &hdr)) + return true; + if (check_add_overflow(hdr, (int)iomsg->controllen, &hdr)) + return true; + + return false; +} + +#ifdef CONFIG_COMPAT +static int __io_compat_msg_copy_hdr(struct io_kiocb *req, + struct io_async_msghdr *iomsg, + struct sockaddr __user **addr, int ddir) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct compat_msghdr msg; + struct compat_iovec __user *uiov; + int ret; + + if (copy_from_user(&msg, sr->umsg_compat, sizeof(msg))) + return -EFAULT; + + ret = __get_compat_msghdr(&iomsg->msg, &msg, addr); + if (ret) + return ret; + + uiov = compat_ptr(msg.msg_iov); + if (req->flags & REQ_F_BUFFER_SELECT) { + compat_ssize_t clen; + + iomsg->free_iov = NULL; + if (msg.msg_iovlen == 0) { + sr->len = 0; + } else if (msg.msg_iovlen > 1) { + return -EINVAL; + } else { + if (!access_ok(uiov, sizeof(*uiov))) + return -EFAULT; + if (__get_user(clen, &uiov->iov_len)) + return -EFAULT; + if (clen < 0) + return -EINVAL; + sr->len = clen; + } + + if (ddir == ITER_DEST && req->flags & REQ_F_APOLL_MULTISHOT) { + iomsg->namelen = msg.msg_namelen; + iomsg->controllen = msg.msg_controllen; + if (io_recvmsg_multishot_overflow(iomsg)) + return -EOVERFLOW; + } + + return 0; + } + + iomsg->free_iov = iomsg->fast_iov; + ret = __import_iovec(ddir, (struct iovec __user *)uiov, msg.msg_iovlen, + UIO_FASTIOV, &iomsg->free_iov, + &iomsg->msg.msg_iter, true); + if (unlikely(ret < 0)) + return ret; + + return 0; +} +#endif + +static int __io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, + struct sockaddr __user **addr, int ddir) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct user_msghdr msg; + int ret; + + if (copy_from_user(&msg, sr->umsg, sizeof(*sr->umsg))) + return -EFAULT; + + ret = __copy_msghdr(&iomsg->msg, &msg, addr); + if (ret) + return ret; + + if (req->flags & REQ_F_BUFFER_SELECT) { + if (msg.msg_iovlen == 0) { + sr->len = iomsg->fast_iov[0].iov_len = 0; + iomsg->fast_iov[0].iov_base = NULL; + iomsg->free_iov = NULL; + } else if (msg.msg_iovlen > 1) { + return -EINVAL; + } else { + if (copy_from_user(iomsg->fast_iov, msg.msg_iov, + sizeof(*msg.msg_iov))) + return -EFAULT; + sr->len = iomsg->fast_iov[0].iov_len; + iomsg->free_iov = NULL; + } + + if (ddir == ITER_DEST && req->flags & REQ_F_APOLL_MULTISHOT) { + iomsg->namelen = msg.msg_namelen; + iomsg->controllen = msg.msg_controllen; + if (io_recvmsg_multishot_overflow(iomsg)) + return -EOVERFLOW; + } + + return 0; + } + + iomsg->free_iov = iomsg->fast_iov; + ret = __import_iovec(ddir, msg.msg_iov, msg.msg_iovlen, UIO_FASTIOV, + &iomsg->free_iov, &iomsg->msg.msg_iter, false); + if (unlikely(ret < 0)) + return ret; + + return 0; +} + +static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, + struct sockaddr __user **addr, int ddir) +{ + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + +#ifdef CONFIG_COMPAT + if (req->ctx->compat) + return __io_compat_msg_copy_hdr(req, iomsg, addr, ddir); +#endif + + return __io_msg_copy_hdr(req, iomsg, addr, ddir); +} + static int io_sendmsg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int ret; - iomsg->msg.msg_name = &iomsg->addr; - iomsg->free_iov = iomsg->fast_iov; - ret = sendmsg_copy_msghdr(&iomsg->msg, sr->umsg, sr->msg_flags, - &iomsg->free_iov); + ret = io_msg_copy_hdr(req, iomsg, NULL, ITER_SOURCE); + if (ret) + return ret; + /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -435,142 +569,21 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) -{ - int hdr; - - if (iomsg->namelen < 0) - return true; - if (check_add_overflow((int)sizeof(struct io_uring_recvmsg_out), - iomsg->namelen, &hdr)) - return true; - if (check_add_overflow(hdr, (int)iomsg->controllen, &hdr)) - return true; - - return false; -} - -static int __io_recvmsg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct user_msghdr msg; - int ret; - - if (copy_from_user(&msg, sr->umsg, sizeof(*sr->umsg))) - return -EFAULT; - - ret = __copy_msghdr(&iomsg->msg, &msg, &iomsg->uaddr); - if (ret) - return ret; - - if (req->flags & REQ_F_BUFFER_SELECT) { - if (msg.msg_iovlen == 0) { - sr->len = iomsg->fast_iov[0].iov_len = 0; - iomsg->fast_iov[0].iov_base = NULL; - iomsg->free_iov = NULL; - } else if (msg.msg_iovlen > 1) { - return -EINVAL; - } else { - if (copy_from_user(iomsg->fast_iov, msg.msg_iov, sizeof(*msg.msg_iov))) - return -EFAULT; - sr->len = iomsg->fast_iov[0].iov_len; - iomsg->free_iov = NULL; - } - - if (req->flags & REQ_F_APOLL_MULTISHOT) { - iomsg->namelen = msg.msg_namelen; - iomsg->controllen = msg.msg_controllen; - if (io_recvmsg_multishot_overflow(iomsg)) - return -EOVERFLOW; - } - } else { - iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ITER_DEST, msg.msg_iov, msg.msg_iovlen, UIO_FASTIOV, - &iomsg->free_iov, &iomsg->msg.msg_iter, - false); - if (ret > 0) - ret = 0; - } - - return ret; -} - -#ifdef CONFIG_COMPAT -static int __io_compat_recvmsg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct compat_msghdr msg; - struct compat_iovec __user *uiov; - int ret; - - if (copy_from_user(&msg, sr->umsg_compat, sizeof(msg))) - return -EFAULT; - - ret = __get_compat_msghdr(&iomsg->msg, &msg, &iomsg->uaddr); - if (ret) - return ret; - - uiov = compat_ptr(msg.msg_iov); - if (req->flags & REQ_F_BUFFER_SELECT) { - compat_ssize_t clen; - - iomsg->free_iov = NULL; - if (msg.msg_iovlen == 0) { - sr->len = 0; - } else if (msg.msg_iovlen > 1) { - return -EINVAL; - } else { - if (!access_ok(uiov, sizeof(*uiov))) - return -EFAULT; - if (__get_user(clen, &uiov->iov_len)) - return -EFAULT; - if (clen < 0) - return -EINVAL; - sr->len = clen; - } - - if (req->flags & REQ_F_APOLL_MULTISHOT) { - iomsg->namelen = msg.msg_namelen; - iomsg->controllen = msg.msg_controllen; - if (io_recvmsg_multishot_overflow(iomsg)) - return -EOVERFLOW; - } - } else { - iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ITER_DEST, (struct iovec __user *)uiov, msg.msg_iovlen, - UIO_FASTIOV, &iomsg->free_iov, - &iomsg->msg.msg_iter, true); - if (ret < 0) - return ret; - } - - return 0; -} -#endif - static int io_recvmsg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg) { - iomsg->msg.msg_name = &iomsg->addr; - iomsg->msg.msg_iter.nr_segs = 0; - -#ifdef CONFIG_COMPAT - if (req->ctx->compat) - return __io_compat_recvmsg_copy_hdr(req, iomsg); -#endif - - return __io_recvmsg_copy_hdr(req, iomsg); + return io_msg_copy_hdr(req, iomsg, &iomsg->uaddr, ITER_DEST); } int io_recvmsg_prep_async(struct io_kiocb *req) { + struct io_async_msghdr *iomsg; int ret; if (!io_msg_alloc_async_prep(req)) return -ENOMEM; - ret = io_recvmsg_copy_hdr(req, req->async_data); + iomsg = req->async_data; + ret = io_recvmsg_copy_hdr(req, iomsg); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; return ret; From patchwork Tue Feb 27 18:46:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13574291 Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0BDC4E1CF for ; Tue, 27 Feb 2024 18:48:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709059735; cv=none; b=EHXaBAOa9iLVp/XL4assKWNSJreeok+wM9zlCY6YYLIxCiaHnTqo1uqW0SKxeaQwpb/Iakv4Z4sL26DXmobV6dp8lNu/NVkIEXVZX/RYy8j6vZGIdGDuZCWf/4mED6x0I3NtULai7xQoxm/u37Ut79sbn+J2eEr7mR9KANBDwdM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709059735; c=relaxed/simple; bh=PTI7AoBBKPlVbATwuk88n8/DPJn7loFHIpoMPNdnVZA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Uzbqp+uhUQan2MzFKHhfj5b8XhAML2Ve81ldHtHVA5QwXv3VY3MAJ42RTBTHgHcrPOhOdFQTmky3S87ud21IhM29bCZtBooE02M9IQPh9giP27c0B9FD6nXKVzurq+S4USg4+Zrq1Nq5cDIcvyzRiGyzUTi7NbyCpu1AzR5PIYM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=khp5BQF4; arc=none smtp.client-ip=209.85.210.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="khp5BQF4" Received: by mail-ot1-f52.google.com with SMTP id 46e09a7af769-6e2b466d213so1087767a34.0 for ; Tue, 27 Feb 2024 10:48:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709059732; x=1709664532; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JAwXJVEwY3pE0HGSRE7b+X6KLN9ymt6KBTvlVLJ9lrg=; b=khp5BQF4RNuzhQ79lziUvMaiMCF2aOd9A0QW47Dl5bGLh8L31sxxQO3/rPc9FD66M5 Ydge8o/1yO0mcNU1xhSDwxUigLtU0twF+Il2CosP2q2uNxgNRH91Rn4yidooA1YQrJCA GWPJQWaUjDmCn91khGsuVg9BRf7PPq3UPGIBN2xH82K9aAaYJPjMZsKzeU/Z/lcpHWU8 MOhIFrh1MEHOb12qHY5dMXRHiRtko8eZTR8MrBC7TNhEbGi5OrYuTLylQot3mErOmxr2 5hibtwN3Sl/uaIrqd1EkmC6lQC1zZfdTO46lX37ksF+8+cNlGqClBbWFLRu9SjIi2Pya 3R0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709059732; x=1709664532; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JAwXJVEwY3pE0HGSRE7b+X6KLN9ymt6KBTvlVLJ9lrg=; b=itxW+mcyoZfs9lU9wMv1IxV8EqeNv2/WdbT9XMsg4B7Qo54VnATAM+BGLasilWvklI t5A9RBKZ1bgsEP2+0uYIOyC/qNAbkqpkqoRPUIs5q9/rZNIHUcHA+SaYlnSJcJ8vbXZW uipPhUSW/0kM3JyqdKl4UvbMQj0/gLhqSCbjNIsWnG3Bf/KYKLsgywXAWec4qf5z5KZb G/+2TVZuI7QcsMQxFw7cBcIM3yvdmQdFIuDEF6rfVPf2ibY2ccWYrjxSfKj2hDDu/rCP jLcEJJbNV7NGifxo++9DEq0lO6Xv40P/Bi2G2v+apIiJUOT0J/hOxB1wOpQmAz5nOFXO CMwQ== X-Gm-Message-State: AOJu0Yyy0qM5kM4WOuVYykPZNDmT60Y9nLvaHWupclr+32bQJHjObD1H 7r+FKX25zebbm1I6pzeZgCpo2yheemwodRXgxyG9H9av0WILQXl1KAsK52RlVB9AtZiDuLBRXNt O X-Google-Smtp-Source: AGHT+IFe1IXNQzuw5caqdR+RfspVKg6bEsMWNeUKIoGTrI56JM8pMRc4Tk6zD5HfdHgtkoGEToPiag== X-Received: by 2002:a05:6830:14:b0:6e2:ead9:f43d with SMTP id c20-20020a056830001400b006e2ead9f43dmr11726380otp.3.1709059732394; Tue, 27 Feb 2024 10:48:52 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id it2-20020a056a00458200b006e543b59587sm2282119pfb.126.2024.02.27.10.48.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 10:48:51 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/3] io_uring/net: move receive multishot out of the generic msghdr path Date: Tue, 27 Feb 2024 11:46:31 -0700 Message-ID: <20240227184845.985268-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227184845.985268-1-axboe@kernel.dk> References: <20240227184845.985268-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the actual user_msghdr / compat_msghdr into the send and receive sides, respectively, so we can move the uaddr receive handling into its own handler, and ditto the multishot with buffer selection logic. Signed-off-by: Jens Axboe --- io_uring/net.c | 161 ++++++++++++++++++++++++++++--------------------- 1 file changed, 91 insertions(+), 70 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 8c64e552a00f..7a07c5563d66 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -204,46 +204,26 @@ static int io_setup_async_msg(struct io_kiocb *req, return -EAGAIN; } -static bool io_recvmsg_multishot_overflow(struct io_async_msghdr *iomsg) -{ - int hdr; - - if (iomsg->namelen < 0) - return true; - if (check_add_overflow((int)sizeof(struct io_uring_recvmsg_out), - iomsg->namelen, &hdr)) - return true; - if (check_add_overflow(hdr, (int)iomsg->controllen, &hdr)) - return true; - - return false; -} - #ifdef CONFIG_COMPAT -static int __io_compat_msg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg, - struct sockaddr __user **addr, int ddir) +static int io_compat_msg_copy_hdr(struct io_kiocb *req, + struct io_async_msghdr *iomsg, + struct compat_msghdr *msg, int ddir) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct compat_msghdr msg; struct compat_iovec __user *uiov; int ret; - if (copy_from_user(&msg, sr->umsg_compat, sizeof(msg))) + if (copy_from_user(msg, sr->umsg_compat, sizeof(*msg))) return -EFAULT; - ret = __get_compat_msghdr(&iomsg->msg, &msg, addr); - if (ret) - return ret; - - uiov = compat_ptr(msg.msg_iov); + uiov = compat_ptr(msg->msg_iov); if (req->flags & REQ_F_BUFFER_SELECT) { compat_ssize_t clen; iomsg->free_iov = NULL; - if (msg.msg_iovlen == 0) { + if (msg->msg_iovlen == 0) { sr->len = 0; - } else if (msg.msg_iovlen > 1) { + } else if (msg->msg_iovlen > 1) { return -EINVAL; } else { if (!access_ok(uiov, sizeof(*uiov))) @@ -255,18 +235,11 @@ static int __io_compat_msg_copy_hdr(struct io_kiocb *req, sr->len = clen; } - if (ddir == ITER_DEST && req->flags & REQ_F_APOLL_MULTISHOT) { - iomsg->namelen = msg.msg_namelen; - iomsg->controllen = msg.msg_controllen; - if (io_recvmsg_multishot_overflow(iomsg)) - return -EOVERFLOW; - } - return 0; } iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ddir, (struct iovec __user *)uiov, msg.msg_iovlen, + ret = __import_iovec(ddir, (struct iovec __user *)uiov, msg->msg_iovlen, UIO_FASTIOV, &iomsg->free_iov, &iomsg->msg.msg_iter, true); if (unlikely(ret < 0)) @@ -276,47 +249,35 @@ static int __io_compat_msg_copy_hdr(struct io_kiocb *req, } #endif -static int __io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, - struct sockaddr __user **addr, int ddir) +static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, + struct user_msghdr *msg, int ddir) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - struct user_msghdr msg; int ret; - if (copy_from_user(&msg, sr->umsg, sizeof(*sr->umsg))) + if (copy_from_user(msg, sr->umsg, sizeof(*sr->umsg))) return -EFAULT; - ret = __copy_msghdr(&iomsg->msg, &msg, addr); - if (ret) - return ret; - if (req->flags & REQ_F_BUFFER_SELECT) { - if (msg.msg_iovlen == 0) { + if (msg->msg_iovlen == 0) { sr->len = iomsg->fast_iov[0].iov_len = 0; iomsg->fast_iov[0].iov_base = NULL; iomsg->free_iov = NULL; - } else if (msg.msg_iovlen > 1) { + } else if (msg->msg_iovlen > 1) { return -EINVAL; } else { - if (copy_from_user(iomsg->fast_iov, msg.msg_iov, - sizeof(*msg.msg_iov))) + if (copy_from_user(iomsg->fast_iov, msg->msg_iov, + sizeof(*msg->msg_iov))) return -EFAULT; sr->len = iomsg->fast_iov[0].iov_len; iomsg->free_iov = NULL; } - if (ddir == ITER_DEST && req->flags & REQ_F_APOLL_MULTISHOT) { - iomsg->namelen = msg.msg_namelen; - iomsg->controllen = msg.msg_controllen; - if (io_recvmsg_multishot_overflow(iomsg)) - return -EOVERFLOW; - } - return 0; } iomsg->free_iov = iomsg->fast_iov; - ret = __import_iovec(ddir, msg.msg_iov, msg.msg_iovlen, UIO_FASTIOV, + ret = __import_iovec(ddir, msg->msg_iov, msg->msg_iovlen, UIO_FASTIOV, &iomsg->free_iov, &iomsg->msg.msg_iter, false); if (unlikely(ret < 0)) return ret; @@ -324,30 +285,34 @@ static int __io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg return 0; } -static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, - struct sockaddr __user **addr, int ddir) +static int io_sendmsg_copy_hdr(struct io_kiocb *req, + struct io_async_msghdr *iomsg) { + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct user_msghdr msg; + int ret; + iomsg->msg.msg_name = &iomsg->addr; iomsg->msg.msg_iter.nr_segs = 0; #ifdef CONFIG_COMPAT - if (req->ctx->compat) - return __io_compat_msg_copy_hdr(req, iomsg, addr, ddir); -#endif + if (unlikely(req->ctx->compat)) { + struct compat_msghdr cmsg; - return __io_msg_copy_hdr(req, iomsg, addr, ddir); -} + ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_SOURCE); + if (unlikely(ret)) + return ret; -static int io_sendmsg_copy_hdr(struct io_kiocb *req, - struct io_async_msghdr *iomsg) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - int ret; + return __get_compat_msghdr(&iomsg->msg, &cmsg, NULL); + } +#endif - ret = io_msg_copy_hdr(req, iomsg, NULL, ITER_SOURCE); - if (ret) + ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_SOURCE); + if (unlikely(ret)) return ret; + ret = __copy_msghdr(&iomsg->msg, &msg, NULL); + /* save msg_control as sys_sendmsg() overwrites it */ sr->msg_control = iomsg->msg.msg_control_user; return ret; @@ -569,10 +534,66 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } +static int io_recvmsg_mshot_prep(struct io_kiocb *req, + struct io_async_msghdr *iomsg, + size_t namelen, size_t controllen) +{ + if ((req->flags & (REQ_F_APOLL_MULTISHOT|REQ_F_BUFFER_SELECT)) == + (REQ_F_APOLL_MULTISHOT|REQ_F_BUFFER_SELECT)) { + int hdr; + + if (unlikely(namelen < 0)) + return -EOVERFLOW; + if (check_add_overflow((int)sizeof(struct io_uring_recvmsg_out), + namelen, &hdr)) + return -EOVERFLOW; + if (check_add_overflow(hdr, (int)controllen, &hdr)) + return -EOVERFLOW; + + iomsg->namelen = namelen; + iomsg->controllen = controllen; + return 0; + } + + return 0; +} + static int io_recvmsg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg) { - return io_msg_copy_hdr(req, iomsg, &iomsg->uaddr, ITER_DEST); + struct user_msghdr msg; + int ret; + + iomsg->msg.msg_name = &iomsg->addr; + iomsg->msg.msg_iter.nr_segs = 0; + +#ifdef CONFIG_COMPAT + if (unlikely(req->ctx->compat)) { + struct compat_msghdr cmsg; + + ret = io_compat_msg_copy_hdr(req, iomsg, &cmsg, ITER_DEST); + if (unlikely(ret)) + return ret; + + ret = __get_compat_msghdr(&iomsg->msg, &cmsg, &iomsg->uaddr); + if (unlikely(ret)) + return ret; + + return io_recvmsg_mshot_prep(req, iomsg, cmsg.msg_namelen, + cmsg.msg_controllen); + } +#endif + + ret = io_msg_copy_hdr(req, iomsg, &msg, ITER_DEST); + if (unlikely(ret)) + return ret; + + ret = __copy_msghdr(&iomsg->msg, &msg, &iomsg->uaddr); + if (unlikely(ret)) + return ret; + + return io_recvmsg_mshot_prep(req, iomsg, msg.msg_namelen, + msg.msg_controllen); } int io_recvmsg_prep_async(struct io_kiocb *req) From patchwork Tue Feb 27 18:46:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13574292 Received: from mail-il1-f180.google.com (mail-il1-f180.google.com [209.85.166.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AE254F896 for ; Tue, 27 Feb 2024 18:48:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709059737; cv=none; b=VvGTMQG7TPVP8ESw/sMjvqbs0eYVYi7erNrNsKDJpOxKzT44lv8iuMfurPwyISTfWYvHVnK+L8Ommhplna5K7PyWcXQUxEQ7soSYJtNEGtOGfXZ1BYkUC8x6/ImdGjGeTPX4R9i99FAXE9Or3KAbS33iGstpmGyNk+teVdvHAjE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709059737; c=relaxed/simple; bh=+TR7Gj4IDHGabPQKAC7eX4UVTyWISGUtkHQ8RKdbcQE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fJ5MFTuPJ7Uv6KHGzjjTz9YbHcTpGo85MHTvscVH84QuUYRfA8T8QfXKxdBhRlWNYRtw095P5ja0GDIpWsdnGyjg7KgoWTkQGHqFsZ/rJXI/GGrVfKckCWOhMQclBtOxJvb1hGQ2ONT9gg5YY42STebmYamEZgaLnqdxD1AFEng= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=CsLsyx1k; arc=none smtp.client-ip=209.85.166.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="CsLsyx1k" Received: by mail-il1-f180.google.com with SMTP id e9e14a558f8ab-35d374bebe3so1489075ab.1 for ; Tue, 27 Feb 2024 10:48:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709059734; x=1709664534; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=p3sTQnA9zVQdLoH6FC615OvBI7Wkv91f4wZIK5mstHo=; b=CsLsyx1kvzA+zbYt/YNwoXtQtH9tAhurfkEL+B4jbGSNg48qpu2fcmuQS52tlkfqkX gpwBMoTJGV2VnWqHounAJHyDLNug4qvxPLZrz6hetwgpKq3knTMNSeu9DKTRmrNe6YE5 vj7TwyLy0EBr8n7QXZcUNFB86nvvi5uyth2iQsXgH9LspvAnhrSRN/8jhfpTQvoAxii+ lGTe9aKZPprm05gO3soXszPcAdWpO8C9u0KDha1rcixbLy35AjUQQ00hJQ8EuNlwKZYd 5tMWJdaS9wyFPgFLN/DG3ZqIZYRjHS7hF84T4v3bbz2pZa0orBYK4CjLfAy8qQk2MqWE eXEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709059734; x=1709664534; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p3sTQnA9zVQdLoH6FC615OvBI7Wkv91f4wZIK5mstHo=; b=wJAsm+cdwSgyL80LKfWE2EeFMi3zPiAzdFr17IdTtqtWpqU7DAgPc0Xm6zLiuLbxst 9hEIVN7ER74w41zeCg8JPqZehdg/2bjV9lRDu1eQ/AaMfEuzGxOVnlwsJKRaNX/+TYUS NfW5W6S+/6CuA0R6MLn3gphUsYnxyEOf1aWZR09uZVEBRpo9e4CTcVuJ+Zi5RcyGAVGZ xWbiDg5uJSix3LsvD+SvtUDO5gJQAUOqnVctlJ3HLwzxgBTYHALlNTKfJfG/h5o9oEeA moEZsbCtB+j8nUHK5PpdMEerpQuTuW+Pm7MFtb6dazM8ffWsJEPiFuo9FgRV28tyQHtG TdnA== X-Gm-Message-State: AOJu0YyRpwOD2uK6oN/teNqPQrECshUYkHRQk1+fzaLr5nG++Boys7uH dddV16mCnnHIjWVqBDQRHlx7U6t59PAirnUBFE5pcVS/2LZsfZUwvu8JaxzuNR3eHdc63nyb8Qm V X-Google-Smtp-Source: AGHT+IFec3DgfjSGRkuStsPiVwx0WR8MXgqoiHuTHdKI3Xaj92LbMeOtJuGm1B2tqx+DQBfjlEsiOg== X-Received: by 2002:a05:6e02:1aa3:b0:365:aeb8:26d1 with SMTP id l3-20020a056e021aa300b00365aeb826d1mr1847773ilv.0.1709059734166; Tue, 27 Feb 2024 10:48:54 -0800 (PST) Received: from localhost.localdomain ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id it2-20020a056a00458200b006e543b59587sm2282119pfb.126.2024.02.27.10.48.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Feb 2024 10:48:52 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] io_uring/net: improve the usercopy for sendmsg/recvmsg Date: Tue, 27 Feb 2024 11:46:32 -0700 Message-ID: <20240227184845.985268-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227184845.985268-1-axboe@kernel.dk> References: <20240227184845.985268-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We're spending a considerable amount of the sendmsg/recvmsg time just copying in the message header. And for provided buffers, the known single entry iovec. Be a bit smarter about it and enable/disable user access around our copying. In a test case that does both sendmsg and recvmsg, the runtime before this change (averaged over multiple runs, very stable times however): Kernel Time Diff ==================================== -git 4720 usec -git+commit 4311 usec -8.7% and looking at a profile diff, we see the following: 0.25% +9.33% [kernel.kallsyms] [k] _copy_from_user 4.47% -3.32% [kernel.kallsyms] [k] __io_msg_copy_hdr.constprop.0 where we drop more than 9% of _copy_from_user() time, and consequently add time to __io_msg_copy_hdr() where the copies are now attributed to, but with a net win of 6%. In comparison, the same test case with send/recv runs in 3745 usec, which is (expectedly) still quite a bit faster. But at least sendmsg/recvmsg is now only ~13% slower, where it was ~21% slower before. Signed-off-by: Jens Axboe --- io_uring/net.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 7a07c5563d66..83fba2882720 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -255,27 +255,42 @@ static int io_msg_copy_hdr(struct io_kiocb *req, struct io_async_msghdr *iomsg, struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); int ret; - if (copy_from_user(msg, sr->umsg, sizeof(*sr->umsg))) + if (!user_access_begin(sr->umsg, sizeof(*sr->umsg))) return -EFAULT; + ret = -EFAULT; + unsafe_get_user(msg->msg_name, &sr->umsg->msg_name, ua_end); + unsafe_get_user(msg->msg_namelen, &sr->umsg->msg_namelen, ua_end); + unsafe_get_user(msg->msg_iov, &sr->umsg->msg_iov, ua_end); + unsafe_get_user(msg->msg_iovlen, &sr->umsg->msg_iovlen, ua_end); + unsafe_get_user(msg->msg_control, &sr->umsg->msg_control, ua_end); + unsafe_get_user(msg->msg_controllen, &sr->umsg->msg_controllen, ua_end); + msg->msg_flags = 0; + if (req->flags & REQ_F_BUFFER_SELECT) { if (msg->msg_iovlen == 0) { sr->len = iomsg->fast_iov[0].iov_len = 0; iomsg->fast_iov[0].iov_base = NULL; iomsg->free_iov = NULL; } else if (msg->msg_iovlen > 1) { - return -EINVAL; + ret = -EINVAL; + goto ua_end; } else { - if (copy_from_user(iomsg->fast_iov, msg->msg_iov, - sizeof(*msg->msg_iov))) - return -EFAULT; + /* we only need the length for provided buffers */ + if (!access_ok(&msg->msg_iov[0].iov_len, sizeof(__kernel_size_t))) + goto ua_end; + unsafe_get_user(iomsg->fast_iov[0].iov_len, + &msg->msg_iov[0].iov_len, ua_end); sr->len = iomsg->fast_iov[0].iov_len; iomsg->free_iov = NULL; } - - return 0; + ret = 0; +ua_end: + user_access_end(); + return ret; } + user_access_end(); iomsg->free_iov = iomsg->fast_iov; ret = __import_iovec(ddir, msg->msg_iov, msg->msg_iovlen, UIO_FASTIOV, &iomsg->free_iov, &iomsg->msg.msg_iter, false);