From patchwork Thu Sep 8 12:20:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0A2CC38145 for ; Thu, 8 Sep 2022 12:22:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231694AbiIHMWr (ORCPT ); Thu, 8 Sep 2022 08:22:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231701AbiIHMWo (ORCPT ); Thu, 8 Sep 2022 08:22:44 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98221F6B99 for ; Thu, 8 Sep 2022 05:22:37 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id y3so37674684ejc.1 for ; Thu, 08 Sep 2022 05:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=D+ecdhJlRbp+3w2pEHxMWQ4ag1rTO2U6MFY9gi4eQAE=; b=pm+jDTOXOeHux5XVYr2rxaUQ3BWV7Uk5d+iS8s4CM1bkeHQMVFX5XTEF+8xSs55xz3 00pn8Jl42gsiisa1NRvLxafG+o4gulmf9S2AAw/CbHHeDQ2KbCBuCfp5cVki1/rUhXWR 42ZLdlgUZ/bLg8LwRaiZ9w+ETOMuQhzK781mjYHXEY4r3sGjqbvmjjwZwUMeFldKzSYi b8hQHYsOZFxS9NqdiY0tyn8gkZrCkw1WsEqbWJOUisFLwZRdtv3bdnObhXLTTJkmxBqP wJbA/cKpy06kadpOe5FwcgwvCbmGssKvEGmDvu6hUdmyCDvHlvdgt7q7rE9Sun16FkIO jzXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=D+ecdhJlRbp+3w2pEHxMWQ4ag1rTO2U6MFY9gi4eQAE=; b=6LUlLhsZRRpZXmQzQQKK8j/YrzgoDDA1qZ15LrmEiyqz0UWxQuFyxeDxTHaObW9dxt SY0TJOdFYRshL7ngxuo6gPrzmAiken3OrB8oKcvOOUxDOFs7J6K/DRpHmxE+qwuS/sWm RnvNkjybX2Cl9/tFy72vZoPQaa0N+sMVacxXNGLymqtuIGzDYFu+JyIPtz3ncoONIrhu DcRpQGKxEV3dO6wBdBEd0g3E5Y73UJRbHskyCSG6aMCEvV0ACoKIpStwJV0onA0r44jx PZl7WwS1PVk0Fm3RNOZywrZPSCqRFWo54OK8JHDzOjnN/YAnccFxpFnHsWnAQH5j4Tkz 9zUQ== X-Gm-Message-State: ACgBeo1W9mXSuvzMEgpqR4mzlqyNY/a6yoci9PmPsgimuPL31tyRMNG1 BJNJoS63/WVQAMhpGt+8h8tG3pXIofI= X-Google-Smtp-Source: AA6agR5csa/8f3C/oWGsD/7gKntbzhYSSV4Ix5Fwf9I4dMgGsmkLs32js8xu3xWCT5tqxEdbpG6omQ== X-Received: by 2002:a17:907:e89:b0:741:9f6d:fb88 with SMTP id ho9-20020a1709070e8900b007419f6dfb88mr5926454ejc.597.1662639755111; Thu, 08 Sep 2022 05:22:35 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:34 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/8] io_uring: kill an outdated comment Date: Thu, 8 Sep 2022 13:20:27 +0100 Message-Id: <38902e7229d68cecd62702436d627d4858b0d9d4.1662639236.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Request referencing has changed a while ago and there is no notion left of submission/completion references, kill an outdated comment. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0482087b7c64..339bc19a708a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1885,10 +1885,6 @@ static void io_queue_async(struct io_kiocb *req, int ret) io_req_task_queue(req); break; case IO_APOLL_ABORTED: - /* - * Queued up for async execution, worker will release - * submit reference when the iocb is actually submitted. - */ io_kbuf_recycle(req, 0); io_queue_iowq(req, NULL); break; From patchwork Thu Sep 8 12:20:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3088CC38145 for ; Thu, 8 Sep 2022 12:22:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231642AbiIHMWs (ORCPT ); Thu, 8 Sep 2022 08:22:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231703AbiIHMWo (ORCPT ); Thu, 8 Sep 2022 08:22:44 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 357B8F6BB5 for ; Thu, 8 Sep 2022 05:22:38 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id t5so24017111edc.11 for ; Thu, 08 Sep 2022 05:22:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=7+lV5qXCTNTjtoJhM9U/RPmNJbFpVO0bVaWeqGA3UqQ=; b=d1J9LYuiN2YnseNjl5OHkPCXb0u0LSJtAZD0Qs1xri80g1+yDudTYGWZ9pRN2M4p4j PgRH4Leg90Xz9sZ6/nq7oJY1aFjcMeKqo7M3npPsMc/lVas+Rr8q2waXUiXUAMdydC9b WmUh5zqCNp2NNBpEkB7FiDUPdslIdLrRRvSSmU5xcPhXWIL+CbPHc0bfYhzOUfLWisEG dHY1/7LDtTab0kThJxBOy0tiHnG0xfEsUnvOGpqpjjOXPYN/zxbnIwWm0H2Zd8FUZo/o OFD721R+5BZ8NsbMByQo8bgPLkEhRatQ9CK8yz/X/DahsFTcXjPaHxtQdVuzztnQlqff AEOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=7+lV5qXCTNTjtoJhM9U/RPmNJbFpVO0bVaWeqGA3UqQ=; b=rGg3pmv7aC8VnOJwk2rWRl8vreoPNGATfsvQmaR6pheSDbwGKwj+GLYQVaJNE8KEal gS+wxzxymI0elNUs6oKnaryVi8m4iDQCB481QWSFBDvkRrYO8Qnj+gZZl2FkwjiOTpyD rn57o+gAfaJIN8Aetyaqo3BGk10C/N6b1kM7202PtuuxwRfvofW+KBwfNVcx1F49ZfL7 19Ogira9inUvFFdj6sye6vT+4lKlxg8zk2PkGNf6PduAfxNQYqrqz8+w6yiMmGy/+lFj CCC/b68lST3e/dXYjmW+UE6VdKPF+1WSsl9KkMZPFt1uEf+muX0ur6XU4sMZYNHeefnN L5ig== X-Gm-Message-State: ACgBeo1/ZGIiHBWBTQk9jgM3G0g7AxRwYOlsEFavbeIGBNCwjI6J5iJE O6Ra5IEty5Y+IKtm+/bU87mtMqtAMCk= X-Google-Smtp-Source: AA6agR7gJWdDLcbo10UQfxn+dWnHGPDwZCk6xtGEuVUE5OxWpcDgXE+Cd1mtVqldfoWFfZqR3DnN8A== X-Received: by 2002:a05:6402:2693:b0:450:a807:6c91 with SMTP id w19-20020a056402269300b00450a8076c91mr2277168edd.33.1662639756013; Thu, 08 Sep 2022 05:22:36 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:35 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/8] io_uring: use io_cq_lock consistently Date: Thu, 8 Sep 2022 13:20:28 +0100 Message-Id: <91699b9a00a07128f7ca66136bdbbfc67a64659e.1662639236.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is one place when we forgot to change hand coded spin locking with io_cq_lock(), change it to be more consistent. Note, the unlock part is already __io_cq_unlock_post(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 339bc19a708a..b5245c5d102c 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1327,7 +1327,7 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx) struct io_wq_work_node *node, *prev; struct io_submit_state *state = &ctx->submit_state; - spin_lock(&ctx->completion_lock); + io_cq_lock(ctx); wq_list_for_each(node, prev, &state->compl_reqs) { struct io_kiocb *req = container_of(node, struct io_kiocb, comp_list); From patchwork Thu Sep 8 12:20:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47387C6FA82 for ; Thu, 8 Sep 2022 12:22:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231701AbiIHMWv (ORCPT ); Thu, 8 Sep 2022 08:22:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231716AbiIHMWr (ORCPT ); Thu, 8 Sep 2022 08:22:47 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B9C511C178 for ; Thu, 8 Sep 2022 05:22:39 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id bj12so37540269ejb.13 for ; Thu, 08 Sep 2022 05:22:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=TfGYSpotZEYF5vK2eLSyOzJp4EwTzlwApilpzC9vhkA=; b=PWiTxdDfR2FdfN7GGQvtwct7KR7BysxxwSJYzlxv7lvUd5OPlslrRlLIK0nPAKiVVh ZnbPifyuu2TpOZ0WT4mKn144O6fMqSsl+Q7+6ywISDxpc1cwR2pPlQnaDqLm57ZSHIcT mikwvt65E/rlUexnjUe/qcchLqaVbwshw9MeTK9zDkYjiTOsBPvNg1s8JCFkAQfTgKti QkiUmZb/7hNOI38IK1JgzucQmxJZQvnxHufG+SDPGjH6/5qx51T8M5UEVU0u3rttVd6p 5cl+q2TmY/ob6GOxQIcqQphl5EYrCaoFQ2MAsqPXGzCGzclZiR+bNTkob0IlOU8rkpEt /YoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=TfGYSpotZEYF5vK2eLSyOzJp4EwTzlwApilpzC9vhkA=; b=kCGDEXinwDuHFls+ht4oUCmH26oyuEjtmswIfgPRmavf0MArYB40cP+0chg628mFLE 591BFS4+I30q8ELJznnjqr98/HGmmpGbapnE7Lz7EIFDW4FRUSj2yhNEJU/0N9GBVwta gpOq7lm3q5C4PA/gbPwkj8KP5sNJJSbF46k7uD2ytWMptxs2ckFDGPWBhhErkdLqHJfC 5EWZQH3q92BbaWcLdIcF1SJxqEv2YjgFo8O2wgz30TladzOVlfSHeCIzZSLsxvVGzVaJ iB9h0I+4Z33DOoX61KsfufNX2AO4UPmAZKIic+BAlP1ijDh9fb2hpoFuoWnovxnW0IhQ 3K+A== X-Gm-Message-State: ACgBeo3yLoSXrlUw2GxRX7tFAJFfSSJClTB9AdXfTjKKc378kapjMVi8 1WuRKqkiFobTCpAkor56/XtVmn1rTqg= X-Google-Smtp-Source: AA6agR6OsC84dKTCA8zulVu7bFyP/cyhXJPv5mWJapsDU2ol4wpPW1qZlCMX+Yiyml10NsD1d8XqLA== X-Received: by 2002:a17:907:a422:b0:73f:18a8:4137 with SMTP id sg34-20020a170907a42200b0073f18a84137mr6094427ejc.10.1662639756992; Thu, 08 Sep 2022 05:22:36 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/8] io_uring/net: reshuffle error handling Date: Thu, 8 Sep 2022 13:20:29 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We should prioritise send/recv retry cases over failures, they're more important. Shuffle -ERESTARTSYS after we handled retries. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 7047c1342541..0bba804a955d 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -291,13 +291,13 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) if (ret < min_ret) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return io_setup_async_msg(req, kmsg, issue_flags); - if (ret == -ERESTARTSYS) - ret = -EINTR; if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; req->flags |= REQ_F_PARTIAL_IO; return io_setup_async_msg(req, kmsg, issue_flags); } + if (ret == -ERESTARTSYS) + ret = -EINTR; req_set_fail(req); } /* fast path, check for non-NULL to avoid function call */ @@ -352,8 +352,6 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) if (ret < min_ret) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return -EAGAIN; - if (ret == -ERESTARTSYS) - ret = -EINTR; if (ret > 0 && io_net_retry(sock, flags)) { sr->len -= ret; sr->buf += ret; @@ -361,6 +359,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) req->flags |= REQ_F_PARTIAL_IO; return -EAGAIN; } + if (ret == -ERESTARTSYS) + ret = -EINTR; req_set_fail(req); } if (ret >= 0) @@ -751,13 +751,13 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) } return ret; } - if (ret == -ERESTARTSYS) - ret = -EINTR; if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; req->flags |= REQ_F_PARTIAL_IO; return io_setup_async_msg(req, kmsg, issue_flags); } + if (ret == -ERESTARTSYS) + ret = -EINTR; req_set_fail(req); } else if ((flags & MSG_WAITALL) && (kmsg->msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC))) { req_set_fail(req); @@ -847,8 +847,6 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) return -EAGAIN; } - if (ret == -ERESTARTSYS) - ret = -EINTR; if (ret > 0 && io_net_retry(sock, flags)) { sr->len -= ret; sr->buf += ret; @@ -856,6 +854,8 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) req->flags |= REQ_F_PARTIAL_IO; return -EAGAIN; } + if (ret == -ERESTARTSYS) + ret = -EINTR; req_set_fail(req); } else if ((flags & MSG_WAITALL) && (msg.msg_flags & (MSG_TRUNC | MSG_CTRUNC))) { out_free: From patchwork Thu Sep 8 12:20:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AB80C6FA86 for ; Thu, 8 Sep 2022 12:22:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231703AbiIHMWw (ORCPT ); Thu, 8 Sep 2022 08:22:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231718AbiIHMWs (ORCPT ); Thu, 8 Sep 2022 08:22:48 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC7CC127570 for ; Thu, 8 Sep 2022 05:22:39 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id bj12so37540341ejb.13 for ; Thu, 08 Sep 2022 05:22:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=yUvpV3mjVhkBX7spqb4A/4OznVW615tmAtfHpse7uww=; b=OVfSI6Vs8AauwJc+3e6qlOXJrdaj6KxAz2mDk+eC3FLIM5pmV6yAZ/x/Kui03Ol8Pi HpTVpsAqHPuXNNPc1LrI5KpSmcqXtVRSPfCGf3g3lmyKsV7+8JrMP06YEv4RkUimOhpS AH7KGaT642khRfTFmAMnuATphmWHlVPxl+azJTruKm6QkIkqOeHSd3SPNwmZfm4ZFtl3 +HCH7drUJXE8cqMlR21/1F7xRppDyZUc7kUlOY+ZCU+dzdioA+Q/Y5q0ghXIR1c8y7GY RR6HyLMVku5i2m4JTLvVXH+kcRsojY+pYhg3l6PRMEOOCnYAeeeDBDsPq6aGN3i45fZ8 SlkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=yUvpV3mjVhkBX7spqb4A/4OznVW615tmAtfHpse7uww=; b=Qvh71t8ijMxtJKFEkoFLDBylqmBIxYl+EQe3XgzC+lsritgL0t0NK1hf/Bs0D6Ixry 0NIRv/WOenEeZqEfMnylPW5EbXNEmguKI+MCOpCDiIld8xLi6bWCFUEm4u2DzFeIe1FC 0GadzX6sXZUnzBQWjMF7rTLDb2TpigihFc6sBzFew9ovrqicGo8B9LdsCXQKT7ylvw03 gVjLt5/bEE4bUdVEaa5Dz9SLTfrl+3ZX7nLrqJlKaWnEQ/7HH78uVFDM9mY1Nq2uJRH+ UKyEGiEZlY8CJlHfIQ736TMIzwGRPLKcPgOqWUtIe+uvMU++xmz6wL9gDaYImKlDCNAp SQcg== X-Gm-Message-State: ACgBeo1gzd0E7PyAaVHgPrKBTlpxRMUIiwI7RSL5Jdt/1aPWTmojfIJY /OaA75xKV1BvWBkTmjjMB4YcXNdv2Yc= X-Google-Smtp-Source: AA6agR5SJxhlL8qXo/DyGzkpGM6O6/jV5MiyqykwrK4aM9Gs5OZ6akuz4qFQCmFlxBXGPymcT7ksmg== X-Received: by 2002:a17:907:3186:b0:777:3fe7:4659 with SMTP id xe6-20020a170907318600b007773fe74659mr1067929ejb.336.1662639757840; Thu, 08 Sep 2022 05:22:37 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 4/8] io_uring/net: use async caches for async prep Date: Thu, 8 Sep 2022 13:20:30 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org send/recv have async_data caches but there are only used from within issue handlers. Extend their use also to ->prep_async, should be handy with links and IOSQE_ASYNC. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 16 +++++++++++++--- io_uring/opdef.c | 2 ++ 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 0bba804a955d..fa54a35191d7 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -126,8 +126,8 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } } -static struct io_async_msghdr *io_recvmsg_alloc_async(struct io_kiocb *req, - unsigned int issue_flags) +static struct io_async_msghdr *io_msg_alloc_async(struct io_kiocb *req, + unsigned int issue_flags) { struct io_ring_ctx *ctx = req->ctx; struct io_cache_entry *entry; @@ -148,6 +148,12 @@ static struct io_async_msghdr *io_recvmsg_alloc_async(struct io_kiocb *req, return NULL; } +static inline struct io_async_msghdr *io_msg_alloc_async_prep(struct io_kiocb *req) +{ + /* ->prep_async is always called from the submission context */ + return io_msg_alloc_async(req, 0); +} + static int io_setup_async_msg(struct io_kiocb *req, struct io_async_msghdr *kmsg, unsigned int issue_flags) @@ -156,7 +162,7 @@ static int io_setup_async_msg(struct io_kiocb *req, if (req_has_async_data(req)) return -EAGAIN; - async_msg = io_recvmsg_alloc_async(req, issue_flags); + async_msg = io_msg_alloc_async(req, issue_flags); if (!async_msg) { kfree(kmsg->free_iov); return -ENOMEM; @@ -217,6 +223,8 @@ int io_sendmsg_prep_async(struct io_kiocb *req) { int ret; + if (!io_msg_alloc_async_prep(req)) + return -ENOMEM; ret = io_sendmsg_copy_hdr(req, req->async_data); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; @@ -504,6 +512,8 @@ int io_recvmsg_prep_async(struct io_kiocb *req) { int ret; + if (!io_msg_alloc_async_prep(req)) + return -ENOMEM; ret = io_recvmsg_copy_hdr(req, req->async_data); if (!ret) req->flags |= REQ_F_NEED_CLEANUP; diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 04693e4a33c7..c6e089900394 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -146,6 +146,7 @@ const struct io_op_def io_op_defs[] = { .unbound_nonreg_file = 1, .pollout = 1, .ioprio = 1, + .manual_alloc = 1, .name = "SENDMSG", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), @@ -163,6 +164,7 @@ const struct io_op_def io_op_defs[] = { .pollin = 1, .buffer_select = 1, .ioprio = 1, + .manual_alloc = 1, .name = "RECVMSG", #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), From patchwork Thu Sep 8 12:20:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10FABC54EE9 for ; Thu, 8 Sep 2022 12:22:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231676AbiIHMWv (ORCPT ); Thu, 8 Sep 2022 08:22:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231719AbiIHMWs (ORCPT ); Thu, 8 Sep 2022 08:22:48 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0151E12C4B4 for ; Thu, 8 Sep 2022 05:22:40 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id gh9so15725883ejc.8 for ; Thu, 08 Sep 2022 05:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=QtGYKdepH6VIklXqjHVUhOxm0sOXFXaUexJ5gRR/+aQ=; b=XTJDQBy0Qar2pFhyrf8mlZV/Q8940Ge0M5sfmws6PD+lHuwA9Y4Mprc130InGMk3Pc S3FI6+LvS9Z++Z5rCzoajlBHkbd1f08S51tMm9tpWlHE95eF4q0zVHZsf5RvjFBSCuib X/GQTCWaghieoDc2JIkBa1E8d5lMJWBrX/v5XiSOPsb0YCo11NC5qd+hMTjZV4z2AQPq ClLjfTjb/y/a4fhw28Dt/gMHa+qLnj/6lg7yn+4DnwBfIYZZ+XV7fk61x9OCCh2+k8X6 fFGoWplSTBrpbQDFmZgWBXO8RsTWuFW5gKdzroktUFNVH4fq572sevfmDEErZ6wiWbSu nm+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=QtGYKdepH6VIklXqjHVUhOxm0sOXFXaUexJ5gRR/+aQ=; b=bzchoNnXIZLDYCEP9b256tx8WX/1Kxs0tZCpeKfFF9Dto5J5/+UG09nJTHe1eJUd1C QuGbJS2LnEp66D7LKfkw2Kh4e1aA+A7WDIFXoIe8JGwIu95VKiiOz9SDAN/hMxqjCy+W brAn7KH2egbm73Ef7ClPy74g2TWhxlKwOc2emnYB8WGRICzd35B6Q/KH+Muov7Uf4I4/ P/a/TFifzPOz1cE1okK1M+xLznnCYNsfEuq0+4/AEO8ZyfZngoIxxXvJdDpQspigpIOg 7NUOK46EYWk3tebT6OD76h46hwLtHc9G32066tkRAgIIvD99mXwuExd1RkLUDrNrw+Pb ATfQ== X-Gm-Message-State: ACgBeo0j6XsOpW4G39P3uHx46CkWMsRQ3DC9wyH/bIxsNA/s/aMBLs0S 841FHzLDnBCDaUsPCm7mGxs+lnYXP2Y= X-Google-Smtp-Source: AA6agR7LWmK/IPgDPLQLGYSC3VCRyXVMiFcBfRV5ZR09Uvsd+NU8m6O4ufNEPMQkPWZi8Zln466zMQ== X-Received: by 2002:a17:907:7241:b0:779:2fc:9a40 with SMTP id ds1-20020a170907724100b0077902fc9a40mr101662ejc.173.1662639758775; Thu, 08 Sep 2022 05:22:38 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 5/8] io_uring/net: io_async_msghdr caches for sendzc Date: Thu, 8 Sep 2022 13:20:31 +0100 Message-Id: <42fa615b6e0be25f47a685c35d7b5e4f1b03d348.1662639236.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We already keep io_async_msghdr caches for normal send/recv requests, use them also for zerocopy send. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index fa54a35191d7..ff1fed00876f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -196,10 +196,9 @@ int io_sendzc_prep_async(struct io_kiocb *req) if (!zc->addr || req_has_async_data(req)) return 0; - if (io_alloc_async_data(req)) + io = io_msg_alloc_async_prep(req); + if (!io) return -ENOMEM; - - io = req->async_data; ret = move_addr_to_kernel(zc->addr, zc->addr_len, &io->addr); return ret; } @@ -212,9 +211,9 @@ static int io_setup_async_addr(struct io_kiocb *req, if (!addr || req_has_async_data(req)) return -EAGAIN; - if (io_alloc_async_data(req)) + io = io_msg_alloc_async(req, issue_flags); + if (!io) return -ENOMEM; - io = req->async_data; memcpy(&io->addr, addr, sizeof(io->addr)); return -EAGAIN; } From patchwork Thu Sep 8 12:20:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30DFCC6FA8A for ; Thu, 8 Sep 2022 12:22:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231718AbiIHMWx (ORCPT ); Thu, 8 Sep 2022 08:22:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231723AbiIHMWt (ORCPT ); Thu, 8 Sep 2022 08:22:49 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB46E1316CA for ; Thu, 8 Sep 2022 05:22:41 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id y3so37675064ejc.1 for ; Thu, 08 Sep 2022 05:22:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=UCrnjc/M5Xx+EyDrYtQMAIxMwOtjvwYqP0u2cAd22I8=; b=D6T7raRpuFsJ27rgeDhm4bviuKPFOiqnsWzPhDSIwWZEYSLZw5sA4pEppSl0iWp+cA RK0Nnv9ssov5u8FDvocbmFI42Vv28cWGqQ7WPIIUo3AMoAY0Cb7K/LOlyd3x/02SQOV4 Wsmnaqd+bAtDQJGk1aZNh8vtJzOEua/QiwJA3qhH4GmPWzfugwLQrlv4H4Z0cg2ZsaY5 Z4bHx7rLqpuP7D7Bj9vK0iW2ETA5YPP0h3R+DJT3ZVbk0ni/c5JpAO8ryvbQ28NHSYxi TWh6etHbZxQS4yO3/kLp6iusoA8Rc2t7wSaQCR+BDyCIVkCCP4jUm2/X0Keoz3YC86LB FaHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=UCrnjc/M5Xx+EyDrYtQMAIxMwOtjvwYqP0u2cAd22I8=; b=cMU+a3u/uMQlaWX5SNC+ckLDapVMTYdT6KrMoDFcJjORHa3LsYtipego5W4HqTI3Xe lsw8HzYhUzJr4+dVz4gaRanL+PwhuVbWFFyiy8JEdfpFC66HKWHi9IxfygpsC51cgxgI B4hSB8OtNT3r/kOLfibHOtm8PfW1Ge0mV9Jo5tz26beW0oKRTzkNtvOejmYbyywkQZgR F/Mh+j8kqGU6JACCnZnm0Bqs33en4M/QCtZyvBFi9EKSC92RbwZIoLMfkMku3upOuUr1 gcWjRt+uWXZLMYUXEsPh8VYwgdoSHcRLj0ZN85v9FFwOg+JNZgjFDs06d73By+gbp3T3 wWAw== X-Gm-Message-State: ACgBeo2G6VfPEULdB0xC9PX8K4b3UX1L9cIQjOhnpOwvMLfVHhMAaUp2 /KTcq70sGiZ2UCA6j+uFLyaXcAYAqjI= X-Google-Smtp-Source: AA6agR6nOzkVl2CTS6DNle+El+0cp0zT1pOxYxYw9/CGOn/5dt5WisM4vuDMryCyrVwdFxhCA6grrg== X-Received: by 2002:a17:906:ef90:b0:730:9cd8:56d7 with SMTP id ze16-20020a170906ef9000b007309cd856d7mr5620858ejb.94.1662639759723; Thu, 08 Sep 2022 05:22:39 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 6/8] io_uring/net: add non-bvec sg chunking callback Date: Thu, 8 Sep 2022 13:20:32 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a sg_from_iter() for when we initiate non-bvec zerocopy sends, which helps us to remove some extra steps from io_sg_from_iter(). The only thing the new function has to do before giving control away to __zerocopy_sg_from_iter() is to check if the skb has managed frags and downgrade them if so. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index ff1fed00876f..4dbdb59968c3 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -948,6 +948,13 @@ int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static int io_sg_from_iter_iovec(struct sock *sk, struct sk_buff *skb, + struct iov_iter *from, size_t length) +{ + skb_zcopy_downgrade_managed(skb); + return __zerocopy_sg_from_iter(NULL, sk, skb, from, length); +} + static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, struct iov_iter *from, size_t length) { @@ -958,13 +965,10 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, ssize_t copied = 0; unsigned long truesize = 0; - if (!shinfo->nr_frags) + if (!frag) shinfo->flags |= SKBFL_MANAGED_FRAG_REFS; - - if (!skb_zcopy_managed(skb) || !iov_iter_is_bvec(from)) { - skb_zcopy_downgrade_managed(skb); + else if (unlikely(!skb_zcopy_managed(skb))) return __zerocopy_sg_from_iter(NULL, sk, skb, from, length); - } bi.bi_size = min(from->count, length); bi.bi_bvec_done = from->iov_offset; @@ -1044,6 +1048,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) (u64)(uintptr_t)zc->buf, zc->len); if (unlikely(ret)) return ret; + msg.sg_from_iter = io_sg_from_iter; } else { ret = import_single_range(WRITE, zc->buf, zc->len, &iov, &msg.msg_iter); @@ -1052,6 +1057,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) ret = io_notif_account_mem(zc->notif, zc->len); if (unlikely(ret)) return ret; + msg.sg_from_iter = io_sg_from_iter_iovec; } msg_flags = zc->msg_flags | MSG_ZEROCOPY; @@ -1062,7 +1068,6 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_flags = msg_flags; msg.msg_ubuf = &io_notif_to_data(zc->notif)->uarg; - msg.sg_from_iter = io_sg_from_iter; ret = sock_sendmsg(sock, &msg); if (unlikely(ret < min_ret)) { From patchwork Thu Sep 8 12:20:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12073C6FA82 for ; Thu, 8 Sep 2022 12:22:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231719AbiIHMWx (ORCPT ); Thu, 8 Sep 2022 08:22:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231730AbiIHMWt (ORCPT ); Thu, 8 Sep 2022 08:22:49 -0400 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A8C91316D1 for ; Thu, 8 Sep 2022 05:22:42 -0700 (PDT) Received: by mail-ej1-x62b.google.com with SMTP id bj12so37540570ejb.13 for ; Thu, 08 Sep 2022 05:22:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=GB9BOyQODze5Kg82+IrDJOynNmHrAMiNDdnDkl3bHr4=; b=p2D+IMj+zJC+TIIteOvukNaHtUlcnpiBGbQceD0tkpCv957T5QX3roOucKQmeCrRi1 ViyL/DOF+uCIK94CiahKVHH3vlGglhGrrAXqWvJxdV3VKzcudWvNOhgoHDIABqeyHl9a p5JnL5ft2N3oYcTr3EgzXSvpKgkSyrw6IEqGuqQEeTe1UAB6o+BkmH8A6bb5xFXMZxKx Glm/UpERZ+Z7SGbMLuJFnlcbLAz/TGfdOo/J4EYJQrsDo0br8TVq/5S/swQUP5DPoEvZ MlaBwGAaPox2CHgb+pLyv8oOTeAaBc3SDf3X9JkDwMI1fF21h8beFVNTcBuh3UiORcxA 4hmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=GB9BOyQODze5Kg82+IrDJOynNmHrAMiNDdnDkl3bHr4=; b=CNxAruZZe/DXnp0G/9O0LyuN91OOiijtzlyicZzQ9UHnyhVXfdI3DmqhDzrWhgJbSz nH511n9zVOoKYAf77zDZTnoPfhds+PI5/FnYzPptTOX3yl8BMlRajqX6uYZm8lNSLXKs Bg72YJOE3t3RX7mC6z/JL7UhNsX7F8d93MQqkAsTKYeMFXFEewrRE3P8YgdL+NuACM3E zEnoxM32WCnH/Pe7BOxOd3ToUdAfwcDcpm/+uTnTE+EwrAEvT3kyz6wPahqyWYxxCJz1 5M6NfHxHMvfDPWDmJc0AkvbWDjGESWeNjezQZttayGla/que8UPEHNGLTttCCniEomm7 tEXA== X-Gm-Message-State: ACgBeo0z+Um+XeSGKJ7TU9FS723I7YfeRAFgxaS3N4cmHcYa2IEZzEp0 ycHLljSZnABtl3EZS0IsjxE3toLMTkM= X-Google-Smtp-Source: AA6agR7JsWOagimuwgu/sCtQWUdXfcX/TlKE/hVlOPHNit6kqIybirqhzDG1AqpzwZIpm8FuijlCAA== X-Received: by 2002:a17:907:7242:b0:741:7cd6:57d5 with SMTP id ds2-20020a170907724200b007417cd657d5mr5802799ejc.419.1662639760559; Thu, 08 Sep 2022 05:22:40 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 7/8] io_uring/net: refactor io_sr_msg types Date: Thu, 8 Sep 2022 13:20:33 +0100 Message-Id: <42c2639d6385b8b2181342d2af3a42d3b1c5bcd2.1662639236.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org In preparation for using struct io_sr_msg for zerocopy sends, clean up types. First, flags can be u16 as it's provided by the userspace in u16 ioprio, as well as addr_len. This saves us 4 bytes. Also use unsigned for size and done_io, both are as well limited to u32. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 4dbdb59968c3..97778cd1306c 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -55,21 +55,21 @@ struct io_sr_msg { struct user_msghdr __user *umsg; void __user *buf; }; + unsigned len; + unsigned done_io; unsigned msg_flags; - unsigned flags; - size_t len; - size_t done_io; + u16 flags; }; struct io_sendzc { struct file *file; void __user *buf; - size_t len; + unsigned len; + unsigned done_io; unsigned msg_flags; - unsigned flags; - unsigned addr_len; + u16 flags; + u16 addr_len; void __user *addr; - size_t done_io; struct io_kiocb *notif; }; From patchwork Thu Sep 8 12:20:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB030C6FA8B for ; Thu, 8 Sep 2022 12:22:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231722AbiIHMWy (ORCPT ); Thu, 8 Sep 2022 08:22:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231727AbiIHMWt (ORCPT ); Thu, 8 Sep 2022 08:22:49 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 470EC1316DC for ; Thu, 8 Sep 2022 05:22:43 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id u9so37667570ejy.5 for ; Thu, 08 Sep 2022 05:22:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=6CCpCBGOxRxa+KZZ3A1bwS94jvsCJ2VLNeWvIj0f7nk=; b=op9L9OVZMwpLPUX1sk1+w0XjzO9cCF/8tnXLcECg8EQyV/bzU9z2WHk8TaCUBMAro+ VlsgOTEDA+sH7g3d18bxM+OIRJ+tZm3vOf31GSUP2S3rbc9Wp3WeT2eC8WIWFgMZ/WHL ja/m7/sR/6vmZMXZHqOumsQZxp3mv3wKmxlXcNrmTyqRal/2qVgJ4MeWgHAJUczVBNU5 rfxDnOKYb8UpbAfPiQEME7d5atV0IAItg8xk6kCwOtnfHR0YcvZGkaty54N/kzMNL+vA zpyfMVRCHIg76r5A6V3zfqhYMwICOpcQR94M8rb9PCCF2Le5nLTOFZ5bGm+nxlgITx9Q pWMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=6CCpCBGOxRxa+KZZ3A1bwS94jvsCJ2VLNeWvIj0f7nk=; b=MZ/HpgaCKIsNdxd7h3c6FFFUo2I1IVpN43mfEvYuHOt6QDXBoYaWHX9i3xkaKizQ1q qHQxftFUpjQFVxjpRMTiyEjENpTBYdEENad84tLkEq2S+BJHAh1jeg3hSsFXg/amFNzN o6NuyZdsLFRPCm35BpOySm4eG/o+LugQ0ECOgqJ9Rk4vxasmU/3MtI93osC02+wcC+sn w43g6xiHAsu3bbe2S0FxXrh2GBGy3yMlgv25bv9WOLbYFB622Xwm78aoobrI6682dNr9 60MIcYfsY4L9TcJjrDuMupEjnVTmLMfAXQwRED2g13dXe+uF0EhL1dWHaA7CMR3vgVQP TP6Q== X-Gm-Message-State: ACgBeo24QcU4mcdhGCG+TYZnqWnHgtMcRXRHOVPirON972tLMt045Bcn BMOTODOEGWTaok0USYpJQtdfak38Tw8= X-Google-Smtp-Source: AA6agR7MUoEJq5S2WKSHSXllnwwUuv7ncyESn768UfZ8xDjYqAKM02yO95bSe8NfxwZCUrS4CH5KBg== X-Received: by 2002:a17:907:6d99:b0:770:7e4c:ba15 with SMTP id sb25-20020a1709076d9900b007707e4cba15mr5777895ejc.259.1662639761401; Thu, 08 Sep 2022 05:22:41 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id p9-20020a17090653c900b0074a82932e3bsm1191791ejo.77.2022.09.08.05.22.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 05:22:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 8/8] io_uring/net: use io_sr_msg for sendzc Date: Thu, 8 Sep 2022 13:20:34 +0100 Message-Id: <408c5b1b2d8869e1a12da5f5a78ed72cac112149.1662639236.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Reuse struct io_sr_msg for zerocopy sends, which is handy. There is only one zerocopy specific field, namely .notif, and we have enough space for it. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 97778cd1306c..acafb2e3dd09 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -59,15 +59,7 @@ struct io_sr_msg { unsigned done_io; unsigned msg_flags; u16 flags; -}; - -struct io_sendzc { - struct file *file; - void __user *buf; - unsigned len; - unsigned done_io; - unsigned msg_flags; - u16 flags; + /* used only for sendzc */ u16 addr_len; void __user *addr; struct io_kiocb *notif; @@ -190,7 +182,7 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, int io_sendzc_prep_async(struct io_kiocb *req) { - struct io_sendzc *zc = io_kiocb_to_cmd(req, struct io_sendzc); + struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *io; int ret; @@ -890,7 +882,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) void io_sendzc_cleanup(struct io_kiocb *req) { - struct io_sendzc *zc = io_kiocb_to_cmd(req, struct io_sendzc); + struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); zc->notif->flags |= REQ_F_CQE_SKIP; io_notif_flush(zc->notif); @@ -899,7 +891,7 @@ void io_sendzc_cleanup(struct io_kiocb *req) int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { - struct io_sendzc *zc = io_kiocb_to_cmd(req, struct io_sendzc); + struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_ring_ctx *ctx = req->ctx; struct io_kiocb *notif; @@ -1009,7 +1001,7 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) { struct sockaddr_storage __address, *addr = NULL; - struct io_sendzc *zc = io_kiocb_to_cmd(req, struct io_sendzc); + struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct msghdr msg; struct iovec iov; struct socket *sock;