From patchwork Wed Sep 21 11:17:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35D14C32771 for ; Wed, 21 Sep 2022 11:20:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229545AbiIULUq (ORCPT ); Wed, 21 Sep 2022 07:20:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229498AbiIULUo (ORCPT ); Wed, 21 Sep 2022 07:20:44 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41E9670E44 for ; Wed, 21 Sep 2022 04:20:43 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id az6so4189153wmb.4 for ; Wed, 21 Sep 2022 04:20:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=uEeWCm94KOo7VZwDqBvTmmSVgBJlmWvoPwkZEcEjcrA=; b=LpZu15pMAJiR5RNtbYdbI9TV53fr3/sLz7AauD1Fcmnk0irl3SntbXOepAzoVHpzqc I65U69Q/32rx5PUAz5mgFuHRuYZFIDWThIvSBWTDmqF/y7XB/bNZKy5tbXKeUSWbSEQ+ 1Os47wgsjyTjxhk0EZFLSrNYiA/sO+OZ8VNcGLu0QCcr1k6nw5uEs34G5/Q2YWwnrBzJ u6sphbD0A2wJWJuTIQmcKPhiqVx8uVzEHh5q08guum8OLa9EShlXlcK+M2DFWIx3yweb KDZf2dH4563tmTE+gxL4felrQ2BeWbBCeoAbxiZS1Chh+m5BlFuaLgfexR3WATDCEkJw 2Vxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=uEeWCm94KOo7VZwDqBvTmmSVgBJlmWvoPwkZEcEjcrA=; b=WmrOsSkgxBcwrf+r7XV3Zq7JhJ6bGqi4YbEfxCBgA6/qwVqZnQ9fQRhNlVLGcyOUrA XCsyPX/R3T2egJUxUmM5NqzOJHShYkAvrxDaOm0uSozDL1ud0I3AX1oCjbRDxtFlUrbj noWavl2tWetejci1pUIFnR+f2kWOvXlglfIaV0Q9A6eHIPkdlhVn5BCC7MonKtNEk8dn fjw0BM3ncCw0jG+RZ9UdAGFBq91BKSsr/Egs5xg0DU4bdb97Yy9duykTQgzYekEKkAu/ WLLSMj+GU6bDM/cU2iVB65TcRY3l6A6Eadc9g69VNAXQnGqJk3CsLGza5SMcC6yvNjEb I2Mg== X-Gm-Message-State: ACrzQf2psfTFJrZmRXodhyKynAeaNa6LUbfbSjZ8M833f32sIJO+AUWh NvDr0OPOOS2Z8GyOeykrCWZQORzSYUg= X-Google-Smtp-Source: AMsMyM5ihFLim9R7SDnN2PC1YLmcBbWjbH8DvLuTOSqgr1nXudgIhpFKmU5TCPYemmmn82qrFUdZIg== X-Received: by 2002:a7b:ce0d:0:b0:3b4:8728:3e7e with SMTP id m13-20020a7bce0d000000b003b487283e7emr5294340wmc.182.1663759241382; Wed, 21 Sep 2022 04:20:41 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 1/9] io_uring: add custom opcode hooks on fail Date: Wed, 21 Sep 2022 12:17:46 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Sometimes we have to do a little bit of a fixup on a request failuer in io_req_complete_failed(). Add a callback in opdef for that. Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 4 ++++ io_uring/opdef.h | 1 + 2 files changed, 5 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 433466455a5f..3875ea897cdf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -865,8 +865,12 @@ inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) void io_req_complete_failed(struct io_kiocb *req, s32 res) { + const struct io_op_def *def = &io_op_defs[req->opcode]; + req_set_fail(req); io_req_set_res(req, res, io_put_kbuf(req, IO_URING_F_UNLOCKED)); + if (def->fail) + def->fail(req); io_req_complete_post(req); } diff --git a/io_uring/opdef.h b/io_uring/opdef.h index 763c6e54e2ee..3efe06d25473 100644 --- a/io_uring/opdef.h +++ b/io_uring/opdef.h @@ -36,6 +36,7 @@ struct io_op_def { int (*issue)(struct io_kiocb *, unsigned int); int (*prep_async)(struct io_kiocb *); void (*cleanup)(struct io_kiocb *); + void (*fail)(struct io_kiocb *); }; extern const struct io_op_def io_op_defs[]; From patchwork Wed Sep 21 11:17:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFAB0ECAAD8 for ; Wed, 21 Sep 2022 11:20:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229788AbiIULUs (ORCPT ); Wed, 21 Sep 2022 07:20:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229498AbiIULUs (ORCPT ); Wed, 21 Sep 2022 07:20:48 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EF996FA31 for ; Wed, 21 Sep 2022 04:20:44 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id i203-20020a1c3bd4000000b003b3df9a5ecbso8842803wma.1 for ; Wed, 21 Sep 2022 04:20:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=7SrEQCUZFJdvUZqJql9L8RzvaOosvdHtufDGMGa9UeI=; b=S48fZWCVUxi9xKrw+Nl5JKAPpHsjXBQ1+jmvL1VmldfxZIAW19DBwEKIcWMmBUofaS Xvum3HlobAQbKeEBcwJT7KNKuTunG/rRRJ0j/y6OTq2yn3H3L23LKsgsxUjt7vC1ZcjE eCBtqcwEC3kXymh0pKWIPRJ5JDMGxzr7x8vcE0SsAq3QVMeAIOBQp3rYOwXSj3Yj3T78 pti64xvIw0S559Y9UfDK4TpcbaYb2aSw+jq6wq/sEkI9qV+t6q1vjnzcA51SNbZL7Qn2 fKJwPFgN33wa22eWhyfxwdnLGpa/wNqf9/m/8jCoMCEXBHOl+ToAJ/V6f5S8gX2Ywloq UG1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=7SrEQCUZFJdvUZqJql9L8RzvaOosvdHtufDGMGa9UeI=; b=dQVS6e7p0Ku7gTIowBSZ3fNwBeerPKNNBjTF7HUeg14h1tU/ePIKGxxhM2x3x5oZOP jAenrAYq829ZnWcerJGKNGAG5PiGd2HQozHVUNCNLR8vkPTwq6j5FmG5ahcfm/oWgPWi m97NwZqWE//Gh/wU/5/6lXODCH83zO31DkfAFx/AichOdG/RLzfwqH6Z+U9Ea/nBvGlv mRQwn7DlARsd3ibZuBi2BhLZi3kG33mJrW4LX6qpuaBWWwsFWMfH10kzRMf6C0bKSEHC q9sLO7VbKXstAXfniy4TYXDQi2aqWNx6iGJ61QeP2gXebjdrlo9Bw1WT9LpC6UJOyeFI IzxA== X-Gm-Message-State: ACrzQf2y7AYVn64m9NrtiXzvdrGAeJdkUpuR6C0a/H4a3nU1Owro/rsd HQw/wM65aEUTNYXhS9mWjJsJ2z13Rrc= X-Google-Smtp-Source: AMsMyM6mmIcXH3/TSCbiwD4CUZ/C0AjGS/0n9Uufd5u5/rV6+Crajf4f0uZJLkBIPQEBM52kE5PotQ== X-Received: by 2002:a05:600c:2043:b0:3b3:c84c:ca33 with SMTP id p3-20020a05600c204300b003b3c84cca33mr5495455wmg.15.1663759242781; Wed, 21 Sep 2022 04:20:42 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/9] io_uring/rw: don't lose partial IO result on fail Date: Wed, 21 Sep 2022 12:17:47 +0100 Message-Id: <05e0879c226bcd53b441bf92868eadd4bf04e2fc.1663668091.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org A partially done read/write may end up in io_req_complete_failed() and loose the result, make sure we return the number of bytes processed. Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov --- io_uring/opdef.c | 6 ++++++ io_uring/rw.c | 8 ++++++++ io_uring/rw.h | 1 + 3 files changed, 15 insertions(+) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index c6e089900394..788393ec3ff4 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -69,6 +69,7 @@ const struct io_op_def io_op_defs[] = { .issue = io_read, .prep_async = io_readv_prep_async, .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, }, [IORING_OP_WRITEV] = { .needs_file = 1, @@ -85,6 +86,7 @@ const struct io_op_def io_op_defs[] = { .issue = io_write, .prep_async = io_writev_prep_async, .cleanup = io_readv_writev_cleanup, + .fail = io_rw_fail, }, [IORING_OP_FSYNC] = { .needs_file = 1, @@ -105,6 +107,7 @@ const struct io_op_def io_op_defs[] = { .name = "READ_FIXED", .prep = io_prep_rw, .issue = io_read, + .fail = io_rw_fail, }, [IORING_OP_WRITE_FIXED] = { .needs_file = 1, @@ -119,6 +122,7 @@ const struct io_op_def io_op_defs[] = { .name = "WRITE_FIXED", .prep = io_prep_rw, .issue = io_write, + .fail = io_rw_fail, }, [IORING_OP_POLL_ADD] = { .needs_file = 1, @@ -275,6 +279,7 @@ const struct io_op_def io_op_defs[] = { .name = "READ", .prep = io_prep_rw, .issue = io_read, + .fail = io_rw_fail, }, [IORING_OP_WRITE] = { .needs_file = 1, @@ -289,6 +294,7 @@ const struct io_op_def io_op_defs[] = { .name = "WRITE", .prep = io_prep_rw, .issue = io_write, + .fail = io_rw_fail, }, [IORING_OP_FADVISE] = { .needs_file = 1, diff --git a/io_uring/rw.c b/io_uring/rw.c index b777c35378b9..8f4e6b3f2b3f 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -954,6 +954,14 @@ static void io_cqring_ev_posted_iopoll(struct io_ring_ctx *ctx) io_cqring_wake(ctx); } +void io_rw_fail(struct io_kiocb *req) +{ + int res; + + res = io_fixup_rw_res(req, req->cqe.res); + io_req_set_res(req, res, req->cqe.flags); +} + int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) { struct io_wq_work_node *pos, *start, *prev; diff --git a/io_uring/rw.h b/io_uring/rw.h index 0204c3fcafa5..3b733f4b610a 100644 --- a/io_uring/rw.h +++ b/io_uring/rw.h @@ -21,3 +21,4 @@ int io_readv_prep_async(struct io_kiocb *req); int io_write(struct io_kiocb *req, unsigned int issue_flags); int io_writev_prep_async(struct io_kiocb *req); void io_readv_writev_cleanup(struct io_kiocb *req); +void io_rw_fail(struct io_kiocb *req); From patchwork Wed Sep 21 11:17:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66249C6FA8E for ; Wed, 21 Sep 2022 11:20:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229498AbiIULUt (ORCPT ); Wed, 21 Sep 2022 07:20:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229673AbiIULUs (ORCPT ); Wed, 21 Sep 2022 07:20:48 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B382A6FA3F for ; Wed, 21 Sep 2022 04:20:45 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id bq9so9348408wrb.4 for ; Wed, 21 Sep 2022 04:20:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=ap2H0ruFKwI5eh8H8ZEfrhtrnIj5BqPVQCmUv7B2rCA=; b=g7W5eSDwC1uF+d0O9+bmHMKQhjzy/PA5bLVlTYhZpgAZsaCgfwdxmQ4oCQwpgDLfZu GxRURsnm6mjdbf1bg7nWk2v36VqX6RBCqPMfJYzZ1XO1ew4xvVAs00m4ZRVqibhhK2i7 1vVD9stKNlPXqkI/AtLcCCKJo435Vj80m5rHmzqB/cDnhtBoFhtZppHzXI5Wap/cZa/U dMxmalDOuYvW8a2NgfJLzbfe3PDbKkMTJpTZ1nyxKnuSYjGkGoFeLwhFhYsJ1XyApHHS 8V1JMX7sUHkYssjnj4SyycRcs6/RM5Pov1TOJ51Jvu3EM7LgA8BtMcQt9TBgJ5KKdEpo 7gqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=ap2H0ruFKwI5eh8H8ZEfrhtrnIj5BqPVQCmUv7B2rCA=; b=gXjacFdd3CZ+i+WYfNjtKMYj0pSJqIxfl9nSRmCVOO0hbMYFG9apMcmUfv/Zn7GyNN ydMnc359dyf7QUlJ25khb3N3gmw2VAzZJrMZQwON5ERZ4uqm3Pm1J2Za/EHOpSx85rFR 0vQmYvnzbMLrl5DHyQR3N6OWnyBAFnn5gu4zijhu3RCxGmZ76nPrBAQU3OvUfIXtvJTG U4qq5D4jOqxJYi3MlC/DX4vjMjBjfaTusrtj6NdywwCC1oeDvER708gWt4oYSwKo8cae laOrmo1L9Bib8TKZOjzVkPo+z7i+0L7ZC3c6NcF6T0iw71C8L/jE45fQzvIk2rHX0u6W Yh5Q== X-Gm-Message-State: ACrzQf3EepNvFSyynNXLZYUIzGEN6e2gCn6qAAK6RTVH1AbMzziDUUI3 P6vG3y9BelJ9+B321sCWnUpZBeZ7rss= X-Google-Smtp-Source: AMsMyM6zYF7LZD0WXGbAE9XZvzFseVXrgpBy4NQiE8iQySaWCwIcoICT1XzjYALdFLGkGw4AjjT4dQ== X-Received: by 2002:a5d:457b:0:b0:22b:24d6:1a9f with SMTP id a27-20020a5d457b000000b0022b24d61a9fmr1423887wrc.201.1663759243844; Wed, 21 Sep 2022 04:20:43 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/9] io_uring/net: don't lose partial send/recv on fail Date: Wed, 21 Sep 2022 12:17:48 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Just as with rw, partial send/recv may end up in io_req_complete_failed() and loose the result, make sure we return the number of bytes processed. Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov --- io_uring/net.c | 10 ++++++++++ io_uring/net.h | 2 ++ io_uring/opdef.c | 4 ++++ 3 files changed, 16 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index 84df6d4253b7..e86a82ef4fbf 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1093,6 +1093,16 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } +void io_sendrecv_fail(struct io_kiocb *req) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + int res = req->cqe.res; + + if (req->flags & REQ_F_PARTIAL_IO) + res = sr->done_io; + io_req_set_res(req, res, req->cqe.flags); +} + int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_accept *accept = io_kiocb_to_cmd(req, struct io_accept); diff --git a/io_uring/net.h b/io_uring/net.h index d744a0a874e7..109ffb3a1a3f 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -43,6 +43,8 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags); int io_recv(struct io_kiocb *req, unsigned int issue_flags); +void io_sendrecv_fail(struct io_kiocb *req); + int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_accept(struct io_kiocb *req, unsigned int issue_flags); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 788393ec3ff4..8d8a0f9bb5b6 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -158,6 +158,7 @@ const struct io_op_def io_op_defs[] = { .issue = io_sendmsg, .prep_async = io_sendmsg_prep_async, .cleanup = io_sendmsg_recvmsg_cleanup, + .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, #endif @@ -176,6 +177,7 @@ const struct io_op_def io_op_defs[] = { .issue = io_recvmsg, .prep_async = io_recvmsg_prep_async, .cleanup = io_sendmsg_recvmsg_cleanup, + .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, #endif @@ -318,6 +320,7 @@ const struct io_op_def io_op_defs[] = { #if defined(CONFIG_NET) .prep = io_sendmsg_prep, .issue = io_send, + .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, #endif @@ -333,6 +336,7 @@ const struct io_op_def io_op_defs[] = { #if defined(CONFIG_NET) .prep = io_recvmsg_prep, .issue = io_recv, + .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, #endif From patchwork Wed Sep 21 11:17:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21451ECAAD8 for ; Wed, 21 Sep 2022 11:20:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229886AbiIULUu (ORCPT ); Wed, 21 Sep 2022 07:20:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59850 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229729AbiIULUs (ORCPT ); Wed, 21 Sep 2022 07:20:48 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4E5771719 for ; Wed, 21 Sep 2022 04:20:46 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id n10so9263974wrw.12 for ; Wed, 21 Sep 2022 04:20:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=vhu/Sfk+5d98pOfwD0Ug0t7w0suS8hFfAmIKFJaf/BI=; b=XlfGp8SXiCwo67QauJt2vOcNWwdGN+GcXWBxfk6i/q+Im/Hff0gHRR357xlxFbExHC sK4Jxak2dKkGPjBMeAfKCZ9BAc2nzDbbn3ttfC6cdUuTq+b889OKOfR4NBPry8Vb/C4e s+YLY1TT4el+XebZwpCSmh/oeS8LYhPBp+6LQS3wpyyOtMXtTzBqzulHKsA5O9YkGXPN h7wB0umoK0VRETIFHV2qB/oNa72LO270iove7PQuAhx7sqstPycH/F1hEw9DSi7DjA0d d9/Xem7coh7O8XdoLmHk+F5vnmvCmjR7yv9uM3hyhKzpokiKQtvsLKEIC5ykljnvSoVU CAhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=vhu/Sfk+5d98pOfwD0Ug0t7w0suS8hFfAmIKFJaf/BI=; b=NRUV68bUAb3KLMzlwXU+EEo/N6mYcdhHnfJrWFhu0HUaeMwdJBSLJEWpeUA4W/VIm6 1Fpw3WnHP/KzWcIjGr+h6WIvdGt2BKOMp8z1qnNQIxLxfAYIChcIbj1z7PyK7M4Juqbl bdICivtNFqL/3yrz8hubsorilHffcAN3TtENmL8qk5L/9B9a3S4fLT+NZE5ZANFd2aci B9x0XiP9WNYPqTA2pHh3rlYx48Iyw6o2C2IVBIR64fQakim0Ffdvdb0IjhryRrFC0LJI jnL/Xx9H/UQYSyBWB6bzbBnZ0AGSizjRQYTwFTuTnCElm9bDKivdXdsjpZ589B/nI2uA ic0g== X-Gm-Message-State: ACrzQf1p/mSx0nERuiXIrSl9qLq+y2mIS3bxzw0JkSQxkJxF2IKlIccK O/Jc+i+mIIG7rOs6lqwiELuNUAhCmS0= X-Google-Smtp-Source: AMsMyM4xhgKcZ/PhPASSsZiRpf34yg/nun83ej+bTW0uD19wayqjfZLGnYV8O8Kj5F762lzC46frjQ== X-Received: by 2002:a05:6000:1cc:b0:22a:e8c6:9f57 with SMTP id t12-20020a05600001cc00b0022ae8c69f57mr12576081wrx.527.1663759244869; Wed, 21 Sep 2022 04:20:44 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 4/9] io_uring/net: don't lose partial send_zc on fail Date: Wed, 21 Sep 2022 12:17:49 +0100 Message-Id: <5673285b5e83e6ceca323727b4ddaa584b5cc91e.1663668091.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Partial zc send may end up in io_req_complete_failed(), which not only would return invalid result but also mask out the notification leading to lifetime issues. Cc: stable@vger.kernel.org Signed-off-by: Pavel Begunkov --- io_uring/net.c | 16 ++++++++++++++++ io_uring/net.h | 1 + io_uring/opdef.c | 1 + 3 files changed, 18 insertions(+) diff --git a/io_uring/net.c b/io_uring/net.c index e86a82ef4fbf..5e7fadefe2d5 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1103,6 +1103,22 @@ void io_sendrecv_fail(struct io_kiocb *req) io_req_set_res(req, res, req->cqe.flags); } +void io_send_zc_fail(struct io_kiocb *req) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + int res = req->cqe.res; + + if (req->flags & REQ_F_PARTIAL_IO) { + if (req->flags & REQ_F_NEED_CLEANUP) { + io_notif_flush(sr->notif); + sr->notif = NULL; + req->flags &= ~REQ_F_NEED_CLEANUP; + } + res = sr->done_io; + } + io_req_set_res(req, res, req->cqe.flags); +} + int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_accept *accept = io_kiocb_to_cmd(req, struct io_accept); diff --git a/io_uring/net.h b/io_uring/net.h index 109ffb3a1a3f..e7366aac335c 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -58,6 +58,7 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags); int io_sendzc(struct io_kiocb *req, unsigned int issue_flags); int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); void io_sendzc_cleanup(struct io_kiocb *req); +void io_send_zc_fail(struct io_kiocb *req); void io_netmsg_cache_free(struct io_cache_entry *entry); #else diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 8d8a0f9bb5b6..f5e7a0e01729 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -497,6 +497,7 @@ const struct io_op_def io_op_defs[] = { .issue = io_sendzc, .prep_async = io_sendzc_prep_async, .cleanup = io_sendzc_cleanup, + .fail = io_send_zc_fail, #else .prep = io_eopnotsupp_prep, #endif From patchwork Wed Sep 21 11:17:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6CADC6FA8E for ; Wed, 21 Sep 2022 11:20:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229729AbiIULUw (ORCPT ); Wed, 21 Sep 2022 07:20:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229805AbiIULUt (ORCPT ); Wed, 21 Sep 2022 07:20:49 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13D3B71982 for ; Wed, 21 Sep 2022 04:20:48 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id t4so273409wmj.5 for ; Wed, 21 Sep 2022 04:20:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=WcE3jJFUvPSbz2D9hJex+1tGIl6RRjHpabbeEgpovDQ=; b=BpWAQldyn2iBOZOrZMCe5W4G9xcGvnx9O5I70h7aYtVZwD3GoJuFiof+xuIEe7KIhO WQDgE9lZrDWNVgxoC7OH/MjXfWcqyq/q3MnX6hcFWsPjVYOjZDG9lMrruLuQ7AR0L5up 0p+ph4IyAtTOlFKV+NiKDHE1vqxffKz1Grf+qjnUaxlQwd3xSI0zlqv7otqen0h4gvHK lNIe/4kV26zwGOsEBDMTZzV6kiC+EgY/3NXYqrMkwVZZAehfObmRha73CRqKICpCOY6M fbtJFmViA0LUws9xIZ7PDV+zb5qtIt1/B1JshP/mTnfydJndfbqhPntBf6nB7yXEP1gT MbTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=WcE3jJFUvPSbz2D9hJex+1tGIl6RRjHpabbeEgpovDQ=; b=iGvZLNf5Oiu9Mlom5Y2PzgpphCs4Mbv13kvlo16mjeWUBg3g2P0bJqBF3Bit4MgtIj yZVXjcdM/yY+XWnnRXby08y9zaLdRq50tN+EswSGmgTLuCBHzQgiEoVziuZUEHVfC3gQ zkCup517HBU6znqhtQZCFs//PspN4Q4TDZocxm8h80hraZJ4NyopJR/FP3rKl1kAP7RI x4kkmacm51FgHDLwJj4WnsRzF73W24FN3E7W8UnV9nkDJu8l5mCe5cpo9C8Xi59D9LdK R5vst5lvSBjXub5GqHb0E2SIYYf8K24EmlFJRtwGHscRNnaiRvo7A+yZ4Atccz2dhWxj oByQ== X-Gm-Message-State: ACrzQf1t2JlW8CswjiaHkJv03TNX7RcD2C1srz+j/wFkuGckT3vG3NY9 lh19NQmfwqVXl3o0O+bZ0uZEQFN/snI= X-Google-Smtp-Source: AMsMyM6lMSLy8cClEsrcrpoTSqew8AFgJ9L0M7ATVgl/FN1n68OFJ7R/S9TPMwyXYWCDiADvM6j6Dg== X-Received: by 2002:a05:600c:3b94:b0:3b4:9cdc:dbd4 with SMTP id n20-20020a05600c3b9400b003b49cdcdbd4mr5603781wms.159.1663759245966; Wed, 21 Sep 2022 04:20:45 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 5/9] io_uring/net: refactor io_setup_async_addr Date: Wed, 21 Sep 2022 12:17:50 +0100 Message-Id: <6bfa9ab810d776853eb26ed59301e2536c3a5471.1663668091.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of passing the right address into io_setup_async_addr() only specify local on-stack storage and let the function infer where to grab it from. It optimises out one local variable we have to deal with. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 5e7fadefe2d5..a190e022a9de 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -196,17 +196,18 @@ int io_sendzc_prep_async(struct io_kiocb *req) } static int io_setup_async_addr(struct io_kiocb *req, - struct sockaddr_storage *addr, + struct sockaddr_storage *addr_storage, unsigned int issue_flags) { + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *io; - if (!addr || req_has_async_data(req)) + if (!sr->addr || req_has_async_data(req)) return -EAGAIN; io = io_msg_alloc_async(req, issue_flags); if (!io) return -ENOMEM; - memcpy(&io->addr, addr, sizeof(io->addr)); + memcpy(&io->addr, addr_storage, sizeof(io->addr)); return -EAGAIN; } @@ -1000,7 +1001,7 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) { - struct sockaddr_storage __address, *addr = NULL; + struct sockaddr_storage __address; struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct msghdr msg; struct iovec iov; @@ -1021,20 +1022,19 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) if (req_has_async_data(req)) { struct io_async_msghdr *io = req->async_data; - msg.msg_name = addr = &io->addr; + msg.msg_name = &io->addr; } else { ret = move_addr_to_kernel(zc->addr, zc->addr_len, &__address); if (unlikely(ret < 0)) return ret; msg.msg_name = (struct sockaddr *)&__address; - addr = &__address; } msg.msg_namelen = zc->addr_len; } if (!(req->flags & REQ_F_POLLED) && (zc->flags & IORING_RECVSEND_POLL_FIRST)) - return io_setup_async_addr(req, addr, issue_flags); + return io_setup_async_addr(req, &__address, issue_flags); if (zc->flags & IORING_RECVSEND_FIXED_BUF) { ret = io_import_fixed(WRITE, &msg.msg_iter, req->imu, @@ -1065,14 +1065,14 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return io_setup_async_addr(req, addr, issue_flags); + return io_setup_async_addr(req, &__address, issue_flags); if (ret > 0 && io_net_retry(sock, msg.msg_flags)) { zc->len -= ret; zc->buf += ret; zc->done_io += ret; req->flags |= REQ_F_PARTIAL_IO; - return io_setup_async_addr(req, addr, issue_flags); + return io_setup_async_addr(req, &__address, issue_flags); } if (ret < 0 && !zc->done_io) zc->notif->flags |= REQ_F_CQE_SKIP; From patchwork Wed Sep 21 11:17:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983606 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAE68C32771 for ; Wed, 21 Sep 2022 11:20:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229913AbiIULUy (ORCPT ); Wed, 21 Sep 2022 07:20:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229891AbiIULUu (ORCPT ); Wed, 21 Sep 2022 07:20:50 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7959070E44 for ; Wed, 21 Sep 2022 04:20:49 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id o5so4218288wms.1 for ; Wed, 21 Sep 2022 04:20:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=jh9KBFYcogDbm7aInFCEJSuzB6ztb3nRXiIBKpYh+zQ=; b=Bg3yQQuBc3eyPYB6EfY0oo91Nvg4KIA94XxCqnoKiqqZC7hZxcHvSlstcl2TycduC1 avoCwczel4VkX94QiwR82VXAovj6vwwOX6Vs6qqKpue4uucEO5M3UEcZuxk5vF6RdU6p 2m2/tbObmcHvfBVhNvMafumC+QIRRW6oLHXuMynBJjnBKCDMa0DJ2vh+IZLDs33w/twY mPyElDF5Y0kmAK4O1FFuyVo8M8kI/XKQFbBaIvCis96Fz+2tyiMatcpurrinlvr+Ih5S UJciZqvxRa7WSnndSaTXapgzpJyJ2KsU97AECGMeFPO2Ehgh+Kp3XNskt0ON6lWszMXD emQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=jh9KBFYcogDbm7aInFCEJSuzB6ztb3nRXiIBKpYh+zQ=; b=SjGoXkn97CzCIuuoxmwU6CEX1ZWBciyGDlPX+1PgFYxNa4uPELVRHoDG3hXeRhSCQ2 7sx/EqkC3wwoGOK00pi9V05yvNgs0WGSi7qIolShLzYDfaaVj5OKaGZbKUUhJxWDd0h+ lrfjpp+EcNJLboqWGD+ZZXpvbzyQVy/bJAWbfAYlqpcW/HtAl/iQb1mJBldgEoyKcSeF J8jNFv8AGqahIBK3VrJWYGnlTVi4juTAcHqx27NH6wYSyPG40Rksc8AWhx7+cEidmE37 knt5A8DDYd/3lNJ1iJsXi62j8Dq2GuzS7OW492eRttxAKsET2FLijQqmuV+PC9hDcsfw M+8g== X-Gm-Message-State: ACrzQf2SNOQvJVoQKj0EvX2XnYG5uuWafV/9jd44c+b1dB1og89FJNfe d1t+F4a+TCsi+VmhZsavuTyAkaTUKHg= X-Google-Smtp-Source: AMsMyM72wskPqcwcQj3eAwJdVxzaZwuhDefVTq+7O3X3LzbBSa4z2L6wsJszP+plVQyX55HKtgWWjA== X-Received: by 2002:a05:600c:4f07:b0:3b4:a5d1:1fea with SMTP id l7-20020a05600c4f0700b003b4a5d11feamr5637830wmq.103.1663759247633; Wed, 21 Sep 2022 04:20:47 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 6/9] io_uring/net: support non-zerocopy sendto Date: Wed, 21 Sep 2022 12:17:51 +0100 Message-Id: <69fbd8b2cb830e57d1bf9ec351e9bf95c5b77e3f.1663668091.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We have normal sends, but what is missing is sendto-like requests. Add sendto() capabilities to IORING_OP_SEND by passing in addr just as we do for IORING_OP_SEND_ZC. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 35 +++++++++++++++++++++++++++++------ io_uring/net.h | 3 ++- io_uring/opdef.c | 5 ++++- 3 files changed, 35 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index a190e022a9de..aa2c819cd85d 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -59,9 +59,10 @@ struct io_sr_msg { unsigned done_io; unsigned msg_flags; u16 flags; - /* used only for sendzc */ + /* initialised and used only by !msg send variants */ u16 addr_len; void __user *addr; + /* used only for send zerocopy */ struct io_kiocb *notif; }; @@ -180,7 +181,7 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, &iomsg->free_iov); } -int io_sendzc_prep_async(struct io_kiocb *req) +int io_send_prep_async(struct io_kiocb *req) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_async_msghdr *io; @@ -234,8 +235,14 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - if (unlikely(sqe->file_index || sqe->addr2)) + if (req->opcode == IORING_OP_SEND) { + if (READ_ONCE(sqe->__pad3[0])) + return -EINVAL; + sr->addr = u64_to_user_ptr(READ_ONCE(sqe->addr2)); + sr->addr_len = READ_ONCE(sqe->addr_len); + } else if (sqe->addr2 || sqe->file_index) { return -EINVAL; + } sr->umsg = u64_to_user_ptr(READ_ONCE(sqe->addr)); sr->len = READ_ONCE(sqe->len); @@ -315,6 +322,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) int io_send(struct io_kiocb *req, unsigned int issue_flags) { + struct sockaddr_storage __address; struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); struct msghdr msg; struct iovec iov; @@ -323,9 +331,23 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) int min_ret = 0; int ret; + if (sr->addr) { + if (req_has_async_data(req)) { + struct io_async_msghdr *io = req->async_data; + + msg.msg_name = &io->addr; + } else { + ret = move_addr_to_kernel(sr->addr, sr->addr_len, &__address); + if (unlikely(ret < 0)) + return ret; + msg.msg_name = (struct sockaddr *)&__address; + } + msg.msg_namelen = sr->addr_len; + } + if (!(req->flags & REQ_F_POLLED) && (sr->flags & IORING_RECVSEND_POLL_FIRST)) - return -EAGAIN; + return io_setup_async_addr(req, &__address, issue_flags); sock = sock_from_file(req->file); if (unlikely(!sock)) @@ -351,13 +373,14 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) ret = sock_sendmsg(sock, &msg); if (ret < min_ret) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return -EAGAIN; + return io_setup_async_addr(req, &__address, issue_flags); + if (ret > 0 && io_net_retry(sock, flags)) { sr->len -= ret; sr->buf += ret; sr->done_io += ret; req->flags |= REQ_F_PARTIAL_IO; - return -EAGAIN; + return io_setup_async_addr(req, &__address, issue_flags); } if (ret == -ERESTARTSYS) ret = -EINTR; diff --git a/io_uring/net.h b/io_uring/net.h index e7366aac335c..488d4dc7eee2 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -31,12 +31,13 @@ struct io_async_connect { int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_shutdown(struct io_kiocb *req, unsigned int issue_flags); -int io_sendzc_prep_async(struct io_kiocb *req); int io_sendmsg_prep_async(struct io_kiocb *req); void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req); int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags); + int io_send(struct io_kiocb *req, unsigned int issue_flags); +int io_send_prep_async(struct io_kiocb *req); int io_recvmsg_prep_async(struct io_kiocb *req); int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index f5e7a0e01729..8fb4d98c9f36 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -316,11 +316,14 @@ const struct io_op_def io_op_defs[] = { .pollout = 1, .audit_skip = 1, .ioprio = 1, + .manual_alloc = 1, .name = "SEND", #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_sendmsg_prep, .issue = io_send, .fail = io_sendrecv_fail, + .prep_async = io_send_prep_async, #else .prep = io_eopnotsupp_prep, #endif @@ -495,7 +498,7 @@ const struct io_op_def io_op_defs[] = { .async_size = sizeof(struct io_async_msghdr), .prep = io_sendzc_prep, .issue = io_sendzc, - .prep_async = io_sendzc_prep_async, + .prep_async = io_send_prep_async, .cleanup = io_sendzc_cleanup, .fail = io_send_zc_fail, #else From patchwork Wed Sep 21 11:17:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32799ECAAD8 for ; Wed, 21 Sep 2022 11:21:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230024AbiIULVE (ORCPT ); Wed, 21 Sep 2022 07:21:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229938AbiIULUx (ORCPT ); Wed, 21 Sep 2022 07:20:53 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B4DC77569 for ; Wed, 21 Sep 2022 04:20:50 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id g3so9261195wrq.13 for ; Wed, 21 Sep 2022 04:20:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=YMalNKBBvx+Fi+YBMCJ2+mo3jcLmAL8upLLfTrb8yzY=; b=EHTadEuz6t74qCV9AIVIeWtyhUrkAp4mNPNuUOHEIoOqsdTBb6Ge78WnGMLcEYAhff jcmKyrWZx78JighWtWlbUIxa+J5WzP9BPvo40vtuAuuFSRXHYByTOIFDqmib254BdCiJ VeOTponjASU4zme9t5rDh8LjdMNTPnqUeLiOfFTLda5T3pfxTR5sUID/q8kfDgZn5BYf xyC5wD8NLh4yUdJglY26Vq8GNUyKnnUHGKU/Qb/hfU+STaEj/PufkDGpntZyGbc+RirP 3Oq/on7N8vWGe8WO89ebSPQKNIa6CJYarDgX6GY2fUhqod0cjMJUt5KvX/aNu4tGA1aD ZEOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=YMalNKBBvx+Fi+YBMCJ2+mo3jcLmAL8upLLfTrb8yzY=; b=lzehhwIN9gbAUkDgUou36yEdtJ/d2F3C2YfxUuCzRHexR5MUJuEm63TXa9rzJ1Hpmj NfJRpA8TF34zOA5Es5J2YICrEKEr88Zi+FE+oRx2kx4k9Gd+2yQA5BSH5/udoY2k0GhG CQMb99cd010CL8xyowFZM8wvRlLty2amHNHdFeWuoxEcckCpC7cDhLvI5yWIIlsbrq7b vyjuv3GcA0bY5StQoLBBYBssp7cVQdcAsQjRlviAziSVH3wJHCdUARMemUIs9ydkNDRb QZMtwHkc++iPQKxg2RNe4QFvhj4OWr5RzMon0uFDaiGYxLkqsQHE0jDRqNo8Rzchdv9g Teag== X-Gm-Message-State: ACrzQf3weWvWfT7CJTkbfW51ODYPj1vJW8dIUUjY519QdaHzZePP4FGy bPev7+sMU2CcsiAYoaWTOOEr4os+qPM= X-Google-Smtp-Source: AMsMyM4mBkmr1H4dHxVA+mjDWtAtxniLgSsAuRr0HNgYg8yHuiV5LObMbM8FRbXcC2iscV6YqEEhbQ== X-Received: by 2002:a05:6000:1090:b0:228:a963:3641 with SMTP id y16-20020a056000109000b00228a9633641mr17162201wrw.289.1663759248793; Wed, 21 Sep 2022 04:20:48 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 7/9] io_uring/net: rename io_sendzc() Date: Wed, 21 Sep 2022 12:17:52 +0100 Message-Id: <265af46829e6076dd220011b1858dc3151969226.1663668091.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Simple renaming of io_sendzc*() functions in preparatio to adding a zerocopy sendmsg variant. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 6 +++--- io_uring/net.h | 6 +++--- io_uring/opdef.c | 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index aa2c819cd85d..06c132edfd91 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -904,7 +904,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) return ret; } -void io_sendzc_cleanup(struct io_kiocb *req) +void io_send_zc_cleanup(struct io_kiocb *req) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -913,7 +913,7 @@ void io_sendzc_cleanup(struct io_kiocb *req) zc->notif = NULL; } -int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) +int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); struct io_ring_ctx *ctx = req->ctx; @@ -1022,7 +1022,7 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, return ret; } -int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) +int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) { struct sockaddr_storage __address; struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); diff --git a/io_uring/net.h b/io_uring/net.h index 488d4dc7eee2..337541f25b79 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -56,9 +56,9 @@ int io_connect_prep_async(struct io_kiocb *req); int io_connect_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_connect(struct io_kiocb *req, unsigned int issue_flags); -int io_sendzc(struct io_kiocb *req, unsigned int issue_flags); -int io_sendzc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); -void io_sendzc_cleanup(struct io_kiocb *req); +int io_send_zc(struct io_kiocb *req, unsigned int issue_flags); +int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); +void io_send_zc_cleanup(struct io_kiocb *req); void io_send_zc_fail(struct io_kiocb *req); void io_netmsg_cache_free(struct io_cache_entry *entry); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 8fb4d98c9f36..36590f9ab37b 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -496,10 +496,10 @@ const struct io_op_def io_op_defs[] = { .manual_alloc = 1, #if defined(CONFIG_NET) .async_size = sizeof(struct io_async_msghdr), - .prep = io_sendzc_prep, - .issue = io_sendzc, + .prep = io_send_zc_prep, + .issue = io_send_zc, .prep_async = io_send_prep_async, - .cleanup = io_sendzc_cleanup, + .cleanup = io_send_zc_cleanup, .fail = io_send_zc_fail, #else .prep = io_eopnotsupp_prep, From patchwork Wed Sep 21 11:17:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E99A3C32771 for ; Wed, 21 Sep 2022 11:21:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230002AbiIULVB (ORCPT ); Wed, 21 Sep 2022 07:21:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229926AbiIULUw (ORCPT ); Wed, 21 Sep 2022 07:20:52 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9286676448 for ; Wed, 21 Sep 2022 04:20:50 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id n10so9264290wrw.12 for ; Wed, 21 Sep 2022 04:20:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=NdzOesns5PuSMPMWnlJY/dJaslxGRHBwKePVvzuqK0k=; b=C2C1+ZWel8AHGalCk5QzGBmlxhg1P0wqu94SsjeCKHUqdGhKfwcUxOEdCpGn4TDCn4 oNGSXRSOkEzgIZ/R4WPQvyxMHrXLAl6vmZB2E/j+i9fVudS0UZVNG1iRiIEeNgLoLpfP PIhIHujt2QFXEXJdHAyOM7jLoZB8wYMqy3mujeotSnI3NZPjwsyGrso+KwdsuVxPnVwk Pl+lcSNM2xG4JXx9jnJhTTThnSvK5OhKMAJCtAbYnel70MWP10RvOnMXoKe1jddAV+/h m51w8IknEYihZkO2D6SWHpLAt6ciC5KTvha3IIqqaeKswlxiYA1xrmpvt+t+Q6X2I+eS cXBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=NdzOesns5PuSMPMWnlJY/dJaslxGRHBwKePVvzuqK0k=; b=AQUzZExzt133LXJ8dd0klAwKtRZ16yciE5rkBE4tcclIfSH8/Lb645DxH2Krbw/NX1 FTHJgzUPe7+U8Yyh3yrDnEFyNI9Q1dAqZ1otcF14ZzPc2f06vQQs3av25DHenERJMurw ItQF2jBaaCZ0FOHdJMUsfHuyciTMM2/BS0QDA8Amn0PTRKvlF2RI4WACwdAFHQZ6mR8e JMVVb32Sfsq8evKAThbfvcQwvhaWv3nEP6YNCyWJmCGvmytwWFbp+tu2ne4lIzWNEJig 6cqGMmUqzCdlu3nUQbMSkC2GJx8FuA1sQahh8mhyTla26KFmiTK3nVYksU5JuysdEKlK t9iw== X-Gm-Message-State: ACrzQf1prrefdiuugO/f0gmxWAQpkqgYwGdQBvwbqLMXPpvtA18lFmtV 8UWsKhx1hBjasIQEykVuZ2Md8LGXHTY= X-Google-Smtp-Source: AMsMyM4gAj3Ctg//m9XNfCS02xs8tA/05xq9rkYTmddIiiyD4pibe4sixBbDpZBX3IMQqua7F7dyig== X-Received: by 2002:a05:6000:1866:b0:228:e373:ad68 with SMTP id d6-20020a056000186600b00228e373ad68mr16414791wri.605.1663759249895; Wed, 21 Sep 2022 04:20:49 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 8/9] io_uring/net: combine fail handlers Date: Wed, 21 Sep 2022 12:17:53 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Merge io_send_zc_fail() into io_sendrecv_fail(), saves a few lines of code and some headache for following patch. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 31 ++++++++++++++++--------------- io_uring/net.h | 1 - io_uring/opdef.c | 2 +- 3 files changed, 17 insertions(+), 17 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 06c132edfd91..4bcc2675001f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -192,6 +192,7 @@ int io_send_prep_async(struct io_kiocb *req) io = io_msg_alloc_async_prep(req); if (!io) return -ENOMEM; + io->free_iov = NULL; ret = move_addr_to_kernel(zc->addr, zc->addr_len, &io->addr); return ret; } @@ -208,6 +209,7 @@ static int io_setup_async_addr(struct io_kiocb *req, io = io_msg_alloc_async(req, issue_flags); if (!io) return -ENOMEM; + io->free_iov = NULL; memcpy(&io->addr, addr_storage, sizeof(io->addr)); return -EAGAIN; } @@ -1119,26 +1121,25 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) void io_sendrecv_fail(struct io_kiocb *req) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *io; int res = req->cqe.res; if (req->flags & REQ_F_PARTIAL_IO) res = sr->done_io; - io_req_set_res(req, res, req->cqe.flags); -} - -void io_send_zc_fail(struct io_kiocb *req) -{ - struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - int res = req->cqe.res; - - if (req->flags & REQ_F_PARTIAL_IO) { - if (req->flags & REQ_F_NEED_CLEANUP) { - io_notif_flush(sr->notif); - sr->notif = NULL; - req->flags &= ~REQ_F_NEED_CLEANUP; - } - res = sr->done_io; + if ((req->flags & REQ_F_NEED_CLEANUP) && + req->opcode == IORING_OP_SEND_ZC) { + /* preserve notification for partial I/O */ + if (res < 0) + sr->notif->flags |= REQ_F_CQE_SKIP; + io_notif_flush(sr->notif); + sr->notif = NULL; } + if (req_has_async_data(req)) { + io = req->async_data; + kfree(io->free_iov); + io->free_iov = NULL; + } + req->flags &= ~REQ_F_NEED_CLEANUP; io_req_set_res(req, res, req->cqe.flags); } diff --git a/io_uring/net.h b/io_uring/net.h index 337541f25b79..45558e2b0a83 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -59,7 +59,6 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags); int io_send_zc(struct io_kiocb *req, unsigned int issue_flags); int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); void io_send_zc_cleanup(struct io_kiocb *req); -void io_send_zc_fail(struct io_kiocb *req); void io_netmsg_cache_free(struct io_cache_entry *entry); #else diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 36590f9ab37b..e9fd940c2ee1 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -500,7 +500,7 @@ const struct io_op_def io_op_defs[] = { .issue = io_send_zc, .prep_async = io_send_prep_async, .cleanup = io_send_zc_cleanup, - .fail = io_send_zc_fail, + .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, #endif From patchwork Wed Sep 21 11:17:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12983609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA103C6FA8E for ; Wed, 21 Sep 2022 11:21:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229892AbiIULVG (ORCPT ); Wed, 21 Sep 2022 07:21:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229959AbiIULU5 (ORCPT ); Wed, 21 Sep 2022 07:20:57 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E135677E80 for ; Wed, 21 Sep 2022 04:20:52 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id t14so9308300wrx.8 for ; Wed, 21 Sep 2022 04:20:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=AwX8MFPz2bqCMB9epVkoln+Im4EAODM6Z9xCifrQzUQ=; b=QgNJieqeMSf3rqhO5yXv5CBQNYFDKVh8mak1bUPEeAgZsCQlHnWVKYbgEIn7h/In4Z 4qB/J7uw7rjnfRpMvyqulY99iaFJ6JvbSixhoJDWjL0xCli+Ff0t+FeMw7GGiSPCjzkN T0IxwYnUIFjypXUKZt39sVdfv88QW0h1brVOHb6AVFUC9m2e4+RUd32fLDwroHlGVdaz TP84cCZEoSxj4pY+M3GHsApjZm4iAQygW4AQZ3qKModpnWUy/Sgn5f/0ZyWYw71mZG4w TMlgSkt6qAd37bNE/mo+J9byqlnweT8GWNk+sdf9CPJRa37q/OZOU7X+sYhTJ9NrbrJS VdSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=AwX8MFPz2bqCMB9epVkoln+Im4EAODM6Z9xCifrQzUQ=; b=LT97F+Jd6NxxSnJvBfPvFLpVe5ucvotVnbllsjS8hB6IvanI4og16gO+8DryLtVSe6 4/Swwltsu/vyMzViXrdvlE+4Zic2KA3c0PVKIpSRQ+oxzoLKZKthPzNZJKgtwS4ZTLgx 2ov35YxkougJEiuUJWygXD2ubEJUwwdU933VLJtTlDqlo2Bx/cWRvBhyi7j5S9awL11S 6Dyo8AuVyUoLMFSIkivQGb0caAVKf0ZLaBO6E3Z9yy7PcwLuyMQCQ+A3x/k9+++oFUAi 30aCO+BsnHIS+UX+FZVXix9RM1DHmpxF753VnreSY+ZNn4J+VqgGheQFm930+OexUyMK l+Ng== X-Gm-Message-State: ACrzQf1wiciDnTx5mzLHriQM+TxPTDbobllw8UnxRD/bcI3vI3TVwbik +ZFcQ5W/3wutr3S3puMpjsIPWmu+JHo= X-Google-Smtp-Source: AMsMyM4tygO+Lz+NgtmuFZ0RItA3G1u3C1rdzW+jNgzIJhkr0rRPK+W3wuSpst5N0/vd2XVsyiJTyw== X-Received: by 2002:a5d:6245:0:b0:225:41ae:a930 with SMTP id m5-20020a5d6245000000b0022541aea930mr17187819wrv.342.1663759251062; Wed, 21 Sep 2022 04:20:51 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.205.62.threembb.co.uk. [188.28.205.62]) by smtp.gmail.com with ESMTPSA id s17-20020a5d6a91000000b00228da845d4dsm2206732wru.94.2022.09.21.04.20.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Sep 2022 04:20:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 9/9] io_uring/net: zerocopy sendmsg Date: Wed, 21 Sep 2022 12:17:54 +0100 Message-Id: <6aabc4bdfc0ec78df6ec9328137e394af9d4e7ef.1663668091.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a zerocopy version of sendmsg. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 1 + io_uring/net.c | 92 +++++++++++++++++++++++++++++++++-- io_uring/net.h | 1 + io_uring/opdef.c | 19 ++++++++ 4 files changed, 108 insertions(+), 5 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 972b179bc07a..92f29d9505a6 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -213,6 +213,7 @@ enum io_uring_op { IORING_OP_SOCKET, IORING_OP_URING_CMD, IORING_OP_SEND_ZC, + IORING_OP_SENDMSG_ZC, /* this goes last, obviously */ IORING_OP_LAST, diff --git a/io_uring/net.c b/io_uring/net.c index 4bcc2675001f..d086f397d4d0 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -909,7 +909,12 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) void io_send_zc_cleanup(struct io_kiocb *req) { struct io_sr_msg *zc = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr *io; + if (req_has_async_data(req)) { + io = req->async_data; + kfree(io->free_iov); + } zc->notif->flags |= REQ_F_CQE_SKIP; io_notif_flush(zc->notif); zc->notif = NULL; @@ -921,8 +926,7 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) struct io_ring_ctx *ctx = req->ctx; struct io_kiocb *notif; - if (READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3) || - READ_ONCE(sqe->__pad3[0])) + if (unlikely(READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3))) return -EINVAL; /* we don't support IOSQE_CQE_SKIP_SUCCESS just yet */ if (req->flags & REQ_F_CQE_SKIP) @@ -941,6 +945,19 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) req->imu = READ_ONCE(ctx->user_bufs[idx]); io_req_set_rsrc_node(req, ctx, 0); } + + if (req->opcode == IORING_OP_SEND_ZC) { + if (READ_ONCE(sqe->__pad3[0])) + return -EINVAL; + zc->addr = u64_to_user_ptr(READ_ONCE(sqe->addr2)); + zc->addr_len = READ_ONCE(sqe->addr_len); + } else { + if (unlikely(sqe->addr2 || sqe->file_index)) + return -EINVAL; + if (unlikely(zc->flags & IORING_RECVSEND_FIXED_BUF)) + return -EINVAL; + } + notif = zc->notif = io_alloc_notif(ctx); if (!notif) return -ENOMEM; @@ -955,8 +972,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (zc->msg_flags & MSG_DONTWAIT) req->flags |= REQ_F_NOWAIT; - zc->addr = u64_to_user_ptr(READ_ONCE(sqe->addr2)); - zc->addr_len = READ_ONCE(sqe->addr_len); zc->done_io = 0; #ifdef CONFIG_COMPAT @@ -1118,6 +1133,73 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } +int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) +{ + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + struct io_async_msghdr iomsg, *kmsg; + struct socket *sock; + unsigned flags, cflags; + int ret, min_ret = 0; + + sock = sock_from_file(req->file); + if (unlikely(!sock)) + return -ENOTSOCK; + + if (req_has_async_data(req)) { + kmsg = req->async_data; + } else { + ret = io_sendmsg_copy_hdr(req, &iomsg); + if (ret) + return ret; + kmsg = &iomsg; + } + + if (!(req->flags & REQ_F_POLLED) && + (sr->flags & IORING_RECVSEND_POLL_FIRST)) + return io_setup_async_msg(req, kmsg, issue_flags); + + flags = sr->msg_flags | MSG_ZEROCOPY; + if (issue_flags & IO_URING_F_NONBLOCK) + flags |= MSG_DONTWAIT; + if (flags & MSG_WAITALL) + min_ret = iov_iter_count(&kmsg->msg.msg_iter); + + kmsg->msg.msg_ubuf = &io_notif_to_data(sr->notif)->uarg; + kmsg->msg.sg_from_iter = io_sg_from_iter_iovec; + ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags); + + if (unlikely(ret < min_ret)) { + if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) + return io_setup_async_msg(req, kmsg, issue_flags); + + if (ret > 0 && io_net_retry(sock, flags)) { + sr->done_io += ret; + req->flags |= REQ_F_PARTIAL_IO; + return io_setup_async_msg(req, kmsg, issue_flags); + } + if (ret < 0 && !sr->done_io) + sr->notif->flags |= REQ_F_CQE_SKIP; + if (ret == -ERESTARTSYS) + ret = -EINTR; + req_set_fail(req); + } + /* fast path, check for non-NULL to avoid function call */ + if (kmsg->free_iov) + kfree(kmsg->free_iov); + + io_netmsg_recycle(req, issue_flags); + if (ret >= 0) + ret += sr->done_io; + else if (sr->done_io) + ret = sr->done_io; + + io_notif_flush(sr->notif); + req->flags &= ~REQ_F_NEED_CLEANUP; + cflags = ret >= 0 ? IORING_CQE_F_MORE : 0; + io_req_set_res(req, ret, cflags); + return IOU_OK; +} + void io_sendrecv_fail(struct io_kiocb *req) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -1127,7 +1209,7 @@ void io_sendrecv_fail(struct io_kiocb *req) if (req->flags & REQ_F_PARTIAL_IO) res = sr->done_io; if ((req->flags & REQ_F_NEED_CLEANUP) && - req->opcode == IORING_OP_SEND_ZC) { + (req->opcode == IORING_OP_SEND_ZC || req->opcode == IORING_OP_SENDMSG_ZC)) { /* preserve notification for partial I/O */ if (res < 0) sr->notif->flags |= REQ_F_CQE_SKIP; diff --git a/io_uring/net.h b/io_uring/net.h index 45558e2b0a83..5ffa11bf5d2e 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -57,6 +57,7 @@ int io_connect_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_connect(struct io_kiocb *req, unsigned int issue_flags); int io_send_zc(struct io_kiocb *req, unsigned int issue_flags); +int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags); int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); void io_send_zc_cleanup(struct io_kiocb *req); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index e9fd940c2ee1..3c930fef255b 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -503,6 +503,25 @@ const struct io_op_def io_op_defs[] = { .fail = io_sendrecv_fail, #else .prep = io_eopnotsupp_prep, +#endif + }, + [IORING_OP_SENDMSG_ZC] = { + .name = "SENDMSG_ZC", + .needs_file = 1, + .unbound_nonreg_file = 1, + .pollout = 1, + .audit_skip = 1, + .ioprio = 1, + .manual_alloc = 1, +#if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), + .prep = io_send_zc_prep, + .issue = io_sendmsg_zc, + .prep_async = io_sendmsg_prep_async, + .cleanup = io_send_zc_cleanup, + .fail = io_sendrecv_fail, +#else + .prep = io_eopnotsupp_prep, #endif }, };