From patchwork Sat May 14 14:20:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12849815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 165CDC433F5 for ; Sat, 14 May 2022 14:20:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233129AbiENOUf (ORCPT ); Sat, 14 May 2022 10:20:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229828AbiENOUd (ORCPT ); Sat, 14 May 2022 10:20:33 -0400 Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1EC42BCF for ; Sat, 14 May 2022 07:20:32 -0700 (PDT) Received: by mail-pg1-x52f.google.com with SMTP id 137so10047681pgb.5 for ; Sat, 14 May 2022 07:20:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pU7A50hXcHq7ng73ERmKvfAIUben4nCE2ouNyg1ycrA=; b=qVjkqMmROebnDmGUGaYAG6U3S2RGwfPcwKmrcc74YhTlURUH4HYVdJnjfGz9s2BgvX SmOLdQCcqldfnkfMcmMApaMJtPlpB7lE+fyk/tSbcM4ULyygD/GsPsXSaJ+AtmOtOSzx tnT8moN/pnyd7IxqgjqlvJseQwa+o8x89F0jtoMzUIbqHoUIY7dozQToY75BIpNHY67T gJ/ccflado8B+ao/g9i0IUJy0yCm6pKy8YZ0uww49tkfASwBF345iNOvibm91tBUKpgC lP64vcGweDFvpz2Vf7pID49OSjj3xCrveJERBBILLkefyugAwcxFc2SXD8eoUPo8VYhw gkbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pU7A50hXcHq7ng73ERmKvfAIUben4nCE2ouNyg1ycrA=; b=T7LSx5V46Keo8eS/An65Rt6Cony+u02069ebu62cmQrlRYvk657i4hkbG0gAIOK3V1 IRk0lbrIo6vdTULbsqtjpIh9z56pCEnQ7aqISc+o8x70aTSKPkxiiaCn3dIzojC7Pq1T I9VR23fEaOoWhPLF4IpWcS9oXgYO0nvL51I/F4tMbYmMD/Z10jcCJhn5+bKM7HFhkWKb b/DHg+Z2nc8cyc3Io56CKs8o3xUXOkVv3CIzdBYpjfMQKJl3oRweXuIedRrUsvpAf/Zn haVQlk2u5QtxQ97Rw9SilrRS1m7yoCBulNtBO3KC3BVHqPMFrFSHD06DLPVy3AHLR0NX uPvw== X-Gm-Message-State: AOAM531zawOmAQIzEwtqYn/uCvTxTPt47PAAibyITCRQ6QHgtE5XTUhY apFXkrLVZiA/izVMamoDxmIi+U5joPYNt+Si X-Google-Smtp-Source: ABdhPJxRAq6CKGQ9wcG36rdN4gtkEy2XZIFAXHY03/GApy/02ClGHPgoJaB2OCFwKGhVtcuX4sGdDg== X-Received: by 2002:a65:6250:0:b0:3c6:8a09:249 with SMTP id q16-20020a656250000000b003c68a090249mr8154153pgv.389.1652538032198; Sat, 14 May 2022 07:20:32 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([203.205.141.27]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b0015e8d4eb27csm3815968plg.198.2022.05.14.07.20.31 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 May 2022 07:20:31 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 1/4] io_uring: add IORING_ACCEPT_MULTISHOT for accept Date: Sat, 14 May 2022 22:20:43 +0800 Message-Id: <20220514142046.58072-2-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220514142046.58072-1-haoxu.linux@gmail.com> References: <20220514142046.58072-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu add an accept_flag IORING_ACCEPT_MULTISHOT for accept, which is to support multishot. Signed-off-by: Hao Xu --- include/uapi/linux/io_uring.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 36ec43dc7bf9..15f821af9242 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -232,6 +232,11 @@ enum { */ #define IORING_RECVSEND_POLL_FIRST (1U << 0) +/* + * accept flags stored in sqe->ioprio + */ +#define IORING_ACCEPT_MULTISHOT (1U << 0) + /* * IO completion data structure (Completion Queue Entry) */ From patchwork Sat May 14 14:20:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12849816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0BC8C433FE for ; Sat, 14 May 2022 14:20:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233074AbiENOUk (ORCPT ); Sat, 14 May 2022 10:20:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232932AbiENOUh (ORCPT ); Sat, 14 May 2022 10:20:37 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D412B29C for ; Sat, 14 May 2022 07:20:34 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id d22so10492643plr.9 for ; Sat, 14 May 2022 07:20:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/MB3cYHRh72iKQ2NuBm+2JqwCkuwriU5DHURNuTzo4Q=; b=g0dr5lRE71lkn3eS8b+u3wYP7rewqhZCS3rdhahWuCFMmV53BThcDs0b7vNutMtwlY rvcLCEzbD7oT4me1jZbBNPDUWjcYECVH7N/F+Hud1AAymyB/xHArf9fe7FQd47N/GxMQ HYywV19qt9R5OH4OMl3kQJCfqYyAV16DipQ6rxaUKtwUrY/14T2EVaQqlmnsFDNwokcS vwrAUbiRqOY4kb94mcwdB6uDvGSSHP96VAMMpEn95UYR11lkBzxUj+yH4cUti0HyhYRo PKv/cDcf3txVpHDnN6zV0whz0QYlrFyojjig8D3ZaLSZCftXb2g/HtmoSqxEvUImEUWu cbCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/MB3cYHRh72iKQ2NuBm+2JqwCkuwriU5DHURNuTzo4Q=; b=AnD9hglu46EdKrZy2qUlu7nVP+TKsrd+CrpMCCFmLShHQEGBlTzkQBrSa3w8iFfmSu vZ/6Q2/vwUdYDgGjaMmcZP8K+5MAoGn8O7n5MIhs4hQlWTSpSsjv4RXXYYdtisgzxAtn rhxuL0WYzMWX1Uk/YfMgUC7fQWAs/mko2+jG5soKrDsVT3jlrwDXqtaN0191Bz/fJh1N IosfB/Szg9DuVkkzRpn1IxGywMVsI9vDsK6PkNWoNeNOlgZhFyTXj6PI4EgC1zuDRuya UrJqv0Bam7r2tda4yrt7o4IDlVgLgAKmUwaYTteR58jv3J9gicp/L1txy/BC7jhmT27L yxwg== X-Gm-Message-State: AOAM531LBXMcxGZYCYY88iQYC98BWgUFENMHFFmNayI/EeSdX54YbJfb hlhnbmIOzP2erTIvFMhUrNvJBJSokst+Lh9M X-Google-Smtp-Source: ABdhPJwt2HKYmmEPtVkMdNzMyQyppCg/TodW2i/HrN47f/gjOzxOKM/boh9Ca42JaL4+UPzaEzVNfg== X-Received: by 2002:a17:902:b906:b0:158:3120:3b69 with SMTP id bf6-20020a170902b90600b0015831203b69mr9190406plb.33.1652538034109; Sat, 14 May 2022 07:20:34 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([203.205.141.27]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b0015e8d4eb27csm3815968plg.198.2022.05.14.07.20.32 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 May 2022 07:20:33 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 2/4] io_uring: add REQ_F_APOLL_MULTISHOT for requests Date: Sat, 14 May 2022 22:20:44 +0800 Message-Id: <20220514142046.58072-3-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220514142046.58072-1-haoxu.linux@gmail.com> References: <20220514142046.58072-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a flag to indicate multishot mode for fast poll. currently only accept use it, but there may be more operations leveraging it in the future. Also add a mask IO_APOLL_MULTI_POLLED which stands for REQ_F_APOLL_MULTI | REQ_F_POLLED, to make the code short and cleaner. Signed-off-by: Hao Xu --- fs/io_uring.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/fs/io_uring.c b/fs/io_uring.c index c4bd0faf0b27..e1e550de5956 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -116,6 +116,8 @@ #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | REQ_F_LINK | REQ_F_HARDLINK |\ IO_REQ_CLEAN_FLAGS) +#define IO_APOLL_MULTI_POLLED (REQ_F_APOLL_MULTISHOT | REQ_F_POLLED) + #define IO_TCTX_REFS_CACHE_NR (1U << 10) struct io_uring { @@ -812,6 +814,7 @@ enum { REQ_F_SINGLE_POLL_BIT, REQ_F_DOUBLE_POLL_BIT, REQ_F_PARTIAL_IO_BIT, + REQ_F_APOLL_MULTISHOT_BIT, /* keep async read/write and isreg together and in order */ REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, @@ -876,6 +879,8 @@ enum { REQ_F_DOUBLE_POLL = BIT(REQ_F_DOUBLE_POLL_BIT), /* request has already done partial IO */ REQ_F_PARTIAL_IO = BIT(REQ_F_PARTIAL_IO_BIT), + /* fast poll multishot mode */ + REQ_F_APOLL_MULTISHOT = BIT(REQ_F_APOLL_MULTISHOT_BIT), }; struct async_poll { From patchwork Sat May 14 14:20:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12849817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC842C433EF for ; Sat, 14 May 2022 14:20:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233122AbiENOUl (ORCPT ); Sat, 14 May 2022 10:20:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233121AbiENOUi (ORCPT ); Sat, 14 May 2022 10:20:38 -0400 Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ED4F2B3 for ; Sat, 14 May 2022 07:20:36 -0700 (PDT) Received: by mail-pf1-x431.google.com with SMTP id k16so3371364pff.5 for ; Sat, 14 May 2022 07:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2FIFHdUPWdaPGI4xDJjIhjnONdaloxgrMvTH+estFXM=; b=M+uHcL66cFvyfd3So8vDo6owMSxqPZaKTCbyUULhErQWXk3REmtrLprojF0NZj/zq4 2vr838tj8O3j8f2VN6OEoWhcM459123Oti9Bc86nhSfyTratc0Tr+Kjmz8wTafrP86sm Q8RXEI38phigQPNLfi7K7oK+xEv4Urq5c0FVvjluSsowNSNo/m1OkSptAi6km0qiXqr+ rjrcDr1wuCO2BysjInSts4TUNsG0hpf8pqsCjziQZ2VnxU4DHudQ8msUW4UKryLEs+j3 cBhCsIMey1YX3/FaUELG0+N3UV9PfbUFPcvr/RvsSFLButnuGI/OT96GOg4Ay18dPM0Y jygw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2FIFHdUPWdaPGI4xDJjIhjnONdaloxgrMvTH+estFXM=; b=XcQuthWg3tp5/dk74tRZHtLYFzu2bdgHE6oDy9T96krnQpYns6aEob0CZl4I2L+5It Ea3Y3eEGvhTfdS4aeTUAxyXR1uwHbbefJU230yXQlMaWtsC3p2GMaLO6tU9Aochur8fm MUQX7zgRH6ItAatuDw8/h4Y0mqj9bJJHvydyByIzbtK5skwqtAyr/95DbaRgxR9kVNDG vGL47suNFUbL/stKBNyHQwuHN4VyEaAo4VlPyj+0zYZA3AiX/30Gwkjj4gCXFVay+dMA wAor8kulniQ1lQwmL7ehsiuJGs+r5AY7nuPdcon8Q2BXqf9aS9aWS9YMA9oa+iYBlpmv IymA== X-Gm-Message-State: AOAM532CWW02NqimshD8gHrzZUsAO6tZ3DywOr6VQM9XphpBFq73V8Kk Uhrm8Hc4mBV7hnf4OHBxCCjOoQnKqvtQB8WZ X-Google-Smtp-Source: ABdhPJxm5nNvMsAsVsswXTHoCKTuxXfK6wCe/deUBt5XqQuzX2qb0mpwaIV2XMjBCwXoUyoy1VUAUw== X-Received: by 2002:a05:6a00:c8f:b0:510:60cf:55fa with SMTP id a15-20020a056a000c8f00b0051060cf55famr9268881pfv.37.1652538036013; Sat, 14 May 2022 07:20:36 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([203.205.141.27]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b0015e8d4eb27csm3815968plg.198.2022.05.14.07.20.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 May 2022 07:20:35 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 3/4] io_uring: let fast poll support multishot Date: Sat, 14 May 2022 22:20:45 +0800 Message-Id: <20220514142046.58072-4-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220514142046.58072-1-haoxu.linux@gmail.com> References: <20220514142046.58072-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu For operations like accept, multishot is a useful feature, since we can reduce a number of accept sqe. Let's integrate it to fast poll, it may be good for other operations in the future. Signed-off-by: Hao Xu --- fs/io_uring.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index e1e550de5956..a83405f657e0 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -6011,6 +6011,7 @@ static void io_poll_remove_entries(struct io_kiocb *req) rcu_read_unlock(); } +static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags); /* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc. @@ -6019,10 +6020,10 @@ static void io_poll_remove_entries(struct io_kiocb *req) * either spurious wakeup or multishot CQE is served. 0 when it's done with * the request, then the mask is stored in req->cqe.res. */ -static int io_poll_check_events(struct io_kiocb *req, bool locked) +static int io_poll_check_events(struct io_kiocb *req, bool *locked) { struct io_ring_ctx *ctx = req->ctx; - int v; + int v, ret; /* req->task == current here, checking PF_EXITING is safe */ if (unlikely(req->task->flags & PF_EXITING)) @@ -6046,23 +6047,37 @@ static int io_poll_check_events(struct io_kiocb *req, bool locked) req->cqe.res = vfs_poll(req->file, &pt) & req->apoll_events; } - /* multishot, just fill an CQE and proceed */ - if (req->cqe.res && !(req->apoll_events & EPOLLONESHOT)) { - __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); + if ((unlikely(!req->cqe.res))) + continue; + if (req->apoll_events & EPOLLONESHOT) + return 0; + + /* multishot, just fill a CQE and proceed */ + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { + __poll_t mask = mangle_poll(req->cqe.res & + req->apoll_events); bool filled; spin_lock(&ctx->completion_lock); - filled = io_fill_cqe_aux(ctx, req->cqe.user_data, mask, - IORING_CQE_F_MORE); + filled = io_fill_cqe_aux(ctx, req->cqe.user_data, + mask, IORING_CQE_F_MORE); io_commit_cqring(ctx); spin_unlock(&ctx->completion_lock); - if (unlikely(!filled)) - return -ECANCELED; - io_cqring_ev_posted(ctx); - } else if (req->cqe.res) { - return 0; + if (filled) { + io_cqring_ev_posted(ctx); + continue; + } + return -ECANCELED; } + io_tw_lock(req->ctx, locked); + if (unlikely(req->task->flags & PF_EXITING)) + return -EFAULT; + ret = io_issue_sqe(req, + IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); + if (ret) + return ret; + /* * Release all references, retry if someone tried to restart * task_work while we were executing it. @@ -6077,7 +6092,7 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) struct io_ring_ctx *ctx = req->ctx; int ret; - ret = io_poll_check_events(req, *locked); + ret = io_poll_check_events(req, locked); if (ret > 0) return; @@ -6102,7 +6117,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) struct io_ring_ctx *ctx = req->ctx; int ret; - ret = io_poll_check_events(req, *locked); + ret = io_poll_check_events(req, locked); if (ret > 0) return; @@ -6343,7 +6358,7 @@ static int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) struct io_ring_ctx *ctx = req->ctx; struct async_poll *apoll; struct io_poll_table ipt; - __poll_t mask = IO_ASYNC_POLL_COMMON | POLLERR; + __poll_t mask = POLLPRI | POLLERR; int ret; if (!def->pollin && !def->pollout) @@ -6352,6 +6367,8 @@ static int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) return IO_APOLL_ABORTED; if ((req->flags & (REQ_F_POLLED|REQ_F_PARTIAL_IO)) == REQ_F_POLLED) return IO_APOLL_ABORTED; + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) + mask |= EPOLLONESHOT; if (def->pollin) { mask |= POLLIN | POLLRDNORM; From patchwork Sat May 14 14:20:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 12849818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CD67C433FE for ; Sat, 14 May 2022 14:20:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233121AbiENOUm (ORCPT ); Sat, 14 May 2022 10:20:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233155AbiENOUk (ORCPT ); Sat, 14 May 2022 10:20:40 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B0629FCB for ; Sat, 14 May 2022 07:20:38 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id v10so10027190pgl.11 for ; Sat, 14 May 2022 07:20:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cg/ciI7F+YQchIATLN/F9rg5PShOW5D8D7HnDBc9m7g=; b=FZzggD67gCOocjHs6WVqYVLIqDZ2lVJwr4vkXrxIETzmGvcfI1AibRImJFTBAL2CeZ c5UA6+XemluRvAM/9GaOwYbxHEn+TCKUc1aV3rHVs7AvTmxxhb7tLekzawcGq0+NgwYb kXpl/QZBMs6vaxNs/GzHRRtyVTrLvC8qIqh7vWTBuAlo+1ZGzaG8TY9qiiukl3hosj9F 6TXqC7bhLB0nKs3IOngTqMCFKGo3VbBg2Kq30nbmkk/NuW9Zal2RALYXW6ox6X2krpKO hbI/BTYHI4IN6KfmV1WiofQ8whamH+/F41QW52ZV5cABztKwxgpHi1mPmeJG3MkCO/bf yjSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cg/ciI7F+YQchIATLN/F9rg5PShOW5D8D7HnDBc9m7g=; b=P6kE86nVeusBSyAwjf73RZR3oW5o6hnsRrB8xTJD93fUIIoUnQT9Nvo1trKnKfTaxl /6+hCM4BiYqgOgtCffee8yfvnVP0vyNpccmUb47ZTKfkHBqpJdxAZMkuHsVKMviRB82P JB+n9Kl2z5rvMINmuIfex/jwYH8kOfyyLBfPeMdUmLIKOqXKhsteXkET6pjpIe44QKIt rNdosyGStTSC8zxQkpi6wBMcuE8R+KWiYKzlOvJ2D75U19Jo0oPthyn5A3ySmrRC2GRt /+FBpYLMRlEXLBriJmcZZeXTw+Yd9hiSaLb6t/ni+ddmxvOB8w702KMzkRYkMCpGpH6l ce4g== X-Gm-Message-State: AOAM533t88NpHUAr8TLQEeESrJiquJA3fCAvKWmqEYQknGBKKJ8kbcMB AH/kkPBGW89nqx9NpeYVk2I496MX6SEMJkxs X-Google-Smtp-Source: ABdhPJwPL0kPkMC0DU3KyFiP7F7DPeNeeFMw7ofvP+cPDCI7itT9g6TySVsgUKoliE2dGWqpmR1dog== X-Received: by 2002:a63:f1e:0:b0:3c1:d54f:fc47 with SMTP id e30-20020a630f1e000000b003c1d54ffc47mr8315403pgl.51.1652538037633; Sat, 14 May 2022 07:20:37 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([203.205.141.27]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b0015e8d4eb27csm3815968plg.198.2022.05.14.07.20.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 14 May 2022 07:20:37 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov Subject: [PATCH 4/4] io_uring: implement multishot mode for accept Date: Sat, 14 May 2022 22:20:46 +0800 Message-Id: <20220514142046.58072-5-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220514142046.58072-1-haoxu.linux@gmail.com> References: <20220514142046.58072-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Refactor io_accept() to support multishot mode. theoretical analysis: 1) when connections come in fast - singleshot: add accept sqe(userspace) --> accept inline ^ | |-----------------| - multishot: add accept sqe(userspace) --> accept inline ^ | |--*--| we do accept repeatedly in * place until get EAGAIN 2) when connections come in at a low pressure similar thing like 1), we reduce a lot of userspace-kernel context switch and useless vfs_poll() tests: Did some tests, which goes in this way: server client(multiple) accept connect read write write read close close Basically, raise up a number of clients(on same machine with server) to connect to the server, and then write some data to it, the server will write those data back to the client after it receives them, and then close the connection after write return. Then the client will read the data and then close the connection. Here I test 10000 clients connect one server, data size 128 bytes. And each client has a go routine for it, so they come to the server in short time. test 20 times before/after this patchset, time spent:(unit cycle, which is the return value of clock()) before: 1930136+1940725+1907981+1947601+1923812+1928226+1911087+1905897+1941075 +1934374+1906614+1912504+1949110+1908790+1909951+1941672+1969525+1934984 +1934226+1914385)/20.0 = 1927633.75 after: 1858905+1917104+1895455+1963963+1892706+1889208+1874175+1904753+1874112 +1874985+1882706+1884642+1864694+1906508+1916150+1924250+1869060+1889506 +1871324+1940803)/20.0 = 1894750.45 (1927633.75 - 1894750.45) / 1927633.75 = 1.65% Signed-off-by: Hao Xu --- fs/io_uring.c | 54 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 48 insertions(+), 6 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index a83405f657e0..d4752e192ef9 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1148,6 +1148,7 @@ static const struct io_op_def io_op_defs[] = { .unbound_nonreg_file = 1, .pollin = 1, .poll_exclusive = 1, + .ioprio = 1, /* used for flags */ }, [IORING_OP_ASYNC_CANCEL] = { .audit_skip = 1, @@ -5762,6 +5763,7 @@ static int io_recv(struct io_kiocb *req, unsigned int issue_flags) static int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_accept *accept = &req->accept; + unsigned flags; if (sqe->len || sqe->buf_index) return -EINVAL; @@ -5770,19 +5772,30 @@ static int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) accept->addr_len = u64_to_user_ptr(READ_ONCE(sqe->addr2)); accept->flags = READ_ONCE(sqe->accept_flags); accept->nofile = rlimit(RLIMIT_NOFILE); + flags = READ_ONCE(sqe->ioprio); + if (flags & ~IORING_ACCEPT_MULTISHOT) + return -EINVAL; accept->file_slot = READ_ONCE(sqe->file_index); - if (accept->file_slot && (accept->flags & SOCK_CLOEXEC)) - return -EINVAL; + if (accept->file_slot) { + if (accept->flags & SOCK_CLOEXEC) + return -EINVAL; + if (flags & IORING_ACCEPT_MULTISHOT && + accept->file_slot != IORING_FILE_INDEX_ALLOC) + return -EINVAL; + } if (accept->flags & ~(SOCK_CLOEXEC | SOCK_NONBLOCK)) return -EINVAL; if (SOCK_NONBLOCK != O_NONBLOCK && (accept->flags & SOCK_NONBLOCK)) accept->flags = (accept->flags & ~SOCK_NONBLOCK) | O_NONBLOCK; + if (flags & IORING_ACCEPT_MULTISHOT) + req->flags |= REQ_F_APOLL_MULTISHOT; return 0; } static int io_accept(struct io_kiocb *req, unsigned int issue_flags) { + struct io_ring_ctx *ctx = req->ctx; struct io_accept *accept = &req->accept; bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK; unsigned int file_flags = force_nonblock ? O_NONBLOCK : 0; @@ -5790,6 +5803,7 @@ static int io_accept(struct io_kiocb *req, unsigned int issue_flags) struct file *file; int ret, fd; +retry: if (!fixed) { fd = __get_unused_fd_flags(accept->flags, accept->nofile); if (unlikely(fd < 0)) @@ -5801,8 +5815,17 @@ static int io_accept(struct io_kiocb *req, unsigned int issue_flags) if (!fixed) put_unused_fd(fd); ret = PTR_ERR(file); - if (ret == -EAGAIN && force_nonblock) - return -EAGAIN; + if (ret == -EAGAIN && force_nonblock) { + /* + * if it's multishot and polled, we don't need to + * return EAGAIN to arm the poll infra since it + * has already been done + */ + if ((req->flags & IO_APOLL_MULTI_POLLED) == + IO_APOLL_MULTI_POLLED) + ret = 0; + return ret; + } if (ret == -ERESTARTSYS) ret = -EINTR; req_set_fail(req); @@ -5813,8 +5836,27 @@ static int io_accept(struct io_kiocb *req, unsigned int issue_flags) ret = io_fixed_fd_install(req, issue_flags, file, accept->file_slot); } - __io_req_complete(req, issue_flags, ret, 0); - return 0; + + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { + __io_req_complete(req, issue_flags, ret, 0); + return 0; + } + if (ret >= 0) { + bool filled; + + spin_lock(&ctx->completion_lock); + filled = io_fill_cqe_aux(ctx, req->cqe.user_data, ret, + IORING_CQE_F_MORE); + io_commit_cqring(ctx); + spin_unlock(&ctx->completion_lock); + if (filled) { + io_cqring_ev_posted(ctx); + goto retry; + } + ret = -ECANCELED; + } + + return ret; } static int io_connect_prep_async(struct io_kiocb *req)