From patchwork Thu Jun 23 13:24:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12892750 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF5EBC433EF for ; Thu, 23 Jun 2022 13:25:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230306AbiFWNZ2 (ORCPT ); Thu, 23 Jun 2022 09:25:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231147AbiFWNZ1 (ORCPT ); Thu, 23 Jun 2022 09:25:27 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E25649C9C for ; Thu, 23 Jun 2022 06:25:26 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id q9so27954707wrd.8 for ; Thu, 23 Jun 2022 06:25:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OpnL0/Edc6qwJvOc9mzgGZ0HMTdXfCTcomSoBlCvR7Q=; b=m/YlUxZc6sv21dQYx+9gNcGEBbQs5n5oiXyViRGxylSJfyJWOTpm+wtzBzIiRrgzXC 9rDMLFepxP17UUV0x499fYS4OMPaOVz++npIy6d0Hj/X+QhvxZDJqLQGhoRjduGurIuw CDSmO2UgTSajFmA2c9OaCEhUt/F9g0LC/GwwTgplSXPVz/2Vqdp+XBWJP+mkF3BXgmiu 5vQO8ucx3woy1Web6n8LSQfl9j7mm9skwg67aol586YmGRMs6+f0Iz8OcCzNfQV7RZ3I essoKyhlr1s0cAGBPP6DgM29pxK9pxioBzEZOwNqWlojVBh/EHaYvuhLSr3c6bylsKhR 0m/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OpnL0/Edc6qwJvOc9mzgGZ0HMTdXfCTcomSoBlCvR7Q=; b=asTxSVobGLmDeEc2hiBwGfJFtCMfNhecKyVrywizfKWhEW/2ZmiMfKlkAz3p+9vQjv 97k7ba0iPi8H1sCJjn4/H7LR/hVx/8PSC9YTohBK5oFqwzrYOrEek4LLOd+cfmCo+/7k feb7fS/FrntpppWukdBSlMtfbvP0m75eUdF4vd5v3WpaLn1KhoGLx0PfDfWtGZxpsAJ2 QI5uhUwApNEuCtdO4gFuc43zlXykEbVIb+GNRg07eNRZfYMsw/3yr6KcXoUby/x7myiQ 0+Ms2lRtOmbHRw5xh+PdzCu7evmrqzGcZjV9Vt8B9UyREeAl9ZwRO0iPi+owOYL3VsCQ KYJw== X-Gm-Message-State: AJIora9Rsg5uhC4diN9xz0f8xQr87CBeMrw68oci9kVXK4Dxfhx8lvTu 5XuDaHmCKnPYNzAn3ay/zP24PvIlNLyUp3rz X-Google-Smtp-Source: AGRyM1ulVIx6pCqksBSkkOYjc7iOKMrD3HkqLryo+wDpTGv3ilcleIfZDZPEq1eN/VuJ2L5xKuTs4Q== X-Received: by 2002:adf:ea43:0:b0:21b:9243:be8c with SMTP id j3-20020adfea43000000b0021b9243be8cmr8075054wrn.650.1655990724769; Thu, 23 Jun 2022 06:25:24 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b0039c5a765388sm3160620wmk.28.2022.06.23.06.25.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 06:25:24 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 1/6] io_uring: clean poll ->private flagging Date: Thu, 23 Jun 2022 14:24:44 +0100 Message-Id: <9a61240555c64ac0b7a9b0eb59a9efeb638a35a4.1655990418.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We store a req pointer in wqe->private but also take one bit to mark double poll entries. Replace macro helpers with inline functions for better type checking and also name the double flag. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index bd3110750cfa..210b174b155b 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -39,6 +39,22 @@ struct io_poll_table { #define IO_POLL_CANCEL_FLAG BIT(31) #define IO_POLL_REF_MASK GENMASK(30, 0) +#define IO_WQE_F_DOUBLE 1 + +static inline struct io_kiocb *wqe_to_req(struct wait_queue_entry *wqe) +{ + unsigned long priv = (unsigned long)wqe->private; + + return (struct io_kiocb *)(priv & ~IO_WQE_F_DOUBLE); +} + +static inline bool wqe_is_double(struct wait_queue_entry *wqe) +{ + unsigned long priv = (unsigned long)wqe->private; + + return priv & IO_WQE_F_DOUBLE; +} + /* * If refs part of ->poll_refs (see IO_POLL_REF_MASK) is 0, it's free. We can * bump it and acquire ownership. It's disallowed to modify requests while not @@ -306,8 +322,6 @@ static void io_poll_cancel_req(struct io_kiocb *req) io_poll_execute(req, 0, 0); } -#define wqe_to_req(wait) ((void *)((unsigned long) (wait)->private & ~1)) -#define wqe_is_double(wait) ((unsigned long) (wait)->private & 1) #define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI) static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, @@ -392,7 +406,7 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, return; } /* mark as double wq entry */ - wqe_private |= 1; + wqe_private |= IO_WQE_F_DOUBLE; req->flags |= REQ_F_DOUBLE_POLL; io_init_poll_iocb(poll, first->events, first->wait.func); *poll_ptr = poll; From patchwork Thu Jun 23 13:24:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12892751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAFE4C43334 for ; Thu, 23 Jun 2022 13:25:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231511AbiFWNZ3 (ORCPT ); Thu, 23 Jun 2022 09:25:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231147AbiFWNZ2 (ORCPT ); Thu, 23 Jun 2022 09:25:28 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 764DD49C94 for ; Thu, 23 Jun 2022 06:25:27 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id z9so11021957wmf.3 for ; Thu, 23 Jun 2022 06:25:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wCYu+CuA4ghhjalWpw26bcKN04Bihmyd2uysIWUIfg4=; b=le/bjXndQ47SINOsDSDvoHKihIDsOQDSDfGirbOds9niu6E/aAp/qKKjpOPUyeFSCE tBnbAQbJM4GAOVUL4PEbyxJSm3InMkuXxx9ZjE2Xk6Qn3DkXkbOXM6DVRCYpr5qqyYDK ylO6aJqRu5Cb+rO1bFLB+sn4ZjvcHJ5zSlJ8n6psl+6+hyVjCQHRKhjpXKZfOos/P+RQ GYLZKyoznOUqhArJYNA33ltG9pWCTWQmUda/sDS7/ECyzwlScc6wtLAqJbX/S0wXeSGU 0hzmQJI54lqWs5g22woerMuAr116loCKS/y5Av1B2MgxymhqG71MXSWXxKDlwBKweou0 fEVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wCYu+CuA4ghhjalWpw26bcKN04Bihmyd2uysIWUIfg4=; b=leeGpNlusx5Dz8QNbngiXCfwGSKA9l84d3UnOEBJt/3TH4YVCaceHuE7LIcs2DspXk VsGoLtekQeCPtB0OD1EX7v6GSpPdzn8cvslPyTWRkU8PF2zco4tSTn3KtF2RhUeKSo2y KvbnjmqEZ3wfgh7Q4f43gVuRJBjcjoWehM25vKWZecUFeoVceKH9aoc9aE13qz9uz4E1 KCSqP7ARn6kbcLuL1DDL/VT2OhrzNLW1hrwfT5OPsfj+PPMl1YhhNwL2EYhBMlP6RmYV UgMXU8Oz3D1S9mTySkxrqeYiqBuN4TpFpGaj6fndOaybgVl41UMwl+EMSbi8WH7XQ4hB I6Tw== X-Gm-Message-State: AJIora8Pk6xnEFkpVQf8QQhCd6kmtakxDopPcA68IIAGKwDzEAbM62ay AciPBAJ65R8gSNDoMLck5CVLl8PylF5k1riE X-Google-Smtp-Source: AGRyM1umSP+E8HKYubWbCxQOXiUubczZe+rnz3iNuY15Y2QwlErP2nTaUhtTAFvputlwk6vJfaatDQ== X-Received: by 2002:a05:600c:1d12:b0:39c:4307:8b10 with SMTP id l18-20020a05600c1d1200b0039c43078b10mr4148439wms.103.1655990725785; Thu, 23 Jun 2022 06:25:25 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b0039c5a765388sm3160620wmk.28.2022.06.23.06.25.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 06:25:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 2/6] io_uring: remove events caching atavisms Date: Thu, 23 Jun 2022 14:24:45 +0100 Message-Id: <12efd4e15c6a90cf9e5b59807cfcb57852b51dc7.1655990418.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Remove events argument from *io_poll_execute(), it's not needed and not used. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 210b174b155b..7de8c52793cd 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -289,8 +289,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) io_req_complete_failed(req, ret); } -static void __io_poll_execute(struct io_kiocb *req, int mask, - __poll_t __maybe_unused events) +static void __io_poll_execute(struct io_kiocb *req, int mask) { io_req_set_res(req, mask, 0); /* @@ -308,18 +307,17 @@ static void __io_poll_execute(struct io_kiocb *req, int mask, io_req_task_work_add(req); } -static inline void io_poll_execute(struct io_kiocb *req, int res, - __poll_t events) +static inline void io_poll_execute(struct io_kiocb *req, int res) { if (io_poll_get_ownership(req)) - __io_poll_execute(req, res, events); + __io_poll_execute(req, res); } static void io_poll_cancel_req(struct io_kiocb *req) { io_poll_mark_cancelled(req); /* kick tw, which should complete the request */ - io_poll_execute(req, 0, 0); + io_poll_execute(req, 0); } #define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI) @@ -334,7 +332,7 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, if (unlikely(mask & POLLFREE)) { io_poll_mark_cancelled(req); /* we have to kick tw in case it's not already */ - io_poll_execute(req, 0, poll->events); + io_poll_execute(req, 0); /* * If the waitqueue is being freed early but someone is already @@ -369,7 +367,7 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, else req->flags &= ~REQ_F_SINGLE_POLL; } - __io_poll_execute(req, mask, poll->events); + __io_poll_execute(req, mask); } return 1; } @@ -487,7 +485,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, req->apoll_events |= EPOLLONESHOT; ipt->error = 0; } - __io_poll_execute(req, mask, poll->events); + __io_poll_execute(req, mask); return 0; } @@ -497,7 +495,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, */ v = atomic_dec_return(&req->poll_refs); if (unlikely(v & IO_POLL_REF_MASK)) - __io_poll_execute(req, 0, poll->events); + __io_poll_execute(req, 0); return 0; } From patchwork Thu Jun 23 13:24:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12892752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB9D1C433EF for ; Thu, 23 Jun 2022 13:25:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231664AbiFWNZb (ORCPT ); Thu, 23 Jun 2022 09:25:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231348AbiFWNZ3 (ORCPT ); Thu, 23 Jun 2022 09:25:29 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F1F249CA2 for ; Thu, 23 Jun 2022 06:25:28 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id r81-20020a1c4454000000b003a0297a61ddso62680wma.2 for ; Thu, 23 Jun 2022 06:25:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CHxW2CtP1hlZ1Q8f7DTNKmbOSEwU/odj12C32OaND+I=; b=HYX44xPspvYu3IXafVq1OaDEE0Dq5XSSWE71//4ZaogDzeLmyeLhsm/CZq6HFk9lkY mxQxQP/fEB8FMfk7VMDDG1FZ2JIpFJeU+/dDZH6qO29dvwkTsXRYnaDZ4yBvjXoP026F TsuSuk7nnGDjZnYTB3tyLQgNO04F4BravPjKas/EgTcHYC7DXyr1YfMkwcyBEzv2AC5g jKEc1Mw6h7brUPL35J56FgdRPIDtMV/8mgHBrCsdoLLSKS8WPM2KqHsfTTqu1DGiJp7L CP7naSHQiLVcekfQfOUvpOkEaMLsAcNiEKjJwWkfDgJ1R3uC3G2Y+6X+UEzQzhwAt1Xm kL8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CHxW2CtP1hlZ1Q8f7DTNKmbOSEwU/odj12C32OaND+I=; b=H5RqNL4pIR+rwdy5DhuQ+AHwMc65ptX68k43jr83kkgOKZAcPGxUJokktCjcrwciu1 VuqCVOB83bJJ3FpqF4Xdrm3O6ty1l7sfxj0TyU/bXJ5r259qoU9wIRbAMxzkPWiTVaSd AMFN3MNBs22mhIvgdNzJRQc6e/7l7EBv1Awzcgf9Qvy1pv1LiyJYO3u8UGgB/Y9wp+pF SBWDzdNH+MfcSif264B1cpe+GtwhFBGYsYjU5DVBQMZnHRoNUxlRgSg535Hz0eorhVLc Hhou+dTVqivACzKXrvdEulNG6coNTSIUKzR3oAk4iEA/B11gdshhoexvs9/srZCMHD/L pP/g== X-Gm-Message-State: AJIora8lZHBYf6KK77pLvwGSRbkO5jHM1sLK4vm4dPKqJXuWQoeQSDPd knXJwo7lWL7u0yGP1jmdC3XZNd+BrlrFYXF+ X-Google-Smtp-Source: AGRyM1u3tK/pf/DZlMWi0KbOH/pqXEgpX/nIj4X610/X0YmUnxosywoJJaaQR49gHZ8fUlddUB7dTg== X-Received: by 2002:a05:600c:1e2a:b0:39c:51f8:80d4 with SMTP id ay42-20020a05600c1e2a00b0039c51f880d4mr4318011wmb.192.1655990726799; Thu, 23 Jun 2022 06:25:26 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b0039c5a765388sm3160620wmk.28.2022.06.23.06.25.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 06:25:26 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 3/6] io_uring: add a helper for apoll alloc Date: Thu, 23 Jun 2022 14:24:46 +0100 Message-Id: <2f93282b47dd678e805dd0d7097f66968ced495c.1655990418.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Extract a helper function for apoll allocation, makes the code easier to read. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 44 ++++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 7de8c52793cd..aef77f2a8a9a 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -508,10 +508,33 @@ static void io_async_queue_proc(struct file *file, struct wait_queue_head *head, __io_queue_proc(&apoll->poll, pt, head, &apoll->double_poll); } +static struct async_poll *io_req_alloc_apoll(struct io_kiocb *req, + unsigned issue_flags) +{ + struct io_ring_ctx *ctx = req->ctx; + struct async_poll *apoll; + + if (req->flags & REQ_F_POLLED) { + apoll = req->apoll; + kfree(apoll->double_poll); + } else if (!(issue_flags & IO_URING_F_UNLOCKED) && + !list_empty(&ctx->apoll_cache)) { + apoll = list_first_entry(&ctx->apoll_cache, struct async_poll, + poll.wait.entry); + list_del_init(&apoll->poll.wait.entry); + } else { + apoll = kmalloc(sizeof(*apoll), GFP_ATOMIC); + if (unlikely(!apoll)) + return NULL; + } + apoll->double_poll = NULL; + req->apoll = apoll; + return apoll; +} + int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) { const struct io_op_def *def = &io_op_defs[req->opcode]; - struct io_ring_ctx *ctx = req->ctx; struct async_poll *apoll; struct io_poll_table ipt; __poll_t mask = POLLPRI | POLLERR | EPOLLET; @@ -546,21 +569,10 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) } if (def->poll_exclusive) mask |= EPOLLEXCLUSIVE; - if (req->flags & REQ_F_POLLED) { - apoll = req->apoll; - kfree(apoll->double_poll); - } else if (!(issue_flags & IO_URING_F_UNLOCKED) && - !list_empty(&ctx->apoll_cache)) { - apoll = list_first_entry(&ctx->apoll_cache, struct async_poll, - poll.wait.entry); - list_del_init(&apoll->poll.wait.entry); - } else { - apoll = kmalloc(sizeof(*apoll), GFP_ATOMIC); - if (unlikely(!apoll)) - return IO_APOLL_ABORTED; - } - apoll->double_poll = NULL; - req->apoll = apoll; + + apoll = io_req_alloc_apoll(req, issue_flags); + if (!apoll) + return IO_APOLL_ABORTED; req->flags |= REQ_F_POLLED; ipt.pt._qproc = io_async_queue_proc; From patchwork Thu Jun 23 13:24:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12892753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5015C43334 for ; Thu, 23 Jun 2022 13:25:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230247AbiFWNZd (ORCPT ); Thu, 23 Jun 2022 09:25:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231228AbiFWNZa (ORCPT ); Thu, 23 Jun 2022 09:25:30 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6212C49C94 for ; Thu, 23 Jun 2022 06:25:29 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id z17so11030751wmi.1 for ; Thu, 23 Jun 2022 06:25:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=K1E1JLCe4USh2kriw3egu2gKlFBp9BC0Bq/laD0bxNE=; b=HMDtfNCV2sLqSQO6JuzTDSn3fFQl+f3NAC1m0lSuhIrF8gO0H67gATLcM3hyqWKK3i yqEnfvUs1rOtQP8B0XBvnigNukWLlasFHy7VR2vGPW7EC3U7kR2gw6cq3AR7RbaKSu8J L0eJivenSjonQas6tos6mNaJuUifH1EW8VLmh8Ig8kwGNAO+FrZQXFTaiFTtQwt7Lg57 JjacqmTT5idjbc7Pn881nH0TaFBIuo/5DnngXOhHymCI/YS/REF1mqnnKYDDz9ff0554 Utr7kf1eRnGHj35eERZdAzDEnIlNZ05b54oa6fjLsq7LZQXCpKCCitjxEyixAun7voe9 QObA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=K1E1JLCe4USh2kriw3egu2gKlFBp9BC0Bq/laD0bxNE=; b=249xgPYLTrIcrYqkELuboGO6feqd8tPH4kMfcGiOmIVCg6US/xsCHyyAQcRwOwLNu3 dbRJEKpWYwwHOHZaaZrTOYy9QmUhPZGyVyCeGzRqhSJ+atfa6ZXhiEo1pNJi6I9ZyEfm RE3bMeskxlepHviikaGp+7wGdPy/KVmVKaUIrLmtuNSzNFVaz2u5KquipfUBbOI36zK3 Bjg5uXIg3TTY+Zwtbo4h0mp9qO40rhOHF5eG4LLJDBiJOD3YaVCSd11XOozIO6s6zqFV DYpiRURuyTaH6eYErjHUhlUJ4fZgY1jLZwjoqKNWUrdMirzNDr5hRAnjw7fYrPMTs9R7 bCtA== X-Gm-Message-State: AJIora+gMsKUcSgc/ZYQjgAk6XhRODJAmFGX2NWFaE+elETdAI0t9s2T ROwwmGb9s9r8yVmgc56Vt+9J0gvojxLbxxJl X-Google-Smtp-Source: AGRyM1veqWP32CFObkXU+/SiHC1oLR9PPbW8iKBBqCjEzfJc7NzSnh3MG2C1HtdqOOzIGVznmuAq5g== X-Received: by 2002:a1c:720f:0:b0:3a0:2ac9:5231 with SMTP id n15-20020a1c720f000000b003a02ac95231mr4171519wmc.39.1655990727732; Thu, 23 Jun 2022 06:25:27 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b0039c5a765388sm3160620wmk.28.2022.06.23.06.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 06:25:27 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 4/6] io_uring: change arm poll return values Date: Thu, 23 Jun 2022 14:24:47 +0100 Message-Id: <529e29e9f97f2e6e383ccd44234d8b576a83a921.1655990418.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The rules for __io_arm_poll_handler()'s result parsing are complicated, as the first step don't pass return a mask but pass back a positive return code and fill ipt->result_mask. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index aef77f2a8a9a..80113b036c88 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -34,6 +34,8 @@ struct io_poll_table { struct io_kiocb *req; int nr_entries; int error; + /* output value, set only if arm poll returns >0 */ + __poll_t result_mask; }; #define IO_POLL_CANCEL_FLAG BIT(31) @@ -462,8 +464,9 @@ static int __io_arm_poll_handler(struct io_kiocb *req, if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { io_poll_remove_entries(req); + ipt->result_mask = mask; /* no one else has access to the req, forget about the ref */ - return mask; + return 1; } if (!mask && unlikely(ipt->error || !ipt->nr_entries)) { @@ -813,7 +816,7 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); if (ret) { - io_req_set_res(req, ret, 0); + io_req_set_res(req, ipt.result_mask, 0); return IOU_OK; } if (ipt.error) { From patchwork Thu Jun 23 13:24:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12892754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CCD9CCA47C for ; Thu, 23 Jun 2022 13:25:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231348AbiFWNZe (ORCPT ); Thu, 23 Jun 2022 09:25:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231147AbiFWNZb (ORCPT ); Thu, 23 Jun 2022 09:25:31 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BA7449C95 for ; Thu, 23 Jun 2022 06:25:30 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id p6-20020a05600c1d8600b003a035657950so515692wms.4 for ; Thu, 23 Jun 2022 06:25:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Jn8QJjicfncUefpIW7W7WlaVDAyYVlVVyqqS00/RmBQ=; b=MmIYcgBR7b5IuSfURDsUgfKK66eVD6uPrZIITgz2vDhdjAhbYSd3HlBplogvkIFffD FGVv/jQHIs4RPSiyBNaVN2T+ItpcBdOwkE6W+O6namlCvDMUhsS1reG5HWUCtg4ZWRJZ fCb39CsXrB6cmau6OQl2rbRH0G32VvNt9adcijooAtkCOQv27Pg8iU29mfiIqViSc/GE NOtBVJwvnPAy1eRkmBCX/vNVIuzPdVkcJBbjJ3mMwxaEtH0WuMj5oRcNoW9N7RzJm05C +gFH0HGbEPKbiS5Fgpkhpnj31NbVwC6XmHY8a8iR1zKYkKJgo5fYd7MYI++m/GjlJBok kVZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Jn8QJjicfncUefpIW7W7WlaVDAyYVlVVyqqS00/RmBQ=; b=Bk3+4H9GoGa+q2CWxpgh+htySmYXle4z+Lf8CsncKRmw7ec4a1XIHRfSIIp409m+DQ gxxdCEF3gsfnapIcZ05mMqb9hkwweAtNhVpV295dewc69CDS4NqiQlQDCMo1v2FreJlB ylTYm98CHYaxgQTNRbne1eDgtcJzQ3gPQCqRULS3WYenzAofp/sjy6e2lUQcSyBHK9pq 0jqywXyeUIpV5FNC3D77KvwNPjd66Hnq7Ck0stEfeJA3rPX6LFtVYURmqbSAw0cbPLiV 6LJgEcAY3zrcyfhWTX7fUPPJNCHiGmPJnJ6gu6KSA8X5K3SJKLZGb1h1QpD2yTa5GqOA XfVg== X-Gm-Message-State: AJIora8F2xbl8h/Hxd2rNQw40CcWHpwfmzjGwz+DHc1ROuZFqUipUqkW 16BHRQSGcIJIkI+kjriA5NVjugfp48L+CcN2 X-Google-Smtp-Source: AGRyM1tqc7+JwDvyAdusxC2UwlJxUWM3xgHJs8t4c/YlMCb3MX3aax42pmay8V8Ham7upBtziaAJ5A== X-Received: by 2002:a05:600c:a41:b0:39c:1512:98bd with SMTP id c1-20020a05600c0a4100b0039c151298bdmr4207426wmq.88.1655990728679; Thu, 23 Jun 2022 06:25:28 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b0039c5a765388sm3160620wmk.28.2022.06.23.06.25.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 06:25:28 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 5/6] io_uring: refactor poll arm error handling Date: Thu, 23 Jun 2022 14:24:48 +0100 Message-Id: <018cacdaef5fe95d7dc56b32e85d752cab7607f6.1655990418.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org __io_arm_poll_handler() errors parsing is a horror, in case it failed it returns 0 and the caller is expected to look at ipt.error, which already led us to a number of problems before. When it returns a valid mask, leave it as it's not, i.e. return 1 and store the mask in ipt.result_mask. In case of a failure that can be handled inline return an error code (negative value), and return 0 if __io_arm_poll_handler() took ownership of the request and will complete it. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 44 +++++++++++++++++++++----------------------- 1 file changed, 21 insertions(+), 23 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 80113b036c88..3f3ae3b1505f 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -435,6 +435,12 @@ static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, (struct io_poll **) &pt->req->async_data); } +/* + * Returns 0 when it's handed over for polling. The caller owns the requests if + * it returns non-zero, but otherwise should not touch it. Negative values + * contain an error code. When the result is >0, the polling has completed + * inline and ipt.result_mask is set to the mask. + */ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll *poll, struct io_poll_table *ipt, __poll_t mask) @@ -461,6 +467,17 @@ static int __io_arm_poll_handler(struct io_kiocb *req, atomic_set(&req->poll_refs, 1); mask = vfs_poll(req->file, &ipt->pt) & poll->events; + if (unlikely(ipt->error || !ipt->nr_entries)) { + io_poll_remove_entries(req); + + if (mask && (poll->events & EPOLLET)) { + ipt->result_mask = mask; + return 1; + } else { + return ipt->error ?: -EINVAL; + } + } + if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { io_poll_remove_entries(req); @@ -469,25 +486,12 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 1; } - if (!mask && unlikely(ipt->error || !ipt->nr_entries)) { - io_poll_remove_entries(req); - if (!ipt->error) - ipt->error = -EINVAL; - return 0; - } - if (req->flags & REQ_F_HASH_LOCKED) io_poll_req_insert_locked(req); else io_poll_req_insert(req); if (mask && (poll->events & EPOLLET)) { - /* can't multishot if failed, just queue the event we've got */ - if (unlikely(ipt->error || !ipt->nr_entries)) { - poll->events |= EPOLLONESHOT; - req->apoll_events |= EPOLLONESHOT; - ipt->error = 0; - } __io_poll_execute(req, mask); return 0; } @@ -582,9 +586,8 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) io_kbuf_recycle(req, issue_flags); ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask); - if (ret || ipt.error) - return ret ? IO_APOLL_READY : IO_APOLL_ABORTED; - + if (ret) + return ret > 0 ? IO_APOLL_READY : IO_APOLL_ABORTED; trace_io_uring_poll_arm(req, mask, apoll->poll.events); return IO_APOLL_OK; } @@ -815,16 +818,11 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) req->flags &= ~REQ_F_HASH_LOCKED; ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); - if (ret) { + if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); return IOU_OK; } - if (ipt.error) { - req_set_fail(req); - return ipt.error; - } - - return IOU_ISSUE_SKIP_COMPLETE; + return ret ?: IOU_ISSUE_SKIP_COMPLETE; } int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) From patchwork Thu Jun 23 13:24:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12892755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BAB6C433EF for ; Thu, 23 Jun 2022 13:25:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231229AbiFWNZg (ORCPT ); Thu, 23 Jun 2022 09:25:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231758AbiFWNZd (ORCPT ); Thu, 23 Jun 2022 09:25:33 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8BCC49C94 for ; Thu, 23 Jun 2022 06:25:31 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id e5so11043413wma.0 for ; Thu, 23 Jun 2022 06:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=F/dOoIiwrGbKhA06TnZaDVvCMhUOp/WZ7raiISfT++k=; b=a6pvkuBIIUGhD2+ndK5W9KzPe2bEjazhx9UmXD869caRL4UBM1lxNOHJHg0s2u4mgh 3TrKvSIyu89UTRTCO5HVkqQyR+Y8sevyQZmkSKrBKsO6xKW78NYYTtdrD0k4/ymtpXjx o3mZbNIBvfn9iUo8mml08if9tKCIupK4Wq4yqgBImjTWbU6Oc3BpxE6JCKQ9yNugoJgC Tkfdixewe+cWhINNQmHZ97AvFNl3E8DjrjczYvTiPWC24rAG6Xa0L+pQHt0fI3hvzbrd tHx3NXbOQAzHFeMsnA4tzwPbe5COz574kCSFfT4uXHkX+z1B+eOAqUoHNLRo1Q5iUrzu CZ6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=F/dOoIiwrGbKhA06TnZaDVvCMhUOp/WZ7raiISfT++k=; b=ROKF6T3NfKFLnsEQ2MiWFeBu7tvW/5sKI9qsVGDEQF+151qrmnDPj2sw0UB7sC/fY/ MLi/7RJPs0h/mpjeqZXvNyoM4URTyrlTPaO2tsS8PWV+80PH6N7+HB8V+dITSpHakUmZ n3LdvnTATvV+0QdBsODtx4TSjN72GLvW1TjLRe/G9X7eAME1RIlkULCCE5qbwumUdMYf ECyrx2jnaHWSw1Z4u+iVq9mgeeGx6+41rV85FVpbD/9XUiBNv12gY+MqSXJN/Ba+MdBE nwwZ5D4583aegR7Q/VObTs1z6FZQreXqJQ94Nmpm29PB9RbInm4XnC7gdbLv+fNguKWt DDdw== X-Gm-Message-State: AJIora8VubQQ0wcKJLebzjJdbZ/zLXDC6cAVpBgxHqCxq2cqtikmentN y17F8RoZZv3yfiArhhOTuVpLSQIm5GFMdwBc X-Google-Smtp-Source: AGRyM1sZKegKhmAAIcNSvYCkZk7TWTXRg/lhJ+aFLc9wXKQ1uU82wiAaCQc6ugmhsXJF8C+ZmMniDA== X-Received: by 2002:a05:600c:4ed0:b0:39c:4acb:4e04 with SMTP id g16-20020a05600c4ed000b0039c4acb4e04mr4105003wmq.63.1655990729850; Thu, 23 Jun 2022 06:25:29 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b0039c5a765388sm3160620wmk.28.2022.06.23.06.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jun 2022 06:25:29 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next v2 6/6] io_uring: optimise submission side poll_refs Date: Thu, 23 Jun 2022 14:24:49 +0100 Message-Id: <8825315d7f5e182ac1578a031e546f79b1c97d01.1655990418.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The final poll_refs put in __io_arm_poll_handler() takes quite some cycles. When we're arming from the original task context task_work won't be run, so in this case we can assume that we won't race with task_works and so not take the initial ownership ref. One caveat is that after arming a poll we may race with it, so we have to add a bunch of io_poll_get_ownership() hidden inside of io_poll_can_finish_inline() whenever we want to complete arming inline. For the same reason we can't just set REQ_F_DOUBLE_POLL in __io_queue_proc() and so need to sync with the first poll entry by taking its wq head lock. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 88 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 67 insertions(+), 21 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 3f3ae3b1505f..eba767594dee 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -34,6 +34,7 @@ struct io_poll_table { struct io_kiocb *req; int nr_entries; int error; + bool owning; /* output value, set only if arm poll returns >0 */ __poll_t result_mask; }; @@ -374,6 +375,27 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, return 1; } +static void io_poll_double_prepare(struct io_kiocb *req) +{ + struct wait_queue_head *head; + struct io_poll *poll = io_poll_get_single(req); + + /* head is RCU protected, see io_poll_remove_entries() comments */ + rcu_read_lock(); + head = smp_load_acquire(&poll->head); + if (head) { + /* + * poll arm may not hold ownership and so race with + * io_poll_wake() by modifying req->flags. There is only one + * poll entry queued, serialise with it by taking its head lock. + */ + spin_lock_irq(&head->lock); + req->flags |= REQ_F_DOUBLE_POLL; + spin_unlock_irq(&head->lock); + } + rcu_read_unlock(); +} + static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, struct wait_queue_head *head, struct io_poll **poll_ptr) @@ -405,16 +427,19 @@ static void __io_queue_proc(struct io_poll *poll, struct io_poll_table *pt, pt->error = -ENOMEM; return; } + + io_poll_double_prepare(req); /* mark as double wq entry */ wqe_private |= IO_WQE_F_DOUBLE; - req->flags |= REQ_F_DOUBLE_POLL; io_init_poll_iocb(poll, first->events, first->wait.func); *poll_ptr = poll; if (req->opcode == IORING_OP_POLL_ADD) req->flags |= REQ_F_ASYNC_DATA; + } else { + /* fine to modify, there is no poll queued to race with us */ + req->flags |= REQ_F_SINGLE_POLL; } - req->flags |= REQ_F_SINGLE_POLL; pt->nr_entries++; poll->head = head; poll->wait.private = (void *) wqe_private; @@ -435,6 +460,12 @@ static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, (struct io_poll **) &pt->req->async_data); } +static bool io_poll_can_finish_inline(struct io_kiocb *req, + struct io_poll_table *pt) +{ + return pt->owning || io_poll_get_ownership(req); +} + /* * Returns 0 when it's handed over for polling. The caller owns the requests if * it returns non-zero, but otherwise should not touch it. Negative values @@ -443,7 +474,8 @@ static void io_poll_queue_proc(struct file *file, struct wait_queue_head *head, */ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll *poll, - struct io_poll_table *ipt, __poll_t mask) + struct io_poll_table *ipt, __poll_t mask, + unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; int v; @@ -452,34 +484,45 @@ static int __io_arm_poll_handler(struct io_kiocb *req, req->work.cancel_seq = atomic_read(&ctx->cancel_seq); io_init_poll_iocb(poll, mask, io_poll_wake); poll->file = req->file; - req->apoll_events = poll->events; ipt->pt._key = mask; ipt->req = req; ipt->error = 0; ipt->nr_entries = 0; - /* - * Take the ownership to delay any tw execution up until we're done - * with poll arming. see io_poll_get_ownership(). + * Polling is either completed here or via task_work, so if we're in the + * task context we're naturally serialised with tw by merit of running + * the same task. When it's io-wq, take the ownership to prevent tw + * from running. However, when we're in the task context, skip taking + * it as an optimisation. + * + * Note: even though the request won't be completed/freed, without + * ownership we still can race with io_poll_wake(). + * io_poll_can_finish_inline() tries to deal with that. */ - atomic_set(&req->poll_refs, 1); + ipt->owning = issue_flags & IO_URING_F_UNLOCKED; + + atomic_set(&req->poll_refs, (int)ipt->owning); mask = vfs_poll(req->file, &ipt->pt) & poll->events; if (unlikely(ipt->error || !ipt->nr_entries)) { io_poll_remove_entries(req); - if (mask && (poll->events & EPOLLET)) { + if (!io_poll_can_finish_inline(req, ipt)) { + io_poll_mark_cancelled(req); + return 0; + } else if (mask && (poll->events & EPOLLET)) { ipt->result_mask = mask; return 1; - } else { - return ipt->error ?: -EINVAL; } + return ipt->error ?: -EINVAL; } if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { + if (!io_poll_can_finish_inline(req, ipt)) + return 0; io_poll_remove_entries(req); ipt->result_mask = mask; /* no one else has access to the req, forget about the ref */ @@ -491,18 +534,21 @@ static int __io_arm_poll_handler(struct io_kiocb *req, else io_poll_req_insert(req); - if (mask && (poll->events & EPOLLET)) { + if (mask && (poll->events & EPOLLET) && + io_poll_can_finish_inline(req, ipt)) { __io_poll_execute(req, mask); return 0; } - /* - * Release ownership. If someone tried to queue a tw while it was - * locked, kick it off for them. - */ - v = atomic_dec_return(&req->poll_refs); - if (unlikely(v & IO_POLL_REF_MASK)) - __io_poll_execute(req, 0); + if (ipt->owning) { + /* + * Release ownership. If someone tried to queue a tw while it was + * locked, kick it off for them. + */ + v = atomic_dec_return(&req->poll_refs); + if (unlikely(v & IO_POLL_REF_MASK)) + __io_poll_execute(req, 0); + } return 0; } @@ -585,7 +631,7 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) io_kbuf_recycle(req, issue_flags); - ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask); + ret = __io_arm_poll_handler(req, &apoll->poll, &ipt, mask, issue_flags); if (ret) return ret > 0 ? IO_APOLL_READY : IO_APOLL_ABORTED; trace_io_uring_poll_arm(req, mask, apoll->poll.events); @@ -817,7 +863,7 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) else req->flags &= ~REQ_F_HASH_LOCKED; - ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); + ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags); if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); return IOU_OK;