From patchwork Wed Nov 30 15:21:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060020 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2F6FC433FE for ; Wed, 30 Nov 2022 15:23:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229680AbiK3PXZ (ORCPT ); Wed, 30 Nov 2022 10:23:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229868AbiK3PXY (ORCPT ); Wed, 30 Nov 2022 10:23:24 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBAAB52171 for ; Wed, 30 Nov 2022 07:23:23 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id ja4-20020a05600c556400b003cf6e77f89cso1670075wmb.0 for ; Wed, 30 Nov 2022 07:23:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ps+n+jwP2OtqvbExQEi1qZlpRpt6nwJt8sdxryK+02Q=; b=GZcylmtvFhBufnMa1VWcvEUz+2D0c3ato8lrhSaxw5+5M/6lClo5UVN8Otoj0iDVaS Y0t5E3KGwK1iuqA8vgNfBbb5b4AzJtfufmdC3cgN8oCLxugkYlCCUPYlkrijxdfTkqiu NB9A3g5BwLPouxO4IXjYvKKezB7cSS9+ohQu8BbYDYvqP+hnLc0cPVTsgmgGLy8F2w8j utuM07HFUaINTNsgmIL/SCIGaozUR4qXrRK/RICCaXYLkeK3VSApjHFfewPXuEx4bDzs ViC5blO2SVrt4dkiH3St0Cwc3/4qiwLuZ9u5j6SdIV9vuVr+V2qfWkdDRijZ+YKL81lF S9jQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ps+n+jwP2OtqvbExQEi1qZlpRpt6nwJt8sdxryK+02Q=; b=EaK7f0slNkx2rw0myt6kcDwaXZkwrNxNfXo+caOkRTiYP42OYjVj2M4rd0uVkYPktG 9TWE7UnAD/UAlVsUg7c61I5HV2fgYzU1Dn2qMlFty/f9Ed8KoLvTbT72QqQ0GjLBBHN1 x387gBuEMUnPXM8XfjQc0w0OVNfOjC5UHPnM+xjZUHass/V/n09hnhRGUW6TTztYpKDs KVE5FLlvYDNIjRT0Qu5t5LPh5Y4qMm9NjLEotKgtX912zi6vVWI6dHfE9p3dQZmXLtbw 2eagWAw248skrnDCGpmA2qN8QI7BURO5w0MlT8x3n1t7yZ4r6VndtAKxBwDYsG9Y0rzO qibw== X-Gm-Message-State: ANoB5pltoJHt9HeZZAdGpgJY+gVtrpYVYJZDUvibhp+iTKSzk0kkvwku z4HGS/jCzJj3gD5Bj03vreoU6kjkND8= X-Google-Smtp-Source: AA0mqf5kH8EJW1iXMYDaRGhf7VtxcUk0wCx6sM5G7oo136RLl+7wMsP6ErVCdVVQzHMnySw0/RV9+w== X-Received: by 2002:a05:600c:154e:b0:3c7:1422:d56a with SMTP id f14-20020a05600c154e00b003c71422d56amr16355689wmg.107.1669821802124; Wed, 30 Nov 2022 07:23:22 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:21 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 1/9] io_uring: kill io_poll_issue's PF_EXITING check Date: Wed, 30 Nov 2022 15:21:51 +0000 Message-Id: <2e9dc998dc07507c759a0c9cb5d2fbea0710d58c.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We don't need to worry about checking PF_EXITING in io_poll_issue(). task works using the function should take care of it and never try to resubmit / retry if the task is dying. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index adecdf65b130..15d285d8ce0f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1808,8 +1808,6 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) int io_poll_issue(struct io_kiocb *req, bool *locked) { io_tw_lock(req->ctx, locked); - if (unlikely(req->task->flags & PF_EXITING)) - return -EFAULT; return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_MULTISHOT| IO_URING_F_COMPLETE_DEFER); } From patchwork Wed Nov 30 15:21:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060022 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6773AC46467 for ; Wed, 30 Nov 2022 15:23:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229877AbiK3PX1 (ORCPT ); Wed, 30 Nov 2022 10:23:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229870AbiK3PXZ (ORCPT ); Wed, 30 Nov 2022 10:23:25 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3E7E74639 for ; Wed, 30 Nov 2022 07:23:24 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id bx10so15549425wrb.0 for ; Wed, 30 Nov 2022 07:23:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gLJAaEfpJqzQKuaMa6gQqKCdtBAxlAHoBY9FLqBsiKY=; b=Ftg2V87aTsXhEdtSmHIMr+4tpTnj8KgXgCcxSCQPqotod0qDi4e0N7luoSul+U23XQ Pn0mFZG+lvQFViL41nzzwxROkg1S5bXi6tdIzkc21Ugm3e9HvpRz8hsi3pDPZ9/wZHJw yT6Lyben+btYVB4yOSPVxRzKqDZmmt38x0a8eJ7HJ0JzW/PAQxAi9UbNhT66GXcixetJ /KuMyN7BCiBIwzCleqNDPfwZ/AOLEXtvSzMPqgz42yDrN5fntZCTgfBr+OAFrxpe2VXo Sgv8Ao+xsx/rv/zv+GTGtVosqknux2jIIMj6FvO5kE+ViAPXrge+4bjszHhKfom9jngd M7Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gLJAaEfpJqzQKuaMa6gQqKCdtBAxlAHoBY9FLqBsiKY=; b=YgDm90uyXdKoXPZaQNS+6X1n4NUvQd3GnhL+Jj1O5fnNsmqxGAPwAJIjHRDmmZy1i8 I53aiX5KnHvAamrUeaBsfh6Drs3bMAoNL6MPEV55AEswDcDOYrWZ1xjgA/79Tr4GIptH M4MW8Qyfm/jSV5P5NLl5VkBpEZ8mBJcraGh0hZoARcgkVse32MNJjwXtWmcJX/6XVBDb sI4FJC77u5zDLCnc8V0PaN9ARJHtCxavkoEK2ZfalmzSAkxg47ZtBrHXB1XatspcTmeA qyVB4dfcmFxTOWOnMqUSuVfze8+3fktiOJyDbjGFQma8/uUYz4zg6CT7mBg+znD1qITQ DcMQ== X-Gm-Message-State: ANoB5pnh6BSyv23MoxkRy8WSdGIKlIUBnNy/ShyAnuyhqVslx+87p0ul trOg2i7TQyoeB6b8OddB1x9WX5fSKjw= X-Google-Smtp-Source: AA0mqf523G64JKnzlmWh3Roci90fRnTbvzrU67YmJaEeDc8hg1nISBJwtvTJBX3n/7F0Mn+ZuLreFA== X-Received: by 2002:a5d:44c8:0:b0:242:2a46:6ff9 with SMTP id z8-20020a5d44c8000000b002422a466ff9mr2618642wrr.371.1669821802895; Wed, 30 Nov 2022 07:23:22 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:22 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 2/9] io_uring: carve io_poll_check_events fast path Date: Wed, 30 Nov 2022 15:21:52 +0000 Message-Id: <8c21c5d5e027e32dc553705e88796dec79ff6f93.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The fast path in io_poll_check_events() is when we have only one (i.e. master) reference. Move all verification, cancellations checks, edge case handling and so on under a common if. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 599ba28c89b2..8987e13d302e 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -247,27 +247,30 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) do { v = atomic_read(&req->poll_refs); - /* tw handler should be the owner, and so have some references */ - if (WARN_ON_ONCE(!(v & IO_POLL_REF_MASK))) - return IOU_POLL_DONE; - if (v & IO_POLL_CANCEL_FLAG) - return -ECANCELED; - /* - * cqe.res contains only events of the first wake up - * and all others are be lost. Redo vfs_poll() to get - * up to date state. - */ - if ((v & IO_POLL_REF_MASK) != 1) - req->cqe.res = 0; - if (v & IO_POLL_RETRY_FLAG) { - req->cqe.res = 0; + if (unlikely(v != 1)) { + /* tw should be the owner and so have some refs */ + if (WARN_ON_ONCE(!(v & IO_POLL_REF_MASK))) + return IOU_POLL_DONE; + if (v & IO_POLL_CANCEL_FLAG) + return -ECANCELED; /* - * We won't find new events that came in between - * vfs_poll and the ref put unless we clear the flag - * in advance. + * cqe.res contains only events of the first wake up + * and all others are to be lost. Redo vfs_poll() to get + * up to date state. */ - atomic_andnot(IO_POLL_RETRY_FLAG, &req->poll_refs); - v &= ~IO_POLL_RETRY_FLAG; + if ((v & IO_POLL_REF_MASK) != 1) + req->cqe.res = 0; + + if (v & IO_POLL_RETRY_FLAG) { + req->cqe.res = 0; + /* + * We won't find new events that came in between + * vfs_poll and the ref put unless we clear the + * flag in advance. + */ + atomic_andnot(IO_POLL_RETRY_FLAG, &req->poll_refs); + v &= ~IO_POLL_RETRY_FLAG; + } } /* the mask was stashed in __io_poll_execute */ From patchwork Wed Nov 30 15:21:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7478C352A1 for ; Wed, 30 Nov 2022 15:23:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229870AbiK3PX2 (ORCPT ); Wed, 30 Nov 2022 10:23:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48208 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229878AbiK3PX0 (ORCPT ); Wed, 30 Nov 2022 10:23:26 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3510872082 for ; Wed, 30 Nov 2022 07:23:25 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id o5so18614763wrm.1 for ; Wed, 30 Nov 2022 07:23:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nmyl7LkvaPhzlhUrU0LzfCAM48z0JEbH4UQCVKBCzOI=; b=CjfiXD0wsH9piZI8ulENctV8dpWYqMhtsbfujUQ48h8FmANWxjJoasDylJBxdFOf+y IM4EGfVUfgQMuv+36Vxx3olcCDFRv5QiKEsUe80TlZ4IDrzbfNNujoptDQ9nmt6h3+Dm NhvTdmAHvssGaCw3/Pkxc/9HS+5p0jrtnQF5B42qWF3dUGrFNr/EtgWIN2VXQ2EWdDfR EvkQmzfPrRVhpGklIjPUpiGMGz10VOZ+LvRA/btXKluxVF+mtFiVhowIQJXtW2l3hACh /ccMX6RNJP2YWH89BB3AhTWmHqhGU7MAq4LYG7RJA9yFEwo+p3c7/Gtpz8mu8k0XtAou D8lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nmyl7LkvaPhzlhUrU0LzfCAM48z0JEbH4UQCVKBCzOI=; b=R5nz5hQcA5BAQZlWZN3VDF9lxLpyqf+krf0z11toreTamU30kxXxrOCTRV+GOKgL/F 9nRh/ktMHGsPxYtOmnK/ogD4wKNN4fm6BM0xcIKDWzS87NC8XEfEPv3mZxsY5FqCzERj wWJWA8l2ncT8Oy+2LKxLwntPU7Xg9/foN+QeR6AtP5s7Fjx6xKbYY7ThIMOn4aaqGErl Ooy+6PE5CprulJg8XA8ifTfN3q6z9RcKcltcMuG30HEvS+DO073N8S66qxwUIbIT3847 k5kdvh+eedmk8QjRTaFqYxWCog9nYe2p6fS+3LvibnMqbjc52+7H5ozCdU6cT4b+VAD0 GqZg== X-Gm-Message-State: ANoB5pnWajcehirrbEhDa0rpRwlJAQX30OQlaTB8q8n+FICIM6KgX9wM vjrUV6YOFV3POZme4rYltZgrxmeyn7c= X-Google-Smtp-Source: AA0mqf5ML+UX00idL1nlEFtWsX6hbLqB7yi9GYE7ISn3ZJef6TKIg1QJNgjCevMfVqPLFLldY2q2PQ== X-Received: by 2002:a5d:61cd:0:b0:241:f8e3:7111 with SMTP id q13-20020a5d61cd000000b00241f8e37111mr21165472wrv.299.1669821803585; Wed, 30 Nov 2022 07:23:23 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:23 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 3/9] io_uring: remove ctx variable in io_poll_check_events Date: Wed, 30 Nov 2022 15:21:53 +0000 Message-Id: <552c1771f8a0e7688afdb4f538ead245f53e80e7.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org ctx is only used by io_poll_check_events() for multishot poll CQE posting, don't save it on stack in advance. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 8987e13d302e..ada0017e3d88 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -237,7 +237,6 @@ enum { */ static int io_poll_check_events(struct io_kiocb *req, bool *locked) { - struct io_ring_ctx *ctx = req->ctx; int v, ret; /* req->task == current here, checking PF_EXITING is safe */ @@ -289,7 +288,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) __poll_t mask = mangle_poll(req->cqe.res & req->apoll_events); - if (!io_aux_cqe(ctx, *locked, req->cqe.user_data, + if (!io_aux_cqe(req->ctx, *locked, req->cqe.user_data, mask, IORING_CQE_F_MORE, false)) { io_req_set_res(req, mask, 0); return IOU_POLL_REMOVE_POLL_USE_RES; From patchwork Wed Nov 30 15:21:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060023 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C98E2C4332F for ; Wed, 30 Nov 2022 15:23:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229897AbiK3PX3 (ORCPT ); Wed, 30 Nov 2022 10:23:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229886AbiK3PX1 (ORCPT ); Wed, 30 Nov 2022 10:23:27 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F401174CE2 for ; Wed, 30 Nov 2022 07:23:25 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id m14so3677657wrh.7 for ; Wed, 30 Nov 2022 07:23:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XxlMuKJgc1CFXoxEQrn/owGBJpxjUhSInGIm+vaV3uE=; b=qLFLxYzU3LQN9aT+VsDRHK9Usjn2o+ph7eVFCEn8ZVr3jmFTgCN67hC2kLHzLbzq1F 2iE50D/i+g4BiNYXNzCksutObbZa/aXGWALvDIHWEoVbjLLYkBYHMy5+FjDEfOwj++R/ VTXJHtuyYX2yNxOlVLzwR1gpvZ+OBXjOqIORE/zWi5PJUUUTy3WmEsOWZ44CS7B8dmPB 1eQeWQczNxuPAffywSihs1ozrLQZoGvWC1w4Fhl81uQhylH3DPOpOXiIYgZFr5p/lIab WDx2JWEIWzcUPBJDa0kfAnS6ks5PeRM6oOEGGy7XqN1jvYeQYnAK5r0pIrZNnPsbfIBT 7XVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XxlMuKJgc1CFXoxEQrn/owGBJpxjUhSInGIm+vaV3uE=; b=kegYShVM1Y27a6i9363GUIMnltffwMEY2zibKJGiqvb/aAFVjO4C7xJOJV+A6iY8AE dcVrYMAigAXYd9NTRij5KEIkP49cV764ljUi1NJMI1h6UBcg5cSum9W3tX85AoUc7mzP qm9AUTAl+Ivelavj6iP1N+v57jxgc3K5qgIDKAdWsSyHPW4CFjCo2ubdGisxjYWaJmgy DvXsb4STJBeQVMlmqmXvVZyu9WV1KZuyK1M8mX42r2qpClJoRabDlF0oj1O0cJ/Ajahg 5MzRb+7iLsEcL+5IjFoTc9xrgW0o/zKy5I6NiAQ7nRutGOYsbfmkjQzG+HDkRkwyti62 /ZmQ== X-Gm-Message-State: ANoB5pkg+udlVxLugU5vHASCaZ47Oqq/aAQ/AxsVfBalYmANQKHphkS6 ddyogUZQMpvb3qgWlpxXJD23jrRO+3w= X-Google-Smtp-Source: AA0mqf6k1SuSy3RFj2kIjtLA/33LaSgj1vTi/71nMVGNoj+f8+UP9aTRnChaNphqrrGfC6+hutWaXA== X-Received: by 2002:a05:6000:3c3:b0:242:378:38d1 with SMTP id b3-20020a05600003c300b00242037838d1mr17910779wrg.352.1669821804348; Wed, 30 Nov 2022 07:23:24 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:24 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 4/9] io_uring: imporve poll warning handling Date: Wed, 30 Nov 2022 15:21:54 +0000 Message-Id: <31edf9f96f05d03ab62c114508a231a2dce434cb.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't try to complete requests if their refs are broken and we've got a warning, it's much better to drop them and potentially leaking than double freeing. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index ada0017e3d88..8f16d2a48ff8 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -249,7 +249,7 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) if (unlikely(v != 1)) { /* tw should be the owner and so have some refs */ if (WARN_ON_ONCE(!(v & IO_POLL_REF_MASK))) - return IOU_POLL_DONE; + return IOU_POLL_NO_ACTION; if (v & IO_POLL_CANCEL_FLAG) return -ECANCELED; /* From patchwork Wed Nov 30 15:21:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EC99C352A1 for ; Wed, 30 Nov 2022 15:23:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229774AbiK3PXa (ORCPT ); Wed, 30 Nov 2022 10:23:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229892AbiK3PX2 (ORCPT ); Wed, 30 Nov 2022 10:23:28 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86E9B769E6 for ; Wed, 30 Nov 2022 07:23:26 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id h11so20345281wrw.13 for ; Wed, 30 Nov 2022 07:23:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zLXpILH9PcccMmd6KkcbcpMJ63pfhz2reMG+GunS118=; b=Sx5AMy3L8jxqD5t1jZt+qxpaWUUji45LC5pgWQTC7q9xi0IR+kefFApl4Uakl9PwBJ 0JsP4DjrEFQn9SYzP9oILju1lJvoxAs94Huo91QqKuWDuiSqxKElxyuyiUPtXknUIntn U04aImEuKHc35sM+GhS/mmYuZrz5yDlkv9/gMW+ImJN2tE1jf3QZtS0gL4vNOVYxjKgq ztgyqgxZFTzrhoNbld/rparfz/S/jwXBaI6Y1uQgpNWLgAyvom6LPEcj+XvXQqqu/zhr Z9D3c7+oWLOi8N5grng8LvILFFEyHyxPoXZvcf69JuFoXnbgJlwLc0VPvICyINpEq6F6 3VwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zLXpILH9PcccMmd6KkcbcpMJ63pfhz2reMG+GunS118=; b=D9bMgPeWb09rhEOaRyrBn6E8xKFzb9xjsqsbCXkHrh+6Pf/rkG0UBilIpVrzzYZMw8 6GN9u+uEl1L4QsHGf+JZJ/o6/iQ0hcvBjnxuFAr/yHVTkKEwMqKtoEPSku3jW35fl3SR I3RSW1xCUGPbZwhcFSto6TIyQ2VV17ej+GqnAABjgy8x1HNs+AlqBPXR7s2oEVdcn8js oBRvuCekzRyi5sbw0U60vwK0JoS17q8PBJkY4Jdso0w61cL760d+cHkcxOGEpLOz9LmY JQtGfPPoP6KjY6WeD8eHE9nKFosKIIYK2c2k5mxR/U54c4o0e000Fr2e792AFzcYDvQJ HYzw== X-Gm-Message-State: ANoB5pk6/i83B3pmIaNlu2mJOSHrnBlJdYjM/raN8upGc8F1HL8VBJkY K4Kbaq5B4aUI6C9TjhpY7yCrvjMogHI= X-Google-Smtp-Source: AA0mqf7B0HYPbJlgTTU8SbCgxD6taAwV92D36eseT5V+Z4RfKiHghX+2Fkh4Vsh2bLD3g9LQIKAqaw== X-Received: by 2002:a5d:464c:0:b0:242:2ac1:375 with SMTP id j12-20020a5d464c000000b002422ac10375mr2388381wrs.432.1669821804950; Wed, 30 Nov 2022 07:23:24 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:24 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 5/9] io_uring: combine poll tw handlers Date: Wed, 30 Nov 2022 15:21:55 +0000 Message-Id: <482e59edb9fc81bd275fdbf486837330fb27120a.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Merge apoll and regular poll tw handlers, it will help with inlining. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 54 +++++++++++++++++++------------------------------ 1 file changed, 21 insertions(+), 33 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 8f16d2a48ff8..ee7da6150ec4 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -321,50 +321,38 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) ret = io_poll_check_events(req, locked); if (ret == IOU_POLL_NO_ACTION) return; - - if (ret == IOU_POLL_DONE) { - struct io_poll *poll = io_kiocb_to_cmd(req, struct io_poll); - req->cqe.res = mangle_poll(req->cqe.res & poll->events); - } else if (ret != IOU_POLL_REMOVE_POLL_USE_RES) { - req->cqe.res = ret; - req_set_fail(req); - } - io_poll_remove_entries(req); io_poll_tw_hash_eject(req, locked); - io_req_set_res(req, req->cqe.res, 0); - io_req_task_complete(req, locked); -} + if (req->opcode == IORING_OP_POLL_ADD) { + if (ret == IOU_POLL_DONE) { + struct io_poll *poll; -static void io_apoll_task_func(struct io_kiocb *req, bool *locked) -{ - int ret; - - ret = io_poll_check_events(req, locked); - if (ret == IOU_POLL_NO_ACTION) - return; - - io_tw_lock(req->ctx, locked); - io_poll_remove_entries(req); - io_poll_tw_hash_eject(req, locked); + poll = io_kiocb_to_cmd(req, struct io_poll); + req->cqe.res = mangle_poll(req->cqe.res & poll->events); + } else if (ret != IOU_POLL_REMOVE_POLL_USE_RES) { + req->cqe.res = ret; + req_set_fail(req); + } - if (ret == IOU_POLL_REMOVE_POLL_USE_RES) + io_req_set_res(req, req->cqe.res, 0); io_req_task_complete(req, locked); - else if (ret == IOU_POLL_DONE) - io_req_task_submit(req, locked); - else - io_req_defer_failed(req, ret); + } else { + io_tw_lock(req->ctx, locked); + + if (ret == IOU_POLL_REMOVE_POLL_USE_RES) + io_req_task_complete(req, locked); + else if (ret == IOU_POLL_DONE) + io_req_task_submit(req, locked); + else + io_req_defer_failed(req, ret); + } } static void __io_poll_execute(struct io_kiocb *req, int mask) { io_req_set_res(req, mask, 0); - - if (req->opcode == IORING_OP_POLL_ADD) - req->io_task_work.func = io_poll_task_func; - else - req->io_task_work.func = io_apoll_task_func; + req->io_task_work.func = io_poll_task_func; trace_io_uring_task_add(req, mask); io_req_task_work_add(req); From patchwork Wed Nov 30 15:21:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3B80C4708A for ; Wed, 30 Nov 2022 15:23:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229868AbiK3PXb (ORCPT ); Wed, 30 Nov 2022 10:23:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229894AbiK3PX2 (ORCPT ); Wed, 30 Nov 2022 10:23:28 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3391177210 for ; Wed, 30 Nov 2022 07:23:27 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id f18so5560337wrj.5 for ; Wed, 30 Nov 2022 07:23:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NKZvNdTv2UR+lj5KsPKwOG/zN0kLMd+y14jmvQ1MFlY=; b=ORbnk+ZpgNEqqfAZdFbRtrSot8of8CrFkhsabWaOFtu4odoGjCmnNt6rsLPTw0xfZ/ W/2FJ2fm5Pqw/11X7cTikoamajfwqTUXnb5NiQ5gQxvpMBrCOVutbCsDxVUIMvGt/uLN QieqdigvZfI3adYmhmO4USP96bv6AFEGH3Z6nIQPGpiWeX0N45Oy7Q7LZ8FHwstWUsg9 hpbY9pWGerbSvyaWs0UvDQ16ocGjeWC7zplbFe4miEp5BitOCTWf5PVZUntkvGYRuSFO rBCc+25/CfK161oRkoYLoAYdMT9vGJwcd+RLgpYTaBkLSxTMw441REnEfed+Er4mn7Ux rWwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NKZvNdTv2UR+lj5KsPKwOG/zN0kLMd+y14jmvQ1MFlY=; b=NB3Q2WeA4VWs6YDP4UGYMdx9OAvwSD/dMiQGLvJoir8gsgdDhywmV3K23cXCZAZVMO uDxQZHq/nHzxvT63Lo+CEouiooGXm8eZ9j9Je64ixnyPJcTj3L1HrpxJMmminTWMBRIp 7iiraMxFrKmqP0QVVBiIBrtARhOg1eHUQgUGhPE7c1RoW89dnChWNnbcaOnmbSf+UkJ1 E7vgsp/+stUf5CSUZiSVwD2Lr7Yxzf/he0x0U+/usdBJtZEsoUzw8UDFzW5MeDu7FUsm GxaLCeKrXtYX4BLpLFCKmAK0jj7YbH8Lv9LGdWHG0OeytqGF5Q1rK4AQEG35quhEB2cf Trkw== X-Gm-Message-State: ANoB5pk7dkP+XPYIXrVyLeWFZwmLJ5YFuTU95pbgBSClR8YDe3/XQhc4 2eQ3zZhr9XQSnphktVs/Rfse7o8LDm4= X-Google-Smtp-Source: AA0mqf51EnaMsUKxzocTCzURQAIYNECjeeof0ps6lGyLBbXCUIABpp0nO2kXDEvJ/qvtQlqjLUvFXQ== X-Received: by 2002:adf:e28b:0:b0:241:c2c3:26c8 with SMTP id v11-20020adfe28b000000b00241c2c326c8mr34788307wri.278.1669821805564; Wed, 30 Nov 2022 07:23:25 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:25 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 6/9] io_uring: don't raw spin unlock to match cq_lock Date: Wed, 30 Nov 2022 15:21:56 +0000 Message-Id: <4ca4f0564492b90214a190cd5b2a6c76522de138.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is one newly added place when we lock ring with io_cq_lock() but unlocking is hand coded calling spin_unlock directly. It's ugly and troublesome in the long run. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 15d285d8ce0f..c30765579a8e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -860,7 +860,7 @@ bool io_aux_cqe(struct io_ring_ctx *ctx, bool defer, u64 user_data, s32 res, u32 io_cq_lock(ctx); __io_flush_post_cqes(ctx); /* no need to flush - flush is deferred */ - spin_unlock(&ctx->completion_lock); + io_cq_unlock(ctx); } /* For defered completions this is not as strict as it is otherwise, diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 062899b1fe86..2277c05f52a6 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -93,6 +93,11 @@ static inline void io_cq_lock(struct io_ring_ctx *ctx) spin_lock(&ctx->completion_lock); } +static inline void io_cq_unlock(struct io_ring_ctx *ctx) +{ + spin_unlock(&ctx->completion_lock); +} + void io_cq_unlock_post(struct io_ring_ctx *ctx); static inline struct io_uring_cqe *io_get_cqe_overflow(struct io_ring_ctx *ctx, From patchwork Wed Nov 30 15:21:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48909C46467 for ; Wed, 30 Nov 2022 15:23:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229883AbiK3PXb (ORCPT ); Wed, 30 Nov 2022 10:23:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229890AbiK3PX3 (ORCPT ); Wed, 30 Nov 2022 10:23:29 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E40117721C for ; Wed, 30 Nov 2022 07:23:27 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id y16so7873030wrm.2 for ; Wed, 30 Nov 2022 07:23:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ApsC/V76AtmxJR1f/OYmi6DVMVs28z9+ywbUb2PrgXo=; b=d8hTzTUUinEhuT7VLFyqOCUWC/9EerJ5vFt768HF26WXNJ/q2GQSx0OrUC5ZSH2BlS mWWKakLsRCu54BzqTUnhpcNGuHRP9W7vEEllFoeshM00rFiksZ0wmbc/Q5SKY86xFNMr 6PcI4vIEckQ0nEKt+1Ymda1CaTQdO5WvVkvFDtf+9Qcy886uc64+YHnLXYJXwtbPFNmP V1TrIOjbvFRzR+mIMwr7IkygbFGdyBQFC8vqYP/y2taooS6Xk5IRQ59tGXiRcr/9EpHj KGvRYLf4dvsVvTC3Vu2NsyBlh7XRvu87nCETR2ySsQVerlSbAy3Ak0L6LZjiwPz5FgjQ AY3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ApsC/V76AtmxJR1f/OYmi6DVMVs28z9+ywbUb2PrgXo=; b=mKlCwLFupgW3kTwq6++OkcCbUJ66qgbovhivsOMEBzafLPfZ3vS2qJTU1NTrym2cZN H2ttY/7Co58Y1isiaz+Pe0G5IKzHjGItGzc48qKOfSI1bewJwwo8J97EruyOZR3QJ1Xw 0M5IplvDjEoI6Q7xb5Iyl/Vyfd5nvrn5uUPdg3XhpGUP4iQHjfQqtzpUnwZGpNwScaf5 fTbpuTQ7jGQxPWb/BgVbPIa/5QeWC15vCqSsEOBMra4A0hKnG7O9IJOtocOFSuFOS2U2 bbe6MH+uQqNDkrt+V9qsCyPuqGPGNsvqzxwIe7YyuX3qke4QXR8jGPZAqjeKrfjQsymb MACw== X-Gm-Message-State: ANoB5pnwAAq1tZLT9VmVasGuk6E1QzUOsogkW3+t9ILt1vwfO7VObGsd rBMxFKFQKi3JtcpwH2Fa4pvwn1/6W2c= X-Google-Smtp-Source: AA0mqf5MrKHEmra6ouulUajNwyzTyIq2kvAmxTvoNxPhkzNqVQZoL0s+x3bzsQ4iCvxtWvCyvl709w== X-Received: by 2002:adf:ed89:0:b0:242:2115:3fc8 with SMTP id c9-20020adfed89000000b0024221153fc8mr5833829wro.78.1669821806364; Wed, 30 Nov 2022 07:23:26 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 7/9] io_uring: improve rsrc quiesce refs checks Date: Wed, 30 Nov 2022 15:21:57 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Do a little bit of refactoring of io_rsrc_ref_quiesce(), flat the data refs checks and so get rid of a conditional weird unlock-else-break construct. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 133608200769..b36d32534165 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -330,17 +330,14 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, ret = wait_for_completion_interruptible(&data->done); if (!ret) { mutex_lock(&ctx->uring_lock); - if (atomic_read(&data->refs) > 0) { - /* - * it has been revived by another thread while - * we were unlocked - */ - mutex_unlock(&ctx->uring_lock); - } else { + if (atomic_read(&data->refs) <= 0) break; - } + /* + * it has been revived by another thread while + * we were unlocked + */ + mutex_unlock(&ctx->uring_lock); } - reinit: atomic_inc(&data->refs); /* wait for all works potentially completing data->done */ From patchwork Wed Nov 30 15:21:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060027 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFDF2C4708C for ; Wed, 30 Nov 2022 15:23:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229878AbiK3PXc (ORCPT ); Wed, 30 Nov 2022 10:23:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229901AbiK3PXa (ORCPT ); Wed, 30 Nov 2022 10:23:30 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DCCC5EFA6 for ; Wed, 30 Nov 2022 07:23:28 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id h11so20345458wrw.13 for ; Wed, 30 Nov 2022 07:23:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PPQcyF5lmjBVrKqNjL4bPYHAxb/9EgcVHkxUrf7lda0=; b=hvOavoZmHuqCxBchdRPgTfoed/sCVO+B3rz/VscC4No5erA/KnTgg2FUBdFcpYOdwt jA4XUaX3+mqa/2RU+vOnpjzWV0XVegJvHOo0uT+bdx4YKa71ZiDroroQpojcp4PlH8eE G4AS5f0JHYftJlVx4PGrYYLL1WlffWN9Zgp0j6Dq2u+zBTdi/xWeZB51KsEFKUgZGq+3 1qrOdCSbQQq2y5XQLUe+TXmzrT6NzqYyEjTZ5P6NswgmoPYD8IwZGJ6kvsky5EuNBz+u IOUcmpuQTNg8f9GuG/YfmfMQ6leLLfCKpWFgyWoIxi1celJNKYrKw8Eay1avYH+quftH wKTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PPQcyF5lmjBVrKqNjL4bPYHAxb/9EgcVHkxUrf7lda0=; b=KfANJ1TngpkdEkqlKMqNBK0grqOCi8HyBQE0POvyJQMEg5XPprW6993Bh4ede8mp2x hlCCZNFa5fzG7dJRAO6DdVxVIJxbtcl4QhYhfcqVALrIy0r22mxsZfYW1d46qh46C38h uAj1I3LCsYPzcx2MR9FzDks8huoQXcMiM8FruUJRlJf10TPkblgj3lmOyvIC2jmTUfCM 9oqMkVoHv3gTeigC0K3czlmoGbO/NK4Ocg2Cob1XurYj0SV3uR5bFaRs920N31OK48fR EViN+gE8x6xcTNvB4JLciOUGgoFkRrxj0Bnhd9O8dT8gi7mcVop32fI5Opp+ncXQaXEG +Ipg== X-Gm-Message-State: ANoB5pldVfYFarewJ5YJDT8x2nTWFE5/J7014RoDPE1WajD3ePP06YPx Bu1sPeOuQ6pXypTf7Yl1sItL7jRhs/A= X-Google-Smtp-Source: AA0mqf7/zdIoBXnaWrVHkHl5gdhQi5WJr3yYoChkZUMlCg3+0v/46XhExB5z/nRmLInrpiqC2LPWLw== X-Received: by 2002:a5d:544c:0:b0:242:65d:c39a with SMTP id w12-20020a5d544c000000b00242065dc39amr16385825wrv.99.1669821807063; Wed, 30 Nov 2022 07:23:27 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:26 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 8/9] io_uring: don't reinstall quiesce node for each tw Date: Wed, 30 Nov 2022 15:21:58 +0000 Message-Id: <3895d3344164cd9b3a0bbb24a6e357e20a13434b.1669821213.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is no need to reinit data and install a new rsrc node every time we get a task_work, it's detrimental, just execute it and conitnue waiting. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index b36d32534165..d25309400a45 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -309,22 +309,27 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, /* As we may drop ->uring_lock, other task may have started quiesce */ if (data->quiesce) return -ENXIO; + ret = io_rsrc_node_switch_start(ctx); + if (ret) + return ret; + io_rsrc_node_switch(ctx, data); + + /* kill initial ref, already quiesced if zero */ + if (atomic_dec_and_test(&data->refs)) + return 0; data->quiesce = true; + mutex_unlock(&ctx->uring_lock); do { - ret = io_rsrc_node_switch_start(ctx); - if (ret) - break; - io_rsrc_node_switch(ctx, data); - - /* kill initial ref, already quiesced if zero */ - if (atomic_dec_and_test(&data->refs)) - break; - mutex_unlock(&ctx->uring_lock); - ret = io_run_task_work_sig(ctx); - if (ret < 0) - goto reinit; + if (ret < 0) { + atomic_inc(&data->refs); + /* wait for all works potentially completing data->done */ + flush_delayed_work(&ctx->rsrc_put_work); + reinit_completion(&data->done); + mutex_lock(&ctx->uring_lock); + break; + } flush_delayed_work(&ctx->rsrc_put_work); ret = wait_for_completion_interruptible(&data->done); @@ -338,14 +343,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, */ mutex_unlock(&ctx->uring_lock); } -reinit: - atomic_inc(&data->refs); - /* wait for all works potentially completing data->done */ - flush_delayed_work(&ctx->rsrc_put_work); - reinit_completion(&data->done); - - mutex_lock(&ctx->uring_lock); - } while (ret >= 0); + } while (1); data->quiesce = false; return ret; From patchwork Wed Nov 30 15:21:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13060028 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEA3DC352A1 for ; Wed, 30 Nov 2022 15:23:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229890AbiK3PXc (ORCPT ); Wed, 30 Nov 2022 10:23:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229900AbiK3PXa (ORCPT ); Wed, 30 Nov 2022 10:23:30 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 633FE74639 for ; Wed, 30 Nov 2022 07:23:28 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id h11so20345522wrw.13 for ; Wed, 30 Nov 2022 07:23:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xWGLAE/PnAOgYj3lvRGSk5vrEcd8TmdYGVPzTcFigGU=; b=efSlq03EwfP1a84dMz6dT+LRTG1F0v1fY42VZHfyiteHL66yPZbS6Wia5XlZzPE3Ci Jrkf9lHWxWe3b+Cxsg97mHuy5UCOTMNxitLuRHaplV5XE5w3EOQuSPD13KY5ArgRpniT cmUQlAw4k0l5YwGQBiHPISV6kap3j7HqBgDoKq9zVKF0HQu2jc9NdwTIbnJcxIRzjrZX djX/HVRUtIfQCGlfR02G81v92S3r7i+B1VmtNZw+fG4LdikMU9VgHK4OHdDs/JFACNba 6fx3zTIkFSaHC70+QUbjXPfJiieSjbJlb40f1VjfCnv7sEo4T3F39oJoz8fL0xLUknUD yHUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xWGLAE/PnAOgYj3lvRGSk5vrEcd8TmdYGVPzTcFigGU=; b=ebUyRy1XebXuTPGIfdU5nG74zqGj0+awCUqVFKN3UYUubibk3wh7oavsbBXEkzgQ7l qZZlRkePCdG7NQvFOnBKVfMIv9jcrJaO6UasVlhJ7pI8lO/VnAM7TXvLtSIDv8+kZERt TF1QSJfqvX57pWGDMjKB7bfs7GhWmSqLHecgXZhEkolt9acaigBUkgBXCrOwhVZ9/fSX c6LDVuN06F1gj0gO5tyVyF7QobzQP48mSKCnsCtL3HpTe2V/CrBn+2OfEqFlazdbjsJe MSdUWKi2ah4NuGJYJi4p84J/rwmWxfaO3B5Qq5Sh5k2o/90JyGf/nLO1N0Xq5D4nIAdB OfHA== X-Gm-Message-State: ANoB5pl7MZbN79Zedv8dKUwqEUJ3BvYQ7lKQxv+B2tMvSj6FkMfaM+Au al1OnQeGK0JQIuxUyzGveNUQKGaZDx8= X-Google-Smtp-Source: AA0mqf4NYyJeCjts9IGtGbAazCZF+NECqo7GGDUgMO+N3RefIrTmzfOV3sCrBiPuWvJg6thfDUVT7w== X-Received: by 2002:a5d:62c2:0:b0:242:146b:7d00 with SMTP id o2-20020a5d62c2000000b00242146b7d00mr10814885wrv.572.1669821807780; Wed, 30 Nov 2022 07:23:27 -0800 (PST) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:97d]) by smtp.gmail.com with ESMTPSA id v14-20020a05600c444e00b003a1980d55c4sm6381844wmn.47.2022.11.30.07.23.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Nov 2022 07:23:27 -0800 (PST) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 9/9] io_uring: reshuffle issue_flags Date: Wed, 30 Nov 2022 15:21:59 +0000 Message-Id: X-Mailer: git-send-email 2.38.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Reshuffle issue flags to keep normal flags separate from the uring_cmd ctx-setup like flags. Shift the second type to the second byte so it's easier to add new ones in the future. Signed-off-by: Pavel Begunkov --- include/linux/io_uring.h | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 0ded9e271523..29e519752da4 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -9,16 +9,15 @@ enum io_uring_cmd_flags { IO_URING_F_COMPLETE_DEFER = 1, IO_URING_F_UNLOCKED = 2, + /* the request is executed from poll, it should not be freed */ + IO_URING_F_MULTISHOT = 4, /* int's last bit, sign checks are usually faster than a bit test */ IO_URING_F_NONBLOCK = INT_MIN, /* ctx state flags, for URING_CMD */ - IO_URING_F_SQE128 = 4, - IO_URING_F_CQE32 = 8, - IO_URING_F_IOPOLL = 16, - - /* the request is executed from poll, it should not be freed */ - IO_URING_F_MULTISHOT = 32, + IO_URING_F_SQE128 = (1 << 8), + IO_URING_F_CQE32 = (1 << 9), + IO_URING_F_IOPOLL = (1 << 10), }; struct io_uring_cmd {