From patchwork Thu Sep 8 15:56:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26609C54EE9 for ; Thu, 8 Sep 2022 15:59:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229637AbiIHP7D (ORCPT ); Thu, 8 Sep 2022 11:59:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230241AbiIHP7C (ORCPT ); Thu, 8 Sep 2022 11:59:02 -0400 Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B26E2BE4F0 for ; Thu, 8 Sep 2022 08:59:01 -0700 (PDT) Received: by mail-ed1-x533.google.com with SMTP id z21so15031332edi.1 for ; Thu, 08 Sep 2022 08:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=nr75665rJFL8Ua9ObjrNRpWClJ7LBHcrJfr1KI7QgOk=; b=DpswcvcQCna07+V8SeOEGFHTO+V3uX4j8HzOMRGehlaAvtEQyEUKNL0vhiTVxbXdwT lX3p4pSdVmhB8UUr07QT84oZDGKF1B/CYcuT6NsnR+/d/+khpnp6LGOomn0oqE+lLIhF fpYei9bWrBDmCKhnXLnU/kGSYtz4LDWHV/NKzX7sWDSaP896wdNCpaTiAVTtHSPJjcO6 x+TjJfWvcsi+8vcpEH2tD5OtIA+4FiUbMQvC7GJJhCSVh34X4Qz0O5v2BkANOEGZhoLP 2f3L+YenPhYQxAkis0ycYm19InUqZ2yEDtD5AqTi5OROFfH4od0N4E7SWgPXs+V2lrTq QrVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=nr75665rJFL8Ua9ObjrNRpWClJ7LBHcrJfr1KI7QgOk=; b=JUy5Gj2d82zCAh7IyN+X/wxwgOzmG36bvwCn2jS8oB7AYL8rV1cGKttSbuJ18J/Vwl 0SfaDPNaPiACQZLQtV6xwEGNR/eyFvN/hP4HTAyGa4gxMkZLRvbvwWELwsuFe7Qsra/D E47rLuDQ+dUqhHUxVxO2LreIkZLEazhv8kzhNs/fyt2FX7ZexR3Lu99kuRUQZKrc3a+7 OUuaO66Jd38HQ8zvMopCmbGgxeS9BSBXoOy0HNyciU3JiUojLiTDP8zxjYFiAeUe3q3z xvYGeMLKfAymIavIwtF0UxDXSYEl6zIrcm8YzLv4Gm1aPTXhE1Drv4g7qRg8bFMRUNql +qIg== X-Gm-Message-State: ACgBeo0oruIJ1mjW0fMoV/fKB7kwoE39FUHvvrYXU96xn2DcXnI4Xg6u PjJisOVMRDhLuxj/AMlFbg0zPfxb000= X-Google-Smtp-Source: AA6agR5tPKIsTcCVCSEGaNxoRqSQ4p6NG7v2exfVrXx8ub5RkE/AsH6KALf2KYIHeKA1J7Rjv9u75A== X-Received: by 2002:a05:6402:510e:b0:448:9d4b:c760 with SMTP id m14-20020a056402510e00b004489d4bc760mr7831528edd.156.1662652739897; Thu, 08 Sep 2022 08:58:59 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id q26-20020a1709060e5a00b0073872f367cesm1392503eji.112.2022.09.08.08.58.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 08:58:59 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Dylan Yudaken Subject: [PATCH 1/6] io_uring: further limit non-owner defer-tw cq waiting Date: Thu, 8 Sep 2022 16:56:52 +0100 Message-Id: <94c83c0a7fe468260ee2ec31bdb0095d6e874ba2.1662652536.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org In case of DEFER_TASK_WORK we try to restrict waiters to only one task, which is also the only submitter; however, we don't do it reliably, which might be very confusing and backfire in the future. E.g. we currently allow multiple tasks in io_iopoll_check(). Fixes: dacbb30102689 ("io_uring: add IORING_SETUP_DEFER_TASKRUN") Signed-off-by: Pavel Begunkov Reviewed-by: Dylan Yudaken --- io_uring/io_uring.c | 6 ++++++ io_uring/io_uring.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0482087b7c64..dc6f64ecd926 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1398,6 +1398,9 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) int ret = 0; unsigned long check_cq; + if (!io_allowed_run_tw(ctx)) + return -EEXIST; + check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) @@ -2386,6 +2389,9 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, ktime_t timeout = KTIME_MAX; int ret; + if (!io_allowed_run_tw(ctx)) + return -EEXIST; + do { /* always run at least 1 task work to process local work */ ret = io_run_task_work_ctx(ctx); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9d89425292b7..4eea0836170e 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -329,4 +329,15 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) return container_of(node, struct io_kiocb, comp_list); } +static inline bool io_allowed_run_tw(struct io_ring_ctx *ctx) +{ + if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) + return true; + if (unlikely(ctx->submitter_task != current)) { + /* maybe this is before any submissions */ + return !ctx->submitter_task; + } + return true; +} + #endif From patchwork Thu Sep 8 15:56:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91DE4C38145 for ; Thu, 8 Sep 2022 15:59:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229504AbiIHP7E (ORCPT ); Thu, 8 Sep 2022 11:59:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230241AbiIHP7D (ORCPT ); Thu, 8 Sep 2022 11:59:03 -0400 Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7237C9CCCC for ; Thu, 8 Sep 2022 08:59:02 -0700 (PDT) Received: by mail-ed1-x52d.google.com with SMTP id e18so25067163edj.3 for ; Thu, 08 Sep 2022 08:59:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=rNeqALpVAO/+TKzL6QzySY7His6uDpVJ/a3IOMvrhUM=; b=AyJxHXZx83BgPSTWK3GKzIaFp2W+0NGspXoBGvyw6T/q1LBFOtIaw+bdpWSjxiWjVQ UfRukVeuvSyJcFoAqkCF28mc5Aj7/E7/728L5xw7zqR1Yx2JsIX/yYG1gBAMMD3eA/y/ uNEejpOsbLnB+DikMd0hQxYm3A7bQQ5SHhvDA7tjLwXP9OMvtkUQNv0ETW1aMkvjnX3L N/O+Mv+zA2P+s+rZZxl/RUN4dRWw0CU1rrEvnNaPMc/5vZ9LoYVOyGdZV59xqS0kpEvv TIXpJCrRZxs3V2sDja0foO21DHKtiJ26NUw+S+KhjlZC9qfeac1VUMlO8KhnvrF0SWjM D2Vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=rNeqALpVAO/+TKzL6QzySY7His6uDpVJ/a3IOMvrhUM=; b=XyUwwdIZWRiMWlofP05eL08vG+IldzCtl0im/e4eG1LSfUP1pWTtq0VbaP6s4KxPqG m07IQYJmDz6VenB4w2MR+VxwtTrRhZABxJx9yBxmK3I4Gx/ZpzkSeXPAbueruro5ehku qx+9Hv2ouo4vbjGmZeHG08SXMehZJhnrsQb2SbeB5cMtwZJvb5sEr+RzWQ+VVn36yQuT yd4W0MBkNBcOHPYpjLp21C5vHBboJWgw6qSK+k2HqRH/X5/KtvS4Dq7GJuU57lNWnsD3 TK/+s7Z66/Ic93izWOMBLORl8maavxRokrbeJlWUjhz+PR+W5cYIJ+sw1q9argGQgzuB PonA== X-Gm-Message-State: ACgBeo0fd+qXNdBBJa/2wkPn7GgpujeXPUZA34oH2CVr44wEiXRJxY8u 5pOy007RdkIQFjW9Pq8WynB5aw6TMHQ= X-Google-Smtp-Source: AA6agR7OFURyitmmo/k6yrCq72/O4j2DtjzIaBj7k51x0Ydc3s1R0/VQUz0lAa6w3nlIr5a4BZmPVg== X-Received: by 2002:aa7:c458:0:b0:44e:9078:5712 with SMTP id n24-20020aa7c458000000b0044e90785712mr8012745edr.25.1662652740765; Thu, 08 Sep 2022 08:59:00 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id q26-20020a1709060e5a00b0073872f367cesm1392503eji.112.2022.09.08.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 08:59:00 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Dylan Yudaken Subject: [PATCH 2/6] io_uring: disallow defer-tw run w/ no submitters Date: Thu, 8 Sep 2022 16:56:53 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We try to restrict CQ waiters when IORING_SETUP_DEFER_TASKRUN is set, but if nothing has been submitted yet it'll allow any waiter, which violates the contract. Fixes: dacbb30102689 ("io_uring: add IORING_SETUP_DEFER_TASKRUN") Signed-off-by: Pavel Begunkov Reviewed-by: Dylan Yudaken --- io_uring/io_uring.c | 7 +------ io_uring/io_uring.h | 9 ++------- 2 files changed, 3 insertions(+), 13 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index dc6f64ecd926..7f60d384e917 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1169,13 +1169,8 @@ int __io_run_local_work(struct io_ring_ctx *ctx, bool locked) int ret; unsigned int loops = 1; - if (unlikely(ctx->submitter_task != current)) { - /* maybe this is before any submissions */ - if (!ctx->submitter_task) - return 0; - + if (unlikely(ctx->submitter_task != current)) return -EEXIST; - } node = io_llist_xchg(&ctx->work_llist, &fake); ret = 0; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 4eea0836170e..d38173b9ac19 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -331,13 +331,8 @@ static inline struct io_kiocb *io_alloc_req(struct io_ring_ctx *ctx) static inline bool io_allowed_run_tw(struct io_ring_ctx *ctx) { - if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN)) - return true; - if (unlikely(ctx->submitter_task != current)) { - /* maybe this is before any submissions */ - return !ctx->submitter_task; - } - return true; + return likely(!(ctx->flags & IORING_SETUP_DEFER_TASKRUN) || + ctx->submitter_task == current); } #endif From patchwork Thu Sep 8 15:56:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0111C54EE9 for ; Thu, 8 Sep 2022 15:59:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230241AbiIHP7F (ORCPT ); Thu, 8 Sep 2022 11:59:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229795AbiIHP7E (ORCPT ); Thu, 8 Sep 2022 11:59:04 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 847A6E55BB for ; Thu, 8 Sep 2022 08:59:03 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id gb36so39048591ejc.10 for ; Thu, 08 Sep 2022 08:59:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=mgVWDGO+sTjtFaqfFa3pHvlo4gSFJ+pkOC743IR4S0A=; b=o7kEKh3Li7g6nSerKYuR02tKSfmvGHmpR+zhctJKMF+gz4vIlhsI8XKt4Lpr/sTzjF Oy70Jy1q225JaceK87UHBA0A7kYtxu2og2GAEVUwW/IPoTg2pyDVUNg920aoL06xrIZm 3AWB8FcHu8wZEw8AcD2miN6xIQSF+vtcxUnF8yijBTr1NNjwxkrAto/t7+09ttS1S+Uh D6lAQw+joiJji/BFAaoG2T0C2UuNVO/DUQNLtMi5ZGmTGAgMVjklL+f5T5p+PBncShyo hJzXrrkim2Qyfi2rbnaFVv4qsLBj9NRofVl7kcDkT1EDO0iVjiOXgMi8yAxo71NPi3d9 SmFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=mgVWDGO+sTjtFaqfFa3pHvlo4gSFJ+pkOC743IR4S0A=; b=hLFLZA0UNLWUBerWvIaeiHVeGylAiC1rY/J7L9PSHSasS5+wGbiz4slBOmVIOauLQj X7OuoiUvSIhV2/C4ESymbjgBIBsdayWlvzsdCx5drpe8+V8tW7tDyqab+BRw3YCqfKaN bxTphRILgZ5v5dfaPQqeIcSsuZZL8zGH9rajCfyI4VY2QCR21GAr37+x9zXT1x3oiAWR knEXubUtMmmTVe1Lb01T51tm1dgyU5foo1yH3kv+jKrAFmrBAPNMWH1bbmPHkwh6zeZk zd4ZEJSLk3jg02dWrApNGl1FlsMUdahg2e8HsYbzBCtSdU8cVPsjCCJUdhR2ERzd9yTG yhQA== X-Gm-Message-State: ACgBeo15pgLPhQ6UsapqtOCRB3/FPgsekehv9JYGgtZ1erUFDhOCc2Eq PPa1ag9vEvPO7VvNVpuHnfeRIoznaL4= X-Google-Smtp-Source: AA6agR4hvOUpK/+fgFlm5oGZZHq/OJDujT2FVrFelWJUSMnIoFhclgtv3XFyOgDweLrQ+NwrzOuOsw== X-Received: by 2002:a17:906:730d:b0:73d:c8a1:a6ae with SMTP id di13-20020a170906730d00b0073dc8a1a6aemr6694199ejc.540.1662652741567; Thu, 08 Sep 2022 08:59:01 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id q26-20020a1709060e5a00b0073872f367cesm1392503eji.112.2022.09.08.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 08:59:01 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Dylan Yudaken Subject: [PATCH 3/6] io_uring/iopoll: fix unexpected returns Date: Thu, 8 Sep 2022 16:56:54 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We may propagate a positive return value of io_run_task_work() out of io_iopoll_check(), which breaks our tests. io_run_task_work() doesn't return anything useful for us, ignore the return value. Fixes: dacbb30102689 ("io_uring: add IORING_SETUP_DEFER_TASKRUN") Signed-off-by: Pavel Begunkov Reviewed-by: Dylan Yudaken --- io_uring/io_uring.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 7f60d384e917..8233a375e8c9 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1435,12 +1435,9 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) u32 tail = ctx->cached_cq_tail; mutex_unlock(&ctx->uring_lock); - ret = io_run_task_work(); + io_run_task_work(); mutex_lock(&ctx->uring_lock); - if (ret < 0) - break; - /* some requests don't go through iopoll_list */ if (tail != ctx->cached_cq_tail || wq_list_empty(&ctx->iopoll_list)) From patchwork Thu Sep 8 15:56:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C303C6FA86 for ; Thu, 8 Sep 2022 15:59:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229795AbiIHP7G (ORCPT ); Thu, 8 Sep 2022 11:59:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230168AbiIHP7F (ORCPT ); Thu, 8 Sep 2022 11:59:05 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B490F7546 for ; Thu, 8 Sep 2022 08:59:04 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id lz22so18247624ejb.3 for ; Thu, 08 Sep 2022 08:59:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=djYdNzXWZEtmnQhM1MYMHWws4q0FI7BUKIiZyuQPvkA=; b=g1Xq2d+q8JLBTZIE0j2J4dUEvqy5lkzYuwyNG9QlUJlIcQm3tsIekgWl8aimIcLrqs AA1HZ4PHe+7cEh8ZCnGb35fBu+aBgrJN7LO/ev1+aWJ3YogpQe8zSb0+pnu5LyK6e5Ki INypVmeupLt/2P0JtNP1LZz6yQ8ZjbaiCx5WROi3iyoouRG/wBBPJ1brXo3w9SvtBwGv H+D9xe7l1TcRBv6pA41hnyH7lc52M/Ec3c6y5oODQ4uF4Zi4ySKESOG3EzQKTEfbWgwA j4bE+MtfnkxG9CwqZn+8tMXyQkiJN9XOIERhlrggxCcOFL1xPMpwCRbAHtJzKVLur3+O NWkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=djYdNzXWZEtmnQhM1MYMHWws4q0FI7BUKIiZyuQPvkA=; b=TUs5gIxXPPOzNX/Dac0hcex8aLWNp6grMucJ96oTy9ZqypP8iXtjXraLLXs6nfB9g/ u8ZmZ8KmNwsqyZfV9ObG+/KVoTaU+Pwtwo+bgaDr7vNyaeELuGZkk5P7PEw1awPn+TtV sqLWklnU5w9PjFhIkFoWzR/Wps4UGpXycQMa9B73h4aWTfN5OyuPdz6+C/jXtX7mWVGz Fdq82/ln+SpXeOt4/PNfoEHCwY+R2Vr7XOEPMu5K/B36lggZ/hpBfq/Byu8AJZZOOHID YtYhgKdrdSddmIFnVb/lOMRoq6FSgN7KF5qdP3COzIUpHthXrpALQtYo/CdmetChLsiM 2+9g== X-Gm-Message-State: ACgBeo2cey6zI36ZcAaXbRrWgfclhDa9kfx6WR/FlWiz1jY2J70aztMw 16LbD2ePXCQmfEADng+GHQ5gTiaSIDY= X-Google-Smtp-Source: AA6agR6MFe7vTAFz0crNTi8xGBvLnqUvC2hRwUdm/PzYzUHVLFs91moi84B089u7fQ3F4cwJ6MTSuA== X-Received: by 2002:a17:906:ef90:b0:730:9d18:17b3 with SMTP id ze16-20020a170906ef9000b007309d1817b3mr6810418ejb.141.1662652742579; Thu, 08 Sep 2022 08:59:02 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id q26-20020a1709060e5a00b0073872f367cesm1392503eji.112.2022.09.08.08.59.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 08:59:02 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Dylan Yudaken Subject: [PATCH 4/6] io_uring/iopoll: unify tw breaking logic Date: Thu, 8 Sep 2022 16:56:55 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Let's keep checks for whether to break the iopoll loop or not same for normal and defer tw, this includes ->cached_cq_tail checks guarding against polling more than asked for. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8233a375e8c9..2fb5f1e78fb2 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1428,21 +1428,21 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) */ if (wq_list_empty(&ctx->iopoll_list) || io_task_work_pending(ctx)) { + u32 tail = ctx->cached_cq_tail; + if (!llist_empty(&ctx->work_llist)) __io_run_local_work(ctx, true); + if (task_work_pending(current) || wq_list_empty(&ctx->iopoll_list)) { - u32 tail = ctx->cached_cq_tail; - mutex_unlock(&ctx->uring_lock); io_run_task_work(); mutex_lock(&ctx->uring_lock); - - /* some requests don't go through iopoll_list */ - if (tail != ctx->cached_cq_tail || - wq_list_empty(&ctx->iopoll_list)) - break; } + /* some requests don't go through iopoll_list */ + if (tail != ctx->cached_cq_tail || + wq_list_empty(&ctx->iopoll_list)) + break; } ret = io_do_iopoll(ctx, !min); if (ret < 0) From patchwork Thu Sep 8 15:56:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E47C9C6FA86 for ; Thu, 8 Sep 2022 15:59:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231237AbiIHP7J (ORCPT ); Thu, 8 Sep 2022 11:59:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230407AbiIHP7G (ORCPT ); Thu, 8 Sep 2022 11:59:06 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68F7DF827F for ; Thu, 8 Sep 2022 08:59:05 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id l14so15622663eja.7 for ; Thu, 08 Sep 2022 08:59:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=v9r22If6h69B1ae0T4+KgnzjvLRV9orbZ+tGlc/KePA=; b=liAwAX8RYVmIYFl6pJUImMIE+VXAw+4dzJsTJf1gNfbORPKRWuIBeWkBPO+55D7cMh kBJ8q5YwVjJC0al34icZSRvxrh2xFkBn6KLCUl2W3T32T2ncdJA/I1bzUWGq8sQ1azi0 hVe+ablw54+cytgjz6vinvq/TBu9ggpGdeFUJyJybQjB+K9u1XsYRmDTTLJjvJx7ZQTF i6cGG8p0lfPWVAxudV+ySR7GN/2p5ledFsg6yDuwJnDFHnzn5qeo6V2dicp5KU3ALuee rcHRXTUN9zi93X1R+tFZb90mORhMQZOPodtDKpLkyVALCWizipH38LtpnID8gGz3mQaI h2nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=v9r22If6h69B1ae0T4+KgnzjvLRV9orbZ+tGlc/KePA=; b=RqKmOecGsqkEs0mLbWQNJiZ/TsSDU/yQX5wFenkkr+wL6PBjIjr3x6EriL/GC+Mpyg G9Wcof/Imh2b/dtj35AU67NEjfHRur+whnOhsfnNPC3j6b3n5LzTwNqm0UGj527YptHv vsG5ab73w+QHuop80m/ocynGq65ESm/8zivyHvzSKDXA/MYOuAKbHOOd7dhVBI1wHjZs xXuQnQvB+1iHcBgzK5tPMATB/NI/1hvCjO3M9XGSrsaDuTTn5dIsv+pUMraP3C5/ovqJ uW+O5ZnZ3OVTSccvo5v5f8ca45Ue4oOak2PaiOsGuGCi8p0VAZ9Inior7iz0cxLndOlI HHQg== X-Gm-Message-State: ACgBeo28dt8eE+nomE5Zs55wKzAa81nI0thKSELSLxqoKbONgu7Rwk1R cKy3EBP4ZLaIY7GQfdpLq95lry+gxRI= X-Google-Smtp-Source: AA6agR5p1ZvUb4chugFj7qMOaj7gPWy6tcbc/XUwuvKD+NYoAkJzE/QVCXD6LhxTNuUx86ndtArrhw== X-Received: by 2002:a17:907:94c2:b0:73d:c534:1ac0 with SMTP id dn2-20020a17090794c200b0073dc5341ac0mr6598776ejc.461.1662652743423; Thu, 08 Sep 2022 08:59:03 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id q26-20020a1709060e5a00b0073872f367cesm1392503eji.112.2022.09.08.08.59.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 08:59:03 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Dylan Yudaken Subject: [PATCH 5/6] io_uring: add fast path for io_run_local_work() Date: Thu, 8 Sep 2022 16:56:56 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We'll grab uring_lock and call __io_run_local_work() with several atomics inside even if there are no task works. Skip it if ->work_llist is empty. Signed-off-by: Pavel Begunkov Reviewed-by: Dylan Yudaken --- io_uring/io_uring.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2fb5f1e78fb2..85b795e4dc6a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1208,6 +1208,9 @@ int io_run_local_work(struct io_ring_ctx *ctx) bool locked; int ret; + if (llist_empty(&ctx->work_llist)) + return 0; + locked = mutex_trylock(&ctx->uring_lock); ret = __io_run_local_work(ctx, locked); if (locked) From patchwork Thu Sep 8 15:56:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12970295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CB07C38145 for ; Thu, 8 Sep 2022 15:59:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230168AbiIHP7H (ORCPT ); Thu, 8 Sep 2022 11:59:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230237AbiIHP7G (ORCPT ); Thu, 8 Sep 2022 11:59:06 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F563E55BB for ; Thu, 8 Sep 2022 08:59:05 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id y3so39153703ejc.1 for ; Thu, 08 Sep 2022 08:59:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=YPoIhoDFX2A1i6MxaTOuMCYaYD7lnGFKcVCbZxdaug4=; b=dtJwG2IQRnGMTPvQFa+5b18Cwss9mEuL70A5ZmbOd6G+23rmBpygtGPIS0dQpcsidY G3Zzi4YOiI1fttMcCBbNeHSS7ilk3WRMP/LpWlWOlh2+/gt8GneBK/XxmbivUjrOXmzm kobncWUSA+uUfAvNElyL9SDzYdkukNYoqXI6GDhLEhzKpVrUeIDsp4kkPs8Qv58lgR7U m8G90Deh4+u1N7tSwdRMD5eJDOj4RyUPbBApb8XtM8tt65gYfiV279CD6EKZLwqELwql 52iEH3WUEHnFJOQocr3fGrvU70HWd0CAWsCwU6gZQ5cFz7p92WHVE61TrAMv19U/8GDM WT2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=YPoIhoDFX2A1i6MxaTOuMCYaYD7lnGFKcVCbZxdaug4=; b=7KFIYnsFv73qi13fT5TfpAkoa8+IYmj2DB8CbOxHGDCYPZbZP7MIDCCooszJqnxIjE m7MTIfp/8m+r6FU/BaMaO6pbjh9ntqi2uA5S+UZS23PTsZaBV6PoguQX8a34d1AApOmY yOzFIUBml2wO1L2YqUaE2yGZkXQ7C9OUh2OVHiAR+D89QLgzvJo7cZU8kVgPqPDmXgrT zLVmR8C40pXxNpJ6bI/HUxbjijb+E32QTLexTmO1lqvAHz5JOTw53rwAo5bDcGzYxCVF JipSd4KZxhygR+bR32YNIJFxauBxLF7WZ0TNRePiYcsYqs0HUfk+JPd+yrv7PVfM4U4S q2Fg== X-Gm-Message-State: ACgBeo1Vr9AQW1OI4WJdCCu3MqPA36Lu4TMTIXxiJPG2w3PAuGGuWXfW Q5x6AcBJm1GzFNCLtT0Czz0d/WHGX68= X-Google-Smtp-Source: AA6agR7ERR9NXv2XvpwnkNKFDrHd2dimxQzRVLUe5OFnXP1JyVV4gbuZYX+2w2WE6kbfLph0BAxOpA== X-Received: by 2002:a17:906:99c5:b0:73d:70c5:1a4f with SMTP id s5-20020a17090699c500b0073d70c51a4fmr6383526ejn.302.1662652744438; Thu, 08 Sep 2022 08:59:04 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:cfb9]) by smtp.gmail.com with ESMTPSA id q26-20020a1709060e5a00b0073872f367cesm1392503eji.112.2022.09.08.08.59.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 08 Sep 2022 08:59:04 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Dylan Yudaken Subject: [PATCH 6/6] io_uring: remove unused return from io_disarm_next Date: Thu, 8 Sep 2022 16:56:57 +0100 Message-Id: <9a441c9a32a58bcc586076fa9a7d0dc33f1fb3cb.1662652536.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We removed conditional io_commit_cqring_flush() guarding against spurious eventfd and the io_disarm_next()'s return value is not used anymore, just void it. Signed-off-by: Pavel Begunkov --- io_uring/timeout.c | 13 +++---------- io_uring/timeout.h | 2 +- 2 files changed, 4 insertions(+), 11 deletions(-) diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 78ea2c64b70e..e8a8c2099480 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -149,11 +149,10 @@ static inline void io_remove_next_linked(struct io_kiocb *req) nxt->link = NULL; } -bool io_disarm_next(struct io_kiocb *req) +void io_disarm_next(struct io_kiocb *req) __must_hold(&req->ctx->completion_lock) { struct io_kiocb *link = NULL; - bool posted = false; if (req->flags & REQ_F_ARM_LTIMEOUT) { link = req->link; @@ -161,7 +160,6 @@ bool io_disarm_next(struct io_kiocb *req) if (link && link->opcode == IORING_OP_LINK_TIMEOUT) { io_remove_next_linked(req); io_req_tw_post_queue(link, -ECANCELED, 0); - posted = true; } } else if (req->flags & REQ_F_LINK_TIMEOUT) { struct io_ring_ctx *ctx = req->ctx; @@ -169,17 +167,12 @@ bool io_disarm_next(struct io_kiocb *req) spin_lock_irq(&ctx->timeout_lock); link = io_disarm_linked_timeout(req); spin_unlock_irq(&ctx->timeout_lock); - if (link) { - posted = true; + if (link) io_req_tw_post_queue(link, -ECANCELED, 0); - } } if (unlikely((req->flags & REQ_F_FAIL) && - !(req->flags & REQ_F_HARDLINK))) { - posted |= (req->link != NULL); + !(req->flags & REQ_F_HARDLINK))) io_fail_links(req); - } - return posted; } struct io_kiocb *__io_disarm_linked_timeout(struct io_kiocb *req, diff --git a/io_uring/timeout.h b/io_uring/timeout.h index 858c62644897..a6939f18313e 100644 --- a/io_uring/timeout.h +++ b/io_uring/timeout.h @@ -27,7 +27,7 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd); __cold bool io_kill_timeouts(struct io_ring_ctx *ctx, struct task_struct *tsk, bool cancel_all); void io_queue_linked_timeout(struct io_kiocb *req); -bool io_disarm_next(struct io_kiocb *req); +void io_disarm_next(struct io_kiocb *req); int io_timeout_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_link_timeout_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);