From patchwork Tue Feb 6 16:24:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547585 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A3872132C0B for ; Tue, 6 Feb 2024 16:27:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236853; cv=none; b=efgZIsNeIw/lQ3lIHX4FMLEQtbYFxF9KXU3dGMPVH8VfU+1Pok7JUEHQpdOHKy/GHLiJn7MfRTtWQg5ffg02R9Q62qKqQ9FUGkVPSIomjGnfCYf6dnPm6WyzXZf2DRQB0scMF5/OEKqHxOknRRVN2F/gHPuf1f+BZGEzp17j/J8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236853; c=relaxed/simple; bh=y23njzJyZvBNI+yR2UGEyk7+lERQN57YsvIg+McJRFA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i+kCWdsltc1gxCLT/YsxtHSgFVqVD/M4oo4LqEnZCtmnfgBaUGhDDp9WA4tXITeLOaog5usOrRA9xZnxMfRs7U2eqIvuOQOgk7e+lDmiZpUmPaVFUi0anAjSFkdqjXDLKrk1p3PgO8hqotgjSGY2m1wsggxv+y+LbT4Ig0ZDBJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=mJvEukce; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="mJvEukce" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso74149539f.0 for ; Tue, 06 Feb 2024 08:27:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236850; x=1707841650; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VmuhgQ1hV2C6uVdY7mANEs6y5wJeAVYVu12IM+UxdUk=; b=mJvEukcet9B8vcgVuy2UBUfpX96Ex24RJfBOL5goI7KrfnQLfyTzXKk7hBMUYUdvaa clwR3a73iD2d/PZ9Qy/6vn7nQBLG+q4ZLa9AI6ATzLYbxKO2PK8YoZpTvwGoJhpXMyty uJJA4TyUoza52nVCmd15qak63fgKqfcnsA1xQxmwVCIwm+1gUCUuaLZU91tWx/SYswTB 7ZUTrwJcLjZGsa9afQPCH753pNVoESIi+kEmmhhYL/EfXt7kPEi6XTy8VrEjnL0BQ4DP dBgePI8xLFhQEBb0WanfGRe9SwBNl05bCQBRuABYAr2gpXCrbI+PbB9YZnD7UHi0LndY kOFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236850; x=1707841650; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VmuhgQ1hV2C6uVdY7mANEs6y5wJeAVYVu12IM+UxdUk=; b=NNuaYdOZEkC0fUEMGCfT2KKnyvMgw2D+pa0w0gDXeyVX4pNZYat1FSOuW3fa8+zVy9 oiSetprsa6pvBgvDeU8XzJxlSjDnyZFcqTLEqqqMOiXplWuXlSaK6TuiUbXfeJY5+jtL bJxakR006C7PK/xi7IpAUcJ6Z13pRzvyXkOhIL48PTBgyjDbMkaIF07K4WcMZatD1Qz4 Lqi5uQqOwDNYvRCak64Jwzqsl6V6IRzHHFNRx05uuTLi8SZKoFOi1eT0Oo87jZcLub90 I1IDP6nlbL9xPEvaFpVnhsnFS1MUk+LCedYh8vDqCTFgTYUmrntb3O/idtjpDC0jvhk2 VmVw== X-Gm-Message-State: AOJu0YyqLD7pjSOon1mzAONdf16vsV+Y/Vu5YwQ0Hp9K+7vC0MJI960d WzEH0jxap4KANu6xfOMT9Tlal3D98cAPHS5exSvjhAfd7p61IoeqPB0W9CIM64jBxOEBLHVbcGH YyBM= X-Google-Smtp-Source: AGHT+IH8tRrmHSIMC7s+zxgNFfNVnkChYYW5peZHy6Ue5AQBF4pWwdQ/cH5BqA7p3sXXR6VhPek0ug== X-Received: by 2002:a5e:990c:0:b0:7c3:f75f:7b12 with SMTP id t12-20020a5e990c000000b007c3f75f7b12mr858400ioj.0.1707236850347; Tue, 06 Feb 2024 08:27:30 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:28 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/10] io_uring/kbuf: cleanup passing back cflags Date: Tue, 6 Feb 2024 09:24:35 -0700 Message-ID: <20240206162726.644202-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We have various functions calculating the CQE cflags we need to pass back, but it's all the same everywhere. Make a number of the putting functions void, and just have the two main helps for this, io_put_kbuf() and io_put_kbuf_comp() calculate the actual mask and pass it back. While at it, cleanup how we put REQ_F_BUFFER_RING buffers. Before this change, we would call into __io_put_kbuf() only to go right back in to the header defined functions. As clearing this type of buffer is just re-assigning the buf_index and incrementing the head, this is very wasteful. Signed-off-by: Jens Axboe --- io_uring/kbuf.c | 14 ++++---------- io_uring/kbuf.h | 41 +++++++++++++++++++++++++++-------------- 2 files changed, 31 insertions(+), 24 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 71880615bb78..ee866d646997 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -102,10 +102,8 @@ bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags) return true; } -unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) +void __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) { - unsigned int cflags; - /* * We can add this buffer back to two lists: * @@ -118,21 +116,17 @@ unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) * We migrate buffers from the comp_list to the issue cache list * when we need one. */ - if (req->flags & REQ_F_BUFFER_RING) { - /* no buffers to recycle for this case */ - cflags = __io_put_kbuf_list(req, NULL); - } else if (issue_flags & IO_URING_F_UNLOCKED) { + if (issue_flags & IO_URING_F_UNLOCKED) { struct io_ring_ctx *ctx = req->ctx; spin_lock(&ctx->completion_lock); - cflags = __io_put_kbuf_list(req, &ctx->io_buffers_comp); + __io_put_kbuf_list(req, &ctx->io_buffers_comp); spin_unlock(&ctx->completion_lock); } else { lockdep_assert_held(&req->ctx->uring_lock); - cflags = __io_put_kbuf_list(req, &req->ctx->io_buffers_cache); + __io_put_kbuf_list(req, &req->ctx->io_buffers_cache); } - return cflags; } static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 53dfaa71a397..f74c910b83f4 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -57,7 +57,7 @@ int io_register_pbuf_status(struct io_ring_ctx *ctx, void __user *arg); void io_kbuf_mmap_list_free(struct io_ring_ctx *ctx); -unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags); +void __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags); bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags); @@ -108,41 +108,54 @@ static inline bool io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags) return false; } -static inline unsigned int __io_put_kbuf_list(struct io_kiocb *req, - struct list_head *list) +static inline void __io_put_kbuf_ring(struct io_kiocb *req) { - unsigned int ret = IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); + if (req->buf_list) { + req->buf_index = req->buf_list->bgid; + req->buf_list->head++; + } + req->flags &= ~REQ_F_BUFFER_RING; +} +static inline void __io_put_kbuf_list(struct io_kiocb *req, + struct list_head *list) +{ if (req->flags & REQ_F_BUFFER_RING) { - if (req->buf_list) { - req->buf_index = req->buf_list->bgid; - req->buf_list->head++; - } - req->flags &= ~REQ_F_BUFFER_RING; + __io_put_kbuf_ring(req); } else { req->buf_index = req->kbuf->bgid; list_add(&req->kbuf->list, list); req->flags &= ~REQ_F_BUFFER_SELECTED; } - - return ret; } static inline unsigned int io_put_kbuf_comp(struct io_kiocb *req) { + unsigned int ret; + lockdep_assert_held(&req->ctx->completion_lock); if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return 0; - return __io_put_kbuf_list(req, &req->ctx->io_buffers_comp); + + ret = IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); + __io_put_kbuf_list(req, &req->ctx->io_buffers_comp); + return ret; } static inline unsigned int io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) { + unsigned int ret; - if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) + if (!(req->flags & (REQ_F_BUFFER_RING | REQ_F_BUFFER_SELECTED))) return 0; - return __io_put_kbuf(req, issue_flags); + + ret = IORING_CQE_F_BUFFER | (req->buf_index << IORING_CQE_BUFFER_SHIFT); + if (req->flags & REQ_F_BUFFER_RING) + __io_put_kbuf_ring(req); + else + __io_put_kbuf(req, issue_flags); + return ret; } #endif From patchwork Tue Feb 6 16:24:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547586 Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96994134751 for ; Tue, 6 Feb 2024 16:27:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236855; cv=none; b=JuWN3oduIiHMPey88cOOZHUS7I7o1e+ZSVQHsSpHi/OFLKuALrNjx7Hp/l0hWsJzTswEUT1NBV53A4iBSKz6QuqSusXSl35btvBENDti3j/5wMwNeLY1+0BfR2htT9gwPxXIBo1XUVAX5nZaPs9ey3uraglh2h7nBwzCN213/v4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236855; c=relaxed/simple; bh=llxWWIJLMUC6G4gnBEi54TwdikqeX/a5DdLpTgHz0YU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oWdpWf59K24nIunsvjt+yMOaY8u6RdM+9Dd6LW5hzPsqE5YpeE28kZvCPg+AQAA3pnhNNCLjtPQkQen42x648PTokKvnaA51atgz3gG1/YMmKkbEz+nATTFPUgvmdAoijdEjKFwDHJ9eZXPzUdc1cB0K1t4Q9AeByYpO/Cc7Z1E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=hqr7BZsj; arc=none smtp.client-ip=209.85.166.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="hqr7BZsj" Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso74150339f.0 for ; Tue, 06 Feb 2024 08:27:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236852; x=1707841652; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eLH6DE53yx0jld1mygpcddfhbdnTIuRzvYOhl0FgAIY=; b=hqr7BZsjeSj8cGAipE2CCV7C12vJiRk4+BghQC0+BbeMXpaGIaV+Re3xMaWogwjCje iCVAAAx9teJDBojVu9auaOTVA0C7cC1u00azu562LUsvZ/347G6OBy1tMJfiKilJ/x4Q II5MMHNuFsqqovfm7JZawqihMqSqUiIsnqmgHpe4rZ3iKoNwguGY1iVvYZU7O6GDxREy YSWt8DWfeqQj3s1LMvOd/y6rSW3sy40leaFXCnAJNRGbiONdV5H6c1I5iqvGOB4nPCTG sCCGtY9JXP8hu9P9/UAdedZ7meYA/uQC1qo6ShnlBdGqoYFzP6S0qiGeZO2KQEamul2b C0kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236852; x=1707841652; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eLH6DE53yx0jld1mygpcddfhbdnTIuRzvYOhl0FgAIY=; b=CjDl1AMHTfCN1L2a3jmvvGBgWSHVuoVngrlwwN850fW/Vpe5N7u21F2vyKWd3GfI34 5WDFXAzSV2lNyFIwya1jiG3frbsDeTR98epYR2AWb4tlSRY4ISwIGALTLfA7Q7X1AU1Z PsduIoTUkEScHPBHo7dXWwu6wVhLLPHoLtjMUoBynJSak6tSXq3eOOvaHfIIzIjCgku3 FxSh3F7e0/KyrLux6/WMjlT8JgxHTRwPXA0IqMkO3eoe0kYVh3MoLucMCaJNc+7ouzfA AQKnXZj/Y++6mxO3vaMNKxTCDz6RWlITPEuyRTdu4qnlJCIjdTeSBLso75BUKoI32Jgq hVvg== X-Gm-Message-State: AOJu0YyxXwp8zYdwn/96AdibxGPKFUBmPolYR9h2jWpFL0WiwIXhwV/g O0Di7IAYgTduqXKtS/9+R3PS9WDISN9GAv3YIF0hMSRwiV4ygzzJ4qk3xAcc6je7XCdOc+fq7yv cvA8= X-Google-Smtp-Source: AGHT+IGHwBwIIYownnFD9Jplqr3dXvtds83VT0eBHZNWF5JRgu3hsJGpj0vE+rDB6SQyYicUbFAB+w== X-Received: by 2002:a6b:670e:0:b0:7c3:eda5:f41a with SMTP id b14-20020a6b670e000000b007c3eda5f41amr2935780ioc.1.1707236852169; Tue, 06 Feb 2024 08:27:32 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:30 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 02/10] io_uring: remove looping around handling traditional task_work Date: Tue, 6 Feb 2024 09:24:36 -0700 Message-ID: <20240206162726.644202-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 A previous commit added looping around handling traditional task_work as an optimization, and while that may seem like a good idea, it's also possible to run into application starvation doing so. If the task_work generation is bursty, we can get very deep task_work queues, and we can end up looping in here for a very long time. One immediately observable problem with that is handling network traffic using provided buffers, where flooding incoming traffic and looping task_work handling will very quickly lead to buffer starvation as we keep running task_work rather than returning to the application so it can handle the associated CQEs and also provide buffers back. Fixes: 3a0c037b0e16 ("io_uring: batch task_work") Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 45 +++++++-------------------------------------- 1 file changed, 7 insertions(+), 38 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9b499864f10d..ae5b38355864 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1175,12 +1175,11 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) static unsigned int handle_tw_list(struct llist_node *node, struct io_ring_ctx **ctx, - struct io_tw_state *ts, - struct llist_node *last) + struct io_tw_state *ts) { unsigned int count = 0; - while (node && node != last) { + do { struct llist_node *next = node->next; struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); @@ -1204,7 +1203,7 @@ static unsigned int handle_tw_list(struct llist_node *node, *ctx = NULL; cond_resched(); } - } + } while (node); return count; } @@ -1223,22 +1222,6 @@ static inline struct llist_node *io_llist_xchg(struct llist_head *head, return xchg(&head->first, new); } -/** - * io_llist_cmpxchg - possibly swap all entries in a lock-less list - * @head: the head of lock-less list to delete all entries - * @old: expected old value of the first entry of the list - * @new: new entry as the head of the list - * - * perform a cmpxchg on the first entry of the list. - */ - -static inline struct llist_node *io_llist_cmpxchg(struct llist_head *head, - struct llist_node *old, - struct llist_node *new) -{ - return cmpxchg(&head->first, old, new); -} - static __cold void io_fallback_tw(struct io_uring_task *tctx, bool sync) { struct llist_node *node = llist_del_all(&tctx->task_list); @@ -1273,9 +1256,7 @@ void tctx_task_work(struct callback_head *cb) struct io_ring_ctx *ctx = NULL; struct io_uring_task *tctx = container_of(cb, struct io_uring_task, task_work); - struct llist_node fake = {}; struct llist_node *node; - unsigned int loops = 0; unsigned int count = 0; if (unlikely(current->flags & PF_EXITING)) { @@ -1283,21 +1264,9 @@ void tctx_task_work(struct callback_head *cb) return; } - do { - loops++; - node = io_llist_xchg(&tctx->task_list, &fake); - count += handle_tw_list(node, &ctx, &ts, &fake); - - /* skip expensive cmpxchg if there are items in the list */ - if (READ_ONCE(tctx->task_list.first) != &fake) - continue; - if (ts.locked && !wq_list_empty(&ctx->submit_state.compl_reqs)) { - io_submit_flush_completions(ctx); - if (READ_ONCE(tctx->task_list.first) != &fake) - continue; - } - node = io_llist_cmpxchg(&tctx->task_list, &fake, NULL); - } while (node != &fake); + node = llist_del_all(&tctx->task_list); + if (node) + count = handle_tw_list(node, &ctx, &ts); ctx_flush_and_put(ctx, &ts); @@ -1305,7 +1274,7 @@ void tctx_task_work(struct callback_head *cb) if (unlikely(atomic_read(&tctx->in_cancel))) io_uring_drop_tctx_refs(current); - trace_io_uring_task_work_run(tctx, count, loops); + trace_io_uring_task_work_run(tctx, count, 1); } static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) From patchwork Tue Feb 6 16:24:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547587 Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BA64134CF1 for ; Tue, 6 Feb 2024 16:27:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236857; cv=none; b=tbxdGBka/DHjSkI2wnIY1vK85gS66qP9lwVl88iET+x2cA4nep8oxsQa1i/Rz1FIgAvjD36sYl9hBKFIdxeeq1PlyMi3hkzFWt2Znmi6K7KkToEPb5Sb33MtWIl/pL+k6FW61R/e3ROVA/gIQdVrfALspYQJAsUCpUBtCUGsTMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236857; c=relaxed/simple; bh=/uHETyTBZ5h7qZwyyhN1Chur+FqysGNL2prClNI50i4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mvhwI2wbjy3F1Agmqn6ITJahRmQ+4+dEF/EdNj4hydEq7AWh6l3mO5M/Ys7QhW65WzEaWBTQ0vni0OzCVEsjo0V3u/zMWG7gDHjpc8SwhqHlq8XICJXKjz9p9wQ3fxDOAa7b9K3WrF1b8uku1F1vfKBCOPcWCjJPpFyKr5+VoRY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=OrkvPyBX; arc=none smtp.client-ip=209.85.166.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="OrkvPyBX" Received: by mail-io1-f48.google.com with SMTP id ca18e2360f4ac-7c3d923f7cbso16620539f.0 for ; Tue, 06 Feb 2024 08:27:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236854; x=1707841654; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G0G9AMUuGqGEDC6zSd+LcpTCMkoymgzlo3crEie7Dr0=; b=OrkvPyBXXJ1qfrBVYEshgLv4wiHebH57J1HnXumXuCwrkXs2NGq7EHuNwZZ6vDYWPR 8I9A9n8TjjcA95M9nvecxcYugDWS3Cmxh5DEps2BxW24RZRtdfGGLjLHaTjuDIDX4k2j OZnTnRmcJZ31DqXMQp8JmWevfouiusJzjmouQd7Z9bI7yiW3YP9JxEbMf+0qt/+TAOnJ TQtCXqiQkGwPqh9PMSRWATJvGiLcTF6/Q+PtnrSReTsaUZBp16Hqi0vNCJUWzowRm8tx 4LyZCXrWc2f8Ym29CllbCmbzaXBqxOPatZOIbPlxsEqImvndZiLHq06dcCQ9jGVF5Yyq //+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236854; x=1707841654; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G0G9AMUuGqGEDC6zSd+LcpTCMkoymgzlo3crEie7Dr0=; b=buiiotaJRIA3KMx6+KwJqWYhoEToH/yEDTRfhksWUzzYzAtNpXyPed5WApf+rQYDmh vp5pUYJ5QxDCKPoEdP5hWM+1+lRe2BqjDYILyDQVS0PiJTJ4QjzuGPbCTeuND2CJLSx3 awGI9s6k5e+BD6h6SMPYFKCPXGdQkHDt+jGZNqRfQuv07ik0ZMUTj5BPnZhuiQ5IT3IF yqi8t6mFdNyyAk4BJr8RdinOBrHx7UbaRr+KiNSs6BUuIVuiUKY0ceIP8RExPkG2sSS4 F5J9xjcUDdF5dTWJ1zRQKT3um76QoK83PPdDI5MCVYasTNFtazpANa4m9q1jmvJnhaeJ QdHA== X-Gm-Message-State: AOJu0YyEP+IUHvuxCofvPxbHdrDFsBdDz+MYsJvnU0wpaxHbg6p1/A8q HlzZv7ZbCvNWdxFN/X6JTJfMtyBpu92wvTJVPShekh5339eHETOvAHLuFnAbTeZHqySXzgexIgW 9aY0= X-Google-Smtp-Source: AGHT+IEQNVU0P/YlEuTbp5ja6SAZzOSTEw4UZSIHRIaxIHf5ygr2pNAFKuVwuE7AKftgghAWyvVy9w== X-Received: by 2002:a5d:938a:0:b0:7c3:f631:a18f with SMTP id c10-20020a5d938a000000b007c3f631a18fmr1205470iol.1.1707236854176; Tue, 06 Feb 2024 08:27:34 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:32 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 03/10] io_uring: remove 'loops' argument from trace_io_uring_task_work_run() Date: Tue, 6 Feb 2024 09:24:37 -0700 Message-ID: <20240206162726.644202-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We no longer loop in task_work handling, hence delete the argument from the tracepoint as it's always 1 and hence not very informative. Signed-off-by: Jens Axboe --- include/trace/events/io_uring.h | 10 +++------- io_uring/io_uring.c | 2 +- 2 files changed, 4 insertions(+), 8 deletions(-) diff --git a/include/trace/events/io_uring.h b/include/trace/events/io_uring.h index 3d7704a52b73..6bb4aaba9e9c 100644 --- a/include/trace/events/io_uring.h +++ b/include/trace/events/io_uring.h @@ -602,29 +602,25 @@ TRACE_EVENT(io_uring_cqe_overflow, * * @tctx: pointer to a io_uring_task * @count: how many functions it ran - * @loops: how many loops it ran * */ TRACE_EVENT(io_uring_task_work_run, - TP_PROTO(void *tctx, unsigned int count, unsigned int loops), + TP_PROTO(void *tctx, unsigned int count), - TP_ARGS(tctx, count, loops), + TP_ARGS(tctx, count), TP_STRUCT__entry ( __field( void *, tctx ) __field( unsigned int, count ) - __field( unsigned int, loops ) ), TP_fast_assign( __entry->tctx = tctx; __entry->count = count; - __entry->loops = loops; ), - TP_printk("tctx %p, count %u, loops %u", - __entry->tctx, __entry->count, __entry->loops) + TP_printk("tctx %p, count %u", __entry->tctx, __entry->count) ); TRACE_EVENT(io_uring_short_write, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ae5b38355864..ced15a13fcbb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1274,7 +1274,7 @@ void tctx_task_work(struct callback_head *cb) if (unlikely(atomic_read(&tctx->in_cancel))) io_uring_drop_tctx_refs(current); - trace_io_uring_task_work_run(tctx, count, 1); + trace_io_uring_task_work_run(tctx, count); } static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) From patchwork Tue Feb 6 16:24:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547588 Received: from mail-io1-f48.google.com (mail-io1-f48.google.com [209.85.166.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CD1B134CFB for ; Tue, 6 Feb 2024 16:27:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236858; cv=none; b=Px2wv9dk3/6rukND9ThIOeYmTog/6C+Z8th2i/pW7+abXuLlvTRZN05v0z48JKZLeVTsm3avKCqOeQeRsZBtA8eMp/hKWeZpXgMQCeimiXu7cP1pr69J/gApAZ1R5PcOjqMzrnHO7gviQhvbGGDZ1aOIGCtwcrnNUY1lvZ9025U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236858; c=relaxed/simple; bh=U2Clrm09ZhoM7Wa+PuhSc+fENSYu0dlRhn130hTzr5U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dDKxQEDrKjiVrLTxZkmUiIHVMHCmtX1bt38UGZ4OUofKp3n0GhGU57SOqB1SId4RPgeOmjPIj2jJTDOc9+JdWEM8CibFok6rzvt/0KeMhvr5bJnro+1/Zi+j6vXiB+G5sSYONzztkL+pGQ0xm2+1acz+iIuJhUQkDMew7exauJU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=e++tOjeG; arc=none smtp.client-ip=209.85.166.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="e++tOjeG" Received: by mail-io1-f48.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so59141939f.0 for ; Tue, 06 Feb 2024 08:27:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236856; x=1707841656; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fMvLhIjj8u1cNNDBwKcaKvfUao8xbMSae4H1f8ns5ZM=; b=e++tOjeGk06Uo09sH2qV2NTlHMlMvl3zO9NtUGfvFY0dayBKWw+ZlrvHRE3oBZtR7p +uYDP/lCvAcdb4W+GRh44T0Gj4MApCzZjfWTWfP5wbLMlqtHOt4kDLDpFrXIT3KajQIa b9lNxdq7VCo8hU3KkQl6B1vOkYBvVKiPvvnl6KM4lGr50hOn67FCG/lfs1V7dNdF39Cm 9t04eHjMsbslolQVxQusj729Xpa7E2Nv4jSsDXOy67usPd4g0HgdangOS+yAzbQDBsTm 3uSzUiKcf+050lmPUkrgaXlQt1wV0odw5Ami4vrZwPDd90bEu1HlQLpJjxKQ3x0TuyVc oTpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236856; x=1707841656; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fMvLhIjj8u1cNNDBwKcaKvfUao8xbMSae4H1f8ns5ZM=; b=REFp4Cf7sJ0jfvPrxw7NLilGsLdYcIX4CNeC8TIoGFhDg6RqIdZ7hN5vxoc9g7mxcG 0ZCDJQv9jCst0hRC2Fg1VzQImDthAp6IQwQCGddPZZa8NpILZOwSEipuH1oxUtT1PFtm e+AX8vXC94qXMa3tnzi1OfpPhQFoy5eTL48dc47l2uH9Io5chzz/fbUQiP0bU7mh8ATv VLDBqmr+3AA3oYXhgrG2Agyk0ld6Xe+OSANO3GpMU8+j4Ux2JFSvdJh6N9zAvJcrJ5Y0 79HZqPprub0KzBbpRbpv+Q9ng3alL2o8wYuff96XAfh2GMDREWNzr9JcaaQwSoeVvoKV 181g== X-Gm-Message-State: AOJu0YwdJe0PHYq7mnBvxzp/ZLcN0LZtP7WKikQhK8L7d6xmGU+k/lkr 9kWHs8MfZVtY5I/JoKfeHbU/UjWX2C8aSdiXywBFnvYxGgnKcORWMMsHF8JBu9xhikK2RFkgAmY ZCkY= X-Google-Smtp-Source: AGHT+IG4cQE56HQ3GoiJkEDBFDpUzrae3W56dJMoLLnJsBMCzEgr0zh8sBsYQkFcQq1XTVnYF49ejw== X-Received: by 2002:a05:6602:123a:b0:7c0:2ea0:b046 with SMTP id z26-20020a056602123a00b007c02ea0b046mr3677335iot.1.1707236856201; Tue, 06 Feb 2024 08:27:36 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:34 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 04/10] io_uring: handle traditional task_work in FIFO order Date: Tue, 6 Feb 2024 09:24:38 -0700 Message-ID: <20240206162726.644202-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For local task_work, which is used if IORING_SETUP_DEFER_TASKRUN is set, we reverse the order of the lockless list before processing the work. This is done to process items in the order in which they were queued, as the llist always adds to the head. Do the same for traditional task_work, so we have the same behavior for both types. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ced15a13fcbb..47d06bc55c95 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1266,7 +1266,7 @@ void tctx_task_work(struct callback_head *cb) node = llist_del_all(&tctx->task_list); if (node) - count = handle_tw_list(node, &ctx, &ts); + count = handle_tw_list(llist_reverse_order(node), &ctx, &ts); ctx_flush_and_put(ctx, &ts); From patchwork Tue Feb 6 16:24:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547589 Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 715A312E1ED for ; Tue, 6 Feb 2024 16:27:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236861; cv=none; b=Abh1SyjdjuUr7d7cr8yHGlUYQ9irRLucrIC4ddlbLsEcem55SiB/xy+N/Mfo5pBtbvtioM4vCffUIiXsUVjBGZ4U05SjmKvnz3kLs/DdabmlFowrjf4l11ncw97JMEliIyHXC0vzkvlQdDKMrA00GKIoM1ehBkovxiZcNOJf3gs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236861; c=relaxed/simple; bh=sHxZJS7kSIuUULAieqdB85lzc84QEvd6GJS4n2E5Rs4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IpRlNbOxlkFPUghE/ZsGHrKK/gLL1EzVDcTbEZ390SSnkLNmlmRorF+z+KVDNkUnfqXybvfBdoopYjR7J7hlOe9Dvv9hGvWmmTo4WVl3riAvFDCbG0lgbAFASBMrPgikjdvT52ECu8zkF04uweIfrTzAu8CmysmWz5dTwZgNv1k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=NNKtRr9B; arc=none smtp.client-ip=209.85.166.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="NNKtRr9B" Received: by mail-io1-f44.google.com with SMTP id ca18e2360f4ac-7bbdd28a52aso98723339f.1 for ; Tue, 06 Feb 2024 08:27:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236858; x=1707841658; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XkbEO3p68qIQIUai79CU/8MPZ1efFTqJP2GlbGjeI6s=; b=NNKtRr9BvnuMtBP2hTzwXPsD6L9sGoTviP1iKXUEJozbSI8Bf1MSHy8IqO0c+WdeDK I+6Wmkex6NsBy3VW+xGYgcveXupg98mP9tPnKjt3ifiumyXXZjx6kuLQenAU9DVCleKw iioli4mV08J3qWBFk6muyojKvcN1OlyeDzXKS5eILdPTqmKt9eno2jH9oG0wWxg+b3NB 0sukfEpwpwaEhxwOsjtoDIUksK9WoH3d55UKHL9p1Upkd8rm0SiEIu5LLsetXKLh87xh fDlalRn0IxKWpYmBi68wCbTVyu++sAAB3D9wds56Da7XuByYmIkfN9LsfymHobhFfwNT vW/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236858; x=1707841658; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XkbEO3p68qIQIUai79CU/8MPZ1efFTqJP2GlbGjeI6s=; b=pKD1pTWqhCjhUagBBPWjfd2t9yeMd06i/RyHaapNrn+gvR99GXZnN7Qgr5UcPegUAL 6QspKVuF6qHFTJAJKTDC8qDGfnHnc0ksS56EvuCPFRLUaseNfyEDPUsHa47AP3UzuYWU gdSqFdvcIsG4KnRlVmBS7/Qdel/Q79NoAjbtmx6BolbILl9X3bdG3iWQpSHnLBUwznNI E9ujqKs3UNoCGrEVHHvrq5Soj7GKrcDPMOo+ngUDaQ4mfSDmakJyEFT3M4pDVz+SwWMQ M9dHSBHnzPynWX8tcYoimJSA8C/W3q+AcRmo0BHmhYzwWylE2TCUw3Cj3VAdsR2HTywE ONfw== X-Gm-Message-State: AOJu0YzYMlCK6GF3rk1FQ6NCgKeC4hVXROT/PRGRcD7EhGvxKmRq2suk YENJKukjsYeZ/zC3dMiH16oML2M5wuxF59+f23lJzxjfV4b02HMtYrFR0FdJ14TBzop8VXXWnTA hvcg= X-Google-Smtp-Source: AGHT+IHEOxMfjlKYM3yv40ZALc2gOo2aSf8sAIZFxvGZC5pGJ7x219iEr5ez4m4PzFR0iKwetGNzGA== X-Received: by 2002:a05:6602:123a:b0:7c0:2ea0:b046 with SMTP id z26-20020a056602123a00b007c02ea0b046mr3677419iot.1.1707236858112; Tue, 06 Feb 2024 08:27:38 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:36 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/10] io_uring: remove next io_kiocb fetch in task_work running Date: Tue, 6 Feb 2024 09:24:39 -0700 Message-ID: <20240206162726.644202-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We just reversed the task_work list and that will have touched requests as well, just get rid of this optimization as it should not make a difference anymore. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 47d06bc55c95..a587b240fa48 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1184,8 +1184,6 @@ static unsigned int handle_tw_list(struct llist_node *node, struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); - prefetch(container_of(next, struct io_kiocb, io_task_work.node)); - if (req->ctx != *ctx) { ctx_flush_and_put(*ctx, ts); *ctx = req->ctx; @@ -1408,7 +1406,6 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts) struct llist_node *next = node->next; struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); - prefetch(container_of(next, struct io_kiocb, io_task_work.node)); INDIRECT_CALL_2(req->io_task_work.func, io_poll_task_func, io_req_rw_complete, req, ts); From patchwork Tue Feb 6 16:24:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547590 Received: from mail-io1-f43.google.com (mail-io1-f43.google.com [209.85.166.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 601E6134CD3 for ; Tue, 6 Feb 2024 16:27:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236863; cv=none; b=eQ6lavkZuZa5HszA0a7xLMRoICuzfVLEp4GIfwpHNhwQNmsOiDqHSIKFszEYxHkfMTqWsQX0gOlYiH9c1RUiN6X6tOpMFP1zHDWmAnARDtafi6rjOhvRttTa10fU6vWeDZO5MUq+9WoizNBenj4a9A5khZfL2V6l0pivKB4LO08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236863; c=relaxed/simple; bh=R/LRAEJbwwD4YpGLmurq5+6JxPXE9VAoQNs2IVMBpN8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SmGgGdqNJsTftqc/wvOg90iYRWo8yqIsqFWQ5tMLiwE/Fw290vx+3No+/frxUGIIZ5M2ANhKeS+y8gK4AFE5YgQqRF1qD+Uakxlm9/UC4UKCU12c3/+fSRuCr+12xqA9HpWLyoecdZeOcdUiUbSmRbhYaN1E0+287TSoP+eMdz8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=CWMxgj1Z; arc=none smtp.client-ip=209.85.166.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="CWMxgj1Z" Received: by mail-io1-f43.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so59143039f.0 for ; Tue, 06 Feb 2024 08:27:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236860; x=1707841660; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fTAxBwS/Fp2VZ+CgFOoX1nsp4BW/P+isZyL7ajilrwQ=; b=CWMxgj1Z9CMWwFle2h60qw/vJwY/uv0JGhOzaM6InT0cSJyz5IdrpJjbWlGJfGeO/6 zdyQ3acLBEtZ2nphqlBBY1B16FgpnAS1RpwIXcBB+FY5tbU00IM6M6uRNdfNn6pMrtHl k/nne3qWI+9ytyxN2Q/nqoaf+tU1MEOWZ/Va4ROVgfzbH943YkG/bsryhpu5lXCQEaRo CMkCmwP+/mD5apvCu2skQYxG5E7dH7/tDte5rikLFez9APsr/oOa4GM/aTREvh/Q0nfr 2pA6obrlwD/izXgQ3BdwQVrH8wVgBsTh1AkOL9611maPdY6R0h7yrjXgXazzxpZACfqB gC5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236860; x=1707841660; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fTAxBwS/Fp2VZ+CgFOoX1nsp4BW/P+isZyL7ajilrwQ=; b=X90wrOtoV61cLrviuu59TosucOaeCU72cUhifblx2+IKelg0GxlSfvEGnqqwyHFdcq LolG8LfFAqGFfd+6Uj1actEl9hWEobNQOuX+0PmJ3CtyVNtaRaIICei0cxeXlzbnGNKY gaG/t8LELWBGGAx2TPAvLPs95p2y/CYqoKmXxm9dasvTK3zcLhapWz5H4T2zG6ik1P9J Z22qDW/K8X+5IWTXRtCuC62a20YYX5F/13JnSOCpY7scHuQ597DnWoZuEAxcdFRn6qU7 m8LIROYl0fbUh7CZ8C84nuqw7sGZNi5Jiv28zDBo0Bb6U4MBTmFM+TQmLrqGVYXHzPJX kefg== X-Gm-Message-State: AOJu0Yw+LsYx66RSgyjHb0huCiQ0ginrFlXOWy68PsZjcbyL5NFfPc4b oB8SrPwP51QOfMZjRRhVKzGIZb0hPPFNBpEXe50JJPKFLYWv/qcz/vR/CHfRiaNq8qTJJktO0+A iMM8= X-Google-Smtp-Source: AGHT+IHXhOKebVgDefnglOOOa+whcp8jJIxR5AgWiIW3eIVJenXtD2Q36WeHwi1ylFqJXxZ1HSvrlg== X-Received: by 2002:a05:6602:123a:b0:7c0:2ea0:b046 with SMTP id z26-20020a056602123a00b007c02ea0b046mr3677517iot.1.1707236860179; Tue, 06 Feb 2024 08:27:40 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:38 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 06/10] io_uring: remove unconditional looping in local task_work handling Date: Tue, 6 Feb 2024 09:24:40 -0700 Message-ID: <20240206162726.644202-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If we have a ton of notifications coming in, we can be looping in here for a long time. This can be problematic for various reasons, mostly because we can starve userspace. If the application is waiting on N events, then only re-run if we need more events. Fixes: c0e0d6ba25f1 ("io_uring: add IORING_SETUP_DEFER_TASKRUN") Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 44 +++++++++++++++++++++++++++++--------------- 1 file changed, 29 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a587b240fa48..ddbce269b6a7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1386,7 +1386,20 @@ static void __cold io_move_task_work_from_local(struct io_ring_ctx *ctx) } } -static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts) +static bool io_run_local_work_continue(struct io_ring_ctx *ctx, int events, + int min_events) +{ + if (llist_empty(&ctx->work_llist)) + return false; + if (events < min_events) + return true; + if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) + atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); + return false; +} + +static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts, + int min_events) { struct llist_node *node; unsigned int loops = 0; @@ -1414,18 +1427,20 @@ static int __io_run_local_work(struct io_ring_ctx *ctx, struct io_tw_state *ts) } loops++; - if (!llist_empty(&ctx->work_llist)) + if (io_run_local_work_continue(ctx, ret, min_events)) goto again; if (ts->locked) { io_submit_flush_completions(ctx); - if (!llist_empty(&ctx->work_llist)) + if (io_run_local_work_continue(ctx, ret, min_events)) goto again; } + trace_io_uring_local_work_run(ctx, ret, loops); return ret; } -static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) +static inline int io_run_local_work_locked(struct io_ring_ctx *ctx, + int min_events) { struct io_tw_state ts = { .locked = true, }; int ret; @@ -1433,20 +1448,20 @@ static inline int io_run_local_work_locked(struct io_ring_ctx *ctx) if (llist_empty(&ctx->work_llist)) return 0; - ret = __io_run_local_work(ctx, &ts); + ret = __io_run_local_work(ctx, &ts, min_events); /* shouldn't happen! */ if (WARN_ON_ONCE(!ts.locked)) mutex_lock(&ctx->uring_lock); return ret; } -static int io_run_local_work(struct io_ring_ctx *ctx) +static int io_run_local_work(struct io_ring_ctx *ctx, int min_events) { struct io_tw_state ts = {}; int ret; ts.locked = mutex_trylock(&ctx->uring_lock); - ret = __io_run_local_work(ctx, &ts); + ret = __io_run_local_work(ctx, &ts, min_events); if (ts.locked) mutex_unlock(&ctx->uring_lock); @@ -1642,7 +1657,7 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) io_task_work_pending(ctx)) { u32 tail = ctx->cached_cq_tail; - (void) io_run_local_work_locked(ctx); + (void) io_run_local_work_locked(ctx, min); if (task_work_pending(current) || wq_list_empty(&ctx->iopoll_list)) { @@ -2486,7 +2501,7 @@ int io_run_task_work_sig(struct io_ring_ctx *ctx) { if (!llist_empty(&ctx->work_llist)) { __set_current_state(TASK_RUNNING); - if (io_run_local_work(ctx) > 0) + if (io_run_local_work(ctx, INT_MAX) > 0) return 0; } if (io_run_task_work() > 0) @@ -2554,7 +2569,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (!io_allowed_run_tw(ctx)) return -EEXIST; if (!llist_empty(&ctx->work_llist)) - io_run_local_work(ctx); + io_run_local_work(ctx, min_events); io_run_task_work(); io_cqring_overflow_flush(ctx); /* if user messes with these they will just get an early return */ @@ -2592,11 +2607,10 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, trace_io_uring_cqring_wait(ctx, min_events); do { + int nr_wait = (int) iowq.cq_tail - READ_ONCE(ctx->rings->cq.tail); unsigned long check_cq; if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - int nr_wait = (int) iowq.cq_tail - READ_ONCE(ctx->rings->cq.tail); - atomic_set(&ctx->cq_wait_nr, nr_wait); set_current_state(TASK_INTERRUPTIBLE); } else { @@ -2615,7 +2629,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, */ io_run_task_work(); if (!llist_empty(&ctx->work_llist)) - io_run_local_work(ctx); + io_run_local_work(ctx, nr_wait); /* * Non-local task_work will be run on exit to userspace, but @@ -3270,7 +3284,7 @@ static __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, if ((ctx->flags & IORING_SETUP_DEFER_TASKRUN) && io_allowed_defer_tw_run(ctx)) - ret |= io_run_local_work(ctx) > 0; + ret |= io_run_local_work(ctx, INT_MAX) > 0; ret |= io_cancel_defer_files(ctx, task, cancel_all); mutex_lock(&ctx->uring_lock); ret |= io_poll_remove_all(ctx, task, cancel_all); @@ -3632,7 +3646,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, * it should handle ownership problems if any. */ if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) - (void)io_run_local_work_locked(ctx); + (void)io_run_local_work_locked(ctx, min_complete); } mutex_unlock(&ctx->uring_lock); } From patchwork Tue Feb 6 16:24:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547591 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51899134CF3 for ; Tue, 6 Feb 2024 16:27:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236865; cv=none; b=TLIhA9G9KgBiV6h/SvU7z7/L/QjYQrU5JE2WzUUBdWiKoKrgWmh7/2cZKw2NLomXNt6S3im91oMMnLfxEoRYdizsrTV5e24r0vAp24MNJMOhV1O1by3zERcqcf0olz3x9hZKWE+qxrPw0Vy2939ZYkNBZlt10sUHDj898YXaTyA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236865; c=relaxed/simple; bh=aY8eL75oLJ6assrEEuO2Iiym9z4MWJRlEcSBeyfX1qw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kZSgmU4kNTWiECF3LiUIBsw94RSDOLu0DvPtLQCWEb04ep7CBwBj837rpSNKtOBng/Ri9GQIPwMRDPbJkIulMeqgahgOZx8PlvP+wU1W2IiymG2NOUDCa735nvG3+1D44fTKbOgk1Nt1G4MqrJBc3AqcWDybYOcOQCGim0HZb8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=nxspfDxQ; arc=none smtp.client-ip=209.85.166.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="nxspfDxQ" Received: by mail-io1-f47.google.com with SMTP id ca18e2360f4ac-7bff2f6080aso40814039f.1 for ; Tue, 06 Feb 2024 08:27:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236862; x=1707841662; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gjZSEeL4+KnrvnzqwSF9/nWH5ab6nBeq1UOj95ia5fI=; b=nxspfDxQYExrI8wS1AkzRCXbHa13852X77NXL7s2dBcgwLRezSWvoy2pIO/KkzDCOi leFXqbgHHlBQCIrTJnQxutInMmZ9z+UKaXZCoPvl4i6msfRL1pBIif2CWLA5+U1PmXHK FFOa25HluR9YyKw5vFg2FBkgHSKVZSVE8xPuJ4jRpb7Yga9OwU1u4Kj8y7PCXvffpfrX WVtnUzwbEfnUcOl3+FoIux5pWOTzZ2xN/9mKIkvEQWQAgwx0nK+KYQGr0zq52MkIwmIP utt+FzZJIZyAyt8jQ5lNwy/za15FpFzT04mv1NsjQg8P08WrsR/da09WD6URH29YqjM5 U4vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236862; x=1707841662; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gjZSEeL4+KnrvnzqwSF9/nWH5ab6nBeq1UOj95ia5fI=; b=Zreh7uf2Bhprrvgqyd5iNNWawKXmt2VadasSibn+3RvHx/aC14pWg8V4lh9TyzoOYl mZ+3xj3lS0gEdzDRrnXG3rU1oWmVzixtEHCKa3LrwWFBKDGlQjs9uyYd6XfgDpoNOa6g UIISX6B5CtUgmzYc5GYcw9vFHAl2SU+K5m2uocB5IheDjmWII3QYQ7l7dOnnvZ2iQ8KF qZXzBGV2W0SEZctmLs6OCaPtxOVMzXHcjPJh98YJPROJvELc5Ips4c95rherM/aMWWt8 q4Ne8wx4p9GY+4K4AZPV1MNyY6rF9NJxBBC8R6kE8Ul1PrvcrTjMIaZo/vYo4VZKUulY mAAg== X-Gm-Message-State: AOJu0YxkosKec4vPCoLV8JSSKdLi+HPAs8NBLU2LpaMVCyJWa3fRfChM pjpQsz22WdA4gbwxSh2au7A7Wr5PVC68jzIftzpGCJpc7LjQ9IIRiBlXP8euCMBckL2ASV6AlDo OsDE= X-Google-Smtp-Source: AGHT+IFoqW6OWqBMtl96ISreoWk0MpcofjJLqe60vPYsDQbGfGQezhyXj/PhsGwNPuaYIov9c2yHdw== X-Received: by 2002:a5d:938a:0:b0:7c3:f631:a18f with SMTP id c10-20020a5d938a000000b007c3f631a18fmr1205820iol.1.1707236862019; Tue, 06 Feb 2024 08:27:42 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:40 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/10] io_uring/poll: improve readability of poll reference decrementing Date: Tue, 6 Feb 2024 09:24:41 -0700 Message-ID: <20240206162726.644202-8-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This overly long line is hard to read. Break it up by AND'ing the ref mask first, then perform the atomic_sub_return() with the value itself. Signed-off-by: Jens Axboe --- io_uring/poll.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 3f3380dc5f68..8e01c2672df4 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -343,8 +343,8 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) * Release all references, retry if someone tried to restart * task_work while we were executing it. */ - } while (atomic_sub_return(v & IO_POLL_REF_MASK, &req->poll_refs) & - IO_POLL_REF_MASK); + v &= IO_POLL_REF_MASK; + } while (atomic_sub_return(v, &req->poll_refs) & IO_POLL_REF_MASK); return IOU_POLL_NO_ACTION; } From patchwork Tue Feb 6 16:24:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547592 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFBED132C0B for ; Tue, 6 Feb 2024 16:27:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236867; cv=none; b=dkQW9K7wgL64WZu23fWwytAcH7LLdvTdFOzLbbq+rgUr5GZSpP7RUS5A0cbtAG5RT/IK6+4Ovg3uc8ZucxV0JLuDoex0BgxPkeGjytv/wlWqcmEQY638xO1nUXvQtw9/4tN5s8SxlKR2WTP4J4L0IlqtN8tuN8dni5zshhlcJaQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236867; c=relaxed/simple; bh=A8v5M+kh6Kj98O4IuUeYD895rOnf2yOV0i92sC0594s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bNqthcsC6nw0Wo5Tz9cCvPASFTKxO4a5juz6eguGcyu7gI0Mw7GtMqwHLqFOxw3uOMH2tLWPamBKHI/h7837Zr6wMT9DYDexqy8v9IU/QzsuDzGoB42i06oVAOKXZgEIRh7Hq/nbhwxijYElDrOdhMxi80fJKUIoLgrcXnWEKnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=rk2DPj9x; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="rk2DPj9x" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-7c3e06c8608so30228539f.1 for ; Tue, 06 Feb 2024 08:27:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236864; x=1707841664; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vbJhfpH4iKSKSZ/16S2TsiEr44tXFsUlzF+G3Ah5qqg=; b=rk2DPj9xUY/dXtSEj5WutGPjSJ5GeBGryHaCz+wQ73/bG0FAw/2sQqTA6eftb6hjFu vAJf0JbHOnY+qGdSiWsTaLeIt3aD0gFhRaWK5RtIV3uXOZYUhih5DEzaubeoEsjbOFsN cKKLgoaqEFS1C6J2Kw2DYmEt3vGPj7Ls2LcPXpWFhr6WlfymwHwdYpSUG6Yn+8dyxG4a QrhEWIjLWHpyRAtuO79aVg42hjBI6OT+ttHwzZkNUgLwmJlYAD+3oKn+tFphvpVk6ycx 8CuSOX1si7Gw9nbJ1daagS4BDpTwgZAZ9gzeZ/ZxTEn4vjP75tv5lCt49aoNBvcjyALT wzUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236864; x=1707841664; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vbJhfpH4iKSKSZ/16S2TsiEr44tXFsUlzF+G3Ah5qqg=; b=bf/wS6ydUZaDPjdaujeDvDAhnAVOxrKPx7RAiKP4dBV7rjcSzyodLyrNhvS+UBgaTR J2CoeMwfQk0tZi5cbSZBYjKwsfRcwdpGq+j6RcY1AMWgQ8HnUXELejGnDEPESeZZ1DYr 0mS34RJ8DHbqJPHMGh18B/9/RAA9vnhMLHdbKnZ4eY4NJBeGDju77Ptq3wkNozfjxCQG BtoKW5uGqFUCgm9RM9L0zkshrc5bxFIuSWHlzjbCpgjw3T43PqQgWGeKhPQTDAfV5Boh Hj6esLY2mnEY0oy6QBctSLxP3NZlI76ixs1zT6sZ0/xcRc14oNgB9tFT6qQUCqN9qrOY i+2A== X-Gm-Message-State: AOJu0Yxztp0EuD/FYGZ9YvWFV8GaUZwn7ZwHaMCckymXddEof6x571GG zUzv4NTPa6OYcM92I9AMJsy3KT2LATM8JOedZjOfCl/AG5jOAmxauSUehqBEnB6iR5xdlYXmd3h LibY= X-Google-Smtp-Source: AGHT+IGSSsVLQiWDs3N6ML78uWrmHKo+Opl7QzDtncSvoCo/Zhf/bdLpoXM+tcVya/YGoGZHQ+rYtA== X-Received: by 2002:a6b:db13:0:b0:7c3:edbb:1816 with SMTP id t19-20020a6bdb13000000b007c3edbb1816mr2791524ioc.2.1707236864514; Tue, 06 Feb 2024 08:27:44 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:42 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/10] io_uring: cleanup handle_tw_list() calling convention Date: Tue, 6 Feb 2024 09:24:42 -0700 Message-ID: <20240206162726.644202-9-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Now that we don't loop around task_work anymore, there's no point in maintaining the ring and locked state outside of handle_tw_list(). Get rid of passing in those pointers (and pointers to pointers) and just do the management internally in handle_tw_list(). Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ddbce269b6a7..df02ed6677c5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1173,10 +1173,10 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) percpu_ref_put(&ctx->refs); } -static unsigned int handle_tw_list(struct llist_node *node, - struct io_ring_ctx **ctx, - struct io_tw_state *ts) +static unsigned int handle_tw_list(struct llist_node *node) { + struct io_ring_ctx *ctx = NULL; + struct io_tw_state ts = { }; unsigned int count = 0; do { @@ -1184,25 +1184,26 @@ static unsigned int handle_tw_list(struct llist_node *node, struct io_kiocb *req = container_of(node, struct io_kiocb, io_task_work.node); - if (req->ctx != *ctx) { - ctx_flush_and_put(*ctx, ts); - *ctx = req->ctx; + if (req->ctx != ctx) { + ctx_flush_and_put(ctx, &ts); + ctx = req->ctx; /* if not contended, grab and improve batching */ - ts->locked = mutex_trylock(&(*ctx)->uring_lock); - percpu_ref_get(&(*ctx)->refs); + ts.locked = mutex_trylock(&ctx->uring_lock); + percpu_ref_get(&ctx->refs); } INDIRECT_CALL_2(req->io_task_work.func, io_poll_task_func, io_req_rw_complete, - req, ts); + req, &ts); node = next; count++; if (unlikely(need_resched())) { - ctx_flush_and_put(*ctx, ts); - *ctx = NULL; + ctx_flush_and_put(ctx, &ts); + ctx = NULL; cond_resched(); } } while (node); + ctx_flush_and_put(ctx, &ts); return count; } @@ -1250,8 +1251,6 @@ static __cold void io_fallback_tw(struct io_uring_task *tctx, bool sync) void tctx_task_work(struct callback_head *cb) { - struct io_tw_state ts = {}; - struct io_ring_ctx *ctx = NULL; struct io_uring_task *tctx = container_of(cb, struct io_uring_task, task_work); struct llist_node *node; @@ -1264,9 +1263,7 @@ void tctx_task_work(struct callback_head *cb) node = llist_del_all(&tctx->task_list); if (node) - count = handle_tw_list(llist_reverse_order(node), &ctx, &ts); - - ctx_flush_and_put(ctx, &ts); + count = handle_tw_list(llist_reverse_order(node)); /* relaxed read is enough as only the task itself sets ->in_cancel */ if (unlikely(atomic_read(&tctx->in_cancel))) From patchwork Tue Feb 6 16:24:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547593 Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F3C812E1ED for ; Tue, 6 Feb 2024 16:27:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236869; cv=none; b=XtH+GaHK093awGJ4UwuQFl0Y+NK7Ds3B2eyneDLv65xx0QJzF+dqE2k27CiM8xixXAhyRWpTSP1hwSdAvdKgr481XBMYq1xLD+HyoshyaoV+L+OSHubVP6ZnYMcKxD8GMxPU6FtbG8aRspAZf7V8SNZtIo+lrSc0i+VhwJjpK84= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236869; c=relaxed/simple; bh=DlYjb0sbIRkTLLxGvUyzYN7iueaFJ0VjPTWMKSPeDpU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vr8yEp3yaxe5EkTSOEONo4uamLQ3D3Cbnw5LkXXecp5cvKBa8hAkxwFRPxrHdjeVH2A5WqtsCj3cftuYUdBhh86qtodyjDFk2F9tKrodpaNsbOoZPAJLfmusfa9gMArwIWkKi8BfXYpdvply5t4RUFhJSVIdJmpDW7+3msGOkYw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=shsoSwgN; arc=none smtp.client-ip=209.85.166.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="shsoSwgN" Received: by mail-io1-f41.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso74155239f.0 for ; Tue, 06 Feb 2024 08:27:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236866; x=1707841666; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aiutwst3+hWMfhhEkUvjQCiZBWfne+AuIbP8Pq0y0gY=; b=shsoSwgNjTzlcdmYVD0LPM5idr63Gxjo/SYtdSF/W7v2HREyo9O5gtixfhl1wwwmqI guIAaD5NwwqtH+pAd6fN5ZjQSNOxljmRpgIqUpbwJo0q261qZvTagd5S0NueDlmWWQ3x mupDMxmdlg0BPTvqP+BMNDUpqlnM3PE/UU8FfZmV8UMnKXk6wiBxoZdpxZyzmQ+J+P1c DDFmalhZyVQFrfaUess+NpG1Tk2UCD27wRsw1h0rtk+vzASAMZpOv6On4p7nK+QDG2dD w5dy5zH8LnCWaWnJ0+jieMyuf4ADZ1bI7fRs51J4D/mh70BfZ2xkX9g4HX/pnIBq/1QH ia1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236866; x=1707841666; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aiutwst3+hWMfhhEkUvjQCiZBWfne+AuIbP8Pq0y0gY=; b=wrk8xP7pKIbQBUb5t0GaKL+Vo4q30xRUzvZpG7SG0GKbRFW3iR4J9cLj2QZhD+uh0l SbpeeNf/BN3x0mM+5tNU6It6kSYcXZY+lqF0DyJQDVpOBVhL7Z2jOaiA685bpEsDZTap pi7sU3VI3i8RfHdYFMSnRxMxm2qoJgesBQuiqtj2QGIjhmTWs9ewPbHSo+7kTZzycdb8 U9qoooo24OnH9Z6b4jQiIzegt/qwB8MH2xeHIjWAvK6I5bH9qa4H2k/JtzXh6ibF2mRx 0Wxdr3vTPfwYFTJZlCkwcQ6MfjO+wmIdm5wzvhrYdbcCDD8sPFk3WxQ8UdNSDm/p2wK1 CS2Q== X-Gm-Message-State: AOJu0YwgAB7JCtjVIHWcAfygP01yu8QKoBRjayLuvHwrAHNBDHhGk82r 5ybq6CxryOnDkokvFp5h8M6O3UhAJMm3UisRzfVcB9a3JycPlBIFihB2JAQtpsd2WwISdI5onYh gHnM= X-Google-Smtp-Source: AGHT+IH2mYCGJZEwqv2o/uOY6+S1TNfy2cU/9OPAxHlP8zVezDaAMxBo24HLfrYDJzn1rLuCE9iuTQ== X-Received: by 2002:a5e:990c:0:b0:7c3:f75f:7b12 with SMTP id t12-20020a5e990c000000b007c3f75f7b12mr859245ioj.0.1707236866314; Tue, 06 Feb 2024 08:27:46 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:44 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/10] io_uring: pass in counter to handle_tw_list() rather than return it Date: Tue, 6 Feb 2024 09:24:43 -0700 Message-ID: <20240206162726.644202-10-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 No functional changes in this patch, just in preparation for returning something other than count from this helper. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index df02ed6677c5..20421bf36cc1 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1173,11 +1173,10 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) percpu_ref_put(&ctx->refs); } -static unsigned int handle_tw_list(struct llist_node *node) +static void handle_tw_list(struct llist_node *node, unsigned int *count) { struct io_ring_ctx *ctx = NULL; struct io_tw_state ts = { }; - unsigned int count = 0; do { struct llist_node *next = node->next; @@ -1195,7 +1194,7 @@ static unsigned int handle_tw_list(struct llist_node *node) io_poll_task_func, io_req_rw_complete, req, &ts); node = next; - count++; + (*count)++; if (unlikely(need_resched())) { ctx_flush_and_put(ctx, &ts); ctx = NULL; @@ -1204,7 +1203,6 @@ static unsigned int handle_tw_list(struct llist_node *node) } while (node); ctx_flush_and_put(ctx, &ts); - return count; } /** @@ -1263,7 +1261,7 @@ void tctx_task_work(struct callback_head *cb) node = llist_del_all(&tctx->task_list); if (node) - count = handle_tw_list(llist_reverse_order(node)); + handle_tw_list(llist_reverse_order(node), &count); /* relaxed read is enough as only the task itself sets ->in_cancel */ if (unlikely(atomic_read(&tctx->in_cancel))) From patchwork Tue Feb 6 16:24:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13547594 Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BE731AB7E2 for ; Tue, 6 Feb 2024 16:27:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236870; cv=none; b=TQyV7YmwK7IEwCjwOKHIO33OUujEQxYb1KVk7QrnwCFGBpBUrtzx5uWhnq3akZbsRL92JBX+qN1GNnaATYw1XaOJhgvxGU5GABAk+mz6A69Ols/dob0Fyf/JYlQipAYVrVP6VYSn6z9V37S2Fx/u4Vi0NEDdWVKd8u1XMU+8iKY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707236870; c=relaxed/simple; bh=ctpng8E34OqQL0TkbrLgWGZdXTSzthGt5vhwv6vtCUU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rPQ++3myeugMW6fhdiwD1jEnnzanYqbVaQ/DwJQE/R5OCgfWNyHRMaJC4LBES5TRRRWwaSeV2meqJyhz+409e3FMFrz/SXTp8eOEiwkCdHx3RceuoVlnEEkoLSYiCV8S1KN2swZERzm14U84HbxYOESoYP2L/f9tANrhpJK4Vb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=CPkMMR0D; arc=none smtp.client-ip=209.85.166.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="CPkMMR0D" Received: by mail-io1-f52.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so59144239f.0 for ; Tue, 06 Feb 2024 08:27:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707236867; x=1707841667; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OOXs2/DYRwIPiYEttFk1hLxTH7tUIHXaBKEkUZ29kqY=; b=CPkMMR0D39KbgpaM5jyp/oexnva7tFkgScfv5mzrgvPIpcfrdEO/1Q6zV2MWWIXaOI v6jNtIHElGRkxgcgvHkuHqgLAY2i2fFqxd0toC/iBuuBvQfjYWAYewHd4fJrSg/Ub+4y h6zsQbsiPlIywQfob0US7Edw+644GrHvcOptF0OKjjz9vcSZ7KqgJDKmKIeWFI/4Ft76 hzW4YWN9G9TkdDkvuHjuCrkaGuIeCJrtoqXwpssfqKkK4i7CrV/NEuo0Q/IgyTCItc+7 0U4prNKJ29hq0H3bnFXZxGX3c+VNylt0WxvcK59lR/4g2GL/AWDD3h3gRcvIrttOFMCQ duhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707236867; x=1707841667; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OOXs2/DYRwIPiYEttFk1hLxTH7tUIHXaBKEkUZ29kqY=; b=SWy3FnagHisWg+7M12B4G9RONIV+BbLs8ibeLp0INoxGqMUBoFuKBRui1VhD8O6mia GEFKtrBTjj8UzG88rong3SgRY9E6BbtIvQrttxEj/BxWoJISX7DNTDo6vczxit7zqU/z opOZBwtb7jBxx5sT7mEikxlC0PoZnL1INb4zUwW47UIqU/dAg3YQKFXjHr4hhRYn+9BK TV+h4abMAHdL9zM4wxLvdbGzl7GCMjz8HFDTi1GBp6DcwHO6ktGU1zPs8IKoVDaULVR0 lzBUHLpe9RPornaDWQH6gVDXqz+pQBwzg+xJzjqcTgrkn3HdpQCY3Ls6iqGJz1w6KNCs 9dew== X-Gm-Message-State: AOJu0Yy6d4F2YpIiD5aOWf8FjP0hKLMI0FSCkaxHdKOpRnXJ1pU9z2IV r4MiDixb6iPMBr+Iizk+BkS2/cR73aDAidGu5rwSAroGn5r3k5CFIR1EsdHsAqJIvcdNpWN/54O zODI= X-Google-Smtp-Source: AGHT+IFklKH8v0uZewFQHDqYukoAG9n8WqD5TdLJgiSU66v5bwecFzSQyC1rIPJWyUHZQbzJCRq6Bg== X-Received: by 2002:a05:6602:70c:b0:7c2:caa4:561a with SMTP id f12-20020a056602070c00b007c2caa4561amr3311550iox.2.1707236867123; Tue, 06 Feb 2024 08:27:47 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id a1-20020a6b6601000000b007bffd556183sm513309ioc.14.2024.02.06.08.27.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Feb 2024 08:27:46 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 10/10] io_uring/sqpoll: manage task_work privately Date: Tue, 6 Feb 2024 09:24:44 -0700 Message-ID: <20240206162726.644202-11-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240206162726.644202-1-axboe@kernel.dk> References: <20240206162726.644202-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Decouple from task_work running, and cap the number of entries we process at the time. If we exceed that number, push remaining entries to a retry list that we'll process first next time. We cap the number of entries to process at 8, which is fairly random. We just want to get enough per-ctx batching here, while not processing endlessly. Since we manually run PF_IO_WORKER related task_work anyway as the task never exits to userspace, with this we no longer need to add an actual task_work item to the per-process list. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 46 +++++++++++++++++++++++++++++++++++---------- io_uring/io_uring.h | 24 +++++++++++++++++------ io_uring/sqpoll.c | 29 +++++++++++++++++++++++++++- 3 files changed, 82 insertions(+), 17 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 20421bf36cc1..9ba2244c624e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1173,7 +1173,14 @@ static void ctx_flush_and_put(struct io_ring_ctx *ctx, struct io_tw_state *ts) percpu_ref_put(&ctx->refs); } -static void handle_tw_list(struct llist_node *node, unsigned int *count) +/* + * Run queued task_work, returning the number of entries processed in *count. + * If more entries than max_entries are available, stop processing once this + * is reached and return the rest of the list. + */ +struct llist_node *io_handle_tw_list(struct llist_node *node, + unsigned int *count, + unsigned int max_entries) { struct io_ring_ctx *ctx = NULL; struct io_tw_state ts = { }; @@ -1200,9 +1207,10 @@ static void handle_tw_list(struct llist_node *node, unsigned int *count) ctx = NULL; cond_resched(); } - } while (node); + } while (node && *count < max_entries); ctx_flush_and_put(ctx, &ts); + return node; } /** @@ -1247,27 +1255,41 @@ static __cold void io_fallback_tw(struct io_uring_task *tctx, bool sync) } } -void tctx_task_work(struct callback_head *cb) +struct llist_node *tctx_task_work_run(struct io_uring_task *tctx, + unsigned int max_entries, + unsigned int *count) { - struct io_uring_task *tctx = container_of(cb, struct io_uring_task, - task_work); struct llist_node *node; - unsigned int count = 0; if (unlikely(current->flags & PF_EXITING)) { io_fallback_tw(tctx, true); - return; + return NULL; } node = llist_del_all(&tctx->task_list); - if (node) - handle_tw_list(llist_reverse_order(node), &count); + if (node) { + node = llist_reverse_order(node); + node = io_handle_tw_list(node, count, max_entries); + } /* relaxed read is enough as only the task itself sets ->in_cancel */ if (unlikely(atomic_read(&tctx->in_cancel))) io_uring_drop_tctx_refs(current); - trace_io_uring_task_work_run(tctx, count); + trace_io_uring_task_work_run(tctx, *count); + return node; +} + +void tctx_task_work(struct callback_head *cb) +{ + struct io_uring_task *tctx; + struct llist_node *ret; + unsigned int count = 0; + + tctx = container_of(cb, struct io_uring_task, task_work); + ret = tctx_task_work_run(tctx, UINT_MAX, &count); + /* can't happen */ + WARN_ON_ONCE(ret); } static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) @@ -1350,6 +1372,10 @@ static void io_req_normal_work_add(struct io_kiocb *req) if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); + /* SQPOLL doesn't need the task_work added, it'll run it itself */ + if (ctx->flags & IORING_SETUP_SQPOLL) + return; + if (likely(!task_work_add(req->task, &tctx->task_work, ctx->notify_method))) return; diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 46795ee462df..38af82788786 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -57,6 +57,8 @@ void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use); void io_req_task_complete(struct io_kiocb *req, struct io_tw_state *ts); void io_req_task_queue_fail(struct io_kiocb *req, int ret); void io_req_task_submit(struct io_kiocb *req, struct io_tw_state *ts); +struct llist_node *io_handle_tw_list(struct llist_node *node, unsigned int *count, unsigned int max_entries); +struct llist_node *tctx_task_work_run(struct io_uring_task *tctx, unsigned int max_entries, unsigned int *count); void tctx_task_work(struct callback_head *cb); __cold void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd); int io_uring_alloc_task_context(struct task_struct *task, @@ -275,6 +277,8 @@ static inline unsigned int io_sqring_entries(struct io_ring_ctx *ctx) static inline int io_run_task_work(void) { + bool ret = false; + /* * Always check-and-clear the task_work notification signal. With how * signaling works for task_work, we can find it set with nothing to @@ -286,18 +290,26 @@ static inline int io_run_task_work(void) * PF_IO_WORKER never returns to userspace, so check here if we have * notify work that needs processing. */ - if (current->flags & PF_IO_WORKER && - test_thread_flag(TIF_NOTIFY_RESUME)) { - __set_current_state(TASK_RUNNING); - resume_user_mode_work(NULL); + if (current->flags & PF_IO_WORKER) { + if (test_thread_flag(TIF_NOTIFY_RESUME)) { + __set_current_state(TASK_RUNNING); + resume_user_mode_work(NULL); + } + if (current->io_uring) { + unsigned int count = 0; + + tctx_task_work_run(current->io_uring, UINT_MAX, &count); + if (count) + ret = true; + } } if (task_work_pending(current)) { __set_current_state(TASK_RUNNING); task_work_run(); - return 1; + ret = true; } - return 0; + return ret; } static inline bool io_task_work_pending(struct io_ring_ctx *ctx) diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c index 65b5dbe3c850..28bf0e085d31 100644 --- a/io_uring/sqpoll.c +++ b/io_uring/sqpoll.c @@ -18,6 +18,7 @@ #include "sqpoll.h" #define IORING_SQPOLL_CAP_ENTRIES_VALUE 8 +#define IORING_TW_CAP_ENTRIES_VALUE 8 enum { IO_SQ_THREAD_SHOULD_STOP = 0, @@ -219,8 +220,31 @@ static bool io_sqd_handle_event(struct io_sq_data *sqd) return did_sig || test_bit(IO_SQ_THREAD_SHOULD_STOP, &sqd->state); } +/* + * Run task_work, processing the retry_list first. The retry_list holds + * entries that we passed on in the previous run, if we had more task_work + * than we were asked to process. Newly queued task_work isn't run until the + * retry list has been fully processed. + */ +static unsigned int io_sq_tw(struct llist_node **retry_list, int max_entries) +{ + struct io_uring_task *tctx = current->io_uring; + unsigned int count = 0; + + if (*retry_list) { + *retry_list = io_handle_tw_list(*retry_list, &count, max_entries); + if (count >= max_entries) + return count; + max_entries -= count; + } + + *retry_list = tctx_task_work_run(tctx, max_entries, &count); + return count; +} + static int io_sq_thread(void *data) { + struct llist_node *retry_list = NULL; struct io_sq_data *sqd = data; struct io_ring_ctx *ctx; unsigned long timeout = 0; @@ -257,7 +281,7 @@ static int io_sq_thread(void *data) if (!sqt_spin && (ret > 0 || !wq_list_empty(&ctx->iopoll_list))) sqt_spin = true; } - if (io_run_task_work()) + if (io_sq_tw(&retry_list, IORING_TW_CAP_ENTRIES_VALUE)) sqt_spin = true; if (sqt_spin || !time_after(jiffies, timeout)) { @@ -312,6 +336,9 @@ static int io_sq_thread(void *data) timeout = jiffies + sqd->sq_thread_idle; } + if (retry_list) + io_sq_tw(&retry_list, UINT_MAX); + io_uring_cancel_generic(true, sqd); sqd->thread = NULL; list_for_each_entry(ctx, &sqd->ctx_list, sqd_list)