From patchwork Fri Jun 9 12:20:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 680C2C7EE29 for ; Fri, 9 Jun 2023 12:30:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239305AbjFIMak (ORCPT ); Fri, 9 Jun 2023 08:30:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238967AbjFIMaj (ORCPT ); Fri, 9 Jun 2023 08:30:39 -0400 Received: from out-28.mta0.migadu.com (out-28.mta0.migadu.com [91.218.175.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B9EE210D for ; Fri, 9 Jun 2023 05:29:58 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313242; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UcRk5qlFQvWJ33hcG3kedIDgrONZn1C9jUn4w2g+xQE=; b=pMVV6ttk9cC1bb9hMbPJ2nhJL4+5iM2MZUL2t5mCo0dMDHDYZF2wWs9FwxyxnaF9fsnbYs cAsk4dLBG8y7YTO3GAthr7JjIK7+l1xk+Ah06/k2MtSibvYeefOjjtJrQTa9UkygKoQvL/ PLhjaK4YODsCHyKkGbzrr4aW1I5Ndw8= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 01/11] io-wq: fix worker counting after worker received exit signal Date: Fri, 9 Jun 2023 20:20:21 +0800 Message-Id: <20230609122031.183730-2-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu acct->nr_workers should be decremented when we break the loop in io_wq_worker(). Fixes: 78f8876c2d9f ("io-wq: exclusively gate signal based exit on get_signal() return") Signed-off-by: Hao Xu --- io_uring/io-wq.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index b2715988791e..b70eebec2845 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -634,6 +634,10 @@ static int io_wq_worker(void *data) if (!get_signal(&ksig)) continue; + + raw_spin_lock(&wq->lock); + acct->nr_workers--; + raw_spin_unlock(&wq->lock); break; } if (!ret) { From patchwork Fri Jun 9 12:20:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69767C7EE43 for ; Fri, 9 Jun 2023 12:29:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240381AbjFIM3d (ORCPT ); Fri, 9 Jun 2023 08:29:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231731AbjFIM3c (ORCPT ); Fri, 9 Jun 2023 08:29:32 -0400 Received: from out-28.mta0.migadu.com (out-28.mta0.migadu.com [IPv6:2001:41d0:1004:224b::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 753943AAE for ; Fri, 9 Jun 2023 05:28:53 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lhlblQ2RrpBpwre9osp84qt/3/ewTJ5oMF21HYvR9Hs=; b=PG7m+3mk7k9WkLMmkTo5aVGlW2dDZtz1au1E4n90cZ/jY+iuFotB5lWXFXf0RIwyR5BoqF 4DycA43tlUQUlhry9QTVnLXvdQzo78x3bR6xW5tWi5evcEHfxky1XIYVZWmhVPErXZx+2n tvkWyRPmXla7Ndju6kl4J3+Eu3eSbbA= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 02/11] io-wq: add a new worker flag to indicate worker exit Date: Fri, 9 Jun 2023 20:20:22 +0800 Message-Id: <20230609122031.183730-3-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new worker flag IO_WORKER_F_EXIT to indicate a worker is going to exit. This is important for fixed workers. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index b70eebec2845..1717f1465613 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -29,6 +29,7 @@ enum { IO_WORKER_F_RUNNING = 2, /* account as running */ IO_WORKER_F_FREE = 4, /* worker on free list */ IO_WORKER_F_BOUND = 8, /* is doing bounded work */ + IO_WORKER_F_EXIT = 16, /* worker is exiting */ }; enum { @@ -592,6 +593,11 @@ static void io_worker_handle_work(struct io_worker *worker) } while (1); } +static bool is_worker_exiting(struct io_worker *worker) +{ + return worker->flags & IO_WORKER_F_EXIT; +} + static int io_wq_worker(void *data) { struct io_worker *worker = data; @@ -609,7 +615,7 @@ static int io_wq_worker(void *data) long ret; set_current_state(TASK_INTERRUPTIBLE); - while (io_acct_run_queue(acct)) + while (!is_worker_exiting(worker) && io_acct_run_queue(acct)) io_worker_handle_work(worker); raw_spin_lock(&wq->lock); @@ -628,6 +634,12 @@ static int io_wq_worker(void *data) raw_spin_unlock(&wq->lock); if (io_run_task_work()) continue; + if (is_worker_exiting(worker)) { + raw_spin_lock(&wq->lock); + acct->nr_workers--; + raw_spin_unlock(&wq->lock); + break; + } ret = schedule_timeout(WORKER_IDLE_TIMEOUT); if (signal_pending(current)) { struct ksignal ksig; From patchwork Fri Jun 9 12:20:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E40B8C7EE43 for ; Fri, 9 Jun 2023 12:34:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231172AbjFIMeH (ORCPT ); Fri, 9 Jun 2023 08:34:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239889AbjFIMeG (ORCPT ); Fri, 9 Jun 2023 08:34:06 -0400 Received: from out-20.mta0.migadu.com (out-20.mta0.migadu.com [IPv6:2001:41d0:1004:224b::14]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6630D2D74 for ; Fri, 9 Jun 2023 05:33:34 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7lBQguwOSNqTMHDu49k7jJ6U7XUSmaL2MWuon1bNrYI=; b=pW0zezHFuhSizOXJSgU9qqdjxlwEH8tCtCuhBQudYD5y/0wQKdvDtjzCLUQs8pSOjPfvgI J77gP4XhXBIseej7ddg5mIKh+cleNsnJOVZBx+FX1/OKwgyYVXZZUBUQ/JEZK4tp4nwVFm vAWsLK4aO8+HrAhW1m9i13XZTA97i0I= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 03/11] io-wq: add a new type io-wq worker Date: Fri, 9 Jun 2023 20:20:23 +0800 Message-Id: <20230609122031.183730-4-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new type io-wq worker IO_WORKER_F_FIXED, this type of worker exists during the whole io-wq lifecycle. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 1717f1465613..7326fef58ca7 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -30,6 +30,7 @@ enum { IO_WORKER_F_FREE = 4, /* worker on free list */ IO_WORKER_F_BOUND = 8, /* is doing bounded work */ IO_WORKER_F_EXIT = 16, /* worker is exiting */ + IO_WORKER_F_FIXED = 32, /* is a fixed worker */ }; enum { @@ -598,6 +599,11 @@ static bool is_worker_exiting(struct io_worker *worker) return worker->flags & IO_WORKER_F_EXIT; } +static bool is_fixed_worker(struct io_worker *worker) +{ + return worker->flags & IO_WORKER_F_FIXED; +} + static int io_wq_worker(void *data) { struct io_worker *worker = data; @@ -622,8 +628,13 @@ static int io_wq_worker(void *data) /* * Last sleep timed out. Exit if we're not the last worker, * or if someone modified our affinity. + * Note: fixed worker always have same lifecycle as io-wq + * itself, and cpu affinity setting doesn't work well for + * fixed worker, they can be manually reset to cpu other than + * the cpuset indicated by io_wq_worker_affinity() */ - if (last_timeout && (exit_mask || acct->nr_workers > 1)) { + if (!is_fixed_worker(worker) && last_timeout && + (exit_mask || acct->nr_workers > 1)) { acct->nr_workers--; raw_spin_unlock(&wq->lock); __set_current_state(TASK_RUNNING); From patchwork Fri Jun 9 12:20:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273878 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F1CEC7EE29 for ; Fri, 9 Jun 2023 12:34:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231244AbjFIMeH (ORCPT ); Fri, 9 Jun 2023 08:34:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239476AbjFIMeG (ORCPT ); Fri, 9 Jun 2023 08:34:06 -0400 Received: from out-28.mta0.migadu.com (out-28.mta0.migadu.com [IPv6:2001:41d0:1004:224b::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 664F730D0 for ; Fri, 9 Jun 2023 05:33:34 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2vOAGi5rd15wmqnwQoNSWtL1Awzw5b0eMRdqHWf5p8g=; b=VgYamFTzVfz9nbUpCPpp/UomqbB6O1fXrkO7TPsto5XpIe7rWOkBI7RzALJFCHuPoFkUOr r50cPRjwW+HGz8cV7i/PMx8TKrbobyR2mhuM6y/Lm60jJImbRhhBYVEwhAcSBoAkt9B3gx vAoHxIsPbm7U3H4JIYN3hoaHh3gXF/I= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 04/11] io-wq: add fixed worker members in io_wq_acct Date: Fri, 9 Jun 2023 20:20:24 +0800 Message-Id: <20230609122031.183730-5-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add fixed worker related members in io_wq_acct. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 7326fef58ca7..bf9e9af8d9ca 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -84,6 +84,8 @@ struct io_wq_acct { raw_spinlock_t lock; struct io_wq_work_list work_list; unsigned long flags; + struct io_worker **fixed_workers; + unsigned int fixed_nr; }; enum { From patchwork Fri Jun 9 12:20:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273866 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CADBBC7EE25 for ; Fri, 9 Jun 2023 12:29:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240446AbjFIM3g (ORCPT ); Fri, 9 Jun 2023 08:29:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238775AbjFIM3d (ORCPT ); Fri, 9 Jun 2023 08:29:33 -0400 Received: from out-20.mta0.migadu.com (out-20.mta0.migadu.com [IPv6:2001:41d0:1004:224b::14]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1618A3C15 for ; Fri, 9 Jun 2023 05:28:54 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313253; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gOeA1+BEjODR6KaxqgAFNYJEiEtPFnsNFz55d6FhHLQ=; b=HR+C9pO9iZ1f5ZPg9Xqy3mkfRvANR+P+ERMBkjxDEj5ZBf+wH+rm8NVWcvcQY7Bs27+09c wsCDfcg1VDG13xC3lGdadketFdmBUdQhVbc34ZmyKgIkKrITRineesP6Z2627gvkwAYzi1 qeJU5GMSH95jDI7OooeKk1Aew4ztZmI= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 05/11] io-wq: add a new parameter for creating a new fixed worker Date: Fri, 9 Jun 2023 20:20:25 +0800 Message-Id: <20230609122031.183730-6-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new parameter when creating new workers to indicate if users want a normal or fixed worker. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index bf9e9af8d9ca..048856eef4d4 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -137,7 +137,7 @@ struct io_cb_cancel_data { bool cancel_all; }; -static bool create_io_worker(struct io_wq *wq, int index); +static bool create_io_worker(struct io_wq *wq, int index, bool fixed); static void io_wq_dec_running(struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wq *wq, struct io_wq_acct *acct, @@ -284,7 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq, * We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct) +static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct, + bool fixed) { /* * Most likely an attempt to queue unbounded work on an io_wq that @@ -302,7 +303,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct) raw_spin_unlock(&wq->lock); atomic_inc(&acct->nr_running); atomic_inc(&wq->worker_refs); - return create_io_worker(wq, acct->index); + return create_io_worker(wq, acct->index, fixed); } static void io_wq_inc_running(struct io_worker *worker) @@ -312,6 +313,11 @@ static void io_wq_inc_running(struct io_worker *worker) atomic_inc(&acct->nr_running); } +static bool is_fixed_worker(struct io_worker *worker) +{ + return worker->flags & IO_WORKER_F_FIXED; +} + static void create_worker_cb(struct callback_head *cb) { struct io_worker *worker; @@ -331,7 +337,7 @@ static void create_worker_cb(struct callback_head *cb) } raw_spin_unlock(&wq->lock); if (do_create) { - create_io_worker(wq, worker->create_index); + create_io_worker(wq, worker->create_index, is_fixed_worker(worker)); } else { atomic_dec(&acct->nr_running); io_worker_ref_put(wq); @@ -398,6 +404,8 @@ static void io_wq_dec_running(struct io_worker *worker) return; if (!io_acct_run_queue(acct)) return; + if (is_fixed_worker(worker)) + return; atomic_inc(&acct->nr_running); atomic_inc(&wq->worker_refs); @@ -601,11 +609,6 @@ static bool is_worker_exiting(struct io_worker *worker) return worker->flags & IO_WORKER_F_EXIT; } -static bool is_fixed_worker(struct io_worker *worker) -{ - return worker->flags & IO_WORKER_F_FIXED; -} - static int io_wq_worker(void *data) { struct io_worker *worker = data; @@ -806,7 +809,7 @@ static void io_workqueue_create(struct work_struct *work) kfree(worker); } -static bool create_io_worker(struct io_wq *wq, int index) +static bool create_io_worker(struct io_wq *wq, int index, bool fixed) { struct io_wq_acct *acct = &wq->acct[index]; struct io_worker *worker; @@ -833,10 +836,14 @@ static bool create_io_worker(struct io_wq *wq, int index) if (index == IO_WQ_ACCT_BOUND) worker->flags |= IO_WORKER_F_BOUND; + if (fixed) + worker->flags |= IO_WORKER_F_FIXED; + tsk = create_io_thread(io_wq_worker, worker, NUMA_NO_NODE); if (!IS_ERR(tsk)) { - io_init_new_worker(wq, worker, tsk); - } else if (!io_should_retry_thread(PTR_ERR(tsk))) { + if (!fixed) + io_init_new_worker(wq, worker, tsk); + } else if (fixed || !io_should_retry_thread(PTR_ERR(tsk))) { kfree(worker); goto fail; } else { @@ -947,7 +954,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work) !atomic_read(&acct->nr_running))) { bool did_create; - did_create = io_wq_create_worker(wq, acct); + did_create = io_wq_create_worker(wq, acct, false); if (likely(did_create)) return; From patchwork Fri Jun 9 12:20:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D431AC7EE29 for ; Fri, 9 Jun 2023 12:29:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239626AbjFIM3b (ORCPT ); Fri, 9 Jun 2023 08:29:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238515AbjFIM3b (ORCPT ); Fri, 9 Jun 2023 08:29:31 -0400 X-Greylist: delayed 491 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Fri, 09 Jun 2023 05:28:52 PDT Received: from out-21.mta0.migadu.com (out-21.mta0.migadu.com [IPv6:2001:41d0:1004:224b::15]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E6463AA9 for ; Fri, 9 Jun 2023 05:28:52 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313257; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JbJwvURb/ngoXMSoDDhj8OOS8eskuwHDmLEoYpaKtzY=; b=nl0yV3eF9RH4oDKrzbNmYPK+4odpxcdtUcoRPooL+gMdNsBqA5ltAnUBZ+3Xjs0vOKlrZl zjzVsvwvSq3hfyr1f8bnafVW5yiG617MaqLFUBiHIb4wCqGW/ZBO5PMWY8Ny3t0Gu11PPW V+6HWW8DiO6n+Ykg57wJzb+JGw/FOFY= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 06/11] io-wq: return io_worker after successful inline worker creation Date: Fri, 9 Jun 2023 20:20:26 +0800 Message-Id: <20230609122031.183730-7-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu After creating a io worker inline successfully, return the io_worker structure. This is used by fixed worker. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 048856eef4d4..4338e5b23b07 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -137,7 +137,7 @@ struct io_cb_cancel_data { bool cancel_all; }; -static bool create_io_worker(struct io_wq *wq, int index, bool fixed); +static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixed); static void io_wq_dec_running(struct io_worker *worker); static bool io_acct_cancel_pending_work(struct io_wq *wq, struct io_wq_acct *acct, @@ -284,8 +284,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq, * We need a worker. If we find a free one, we're good. If not, and we're * below the max number of workers, create one. */ -static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct, - bool fixed) +static struct io_worker *io_wq_create_worker(struct io_wq *wq, + struct io_wq_acct *acct, bool fixed) { /* * Most likely an attempt to queue unbounded work on an io_wq that @@ -297,7 +297,7 @@ static bool io_wq_create_worker(struct io_wq *wq, struct io_wq_acct *acct, raw_spin_lock(&wq->lock); if (acct->nr_workers >= acct->max_workers) { raw_spin_unlock(&wq->lock); - return true; + return NULL; } acct->nr_workers++; raw_spin_unlock(&wq->lock); @@ -809,11 +809,11 @@ static void io_workqueue_create(struct work_struct *work) kfree(worker); } -static bool create_io_worker(struct io_wq *wq, int index, bool fixed) +static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixed) { struct io_wq_acct *acct = &wq->acct[index]; struct io_worker *worker; - struct task_struct *tsk; + struct task_struct *tsk = NULL; __set_current_state(TASK_RUNNING); @@ -825,7 +825,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) acct->nr_workers--; raw_spin_unlock(&wq->lock); io_worker_ref_put(wq); - return false; + return tsk ? (struct io_worker *)tsk : ERR_PTR(-ENOMEM); } refcount_set(&worker->ref, 1); @@ -841,8 +841,8 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) tsk = create_io_thread(io_wq_worker, worker, NUMA_NO_NODE); if (!IS_ERR(tsk)) { - if (!fixed) - io_init_new_worker(wq, worker, tsk); + io_init_new_worker(wq, worker, tsk); + return worker; } else if (fixed || !io_should_retry_thread(PTR_ERR(tsk))) { kfree(worker); goto fail; @@ -851,7 +851,7 @@ static bool create_io_worker(struct io_wq *wq, int index, bool fixed) schedule_work(&worker->work); } - return true; + return (struct io_worker *)tsk; } /* From patchwork Fri Jun 9 12:20:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273868 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D02A5C83005 for ; Fri, 9 Jun 2023 12:29:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231731AbjFIM3i (ORCPT ); Fri, 9 Jun 2023 08:29:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39864 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240506AbjFIM3e (ORCPT ); Fri, 9 Jun 2023 08:29:34 -0400 Received: from out-46.mta0.migadu.com (out-46.mta0.migadu.com [IPv6:2001:41d0:1004:224b::2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 636233C19 for ; Fri, 9 Jun 2023 05:28:56 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=a5OtU3AeT801fzenuZasRsfvUafYbVDhac6dEZreRlU=; b=bXyer4xJN+MgIvHx/TnVl8zyTaENMHEEegnOgqtNs9ZMVEFu1G2UN5EBUHqsO0MOVOeFsM /8fT10lSyHYnZDyrjRywB1FTHZnCGck989Uzkbpu1tJv4CNHtUuN908MpGce5nDEkxYrVT a/wT3BDpEthsz3iFJc+8crag3sYtbxs= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 07/11] io_uring: add new api to register fixed workers Date: Fri, 9 Jun 2023 20:20:27 +0800 Message-Id: <20230609122031.183730-8-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new api to register fixed workers. The api is designed to register fixed workers for the current task. For simplicity, it doesn't allow worker number update. We have a separate unregister api to uninstall all the fixed workers. And then we can register different number of fixed workers again. Signed-off-by: Hao Xu --- include/uapi/linux/io_uring.h | 9 ++++ io_uring/io-wq.c | 85 +++++++++++++++++++++++++++++++++++ io_uring/io-wq.h | 1 + io_uring/io_uring.c | 71 +++++++++++++++++++++++++++++ 4 files changed, 166 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index f222d263bc55..6dc43be5009d 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -535,6 +535,9 @@ enum { /* register a range of fixed file slots for automatic slot allocation */ IORING_REGISTER_FILE_ALLOC_RANGE = 25, + /* set/get number of fixed workers */ + IORING_REGISTER_IOWQ_FIXED_WORKERS = 26, + /* this goes last */ IORING_REGISTER_LAST, @@ -715,6 +718,12 @@ struct io_uring_recvmsg_out { __u32 flags; }; +struct io_uring_fixed_worker_arg { + __u32 nr_workers; + __u32 resv; + __u64 resv2[3]; +}; + #ifdef __cplusplus } #endif diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 4338e5b23b07..28f13c1c38f4 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -1371,6 +1371,91 @@ int io_wq_max_workers(struct io_wq *wq, int *new_count) return 0; } +static void io_wq_clean_fixed_workers(struct io_wq *wq) +{ + int i, j; + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + struct io_wq_acct *acct = &wq->acct[i]; + struct io_worker **workers = acct->fixed_workers; + + if (!workers) + continue; + + for (j = 0; j < acct->fixed_nr; j++) { + if (!workers[j]) + continue; + workers[j]->flags |= IO_WORKER_F_EXIT; + wake_up_process(worker->task); + } + kfree(workers); + } +} + +/* + * Set number of fixed workers. + */ +int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *count) +{ + struct io_wq_acct *acct; + int i, j, ret = 0; + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + if (wq->acct[i].fixed_nr) { + ret = -EBUSY; + break; + } + } + if (ret) + return ret; + + BUILD_BUG_ON((int) IO_WQ_ACCT_BOUND != (int) IO_WQ_BOUND); + BUILD_BUG_ON((int) IO_WQ_ACCT_UNBOUND != (int) IO_WQ_UNBOUND); + BUILD_BUG_ON((int) IO_WQ_ACCT_NR != 2); + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + if (count[i].nr_workers > task_rlimit(current, RLIMIT_NPROC)) + count[i].nr_workers = + task_rlimit(current, RLIMIT_NPROC); + } + + rcu_read_lock(); + + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + unsigned int nr = count[i].nr_workers; + + acct = &wq->acct[i]; + acct->fixed_nr = nr; + acct->fixed_workers = kcalloc(nr, sizeof(struct io_worker *), + GFP_KERNEL); + if (!acct->fixed_workers) { + ret = -ENOMEM; + break; + } + + for (j = 0; j < nr; j++) { + struct io_worker *worker = + io_wq_create_worker(wq, acct, true); + if (IS_ERR(worker)) { + ret = PTR_ERR(worker); + break; + } + acct->fixed_workers[j] = worker; + } + if (j < nr) + break; + } + rcu_read_unlock(); + + if (ret) + goto err; + return 0; + +err: + io_wq_clean_fixed_workers(wq); + return ret; +} + static __init int io_wq_init(void) { int ret; diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 31228426d192..88a1ee9fde24 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -52,6 +52,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val); int io_wq_cpu_affinity(struct io_wq *wq, cpumask_var_t mask); int io_wq_max_workers(struct io_wq *wq, int *new_count); +int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *count); static inline bool io_wq_is_hashed(struct io_wq_work *work) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index c99a7a0c3f21..bb8342b4a2c6 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -4351,6 +4351,71 @@ static __cold int io_register_iowq_max_workers(struct io_ring_ctx *ctx, return ret; } +/* + * note: this function sets fixed workers for a single task, so every + * task which wants to set the fixed workers has to call this function + */ +static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx, + void __user *arg, int nr_args) + __must_hold(&ctx->uring_lock) +{ + struct io_uring_task *tctx = NULL; + struct io_sq_data *sqd = NULL; + struct io_uring_fixed_worker_arg *res; + size_t size; + int i, ret; + bool zero = true; + + size = array_size(nr_args, sizeof(*res)); + if (size == SIZE_MAX) + return -EOVERFLOW; + + res = memdup_user(arg, size); + if (IS_ERR(res)) + return PTR_ERR(res); + + for (i = 0; i < nr_args; i++) { + if (res[i].nr_workers) { + zero = false; + break; + } + } + + if (zero) + return 0; + + if (ctx->flags & IORING_SETUP_SQPOLL) { + sqd = ctx->sq_data; + if (sqd) { + /* + * Observe the correct sqd->lock -> ctx->uring_lock + * ordering. Fine to drop uring_lock here, we hold + * a ref to the ctx. + */ + refcount_inc(&sqd->refs); + mutex_unlock(&ctx->uring_lock); + mutex_lock(&sqd->lock); + mutex_lock(&ctx->uring_lock); + if (sqd->thread) + tctx = sqd->thread->io_uring; + } + } else { + tctx = current->io_uring; + } + + if (tctx && tctx->io_wq) + ret = io_wq_fixed_workers(tctx->io_wq, res); + else + ret = -EFAULT; + + if (sqd) { + mutex_unlock(&sqd->lock); + io_put_sq_data(sqd); + } + + return ret; +} + static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -4509,6 +4574,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_file_alloc_range(ctx, arg); break; + case IORING_REGISTER_IOWQ_FIXED_WORKERS: + ret = -EINVAL; + if (!arg || nr_args != 2) + break; + ret = io_register_iowq_fixed_workers(ctx, arg, nr_args); + break; default: ret = -EINVAL; break; From patchwork Fri Jun 9 12:20:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273870 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5EBEC7EE25 for ; Fri, 9 Jun 2023 12:30:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238967AbjFIMak (ORCPT ); Fri, 9 Jun 2023 08:30:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239082AbjFIMaj (ORCPT ); Fri, 9 Jun 2023 08:30:39 -0400 X-Greylist: delayed 556 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Fri, 09 Jun 2023 05:29:56 PDT Received: from out-21.mta0.migadu.com (out-21.mta0.migadu.com [91.218.175.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0097B1FE9 for ; Fri, 9 Jun 2023 05:29:56 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313262; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gaD/cBBg+X5hxLR5Nw1m3BpyHfWuwSitgkQ8D/QMqj4=; b=JEwQ1VgC7K/GGROyMsqttASO9IN751Mi+cZEJnurv/sCUJytX/t+3xjkatsL0c0Pe5i5uN 7Y5Y8Ti5us3yHBzP+iIzOL0DutN6q3OR4qAYjUK5Mcs5lGBzNzE5WxcNvqn+mPCbt4idCP 1ErXvrK2LvSl+AvtJc0cFCSM84uhljo= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 08/11] io_uring: add function to unregister fixed workers Date: Fri, 9 Jun 2023 20:20:28 +0800 Message-Id: <20230609122031.183730-9-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new register api to unregister fixed workers. Signed-off-by: Hao Xu --- include/uapi/linux/io_uring.h | 3 +++ io_uring/io-wq.c | 50 ++++++++++++++++++++++++++++++++++- io_uring/io-wq.h | 1 + io_uring/io_uring.c | 45 +++++++++++++++++++++++++++++++ 4 files changed, 98 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 6dc43be5009d..b0a6e3106b42 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -538,6 +538,9 @@ enum { /* set/get number of fixed workers */ IORING_REGISTER_IOWQ_FIXED_WORKERS = 26, + /* destroy fixed workers */ + IORING_UNREGISTER_IOWQ_FIXED_WORKERS = 27, + /* this goes last */ IORING_REGISTER_LAST, diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 28f13c1c38f4..f39e6b931d17 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -1386,7 +1386,7 @@ static void io_wq_clean_fixed_workers(struct io_wq *wq) if (!workers[j]) continue; workers[j]->flags |= IO_WORKER_F_EXIT; - wake_up_process(worker->task); + wake_up_process(workers[j]->task); } kfree(workers); } @@ -1456,6 +1456,54 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *coun return ret; } +/* + * destroy fixed workers. + */ +int io_wq_destroy_fixed_workers(struct io_wq *wq) +{ + int i, j; + + raw_spin_lock(&wq->lock); + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + if (wq->acct[i].fixed_nr) + break; + } + raw_spin_unlock(&wq->lock); + if (i == IO_WQ_ACCT_NR) + return -EFAULT; + + BUILD_BUG_ON((int) IO_WQ_ACCT_BOUND != (int) IO_WQ_BOUND); + BUILD_BUG_ON((int) IO_WQ_ACCT_UNBOUND != (int) IO_WQ_UNBOUND); + BUILD_BUG_ON((int) IO_WQ_ACCT_NR != 2); + + rcu_read_lock(); + raw_spin_lock(&wq->lock); + for (i = 0; i < IO_WQ_ACCT_NR; i++) { + struct io_wq_acct *acct = &wq->acct[i]; + struct io_worker **workers = acct->fixed_workers; + unsigned int nr = acct->fixed_nr; + + if (!nr) + continue; + + for (j = 0; j < nr; j++) { + struct io_worker *worker = workers[j]; + + BUG_ON(!worker); + BUG_ON(!worker->task); + + workers[j]->flags |= IO_WORKER_F_EXIT; + wake_up_process(worker->task); + } + // wait for all workers exit + kfree(workers); + } + raw_spin_unlock(&wq->lock); + rcu_read_unlock(); + + return 0; +} + static __init int io_wq_init(void) { int ret; diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 88a1ee9fde24..15e93af36511 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -53,6 +53,7 @@ void io_wq_hash_work(struct io_wq_work *work, void *val); int io_wq_cpu_affinity(struct io_wq *wq, cpumask_var_t mask); int io_wq_max_workers(struct io_wq *wq, int *new_count); int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *count); +int io_wq_destroy_fixed_workers(struct io_wq *wq); static inline bool io_wq_is_hashed(struct io_wq_work *work) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bb8342b4a2c6..b37224cc1d05 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -4416,6 +4416,45 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx, return ret; } +static __cold int io_unregister_iowq_fixed_workers(struct io_ring_ctx *ctx) + __must_hold(&ctx->uring_lock) +{ + struct io_uring_task *tctx = NULL; + struct io_sq_data *sqd = NULL; + int ret; + + if (ctx->flags & IORING_SETUP_SQPOLL) { + sqd = ctx->sq_data; + if (sqd) { + /* + * Observe the correct sqd->lock -> ctx->uring_lock + * ordering. Fine to drop uring_lock here, we hold + * a ref to the ctx. + */ + refcount_inc(&sqd->refs); + mutex_unlock(&ctx->uring_lock); + mutex_lock(&sqd->lock); + mutex_lock(&ctx->uring_lock); + if (sqd->thread) + tctx = sqd->thread->io_uring; + } + } else { + tctx = current->io_uring; + } + + if (tctx && tctx->io_wq) + ret = io_wq_destroy_fixed_workers(tctx->io_wq); + else + ret = -EFAULT; + + if (sqd) { + mutex_unlock(&sqd->lock); + io_put_sq_data(sqd); + } + + return ret; +} + static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -4580,6 +4619,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_iowq_fixed_workers(ctx, arg, nr_args); break; + case IORING_UNREGISTER_IOWQ_FIXED_WORKERS: + ret = -EINVAL; + if (arg || nr_args) + break; + ret = io_unregister_iowq_fixed_workers(ctx); + break; default: ret = -EINVAL; break; From patchwork Fri Jun 9 12:20:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D8D0C7EE25 for ; Fri, 9 Jun 2023 12:30:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231612AbjFIMas (ORCPT ); Fri, 9 Jun 2023 08:30:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239082AbjFIMar (ORCPT ); Fri, 9 Jun 2023 08:30:47 -0400 Received: from out-28.mta0.migadu.com (out-28.mta0.migadu.com [91.218.175.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7B3E4273D for ; Fri, 9 Jun 2023 05:30:08 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313265; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GUYNkZQUX2Qc9IRW4xhjXCnw3AmWpQKAecn+j1go0VI=; b=NtanpO1rfJFF0k53nnSadLV7httda9WdZlyRG6wglgjI+29LrKDznS1Xvz5MN0++5ih4Wq ANKU8/ER8g0ZuHsS3oMeNlVixE1HH7oTdya8qKNLESFg8rHTuYvGy0CzGu/qeYkzUFYCUC XAW7AJm0xHWd6BWTxjd43DsyfzFG7rM= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 09/11] io-wq: add strutures to allow to wait fixed workers exit Date: Fri, 9 Jun 2023 20:20:29 +0800 Message-Id: <20230609122031.183730-10-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu When unregister fixed workers, there should be a way to allow us to wait all the fixed workers exit. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 72 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 20 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index f39e6b931d17..61cf6da2c72f 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -108,6 +108,10 @@ struct io_wq { atomic_t worker_refs; struct completion worker_done; + atomic_t fixed_worker_refs; + struct completion fixed_worker_done; + bool fixed_comp_init; + struct hlist_node cpuhp_node; struct task_struct *task; @@ -172,10 +176,25 @@ static inline struct io_wq_acct *io_wq_get_acct(struct io_worker *worker) return io_get_acct(worker->wq, worker->flags & IO_WORKER_F_BOUND); } -static void io_worker_ref_put(struct io_wq *wq) +static void io_worker_ref_get(struct io_wq *wq, bool fixed) +{ + atomic_inc(&wq->worker_refs); + if (fixed) + atomic_inc(&wq->fixed_worker_refs); +} + +static void io_worker_ref_put(struct io_wq *wq, bool fixed) { if (atomic_dec_and_test(&wq->worker_refs)) complete(&wq->worker_done); + + if (fixed && atomic_dec_and_test(&wq->fixed_worker_refs)) + complete(&wq->fixed_worker_done); +} + +static bool is_fixed_worker(struct io_worker *worker) +{ + return worker->flags & IO_WORKER_F_FIXED; } static void io_worker_cancel_cb(struct io_worker *worker) @@ -187,7 +206,7 @@ static void io_worker_cancel_cb(struct io_worker *worker) raw_spin_lock(&wq->lock); acct->nr_workers--; raw_spin_unlock(&wq->lock); - io_worker_ref_put(wq); + io_worker_ref_put(wq, is_fixed_worker(worker)); clear_bit_unlock(0, &worker->create_state); io_worker_release(worker); } @@ -205,6 +224,7 @@ static bool io_task_worker_match(struct callback_head *cb, void *data) static void io_worker_exit(struct io_worker *worker) { struct io_wq *wq = worker->wq; + bool fixed = is_fixed_worker(worker); while (1) { struct callback_head *cb = task_work_cancel_match(wq->task, @@ -230,7 +250,7 @@ static void io_worker_exit(struct io_worker *worker) preempt_enable(); kfree_rcu(worker, rcu); - io_worker_ref_put(wq); + io_worker_ref_put(wq, fixed); do_exit(0); } @@ -302,7 +322,7 @@ static struct io_worker *io_wq_create_worker(struct io_wq *wq, acct->nr_workers++; raw_spin_unlock(&wq->lock); atomic_inc(&acct->nr_running); - atomic_inc(&wq->worker_refs); + io_worker_ref_get(wq, fixed); return create_io_worker(wq, acct->index, fixed); } @@ -313,11 +333,6 @@ static void io_wq_inc_running(struct io_worker *worker) atomic_inc(&acct->nr_running); } -static bool is_fixed_worker(struct io_worker *worker) -{ - return worker->flags & IO_WORKER_F_FIXED; -} - static void create_worker_cb(struct callback_head *cb) { struct io_worker *worker; @@ -325,8 +340,10 @@ static void create_worker_cb(struct callback_head *cb) struct io_wq_acct *acct; bool do_create = false; + bool fixed; worker = container_of(cb, struct io_worker, create_work); + fixed = is_fixed_worker(worker); wq = worker->wq; acct = &wq->acct[worker->create_index]; raw_spin_lock(&wq->lock); @@ -337,10 +354,10 @@ static void create_worker_cb(struct callback_head *cb) } raw_spin_unlock(&wq->lock); if (do_create) { - create_io_worker(wq, worker->create_index, is_fixed_worker(worker)); + create_io_worker(wq, worker->create_index, fixed); } else { atomic_dec(&acct->nr_running); - io_worker_ref_put(wq); + io_worker_ref_put(wq, fixed); } clear_bit_unlock(0, &worker->create_state); io_worker_release(worker); @@ -351,6 +368,7 @@ static bool io_queue_worker_create(struct io_worker *worker, task_work_func_t func) { struct io_wq *wq = worker->wq; + bool fixed = is_fixed_worker(worker); /* raced with exit, just ignore create call */ if (test_bit(IO_WQ_BIT_EXIT, &wq->state)) @@ -367,7 +385,7 @@ static bool io_queue_worker_create(struct io_worker *worker, test_and_set_bit_lock(0, &worker->create_state)) goto fail_release; - atomic_inc(&wq->worker_refs); + io_worker_ref_get(wq, fixed); init_task_work(&worker->create_work, func); worker->create_index = acct->index; if (!task_work_add(wq->task, &worker->create_work, TWA_SIGNAL)) { @@ -379,16 +397,16 @@ static bool io_queue_worker_create(struct io_worker *worker, */ if (test_bit(IO_WQ_BIT_EXIT, &wq->state)) io_wq_cancel_tw_create(wq); - io_worker_ref_put(wq); + io_worker_ref_put(wq, fixed); return true; } - io_worker_ref_put(wq); + io_worker_ref_put(wq, fixed); clear_bit_unlock(0, &worker->create_state); fail_release: io_worker_release(worker); fail: atomic_dec(&acct->nr_running); - io_worker_ref_put(wq); + io_worker_ref_put(wq, fixed); return false; } @@ -408,7 +426,7 @@ static void io_wq_dec_running(struct io_worker *worker) return; atomic_inc(&acct->nr_running); - atomic_inc(&wq->worker_refs); + io_worker_ref_get(wq, false); io_queue_worker_create(worker, acct, create_worker_cb); } @@ -790,7 +808,7 @@ static void create_worker_cont(struct callback_head *cb) } else { raw_spin_unlock(&wq->lock); } - io_worker_ref_put(wq); + io_worker_ref_put(wq, is_fixed_worker(worker)); kfree(worker); return; } @@ -824,7 +842,7 @@ static struct io_worker *create_io_worker(struct io_wq *wq, int index, bool fixe raw_spin_lock(&wq->lock); acct->nr_workers--; raw_spin_unlock(&wq->lock); - io_worker_ref_put(wq); + io_worker_ref_put(wq, fixed); return tsk ? (struct io_worker *)tsk : ERR_PTR(-ENOMEM); } @@ -1243,7 +1261,7 @@ static void io_wq_exit_workers(struct io_wq *wq) rcu_read_lock(); io_wq_for_each_worker(wq, io_wq_worker_wake, NULL); rcu_read_unlock(); - io_worker_ref_put(wq); + io_worker_ref_put(wq, false); wait_for_completion(&wq->worker_done); spin_lock_irq(&wq->hash->wait.lock); @@ -1390,6 +1408,7 @@ static void io_wq_clean_fixed_workers(struct io_wq *wq) } kfree(workers); } + wait_for_completion(&wq->fixed_worker_done); } /* @@ -1421,6 +1440,13 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *coun rcu_read_lock(); + atomic_set(&wq->fixed_worker_refs, 1); + if (wq->fixed_comp_init) { + reinit_completion(&wq->fixed_worker_done); + } else { + init_completion(&wq->fixed_worker_done); + wq->fixed_comp_init = true; + } for (i = 0; i < IO_WQ_ACCT_NR; i++) { unsigned int nr = count[i].nr_workers; @@ -1495,12 +1521,18 @@ int io_wq_destroy_fixed_workers(struct io_wq *wq) workers[j]->flags |= IO_WORKER_F_EXIT; wake_up_process(worker->task); } - // wait for all workers exit kfree(workers); } raw_spin_unlock(&wq->lock); rcu_read_unlock(); + // decrement the init reference + if (atomic_dec_and_test(&wq->fixed_worker_refs)) + complete(&wq->fixed_worker_done); + + wait_for_completion(&wq->fixed_worker_done); + wq->fixed_comp_init = false; + return 0; } From patchwork Fri Jun 9 12:20:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A9E0C7EE43 for ; Fri, 9 Jun 2023 12:29:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240167AbjFIM3i (ORCPT ); Fri, 9 Jun 2023 08:29:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39858 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231731AbjFIM3e (ORCPT ); Fri, 9 Jun 2023 08:29:34 -0400 Received: from out-46.mta0.migadu.com (out-46.mta0.migadu.com [IPv6:2001:41d0:1004:224b::2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32DA03C1F for ; Fri, 9 Jun 2023 05:28:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M36Jp2AxZkBfLvbvl2HSzo9LX0/6SxFY++SadI4YEpU=; b=sH53kXNSjeKiQd27BwhD2i4QH+Ooj4w4w449tiId73+FG0jrmBblPJH22ZDaK3att8tx88 XWssYBEqvN10UD4UgxptngknKn9DLkz7SVeonq5Ce5bTwPxGQ7EKjOzuhI/6ixBGJufPbm VG3T87RasiNjaKcD1xaVcVyMisvpL6Q= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 10/11] io-wq: distinguish fixed worker by its name Date: Fri, 9 Jun 2023 20:20:30 +0800 Message-Id: <20230609122031.183730-11-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Distinguish fixed workers and normal workers by their names. Signed-off-by: Hao Xu --- io_uring/io-wq.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 61cf6da2c72f..7a9e5fa19b81 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -634,10 +634,12 @@ static int io_wq_worker(void *data) struct io_wq *wq = worker->wq; bool exit_mask = false, last_timeout = false; char buf[TASK_COMM_LEN]; + bool fixed = is_fixed_worker(worker); worker->flags |= (IO_WORKER_F_UP | IO_WORKER_F_RUNNING); - snprintf(buf, sizeof(buf), "iou-wrk-%d", wq->task->pid); + snprintf(buf, sizeof(buf), fixed ? "iou-fixed-%d" : "iou-wrk-%d", + wq->task->pid); set_task_comm(current, buf); while (!test_bit(IO_WQ_BIT_EXIT, &wq->state)) { @@ -656,7 +658,7 @@ static int io_wq_worker(void *data) * fixed worker, they can be manually reset to cpu other than * the cpuset indicated by io_wq_worker_affinity() */ - if (!is_fixed_worker(worker) && last_timeout && + if (!fixed && last_timeout && (exit_mask || acct->nr_workers > 1)) { acct->nr_workers--; raw_spin_unlock(&wq->lock); From patchwork Fri Jun 9 12:20:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hao Xu X-Patchwork-Id: 13273865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA4E2C7EE29 for ; Fri, 9 Jun 2023 12:29:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240524AbjFIM3g (ORCPT ); Fri, 9 Jun 2023 08:29:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240531AbjFIM3d (ORCPT ); Fri, 9 Jun 2023 08:29:33 -0400 Received: from out-28.mta0.migadu.com (out-28.mta0.migadu.com [IPv6:2001:41d0:1004:224b::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D91473C14 for ; Fri, 9 Jun 2023 05:28:54 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1686313270; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CbfluvVmDKK5YgTgobgtoj9vZ7mG0cgUcWql500V/a8=; b=Ne6l0UK0g9XrDmTYsKwrL+MyUfHCmdsY0TU31AneYWW7+maZ4wfM/aM3oAxUzHZh4clgMF QDR3aSzxMqYOVGKrOZrzSGsnCmyl2v31HAyg820hgwYAoVBGUURAxcthv7b0qaYStOMxqX 05x+fJzy+h5Yg0do6qyPiVeqB4t/SWU= From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , Wanpeng Li , linux-fsdevel@vger.kernel.org Subject: [PATCH 11/11] io_uring: add IORING_SETUP_FIXED_WORKER_ONLY and its friend Date: Fri, 9 Jun 2023 20:20:31 +0800 Message-Id: <20230609122031.183730-12-hao.xu@linux.dev> In-Reply-To: <20230609122031.183730-1-hao.xu@linux.dev> References: <20230609122031.183730-1-hao.xu@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new setup flag to indicate that the uring instance only use fixed workers as async offload threads. Add a work flag and its code logic as well. Signed-off-by: Hao Xu --- include/uapi/linux/io_uring.h | 10 +++++++++- io_uring/io-wq.c | 18 +++++++++++++----- io_uring/io-wq.h | 1 + io_uring/io_uring.c | 24 +++++++++++++++++++----- 4 files changed, 42 insertions(+), 11 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index b0a6e3106b42..900fedaa5692 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -185,6 +185,11 @@ enum { */ #define IORING_SETUP_REGISTERED_FD_ONLY (1U << 15) +/* + * this ring instance only use fixed worker for async offload. + */ +#define IORING_SETUP_FIXED_WORKER_ONLY (1U << 16) + enum io_uring_op { IORING_OP_NOP, IORING_OP_READV, @@ -721,9 +726,12 @@ struct io_uring_recvmsg_out { __u32 flags; }; +#define IORING_FIXED_WORKER_F_ONLY (1U << 0) +#define IORING_FIXED_WORKER_F_VALID (IORING_FIXED_WORKER_F_ONLY) + struct io_uring_fixed_worker_arg { __u32 nr_workers; - __u32 resv; + __u32 flags; __u64 resv2[3]; }; diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 7a9e5fa19b81..98a16abb2944 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -272,7 +272,7 @@ static inline bool io_acct_run_queue(struct io_wq_acct *acct) * caller must create one. */ static bool io_wq_activate_free_worker(struct io_wq *wq, - struct io_wq_acct *acct) + struct io_wq_acct *acct, bool fixed) __must_hold(RCU) { struct hlist_nulls_node *n; @@ -286,7 +286,8 @@ static bool io_wq_activate_free_worker(struct io_wq *wq, hlist_nulls_for_each_entry_rcu(worker, n, &wq->free_list, nulls_node) { if (!io_worker_get(worker)) continue; - if (io_wq_get_acct(worker) != acct) { + if (io_wq_get_acct(worker) != acct || + (fixed && !is_fixed_worker(worker))) { io_worker_release(worker); continue; } @@ -492,6 +493,9 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct, work = container_of(node, struct io_wq_work, list); + if ((work->flags & IO_WQ_WORK_FIXED) && !is_fixed_worker(worker)) + continue; + /* not hashed, can run anytime */ if (!io_wq_is_hashed(work)) { wq_list_del(&acct->work_list, node, prev); @@ -946,7 +950,7 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work) struct io_wq_acct *acct = io_work_get_acct(wq, work); struct io_cb_cancel_data match; unsigned work_flags = work->flags; - bool do_create; + bool do_create, fixed = work_flags & IO_WQ_WORK_FIXED; /* * If io-wq is exiting for this task, or if the request has explicitly @@ -965,11 +969,14 @@ void io_wq_enqueue(struct io_wq *wq, struct io_wq_work *work) raw_spin_lock(&wq->lock); rcu_read_lock(); - do_create = !io_wq_activate_free_worker(wq, acct); + do_create = !io_wq_activate_free_worker(wq, acct, fixed); rcu_read_unlock(); raw_spin_unlock(&wq->lock); + if (fixed) + return; + if (do_create && ((work_flags & IO_WQ_WORK_CONCURRENT) || !atomic_read(&acct->nr_running))) { bool did_create; @@ -1155,7 +1162,7 @@ static int io_wq_hash_wake(struct wait_queue_entry *wait, unsigned mode, struct io_wq_acct *acct = &wq->acct[i]; if (test_and_clear_bit(IO_ACCT_STALLED_BIT, &acct->flags)) - io_wq_activate_free_worker(wq, acct); + io_wq_activate_free_worker(wq, acct, false); } rcu_read_unlock(); return 1; @@ -1477,6 +1484,7 @@ int io_wq_fixed_workers(struct io_wq *wq, struct io_uring_fixed_worker_arg *coun if (ret) goto err; + return 0; err: diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 15e93af36511..d81d5f9aa602 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -11,6 +11,7 @@ enum { IO_WQ_WORK_HASHED = 2, IO_WQ_WORK_UNBOUND = 4, IO_WQ_WORK_CONCURRENT = 16, + IO_WQ_WORK_FIXED = 32, IO_WQ_HASH_SHIFT = 24, /* upper 8 bits are used for hash key */ }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b37224cc1d05..bf8232906605 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -479,6 +479,9 @@ void io_queue_iowq(struct io_kiocb *req, struct io_tw_state *ts_dont_use) if (WARN_ON_ONCE(!same_thread_group(req->task, current))) req->work.flags |= IO_WQ_WORK_CANCEL; + if (req->ctx->flags & IORING_SETUP_FIXED_WORKER_ONLY) + req->work.flags |= IO_WQ_WORK_FIXED; + trace_io_uring_queue_async_work(req, io_wq_is_hashed(&req->work)); io_wq_enqueue(tctx->io_wq, &req->work); if (link) @@ -1971,7 +1974,12 @@ struct io_wq_work *io_wq_free_work(struct io_wq_work *work) struct io_kiocb *req = container_of(work, struct io_kiocb, work); req = io_put_req_find_next(req); - return req ? &req->work : NULL; + if (req) { + req->work.flags |= IO_WQ_WORK_FIXED; + return &req->work; + } + + return NULL; } void io_wq_submit_work(struct io_wq_work *work) @@ -4364,7 +4372,7 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx, struct io_uring_fixed_worker_arg *res; size_t size; int i, ret; - bool zero = true; + bool zero = true, fixed_only = false; size = array_size(nr_args, sizeof(*res)); if (size == SIZE_MAX) @@ -4375,15 +4383,20 @@ static __cold int io_register_iowq_fixed_workers(struct io_ring_ctx *ctx, return PTR_ERR(res); for (i = 0; i < nr_args; i++) { - if (res[i].nr_workers) { + if (res[i].flags & ~IORING_FIXED_WORKER_F_VALID) + return -EINVAL; + if (res[i].flags & IORING_FIXED_WORKER_F_ONLY) + fixed_only = true; + if (res[i].nr_workers) zero = false; - break; - } } if (zero) return 0; + if (fixed_only) + ctx->flags |= IORING_SETUP_FIXED_WORKER_ONLY; + if (ctx->flags & IORING_SETUP_SQPOLL) { sqd = ctx->sq_data; if (sqd) { @@ -4423,6 +4436,7 @@ static __cold int io_unregister_iowq_fixed_workers(struct io_ring_ctx *ctx) struct io_sq_data *sqd = NULL; int ret; + ctx->flags &= ~IORING_SETUP_FIXED_WORKER_ONLY; if (ctx->flags & IORING_SETUP_SQPOLL) { sqd = ctx->sq_data; if (sqd) {