From patchwork Mon Sep 30 20:37:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13817066 Received: from mail-il1-f182.google.com (mail-il1-f182.google.com [209.85.166.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2F432C190 for ; Mon, 30 Sep 2024 20:40:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728827; cv=none; b=HqgsqfiE191NyJAaq4N7aQOY4wBc/CPkpn39an+nGgWjPDyOE/Ld4wBE+wTiX2nIo67QPVOJ9Eoyp35dFiGdBcvJMA7HG1mGns2yT2WBl5uJ5PaZaTuTuZ3bjnJBwtZYxwbEOij2aC8BO4xVPbmUXyRwPiZRLDX3cRC2taCE0BQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728827; c=relaxed/simple; bh=7f3B/9vwhvFQH3WS9Kwb/5kXTSA7kbzx32roevP58ek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h9+Q+Md9D7o4KzTap2DVMgMWiV7dTDkyvQ9CKnUB1e20SUbvNYG7Maxq71ayCUv8pUtiHGqCeUXzJ9p4CpGvl6AzGYqG5cMmutDAHywsOAcmB2CXIRya4oAUJIq7M48t0m9/w3LneEfu+HMiSrrOuuBi1DFMhbLZ7ZOXwZlUx9A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=aYaXsE/Q; arc=none smtp.client-ip=209.85.166.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="aYaXsE/Q" Received: by mail-il1-f182.google.com with SMTP id e9e14a558f8ab-3a3525ba6aaso5803315ab.2 for ; Mon, 30 Sep 2024 13:40:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1727728823; x=1728333623; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uzedQLOXkQza5ujxTJxGLPoQiEEKfSfG+LpallEDVjg=; b=aYaXsE/QTsX8DH8kPYIMi5gbVrZEt2+QZ6GI4XjZpc1gbs0d+vHj+EyU6BLC/8UtSN n4L1qnYlKtqwIfcCkVrzawaWF5Qg7xJgj7b/JI6vaG4z1L3IN76iWSLJrEkY7HN8C6tP LO4JaRolXSLcx8RBU+jSsWsHXZ2QfzFxv5wxYAuCfwZNLSUDba5+170TFLkpasA80mce kiVc9w1mRokBDzb1MzOw2mx2Esq61J8JDjWRERCOF8+oR6cOEWqM6kodJtHLhrl4kBAc 2mcTG83TCERgKJs/t1FyhQtyAze5jSfXMhnFYstjtM9uymXSFTzGcgCRcYK1kJUcaCUE Rp9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727728823; x=1728333623; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uzedQLOXkQza5ujxTJxGLPoQiEEKfSfG+LpallEDVjg=; b=IgfDbu4CxGj+BsLGa3gPpoxNNUzaxwUMDRh+yMJlunejT25guGVz+rrhZlD2lw0/Q5 nCpzzVo+SL5xXVO8rycz08fa+nqx9Xf8SikSU+dKZE9Bg0dPSYpXStL1/+FgycK+CqZM AJJk1ibxiwj0II890E6fwW5SqUymCK7XOSzsP//dZxKanTktoBafrPT/yChXp5wz45n3 bAQwN4eGTTCZotGSIuqph3DkODhZH01y7xt7CECSkqgufzX06GISiE2hDDJsTwoU+7ps 5UheDyFjLeaV0LlCjS8qNP0Q/wF8C/UTWRjBX97jUfErrv8WqTF/MrussoxwIVBLjobE y0lg== X-Gm-Message-State: AOJu0YxKkZ3CyqA/43j6+Sy7XcFTKTlgrEr7M191aRcBQqkJJxrh9Gg4 /0PKhV2qCN047EHnlf6fBK0PCttBHKrh1Vi+YrvIxz79bbxTJ3jSfaIf406199IMBg+Qga/+OYM sefo= X-Google-Smtp-Source: AGHT+IEjt0GzZ6RVff1vWv/hbhrJHRkN4YWl3iBXXG5hLVokiHew4mSQIDK68JjUBWMcdUIlORYH2A== X-Received: by 2002:a92:c266:0:b0:3a0:9fc6:5437 with SMTP id e9e14a558f8ab-3a3451a2361mr106267025ab.18.1727728822877; Mon, 30 Sep 2024 13:40:22 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a344d60728sm26430175ab.2.2024.09.30.13.40.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 13:40:21 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/5] io_uring/poll: remove 'ctx' argument from io_poll_req_delete() Date: Mon, 30 Sep 2024 14:37:45 -0600 Message-ID: <20240930204018.109617-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240930204018.109617-1-axboe@kernel.dk> References: <20240930204018.109617-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's always req->ctx being used anyway, having this as a separate argument (that is then not even used) just makes it more confusing. Signed-off-by: Jens Axboe --- io_uring/poll.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 1f63b60e85e7..175c279e59ea 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -129,7 +129,7 @@ static void io_poll_req_insert(struct io_kiocb *req) spin_unlock(&hb->lock); } -static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) +static void io_poll_req_delete(struct io_kiocb *req) { struct io_hash_table *table = &req->ctx->cancel_table; u32 index = hash_long(req->cqe.user_data, table->hash_bits); @@ -165,7 +165,7 @@ static void io_poll_tw_hash_eject(struct io_kiocb *req, struct io_tw_state *ts) hash_del(&req->hash_node); req->flags &= ~REQ_F_HASH_LOCKED; } else { - io_poll_req_delete(req, ctx); + io_poll_req_delete(req); } } From patchwork Mon Sep 30 20:37:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13817067 Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A97C199FAF for ; Mon, 30 Sep 2024 20:40:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728829; cv=none; b=nvLTz9C+KVBcnAaqNy7iiCv2lgn6vlgoFrGlVuPyPqLjXCxhMFuy5p31lNVfZn51NUOLN62IDezVVNVC4rlgQKCJayDAwtrXbTwBy96xlHbVu8MhviETJnYq+utba+Dl8+PdbdwFRXqmPSKPhDzjCe/j+uvw/UE2CnFyu/xT0To= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728829; c=relaxed/simple; bh=AHcnLaXM698lQPMYEtrGO8ii8vBk6G9LhwjUbBlLlhw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R6K85Ho2xF6xC6YpBW5YPykEKMV9mBIRZsxOkcQyu95sdZ4DyVG8qEeWJO8LC1s2+QY0MlPgQnkeSF/lv6BbVluIZ0/BUxbn07Gq4GFNQZVlV0NY+2wxAeabs1FTPJmRU1/SXKM4bBO2yk1cG9/zSBljF49thjODpz4iLLmGwaw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=RUfUwS11; arc=none smtp.client-ip=209.85.166.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="RUfUwS11" Received: by mail-io1-f46.google.com with SMTP id ca18e2360f4ac-82ce1cd202cso278055139f.0 for ; Mon, 30 Sep 2024 13:40:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1727728825; x=1728333625; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PC7IMRLmnudLU0k1domBB6+v/EkL/Zxt7NFChK5oTY0=; b=RUfUwS117iM5vlwgnT8HR0xT7IaGXakES7fzKUAbZJTSHLq3/CElLRbIT8SCnHhBHx gx2OV1o2qHXTiYfOMAxmwpu3UVHuXblsUkNTfx21s7TTJwwlobIuSFp5bANQEoOFc+Av DiwWJjEbo4jei3UCthnb84wCO2U7msIs7jkbOkrausEfHLI8R2Puj/qqlFHrdSlBvdzu 1xBiLOp472FyZctWnenwIgbQagaAFVl1ZTPkSMBm26NDx//kf5XimfHWauQGPOqSvXJ5 vFLJyhiFHu139/HHIjaaURnZSe53fctTv1QrgAmZkHXmjSKDpr5ad1ZWaR5Qu5ylGp2U cHQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727728825; x=1728333625; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PC7IMRLmnudLU0k1domBB6+v/EkL/Zxt7NFChK5oTY0=; b=ifIPNXLQnu55mC/+vbctFjwxGR5T3rJDEnZqjNRAURZfbL2w/kin49H/fBGYNqK5jz gzoz3lEumsp9LBFUFS5duEEEuNHkxnc7YKKtvSJ4mAy8RdIUmOFxkGijbLIcpQBi4hAO 6/Q0NiZ0iXVYGgiU6iqvUYY1qWvgU2Iggl6Rhi7M8rK3Q7kF/4w/7o6MtZbjCISBzLzM uIipqcFEdWcSgdWJ/SLzLtxFdQaSWRHdRfKC7+2c8hTH5VnUMd9pCYgBX+4Kg151E0Zx b4n3VHOyLFRIUUlSDhymCPQj/DNReD1Dz9gl5HkumHhAFZLodTNB6D3oCKRl4Ld6zThj 3hYQ== X-Gm-Message-State: AOJu0YyeAMAb643maZB6goU/U8xJrSo0crW/JdRKIs5DlKv4H1MluSxe tRMKgXQg200nW/62npNYlstc/JOa0jrTis5n2JvGZ9tNZf5RkY2zyDzlKLvG5bm3UuhCmyChSU/ sKBE= X-Google-Smtp-Source: AGHT+IHqR7M+bWLd5ZoQMRwgkPjtRG8jTo8nwmrGXIFN/ocTzDocX9IgwuKZSQnXW2FK3XiRpWLkAA== X-Received: by 2002:a92:b712:0:b0:3a3:6045:f8bd with SMTP id e9e14a558f8ab-3a36045fa4cmr1926645ab.5.1727728824802; Mon, 30 Sep 2024 13:40:24 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a344d60728sm26430175ab.2.2024.09.30.13.40.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 13:40:23 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/5] io_uring/poll: get rid of unlocked cancel hash Date: Mon, 30 Sep 2024 14:37:46 -0600 Message-ID: <20240930204018.109617-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240930204018.109617-1-axboe@kernel.dk> References: <20240930204018.109617-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring maintains two hash lists of inflight requests: 1) ctx->cancel_table_locked. This is used when the caller has the ctx->uring_lock held already. This is only an issue side parameter, as removal or task_work will always have it held. 2) ctx->cancel_table. This is used when the issuer does NOT have the ctx->uring_lock held, and relies on the table spinlocks for access. However, it's pretty trivial to simply grab the lock in the one spot where we care about it, for insertion. With that, we can kill the unlocked table (and get rid of the _locked postfix for the other one). Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 6 +- io_uring/fdinfo.c | 11 +-- io_uring/io_uring.c | 4 - io_uring/poll.c | 139 ++++++++------------------------- 4 files changed, 36 insertions(+), 124 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 4b9ba523978d..d8ca27da1341 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -291,7 +291,7 @@ struct io_ring_ctx { struct xarray io_bl_xa; - struct io_hash_table cancel_table_locked; + struct io_hash_table cancel_table; struct io_alloc_cache apoll_cache; struct io_alloc_cache netmsg_cache; struct io_alloc_cache rw_cache; @@ -342,7 +342,6 @@ struct io_ring_ctx { struct list_head io_buffers_comp; struct list_head cq_overflow_list; - struct io_hash_table cancel_table; struct hlist_head waitid_list; @@ -459,7 +458,6 @@ enum { REQ_F_DOUBLE_POLL_BIT, REQ_F_APOLL_MULTISHOT_BIT, REQ_F_CLEAR_POLLIN_BIT, - REQ_F_HASH_LOCKED_BIT, /* keep async read/write and isreg together and in order */ REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, @@ -534,8 +532,6 @@ enum { REQ_F_APOLL_MULTISHOT = IO_REQ_FLAG(REQ_F_APOLL_MULTISHOT_BIT), /* recvmsg special flag, clear EPOLLIN */ REQ_F_CLEAR_POLLIN = IO_REQ_FLAG(REQ_F_CLEAR_POLLIN_BIT), - /* hashed into ->cancel_hash_locked, protected by ->uring_lock */ - REQ_F_HASH_LOCKED = IO_REQ_FLAG(REQ_F_HASH_LOCKED_BIT), /* don't use lazy poll wake for this request */ REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), /* file is pollable */ diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index 6b1247664b35..a6bac533edbe 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -190,22 +190,13 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file) } seq_puts(m, "PollList:\n"); - for (i = 0; i < (1U << ctx->cancel_table.hash_bits); i++) { + for (i = 0; has_lock && i < (1U << ctx->cancel_table.hash_bits); i++) { struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; - struct io_hash_bucket *hbl = &ctx->cancel_table_locked.hbs[i]; struct io_kiocb *req; - spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) seq_printf(m, " op=%d, task_works=%d\n", req->opcode, task_work_pending(req->task)); - spin_unlock(&hb->lock); - - if (!has_lock) - continue; - hlist_for_each_entry(req, &hbl->list, hash_node) - seq_printf(m, " op=%d, task_works=%d\n", req->opcode, - task_work_pending(req->task)); } if (has_lock) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index feb61d68dca6..6685932aea9b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -294,8 +294,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) hash_bits = clamp(hash_bits, 1, 8); if (io_alloc_hash_table(&ctx->cancel_table, hash_bits)) goto err; - if (io_alloc_hash_table(&ctx->cancel_table_locked, hash_bits)) - goto err; if (percpu_ref_init(&ctx->refs, io_ring_ctx_ref_free, 0, GFP_KERNEL)) goto err; @@ -358,7 +356,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) io_alloc_cache_free(&ctx->msg_cache, io_msg_cache_free); io_futex_cache_free(ctx); kfree(ctx->cancel_table.hbs); - kfree(ctx->cancel_table_locked.hbs); xa_destroy(&ctx->io_bl_xa); kfree(ctx); return NULL; @@ -2771,7 +2768,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_wq_put_hash(ctx->hash_map); io_napi_free(ctx); kfree(ctx->cancel_table.hbs); - kfree(ctx->cancel_table_locked.hbs); xa_destroy(&ctx->io_bl_xa); kfree(ctx); } diff --git a/io_uring/poll.c b/io_uring/poll.c index 175c279e59ea..69382da48c00 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -122,28 +122,6 @@ static void io_poll_req_insert(struct io_kiocb *req) { struct io_hash_table *table = &req->ctx->cancel_table; u32 index = hash_long(req->cqe.user_data, table->hash_bits); - struct io_hash_bucket *hb = &table->hbs[index]; - - spin_lock(&hb->lock); - hlist_add_head(&req->hash_node, &hb->list); - spin_unlock(&hb->lock); -} - -static void io_poll_req_delete(struct io_kiocb *req) -{ - struct io_hash_table *table = &req->ctx->cancel_table; - u32 index = hash_long(req->cqe.user_data, table->hash_bits); - spinlock_t *lock = &table->hbs[index].lock; - - spin_lock(lock); - hash_del(&req->hash_node); - spin_unlock(lock); -} - -static void io_poll_req_insert_locked(struct io_kiocb *req) -{ - struct io_hash_table *table = &req->ctx->cancel_table_locked; - u32 index = hash_long(req->cqe.user_data, table->hash_bits); lockdep_assert_held(&req->ctx->uring_lock); @@ -154,19 +132,14 @@ static void io_poll_tw_hash_eject(struct io_kiocb *req, struct io_tw_state *ts) { struct io_ring_ctx *ctx = req->ctx; - if (req->flags & REQ_F_HASH_LOCKED) { - /* - * ->cancel_table_locked is protected by ->uring_lock in - * contrast to per bucket spinlocks. Likely, tctx_task_work() - * already grabbed the mutex for us, but there is a chance it - * failed. - */ - io_tw_lock(ctx, ts); - hash_del(&req->hash_node); - req->flags &= ~REQ_F_HASH_LOCKED; - } else { - io_poll_req_delete(req); - } + /* + * ->cancel_table_locked is protected by ->uring_lock in + * contrast to per bucket spinlocks. Likely, tctx_task_work() + * already grabbed the mutex for us, but there is a chance it + * failed. + */ + io_tw_lock(ctx, ts); + hash_del(&req->hash_node); } static void io_init_poll_iocb(struct io_poll *poll, __poll_t events) @@ -563,12 +536,13 @@ static bool io_poll_can_finish_inline(struct io_kiocb *req, return pt->owning || io_poll_get_ownership(req); } -static void io_poll_add_hash(struct io_kiocb *req) +static void io_poll_add_hash(struct io_kiocb *req, unsigned int issue_flags) { - if (req->flags & REQ_F_HASH_LOCKED) - io_poll_req_insert_locked(req); - else - io_poll_req_insert(req); + struct io_ring_ctx *ctx = req->ctx; + + io_ring_submit_lock(ctx, issue_flags); + io_poll_req_insert(req); + io_ring_submit_unlock(ctx, issue_flags); } /* @@ -605,11 +579,6 @@ static int __io_arm_poll_handler(struct io_kiocb *req, ipt->owning = issue_flags & IO_URING_F_UNLOCKED; atomic_set(&req->poll_refs, (int)ipt->owning); - /* io-wq doesn't hold uring_lock */ - if (issue_flags & IO_URING_F_UNLOCKED) - req->flags &= ~REQ_F_HASH_LOCKED; - - /* * Exclusive waits may only wake a limited amount of entries * rather than all of them, this may interfere with lazy @@ -638,7 +607,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, if (mask && ((poll->events & (EPOLLET|EPOLLONESHOT)) == (EPOLLET|EPOLLONESHOT))) { if (!io_poll_can_finish_inline(req, ipt)) { - io_poll_add_hash(req); + io_poll_add_hash(req, issue_flags); return 0; } io_poll_remove_entries(req); @@ -647,7 +616,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 1; } - io_poll_add_hash(req); + io_poll_add_hash(req, issue_flags); if (mask && (poll->events & EPOLLET) && io_poll_can_finish_inline(req, ipt)) { @@ -720,12 +689,6 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) __poll_t mask = POLLPRI | POLLERR | EPOLLET; int ret; - /* - * apoll requests already grab the mutex to complete in the tw handler, - * so removal from the mutex-backed hash is free, use it by default. - */ - req->flags |= REQ_F_HASH_LOCKED; - if (!def->pollin && !def->pollout) return IO_APOLL_ABORTED; if (!io_file_can_poll(req)) @@ -761,18 +724,22 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) return IO_APOLL_OK; } -static __cold bool io_poll_remove_all_table(struct task_struct *tsk, - struct io_hash_table *table, - bool cancel_all) +/* + * Returns true if we found and killed one or more poll requests + */ +__cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, + bool cancel_all) { - unsigned nr_buckets = 1U << table->hash_bits; + unsigned nr_buckets = 1U << ctx->cancel_table.hash_bits; struct hlist_node *tmp; struct io_kiocb *req; bool found = false; int i; + lockdep_assert_held(&ctx->uring_lock); + for (i = 0; i < nr_buckets; i++) { - struct io_hash_bucket *hb = &table->hbs[i]; + struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; spin_lock(&hb->lock); hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) { @@ -787,28 +754,13 @@ static __cold bool io_poll_remove_all_table(struct task_struct *tsk, return found; } -/* - * Returns true if we found and killed one or more poll requests - */ -__cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, - bool cancel_all) - __must_hold(&ctx->uring_lock) -{ - bool ret; - - ret = io_poll_remove_all_table(tsk, &ctx->cancel_table, cancel_all); - ret |= io_poll_remove_all_table(tsk, &ctx->cancel_table_locked, cancel_all); - return ret; -} - static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, - struct io_hash_table *table, struct io_hash_bucket **out_bucket) { struct io_kiocb *req; - u32 index = hash_long(cd->data, table->hash_bits); - struct io_hash_bucket *hb = &table->hbs[index]; + u32 index = hash_long(cd->data, ctx->cancel_table.hash_bits); + struct io_hash_bucket *hb = &ctx->cancel_table.hbs[index]; *out_bucket = NULL; @@ -831,17 +783,16 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd, - struct io_hash_table *table, struct io_hash_bucket **out_bucket) { - unsigned nr_buckets = 1U << table->hash_bits; + unsigned nr_buckets = 1U << ctx->cancel_table.hash_bits; struct io_kiocb *req; int i; *out_bucket = NULL; for (i = 0; i < nr_buckets; i++) { - struct io_hash_bucket *hb = &table->hbs[i]; + struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { @@ -866,17 +817,16 @@ static int io_poll_disarm(struct io_kiocb *req) return 0; } -static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, - struct io_hash_table *table) +static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) { struct io_hash_bucket *bucket; struct io_kiocb *req; if (cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP | IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd, table, &bucket); + req = io_poll_file_find(ctx, cd, &bucket); else - req = io_poll_find(ctx, false, cd, table, &bucket); + req = io_poll_find(ctx, false, cd, &bucket); if (req) io_poll_cancel_req(req); @@ -890,12 +840,8 @@ int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, { int ret; - ret = __io_poll_cancel(ctx, cd, &ctx->cancel_table); - if (ret != -ENOENT) - return ret; - io_ring_submit_lock(ctx, issue_flags); - ret = __io_poll_cancel(ctx, cd, &ctx->cancel_table_locked); + ret = __io_poll_cancel(ctx, cd); io_ring_submit_unlock(ctx, issue_flags); return ret; } @@ -972,13 +918,6 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) ipt.pt._qproc = io_poll_queue_proc; - /* - * If sqpoll or single issuer, there is no contention for ->uring_lock - * and we'll end up holding it in tw handlers anyway. - */ - if (req->ctx->flags & (IORING_SETUP_SQPOLL|IORING_SETUP_SINGLE_ISSUER)) - req->flags |= REQ_F_HASH_LOCKED; - ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags); if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); @@ -997,26 +936,16 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) int ret2, ret = 0; io_ring_submit_lock(ctx, issue_flags); - preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table, &bucket); + preq = io_poll_find(ctx, true, &cd, &bucket); ret2 = io_poll_disarm(preq); if (bucket) spin_unlock(&bucket->lock); if (!ret2) goto found; - if (ret2 != -ENOENT) { - ret = ret2; - goto out; - } - - preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table_locked, &bucket); - ret2 = io_poll_disarm(preq); - if (bucket) - spin_unlock(&bucket->lock); if (ret2) { ret = ret2; goto out; } - found: if (WARN_ON_ONCE(preq->opcode != IORING_OP_POLL_ADD)) { ret = -EFAULT; From patchwork Mon Sep 30 20:37:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13817068 Received: from mail-il1-f179.google.com (mail-il1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2E6021373 for ; Mon, 30 Sep 2024 20:40:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728829; cv=none; b=kPle+5cjANP5gch+eRDBZifUZBuNZXrKkVFFmzoh3XZDHusa11sMp2dpH2IleLhLSof8t4imVn2CR296PXmFLPq/+r9OkJzxUcvfhaAJVO90XR9YCTA/cZPIKEW6QWWYN4b983SUMPbOyBhueVBjzKcIs4xEang9Wdr7Xn0eQRA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728829; c=relaxed/simple; bh=t7dgvkj5xpodNXZOK/IWcfBh4eHIdO52KtvIie4ZSNk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NB2s5z/wrpPgy6DlVmwedYlE4gBa5wXmYne1s98n0MjtzfPoDiaXPyIGAAcHqqnR1jvsVV4DalAwYngpUDRe+kfIA6qKOTwSLD6LpBW9nxGuWZ184EZOrrX5jTnwJnTZAmtiN+BnLmOE79B+sZ7pOYTMM/36gx+vG7qL1kYmROE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=I9mh+We2; arc=none smtp.client-ip=209.85.166.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="I9mh+We2" Received: by mail-il1-f179.google.com with SMTP id e9e14a558f8ab-3a274ef3bdeso18282935ab.2 for ; Mon, 30 Sep 2024 13:40:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1727728827; x=1728333627; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NcT3xb4WXjYKAJLK56A3Ii5F4kRF9uFA+kKq+uCfFvE=; b=I9mh+We2Ygzh5xthAqdyEt/JRUDWWsKfext5PowHOm6hRMv4euTY7hAemhCPQsvs5t sMCJn+hnyLZNoEKrDsnjdNbzKwiLADvuCgoQFfuOX/focrydbuhrMLl/rnwHcyyUDPaE XZKTHycqxu8/F/sEnM8J5T4aNmxvl5wSyWuMRSDooX1skFsvdBn6Gaur1sbkUNRoQYZp SaZZr1dmgLIrgJUqLGk1+aMv5Zab6SVx5wAZgrp29M0c1qj+WFZLsXFpwv/0rTEGyLMy RZxp0CGtxhX+ManSZRRbMzzrBYPSskmh8Tjp/EgVPUsrOV5b7TKF1o1i87ltKQDjg8tk PLHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727728827; x=1728333627; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NcT3xb4WXjYKAJLK56A3Ii5F4kRF9uFA+kKq+uCfFvE=; b=xGnjfXrH0Y4zpHr7cZXelM3Nbjj6nc3LSqFAKIgy3q+YMDNcSaExxxf1RBm6hDZMhs PrB1TLdda7gJdzRQ10kFjm8mxWDPwo4UdMFOCDfa3RPrLpSC5LNXhUkFNzC3L0eTb1pQ xxL5O/TntqzzhROH6U38L984cUTyaORXBhP66ftLTXbdXNM5YpZAZOhod9103JLcrcPQ qnGo73nTQWyXIBElW44TICbu1WPX8mKrtctDcaxkPDI8nsYDV+d3DGrIS429Dg3FLdXV p2yY6e/m68ulxNsph2feKfazWYULz8c57OlSnAAD359fRBFCDSlR6+NNP2AMV2O9xbmT iyFw== X-Gm-Message-State: AOJu0Yw/2GbBG5KO/3kx6pyPHiIO6DVu8WFmL1x/rROdgiu1W1cwp9Jo DPnFILahaz1t+gN3IjdA/HulnEdx515xebV9RqNuG2COpxfiVFKKf8zuQXZtvqS3bHCLAOgmU9i Npms= X-Google-Smtp-Source: AGHT+IEDsWdTHs0dNRFVIsCUZWw3Q0C0rvMx42yAFswqcVebrlTTDIofheMHv/unA0f+CtEjsL6/Lw== X-Received: by 2002:a05:6e02:2187:b0:3a1:a69f:9391 with SMTP id e9e14a558f8ab-3a345179f16mr97819855ab.13.1727728826696; Mon, 30 Sep 2024 13:40:26 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a344d60728sm26430175ab.2.2024.09.30.13.40.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 13:40:25 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/5] io_uring/poll: get rid of io_poll_tw_hash_eject() Date: Mon, 30 Sep 2024 14:37:47 -0600 Message-ID: <20240930204018.109617-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240930204018.109617-1-axboe@kernel.dk> References: <20240930204018.109617-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It serves no purposes anymore, all it does is delete the hash list entry. task_work always has the ring locked. Signed-off-by: Jens Axboe --- io_uring/poll.c | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 69382da48c00..a7d7fa844729 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -128,20 +128,6 @@ static void io_poll_req_insert(struct io_kiocb *req) hlist_add_head(&req->hash_node, &table->hbs[index].list); } -static void io_poll_tw_hash_eject(struct io_kiocb *req, struct io_tw_state *ts) -{ - struct io_ring_ctx *ctx = req->ctx; - - /* - * ->cancel_table_locked is protected by ->uring_lock in - * contrast to per bucket spinlocks. Likely, tctx_task_work() - * already grabbed the mutex for us, but there is a chance it - * failed. - */ - io_tw_lock(ctx, ts); - hash_del(&req->hash_node); -} - static void io_init_poll_iocb(struct io_poll *poll, __poll_t events) { poll->head = NULL; @@ -336,7 +322,8 @@ void io_poll_task_func(struct io_kiocb *req, struct io_tw_state *ts) return; } io_poll_remove_entries(req); - io_poll_tw_hash_eject(req, ts); + /* task_work always has ->uring_lock held */ + hash_del(&req->hash_node); if (req->opcode == IORING_OP_POLL_ADD) { if (ret == IOU_POLL_DONE) { From patchwork Mon Sep 30 20:37:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13817069 Received: from mail-il1-f171.google.com (mail-il1-f171.google.com [209.85.166.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8812199FAF for ; Mon, 30 Sep 2024 20:40:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728832; cv=none; b=IQantSWaJuVB0jkL3i/r5IaMiLxgAf8d3IXPcehnSpsgTEvTo3eFyXBMiCviy4/BiBMrDlZ1rWn5ZqkjFo/LuCUPPKe3lcl20fQSpQUD+eTdLXB1KVhZQs9JB0vYNiW1oH0+TYOQN/6TLvr7ab+iqazpECXMgfyMYi1ElikXXRI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728832; c=relaxed/simple; bh=mCuQ4XeoaGOEiJ6kvrbrA3dMH2ptmBDDdJOaZeeZq4U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e4+XiabZAKnH5QKf5niSGio5nw17Kcc5thovu2SwTu17zJOI+p+3WXjzw5wVk11dy/LNPqqgYahn8KO28PdAuPIFTon7ccj5jnDcnhRC/dqbjR02oIw0P/RafaYOSztmJNOPMvJzfXqfrqD8sjV4SnHprgtwrHyJsINydOtel98= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=VyQhJaY1; arc=none smtp.client-ip=209.85.166.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="VyQhJaY1" Received: by mail-il1-f171.google.com with SMTP id e9e14a558f8ab-3a3525ba6aaso5803545ab.2 for ; Mon, 30 Sep 2024 13:40:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1727728828; x=1728333628; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GUh7ByU+LqT2NJHz3+ivI+J6VF4Zj/o/zOLq9N3AmIY=; b=VyQhJaY1LXP3oU6XY3N3mNLbqnxBDXbxWia7T9zw1J6YMpG9xlZNaw4OR5WLWOHaWl FJGHH4W721X9epfXtwDvqXBjlZKOdw7hqDEv5H0ihz+NSlTdspuVOXCBuSQURr3CIANJ FuKyDJAVpMmf5UoDtHkX6Yl/JitQKsjVCl0Sjcq91WfLhvHJxm32b6HJ4AKxtst7118z 33M+QobBiFqOVErDJ59EChlm5PDmi/4FnmIhjgLbGN6NgvcpyjF21sfRM2CFHeQo0Fc3 0SFjrYz5m0m/uQ//w5wASrAUGO36tbfilZKBI6A5GmgW+YkkkN8DEKMxLk6Cp32awtAm o7jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727728828; x=1728333628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GUh7ByU+LqT2NJHz3+ivI+J6VF4Zj/o/zOLq9N3AmIY=; b=Lj+fOKpf4dLFAb7P2AQu+/xCR1BSZzpJwXxLf4cYEzj3DRWpZ3b+SGu9WmoEEmpuw0 eGEbilqY9x/zt99MiTYlDigu7kUMThfYaqB+8LktvzD7MlLEjNvg5sOlOZBZdQwBjIZ1 0dfklmWaOPU17doYMQZTHIP1cWBnyT9qCh9iU8eY0xKHVyVJd7WzuMzL60ZvpTMmY8/s VtrMP+w0RQa7ucHeKXLH66fcHi89ZIC07UgDdHNBhS33wA9+8aizqYS3bdfmsyNryOZV fqv1wtlmklE1p17d7FbDxQjDL3ggSbtOmgqxChvPpP7w18omX3T8qI12cLf8IoiJKkAQ 6XhA== X-Gm-Message-State: AOJu0YxC8R0u8nfvpaaZMTfCa3yVE4GBuqSjPwanU57Q2qTlpyVGTTU1 k8XUd1bKZxqlvcmW6UHBrinBssXtJ300TH8CKwpNI06jSZqgWKL4bW9BnAdFRq95T+30woPwHNe ghgg= X-Google-Smtp-Source: AGHT+IHdcUcj0r4ScQgh7Ki0CDe8HpsuuKEHMjGPQ+q4R/LlRUfFwxofUMshYlhSoR0dHebp51WAEw== X-Received: by 2002:a05:6e02:1a2c:b0:3a0:aac2:a0a4 with SMTP id e9e14a558f8ab-3a34515d38dmr100634725ab.9.1727728828463; Mon, 30 Sep 2024 13:40:28 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a344d60728sm26430175ab.2.2024.09.30.13.40.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 13:40:26 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/5] io_uring/poll: get rid of per-hashtable bucket locks Date: Mon, 30 Sep 2024 14:37:48 -0600 Message-ID: <20240930204018.109617-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240930204018.109617-1-axboe@kernel.dk> References: <20240930204018.109617-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Any access to the table is protected by ctx->uring_lock now anyway, the per-bucket locking doesn't buy us anything. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 1 - io_uring/cancel.c | 4 +--- io_uring/poll.c | 39 +++++++++------------------------- 3 files changed, 11 insertions(+), 33 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index d8ca27da1341..9c7e1d3f06e5 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -67,7 +67,6 @@ struct io_file_table { }; struct io_hash_bucket { - spinlock_t lock; struct hlist_head list; } ____cacheline_aligned_in_smp; diff --git a/io_uring/cancel.c b/io_uring/cancel.c index a6e58a20efdd..755dd5506a5f 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -236,10 +236,8 @@ void init_hash_table(struct io_hash_table *table, unsigned size) { unsigned int i; - for (i = 0; i < size; i++) { - spin_lock_init(&table->hbs[i].lock); + for (i = 0; i < size; i++) INIT_HLIST_HEAD(&table->hbs[i].list); - } } static int __io_sync_cancel(struct io_uring_task *tctx, diff --git a/io_uring/poll.c b/io_uring/poll.c index a7d7fa844729..63f9461aa9b6 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -728,7 +728,6 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, for (i = 0; i < nr_buckets; i++) { struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; - spin_lock(&hb->lock); hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) { if (io_match_task_safe(req, tsk, cancel_all)) { hlist_del_init(&req->hash_node); @@ -736,22 +735,17 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, found = true; } } - spin_unlock(&hb->lock); } return found; } static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, - struct io_cancel_data *cd, - struct io_hash_bucket **out_bucket) + struct io_cancel_data *cd) { struct io_kiocb *req; u32 index = hash_long(cd->data, ctx->cancel_table.hash_bits); struct io_hash_bucket *hb = &ctx->cancel_table.hbs[index]; - *out_bucket = NULL; - - spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { if (cd->data != req->cqe.user_data) continue; @@ -761,34 +755,25 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, if (io_cancel_match_sequence(req, cd->seq)) continue; } - *out_bucket = hb; return req; } - spin_unlock(&hb->lock); return NULL; } static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, - struct io_cancel_data *cd, - struct io_hash_bucket **out_bucket) + struct io_cancel_data *cd) { unsigned nr_buckets = 1U << ctx->cancel_table.hash_bits; struct io_kiocb *req; int i; - *out_bucket = NULL; - for (i = 0; i < nr_buckets; i++) { struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; - spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { - if (io_cancel_req_match(req, cd)) { - *out_bucket = hb; + if (io_cancel_req_match(req, cd)) return req; - } } - spin_unlock(&hb->lock); } return NULL; } @@ -806,20 +791,19 @@ static int io_poll_disarm(struct io_kiocb *req) static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) { - struct io_hash_bucket *bucket; struct io_kiocb *req; if (cd->flags & (IORING_ASYNC_CANCEL_FD | IORING_ASYNC_CANCEL_OP | IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd, &bucket); + req = io_poll_file_find(ctx, cd); else - req = io_poll_find(ctx, false, cd, &bucket); + req = io_poll_find(ctx, false, cd); - if (req) + if (req) { io_poll_cancel_req(req); - if (bucket) - spin_unlock(&bucket->lock); - return req ? 0 : -ENOENT; + return 0; + } + return -ENOENT; } int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, @@ -918,15 +902,12 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) struct io_poll_update *poll_update = io_kiocb_to_cmd(req, struct io_poll_update); struct io_ring_ctx *ctx = req->ctx; struct io_cancel_data cd = { .ctx = ctx, .data = poll_update->old_user_data, }; - struct io_hash_bucket *bucket; struct io_kiocb *preq; int ret2, ret = 0; io_ring_submit_lock(ctx, issue_flags); - preq = io_poll_find(ctx, true, &cd, &bucket); + preq = io_poll_find(ctx, true, &cd); ret2 = io_poll_disarm(preq); - if (bucket) - spin_unlock(&bucket->lock); if (!ret2) goto found; if (ret2) { From patchwork Mon Sep 30 20:37:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13817070 Received: from mail-il1-f178.google.com (mail-il1-f178.google.com [209.85.166.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77A58199E8E for ; Mon, 30 Sep 2024 20:40:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728834; cv=none; b=Z8lOVyb0Pp1g7tT+9YVkuu55o2D4jFF0AnzOYedrsRgH8YFmLwDjLfUs+mBL6lhUFl29SlfGrjTp8HKuFvsY+0MF2trpLlR66hCKN+oogHEZprirH76eJx52BNx3K1m5zL4AlOBmLsO1JJhKgGD8DAh2s2tUYYc9TmUVnR/wzhI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727728834; c=relaxed/simple; bh=lp05/IfRAcCB864MdOpi4/54CxOQa2AO+8OgPS0E+sI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Rdo+muu2kcPK+9Wk6u3/dTuKt41efqnSOZMJO2ZoTiNVaFmuqq7bhzAnKf36+I1xsd8HN+muGPnlszaXJjd5Zva0VfL9XL8fxPZTy/5H2tEAm9SlMq+omyI11Ebe3IkUwMiJ1act3m+UnYxwboEoemYl/i7a5v0j1+zl6m0L790= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=29f+CWvV; arc=none smtp.client-ip=209.85.166.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="29f+CWvV" Received: by mail-il1-f178.google.com with SMTP id e9e14a558f8ab-3a0be4d803eso15928555ab.0 for ; Mon, 30 Sep 2024 13:40:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1727728831; x=1728333631; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=noqafkmXQlFniP1vN6HSn+4V1mhVCqCJBxND01DbHbQ=; b=29f+CWvVE/bEf1g8FPGEK0ypJnLe5gBfZW21AO7WNZZNfuGIBqkjHx5EVx9o3+A2Sn Jm+DunWserPCgPgF6ocvtVFxnujffjRZq1T8My1g+2VlrMs0zYT2SSbECpie6mx7zl39 GaY2SUech9qfxW6I7uDbgrgHaiKHQlbo8LSROZ7g5i6Oa6oB9Zs1ZNWs7Vn5M4vaiqa2 ALz4JNijuj2il41deoxwhr+9vtsgpy/fQIBwuS56hUiyd8GDsTwXhB81p+dgv10JN5Pz PWY1a3SLRhG6fJ5qRUhGEZvbeGGF7QuQE2HKs1LcJS+DHQIe20+qNsI1tki9BWWHWaw6 K2EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727728831; x=1728333631; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=noqafkmXQlFniP1vN6HSn+4V1mhVCqCJBxND01DbHbQ=; b=UNzUe3AyjyaZfgczJx8eHQGQAZ1m9pFMd4ySQU0V097bpAhtuuGNrhzE581XMX45Bi 8yd73mWEbz3qCZ7kJ9E10FsaqWqPdhgS9o2ifBWSBBYKQYkhwS+gjsUJDvKy+tZyPnI3 Dixpb46J8ErGp6FX2E+FqFLstIlTHGdC8jBGtFDK0kOmJpeJKo/C0+EE/nS0p8J8TtQr dYoixrpmDCjfpCMtBvZpup64qbk0kEZGtaiaviqANf0L3Le/Wy1GNCofpAqzKjpPEt+G XFtrUf/JwnuCfSfRI8fLUVaCYpJj73kbYnSg8AiWbO0x0zQmGk6oZK9ldGkvr0pK9vKK ZUzw== X-Gm-Message-State: AOJu0YwIy4uvX/MSMgD04rYRaYl5j2nSRT1ZY+B4A+/mw1nRTnEqhyXo Es1HINcV+hwoOJzXXfwD5YWd+JnjsIbMgYyakjZLL8o2lgupxEyN7dyhG93JEoyXw1RCl79lUcU iC3k= X-Google-Smtp-Source: AGHT+IFJdxnYLOehQ3fMkpTkpFWOLC2rKbpsaVJ9ZSUAahGueAdWlU91yc5H5ob9FpjxFimLfNBIAQ== X-Received: by 2002:a92:c266:0:b0:3a0:8ecb:a1dc with SMTP id e9e14a558f8ab-3a34517fac8mr114447175ab.15.1727728831070; Mon, 30 Sep 2024 13:40:31 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a344d60728sm26430175ab.2.2024.09.30.13.40.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 13:40:28 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/5] io_uring/cancel: get rid of init_hash_table() helper Date: Mon, 30 Sep 2024 14:37:49 -0600 Message-ID: <20240930204018.109617-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240930204018.109617-1-axboe@kernel.dk> References: <20240930204018.109617-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 All it does is initialize the lists, just move the INIT_HLIST_HEAD() into the one caller. Signed-off-by: Jens Axboe --- io_uring/cancel.c | 8 -------- io_uring/cancel.h | 1 - io_uring/io_uring.c | 4 +++- 3 files changed, 3 insertions(+), 10 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 755dd5506a5f..cc3475b22ae5 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -232,14 +232,6 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -void init_hash_table(struct io_hash_table *table, unsigned size) -{ - unsigned int i; - - for (i = 0; i < size; i++) - INIT_HLIST_HEAD(&table->hbs[i].list); -} - static int __io_sync_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd, int fd) { diff --git a/io_uring/cancel.h b/io_uring/cancel.h index b33995e00ba9..bbfea2cd00ea 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -20,7 +20,6 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd, unsigned int issue_flags); -void init_hash_table(struct io_hash_table *table, unsigned size); int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg); bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6685932aea9b..469900007eae 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -263,13 +263,15 @@ static int io_alloc_hash_table(struct io_hash_table *table, unsigned bits) { unsigned hash_buckets = 1U << bits; size_t hash_size = hash_buckets * sizeof(table->hbs[0]); + int i; table->hbs = kmalloc(hash_size, GFP_KERNEL); if (!table->hbs) return -ENOMEM; table->hash_bits = bits; - init_hash_table(table, hash_buckets); + for (i = 0; i < hash_buckets; i++) + INIT_HLIST_HEAD(&table->hbs[i].list); return 0; }