From patchwork Tue Jun 14 12:29:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881114 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 370FAC43334 for ; Tue, 14 Jun 2022 12:33:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243039AbiFNMdq (ORCPT ); Tue, 14 Jun 2022 08:33:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243885AbiFNMdS (ORCPT ); Tue, 14 Jun 2022 08:33:18 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4CAE4B1C0 for ; Tue, 14 Jun 2022 05:30:37 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id e5so4586471wma.0 for ; Tue, 14 Jun 2022 05:30:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ORixVfQAPkrUEdbPULkgFBU4hdADQl3LSugR9FT2k0U=; b=n7UD8aRnamOC+UGlbr/BPuemAAj9OMj6NsuyC3hj/J+1SP+dwfYw30DfTZdQ/F371I hdswDptB/Nd5u/3bHsKzf1bKG0Mst5MuG2vyvuqHUf6pPzwPGTLh/J7fRNfpmoDDnYXy +vsoN4CEXQsWbwMeIcHs9+PbTVi4RzzIjTVpags2wkp8Xi+AQz4D9YtF6O4dntxLDRDJ jaoM/Izg8zDXfI0g1mX2wderY0HSWvt8vA25qwnTs9HhiJk+K95o95UZFxoDI5G8ze88 H7QeQ3EjrgRNorge0CTh5VbLPt4zKjqjALHucv1vSE9VXCS1AE+e6o2KgA8xqzznthU+ KIoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ORixVfQAPkrUEdbPULkgFBU4hdADQl3LSugR9FT2k0U=; b=O8NLVjDTWmflioHpNYRhOacHaHP2iTyVZsXbYL+qHTQPwLgwMYUfl1efiPRe86DQ1Y 4VnXhMwajgStrFRgVz3U+n5tFpVqJd1cAOnFOPlqV6jJCJn2uttO9huncXGxJV8h0zgM IPJXbPbaBPwUoO/aYE2Nz4Bizj7W6XjDPQZfsNIX/UKtEFnKU138f/XPayyMFProHTtk 4cJQHUjBK5PsJwY6E/klXqm472Dwp6QIYCDQeZuubnJhAqrULSD0+0SrnkqCLQOj351J 2NLPHH7TpUtSb/NJaiczddsLZWYjO9zeF9iEyIX0sQiJdGqUmx747ti2IYiQ9sSgVGUP 66KA== X-Gm-Message-State: AOAM530wMlSsvGwZ4uszCSgdawLoEIljJER0w1r64Y1pdpTxoHS/lFQ1 asJSBBL9aCvccUq63ufFbzGquellEXO77w== X-Google-Smtp-Source: ABdhPJzjtV1QJ4fuigd7ZVUU3k8E7D8n1iqlYccE++mDVcHH4IRZhA+t3wISoJxfV16gDyui3lGsmA== X-Received: by 2002:a1c:4d05:0:b0:39c:5932:d9f1 with SMTP id o5-20020a1c4d05000000b0039c5932d9f1mr3939456wmh.52.1655209835880; Tue, 14 Jun 2022 05:30:35 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:35 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 01/25] io_uring: make reg buf init consistent Date: Tue, 14 Jun 2022 13:29:39 +0100 Message-Id: <1835f8612fd77ed79712f566778cad6691d41c06.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The default (i.e. empty) state of register buffer is dummy_ubuf, so set it to dummy on init instead of NULL. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index fef46972c327..fd1323482030 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -567,7 +567,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, io_buffer_unmap(ctx, &imu); break; } - ctx->user_bufs[i] = NULL; + ctx->user_bufs[i] = ctx->dummy_ubuf; needs_switch = true; } @@ -1200,14 +1200,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, size_t size; int ret, nr_pages, i; - if (!iov->iov_base) { - *pimu = ctx->dummy_ubuf; + *pimu = ctx->dummy_ubuf; + if (!iov->iov_base) return 0; - } - *pimu = NULL; ret = -ENOMEM; - pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, &nr_pages); if (IS_ERR(pages)) { From patchwork Tue Jun 14 12:29:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881115 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02F0CC43334 for ; Tue, 14 Jun 2022 12:33:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242647AbiFNMdu (ORCPT ); Tue, 14 Jun 2022 08:33:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245251AbiFNMdY (ORCPT ); Tue, 14 Jun 2022 08:33:24 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4EE044A01 for ; Tue, 14 Jun 2022 05:30:39 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id x6-20020a1c7c06000000b003972dfca96cso4712771wmc.4 for ; Tue, 14 Jun 2022 05:30:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ToujjDMrgxjXImVoVU6hciJcYYHsx0thIW3gWEU5SEA=; b=nTcDlCuG/00HkgwY92Q3tY5WQhp7nXfww4UCWNXZJ/2WFTibK0VHf9F6pvqlXcTqWj aHMsLQUrakD9XYjCeiAl486PJ3yLuPXUs2NG9SIBR0v3nftF12Tp7Xp5BRsp/AnrObpf fo6nl8YMQKPx27JUxEqwE4ZY4/Nz5IZcYa1peHGGAZs4o9MBzI8tTwF8uCg25fQj7vLC psXmkl77FGQg6KvzO7y1AC5OUDY8i9UDscceWOtoYj8W1YtLgUPMqqs9WVRvOnISIYdc /8BD+LCLpkPr5tesvgYSyaIBpAf8kC+IC02RCyCEdNe4khdRcWsYWOalm+6D2ZXHG0+p e5/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ToujjDMrgxjXImVoVU6hciJcYYHsx0thIW3gWEU5SEA=; b=mCSpMDN59Y1scAxpxgVkb3+UtS6ynaqaVpXUOMqSET+hqAVLWQNOwLERDV1oze3VZI QoVO/QthbpjvO+9ivF9+vDt6Gbm4XEUY4AKQ9aiLm7NfmiCIAmorzD4mwCKzgxP0gaQX KJbYwbfQ9qck9fjoUbCTAPbX18cAmRlpgq69v03Z7nUeKzFl9oihMw4xq9YXm7uQ0rFj dKc3qSgz25zWzO/rybDWT3PHNETLIzNVMEJbc7QcKBOfHCCzP9K1MQsTMwGhh1OckUoN WXj0YXzSEdjxr5o0yhjpkbf2/TNUNDSuJbLG++//jPhJvy9N3us9OF+pZr90v6V8B9tS Cn9g== X-Gm-Message-State: AOAM531olpJt7lextwWdCVlhh8fHbURvJgSjxsXwGByuIxFQbm1i/Nrb Y/uEJkEujvARB349Ofqx6g0n8y4hcwh/SA== X-Google-Smtp-Source: ABdhPJz26FqgShcnU56x4IISwHonAR5Nhy3xktetYZINkroEmFjTmf/iGKMt0nBP+9chJhI+eY+hHA== X-Received: by 2002:a05:600c:601f:b0:39c:416c:4069 with SMTP id az31-20020a05600c601f00b0039c416c4069mr3867322wmb.85.1655209837312; Tue, 14 Jun 2022 05:30:37 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:36 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 02/25] io_uring: move defer_list to slow data Date: Tue, 14 Jun 2022 13:29:40 +0100 Message-Id: <097c9f3eb3f8ab052e10160567928ec8df0f1303.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org draining is slow path, move defer_list to the end where slow data lives inside the context. Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 7c22cf35a7e2..52e91c3df8d5 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -160,7 +160,6 @@ struct io_ring_ctx { struct io_uring_sqe *sq_sqes; unsigned cached_sq_head; unsigned sq_entries; - struct list_head defer_list; /* * Fixed resources fast path, should be accessed only under @@ -272,8 +271,12 @@ struct io_ring_ctx { struct work_struct exit_work; struct list_head tctx_list; struct completion ref_comp; + + /* io-wq management, e.g. thread count */ u32 iowq_limits[2]; bool iowq_limits_set; + + struct list_head defer_list; }; }; From patchwork Tue Jun 14 12:29:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881116 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6434FC433EF for ; Tue, 14 Jun 2022 12:33:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245346AbiFNMdv (ORCPT ); Tue, 14 Jun 2022 08:33:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245727AbiFNMd1 (ORCPT ); Tue, 14 Jun 2022 08:33:27 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92FD145063 for ; Tue, 14 Jun 2022 05:30:40 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id q15so11035551wrc.11 for ; Tue, 14 Jun 2022 05:30:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=43ATD8fKnmjyJdZ2R6qxl1QYnz/Icg31elC1cRBKr+w=; b=Gna1ys5pfe3cJOu1n9NdsRTUGvJhdA5kqAw2G4tlYv4x1/52D5HMpeKTuiSIndkpTT IWfr94ieH2OEB5R7UW0UJfTNqXdP8V2+bctQ1i29qSvsWe9Xp7AtG368QLez1PHOHG4x sEI5v1e7WznGsNyuoyp42j24U56QAglG6XXCJWgKW0szn02YVrIH/LIc+GC9pEETPwWc BVsayvpDp2S2spAT0Ol5E/ZdIY/uEky2rOS5judGmm2mAsgfO1ekv4NYpECKVPrfeO5p VhOioUJ80It2DSr75DfQAq0OU3NlYZoH39pLGo8I4CARPFTZXQBWGCMwyu4lc/Qi+eIj T21A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=43ATD8fKnmjyJdZ2R6qxl1QYnz/Icg31elC1cRBKr+w=; b=F6nRlhBQ8U4UOul/oqIGuLwZNKPQahvLyYrn/SZ8FzRCK+GsfIbWZtxAVpZ9IoisyI ietFwPYlKDtLGuVWwkwzfUzTS7HnU4Z6x00SNPyps3PhD4pS1QGl18z4ELzAlbZ7Cs5Q pBoU/NO2O1W+QeDHDMdcajeOp5k5A9WvhYxu0IVFDmpQGeihNP2RGcIJbY7y2pZua5ll MiClp2FoB8H33TPFD0N4ocL4tgAKmn6Z7qSiFJiKOtAZKL5eWgrab7S2i77KWV0uUN+f eXzGWLkEKdln+olx8NI0xI/LmK9sMgKIWDV514DILcetq7Z+nwHly/wdM8uVXkTuEy0r KjoA== X-Gm-Message-State: AJIora/FI66TJ5ja2v8O3AHThKxWUx5biGB91/qubBP5E8gfH6JwVwzc DSKFcnwMxnx8k1BspcNCcIGcwiAMgyK6sQ== X-Google-Smtp-Source: AGRyM1vS/ytzSJDnimxpkRfjdk9iGlyUfaSj95AHRK9YVtLAUh9+/nSWph2DvHycCRjcFDdRFrju1Q== X-Received: by 2002:adf:f584:0:b0:218:5a97:8f05 with SMTP id f4-20020adff584000000b002185a978f05mr4658518wro.333.1655209838539; Tue, 14 Jun 2022 05:30:38 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 03/25] io_uring: better caching for ctx timeout fields Date: Tue, 14 Jun 2022 13:29:41 +0100 Message-Id: <51f0bc096197ab07fbc54b975dafbd22a31a634c.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Following timeout fields access patterns, move all of them into a separate cache line inside ctx, so they don't intervene with normal completion caching, especially since timeout removals and completion are separated and the later is done via tw. It also sheds some bytes from io_ring_ctx, 1216B -> 1152B Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 52e91c3df8d5..4f52dcbbda56 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -179,8 +179,6 @@ struct io_ring_ctx { struct xarray io_bl_xa; struct list_head io_buffers_cache; - struct list_head timeout_list; - struct list_head ltimeout_list; struct list_head cq_overflow_list; struct list_head apoll_cache; struct xarray personalities; @@ -213,15 +211,11 @@ struct io_ring_ctx { struct io_ev_fd __rcu *io_ev_fd; struct wait_queue_head cq_wait; unsigned cq_extra; - atomic_t cq_timeouts; - unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; struct { spinlock_t completion_lock; - spinlock_t timeout_lock; - /* * ->iopoll_list is protected by the ctx->uring_lock for * io_uring instances that don't use IORING_SETUP_SQPOLL. @@ -253,6 +247,15 @@ struct io_ring_ctx { struct list_head io_buffers_pages; }; + /* timeouts */ + struct { + spinlock_t timeout_lock; + atomic_t cq_timeouts; + struct list_head timeout_list; + struct list_head ltimeout_list; + unsigned cq_last_tm_flush; + } ____cacheline_aligned_in_smp; + /* Keep this last, we don't need it for the fast path */ struct { #if defined(CONFIG_UNIX) From patchwork Tue Jun 14 12:29:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881117 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C92CC43334 for ; Tue, 14 Jun 2022 12:33:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1353854AbiFNMd5 (ORCPT ); Tue, 14 Jun 2022 08:33:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236258AbiFNMd3 (ORCPT ); Tue, 14 Jun 2022 08:33:29 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 351F94B1C9 for ; Tue, 14 Jun 2022 05:30:42 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id k19so11059011wrd.8 for ; Tue, 14 Jun 2022 05:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=D8sFDmp4NCj8bAwGZ38Oxd60P7D2GVHJ5ZCu/JRHls8=; b=Sd2BoO9uPXJfiWiTr5JcFwnY7ytE5K28EVFwgVpjKyK4XlVJUsLRV70zc47dRlZhq3 FjgqjPkFXE6qV0EOZnI8724xkWHbO3wOBoQT3hvNcKzkaLlDiUNbnOdCdXYb6uxOpofw IIalwZlFxcyXDLo7sciHYCIGAlna32x7V9p/TNzvjy1hfD0WMxBMadcOOkuQ7+U/oCW5 5UCesFYt/zI2yRqC37gUmjMohk+PU+AsaEHzTXSMXKU5kM9mrsl1ijI57EKa/kpsxZ8x 8uRQNtnqVymDDjDK/iSdSE+c1vUsal4j7lq0T/Mdo8QM28gna+ytkGqkZJsECzPh99/n geNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D8sFDmp4NCj8bAwGZ38Oxd60P7D2GVHJ5ZCu/JRHls8=; b=2y7petXuyTCsH0X/O+26ytXZR+Q9KgC1MOVjinc414q1jSXH1p72gDc/o/zvXb27CG AoKgpd+FJypUMgGTa2Smz6xOMvBJW0nQ7u45kgP1lnvxy09UZRMcnySleDAMGNNCZcJ0 TQ14RzFqZ12OlPBjvM1LIGPzfJsnpAPZLrZFyYvrXPwyDBalXoYWqF+1rFxpvWF19mCk VEyUmSjjKjAqZK7Im+CVeXT0axaxieCzQQ8hY/3hYmn37GlcUoPvPdre54mOgjAekZ+R BhQsckdt2EddMvwglfpJEQzHeOw3OTKDmrGpsJXNOsakkFKg3rY0oGoGxmvtZHSIWwwG W9QA== X-Gm-Message-State: AJIora//KuunKMRdwwp1uVC60Q7nmJrJoDzHpl+a7knszwNxVKN1rmcU 1h8A5A2zIujmNSjrsxB006l1XeteaVhJkg== X-Google-Smtp-Source: AGRyM1sAVWyXUL73oAUOVdxhbmDL+pNKmGw9HuLn/a8zYg7zgzp1GFnqGmPLZxW11DnoMiGE3967fw== X-Received: by 2002:adf:e5d0:0:b0:210:313a:e4dc with SMTP id a16-20020adfe5d0000000b00210313ae4dcmr4655230wrn.152.1655209839897; Tue, 14 Jun 2022 05:30:39 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 04/25] io_uring: refactor ctx slow data placement Date: Tue, 14 Jun 2022 13:29:42 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Shove all slow path data at the end of ctx and get rid of extra indention. Signed-off-by: Pavel Begunkov --- io_uring/io_uring_types.h | 81 +++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 42 deletions(-) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 4f52dcbbda56..ca8e25992ece 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -183,7 +183,6 @@ struct io_ring_ctx { struct list_head apoll_cache; struct xarray personalities; u32 pers_next; - unsigned sq_thread_idle; } ____cacheline_aligned_in_smp; /* IRQ completion list, under ->completion_lock */ @@ -230,23 +229,6 @@ struct io_ring_ctx { struct list_head io_buffers_comp; } ____cacheline_aligned_in_smp; - struct io_restriction restrictions; - - /* slow path rsrc auxilary data, used by update/register */ - struct { - struct io_rsrc_node *rsrc_backup_node; - struct io_mapped_ubuf *dummy_ubuf; - struct io_rsrc_data *file_data; - struct io_rsrc_data *buf_data; - - struct delayed_work rsrc_put_work; - struct llist_head rsrc_put_llist; - struct list_head rsrc_ref_list; - spinlock_t rsrc_ref_lock; - - struct list_head io_buffers_pages; - }; - /* timeouts */ struct { spinlock_t timeout_lock; @@ -257,30 +239,45 @@ struct io_ring_ctx { } ____cacheline_aligned_in_smp; /* Keep this last, we don't need it for the fast path */ - struct { - #if defined(CONFIG_UNIX) - struct socket *ring_sock; - #endif - /* hashed buffered write serialization */ - struct io_wq_hash *hash_map; - - /* Only used for accounting purposes */ - struct user_struct *user; - struct mm_struct *mm_account; - - /* ctx exit and cancelation */ - struct llist_head fallback_llist; - struct delayed_work fallback_work; - struct work_struct exit_work; - struct list_head tctx_list; - struct completion ref_comp; - - /* io-wq management, e.g. thread count */ - u32 iowq_limits[2]; - bool iowq_limits_set; - - struct list_head defer_list; - }; + + struct io_restriction restrictions; + + /* slow path rsrc auxilary data, used by update/register */ + struct io_rsrc_node *rsrc_backup_node; + struct io_mapped_ubuf *dummy_ubuf; + struct io_rsrc_data *file_data; + struct io_rsrc_data *buf_data; + + struct delayed_work rsrc_put_work; + struct llist_head rsrc_put_llist; + struct list_head rsrc_ref_list; + spinlock_t rsrc_ref_lock; + + struct list_head io_buffers_pages; + + #if defined(CONFIG_UNIX) + struct socket *ring_sock; + #endif + /* hashed buffered write serialization */ + struct io_wq_hash *hash_map; + + /* Only used for accounting purposes */ + struct user_struct *user; + struct mm_struct *mm_account; + + /* ctx exit and cancelation */ + struct llist_head fallback_llist; + struct delayed_work fallback_work; + struct work_struct exit_work; + struct list_head tctx_list; + struct completion ref_comp; + + /* io-wq management, e.g. thread count */ + u32 iowq_limits[2]; + bool iowq_limits_set; + + struct list_head defer_list; + unsigned sq_thread_idle; }; enum { From patchwork Tue Jun 14 12:29:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881118 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D802C433EF for ; Tue, 14 Jun 2022 12:34:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242448AbiFNMd6 (ORCPT ); Tue, 14 Jun 2022 08:33:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236203AbiFNMdj (ORCPT ); Tue, 14 Jun 2022 08:33:39 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B2BF4B1E3 for ; Tue, 14 Jun 2022 05:30:43 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id q15so11035735wrc.11 for ; Tue, 14 Jun 2022 05:30:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZERg2Xd6SMB5hSrov8OFN4NG5smSNMpMwnWVwGCVfsI=; b=TMF70ux0ORmxHcwA9Muq6mgAIcWz+yJpMzv9ijdikxSCoMmb0Udhtp3Ydb3I2p3qtu LJjc2OJXu53qCvTPces7PzxVVXjBy3Ac5wvvwvfAxTCjRsAgiJirY54qOB5xenw0R83d +/dmwKzG3g4NgC3dBtIGZO6UXHaC1eFuCkWipZjZhSAPHKULhzc/SD9SheOcv7Qm/cDe 5TdhEGwUSaLUsoSQPJkzn+AQ1geqRqJ8V0djrN5u6Ex6x8g9MIy4JagtseGrEWRSu3v7 9pYNoOk0VPJQywhdHexcGO9m8xzKfC++kLCjRp4oWOUZavo2mk0Y/op/j+hCWcYs8Yug Y+lQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZERg2Xd6SMB5hSrov8OFN4NG5smSNMpMwnWVwGCVfsI=; b=uiRB/Em90K8EKooV2P+51T5tdQmDrP/foSwNqa0wOiIL9/Q3lU5+wAxQbzt1FmWJi8 fW6VZT4PbwdXZHqRGLORj94O8y+5/rVxJHBlCj9Tr5AoHkePTBSINL0E3iR+Wy1/1Dpi Sgz5I2ZJBMv/0yYYVRbyWLfB/aOmbMXRaZlgcYnJFzTHa9Zk45loDqHxa3xhnyqfBUSu FN83h8rjZhE2jayIBQouietlVZoSkHxzH8Rl6vcrm7JiV4vhRhFkcEQTsZ7USiCoOMEn RLrqpYSVYsHV4Q09GP5WoBrcPjOcsGxtzfD+NNhXrIef9ScrW94IsqepyoAGSQLL+QHw l5fA== X-Gm-Message-State: AJIora+7S1ABF7zeaqhP6Q8kmw3ROEJJiBNk/hV0or+q2qc2jUfdtXTS ohCkpPu0yzLGVeCZoi1XMYnMa8tQBqcvpQ== X-Google-Smtp-Source: AGRyM1vhhogDCG+9fI8tX2qtX7pnPpP15MpF34zHouKVCPi2FfntlG4FTLPjWgfVm8fLlDiMGuuM6Q== X-Received: by 2002:a5d:6d0c:0:b0:216:2433:5317 with SMTP id e12-20020a5d6d0c000000b0021624335317mr4705534wrq.263.1655209841142; Tue, 14 Jun 2022 05:30:41 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:40 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 05/25] io_uring: move cancel_seq out of io-wq Date: Tue, 14 Jun 2022 13:29:43 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io-wq doesn't use ->cancel_seq, it's only important to io_uring and should be stored there. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 4 ++-- io_uring/io-wq.h | 1 - io_uring/io_uring.c | 2 +- io_uring/io_uring_types.h | 1 + io_uring/poll.c | 10 +++++----- io_uring/timeout.c | 4 ++-- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 83cceb52d82d..2e72231882b7 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -44,9 +44,9 @@ static bool io_cancel_cb(struct io_wq_work *work, void *data) return false; } if (cd->flags & (IORING_ASYNC_CANCEL_ALL|IORING_ASYNC_CANCEL_ANY)) { - if (cd->seq == req->work.cancel_seq) + if (cd->seq == req->cancel_seq) return false; - req->work.cancel_seq = cd->seq; + req->cancel_seq = cd->seq; } return true; } diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index ba6eee76d028..dbecd27656c7 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -155,7 +155,6 @@ struct io_wq_work_node *wq_stack_extract(struct io_wq_work_node *stack) struct io_wq_work { struct io_wq_work_node list; unsigned flags; - int cancel_seq; }; static inline struct io_wq_work *wq_next_work(struct io_wq_work *work) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6a94d1682aaf..af9188c8e2eb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -849,7 +849,7 @@ static void io_prep_async_work(struct io_kiocb *req) req->work.list.next = NULL; req->work.flags = 0; - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); + req->cancel_seq = atomic_read(&ctx->cancel_seq); if (req->flags & REQ_F_FORCE_ASYNC) req->work.flags |= IO_WQ_WORK_CONCURRENT; diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index ca8e25992ece..25e07c3f7b2a 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -486,6 +486,7 @@ struct io_kiocb { /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; struct io_wq_work work; + int cancel_seq; }; struct io_cancel_data { diff --git a/io_uring/poll.c b/io_uring/poll.c index 0df5eca93b16..b46973140ffd 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -405,7 +405,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, int v; INIT_HLIST_NODE(&req->hash_node); - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); + req->cancel_seq = atomic_read(&ctx->cancel_seq); io_init_poll_iocb(poll, mask, io_poll_wake); poll->file = req->file; @@ -565,9 +565,9 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, if (poll_only && req->opcode != IORING_OP_POLL_ADD) continue; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { - if (cd->seq == req->work.cancel_seq) + if (cd->seq == req->cancel_seq) continue; - req->work.cancel_seq = cd->seq; + req->cancel_seq = cd->seq; } return req; } @@ -589,9 +589,9 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, if (!(cd->flags & IORING_ASYNC_CANCEL_ANY) && req->file != cd->file) continue; - if (cd->seq == req->work.cancel_seq) + if (cd->seq == req->cancel_seq) continue; - req->work.cancel_seq = cd->seq; + req->cancel_seq = cd->seq; return req; } } diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 69cca42d6835..89000aae65d9 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -227,9 +227,9 @@ static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx, cd->data != tmp->cqe.user_data) continue; if (cd->flags & (IORING_ASYNC_CANCEL_ALL|IORING_ASYNC_CANCEL_ANY)) { - if (cd->seq == tmp->work.cancel_seq) + if (cd->seq == tmp->cancel_seq) continue; - tmp->work.cancel_seq = cd->seq; + tmp->cancel_seq = cd->seq; } req = tmp; break; From patchwork Tue Jun 14 12:29:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881119 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC0BAC43334 for ; Tue, 14 Jun 2022 12:34:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241187AbiFNMd7 (ORCPT ); Tue, 14 Jun 2022 08:33:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235189AbiFNMdk (ORCPT ); Tue, 14 Jun 2022 08:33:40 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4729C4B1EB for ; Tue, 14 Jun 2022 05:30:44 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id k19so11059150wrd.8 for ; Tue, 14 Jun 2022 05:30:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tcnT183OYNPa4vdT2GoOS6WMF6bmKM6MUsSlmmhun7A=; b=hlX0CdHLf6LcGW5CKOCY2RKXzHPOOz8THFZNt0wiqq/l5jhfNAKXCCtzaCfVoQP+lb kr9kG4rzaLnT9q/85r2NPTsmuj3YfGnlZ62Yhe42lxZ8tzU+0RN8ow2hc430V/MB5GAS vxPiDtNOn8y6Iu6FQKYvnL91b6VUjBPRucSh8I0O4LXA9JHlzTlkKIKw+B65KGyHM9ea yJflp1jWpG+81q0NCLpxio7ibtlWBP+V88UW6CZUQIeEQ0zavu03X0iQy2EAmevk/oIO wrs5d00PSZ1PVRjXQLZZFCghKUGKKbNtEFRv4rTCTdWfut1rYInHUf0T8VJC27r6ccpg 88QQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tcnT183OYNPa4vdT2GoOS6WMF6bmKM6MUsSlmmhun7A=; b=0QjM6NpZMQQ7czAuizQ2Pskado1uiM4/quM2OJeqs+wwk5uIAf6QjY0Ptjew9hg6ff 1nM5ijVvaZhxod2FXmiIi7dUJB5spq2sksjpPFWVe0aIVLzzfV3b2enS7Q+7gxc8O3CF ld0I4L2HFpoOjANqTXtS5B2YKuGWHhdX2ulTEAxcUuYoIgZ5I2Yu0s2HNKOoM8p9MJLJ nd945iMwbzFAR/K013zYR8LJz3/Qe8AyHRSHk8gn7T8gRZgoutRVKMSfHFF3Oxa/FVyI dtBM6gD3DHAZm9Kv0j3SRqs3OzmtD/VhScsPRycAa7OF+GB1m7WW1pWWVZ9vbjsPa8Ij /IKQ== X-Gm-Message-State: AJIora/55sQrVcbPp7pF9Q9CNNyGJpl/MXOPnbhx/0jC2pTIGaRpEoeK G7R/NDqSnG9tXkLOl+C2vhpRAGOXddkzKQ== X-Google-Smtp-Source: AGRyM1veVvLJqJenIPmY7d6F8waNbakyiOsDBFctjMPiBmPuSRQnFYlN7yfl/U/uhV435yGyiDT8Nw== X-Received: by 2002:a05:6000:156d:b0:210:3125:6012 with SMTP id 13-20020a056000156d00b0021031256012mr4525595wrz.357.1655209842417; Tue, 14 Jun 2022 05:30:42 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 06/25] io_uring: move small helpers to headers Date: Tue, 14 Jun 2022 13:29:44 +0100 Message-Id: <263af7a3f2f40130c26969877381567ce66870d9.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There is a bunch of inline helpers that will be useful not only to the core of io_uring, move them to headers. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 22 ---------------------- io_uring/io_uring.h | 22 ++++++++++++++++++++++ 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index af9188c8e2eb..2f5fd1749c4a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -616,14 +616,6 @@ struct sock *io_uring_get_socket(struct file *file) } EXPORT_SYMBOL(io_uring_get_socket); -static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) -{ - if (!*locked) { - mutex_lock(&ctx->uring_lock); - *locked = true; - } -} - static inline void io_submit_flush_completions(struct io_ring_ctx *ctx) { if (!wq_list_empty(&ctx->submit_state.compl_reqs)) @@ -879,15 +871,6 @@ static void io_prep_async_link(struct io_kiocb *req) } } -static inline void io_req_add_compl_list(struct io_kiocb *req) -{ - struct io_submit_state *state = &req->ctx->submit_state; - - if (!(req->flags & REQ_F_CQE_SKIP)) - state->flush_cqes = true; - wq_list_add_tail(&req->comp_list, &state->compl_reqs); -} - void io_queue_iowq(struct io_kiocb *req, bool *dont_use) { struct io_kiocb *link = io_prep_linked_timeout(req); @@ -1293,11 +1276,6 @@ static void io_req_complete_post32(struct io_kiocb *req, u64 extra1, u64 extra2) io_cqring_ev_posted(ctx); } -static inline void io_req_complete_state(struct io_kiocb *req) -{ - req->flags |= REQ_F_COMPLETE_INLINE; -} - inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) { if (issue_flags & IO_URING_F_COMPLETE_DEFER) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 3660df80e589..26b669746d61 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -193,6 +193,28 @@ static inline bool io_run_task_work(void) return false; } +static inline void io_req_complete_state(struct io_kiocb *req) +{ + req->flags |= REQ_F_COMPLETE_INLINE; +} + +static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) +{ + if (!*locked) { + mutex_lock(&ctx->uring_lock); + *locked = true; + } +} + +static inline void io_req_add_compl_list(struct io_kiocb *req) +{ + struct io_submit_state *state = &req->ctx->submit_state; + + if (!(req->flags & REQ_F_CQE_SKIP)) + state->flush_cqes = true; + wq_list_add_tail(&req->comp_list, &state->compl_reqs); +} + int io_run_task_work_sig(void); void io_req_complete_failed(struct io_kiocb *req, s32 res); void __io_req_complete32(struct io_kiocb *req, unsigned int issue_flags, From patchwork Tue Jun 14 12:29:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881120 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33C24C43334 for ; Tue, 14 Jun 2022 12:34:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235345AbiFNMd7 (ORCPT ); Tue, 14 Jun 2022 08:33:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243863AbiFNMdm (ORCPT ); Tue, 14 Jun 2022 08:33:42 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E47A4B1FE for ; Tue, 14 Jun 2022 05:30:45 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id e5so4586644wma.0 for ; Tue, 14 Jun 2022 05:30:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=79kYGB1MuhBVdSp/oRQj0eo+ojhHpblZthnUV8e/Yck=; b=Ug/njMd0L0l6sDO0E3omxepK8SKj0RH0+yxjCQd4mL9Z0uTBTMdcKy8hjOw8+ttZ5R 7sIdO0xMS4LeqZLd2SwuC+BKBmKsWVKnD9yubN0U+ViBMjwOI6pEfQHTv/FsGAUDjlkA bgAKuFBENiTAV53CbpR7VTGv26a1J3ePfG6zRomRo40J3LEkp1yTyTi1Zgl/UhofMp/I 9sX/AC1Xc3L0jCqx6C7e8jqDUhOBTOWBaH31mHFTQchHYXSDvhESjYm55wYUjw8VVlxO SVHfb+ty4X5JvtjJAsFx/aOZCh1owLH2eG49+2DhXMR0qQQrHyflDNl6AFPpwtPHgwzN C1CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=79kYGB1MuhBVdSp/oRQj0eo+ojhHpblZthnUV8e/Yck=; b=b+HZpyLU8j7QlUhd9fWof15nggKi7MoLZ/Nba/LiQ3s7w/N790ycy3bkyObj/AqZDN QoV8sHqsBzeVPN3+6Bk/UXv5c0bujJT+hihdTeiLKaPxiu6Wnl4v5YnHJkgcYJhLqXyL gZkPggJil36/5f1ETfAiif81eGtq+KoXWy2YEBV22VNQGw6wYpfPgfO7Jh+SQD4tzaes TFCu7YUsFFk8ZXvcmagO2Ion0lQ1FvfYR6I5sBea3tjK3Ogcxl7deDozFHqIVqUdoU6j HRil5C+Age/Iui8tWao3azKfLKm3I52UdDR85ZDpog6MQwp0lc8S2Cx3PwRt83pqs8Js C/0g== X-Gm-Message-State: AOAM533g24SfcGniGNZpPbYTtf6+/qTblcA8HjugZ2CXcej8iJod9jkX NpxUnsuPeSwGXfQsLnFypapVhNFhVh3QWQ== X-Google-Smtp-Source: ABdhPJx/Tkrj6dvGiLtf3j2O9F4wwbbQsS8hHFnQBJPENhf+KZjTiFFle02+bKR8K4p/5m2xdhewtw== X-Received: by 2002:a05:600c:8a6:b0:39c:5682:32d with SMTP id l38-20020a05600c08a600b0039c5682032dmr3884388wmp.126.1655209843581; Tue, 14 Jun 2022 05:30:43 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:43 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 07/25] io_uring: inline ->registered_rings Date: Tue, 14 Jun 2022 13:29:45 +0100 Message-Id: <3b98818d59d19262712dea1893c87dc186a584e8.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There can be only 16 registered rings, no need to allocate an array for them separately but store it in tctx. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - io_uring/tctx.c | 9 --------- io_uring/tctx.h | 3 ++- 3 files changed, 2 insertions(+), 11 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2f5fd1749c4a..2efe6bd16e07 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2840,7 +2840,6 @@ void __io_uring_free(struct task_struct *tsk) WARN_ON_ONCE(tctx->io_wq); WARN_ON_ONCE(tctx->cached_refs); - kfree(tctx->registered_rings); percpu_counter_destroy(&tctx->inflight); kfree(tctx); tsk->io_uring = NULL; diff --git a/io_uring/tctx.c b/io_uring/tctx.c index f3262eef55d4..6adf659687f8 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -55,16 +55,8 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, if (unlikely(!tctx)) return -ENOMEM; - tctx->registered_rings = kcalloc(IO_RINGFD_REG_MAX, - sizeof(struct file *), GFP_KERNEL); - if (unlikely(!tctx->registered_rings)) { - kfree(tctx); - return -ENOMEM; - } - ret = percpu_counter_init(&tctx->inflight, 0, GFP_KERNEL); if (unlikely(ret)) { - kfree(tctx->registered_rings); kfree(tctx); return ret; } @@ -73,7 +65,6 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, if (IS_ERR(tctx->io_wq)) { ret = PTR_ERR(tctx->io_wq); percpu_counter_destroy(&tctx->inflight); - kfree(tctx->registered_rings); kfree(tctx); return ret; } diff --git a/io_uring/tctx.h b/io_uring/tctx.h index f4964e40d07e..7684713e950f 100644 --- a/io_uring/tctx.h +++ b/io_uring/tctx.h @@ -20,8 +20,9 @@ struct io_uring_task { struct io_wq_work_list task_list; struct io_wq_work_list prio_task_list; struct callback_head task_work; - struct file **registered_rings; bool task_running; + + struct file *registered_rings[IO_RINGFD_REG_MAX]; }; struct io_tctx_node { From patchwork Tue Jun 14 12:29:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881121 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A1B7C43334 for ; Tue, 14 Jun 2022 12:34:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240854AbiFNMeE (ORCPT ); Tue, 14 Jun 2022 08:34:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244265AbiFNMdm (ORCPT ); Tue, 14 Jun 2022 08:33:42 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A62D64B411 for ; Tue, 14 Jun 2022 05:30:46 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id n185so4556318wmn.4 for ; Tue, 14 Jun 2022 05:30:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2CCQsCTUyQprcqQdLzZjdfjxbl316XgfMbO5tlRT3XQ=; b=JuD9WnmYUsWoW1ZJczovNjH2Wqh51+TEM+Zmaw0XNnFPByMhEuHzwTcHYdyScbxLlo UrP5SwI4WrQzDXuGhjiKJO5bjYdkJwhbvw8jySYoVYTsDBnkoqn3DqCQ/kaMa8WK5HVD sDE153HUbQdvwMUo1G8N1QyoY0xw3WpuFzWfscyzBKdMkiDh+9vXA1AAwI3JLQk73xGf LjGr09jFyU8ALhUhuzDuVWWxkmTCL1xQkRDVjuJEph5QFw0hROkMSRDWaGuUDkUEF+TE yjRQJe3BVEEoc8lG8H+pGSaNstEFuHulQSNKGFABytZmurOMQXb+QGQpO8TP/8osP6fP j2GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2CCQsCTUyQprcqQdLzZjdfjxbl316XgfMbO5tlRT3XQ=; b=ZFwxgRx259BHS0g5BL0h+LS4B5+mJtPj5jTzB5OyeZXlq1IAM2G9Q8NclvX1BLvtt1 8LjsH7eCAtTyxRbd2Nw/MhCRtuWUE6PQAnZxf8zqmXghzxdgZ2jIcYlYHvdRA7hWRhpi Hp4FvEOV7x0nDp+8umowXMfozvKOH9zzFrrbRGUlNmHCA2sQtmJmDJBshM3E2NkJGEzf mCcQCzyhXied/c8T/SfKigjU/1EidFH3TCnonlvZ3O/Zp1zABYYchw7QqBeJofS0wd4R 7cW1vFEU7keQ7WXAoTYjHqBjg04rW0uAu0olNrQCgrgk266hbSc1JYT6k7WuQVFHGqT1 zUmw== X-Gm-Message-State: AOAM53039LnVGPZQZa6nVnSBbewD50N7RcYSOHsbE1xY6gPSKLN2sj+Y Cht3u2pISrZmhdCS6KGP6W0fF//M7zhXgQ== X-Google-Smtp-Source: ABdhPJyYMaMqjh/UKNWiBJX0HsHj+9YVK9PjmxM3z/0bwuxh6HMjMa09Q6ULhgNThsLE2EaH7agQ3Q== X-Received: by 2002:a05:600c:1c18:b0:39c:4f19:c37a with SMTP id j24-20020a05600c1c1800b0039c4f19c37amr3937284wms.69.1655209844882; Tue, 14 Jun 2022 05:30:44 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 08/25] io_uring: don't set REQ_F_COMPLETE_INLINE in tw Date: Tue, 14 Jun 2022 13:29:46 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_req_task_complete() enqueues requests for state completion itself, no need for REQ_F_COMPLETE_INLINE, which is only serve the purpose of not bloating the kernel. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2efe6bd16e07..8ce8d2516704 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1864,7 +1864,6 @@ inline void io_req_task_complete(struct io_kiocb *req, bool *locked) { if (*locked) { req->cqe.flags |= io_put_kbuf(req, 0); - io_req_complete_state(req); io_req_add_compl_list(req); } else { req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED); From patchwork Tue Jun 14 12:29:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C092BC43334 for ; Tue, 14 Jun 2022 12:34:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234267AbiFNMeO (ORCPT ); Tue, 14 Jun 2022 08:34:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232988AbiFNMdo (ORCPT ); Tue, 14 Jun 2022 08:33:44 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 770F84B429 for ; Tue, 14 Jun 2022 05:30:48 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id u8so11042186wrm.13 for ; Tue, 14 Jun 2022 05:30:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5BK3pf+Ob1QUOBk18Dp6t/3CfaS7xXLjPvB8wT77wB0=; b=gHW2Yc1Q3mf2pomD74eLIngyMMfcjA7psHXSdtKMOg1DEC8Y2FspfFdpSirnWL3LjC 1oes8wkm3NwM9BUzVdw443qRkCcCOZz0XMZEOVGoL5gT0ALJiptGZX2RM58S2chqTHsO Y320LqI/1QmdnDxqNHBf1HmhqWwWdXUmJLz6FLSoRN0ZNPp5EULwPa9hO2Jc+78pNkCp bwcalvBnWa9xc6otdN4ClkaVyMOadAAHVkqB3Jyo4UBDbGL/DaZq4qMkIMiH1Dgd21lt 9V7THk7u0gt6ctkMoIp2B5IgoDVfTpb2Zl4VJHO7lNHkCfrt5u7oJ1qCnFpNBF6scClf RiBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5BK3pf+Ob1QUOBk18Dp6t/3CfaS7xXLjPvB8wT77wB0=; b=Z8U7IVXj6ji9DjRNvgn0ggmEcCe41Uo+fEiNq4rsNEbXQUIoZc1ue2MPuESyoT/nea GAcTkLLhDjkQpCTqIt+LTH7dVvazC6XrjYFm/gyxf7W2x5qHz4QXINw+o+cXmAXxqDCX dNyKfeK1R2YT/1KHGxrFjmveEb3UMV5j9nwr3txrCGEKK1/W0KVeu5cIIQtXAR5TNWlf tm0uiTQg/iLLGoXoUsKZ36ZZdpQzwgMEwskloInjFRqq2FfkJcs6h6WRfbBxTkan3pbW JIdjpnp1ubmqZVBSbupnvNhfRRjLq74US3IRZxiHhEQFbLwAa5WqqWCCm1N6Vb9VZqFJ JhNQ== X-Gm-Message-State: AJIora9PfzbgD4SctjiW2K3JZrmQrfj0wx3LDnTBKTOLo7CrO62DmQC+ 7XIWgZ+LsIYVvIp4Aw2VkLCXHTKHgkfwFA== X-Google-Smtp-Source: AGRyM1tcbRFwVYIPXaW+CRBc4OIUiWKo1cGTRzN7VKOZVMcHqoIkok+gIh2/fcrKZHTrAdJvadiyTg== X-Received: by 2002:a5d:6d8c:0:b0:217:a419:c3f8 with SMTP id l12-20020a5d6d8c000000b00217a419c3f8mr4649548wrs.260.1655209846292; Tue, 14 Jun 2022 05:30:46 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:45 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 09/25] io_uring: never defer-complete multi-apoll Date: Tue, 14 Jun 2022 13:29:47 +0100 Message-Id: <1bb69d5cfe760bec1b2b2a6405ccbbede37cb9d7.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Luckily, nnobody completes multi-apoll requests outside the polling functions, but don't set IO_URING_F_COMPLETE_DEFER in any case as there is nobody who is catching REQ_F_COMPLETE_INLINE, and so will leak requests if used. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8ce8d2516704..5156844ca2bb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2149,7 +2149,7 @@ int io_poll_issue(struct io_kiocb *req, bool *locked) io_tw_lock(req->ctx, locked); if (unlikely(req->task->flags & PF_EXITING)) return -EFAULT; - return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); + return io_issue_sqe(req, IO_URING_F_NONBLOCK); } struct io_wq_work *io_wq_free_work(struct io_wq_work *work) From patchwork Tue Jun 14 12:29:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12BB6C43334 for ; Tue, 14 Jun 2022 12:34:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243296AbiFNMeY (ORCPT ); Tue, 14 Jun 2022 08:34:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243075AbiFNMdp (ORCPT ); Tue, 14 Jun 2022 08:33:45 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E01C4B42C for ; Tue, 14 Jun 2022 05:30:49 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id m125-20020a1ca383000000b0039c63fe5f64so4731124wme.0 for ; Tue, 14 Jun 2022 05:30:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lDOepxPFCtlbou3NV1mYJpDfthiiMh/+cWslKUPd3nw=; b=PaA2PEcaqpf8zNxLiRGz0FCKXMQrimxszEFMFv1ZWcvlPklkGW+pH9GgBF/72+w20O OBpTNccsAN3aTySKJacIZzU7ISAW/Yl850aQbzoc6OZCmOgfCs2yARxJz4DPRqfs2T4i yyO9CJwZY63/8RjYNs/LNkyCXnp3di8w+5d4IgmmgaSH+aQvXV15UZwctJjFtnoIFYPO uf/xjD0kAi6OaRu2Bhy3eq44E8GBjbFphgla0mhTWEZZOL7KjYdiDKkdMrKLFZGBT4cD E5Kt4kbVKJVQRtOA0Xpa/zh3oqYsWFKlmT5udgJqwE/FkUj2Ruy67qzcefSyv5f5G2wA /Fgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lDOepxPFCtlbou3NV1mYJpDfthiiMh/+cWslKUPd3nw=; b=ycMtj86AO+VotFw418BAEKXLuaStG2NzdZ1jWwFoT93U7ap1yQC09XEdLjaA55mGmC QExDqE0YVvIiSJwugSVVsbGD3MN1rIiwFHJGnx/cJmZeNIYhIzhCmAG4uOK2fNnxHzbH 2anpTTssr1GdN9Ewi++YwxKUR+fytaBecdyWn2RYNXD4GLXHoQTLUFwBIzskhXziVMV7 Avssf7okzAP50GO2PvMbBScrnCgkh5BwcI91fc6kp2nn/axY2w2tLKlYs2addphXLjdp 4JIVeXzRNUclfzOKRRKtkmWO5X1SoCnDJ1ukP0Uss4nTbKqbMZKQ7+MbdnWbqfFn+8h0 2sjA== X-Gm-Message-State: AOAM530XSlMZyb6mQ6eIgg0US/v0S3Xq1wVCSF6XXMfwMEyEziRu0Lav M3vxzc2gs9e70DNWL2TDa6f4fqrlWoFoeQ== X-Google-Smtp-Source: ABdhPJzLgy/Pu/rruOs2FlE0fqnwtTACFsKG3goLm24dssi3jaZJzFJwglc2IUWVBUs0pb26gOsdcA== X-Received: by 2002:a7b:c3d1:0:b0:39c:64d0:b8c2 with SMTP id t17-20020a7bc3d1000000b0039c64d0b8c2mr3969898wmj.195.1655209847489; Tue, 14 Jun 2022 05:30:47 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:46 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 10/25] io_uring: kill REQ_F_COMPLETE_INLINE Date: Tue, 14 Jun 2022 13:29:48 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org REQ_F_COMPLETE_INLINE is only needed to delay queueing into the completion list to io_queue_sqe() as __io_req_complete() is inlined and we don't want to bloat the kernel. As now we complete in a more centralised fashion in io_issue_sqe() we can get rid of the flag and queue to the list directly. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 20 ++++++++------------ io_uring/io_uring.h | 5 ----- io_uring/io_uring_types.h | 3 --- 3 files changed, 8 insertions(+), 20 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 5156844ca2bb..6c48d0c6dcd5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1278,17 +1278,14 @@ static void io_req_complete_post32(struct io_kiocb *req, u64 extra1, u64 extra2) inline void __io_req_complete(struct io_kiocb *req, unsigned issue_flags) { - if (issue_flags & IO_URING_F_COMPLETE_DEFER) - io_req_complete_state(req); - else - io_req_complete_post(req); + io_req_complete_post(req); } void __io_req_complete32(struct io_kiocb *req, unsigned int issue_flags, u64 extra1, u64 extra2) { if (issue_flags & IO_URING_F_COMPLETE_DEFER) { - io_req_complete_state(req); + io_req_add_compl_list(req); req->extra1 = extra1; req->extra2 = extra2; } else { @@ -2132,9 +2129,12 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if (creds) revert_creds(creds); - if (ret == IOU_OK) - __io_req_complete(req, issue_flags); - else if (ret != IOU_ISSUE_SKIP_COMPLETE) + if (ret == IOU_OK) { + if (issue_flags & IO_URING_F_COMPLETE_DEFER) + io_req_add_compl_list(req); + else + io_req_complete_post(req); + } else if (ret != IOU_ISSUE_SKIP_COMPLETE) return ret; /* If the op doesn't have a file, we're not polling for it */ @@ -2299,10 +2299,6 @@ static inline void io_queue_sqe(struct io_kiocb *req) ret = io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_COMPLETE_DEFER); - if (req->flags & REQ_F_COMPLETE_INLINE) { - io_req_add_compl_list(req); - return; - } /* * We async punt it if the file wasn't marked NOWAIT, or if the file * doesn't support non-blocking read/write attempts diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 26b669746d61..2141519e995a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -193,11 +193,6 @@ static inline bool io_run_task_work(void) return false; } -static inline void io_req_complete_state(struct io_kiocb *req) -{ - req->flags |= REQ_F_COMPLETE_INLINE; -} - static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked) { if (!*locked) { diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 25e07c3f7b2a..3cf06f4a4d2e 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -299,7 +299,6 @@ enum { REQ_F_POLLED_BIT, REQ_F_BUFFER_SELECTED_BIT, REQ_F_BUFFER_RING_BIT, - REQ_F_COMPLETE_INLINE_BIT, REQ_F_REISSUE_BIT, REQ_F_CREDS_BIT, REQ_F_REFCOUNT_BIT, @@ -353,8 +352,6 @@ enum { REQ_F_BUFFER_SELECTED = BIT(REQ_F_BUFFER_SELECTED_BIT), /* buffer selected from ring, needs commit */ REQ_F_BUFFER_RING = BIT(REQ_F_BUFFER_RING_BIT), - /* completion is deferred through io_comp_state */ - REQ_F_COMPLETE_INLINE = BIT(REQ_F_COMPLETE_INLINE_BIT), /* caller should reissue async */ REQ_F_REISSUE = BIT(REQ_F_REISSUE_BIT), /* supports async reads/writes */ From patchwork Tue Jun 14 12:29:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4216BCCA47A for ; Tue, 14 Jun 2022 12:34:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355711AbiFNMeZ (ORCPT ); Tue, 14 Jun 2022 08:34:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243391AbiFNMdp (ORCPT ); Tue, 14 Jun 2022 08:33:45 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9734F4B439 for ; Tue, 14 Jun 2022 05:30:50 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id v14so11084548wra.5 for ; Tue, 14 Jun 2022 05:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TB/1dEsnzFxD0vauIvqVRckM4LeY6dlF3gUvCxeBMs0=; b=WN6tZRLO0TM2yBLdd+enYfc89A9Y2ui3aSt2F1RIOufNkdN5iGXF+g6kOEDnr30I86 OgGNu+s5pENC5psU3EWw+QQOzDf4gLUdqmFgbeNWwyjrzo0nMxXTUaWtzzQduAD+jdQH TEbMHdGTqnGxbu2YoyJW1MfKmDLfV5we8jlyvTNAtiru/ReJodyU9hToqtLVVDelz9/l F9l+0Q8r08SQCwW3Gye8RyGuN2KfNh4ZTrY8coxU823f6jArSJsn9aTI8R7HnUakpSoz zOeDOwNqxPw06uaGO90bzbD+NQ/UYILQD8df0A5LQVL4B4YoFLfuWRYX351CgpWvgG2B kf4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TB/1dEsnzFxD0vauIvqVRckM4LeY6dlF3gUvCxeBMs0=; b=T7LNgUGiyS+/3wYm886ctYGOyJdwKAzECQkIhH6OPlRWHvm6f3m6VfDGr9VUD9rePT NK8+GQV0/35KmpxrpwxsCKOcnjxJYgIu/rYNYYmLwAstULznCWyuLSiKx/ekhbI8fCQ4 U62LzdR9zPvxIIa+JEftp6g7Ub1OWxMMnhbK1B4qKe95SVDFQYGmdrp4onvLGryevE5h hZZbzuwcIbym2IIQ0MzbrRF9bQafqN429WjRzEm5hoqh0dt/zMZV2INMoT2iXfR1keTA 4KoRJf1GipQJ2/DCD4lVAsNQ7+RLrSh4+NBw89g+FPhwsJwl+vaLEXM3ueFG6/FPbMUg R85g== X-Gm-Message-State: AJIora8OvJdCTk7y9Aw1UW5DHYf5OBjyOdzP7GmW87x+4XS97ES2VCIt kTFuCskFh+9v6Yb52S5QtfgHwwMuFBElNA== X-Google-Smtp-Source: AGRyM1sVQrWiZ4+QZnlzgBbsvAvCe6JrYZXbrWAbjXiOtdkZMTYELKltc9Doon/2sWL0oOZCxYDPXQ== X-Received: by 2002:adf:ecce:0:b0:219:e5a7:64a8 with SMTP id s14-20020adfecce000000b00219e5a764a8mr4787538wro.278.1655209848684; Tue, 14 Jun 2022 05:30:48 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 11/25] io_uring: refactor io_req_task_complete() Date: Tue, 14 Jun 2022 13:29:49 +0100 Message-Id: <517340728990899f3bdd23a2b18929409cae0718.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Clean up io_req_task_complete() and deduplicate io_put_kbuf() calls. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 6c48d0c6dcd5..f3cae98471b8 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1857,15 +1857,19 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) return ret; } -inline void io_req_task_complete(struct io_kiocb *req, bool *locked) + +void io_req_task_complete(struct io_kiocb *req, bool *locked) { - if (*locked) { - req->cqe.flags |= io_put_kbuf(req, 0); + if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { + unsigned issue_flags = *locked ? IO_URING_F_UNLOCKED : 0; + + req->cqe.flags |= io_put_kbuf(req, issue_flags); + } + + if (*locked) io_req_add_compl_list(req); - } else { - req->cqe.flags |= io_put_kbuf(req, IO_URING_F_UNLOCKED); + else io_req_complete_post(req); - } } /* From patchwork Tue Jun 14 12:29:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C2C2C433EF for ; Tue, 14 Jun 2022 12:34:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244404AbiFNMe0 (ORCPT ); Tue, 14 Jun 2022 08:34:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244942AbiFNMdv (ORCPT ); Tue, 14 Jun 2022 08:33:51 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF1D14B850 for ; Tue, 14 Jun 2022 05:30:51 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id l2-20020a05600c4f0200b0039c55c50482so6160899wmq.0 for ; Tue, 14 Jun 2022 05:30:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fBuzqx+gzC5/J7sqLEp0sAY1HfeFLOvDJjV1YuH4kBU=; b=bMqwj6XSkvQX4CWwa0hFVQ4ImMNMs96CmxzdwYKER52tRrGxzveUlZieIHF23UBFxy MlzIKBu5mGMEA0IwsLy2sW0lJN0hSZHR6R6pkTXq8TSmv3qsgSJoaBZh0/fHhFGkzO/q Qo4hkP1ZX7BJtjU6+F9DnHP6KXXed5CgkANiEeys5fMSlOkbcxdJpuGsjSFjyWdHYHV7 ePvLWDxrDhGc8X3+4YJ8L5yNuyZE6eZm7DKom2tquTb5CmuD07WUcX63kDSthhmejr11 0H2ibYuxvXXHSbrqM4OlDhTAZjGVGe8aNN1E9rsTjH+Afaegq9GaTj4MEX04U/DvBNFb j76w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fBuzqx+gzC5/J7sqLEp0sAY1HfeFLOvDJjV1YuH4kBU=; b=NJpaWtHtHsCAw3l29VhEn+JgvPZOBnugq1FQPvDMWBWGXKak+6d9n439vl/i0E4qYK R+wHpO1ohSvhc52WO0si3RStDqk90y5vXniHamvr1V8Ppiay/IzfeoyfDDAzcLmBSwEe 0K2v8rOHn8jLNGkXv+sK9ate2V4+3lQc6RwopXRon0J6T07UVLMM/SXseon4eIYADAh5 cxgp/lNuGysKktadYkIbBT5JgX2RKPfDGwSI3xSzyv2tAgWtzWViZXovKTsGTnoJJcsx +mOK6OmLiVk0W6Tbqp0/UqbTrnNqzhCRoGn35n+MhV8GN+fQulo6NigJkcAeycS0Anqm RcjA== X-Gm-Message-State: AOAM531/GjRl8GUSsxVp+ady3IOMayctP98wpKMhz+vZhjx2TGzHVb2f rN2BWQQaFG4M+uTO719Au0m1e7WaJMisPQ== X-Google-Smtp-Source: ABdhPJydbLtvL9CooGEWOM1X7dIkkJNnqUOnr1v3in/bOboA2sOEEkdIiXokmvf/r43l/A9dvux9iA== X-Received: by 2002:a7b:c205:0:b0:39c:506d:e294 with SMTP id x5-20020a7bc205000000b0039c506de294mr3869257wmi.159.1655209849894; Tue, 14 Jun 2022 05:30:49 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:49 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 12/25] io_uring: don't inline io_put_kbuf Date: Tue, 14 Jun 2022 13:29:50 +0100 Message-Id: <84e7d777da1c5cff7d34ec7fa65ad93789a09baf.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_put_kbuf() is huge, don't bloat the kernel with inlining. Signed-off-by: Pavel Begunkov --- io_uring/kbuf.c | 33 +++++++++++++++++++++++++++++++++ io_uring/kbuf.h | 38 ++++++-------------------------------- 2 files changed, 39 insertions(+), 32 deletions(-) diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 9cdbc018fd64..6f2adb481a0d 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -81,6 +81,39 @@ static int io_buffer_add_list(struct io_ring_ctx *ctx, return xa_err(xa_store(&ctx->io_bl_xa, bgid, bl, GFP_KERNEL)); } +unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) +{ + unsigned int cflags; + + /* + * We can add this buffer back to two lists: + * + * 1) The io_buffers_cache list. This one is protected by the + * ctx->uring_lock. If we already hold this lock, add back to this + * list as we can grab it from issue as well. + * 2) The io_buffers_comp list. This one is protected by the + * ctx->completion_lock. + * + * We migrate buffers from the comp_list to the issue cache list + * when we need one. + */ + if (req->flags & REQ_F_BUFFER_RING) { + /* no buffers to recycle for this case */ + cflags = __io_put_kbuf_list(req, NULL); + } else if (issue_flags & IO_URING_F_UNLOCKED) { + struct io_ring_ctx *ctx = req->ctx; + + spin_lock(&ctx->completion_lock); + cflags = __io_put_kbuf_list(req, &ctx->io_buffers_comp); + spin_unlock(&ctx->completion_lock); + } else { + lockdep_assert_held(&req->ctx->uring_lock); + + cflags = __io_put_kbuf_list(req, &req->ctx->io_buffers_cache); + } + return cflags; +} + static void __user *io_provided_buffer_select(struct io_kiocb *req, size_t *len, struct io_buffer_list *bl) { diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index 80b6df2c7535..5da3d4039aed 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -47,6 +47,8 @@ int io_provide_buffers(struct io_kiocb *req, unsigned int issue_flags); int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); int io_unregister_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg); +unsigned int __io_put_kbuf(struct io_kiocb *req, unsigned issue_flags); + static inline bool io_do_buffer_select(struct io_kiocb *req) { if (!(req->flags & REQ_F_BUFFER_SELECT)) @@ -70,7 +72,8 @@ static inline void io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags) __io_kbuf_recycle(req, issue_flags); } -static unsigned int __io_put_kbuf(struct io_kiocb *req, struct list_head *list) +static inline unsigned int __io_put_kbuf_list(struct io_kiocb *req, + struct list_head *list) { if (req->flags & REQ_F_BUFFER_RING) { if (req->buf_list) @@ -90,44 +93,15 @@ static inline unsigned int io_put_kbuf_comp(struct io_kiocb *req) if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return 0; - return __io_put_kbuf(req, &req->ctx->io_buffers_comp); + return __io_put_kbuf_list(req, &req->ctx->io_buffers_comp); } static inline unsigned int io_put_kbuf(struct io_kiocb *req, unsigned issue_flags) { - unsigned int cflags; if (!(req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING))) return 0; - - /* - * We can add this buffer back to two lists: - * - * 1) The io_buffers_cache list. This one is protected by the - * ctx->uring_lock. If we already hold this lock, add back to this - * list as we can grab it from issue as well. - * 2) The io_buffers_comp list. This one is protected by the - * ctx->completion_lock. - * - * We migrate buffers from the comp_list to the issue cache list - * when we need one. - */ - if (req->flags & REQ_F_BUFFER_RING) { - /* no buffers to recycle for this case */ - cflags = __io_put_kbuf(req, NULL); - } else if (issue_flags & IO_URING_F_UNLOCKED) { - struct io_ring_ctx *ctx = req->ctx; - - spin_lock(&ctx->completion_lock); - cflags = __io_put_kbuf(req, &ctx->io_buffers_comp); - spin_unlock(&ctx->completion_lock); - } else { - lockdep_assert_held(&req->ctx->uring_lock); - - cflags = __io_put_kbuf(req, &req->ctx->io_buffers_cache); - } - - return cflags; + return __io_put_kbuf(req, issue_flags); } #endif From patchwork Tue Jun 14 12:29:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2FAAC43334 for ; Tue, 14 Jun 2022 12:34:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245146AbiFNMe0 (ORCPT ); Tue, 14 Jun 2022 08:34:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229720AbiFNMd4 (ORCPT ); Tue, 14 Jun 2022 08:33:56 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E3004B864 for ; Tue, 14 Jun 2022 05:30:53 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id m39-20020a05600c3b2700b0039c511ebbacso6133314wms.3 for ; Tue, 14 Jun 2022 05:30:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R4fEAW3IwnjCgPgFIiQUdI5m2WItkDGjr8AmxNsgZaI=; b=PHPZ8jjekPVuUliMXV5pGuBP/yF1BZuT0iorpX/003+XR5JkQ45gMD64UsgGzXA24h nwNTybqwZT5I6T2vNHRGDoeAP9FFOzq2Gh24lfv+FZJG/JMsw8yxYqGlOqgjB8VsN3+x G7EcacKwJG467tuiW4GUB/gkXvYVz5AdKiB9Lr4HKNIgDRLN7OblQ/NIBivWdEFh3yk8 MtlX1BS/pqhdcrOlLK5J3H4Oqt+9Y+F9KfY+cDXEu1oeYAa6KpCtzwGtJx+teiSQHKSL UTbPuBn7Lt3wRRIm7O7P/H6UmJiMd1oyrYjEm8BTFpj3qYe10VtAuBonvIWOso3+YWF4 l6aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R4fEAW3IwnjCgPgFIiQUdI5m2WItkDGjr8AmxNsgZaI=; b=FdYznlxelBQVM0+wqiRIErmUhKyeuHrv/f0yn4NM0iRo9sNpE9paqe71hlpSI6xCGs Vf3uJ2pJkg2meNqnpcSgJHtS97GvQSC1SqPUcKCrHoZZ972cVUOHF0gj+R+5KErDhj3V IkSW8NFnn503wA3Vq8UD/ivKtNIY4dQhn49jE/bnKEyZIgN4COhqf6lrfvE7y9lLG2WU +XnGCdW0H3pktYDHakUhym5ZNT1GRpYuJwGmBlP+Gm5tvXkPpNNb9EIUTKgVSBWyzH8J qM9Pdvb4FPZnH98uJ4SKrJQjGlykKM0mUeK86w78dbZFTeDA3Ogpsk50Tt+bY/PjWdHI TWDw== X-Gm-Message-State: AOAM530QjvGeLTubTBZdyKxkWmdfsT5WXg3iOKrt6XfuvT14GVUwpwus Yb9oVeSVL7rQtCWI0CM8/w548d8N24YRBw== X-Google-Smtp-Source: ABdhPJwKKhI0KlomPet6ynDBihho4ZpZapSjOOFXa0QbXzboL48tDBHolKtJQp0y+o/1A1B3tK3ZZg== X-Received: by 2002:a05:600c:2194:b0:39c:419c:1a24 with SMTP id e20-20020a05600c219400b0039c419c1a24mr3900533wme.186.1655209851382; Tue, 14 Jun 2022 05:30:51 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 13/25] io_uring: remove check_cq checking from hot paths Date: Tue, 14 Jun 2022 13:29:51 +0100 Message-Id: <35d95aeb2c0bb749398f8eeb05f3006391469d0b.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org All ctx->check_cq events are slow path, don't test every single flag one by one in the hot path, but add a common guarding if. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index f3cae98471b8..bd2aab895e29 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1807,24 +1807,25 @@ static int io_iopoll_check(struct io_ring_ctx *ctx, long min) int ret = 0; unsigned long check_cq; + check_cq = READ_ONCE(ctx->check_cq); + if (unlikely(check_cq)) { + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + __io_cqring_overflow_flush(ctx, false); + /* + * Similarly do not spin if we have not informed the user of any + * dropped CQE. + */ + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) + return -EBADR; + } /* * Don't enter poll loop if we already have events pending. * If we do, we can potentially be spinning for commands that * already triggered a CQE (eg in error). */ - check_cq = READ_ONCE(ctx->check_cq); - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - __io_cqring_overflow_flush(ctx, false); if (io_cqring_events(ctx)) return 0; - /* - * Similarly do not spin if we have not informed the user of any - * dropped CQE. - */ - if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))) - return -EBADR; - do { /* * If a submit got punted to a workqueue, we can have the @@ -2752,12 +2753,15 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, ret = io_run_task_work_sig(); if (ret || io_should_wake(iowq)) return ret; + check_cq = READ_ONCE(ctx->check_cq); - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) - return 1; - if (unlikely(check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT))) - return -EBADR; + if (unlikely(check_cq)) { + /* let the caller flush overflows, retry */ + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + return 1; + if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) + return -EBADR; + } if (!schedule_hrtimeout(&timeout, HRTIMER_MODE_ABS)) return -ETIME; return 1; From patchwork Tue Jun 14 12:29:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94DEFCCA47A for ; Tue, 14 Jun 2022 12:34:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240593AbiFNMe1 (ORCPT ); Tue, 14 Jun 2022 08:34:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236258AbiFNMd5 (ORCPT ); Tue, 14 Jun 2022 08:33:57 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C77874B878 for ; Tue, 14 Jun 2022 05:30:54 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id w17so3574055wrg.7 for ; Tue, 14 Jun 2022 05:30:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QBEXlpK/N89FNLVTHLSAOv6cYdu1Mi+Y2oi1hv0ROqg=; b=nRRw0B7Cyd5L/fKnw9F0cKxZ/WoyexHat7DHrAI6N0/te61smXxwFCxSXKXSDwPpFF e3pf6PyynpdiGzFqMJxBDgbVo4eZ63rjZrfcmVc+mm3nK6GNtZcdks+6AIqHxEE+jT6H 2M/NzUrnB3mniNcscN0iSlOSpzfOfB0HMnNxfrmBH9uuRLsb5AH1N6Wxabh2oT3Kex1D LN4qo3sk7Y7oRaBX7FlEwmHO7B5epLxTHxCvk6acL44ohk8r6skLzxs1wEegkN8eI2/c dc7mYyfpWArfpiSQN19Bdm16DbCyNVA4v72OXFVtKgogKTkiEns6YfvN+dH4DbabZnSq y/6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QBEXlpK/N89FNLVTHLSAOv6cYdu1Mi+Y2oi1hv0ROqg=; b=mYOUqsrY50k33MYb77/9NN2ijAC2S5hK2sSJJjFtEtA8oAujqplBGZvtCipWedNg7h imfIxGtmDQPO0Jr1ZozCNJflgm959JT1FDcx4dtV+g8D0+BHiTBy9JQqUiYzZYHInsm+ NN9rySaNnpRNisf4bR9Lownonj7J/AiSd3+YgXNmiATZH1Q+/fubio6z3QztU3cK0WKt B3an1mf+RsDWamJMO98xZFY7NRLh0rbB8+G7+4f/JFUMYrImV3e0EDTACV8gpPUxvjr0 R9t2tSXa5XIZtMZcXBh6IZ2q8YIIqnXK5q3LgKarTkA94jqWiQRKnjoAYcIKIVqv11h8 U6ag== X-Gm-Message-State: AJIora+lypJJcleXAuNHP3y0ztbzlbK6FJxN696+b0ym+ZPzxgshIVWs 8IOpyjv7xI9ucaquxAQLOnkrfsmd/K432A== X-Google-Smtp-Source: AGRyM1uiayyA0tNM5tj8npnH1vzMUY/05KvdjUYVZzsJ+n4rGceMVASPR0bvXRe9eeUSSy3a7preZw== X-Received: by 2002:a5d:64c4:0:b0:219:b73f:48a7 with SMTP id f4-20020a5d64c4000000b00219b73f48a7mr4677687wri.180.1655209852752; Tue, 14 Jun 2022 05:30:52 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:52 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Hao Xu Subject: [PATCH for-next 14/25] io_uring: poll: remove unnecessary req->ref set Date: Tue, 14 Jun 2022 13:29:52 +0100 Message-Id: <2cb6a90af34475a9ae2c2d1aca55a9edb3d3975e.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu We now don't need to set req->refcount for poll requests since the reworked poll code ensures no request release race. Signed-off-by: Hao Xu Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index b46973140ffd..be3dd77fa21c 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -683,7 +683,6 @@ int io_poll_add_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if ((flags & IORING_POLL_ADD_MULTI) && (req->flags & REQ_F_CQE_SKIP)) return -EINVAL; - io_req_set_refcount(req); req->apoll_events = poll->events = io_poll_parse_events(sqe, flags); return 0; } From patchwork Tue Jun 14 12:29:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04831C43334 for ; Tue, 14 Jun 2022 12:34:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355092AbiFNMea (ORCPT ); Tue, 14 Jun 2022 08:34:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243599AbiFNMeA (ORCPT ); Tue, 14 Jun 2022 08:34:00 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C88B457A8 for ; Tue, 14 Jun 2022 05:30:56 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id w17so3574141wrg.7 for ; Tue, 14 Jun 2022 05:30:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Z06udJtnFtpH/wHY1WkD2gKtH1iQkgAVtqfz/r7EviA=; b=VZegUOGZPlTj9S3Ck1koofFAlZp5PE61v42Se9GFCDUYhepuBoyjBCFlXxNvh5X3Jl 51VwYVFkPj50+f0AuPyQN+7ZseoosQag063DJMw1AMhu5E5hQw/vJpc729Opt8PzwSip pD/LnagtskLt0h0AR6t7Xl2UxFDnuhueu4tCu0FeNiJvihktSfBC+5i7TKWLMEHybIfC hZ3QclRIQy7rrDbzKa8FbV0oVcjMRwYNB7iVUgfDvNBq3wVEad9Vse/khoOa6NvOyABv wGW/XX5ErWomjLmWdB0Wn1qFUnz8Bq6kuig9Mm48UkDLj2kUgtw14stIqNisi1lUV4S4 IY4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Z06udJtnFtpH/wHY1WkD2gKtH1iQkgAVtqfz/r7EviA=; b=DWhOx3MxI1UZB2UmF53KCfBWDyZrJ2rHTpn9e33Hk/scdtT5YEhgPItEBKEvAb6uRu M2W0gBtKCaiJWRJJStaN3a773ayVruGxWi2Y9Gk8Yuu3UcsCXROz+KXxpFutptrE1gkb NcwyJ70zYlU/QRP2LWqDkp3PV7AIFQM2G3HWJQzlIv8bO2mhmUsrFn2nXn4eGkkhAyyG 6O2ZdR9gWjyJSOvmuD1Iy9bQi7DulzYl3dPW2oFAq2Sc4t+VnbADOBdvbU9/Ym/JM69i a0IuIZuo/KStM7tC68RYT+JbUGkFnSRD6J4XwWKGBQZilxcj+MxWrmm4wXkqiEJwccjU pNdg== X-Gm-Message-State: AJIora/SlsvY+RJW6nxyeRCRw7/y2gh4cC8UzxhmxE0I2ERPD3TczdSR JejYk9aauxIyak468AZhEYPZxEJNMIql2w== X-Google-Smtp-Source: AGRyM1txsz7RAEkfaiKYtkISzEqgrVREjSB7LeBheZW2lUvu2L8fNnU9hnHRlaezqj3Ak7FtncaV/A== X-Received: by 2002:adf:f688:0:b0:215:6e4d:4103 with SMTP id v8-20020adff688000000b002156e4d4103mr4612028wrp.372.1655209854095; Tue, 14 Jun 2022 05:30:54 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:53 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Hao Xu Subject: [PATCH for-next 15/25] io_uring: switch cancel_hash to use per entry spinlock Date: Tue, 14 Jun 2022 13:29:53 +0100 Message-Id: <2fedbcc47d82f5c98ccf94b6bdcada14f9ebdd59.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org From: Hao Xu Add a new io_hash_bucket structure so that each bucket in cancel_hash has separate spinlock. Use per entry lock for cancel_hash, this removes some completion lock invocation and remove contension between different cancel_hash entries. Signed-off-by: Hao Xu Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 14 ++++++- io_uring/cancel.h | 6 +++ io_uring/fdinfo.c | 9 +++-- io_uring/io_uring.c | 8 ++-- io_uring/io_uring_types.h | 2 +- io_uring/poll.c | 80 ++++++++++++++++++++++++--------------- 6 files changed, 79 insertions(+), 40 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 2e72231882b7..54a5e0a86b61 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -93,14 +93,14 @@ int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) if (!ret) return 0; - spin_lock(&ctx->completion_lock); ret = io_poll_cancel(ctx, cd); if (ret != -ENOENT) goto out; + spin_lock(&ctx->completion_lock); if (!(cd->flags & IORING_ASYNC_CANCEL_FD)) ret = io_timeout_cancel(ctx, cd); -out: spin_unlock(&ctx->completion_lock); +out: return ret; } @@ -192,3 +192,13 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) io_req_set_res(req, ret, 0); return IOU_OK; } + +void init_hash_table(struct io_hash_bucket *hash_table, unsigned size) +{ + unsigned int i; + + for (i = 0; i < size; i++) { + spin_lock_init(&hash_table[i].lock); + INIT_HLIST_HEAD(&hash_table[i].list); + } +} diff --git a/io_uring/cancel.h b/io_uring/cancel.h index 4f35d8696325..556a7dcf160e 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -4,3 +4,9 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd); +void init_hash_table(struct io_hash_bucket *hash_table, unsigned size); + +struct io_hash_bucket { + spinlock_t lock; + struct hlist_head list; +} ____cacheline_aligned_in_smp; diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index fcedde4b4b1e..f941c73f5502 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -13,6 +13,7 @@ #include "io_uring.h" #include "sqpoll.h" #include "fdinfo.h" +#include "cancel.h" #ifdef CONFIG_PROC_FS static __cold int io_uring_show_cred(struct seq_file *m, unsigned int id, @@ -157,17 +158,19 @@ static __cold void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, mutex_unlock(&ctx->uring_lock); seq_puts(m, "PollList:\n"); - spin_lock(&ctx->completion_lock); for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list = &ctx->cancel_hash[i]; + struct io_hash_bucket *hb = &ctx->cancel_hash[i]; struct io_kiocb *req; - hlist_for_each_entry(req, list, hash_node) + spin_lock(&hb->lock); + hlist_for_each_entry(req, &hb->list, hash_node) seq_printf(m, " op=%d, task_works=%d\n", req->opcode, task_work_pending(req->task)); + spin_unlock(&hb->lock); } seq_puts(m, "CqOverflowList:\n"); + spin_lock(&ctx->completion_lock); list_for_each_entry(ocqe, &ctx->cq_overflow_list, list) { struct io_uring_cqe *cqe = &ocqe->cqe; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bd2aab895e29..4e0f60ffc975 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -717,11 +717,13 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) if (hash_bits <= 0) hash_bits = 1; ctx->cancel_hash_bits = hash_bits; - ctx->cancel_hash = kmalloc((1U << hash_bits) * sizeof(struct hlist_head), - GFP_KERNEL); + ctx->cancel_hash = + kmalloc((1U << hash_bits) * sizeof(struct io_hash_bucket), + GFP_KERNEL); if (!ctx->cancel_hash) goto err; - __hash_init(ctx->cancel_hash, 1U << hash_bits); + + init_hash_table(ctx->cancel_hash, 1U << hash_bits); ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 3cf06f4a4d2e..0dd3ab7eec4c 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -222,7 +222,7 @@ struct io_ring_ctx { * manipulate the list, hence no extra locking is needed there. */ struct io_wq_work_list iopoll_list; - struct hlist_head *cancel_hash; + struct io_hash_bucket *cancel_hash; unsigned cancel_hash_bits; bool poll_multi_queue; diff --git a/io_uring/poll.c b/io_uring/poll.c index be3dd77fa21c..ab4be4173cf3 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -19,6 +19,7 @@ #include "opdef.h" #include "kbuf.h" #include "poll.h" +#include "cancel.h" struct io_poll_update { struct file *file; @@ -73,10 +74,22 @@ static struct io_poll *io_poll_get_single(struct io_kiocb *req) static void io_poll_req_insert(struct io_kiocb *req) { struct io_ring_ctx *ctx = req->ctx; - struct hlist_head *list; + u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); + struct io_hash_bucket *hb = &ctx->cancel_hash[index]; - list = &ctx->cancel_hash[hash_long(req->cqe.user_data, ctx->cancel_hash_bits)]; - hlist_add_head(&req->hash_node, list); + spin_lock(&hb->lock); + hlist_add_head(&req->hash_node, &hb->list); + spin_unlock(&hb->lock); +} + +static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) +{ + u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); + spinlock_t *lock = &ctx->cancel_hash[index].lock; + + spin_lock(lock); + hash_del(&req->hash_node); + spin_unlock(lock); } static void io_init_poll_iocb(struct io_poll *poll, __poll_t events, @@ -220,8 +233,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) } io_poll_remove_entries(req); + io_poll_req_delete(req, ctx); spin_lock(&ctx->completion_lock); - hash_del(&req->hash_node); req->cqe.flags = 0; __io_req_complete_post(req); io_commit_cqring(ctx); @@ -231,7 +244,6 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) static void io_apoll_task_func(struct io_kiocb *req, bool *locked) { - struct io_ring_ctx *ctx = req->ctx; int ret; ret = io_poll_check_events(req, locked); @@ -239,9 +251,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) return; io_poll_remove_entries(req); - spin_lock(&ctx->completion_lock); - hash_del(&req->hash_node); - spin_unlock(&ctx->completion_lock); + io_poll_req_delete(req, req->ctx); if (!ret) io_req_task_submit(req, locked); @@ -435,9 +445,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 0; } - spin_lock(&ctx->completion_lock); io_poll_req_insert(req); - spin_unlock(&ctx->completion_lock); if (mask && (poll->events & EPOLLET)) { /* can't multishot if failed, just queue the event we've got */ @@ -534,32 +542,31 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, bool found = false; int i; - spin_lock(&ctx->completion_lock); for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list; + struct io_hash_bucket *hb = &ctx->cancel_hash[i]; - list = &ctx->cancel_hash[i]; - hlist_for_each_entry_safe(req, tmp, list, hash_node) { + spin_lock(&hb->lock); + hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) { if (io_match_task_safe(req, tsk, cancel_all)) { hlist_del_init(&req->hash_node); io_poll_cancel_req(req); found = true; } } + spin_unlock(&hb->lock); } - spin_unlock(&ctx->completion_lock); return found; } static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd) - __must_hold(&ctx->completion_lock) { - struct hlist_head *list; struct io_kiocb *req; + u32 index = hash_long(cd->data, ctx->cancel_hash_bits); + struct io_hash_bucket *hb = &ctx->cancel_hash[index]; - list = &ctx->cancel_hash[hash_long(cd->data, ctx->cancel_hash_bits)]; - hlist_for_each_entry(req, list, hash_node) { + spin_lock(&hb->lock); + hlist_for_each_entry(req, &hb->list, hash_node) { if (cd->data != req->cqe.user_data) continue; if (poll_only && req->opcode != IORING_OP_POLL_ADD) @@ -571,21 +578,21 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, } return req; } + spin_unlock(&hb->lock); return NULL; } static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd) - __must_hold(&ctx->completion_lock) { struct io_kiocb *req; int i; for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct hlist_head *list; + struct io_hash_bucket *hb = &ctx->cancel_hash[i]; - list = &ctx->cancel_hash[i]; - hlist_for_each_entry(req, list, hash_node) { + spin_lock(&hb->lock); + hlist_for_each_entry(req, &hb->list, hash_node) { if (!(cd->flags & IORING_ASYNC_CANCEL_ANY) && req->file != cd->file) continue; @@ -594,12 +601,12 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, req->cancel_seq = cd->seq; return req; } + spin_unlock(&hb->lock); } return NULL; } static bool io_poll_disarm(struct io_kiocb *req) - __must_hold(&ctx->completion_lock) { if (!io_poll_get_ownership(req)) return false; @@ -609,17 +616,23 @@ static bool io_poll_disarm(struct io_kiocb *req) } int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) - __must_hold(&ctx->completion_lock) { struct io_kiocb *req; + u32 index; + spinlock_t *lock; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) req = io_poll_file_find(ctx, cd); else req = io_poll_find(ctx, false, cd); - if (!req) + if (!req) { return -ENOENT; + } else { + index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); + lock = &ctx->cancel_hash[index].lock; + } io_poll_cancel_req(req); + spin_unlock(lock); return 0; } @@ -713,18 +726,23 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) struct io_poll_update *poll_update = io_kiocb_to_cmd(req); struct io_cancel_data cd = { .data = poll_update->old_user_data, }; struct io_ring_ctx *ctx = req->ctx; + u32 index = hash_long(cd.data, ctx->cancel_hash_bits); + spinlock_t *lock = &ctx->cancel_hash[index].lock; struct io_kiocb *preq; int ret2, ret = 0; bool locked; - spin_lock(&ctx->completion_lock); preq = io_poll_find(ctx, true, &cd); - if (!preq || !io_poll_disarm(preq)) { - spin_unlock(&ctx->completion_lock); - ret = preq ? -EALREADY : -ENOENT; + if (!preq) { + ret = -ENOENT; + goto out; + } + ret2 = io_poll_disarm(preq); + spin_unlock(lock); + if (!ret2) { + ret = -EALREADY; goto out; } - spin_unlock(&ctx->completion_lock); if (poll_update->update_events || poll_update->update_user_data) { /* only mask one event flags, keep behavior flags */ From patchwork Tue Jun 14 12:29:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 758C7C433EF for ; Tue, 14 Jun 2022 12:34:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356350AbiFNMec (ORCPT ); Tue, 14 Jun 2022 08:34:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33652 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243825AbiFNMeA (ORCPT ); Tue, 14 Jun 2022 08:34:00 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D10174BB86 for ; Tue, 14 Jun 2022 05:30:56 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id m16-20020a7bca50000000b0039c8a224c95so3819987wml.2 for ; Tue, 14 Jun 2022 05:30:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aiDnDEFv3waDefowu5Uws7FQ5cxLr1/URoFOkJXGFhw=; b=WKuxUytv5fxlW+72ObSGz/d8M0pmIyboKKh+OhB7lsZVy6b3i0OtEnXFN5cfVet9GJ XpPMCikqhhY3kkujvHvHQrSvyXbl18m39IA/meDkowKH9h+vbU94Zqg4dAIrFq8GUD5M THV2jKXKxNCYZLsRwtc9GYOQ+bs4qi3OcNVjj1UO3cYJ2rHAqz/JulrAZTKPhFYctj+t iqVXPgG4SNZeDAdlkmpwFtvFbHYwuehHJAGOnG/21Drp6VP3GjipKwhgV9nmqPf+j2cI TJ9ae34L/BV0t8j//SlOLafyn92JIfE0OdheRbsU39xdq7Z8YXjZr93KBvV1JbCyToK5 zfPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aiDnDEFv3waDefowu5Uws7FQ5cxLr1/URoFOkJXGFhw=; b=T8bKmHY7e25+9gxzJN6xwPixe6SkhMqUz/JWmvJ2kGLdW56XscKDPqXNTxqiwbL49q xFwP9pnSAwhs61sEn+eUB2Jsm/D5mIljGc6JxeGIi6GPRhK2F5CpEEkZ2j2Dp46PzaE0 iLCLCMOG7kvTVKhyRY0mqymc56fE/2Z4wSPEZfutITtW0FnGQ0wfnxVvML7Ct/kJnppw q+wfSYqkLwSKz1LkoDdQ14VPLt2RB9jj3RF0YuZUBADh06YUX0FC40YSzN5D0CechbB6 t04yRSrnI0x2nwgC4cqFf2Xn+Srf+e4Ug0elKoRLNcpSKLEANB4yYRNJ3LuIO2ZD6KA/ q1Pg== X-Gm-Message-State: AOAM5323yaDP6JPruz/DpL6ildJsnpzXCDbQ1DGfdtxjMk1NuXkcCb+s RUdaq7Tvejw1GUHQXTt9+3AhvPRmpybnQQ== X-Google-Smtp-Source: ABdhPJzOUXePin1Syi7e4bfS/NBdykbFU4aCee8W3n50Bo08Ad1/HRXFqlZ1mg+5FLNWgzwbaLsH/w== X-Received: by 2002:a1c:3b56:0:b0:39c:472a:4aff with SMTP id i83-20020a1c3b56000000b0039c472a4affmr3938273wma.23.1655209855089; Tue, 14 Jun 2022 05:30:55 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:54 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 16/25] io_uring: pass poll_find lock back Date: Tue, 14 Jun 2022 13:29:54 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of using implicit knowledge of what is locked or not after io_poll_find() and co returns, pass back a pointer to the locked bucket if any. If set the user must to unlock the spinlock. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 46 ++++++++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index ab4be4173cf3..ad25a964d4ed 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -559,12 +559,15 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, } static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, - struct io_cancel_data *cd) + struct io_cancel_data *cd, + struct io_hash_bucket **out_bucket) { struct io_kiocb *req; u32 index = hash_long(cd->data, ctx->cancel_hash_bits); struct io_hash_bucket *hb = &ctx->cancel_hash[index]; + *out_bucket = NULL; + spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { if (cd->data != req->cqe.user_data) @@ -576,6 +579,7 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, continue; req->cancel_seq = cd->seq; } + *out_bucket = hb; return req; } spin_unlock(&hb->lock); @@ -583,11 +587,14 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, } static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, - struct io_cancel_data *cd) + struct io_cancel_data *cd, + struct io_hash_bucket **out_bucket) { struct io_kiocb *req; int i; + *out_bucket = NULL; + for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { struct io_hash_bucket *hb = &ctx->cancel_hash[i]; @@ -599,6 +606,7 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, if (cd->seq == req->cancel_seq) continue; req->cancel_seq = cd->seq; + *out_bucket = hb; return req; } spin_unlock(&hb->lock); @@ -617,23 +625,19 @@ static bool io_poll_disarm(struct io_kiocb *req) int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) { + struct io_hash_bucket *bucket; struct io_kiocb *req; - u32 index; - spinlock_t *lock; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd); + req = io_poll_file_find(ctx, cd, &bucket); else - req = io_poll_find(ctx, false, cd); - if (!req) { - return -ENOENT; - } else { - index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); - lock = &ctx->cancel_hash[index].lock; - } - io_poll_cancel_req(req); - spin_unlock(lock); - return 0; + req = io_poll_find(ctx, false, cd, &bucket); + + if (req) + io_poll_cancel_req(req); + if (bucket) + spin_unlock(&bucket->lock); + return req ? 0 : -ENOENT; } static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, @@ -726,19 +730,21 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) struct io_poll_update *poll_update = io_kiocb_to_cmd(req); struct io_cancel_data cd = { .data = poll_update->old_user_data, }; struct io_ring_ctx *ctx = req->ctx; - u32 index = hash_long(cd.data, ctx->cancel_hash_bits); - spinlock_t *lock = &ctx->cancel_hash[index].lock; + struct io_hash_bucket *bucket; struct io_kiocb *preq; int ret2, ret = 0; bool locked; - preq = io_poll_find(ctx, true, &cd); + preq = io_poll_find(ctx, true, &cd, &bucket); + if (preq) + ret2 = io_poll_disarm(preq); + if (bucket) + spin_unlock(&bucket->lock); + if (!preq) { ret = -ENOENT; goto out; } - ret2 = io_poll_disarm(preq); - spin_unlock(lock); if (!ret2) { ret = -EALREADY; goto out; From patchwork Tue Jun 14 12:29:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AA73C43334 for ; Tue, 14 Jun 2022 12:34:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232988AbiFNMed (ORCPT ); Tue, 14 Jun 2022 08:34:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244265AbiFNMeL (ORCPT ); Tue, 14 Jun 2022 08:34:11 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1752E4BB8F for ; Tue, 14 Jun 2022 05:30:58 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id a10so4563015wmj.5 for ; Tue, 14 Jun 2022 05:30:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=d4/MXjfOfkv5/D7WY7pe9UNNIj5LXSJkYJ36ARTYKgw=; b=dA4JL5MQqYQDxQ8LNUlsxRKdmWybGFh5ioirlz+CEBN+88glF4wWxUxFKA8DXOCO10 uEJT/o+W26lKlDD2KWoN2yxqbYWjIEtSG0/zyvEZt+xnIAe07PW85sLFEkOOtLPLe3F9 4xPGWKR+Z210XpBWoCA5tl2025iiUKOAeQZlaND85GGCmOdGJbudW26lktKhzmeVwxy+ tJ4uGJOqs/BtXwdGxeJuDv1jH4UjCubm8q4zZySFzrqWL5Sb/X3ty11Szc90J51ztILq KeY05SGeYhmlO1zGmdVz3FTsZ4iwp8vXvHhIlpbFfNWgX7/bpc6Go624qCY5B5LKVvIp vPrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=d4/MXjfOfkv5/D7WY7pe9UNNIj5LXSJkYJ36ARTYKgw=; b=lMi9cxE5tSmuLoJV7mIMv7dbdTBJyQMmhJa3dZfhOS5cQzsav/b2iuq27c/YPHxSDd UZLQQkhZbkMJOTBuGRc1v6cB74W2yIPs1V3vNyG36ZnqKLsBZ2cERzeh11YJTXHpnk2m ZeHpK7TTSEVuhCMyDsj2i+LBuYxN1Nqw9t62PHrlG1vrMYUL+beCRjMu21dMF/I+1gNL vsVAbsZpOCAqmyOOPAzY1M1lnBwZsCq6swSDsY4A7N+aA5tkXBh6mpsbx7GfrUwI4WJx DdQiXzp5W25Ba9l+r05oTqwGBzXoKzZ8NPztw7a2sl6QIWO9MsL3ZikY/jT7lHSAFBjt QvcA== X-Gm-Message-State: AOAM532Tfn7ZFzPYi2yKrYSsf97AVXriqmWE4kYpDXQegGiMmxSx8Y1B m/dc0UhdeHTq56QzKJaFE3pl5IGFPw7jMQ== X-Google-Smtp-Source: ABdhPJwW+H/vzG/Kgi+vOG0+YQRvQeRpXKNvRnm61LMIExt/4Oc/VAHDNYFRcsGUDORY1fmuyCFhTA== X-Received: by 2002:a05:600c:5021:b0:39c:6571:e0b0 with SMTP id n33-20020a05600c502100b0039c6571e0b0mr3887439wmr.177.1655209856403; Tue, 14 Jun 2022 05:30:56 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:55 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 17/25] io_uring: clean up io_try_cancel Date: Tue, 14 Jun 2022 13:29:55 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Get rid of an unnecessary extra goto in io_try_cancel() and simplify the function. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 54a5e0a86b61..2f157aebd3a0 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -95,12 +95,12 @@ int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) ret = io_poll_cancel(ctx, cd); if (ret != -ENOENT) - goto out; + return ret; + spin_lock(&ctx->completion_lock); if (!(cd->flags & IORING_ASYNC_CANCEL_FD)) ret = io_timeout_cancel(ctx, cd); spin_unlock(&ctx->completion_lock); -out: return ret; } From patchwork Tue Jun 14 12:29:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51F9DC43334 for ; Tue, 14 Jun 2022 12:34:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233873AbiFNMeg (ORCPT ); Tue, 14 Jun 2022 08:34:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33048 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244298AbiFNMeM (ORCPT ); Tue, 14 Jun 2022 08:34:12 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0C3E4BB97 for ; Tue, 14 Jun 2022 05:30:59 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id x17so11078046wrg.6 for ; Tue, 14 Jun 2022 05:30:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n/nGsucu9f3Io3BE8Nz2zIpgpyCeUl8M+4ygHEUZXIc=; b=l9higdNwgRRofLokarXDlvp/3N5y/qtPuH06WpDzWUDGy+sJkl9HhbthUTpUUSkaMI /B9LPjGDphXBIdOdQNehk20Z8MATYCdfKPRTf4/+b1l22mMpFSh6zW3C9vR2UCtLfJxx qF63+fa0yFo3M7sgspwWqqqoLoe/Kb+9Wk4JYzcgPa7yZOYyYZVcSY9CJ2KyIeYMxm5h dwvR8SNNGTN+pa2I92vu6QzDKcIPJgWXPkdZxx8exmsGYZZCSA1zUIBuFQeeL/e6JtQG d4ePRFsPMEIgW/9p9h9CWUYMoZxiHaOHc7lucJvRguNM7DAP78VEWOhIRBpCG0DUEkRh I87w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n/nGsucu9f3Io3BE8Nz2zIpgpyCeUl8M+4ygHEUZXIc=; b=t0UgF3Y4vP+eRybYJwkZv+HTyKAh+sCKdskeLizDa+sJB6RlFWQPNl7uNcr2YR4b6v LSIv0z0w1osAw6ksgS+j8u1Zmc+4lhoP2dy4StI7ofaseqayjjmV1QByx+/DY/Wmx1Km NocdcepbmFX3e5WE+Ib1izaWXcc8QdnkqKUyHPa5NqwObEHfhAApB77mKDLrZSWfdRmq LclWRehgpa4T0Mt5S/7VMhxu3wR+b0ND21nGtBsiyl+NUSGaenFmO7civJZxeZsSvKud 0ac36M8FRZGTb2EhqwauTLtuMHknyTOx2CMTDG0l8ToA1TJL+SsgrpaBpDLuaJI02YPh 9KAg== X-Gm-Message-State: AJIora/OazvWGdoPblL1Nqxmyljq+3LeqjQA+yiIRITFhIAIM6R+t/p7 InjV2WNh71K60nV6gRc844+dM9ZN43izFA== X-Google-Smtp-Source: AGRyM1u9ErXwsBlL91PbQaWhZuTkz5uF26xrdrpO+ABRRQK1RICnDv6MSD8Bq79jdLPX+86KOwWnAg== X-Received: by 2002:a05:6000:156d:b0:210:3125:6012 with SMTP id 13-20020a056000156d00b0021031256012mr4526682wrz.357.1655209857888; Tue, 14 Jun 2022 05:30:57 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:57 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 18/25] io_uring: limit number hash buckets Date: Tue, 14 Jun 2022 13:29:56 +0100 Message-Id: <94334f242fa236c77e1f2c32a9b0108931e285a2.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't allocate to many hash/cancellation buckets, there might be too many, clamp it to 8 bits, or 256 * 64B = 16KB. We don't usually have too many requests, and 256 buckets should be enough, especially since we do hash search only in the cancellation path. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 4e0f60ffc975..a203943f3d71 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -710,12 +710,12 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) /* * Use 5 bits less than the max cq entries, that should give us around - * 32 entries per hash list if totally full and uniformly spread. + * 32 entries per hash list if totally full and uniformly spread, but + * don't keep too many buckets to not overconsume memory. */ - hash_bits = ilog2(p->cq_entries); - hash_bits -= 5; - if (hash_bits <= 0) - hash_bits = 1; + hash_bits = ilog2(p->cq_entries) - 5; + hash_bits = clamp(hash_bits, 1, 8); + ctx->cancel_hash_bits = hash_bits; ctx->cancel_hash = kmalloc((1U << hash_bits) * sizeof(struct io_hash_bucket), From patchwork Tue Jun 14 12:29:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E2BACCA47C for ; Tue, 14 Jun 2022 12:34:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238220AbiFNMef (ORCPT ); Tue, 14 Jun 2022 08:34:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239463AbiFNMeW (ORCPT ); Tue, 14 Jun 2022 08:34:22 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 143644BBA1 for ; Tue, 14 Jun 2022 05:31:00 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id m32-20020a05600c3b2000b0039756bb41f2so4707088wms.3 for ; Tue, 14 Jun 2022 05:31:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VdgQMW5jxhkuYvwndjirbk6yyzG8N1G9hnlDD6URUMA=; b=fVCBvm47gXMMrJzj71SNlSolJvuiLlBHc3cSeoYssM+tq1s5T9ynml3PmMt+8H3Y9T V4+5Cj0Wy5YYUtKACEI0Ss+HNBejPKamoE2YEoSmhMf+9AJQCuNQJFW5Zp7M1WS+go7+ /aVnu6lk0A9oPxH+z/pg9sPcnrHWhq1q7owPvUbHbbLR0BLRw8K/5PyltjnKKWkelULo BuFTxuws1pxGKVAJuUW17SztRmlqIt/6Im5oDdRlvgPrKg1zGaG/YP7VSMgTy3KGxW34 ceJsCIiWeOjfc90AecLAzWCPxeqUbEAzSXkgF82ejR1pLLk5bdevfe+QiqYuMyhxQOHl qcIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VdgQMW5jxhkuYvwndjirbk6yyzG8N1G9hnlDD6URUMA=; b=DDqyEjskTI4JNMuBb6obJZoMVj8MI0bHyTLYKpviLrdMcn9j3y0oNDbBioAgfRLKm5 /9olw5Jh0uxEex/7xYgj6QcjsrPO78z4+gr+rnQkNruXN8iyRjYBJK+oGHsT4tjOErVP HOHBQ1KoQk6YKbSAV/d2M6WaBkW/2kpJBIR8LEFjTeKFzq7vYPfzNUEuRbCuj7qBc082 is2zKwG51iEQuO5bTyaUBjLV3YID4KpkflKIA6s8fVpa0l0vBk9pCTikuHC+7Nj7gE59 D8QQEo4xriooGzCI1SS2MLkvHX3Vse83JjLGut7WB+gvnIMjX8lD6LDgiZsYn5Uu+UG3 6Idg== X-Gm-Message-State: AOAM530nXe0vAVg65WS6bPxz8Fl7uJAxCkRAqu+oOWkSNLij4eEIxODV 0db2j/nooOPjyHWHs9vdXa7lDdwN7/JyYw== X-Google-Smtp-Source: ABdhPJyFT+s6sXcTlxPk/OwbTJ/f65jkAL9P1mU3kcqSLcSXHPwgeBvF7fRBfZb5WR6/56xjjRRiPw== X-Received: by 2002:a7b:c4d8:0:b0:39c:97ed:baa3 with SMTP id g24-20020a7bc4d8000000b0039c97edbaa3mr3912809wmk.58.1655209859274; Tue, 14 Jun 2022 05:30:59 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:58 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 19/25] io_uring: clean up io_ring_ctx_alloc Date: Tue, 14 Jun 2022 13:29:57 +0100 Message-Id: <1dfcb8d34f9e51d6da65cb6aac8e943dfcef86e3.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a variable for the number of hash buckets in io_ring_ctx_alloc(), makes it more readable. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a203943f3d71..89696efcead4 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -700,6 +700,8 @@ static __cold void io_fallback_req_func(struct work_struct *work) static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) { struct io_ring_ctx *ctx; + unsigned hash_buckets; + size_t hash_size; int hash_bits; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); @@ -715,15 +717,15 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) */ hash_bits = ilog2(p->cq_entries) - 5; hash_bits = clamp(hash_bits, 1, 8); + hash_buckets = 1U << hash_bits; + hash_size = hash_buckets * sizeof(struct io_hash_bucket); ctx->cancel_hash_bits = hash_bits; - ctx->cancel_hash = - kmalloc((1U << hash_bits) * sizeof(struct io_hash_bucket), - GFP_KERNEL); + ctx->cancel_hash = kmalloc(hash_size, GFP_KERNEL); if (!ctx->cancel_hash) goto err; - init_hash_table(ctx->cancel_hash, 1U << hash_bits); + init_hash_table(ctx->cancel_hash, hash_buckets); ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) From patchwork Tue Jun 14 12:29:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE0DFC433EF for ; Tue, 14 Jun 2022 12:34:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242808AbiFNMeg (ORCPT ); Tue, 14 Jun 2022 08:34:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1355284AbiFNMeX (ORCPT ); Tue, 14 Jun 2022 08:34:23 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39BF94BBB1 for ; Tue, 14 Jun 2022 05:31:02 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id m32-20020a05600c3b2000b0039756bb41f2so4707111wms.3 for ; Tue, 14 Jun 2022 05:31:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zrn7uBYKLng0XHi7RMiiU3T4m2AfPpnQM1wA3OWR3wg=; b=RQ9uiRkk3D2d/Y/TWEyhipyB3ehxQ2F7AQAlyRPIs8S+p1XK9FGVsQW+GyVeBbDNoE 0XThRMrJNj6SvYNBR7YlbujC7oaCGo0q30X0Db+3eR1sTS8qkMLMYneGW+j/dAVbUI9S Sgoyo2RPVMHCStYXmxM4r8wDLfX0jrC9C6MGsOYTyoC11klP3LzrPOjlvWnBSrJG4+Aw 4f6e8dBgMogvMeFp+hveCdiaAKZI+Q2BQBjDFWW3J41P4pAzESQiCEKhioNddHAsz5Xe HgxcI1/2/dZqAVLQBCXZu2euhrmKU3eE3wRr3+wgAawZJRYYVs5E46GPSWmBBjJ0+g/R 6SIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zrn7uBYKLng0XHi7RMiiU3T4m2AfPpnQM1wA3OWR3wg=; b=gyGzWrSfH978ZN3ewaxVkZ0j56XeILoU/TaeUXeFkcKt4GlWF/bgp8L1eQEKLasXyF mkfHtomVoJLQ14qsii+O+uQ0E/T4q2Yw8r7yUxa7sxhWZSNBiJ5+1oMPqW7GaACof06Z dptlSNLCggxDa8EM6yK/Sw8v0+jUgozazLg227BTLLk9jRXR5uCqo0BPMtJ+zhdzRTSZ owCHinRSXXfqY1KruWO6pLYCpH7HGYYv5LEmDs4b5+abbOFLBGynf+G7UoOqboQAt6I7 p0lxJ2R3AdSmeeoQ91C7Vi6hxCkDCdt9VMsUqLOOkB5gMUkbzOjmGriGuihgig/02Hnm hL0g== X-Gm-Message-State: AOAM533hXXHmLb86+P9USGbPKdslbMze2aIuf4EdVn03Yo63N0QKiM3P vYCuI2bybc9Sd5WpNCB902koUrJvPB+t0A== X-Google-Smtp-Source: ABdhPJxF0R7zhHAezwx4KhdekrUB1Wo/7YUbjq7q12zEC42mFrR2mQvNAIGGAxCjALf/TiQBXzOPNw== X-Received: by 2002:a05:600c:3d18:b0:39c:474c:eb with SMTP id bh24-20020a05600c3d1800b0039c474c00ebmr3891085wmb.87.1655209860406; Tue, 14 Jun 2022 05:31:00 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.30.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:30:59 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 20/25] io_uring: use state completion infra for poll reqs Date: Tue, 14 Jun 2022 13:29:58 +0100 Message-Id: <8ef49e3c8619395fe2d5d7e00e69f0af8d5cc9e2.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Use io_req_task_complete() for poll request completions, so it can utilise state completions and save lots of unnecessary locking. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index ad25a964d4ed..2688201e872a 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -234,12 +234,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) io_poll_remove_entries(req); io_poll_req_delete(req, ctx); - spin_lock(&ctx->completion_lock); - req->cqe.flags = 0; - __io_req_complete_post(req); - io_commit_cqring(ctx); - spin_unlock(&ctx->completion_lock); - io_cqring_ev_posted(ctx); + io_req_set_res(req, req->cqe.res, 0); + io_req_task_complete(req, locked); } static void io_apoll_task_func(struct io_kiocb *req, bool *locked) From patchwork Tue Jun 14 12:29:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38E0BC43334 for ; Tue, 14 Jun 2022 12:34:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1352675AbiFNMeh (ORCPT ); Tue, 14 Jun 2022 08:34:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243885AbiFNMeZ (ORCPT ); Tue, 14 Jun 2022 08:34:25 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68CF9457AB for ; Tue, 14 Jun 2022 05:31:03 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id r186-20020a1c44c3000000b0039c76434147so798890wma.1 for ; Tue, 14 Jun 2022 05:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5RsHa8GytWmpiECLpptucUddBCUqGE5f7s10JlyPkE4=; b=C/yZqIa4hThHtfLbZV53WgbndWgvbd+T+/HU+wsOJQW+A1vGdk4deckwtgxHO+FeL4 8Ji6nJIBX0hIy0kFC/swpdUmCsXebjlxCzQJoHCAJOci0L2t7WEIKFZ1kJY8EScwJZt6 YS9fTb8hvR5jyN9LON0fUJ3olr5jzun0mX82ObtoUIWIDG/A54Ssci3e6wSt52NKz2rj k2PKDp+0+g/6fOBl+dg04Xhq904lFbZRaviJRfoYa4hGjR/X2tHVITZAW5yqpYR51x5P 0jb7tR1AL9mpBSKxDDg620mCpLp2AbremMJT+VXXgJa9VIwbcKfMJLZkWbW5TGKR0ALA ZOcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5RsHa8GytWmpiECLpptucUddBCUqGE5f7s10JlyPkE4=; b=sYfzR9WLI0x3gUD92AA7e6lSziYjRcW2kkYxJtUlxLMX7ZhK8sVZ3knjdqk6Kkc5M/ XHrIiTsDrzBITqwKkAN3bdCLRlwvDBoRBDiLeLqYBldBRdFz4Hmd3KLRtjKzRE3yuHH8 NDtIcAMKspJ4C5YH6AKSmF69LZZFvKJpFVkvShA9Ibuh0yNPBEM1HP9NaghHcPpIu519 XC7PhRrn/1LuzQCujmVh5JCU8znNDGfJ8044J4fRW8rPx3Aug9Z5RkH4LKIhhwp+WRkf uwE2n9gWq1UlTaesoU2oq5fXpEYLvqXHq/MXwBRD1VjM2Uu0Ijr/52QoYfjhE7R8i95t vK8Q== X-Gm-Message-State: AOAM530BdW8nxFigOZ+TKhPqoX81kQrZGwKwz8DXZaPTlr4nDRVdcSzv Mhy65pIUs2BIO4JT6cOzCX2WOC7iXZjIYw== X-Google-Smtp-Source: ABdhPJwaODPwqem1N84ZiFkd8tmjQ2YY+jK2vu1gf+Em4ZRddpaNbOfLrCEceaI2CLDi8Z0ZZ5vCow== X-Received: by 2002:a05:600c:501f:b0:39d:a3d:e919 with SMTP id n31-20020a05600c501f00b0039d0a3de919mr3934482wmr.132.1655209861508; Tue, 14 Jun 2022 05:31:01 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.31.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:31:00 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 21/25] io_uring: add IORING_SETUP_SINGLE_ISSUER Date: Tue, 14 Jun 2022 13:29:59 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a new IORING_SETUP_SINGLE_ISSUER flag and the userspace visible part of it, i.e. put limitations of submitters. Also, don't allow it together with IOPOLL as we're not going to put it to good use. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 5 ++++- io_uring/io_uring.c | 9 ++++++--- io_uring/io_uring_types.h | 1 + io_uring/tctx.c | 25 +++++++++++++++++++++++-- io_uring/tctx.h | 4 ++-- 5 files changed, 36 insertions(+), 8 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index a41ddb8c5e1f..a3a691340d3e 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -138,9 +138,12 @@ enum { * IORING_SQ_TASKRUN in the sq ring flags. Not valid with COOP_TASKRUN. */ #define IORING_SETUP_TASKRUN_FLAG (1U << 9) - #define IORING_SETUP_SQE128 (1U << 10) /* SQEs are 128 byte */ #define IORING_SETUP_CQE32 (1U << 11) /* CQEs are 32 byte */ +/* + * Only one task is allowed to submit requests + */ +#define IORING_SETUP_SINGLE_ISSUER (1U << 12) enum io_uring_op { IORING_OP_NOP, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 89696efcead4..af0bd74e78fa 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3020,6 +3020,8 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_destroy_buffers(ctx); if (ctx->sq_creds) put_cred(ctx->sq_creds); + if (ctx->submitter_task) + put_task_struct(ctx->submitter_task); /* there are no registered resources left, nobody uses it */ if (ctx->rsrc_node) @@ -3615,7 +3617,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, } ret = to_submit; } else if (to_submit) { - ret = io_uring_add_tctx_node(ctx); + ret = __io_uring_add_tctx_node(ctx, false); if (unlikely(ret)) goto out; @@ -3752,7 +3754,7 @@ static int io_uring_install_fd(struct io_ring_ctx *ctx, struct file *file) if (fd < 0) return fd; - ret = io_uring_add_tctx_node(ctx); + ret = __io_uring_add_tctx_node(ctx, false); if (ret) { put_unused_fd(fd); return ret; @@ -3972,7 +3974,8 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) IORING_SETUP_CLAMP | IORING_SETUP_ATTACH_WQ | IORING_SETUP_R_DISABLED | IORING_SETUP_SUBMIT_ALL | IORING_SETUP_COOP_TASKRUN | IORING_SETUP_TASKRUN_FLAG | - IORING_SETUP_SQE128 | IORING_SETUP_CQE32)) + IORING_SETUP_SQE128 | IORING_SETUP_CQE32 | + IORING_SETUP_SINGLE_ISSUER)) return -EINVAL; return io_uring_create(entries, &p, params); diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index 0dd3ab7eec4c..faa1477bd754 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -241,6 +241,7 @@ struct io_ring_ctx { /* Keep this last, we don't need it for the fast path */ struct io_restriction restrictions; + struct task_struct *submitter_task; /* slow path rsrc auxilary data, used by update/register */ struct io_rsrc_node *rsrc_backup_node; diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 6adf659687f8..e3138a36cf39 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -81,12 +81,32 @@ __cold int io_uring_alloc_task_context(struct task_struct *task, return 0; } -int __io_uring_add_tctx_node(struct io_ring_ctx *ctx) +static int io_register_submitter(struct io_ring_ctx *ctx) +{ + int ret = 0; + + mutex_lock(&ctx->uring_lock); + if (!ctx->submitter_task) + ctx->submitter_task = get_task_struct(current); + else if (ctx->submitter_task != current) + ret = -EEXIST; + mutex_unlock(&ctx->uring_lock); + + return ret; +} + +int __io_uring_add_tctx_node(struct io_ring_ctx *ctx, bool submitter) { struct io_uring_task *tctx = current->io_uring; struct io_tctx_node *node; int ret; + if ((ctx->flags & IORING_SETUP_SINGLE_ISSUER) && submitter) { + ret = io_register_submitter(ctx); + if (ret) + return ret; + } + if (unlikely(!tctx)) { ret = io_uring_alloc_task_context(current, ctx); if (unlikely(ret)) @@ -120,7 +140,8 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx) list_add(&node->ctx_node, &ctx->tctx_list); mutex_unlock(&ctx->uring_lock); } - tctx->last = ctx; + if (submitter) + tctx->last = ctx; return 0; } diff --git a/io_uring/tctx.h b/io_uring/tctx.h index 7684713e950f..dde82ce4d8e2 100644 --- a/io_uring/tctx.h +++ b/io_uring/tctx.h @@ -34,7 +34,7 @@ struct io_tctx_node { int io_uring_alloc_task_context(struct task_struct *task, struct io_ring_ctx *ctx); void io_uring_del_tctx_node(unsigned long index); -int __io_uring_add_tctx_node(struct io_ring_ctx *ctx); +int __io_uring_add_tctx_node(struct io_ring_ctx *ctx, bool submitter); void io_uring_clean_tctx(struct io_uring_task *tctx); void io_uring_unreg_ringfd(void); @@ -52,5 +52,5 @@ static inline int io_uring_add_tctx_node(struct io_ring_ctx *ctx) if (likely(tctx && tctx->last == ctx)) return 0; - return __io_uring_add_tctx_node(ctx); + return __io_uring_add_tctx_node(ctx, true); } From patchwork Tue Jun 14 12:30:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0342DCCA47C for ; Tue, 14 Jun 2022 12:34:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243679AbiFNMeh (ORCPT ); Tue, 14 Jun 2022 08:34:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243562AbiFNMeZ (ORCPT ); Tue, 14 Jun 2022 08:34:25 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68D504B86F for ; Tue, 14 Jun 2022 05:31:03 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id m125-20020a1ca383000000b0039c63fe5f64so4731124wme.0 for ; Tue, 14 Jun 2022 05:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZkF0//vS8P5S/motkhyB8BSxXm8jA0xCFMYcO1lVP80=; b=FLSAmJFnXDo9VqrHKzAXTLRo+bBnC+/da/43PqD/IQ7wxuUdCHBR//NF3v++iYzUFH AxJt7ZPUuA7IWD5buKn1uLZj3eRg9KAWuuIo5C/+OrzMBdqHqXbjwahiko9DJ83/nOP2 SkRNrb1sxfekN9lDUdRuBCoEIT4mdz8qhNHFiywY+5Hxd14CakRZAlfvEljOUOmTri7B xqM4viwH8zSFaN/z3QOyCbt2v/xoiQ2+3R0E8HQ+x3gZcx4/IYZdaM+n3cURavxSxA+2 XY++quULOwfNFfT/TOL1Ca5TtthK9XlNuLJqXLMVxg+/0E46Cj1wHF+xtSsZ1N9b4Jql 23zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZkF0//vS8P5S/motkhyB8BSxXm8jA0xCFMYcO1lVP80=; b=OCE4y47aFnsOwBwSWvoCdskLltmtvaawoKYCzUbzB3jEfd84ezJzqhieHqwv2hc5lP 52q/MI/hI3K2dm3ktWUPJMxBW1wmOEuR7mD3Y4i2M1vA6nASgFfLjVd0mVhxP/lmPoaq aZr46ukLB1xMwlxVudJaOzbkrKEY2299X5RXkao1Ny/8BLjYiqIJqpkKFLSXqrhVLe0Y HY1MV7IRmuc2wh7eJTcCMrwoWeKQAAHYTxJCO+3EeJf6camdt7xnXv5YGi2j37bAwMZv OcjyFs9TJJDhTIaVsKvjrioxt4n5MxyLL9bASgcXetNfHnNCKyIaveMxPbWB6LDtOWNY wg9w== X-Gm-Message-State: AOAM531Rr3KieqS99AiOazAdLKbWzUdaGloWYn7IyWlyh4vAVuqJPJwC iTUraF1V+HLYuDP+gAHMutqTfQdUOrzI3g== X-Google-Smtp-Source: ABdhPJzsaOl28+DXEVBZxr17oxsJ+Im0UfdYOoDjBfRRlCX2XNmWHRS5x0nygldfHP6m4eQTmiCQrA== X-Received: by 2002:a05:600c:190d:b0:39c:8216:f53d with SMTP id j13-20020a05600c190d00b0039c8216f53dmr3962769wmq.108.1655209862637; Tue, 14 Jun 2022 05:31:02 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.31.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:31:02 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 22/25] io_uring: pass hash table into poll_find Date: Tue, 14 Jun 2022 13:30:00 +0100 Message-Id: <7584a10fa437dc45e3e75141cc9d3f5596bc180c.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org In preparation for having multiple cancellation hash tables, pass a table pointer into io_poll_find() and other poll cancel functions. Signed-off-by: Pavel Begunkov --- io_uring/poll.c | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index 2688201e872a..b57e39082bac 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -556,11 +556,12 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, + struct io_hash_bucket hash_table[], struct io_hash_bucket **out_bucket) { struct io_kiocb *req; u32 index = hash_long(cd->data, ctx->cancel_hash_bits); - struct io_hash_bucket *hb = &ctx->cancel_hash[index]; + struct io_hash_bucket *hb = &hash_table[index]; *out_bucket = NULL; @@ -584,6 +585,7 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + struct io_hash_bucket hash_table[], struct io_hash_bucket **out_bucket) { struct io_kiocb *req; @@ -592,7 +594,7 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, *out_bucket = NULL; for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &ctx->cancel_hash[i]; + struct io_hash_bucket *hb = &hash_table[i]; spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { @@ -619,15 +621,16 @@ static bool io_poll_disarm(struct io_kiocb *req) return true; } -int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) +static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + struct io_hash_bucket hash_table[]) { struct io_hash_bucket *bucket; struct io_kiocb *req; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd, &bucket); + req = io_poll_file_find(ctx, cd, ctx->cancel_hash, &bucket); else - req = io_poll_find(ctx, false, cd, &bucket); + req = io_poll_find(ctx, false, cd, ctx->cancel_hash, &bucket); if (req) io_poll_cancel_req(req); @@ -636,6 +639,11 @@ int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) return req ? 0 : -ENOENT; } +int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) +{ + return __io_poll_cancel(ctx, cd, ctx->cancel_hash); +} + static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, unsigned int flags) { @@ -731,7 +739,7 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) int ret2, ret = 0; bool locked; - preq = io_poll_find(ctx, true, &cd, &bucket); + preq = io_poll_find(ctx, true, &cd, ctx->cancel_hash, &bucket); if (preq) ret2 = io_poll_disarm(preq); if (bucket) From patchwork Tue Jun 14 12:30:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7195C433EF for ; Tue, 14 Jun 2022 12:34:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356483AbiFNMej (ORCPT ); Tue, 14 Jun 2022 08:34:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244909AbiFNMe0 (ORCPT ); Tue, 14 Jun 2022 08:34:26 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A8FB46645 for ; Tue, 14 Jun 2022 05:31:05 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id x6-20020a1c7c06000000b003972dfca96cso4713513wmc.4 for ; Tue, 14 Jun 2022 05:31:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pI53pa4TxVJKzdBQkeVibcLg0pL+OIEI3I8Cb450LDs=; b=JNBPnEY8mXDmcIndU0yNPbhJvi6izUri8GVARuBXzuWy2St4wbFuvGasEXg7vKTrdv 9nbI0Jql/6MS6E0hDkAiT7pm7V1fj0ngt3YeZptXHve7eHQppoUK1wB1mmLaWNeRvwmt 13jrQrCdgO0RVUlRU5aGDDYYl+PZVLyBEWRTGNBS6igDAV+CCgBy2LdYg6u60GjiOgW5 zlqwk/DsG8UIPs3DTAOwPvm3iNRcr/AF9Y64uDjhddrGrRHtzL4JGuqMUeconYJbj0L1 SnhX5bmoSLfEQxld6z/CwRm1yY32YoYoXHizxbvy7Y1okH4Hgwh9JF8QLoOAhNs9v2bO zPRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pI53pa4TxVJKzdBQkeVibcLg0pL+OIEI3I8Cb450LDs=; b=T7NxlNiDoOQkEaHtc7g9v7uy27SYDOI1HqsvIhsYdnqDCXUHdWanoMFxjBMFYSFFJm 884x32KNO3xdnUFKalgFekXlhn8pDyXl+If2CVaj8SjVXj8R2JqXO1IqMHkX3yPdB8Kd lrrJfDyKL3UrOgx1QDHiR4qfo7sMT3G/7C3FI+NpdOgIih2sa4j3FJJyIcfj7R4vWfLv HX5MZRgr2N19lDv9XMZ+KvSLBXPz9p+58KHZVYCpS+Cj3svrA4WMowMe/Prr1jldtRdp iaCobDybYU38rKebKkMdE/WWzz5VSsOyNDUGl7iwHPpE78eGifR/nuytAKjPgeXKro0K l3OA== X-Gm-Message-State: AOAM532u38Aa4yqxmvbFdszDPRByBAyxfkLHdkwXl/DlXwhSLhyjJcI1 e4Q7U4H+qgLF/cjQ+AeC4HA0dHipiqfIjw== X-Google-Smtp-Source: ABdhPJyxpOJ1s8T4eBaCWN8KE2UmGVOCATtvQNluB5S5hj+PS0av2peLRNOfnbSh0j2DkfHSq2J76w== X-Received: by 2002:a05:600c:601f:b0:39c:416c:4069 with SMTP id az31-20020a05600c601f00b0039c416c4069mr3869310wmb.85.1655209863780; Tue, 14 Jun 2022 05:31:03 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.31.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:31:03 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 23/25] io_uring: introduce a struct for hash table Date: Tue, 14 Jun 2022 13:30:01 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Instead of passing around a pointer to hash buckets, add a bit of type safety and wrap it into a structure. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 6 +++--- io_uring/cancel.h | 7 +------ io_uring/fdinfo.c | 4 ++-- io_uring/io_uring.c | 29 ++++++++++++++++------------ io_uring/io_uring_types.h | 13 +++++++++++-- io_uring/poll.c | 40 +++++++++++++++++++++------------------ 6 files changed, 56 insertions(+), 43 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 2f157aebd3a0..1ee7bacd447f 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -193,12 +193,12 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -void init_hash_table(struct io_hash_bucket *hash_table, unsigned size) +void init_hash_table(struct io_hash_table *table, unsigned size) { unsigned int i; for (i = 0; i < size; i++) { - spin_lock_init(&hash_table[i].lock); - INIT_HLIST_HEAD(&hash_table[i].list); + spin_lock_init(&table->hbs[i].lock); + INIT_HLIST_HEAD(&table->hbs[i].list); } } diff --git a/io_uring/cancel.h b/io_uring/cancel.h index 556a7dcf160e..fd4cb1a2595d 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -4,9 +4,4 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd); -void init_hash_table(struct io_hash_bucket *hash_table, unsigned size); - -struct io_hash_bucket { - spinlock_t lock; - struct hlist_head list; -} ____cacheline_aligned_in_smp; +void init_hash_table(struct io_hash_table *table, unsigned size); diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index f941c73f5502..344e7d90d557 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -158,8 +158,8 @@ static __cold void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, mutex_unlock(&ctx->uring_lock); seq_puts(m, "PollList:\n"); - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &ctx->cancel_hash[i]; + for (i = 0; i < (1U << ctx->cancel_table.hash_bits); i++) { + struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i]; struct io_kiocb *req; spin_lock(&hb->lock); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index af0bd74e78fa..64a945fe47f5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -697,11 +697,23 @@ static __cold void io_fallback_req_func(struct work_struct *work) percpu_ref_put(&ctx->refs); } +static int io_alloc_hash_table(struct io_hash_table *table, unsigned bits) +{ + unsigned hash_buckets = 1U << bits; + size_t hash_size = hash_buckets * sizeof(table->hbs[0]); + + table->hbs = kmalloc(hash_size, GFP_KERNEL); + if (!table->hbs) + return -ENOMEM; + + table->hash_bits = bits; + init_hash_table(table, hash_buckets); + return 0; +} + static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) { struct io_ring_ctx *ctx; - unsigned hash_buckets; - size_t hash_size; int hash_bits; ctx = kzalloc(sizeof(*ctx), GFP_KERNEL); @@ -717,16 +729,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) */ hash_bits = ilog2(p->cq_entries) - 5; hash_bits = clamp(hash_bits, 1, 8); - hash_buckets = 1U << hash_bits; - hash_size = hash_buckets * sizeof(struct io_hash_bucket); - - ctx->cancel_hash_bits = hash_bits; - ctx->cancel_hash = kmalloc(hash_size, GFP_KERNEL); - if (!ctx->cancel_hash) + if (io_alloc_hash_table(&ctx->cancel_table, hash_bits)) goto err; - init_hash_table(ctx->cancel_hash, hash_buckets); - ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) goto err; @@ -767,7 +772,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) return ctx; err: kfree(ctx->dummy_ubuf); - kfree(ctx->cancel_hash); + kfree(ctx->cancel_table.hbs); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); kfree(ctx); @@ -3050,7 +3055,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_req_caches_free(ctx); if (ctx->hash_map) io_wq_put_hash(ctx->hash_map); - kfree(ctx->cancel_hash); + kfree(ctx->cancel_table.hbs); kfree(ctx->dummy_ubuf); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index faa1477bd754..d1ce3735aca1 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -7,6 +7,16 @@ #include "io-wq.h" #include "filetable.h" +struct io_hash_bucket { + spinlock_t lock; + struct hlist_head list; +} ____cacheline_aligned_in_smp; + +struct io_hash_table { + struct io_hash_bucket *hbs; + unsigned hash_bits; +}; + struct io_uring { u32 head ____cacheline_aligned_in_smp; u32 tail ____cacheline_aligned_in_smp; @@ -222,8 +232,7 @@ struct io_ring_ctx { * manipulate the list, hence no extra locking is needed there. */ struct io_wq_work_list iopoll_list; - struct io_hash_bucket *cancel_hash; - unsigned cancel_hash_bits; + struct io_hash_table cancel_table; bool poll_multi_queue; struct list_head io_buffers_comp; diff --git a/io_uring/poll.c b/io_uring/poll.c index b57e39082bac..2f49d15cfcce 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -73,9 +73,9 @@ static struct io_poll *io_poll_get_single(struct io_kiocb *req) static void io_poll_req_insert(struct io_kiocb *req) { - struct io_ring_ctx *ctx = req->ctx; - u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); - struct io_hash_bucket *hb = &ctx->cancel_hash[index]; + struct io_hash_table *table = &req->ctx->cancel_table; + u32 index = hash_long(req->cqe.user_data, table->hash_bits); + struct io_hash_bucket *hb = &table->hbs[index]; spin_lock(&hb->lock); hlist_add_head(&req->hash_node, &hb->list); @@ -84,8 +84,9 @@ static void io_poll_req_insert(struct io_kiocb *req) static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) { - u32 index = hash_long(req->cqe.user_data, ctx->cancel_hash_bits); - spinlock_t *lock = &ctx->cancel_hash[index].lock; + struct io_hash_table *table = &req->ctx->cancel_table; + u32 index = hash_long(req->cqe.user_data, table->hash_bits); + spinlock_t *lock = &table->hbs[index].lock; spin_lock(lock); hash_del(&req->hash_node); @@ -533,13 +534,15 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, bool cancel_all) { + struct io_hash_table *table = &ctx->cancel_table; + unsigned nr_buckets = 1U << table->hash_bits; struct hlist_node *tmp; struct io_kiocb *req; bool found = false; int i; - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &ctx->cancel_hash[i]; + for (i = 0; i < nr_buckets; i++) { + struct io_hash_bucket *hb = &table->hbs[i]; spin_lock(&hb->lock); hlist_for_each_entry_safe(req, tmp, &hb->list, hash_node) { @@ -556,12 +559,12 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, - struct io_hash_bucket hash_table[], + struct io_hash_table *table, struct io_hash_bucket **out_bucket) { struct io_kiocb *req; - u32 index = hash_long(cd->data, ctx->cancel_hash_bits); - struct io_hash_bucket *hb = &hash_table[index]; + u32 index = hash_long(cd->data, table->hash_bits); + struct io_hash_bucket *hb = &table->hbs[index]; *out_bucket = NULL; @@ -585,16 +588,17 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, struct io_cancel_data *cd, - struct io_hash_bucket hash_table[], + struct io_hash_table *table, struct io_hash_bucket **out_bucket) { + unsigned nr_buckets = 1U << table->hash_bits; struct io_kiocb *req; int i; *out_bucket = NULL; - for (i = 0; i < (1U << ctx->cancel_hash_bits); i++) { - struct io_hash_bucket *hb = &hash_table[i]; + for (i = 0; i < nr_buckets; i++) { + struct io_hash_bucket *hb = &table->hbs[i]; spin_lock(&hb->lock); hlist_for_each_entry(req, &hb->list, hash_node) { @@ -622,15 +626,15 @@ static bool io_poll_disarm(struct io_kiocb *req) } static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, - struct io_hash_bucket hash_table[]) + struct io_hash_table *table) { struct io_hash_bucket *bucket; struct io_kiocb *req; if (cd->flags & (IORING_ASYNC_CANCEL_FD|IORING_ASYNC_CANCEL_ANY)) - req = io_poll_file_find(ctx, cd, ctx->cancel_hash, &bucket); + req = io_poll_file_find(ctx, cd, table, &bucket); else - req = io_poll_find(ctx, false, cd, ctx->cancel_hash, &bucket); + req = io_poll_find(ctx, false, cd, table, &bucket); if (req) io_poll_cancel_req(req); @@ -641,7 +645,7 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) { - return __io_poll_cancel(ctx, cd, ctx->cancel_hash); + return __io_poll_cancel(ctx, cd, &ctx->cancel_table); } static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, @@ -739,7 +743,7 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) int ret2, ret = 0; bool locked; - preq = io_poll_find(ctx, true, &cd, ctx->cancel_hash, &bucket); + preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table, &bucket); if (preq) ret2 = io_poll_disarm(preq); if (bucket) From patchwork Tue Jun 14 12:30:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64EFAC43334 for ; Tue, 14 Jun 2022 12:34:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244453AbiFNMei (ORCPT ); Tue, 14 Jun 2022 08:34:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244877AbiFNMe0 (ORCPT ); Tue, 14 Jun 2022 08:34:26 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A795446673 for ; Tue, 14 Jun 2022 05:31:05 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id w17so3574055wrg.7 for ; Tue, 14 Jun 2022 05:31:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=62dWMekHuhr5uFwrGJkmLtFH6YzJbWWvh8PXlOiIzdE=; b=HIB/T825Z1g8+8Shi0uhyXRzr+e8hseyNEl7uoYbz1k9nwBj9Xjy6Qz3td3w+JAcVz jf9Qj5/6pRb+s5EemMSYO3Ro+tE7qbQ2SV6owvfIM3ERAEFCu3P7MNQcp0+aRBEgLbRj LO4uSJrHeFTsoxrerz7Kgt0CsqZVAyjajUyx2UzFuiFS0fDkGRInzbydbzsQyBssyu9i 1holW9bjkt8RvhfqhJCMjvYxthsBzWFcpMkytYfiaFM7xlqSMgBuNoO3AVEqV9tw2gqV YMNldpMsGPyfv3BQNfN9HLAcRDy1YkyfLMsxVWcvQ+r1P4c6k4ReT33olsesO0tpAwfQ 6ROA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=62dWMekHuhr5uFwrGJkmLtFH6YzJbWWvh8PXlOiIzdE=; b=HJdAY2vNeAOPLEkdRf6mhJHUARh7yQdM20IGOSDUuLgDfU2Oe4A4W14OVQFBMCykaD RpUg0G+C2O/0ghiBLt3tfq7aw3KYor0/FkgAlWwnNk8NLP9dYOuJIpswx5eDRw8ry0x4 jXk1E6i8BhZnlSM+m7TheItxi2yq43q5O7KkWVFW8+3/HbqZ3rom5wYj/qyjNfB6lIdH wLYOEeLHmivXU/DjXvpgiilK9F3wPns4uBA1+vPOCPQVdRgZ9XKGN2w3pfvq95JlMEYL f3ZYsPPaiqew336xOeZI0LDurp8LAqIU/kfLzKUVdUFHJcnMIeR2Hti1L6XGZmB7iTRk FBfQ== X-Gm-Message-State: AJIora+GzeV0nn70yPdEwLvNqPfSkwNaVf2sP0PDCEXfXWxZXilIJyMP jUqeGmOHZ3hyD6syBNQeu4jPpbXeQsnsTA== X-Google-Smtp-Source: AGRyM1s3j7H2Cl96Pgeg0UEv7DN0FdKnRlcEErbrQ3n6q+FhuHiW6vA+qfd874ZdXehB6q3ZdHKbGQ== X-Received: by 2002:adf:e10d:0:b0:20c:dc8f:e5a5 with SMTP id t13-20020adfe10d000000b0020cdc8fe5a5mr4736778wrz.265.1655209864981; Tue, 14 Jun 2022 05:31:04 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.31.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:31:04 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 24/25] io_uring: propagate locking state to poll cancel Date: Tue, 14 Jun 2022 13:30:02 +0100 Message-Id: <19652167659a9e397fecd155ea152713791bd826.1655209709.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Poll cancellation will be soon need to grab ->uring_lock inside, pass the locking state, i.e. issue_flags, inside the cancellation functions. Signed-off-by: Pavel Begunkov --- io_uring/cancel.c | 7 ++++--- io_uring/cancel.h | 3 ++- io_uring/poll.c | 3 ++- io_uring/poll.h | 3 ++- io_uring/timeout.c | 3 ++- 5 files changed, 12 insertions(+), 7 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 1ee7bacd447f..14fcd4fd10d9 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -78,7 +78,8 @@ static int io_async_cancel_one(struct io_uring_task *tctx, return ret; } -int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) +int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd, + unsigned issue_flags) { struct io_ring_ctx *ctx = req->ctx; int ret; @@ -93,7 +94,7 @@ int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd) if (!ret) return 0; - ret = io_poll_cancel(ctx, cd); + ret = io_poll_cancel(ctx, cd, issue_flags); if (ret != -ENOENT) return ret; @@ -136,7 +137,7 @@ static int __io_async_cancel(struct io_cancel_data *cd, struct io_kiocb *req, int ret, nr = 0; do { - ret = io_try_cancel(req, cd); + ret = io_try_cancel(req, cd, issue_flags); if (ret == -ENOENT) break; if (!all) diff --git a/io_uring/cancel.h b/io_uring/cancel.h index fd4cb1a2595d..8dd259dc383e 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -3,5 +3,6 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags); -int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd); +int io_try_cancel(struct io_kiocb *req, struct io_cancel_data *cd, + unsigned int issue_flags); void init_hash_table(struct io_hash_table *table, unsigned size); diff --git a/io_uring/poll.c b/io_uring/poll.c index 2f49d15cfcce..db7357de7f8c 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -643,7 +643,8 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, return req ? 0 : -ENOENT; } -int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) +int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned issue_flags) { return __io_poll_cancel(ctx, cd, &ctx->cancel_table); } diff --git a/io_uring/poll.h b/io_uring/poll.h index cc75c1567a84..fa3e19790281 100644 --- a/io_uring/poll.h +++ b/io_uring/poll.h @@ -24,7 +24,8 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags); int io_poll_remove_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags); -int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd); +int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, + unsigned issue_flags); int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags); bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, bool cancel_all); diff --git a/io_uring/timeout.c b/io_uring/timeout.c index 89000aae65d9..0b3751959e88 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -262,6 +262,7 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) { + unsigned issue_flags = *locked ? 0 : IO_URING_F_UNLOCKED; struct io_timeout *timeout = io_kiocb_to_cmd(req); struct io_kiocb *prev = timeout->prev; int ret = -ENOENT; @@ -273,7 +274,7 @@ static void io_req_task_link_timeout(struct io_kiocb *req, bool *locked) .data = prev->cqe.user_data, }; - ret = io_try_cancel(req, &cd); + ret = io_try_cancel(req, &cd, issue_flags); } io_req_set_res(req, ret ?: -ETIME, 0); io_req_complete_post(req); From patchwork Tue Jun 14 12:30:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12881138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0707C43334 for ; Tue, 14 Jun 2022 12:34:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1356534AbiFNMel (ORCPT ); Tue, 14 Jun 2022 08:34:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245682AbiFNMe1 (ORCPT ); Tue, 14 Jun 2022 08:34:27 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2678647569 for ; Tue, 14 Jun 2022 05:31:07 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id i17-20020a7bc951000000b0039c4760ec3fso835953wml.0 for ; Tue, 14 Jun 2022 05:31:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4yIX3gVkG6/uT9LgOTEi4hy6UN+9Z2Jm3ZZftf3P9LE=; b=EUEp4I6zysGh3HKV/yU/NMXoI/J5mnVoMPshnzcRU6kglCYT8ogf7x4mLl2FLHqnja 3KtTNT8N6CH4GwnONnWOQq5EYO0IPtPTMn9c1jEOCWw+uuHRKPOX1edPbhspeMgjQnRp Qem5AIjpiC0zDhkSUk2jAMqEjLAyBgsLalkLni9xHm+zIXc9f6BujWyEY0pihFrS66Qc BRGuIWRDcXuFSkpCnTxNWYrf9kiNzoy3mZanhFbwcrQ/kxh4jQIgmQAoD0PVqGPl/h7Y QqcR5Kl+jYtRi7QdiqtrgSNYDI0Hnq+5AUX1IC4qsBrGZwwf8aqUL6itIURPn++aBCWy fRjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4yIX3gVkG6/uT9LgOTEi4hy6UN+9Z2Jm3ZZftf3P9LE=; b=kr4CjJKRKSlyZXkwGOWWl/mupAO7dct/0ITZ+MZ7z/5B1agqXI0rFMoo7ZllFvdBeI EBR61TWrDLkk1lk1xxBX7JHICUcpTwx+Ax2FnsR4VqcBvkZYufT2kxd6bR549mGOZu4E HDhEqjUlbkesqyZJHLTpr7ZjGaN9gazcJg/bmgCUOl01EqbX2DtaiARXOxedErlcTjMz 3jF+wJUvv10V/PqBAkUhWjsbVmecwEBDpJUUN1Kf8lIVy49O9ZXnsBTLS3Mmlo611l9g ObRd6cp8Dnp4IZNU5l/6eTIDqUmxPHH2UpzUTIuyGBaVerfV9V4i1eiOrvSlPJjNQOpu WX6A== X-Gm-Message-State: AOAM533imeGgE4uSSMb/4l9lceVckwiLWh7HcANNoz3c4dH9gpjdUmXE U6jXsFd1an7yyq3FO6DW3dm8dw+I+DqBuQ== X-Google-Smtp-Source: ABdhPJypoaHFONLPmF/fTnI7La52VETRFWoJPRz3VXXtMvblgjCctzBHZ8ASiVtzg1bnfMhGY2Sf8Q== X-Received: by 2002:a7b:c3d1:0:b0:39c:64d0:b8c2 with SMTP id t17-20020a7bc3d1000000b0039c64d0b8c2mr3971358wmj.195.1655209866143; Tue, 14 Jun 2022 05:31:06 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.125.106.threembb.co.uk. [188.28.125.106]) by smtp.gmail.com with ESMTPSA id t7-20020a05600c198700b0039c5fb1f592sm12410651wmq.14.2022.06.14.05.31.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 05:31:05 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH for-next 25/25] io_uring: mutex locked poll hashing Date: Tue, 14 Jun 2022 13:30:03 +0100 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Currently we do two extra spin lock/unlock pairs to add a poll/apoll request to the cancellation hash table and remove it from there. On the submission side we often already hold ->uring_lock and tw completion is likely to hold it as well. Add a second cancellation hash table protected by ->uring_lock. In concerns for latency because of a need to have the mutex locked on the completion side, use the new table only in following cases: 1) IORING_SETUP_SINGLE_ISSUER: only one task grabs uring_lock, so there is no contention and so the main tw hander will always end up grabbing it before calling into callbacks. 2) IORING_SETUP_SQPOLL: same as with single issuer, only one task is using ->uring_lock. 3) apoll: we normally grab the lock on the completion side anyway to execute the request, so it's free. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 9 +++- io_uring/io_uring_types.h | 4 ++ io_uring/poll.c | 111 ++++++++++++++++++++++++++++++-------- 3 files changed, 102 insertions(+), 22 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 64a945fe47f5..169947b3679e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -731,6 +731,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) hash_bits = clamp(hash_bits, 1, 8); if (io_alloc_hash_table(&ctx->cancel_table, hash_bits)) goto err; + if (io_alloc_hash_table(&ctx->cancel_table_locked, hash_bits)) + goto err; ctx->dummy_ubuf = kzalloc(sizeof(*ctx->dummy_ubuf), GFP_KERNEL); if (!ctx->dummy_ubuf) @@ -773,6 +775,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) err: kfree(ctx->dummy_ubuf); kfree(ctx->cancel_table.hbs); + kfree(ctx->cancel_table_locked.hbs); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); kfree(ctx); @@ -3056,6 +3059,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) if (ctx->hash_map) io_wq_put_hash(ctx->hash_map); kfree(ctx->cancel_table.hbs); + kfree(ctx->cancel_table_locked.hbs); kfree(ctx->dummy_ubuf); kfree(ctx->io_bl); xa_destroy(&ctx->io_bl_xa); @@ -3217,12 +3221,13 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx) __io_cqring_overflow_flush(ctx, true); xa_for_each(&ctx->personalities, index, creds) io_unregister_personality(ctx, index); + if (ctx->rings) + io_poll_remove_all(ctx, NULL, true); mutex_unlock(&ctx->uring_lock); /* failed during ring init, it couldn't have issued any requests */ if (ctx->rings) { io_kill_timeouts(ctx, NULL, true); - io_poll_remove_all(ctx, NULL, true); /* if we failed setting up the ctx, we might not have any rings */ io_iopoll_try_reap_events(ctx); } @@ -3347,7 +3352,9 @@ static __cold void io_uring_try_cancel_requests(struct io_ring_ctx *ctx, } ret |= io_cancel_defer_files(ctx, task, cancel_all); + mutex_lock(&ctx->uring_lock); ret |= io_poll_remove_all(ctx, task, cancel_all); + mutex_unlock(&ctx->uring_lock); ret |= io_kill_timeouts(ctx, task, cancel_all); if (task) ret |= io_run_task_work(); diff --git a/io_uring/io_uring_types.h b/io_uring/io_uring_types.h index d1ce3735aca1..6c3c3c88223a 100644 --- a/io_uring/io_uring_types.h +++ b/io_uring/io_uring_types.h @@ -189,6 +189,7 @@ struct io_ring_ctx { struct xarray io_bl_xa; struct list_head io_buffers_cache; + struct io_hash_table cancel_table_locked; struct list_head cq_overflow_list; struct list_head apoll_cache; struct xarray personalities; @@ -323,6 +324,7 @@ enum { /* keep async read/write and isreg together and in order */ REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, + REQ_F_HASH_LOCKED_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -388,6 +390,8 @@ enum { REQ_F_APOLL_MULTISHOT = BIT(REQ_F_APOLL_MULTISHOT_BIT), /* recvmsg special flag, clear EPOLLIN */ REQ_F_CLEAR_POLLIN = BIT(REQ_F_CLEAR_POLLIN_BIT), + /* hashed into ->cancel_hash_locked */ + REQ_F_HASH_LOCKED = BIT(REQ_F_HASH_LOCKED_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, bool *locked); diff --git a/io_uring/poll.c b/io_uring/poll.c index db7357de7f8c..5eb805b97f5d 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -93,6 +93,26 @@ static void io_poll_req_delete(struct io_kiocb *req, struct io_ring_ctx *ctx) spin_unlock(lock); } +static void io_poll_req_insert_locked(struct io_kiocb *req) +{ + struct io_hash_table *table = &req->ctx->cancel_table_locked; + u32 index = hash_long(req->cqe.user_data, table->hash_bits); + + hlist_add_head(&req->hash_node, &table->hbs[index].list); +} + +static void io_poll_tw_hash_eject(struct io_kiocb *req, bool *locked) +{ + struct io_ring_ctx *ctx = req->ctx; + + if (req->flags & REQ_F_HASH_LOCKED) { + io_tw_lock(ctx, locked); + hash_del(&req->hash_node); + } else { + io_poll_req_delete(req, ctx); + } +} + static void io_init_poll_iocb(struct io_poll *poll, __poll_t events, wait_queue_func_t wake_func) { @@ -217,7 +237,6 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked) static void io_poll_task_func(struct io_kiocb *req, bool *locked) { - struct io_ring_ctx *ctx = req->ctx; int ret; ret = io_poll_check_events(req, locked); @@ -234,7 +253,8 @@ static void io_poll_task_func(struct io_kiocb *req, bool *locked) } io_poll_remove_entries(req); - io_poll_req_delete(req, ctx); + io_poll_tw_hash_eject(req, locked); + io_req_set_res(req, req->cqe.res, 0); io_req_task_complete(req, locked); } @@ -248,7 +268,7 @@ static void io_apoll_task_func(struct io_kiocb *req, bool *locked) return; io_poll_remove_entries(req); - io_poll_req_delete(req, req->ctx); + io_poll_tw_hash_eject(req, locked); if (!ret) io_req_task_submit(req, locked); @@ -442,7 +462,10 @@ static int __io_arm_poll_handler(struct io_kiocb *req, return 0; } - io_poll_req_insert(req); + if (req->flags & REQ_F_HASH_LOCKED) + io_poll_req_insert_locked(req); + else + io_poll_req_insert(req); if (mask && (poll->events & EPOLLET)) { /* can't multishot if failed, just queue the event we've got */ @@ -480,6 +503,15 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) __poll_t mask = POLLPRI | POLLERR | EPOLLET; int ret; + /* + * apoll requests already grab the mutex to complete in the tw handler, + * so removal from the mutex-backed hash is free, use it by default. + */ + if (issue_flags & IO_URING_F_UNLOCKED) + req->flags &= ~REQ_F_HASH_LOCKED; + else + req->flags |= REQ_F_HASH_LOCKED; + if (!def->pollin && !def->pollout) return IO_APOLL_ABORTED; if (!file_can_poll(req->file)) @@ -528,13 +560,10 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) return IO_APOLL_OK; } -/* - * Returns true if we found and killed one or more poll requests - */ -__cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, - bool cancel_all) +static __cold bool io_poll_remove_all_table(struct task_struct *tsk, + struct io_hash_table *table, + bool cancel_all) { - struct io_hash_table *table = &ctx->cancel_table; unsigned nr_buckets = 1U << table->hash_bits; struct hlist_node *tmp; struct io_kiocb *req; @@ -557,6 +586,17 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, return found; } +/* + * Returns true if we found and killed one or more poll requests + */ +__cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct task_struct *tsk, + bool cancel_all) + __must_hold(&ctx->uring_lock) +{ + return io_poll_remove_all_table(tsk, &ctx->cancel_table, cancel_all) | + io_poll_remove_all_table(tsk, &ctx->cancel_table_locked, cancel_all); +} + static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, struct io_cancel_data *cd, struct io_hash_table *table, @@ -616,13 +656,15 @@ static struct io_kiocb *io_poll_file_find(struct io_ring_ctx *ctx, return NULL; } -static bool io_poll_disarm(struct io_kiocb *req) +static int io_poll_disarm(struct io_kiocb *req) { + if (!req) + return -ENOENT; if (!io_poll_get_ownership(req)) - return false; + return -EALREADY; io_poll_remove_entries(req); hash_del(&req->hash_node); - return true; + return 0; } static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, @@ -646,7 +688,16 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, int io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd, unsigned issue_flags) { - return __io_poll_cancel(ctx, cd, &ctx->cancel_table); + int ret; + + ret = __io_poll_cancel(ctx, cd, &ctx->cancel_table); + if (ret != -ENOENT) + return ret; + + io_ring_submit_lock(ctx, issue_flags); + ret = __io_poll_cancel(ctx, cd, &ctx->cancel_table_locked); + io_ring_submit_unlock(ctx, issue_flags); + return ret; } static __poll_t io_poll_parse_events(const struct io_uring_sqe *sqe, @@ -721,6 +772,16 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) ipt.pt._qproc = io_poll_queue_proc; + /* + * If sqpoll or single issuer, there is no contention for ->uring_lock + * and we'll end up holding it in tw handlers anyway. + */ + if (!(issue_flags & IO_URING_F_UNLOCKED) && + (req->ctx->flags & (IORING_SETUP_SQPOLL | IORING_SETUP_SINGLE_ISSUER))) + req->flags |= REQ_F_HASH_LOCKED; + else + req->flags &= ~REQ_F_HASH_LOCKED; + ret = __io_arm_poll_handler(req, poll, &ipt, poll->events); if (ipt.error) { return ipt.error; @@ -745,20 +806,28 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags) bool locked; preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table, &bucket); - if (preq) - ret2 = io_poll_disarm(preq); + ret2 = io_poll_disarm(preq); if (bucket) spin_unlock(&bucket->lock); - - if (!preq) { - ret = -ENOENT; + if (!ret2) + goto found; + if (ret2 != -ENOENT) { + ret = ret2; goto out; } - if (!ret2) { - ret = -EALREADY; + + io_ring_submit_lock(ctx, issue_flags); + preq = io_poll_find(ctx, true, &cd, &ctx->cancel_table_locked, &bucket); + ret2 = io_poll_disarm(preq); + if (bucket) + spin_unlock(&bucket->lock); + io_ring_submit_unlock(ctx, issue_flags); + if (ret2) { + ret = ret2; goto out; } +found: if (poll_update->update_events || poll_update->update_user_data) { /* only mask one event flags, keep behavior flags */ if (poll_update->update_events) {