From patchwork Thu Aug 24 22:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364840 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0D6DEE49A0 for ; Thu, 24 Aug 2023 22:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234851AbjHXWzm (ORCPT ); Thu, 24 Aug 2023 18:55:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244010AbjHXWz2 (ORCPT ); Thu, 24 Aug 2023 18:55:28 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7A651FD6 for ; Thu, 24 Aug 2023 15:55:20 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-99bc9e3cbf1so86996766b.0 for ; Thu, 24 Aug 2023 15:55:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917719; x=1693522519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TNuX+XOKOpOPnPfeKjKStgxGv91vortKXWoWHq+NGN8=; b=GN7Y51rkxh/KDEYJopcAC0JCE27tT3N8tM7IhoR+81h6Lp3NJDp7fWI28+2C9jbsE/ 5oegkyfnn3A/ffz8pDOOKtYjNkUl3uILDcvZ1q95F5yuA+6a7q4Sd2fWSSz2zsBBjEH7 v3GaiR/YzYqKAkSU/cdJUDxoHubEPbdaLTCokhsbx/71rbvWAG1orcQxv7t5s3Q+eElW 1fLhCrE94RH0tTHgYzezABuneZEhJqbxq4/McgsNYAni9rbRojHE0vJ2OkIVh0nX9aBJ 7otJ1W4gp7buPvCqvReJASaR9CCTDSdiWdmkI9WvimzSUmDz0ZRKqyBHrx/Sw+tQwy96 +RNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917719; x=1693522519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TNuX+XOKOpOPnPfeKjKStgxGv91vortKXWoWHq+NGN8=; b=OhEtAxO1UhZlsF+fwzKdoEjmfsIo6O3rR2FqWi+c/6QlBjOBQtkc/sE6P4vxLg+JMg pe1ZO4XPyMnNzrEUMrm61ALvHbsa5iZC1wKiwLbgWObS941YGzSkI+tHr8zGgk+d6+jQ vHwh1wzgekYK37OVSCqy2Xjn4PYyXDv9JZdrDtBWXFR3h5lJEnwQjV5i99vbM47QeKbZ WZQK2+9/4IXsdPOCDpJ9Tr68hiWDRlcGtMuz/x03KD+JZ1no2ifH+ZEriNvN3aZlNm8U 2XGwP7xww8p2G3MgQCZkjOnYnDJMkXjFSwqwLG96TkPCS4Jnzc6eIRy1QPylrA2UOund p2MQ== X-Gm-Message-State: AOJu0YzMime8hXzOvK+Zv452Uamon3ILb9xUsoWi96EhoUew1QSOfwDV lTCtmQyh0a/ycS4YSXhJUMlJWRSdc/Y= X-Google-Smtp-Source: AGHT+IEV2wWabq7ntU5I449SamtsFWtTSO6aZzb8f78mtoI0n8IAlfLbCTifxplbWkayRUlRyKhOow== X-Received: by 2002:a17:907:1c84:b0:9a1:c32c:a69c with SMTP id nb4-20020a1709071c8400b009a1c32ca69cmr9542348ejc.17.1692917718748; Thu, 24 Aug 2023 15:55:18 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:18 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 01/15] io_uring: improve cqe !tracing hot path Date: Thu, 24 Aug 2023 23:53:23 +0100 Message-ID: <555d8312644b3776f4be7e23f9b92943875c4bc7.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org While looking at io_fill_cqe_req()'s asm I stumbled on our trace points turning into the chunk below: trace_io_uring_complete(req->ctx, req, req->cqe.user_data, req->cqe.res, req->cqe.flags, req->extra1, req->extra2); io_uring/io_uring.c:898: trace_io_uring_complete(req->ctx, req, req->cqe.user_data, movq 232(%rbx), %rdi # req_44(D)->big_cqe.extra2, _5 movq 224(%rbx), %rdx # req_44(D)->big_cqe.extra1, _6 movl 84(%rbx), %r9d # req_44(D)->cqe.D.81184.flags, _7 movl 80(%rbx), %r8d # req_44(D)->cqe.res, _8 movq 72(%rbx), %rcx # req_44(D)->cqe.user_data, _9 movq 88(%rbx), %rsi # req_44(D)->ctx, _10 ./arch/x86/include/asm/jump_label.h:27: asm_volatile_goto("1:" 1:jmp .L1772 # objtool NOPs this # ... It does a jump_label for actual tracing, but those 6 moves will stay there in the hottest io_uring path. As an optimisation, add a trace_io_uring_complete_enabled() check, which is also uses jump_labels, it tricks the compiler into behaving. It removes the junk without changing anything else int the hot path. Note: apparently, it's not only me noticing it, and people are also working it around. We should remove the check when it's solved generically or rework tracing. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 3e6ff3cd9a24..465598223386 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -145,10 +145,11 @@ static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, struct io_kiocb *req if (unlikely(!cqe)) return false; - trace_io_uring_complete(req->ctx, req, req->cqe.user_data, - req->cqe.res, req->cqe.flags, - (req->flags & REQ_F_CQE32_INIT) ? req->extra1 : 0, - (req->flags & REQ_F_CQE32_INIT) ? req->extra2 : 0); + if (trace_io_uring_complete_enabled()) + trace_io_uring_complete(req->ctx, req, req->cqe.user_data, + req->cqe.res, req->cqe.flags, + (req->flags & REQ_F_CQE32_INIT) ? req->extra1 : 0, + (req->flags & REQ_F_CQE32_INIT) ? req->extra2 : 0); memcpy(cqe, &req->cqe, sizeof(*cqe)); From patchwork Thu Aug 24 22:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ED28C6FA8F for ; Thu, 24 Aug 2023 22:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239486AbjHXWzm (ORCPT ); Thu, 24 Aug 2023 18:55:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244011AbjHXWz2 (ORCPT ); Thu, 24 Aug 2023 18:55:28 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51EAD1FD7 for ; Thu, 24 Aug 2023 15:55:21 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-98377c5d53eso33868666b.0 for ; Thu, 24 Aug 2023 15:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917719; x=1693522519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iUl2RRSOOJ1zzBFK6rQynnQ5/OpqN87Xbc+dPAhq4HU=; b=Ecu7Kq/kP7NS+7KT6NUec09qbeGoViEs0Hji8YEosb2ZHpaFJ7OrPAWe6urZD0Y617 bDpAuBIwIDhhdbyrrW/LBimK9BIX8NNrGDyOWlCZKSsCjVVVuIafvt6MQe4hDVP//JIG NcwjqtAFQqlfJa6pYFxE7mAisScC5mhDrjaAn9ef0PWPxoJiTlNn1MJH08QaixXiTqjc Dj0zOSPBqyCThHm5+RnezRebaEqsZnwTQiYwZ456gEamRrJPS0wi965j0lhkJE49s0vR pRqS0SInfq7253gKzdat0cnyVmb5mVE8Vzy4m6rFW6w/qPUsj80IZ4mcc4QWpq7i5C+C U7AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917719; x=1693522519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iUl2RRSOOJ1zzBFK6rQynnQ5/OpqN87Xbc+dPAhq4HU=; b=fMWBmJFkLeQt5D9ynAsIOQ4Ege7+CQgB7vBTU4LzJhpvLL89Org49VKZ0y3X+gyOXm +qr9pucIkpnZ9BFf59TLJGFfGPgiG6l2wFkLSPsd8IZ3zWP7zRdiDGvGNDk8ZOKfL/y/ kjZHEybx59cq+OEUKk2Calm70c3PGZI+EUwNQh10MynM2Y5Cv216u9mPOLvXqk3yet4k JP+eCk5q8AT3iIlhrmvYyIy39wA06AsmLMI3w3NKcbsEjMOwS+HR4WUyNRhOt4q5MAfX wbESaQc/xlhdBavgntzTKcrLlf4t89F50n1LubBjq/oQ5oxPffe+L7vdIBOwIGYuOfdu zzVg== X-Gm-Message-State: AOJu0YyCr2a1MmWHPCra+8RCBxm2/2ibDu1Gl3HykY4D/Q62JBpMBq5c lA3JQzCEiIlVeHRJGRhQwdM/1uteJS8= X-Google-Smtp-Source: AGHT+IFf410cCv/xJNu379+Qmg18Ad+/Mwj+tl/b8Mb9yS+XqA5kuaMQ3Skl8N2MLU+7mEJxTLxlCQ== X-Received: by 2002:a17:907:2711:b0:9a1:e66e:b69a with SMTP id w17-20020a170907271100b009a1e66eb69amr4099791ejk.21.1692917719517; Thu, 24 Aug 2023 15:55:19 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:19 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 02/15] io_uring: cqe init hardening Date: Thu, 24 Aug 2023 23:53:24 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_kiocb::cqe stores the completion info which we'll memcpy to userspace, and we rely on callbacks and other later steps to populate it with right values. We have never had problems with that, but it would still be safer to zero it on allocation. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e1a23f4993d3..3e0fe1ebbc10 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1056,7 +1056,7 @@ static void io_preinit_req(struct io_kiocb *req, struct io_ring_ctx *ctx) req->link = NULL; req->async_data = NULL; /* not necessary, but safer to zero */ - req->cqe.res = 0; + memset(&req->cqe, 0, sizeof(req->cqe)); } static void io_flush_cached_locked_reqs(struct io_ring_ctx *ctx, From patchwork Thu Aug 24 22:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364844 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF666EE49A6 for ; Thu, 24 Aug 2023 22:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243951AbjHXWzn (ORCPT ); Thu, 24 Aug 2023 18:55:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244019AbjHXWzb (ORCPT ); Thu, 24 Aug 2023 18:55:31 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E1BE1FDE for ; Thu, 24 Aug 2023 15:55:22 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-99c136ee106so32795366b.1 for ; Thu, 24 Aug 2023 15:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917720; x=1693522520; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1G/AFWb5iyL48Qc9HzbF86s/wLNrCJOiAeFSM8nteRg=; b=DE0tfFnKkSWmAzFBZznvuhWTz2UgV06UcLKMpMRcLOIVopjcz3MXyL9DqlFByol85M RsDvdZZRge1qKqG20DehxcDZscooQzHQ6kiQ50gIjIng+Z+XEKh5GntkxRqr30/4dVFt FSbOtX6yHc9Bl3uOLuHDPFlqKDONV5aCaj69+3EvramxgKzbMJ+12/gA4Fo+YhqcNX9X UMFwU50XHBrd6wpV/7FoeWr6lV8+QHmry7mNeXT/w5q7rWgzCKhCvSsb2CFRawjECmiL iAdAf0f6z6nXJo4wptHXZz6MO0AQSJt8dsQXaWpKMmNtjaCLx355PozJlqngHp3MCsS2 wF+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917720; x=1693522520; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1G/AFWb5iyL48Qc9HzbF86s/wLNrCJOiAeFSM8nteRg=; b=EHlNab1Zm8EBcBhMCuaolbDAbYopTxugESlIvDUdm07pAUo7Dl1Zcu2z1ieJVn6fbf +kCWv6Y0/jgetsP0EY0nU2iUkRak9UyEn1IfiM36Qp9WntFy1vIc4SVi8yfvOOfoBMWL Jtc9Yi0GY4KwtSf3tZ75QhKH6uwz7slTaIcFYqVgZSC8IZnrZDSd6IUi9WuZdONFw1nk tYuI/2LNgQmX3grtffvE6y5gSme+2e+SxjI8Owfxwy0jasB2LX3rMIOgWa1puB2QCj9x dUU3QfN+TOf/pXClACUKhU8NRuHCOwnSuTcryV9XUa4nLANpLBzrPTiSzLVFc1XlRtZ5 Ax4w== X-Gm-Message-State: AOJu0YwY8yJI/nv4Ci/pDkyTnS3Zryyuypq4sDmrHbSjlbi9L1CY7G8k l4AwWxvCLFJ+K0OZqa6aTM/Z6JMPkkg= X-Google-Smtp-Source: AGHT+IF9jUv0l9seJq+bCEq8PmDkvGQsZ5stZ62x46MhR29Wh+cRoKsIcI68BlnOwu+oo+5fednBig== X-Received: by 2002:a17:906:7699:b0:9a2:1e03:1573 with SMTP id o25-20020a170906769900b009a21e031573mr2187838ejm.65.1692917720332; Thu, 24 Aug 2023 15:55:20 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:20 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 03/15] io_uring: simplify big_cqe handling Date: Thu, 24 Aug 2023 23:53:25 +0100 Message-ID: <447aa1b2968978c99e655ba88db536e903df0fe9.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Don't keep big_cqe bits of req in a union with hash_node, find a separate space for it. It's bit safer, but also if we keep it always initialised, we can get rid of ugly REQ_F_CQE32_INIT handling. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 16 ++++++---------- io_uring/io_uring.c | 8 +++----- io_uring/io_uring.h | 15 +++------------ io_uring/uring_cmd.c | 5 ++--- 4 files changed, 14 insertions(+), 30 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index f04ce513fadb..9795eda529f7 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -409,7 +409,6 @@ enum { REQ_F_SINGLE_POLL_BIT, REQ_F_DOUBLE_POLL_BIT, REQ_F_PARTIAL_IO_BIT, - REQ_F_CQE32_INIT_BIT, REQ_F_APOLL_MULTISHOT_BIT, REQ_F_CLEAR_POLLIN_BIT, REQ_F_HASH_LOCKED_BIT, @@ -479,8 +478,6 @@ enum { REQ_F_PARTIAL_IO = BIT(REQ_F_PARTIAL_IO_BIT), /* fast poll multishot mode */ REQ_F_APOLL_MULTISHOT = BIT(REQ_F_APOLL_MULTISHOT_BIT), - /* ->extra1 and ->extra2 are initialised */ - REQ_F_CQE32_INIT = BIT(REQ_F_CQE32_INIT_BIT), /* recvmsg special flag, clear EPOLLIN */ REQ_F_CLEAR_POLLIN = BIT(REQ_F_CLEAR_POLLIN_BIT), /* hashed into ->cancel_hash_locked, protected by ->uring_lock */ @@ -579,13 +576,7 @@ struct io_kiocb { struct io_task_work io_task_work; unsigned nr_tw; /* for polled requests, i.e. IORING_OP_POLL_ADD and async armed poll */ - union { - struct hlist_node hash_node; - struct { - u64 extra1; - u64 extra2; - }; - }; + struct hlist_node hash_node; /* internal polling, see IORING_FEAT_FAST_POLL */ struct async_poll *apoll; /* opcode allocated if it needs to store data for async defer */ @@ -595,6 +586,11 @@ struct io_kiocb { /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; struct io_wq_work work; + + struct { + u64 extra1; + u64 extra2; + } big_cqe; }; struct io_overflow_cqe { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3e0fe1ebbc10..0aeb33256a6d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -807,13 +807,10 @@ static bool io_cqring_event_overflow(struct io_ring_ctx *ctx, u64 user_data, void io_req_cqe_overflow(struct io_kiocb *req) { - if (!(req->flags & REQ_F_CQE32_INIT)) { - req->extra1 = 0; - req->extra2 = 0; - } io_cqring_event_overflow(req->ctx, req->cqe.user_data, req->cqe.res, req->cqe.flags, - req->extra1, req->extra2); + req->big_cqe.extra1, req->big_cqe.extra2); + memset(&req->big_cqe, 0, sizeof(req->big_cqe)); } /* @@ -1057,6 +1054,7 @@ static void io_preinit_req(struct io_kiocb *req, struct io_ring_ctx *ctx) req->async_data = NULL; /* not necessary, but safer to zero */ memset(&req->cqe, 0, sizeof(req->cqe)); + memset(&req->big_cqe, 0, sizeof(req->big_cqe)); } static void io_flush_cached_locked_reqs(struct io_ring_ctx *ctx, diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 465598223386..9b5dfb6ef484 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -148,21 +148,12 @@ static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, struct io_kiocb *req if (trace_io_uring_complete_enabled()) trace_io_uring_complete(req->ctx, req, req->cqe.user_data, req->cqe.res, req->cqe.flags, - (req->flags & REQ_F_CQE32_INIT) ? req->extra1 : 0, - (req->flags & REQ_F_CQE32_INIT) ? req->extra2 : 0); + req->big_cqe.extra1, req->big_cqe.extra2); memcpy(cqe, &req->cqe, sizeof(*cqe)); - if (ctx->flags & IORING_SETUP_CQE32) { - u64 extra1 = 0, extra2 = 0; - - if (req->flags & REQ_F_CQE32_INIT) { - extra1 = req->extra1; - extra2 = req->extra2; - } - - WRITE_ONCE(cqe->big_cqe[0], extra1); - WRITE_ONCE(cqe->big_cqe[1], extra2); + memcpy(cqe->big_cqe, &req->big_cqe, sizeof(*cqe)); + memset(&req->big_cqe, 0, sizeof(req->big_cqe)); } return true; } diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 8e7a03c1b20e..537795fddc87 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -43,9 +43,8 @@ EXPORT_SYMBOL_GPL(io_uring_cmd_do_in_task_lazy); static inline void io_req_set_cqe32_extra(struct io_kiocb *req, u64 extra1, u64 extra2) { - req->extra1 = extra1; - req->extra2 = extra2; - req->flags |= REQ_F_CQE32_INIT; + req->big_cqe.extra1 = extra1; + req->big_cqe.extra2 = extra2; } /* From patchwork Thu Aug 24 22:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C8FBC88CB9 for ; Thu, 24 Aug 2023 22:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243938AbjHXWzn (ORCPT ); Thu, 24 Aug 2023 18:55:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244023AbjHXWzc (ORCPT ); Thu, 24 Aug 2023 18:55:32 -0400 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E9B21FC3 for ; Thu, 24 Aug 2023 15:55:23 -0700 (PDT) Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2bbbda48904so4370761fa.2 for ; Thu, 24 Aug 2023 15:55:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917721; x=1693522521; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+MCzLayGP6DIzHUSXyvUW9IwK0Due1vKOjR85NCwaD0=; b=LmuKzJRuH0jR39EmS763qyYUd/xwqNgMEJhXIjFnrTXUR3rbhqBCg/qz4jIZuXPLP4 4L+sE5u3IHiYZBMcMP33cTSNke2hSTYtoX1KQvphVUIt//2EleqQqwpRsQBxYPV+rCMf czwQJhuZt9TLwRbnsM705I6bwmJGltGIQ64NbgzwkC9RijRr9uqmE3LGJh4nxLiGd7FL 0qOzulwfme/wqIGo5vpRYvv2juLGW1w1fW/MSNguM31GaQqPros6bMhUVB928zUp7heM a6rfAjIUQWb10RE7P5rSpMbAEdG4LrGerOsXtoeUKs1IoxFS3SFw0Tz1PhwXP25Zlini FTgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917721; x=1693522521; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+MCzLayGP6DIzHUSXyvUW9IwK0Due1vKOjR85NCwaD0=; b=Shd0AlOp9hFBno9FGVbNUP2kKfUHVrXOSRlNDAKoCiSth7w6XWQzsQ4Jc1dZ18lZNn D2RBsTTb7+jZLAiL9lMTfmap5kqsR1m1CquZKDYY3uhKingtsM/hWhUByJkqOxGUDqr/ a0wO8/pYX3w612LKcZ2D/E2FtdKdmsb/SyzpgH3TsCT7/rIvqmxNJ9zSEVvebbyB0ZV6 tSfnEPQ/1YjV+4d3pqjqYVX/JBLjuKZbOOLCu3VGhzVhpuEmxD5Yl4cCIHuPoII1dFO9 M6kjSfKxMjMNzjNum4KGDpoWk7/XDh4jadPwU0fRySrP1C5Ol4AUJxa/bkaQNdZ42sK3 OQpw== X-Gm-Message-State: AOJu0YwhNiuKxdlvc0uHEg6dpj2KXn2VRehyMz2NaQQrbyPoc2nlMZ51 qQXt+V8T1NfXH4dUxMiGV4sJJi7LjsY= X-Google-Smtp-Source: AGHT+IHWqPkpais3IjixT599RTDAtOm+7gSLec2Bfx7kIzoNevzr1ZmJxS7Dxx+lSHNJG4A8u/UwYQ== X-Received: by 2002:a2e:87d6:0:b0:2bc:bece:2264 with SMTP id v22-20020a2e87d6000000b002bcbece2264mr9864294ljj.50.1692917721205; Thu, 24 Aug 2023 15:55:21 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:20 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 04/15] io_uring: refactor __io_get_cqe() Date: Thu, 24 Aug 2023 23:53:26 +0100 Message-ID: <74dc8fdf2657e438b2e05e1d478a3596924604e9.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Make __io_get_cqe simpler by not grabbing the cqe from refilled cached, but letting io_get_cqe() do it for us. That's cleaner and removes some duplication. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 13 ++++--------- io_uring/io_uring.h | 23 ++++++++++++----------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0aeb33256a6d..de05831eeca7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -818,7 +818,7 @@ void io_req_cqe_overflow(struct io_kiocb *req) * control dependency is enough as we're using WRITE_ONCE to * fill the cq entry */ -struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow) +bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow) { struct io_rings *rings = ctx->rings; unsigned int off = ctx->cached_cq_tail & (ctx->cq_entries - 1); @@ -830,7 +830,7 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow) * Force overflow the completion. */ if (!overflow && (ctx->check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))) - return NULL; + return false; /* userspace may cheat modifying the tail, be safe and do min */ queued = min(__io_cqring_events(ctx), ctx->cq_entries); @@ -838,7 +838,7 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow) /* we need a contiguous range, limit based on the current array offset */ len = min(free, ctx->cq_entries - off); if (!len) - return NULL; + return false; if (ctx->flags & IORING_SETUP_CQE32) { off <<= 1; @@ -847,12 +847,7 @@ struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow) ctx->cqe_cached = &rings->cqes[off]; ctx->cqe_sentinel = ctx->cqe_cached + len; - - ctx->cached_cq_tail++; - ctx->cqe_cached++; - if (ctx->flags & IORING_SETUP_CQE32) - ctx->cqe_cached++; - return &rings->cqes[off]; + return true; } static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9b5dfb6ef484..9c80d20fe18f 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -38,7 +38,7 @@ enum { IOU_STOP_MULTISHOT = -ECANCELED, }; -struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx, bool overflow); +bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow); void io_req_cqe_overflow(struct io_kiocb *req); int io_run_task_work_sig(struct io_ring_ctx *ctx); void io_req_defer_failed(struct io_kiocb *req, s32 res); @@ -112,19 +112,20 @@ static inline void io_req_task_work_add(struct io_kiocb *req) static inline struct io_uring_cqe *io_get_cqe_overflow(struct io_ring_ctx *ctx, bool overflow) { - io_lockdep_assert_cq_locked(ctx); + struct io_uring_cqe *cqe; - if (likely(ctx->cqe_cached < ctx->cqe_sentinel)) { - struct io_uring_cqe *cqe = ctx->cqe_cached; + io_lockdep_assert_cq_locked(ctx); - ctx->cached_cq_tail++; - ctx->cqe_cached++; - if (ctx->flags & IORING_SETUP_CQE32) - ctx->cqe_cached++; - return cqe; + if (unlikely(ctx->cqe_cached >= ctx->cqe_sentinel)) { + if (unlikely(!io_cqe_cache_refill(ctx, overflow))) + return NULL; } - - return __io_get_cqe(ctx, overflow); + cqe = ctx->cqe_cached; + ctx->cached_cq_tail++; + ctx->cqe_cached++; + if (ctx->flags & IORING_SETUP_CQE32) + ctx->cqe_cached++; + return cqe; } static inline struct io_uring_cqe *io_get_cqe(struct io_ring_ctx *ctx) From patchwork Thu Aug 24 22:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7B28EE49AB for ; Thu, 24 Aug 2023 22:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243968AbjHXWzo (ORCPT ); Thu, 24 Aug 2023 18:55:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244024AbjHXWzc (ORCPT ); Thu, 24 Aug 2023 18:55:32 -0400 Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 358E41FC4 for ; Thu, 24 Aug 2023 15:55:24 -0700 (PDT) Received: by mail-ed1-x536.google.com with SMTP id 4fb4d7f45d1cf-529fb2c6583so537149a12.1 for ; Thu, 24 Aug 2023 15:55:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917722; x=1693522522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ofRW9/wUGWpYyVONjUTn3Q1b6uINI4jGUtdfvGoE5vc=; b=nYLiIhJNECLHJxPonMWN3yRXfIQ8q1wr5vnivlZX0+pG2trL8SScuet9zBKObhPLxp l6S4DjWUYBtInX9Ym73tQRJtSPNCdyxBL+ouAy/L5RqzbDFwddiHyagioqO4BoeZaOO6 FFC/SodjAhk2PQnIaxepoMZa49Jru7XE5HdWP4EHA6p8VTHkf8F8XlpAqm416boCkMiu 2Jic65/+olut6dPBLSSuEXGVtErH3Z4RpCWIZGaS+GWjfAm3EDPaxyBEMqY4EriFpI1n ovkpP5T1ecYP2HYxvBd9tF2smEw+JUKLjIhd5cndOruY7ejbx/6dR6eqwOqhtuVBppPA AT5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917722; x=1693522522; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ofRW9/wUGWpYyVONjUTn3Q1b6uINI4jGUtdfvGoE5vc=; b=FysfPNJxPWwh6Lylmgh6Nu9eQ9BmLctsiceH8gKFFTA3zdKENLUCeajez6TZcmNesa rAnTxIF/g2Yv8fEbC2LwZAL5nomm8Xo1h6A0eRVIy1rtY4dYy4/UYKyVP9DEeMXYfbnK DcGwKZxp7WSbuWc2UYNuObHgCxbsOkTatvKVyjLEU/T2W3gezK6cRs8o1P+ATz/VXsi8 +mdZ5BNeNcucqwJDp12QIttOZ+oizZUsww//WWqi/whiXwnPpQN87/1YeLaOddM38Fw+ Lxth0WSr1XyGUcRxhqpx/q67dp7PPERp5raTZg37dFY46uYxTwxpyisvfiIWVfsugFfj ClWw== X-Gm-Message-State: AOJu0Yyfsga6aj4ccyK1iyNbRM9qJ2LIlowJpdya7qaNOimhDvlb+jnQ +xsXFmH0nC4Fxwg7jdZD+i2e3+khuFE= X-Google-Smtp-Source: AGHT+IGfNYk06nQxsak/4bCdpxun/PbHqZNSohSW5GzQ5+T18uoql8jMsBNrAaFvjh00VRwSw+qCxQ== X-Received: by 2002:a17:906:105c:b0:9a1:72f9:49fe with SMTP id j28-20020a170906105c00b009a172f949femr13780203ejj.65.1692917722274; Thu, 24 Aug 2023 15:55:22 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:21 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 05/15] io_uring: optimise extra io_get_cqe null check Date: Thu, 24 Aug 2023 23:53:27 +0100 Message-ID: <322ea4d3377d3d4efd8ae90ab8ed28a99f518210.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org If the cached cqe check passes in io_get_cqe*() it already means that the cqe we return is valid and non-zero, however the compiler is unable to optimise null checks like in io_fill_cqe_req(). Do a bit of trickery, return success/fail boolean from io_get_cqe*() and store cqe in the cqe parameter. That makes it do the right thing, erasing the check together with the introduced indirection. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 7 +++---- io_uring/io_uring.h | 20 +++++++++----------- 2 files changed, 12 insertions(+), 15 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index de05831eeca7..cfc2dc8c4b2f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -683,10 +683,10 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx) io_cq_lock(ctx); while (!list_empty(&ctx->cq_overflow_list)) { - struct io_uring_cqe *cqe = io_get_cqe_overflow(ctx, true); + struct io_uring_cqe *cqe; struct io_overflow_cqe *ocqe; - if (!cqe) + if (!io_get_cqe_overflow(ctx, &cqe, true)) break; ocqe = list_first_entry(&ctx->cq_overflow_list, struct io_overflow_cqe, list); @@ -862,8 +862,7 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res, * submission (by quite a lot). Increment the overflow count in * the ring. */ - cqe = io_get_cqe(ctx); - if (likely(cqe)) { + if (likely(io_get_cqe(ctx, &cqe))) { trace_io_uring_complete(ctx, NULL, user_data, res, cflags, 0, 0); WRITE_ONCE(cqe->user_data, user_data); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9c80d20fe18f..2960e35b32a5 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -109,28 +109,27 @@ static inline void io_req_task_work_add(struct io_kiocb *req) #define io_for_each_link(pos, head) \ for (pos = (head); pos; pos = pos->link) -static inline struct io_uring_cqe *io_get_cqe_overflow(struct io_ring_ctx *ctx, - bool overflow) +static inline bool io_get_cqe_overflow(struct io_ring_ctx *ctx, + struct io_uring_cqe **ret, + bool overflow) { - struct io_uring_cqe *cqe; - io_lockdep_assert_cq_locked(ctx); if (unlikely(ctx->cqe_cached >= ctx->cqe_sentinel)) { if (unlikely(!io_cqe_cache_refill(ctx, overflow))) - return NULL; + return false; } - cqe = ctx->cqe_cached; + *ret = ctx->cqe_cached; ctx->cached_cq_tail++; ctx->cqe_cached++; if (ctx->flags & IORING_SETUP_CQE32) ctx->cqe_cached++; - return cqe; + return true; } -static inline struct io_uring_cqe *io_get_cqe(struct io_ring_ctx *ctx) +static inline bool io_get_cqe(struct io_ring_ctx *ctx, struct io_uring_cqe **ret) { - return io_get_cqe_overflow(ctx, false); + return io_get_cqe_overflow(ctx, ret, false); } static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, struct io_kiocb *req) @@ -142,8 +141,7 @@ static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, struct io_kiocb *req * submission (by quite a lot). Increment the overflow count in * the ring. */ - cqe = io_get_cqe(ctx); - if (unlikely(!cqe)) + if (unlikely(!io_get_cqe(ctx, &cqe))) return false; if (trace_io_uring_complete_enabled()) From patchwork Thu Aug 24 22:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1F7CEE49A5 for ; Thu, 24 Aug 2023 22:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243960AbjHXWzn (ORCPT ); Thu, 24 Aug 2023 18:55:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244025AbjHXWzc (ORCPT ); Thu, 24 Aug 2023 18:55:32 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDB651FC7 for ; Thu, 24 Aug 2023 15:55:24 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id a640c23a62f3a-99bf1f632b8so35645066b.1 for ; Thu, 24 Aug 2023 15:55:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917723; x=1693522523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fWUAJWGai+zIRk9xtvb8L4N1KU3DAFreaKDKoivVNNU=; b=btNGfqOVCCzNcZwXnupcouLsdIkJvt+0UPTCqGEh/MQ7bskJj7h5IPXPa/k2IZGTES 3HR2C2sb5QmYxiE29jrjmTIop4D6GmOMV32ckQeHHb2TSZ/k42BFCqOjydF1qQfnw1Vr FvXEZbxMAW3npBu4JrVYEpqb5gW72MEEqoVpn8HpGBoHgSRgjoLZj8uegseKb8BHg6l+ 235TNnoCQKJ11vXsd0uXe3XTQ737ga66cIKbH+ObTO1pB1v2ic7BN6gdFAWyiavRjXC1 P44XQRb7K2YyHkitdTADUACVhVUnLqeAmK42atVqZ1d8AR6AxRVeSOvQf+EmhRtYmWDY ZjWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917723; x=1693522523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fWUAJWGai+zIRk9xtvb8L4N1KU3DAFreaKDKoivVNNU=; b=GYGecxcxjKImrIl1KpXeUTqddG678h5cPWAeGDuqUZ7AdQ4L/uPGpqK8P1rr2yUO50 ZreLzY988Uj179r8vv+kx9l1H+UQVPgyR4JoFgIdE3P7jsY+EbVU/mEhsns0nw7ifvsk 54qmAWL6pciOH3tFXTmaNB9YxECJ7e/BvGzZO6vfxs6IlugqzDjd1GLkjTWNnR8E8nqL pYIo3H/eeIC7v6rfROuRjZg5VvvZgZyVs6BF1n+W28gGZDuRVlQejDttq5VagDwf69gM J2u0m8MO3ybkIk5u0JUp63+pADkCLreTQfuGw6Ei6aAjF+a8RO7My/fLAuQp13IwUZR0 BoSQ== X-Gm-Message-State: AOJu0YwGC2JSwOJzjLAHKBsDjVhWNRUSQLwFSlSYgGdUOUFpEz6FwYP8 QiINXkxNggOZtac38YOw+mQIrJ2lNEU= X-Google-Smtp-Source: AGHT+IEHNSH6N2d7VMr9KcbODujE42TOKx3FHGdm0i88RdQ+SKwIdR56ewfbvwBmwTQqzFOH2iQN0A== X-Received: by 2002:a17:906:2011:b0:99c:56d1:7c71 with SMTP id 17-20020a170906201100b0099c56d17c71mr13242744ejo.26.1692917723132; Thu, 24 Aug 2023 15:55:23 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:22 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 06/15] io_uring: reorder cqring_flush and wakeups Date: Thu, 24 Aug 2023 23:53:28 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Unlike in the past, io_commit_cqring_flush() doesn't do anything that may need io_cqring_wake() to be issued after, all requests it completes will go via task_work. Do io_commit_cqring_flush() after io_cqring_wake() to clean up __io_cq_unlock_post(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 14 +++----------- io_uring/rw.c | 2 +- 2 files changed, 4 insertions(+), 12 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cfc2dc8c4b2f..7c1ef5b6628d 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -629,19 +629,11 @@ static inline void io_cq_lock(struct io_ring_ctx *ctx) static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) { io_commit_cqring(ctx); - - if (ctx->task_complete) { - /* - * ->task_complete implies that only current might be waiting - * for CQEs, and obviously, we currently don't. No one is - * waiting, wakeups are futile, skip them. - */ - io_commit_cqring_flush(ctx); - } else { + if (!ctx->task_complete) { spin_unlock(&ctx->completion_lock); - io_commit_cqring_flush(ctx); io_cqring_wake(ctx); } + io_commit_cqring_flush(ctx); } static void io_cq_unlock_post(struct io_ring_ctx *ctx) @@ -649,8 +641,8 @@ static void io_cq_unlock_post(struct io_ring_ctx *ctx) { io_commit_cqring(ctx); spin_unlock(&ctx->completion_lock); - io_commit_cqring_flush(ctx); io_cqring_wake(ctx); + io_commit_cqring_flush(ctx); } /* Returns true if there are no backlogged entries after the flush */ diff --git a/io_uring/rw.c b/io_uring/rw.c index 9b51afdae505..20140d3505f1 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -985,9 +985,9 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) static void io_cqring_ev_posted_iopoll(struct io_ring_ctx *ctx) { - io_commit_cqring_flush(ctx); if (ctx->flags & IORING_SETUP_SQPOLL) io_cqring_wake(ctx); + io_commit_cqring_flush(ctx); } void io_rw_fail(struct io_kiocb *req) From patchwork Thu Aug 24 22:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364846 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24140EE49AD for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243973AbjHXWzp (ORCPT ); Thu, 24 Aug 2023 18:55:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244033AbjHXWzf (ORCPT ); Thu, 24 Aug 2023 18:55:35 -0400 Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1816A1FD3 for ; Thu, 24 Aug 2023 15:55:26 -0700 (PDT) Received: by mail-ej1-x62a.google.com with SMTP id a640c23a62f3a-99c1f6f3884so35181466b.0 for ; Thu, 24 Aug 2023 15:55:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917724; x=1693522524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IgZ/67ABnBamuCv5GTpb6bProem6msgVgR9jIq/TUv4=; b=sxd7AiU7ZPL+TShbFg4l+3e5oI8hDkEplvoAoeVGIHI9MEBRGHYApHJ62FhY2SdHWB nlc6BiyiP/0dA9c0n66KWsh3f7kMih5NcpUMhkjUB+OSnxnMkmLw0f/rpOSoLyYVlKfr B9CU2dG0CiBewLh3DJ16g3lEjbsJ1nzsNpadifUjQYYeP9xWDV3W9G+66qbL3Ir/iRLv //+gCWPuXoUdDCJrbJ5XUJokrnnWXJy0obteTsRYyjh6rDHDrwRGHCokZORCX7zSq41n mSekAS5iuf97zDHucDfTCsPrb2Nfl5JK7KwWrLbsVq6rvVT3AJwvxnaV4/IJGxhwzqc4 WVgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917724; x=1693522524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IgZ/67ABnBamuCv5GTpb6bProem6msgVgR9jIq/TUv4=; b=GBLlo5zpUW9TU27DfuHFokgqMTHpfvWpA0BmrTkkeSuUSQLMG+H6pEJ5vTzJ9GA3J7 +xR7Y7aG9f1fMFtjjvfn6jALNMgyc8pwbRSxgpDeJNVV2XCI6VwxObXhqGQZSLDZrgWY NBPxletn95BuouXnK7G1s87xBOkikEj/X+DqXX+lRQtiqCIBcexbidJep6qB83ZUYgfz cZC7HieyT3eXJpDHGNsNrVIOkv7h+k9whn7IxRZuZyJT/Vt4iaMCp+JjPjTkdB8n0VNx c4ZxGYhHRdQ5K1UvclJJ4m5UW919Coc3koiDcnBe5OSgOh3ve4IOgeIY9pyfmPfgu9zc Kd4g== X-Gm-Message-State: AOJu0Yy5u+OCkiqI1sQURZcOzVbVLzA8QyD1uinmYNeVNrVuRd1ibwg7 RsbOargldM3+EewNQ1lFMytEfdoFJKo= X-Google-Smtp-Source: AGHT+IGvLUKG9Lt4y8d51sYJ2LOtZWu8iqG5+NORwXkd3AFDS/kFQ65EymKi1P5M0Hg29B8zb8djeA== X-Received: by 2002:a17:906:6b92:b0:969:93f2:259a with SMTP id l18-20020a1709066b9200b0096993f2259amr12132165ejr.73.1692917723924; Thu, 24 Aug 2023 15:55:23 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 07/15] io_uring: merge iopoll and normal completion paths Date: Thu, 24 Aug 2023 23:53:29 +0100 Message-ID: <3840473f5e8a960de35b77292026691880f6bdbc.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org io_do_iopoll() and io_submit_flush_completions() are pretty similar, both filling CQEs and then free a list of requests. Don't duplicate it and make iopoll use __io_submit_flush_completions(), which also helps with inlining and other optimisations. For that, we need to first find all completed iopoll requests and splice them from the iopoll list and then pass it down. This adds one extra list traversal, which should be fine as requests will stay hot in cache. CQ locking is already conditional, introduce ->lockless_cq and skip locking for IOPOLL as it's protected by ->uring_lock. We also add a wakeup optimisation for IOPOLL to __io_cq_unlock_post(), so it works just like io_cqring_ev_posted_iopoll(). Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 18 ++++++++++++------ io_uring/io_uring.h | 2 +- io_uring/rw.c | 24 +++++------------------- 4 files changed, 19 insertions(+), 26 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9795eda529f7..c0c03d8059df 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -205,6 +205,7 @@ struct io_ring_ctx { unsigned int has_evfd: 1; /* all CQEs should be posted only by the submitter task */ unsigned int task_complete: 1; + unsigned int lockless_cq: 1; unsigned int syscall_iopoll: 1; unsigned int poll_activated: 1; unsigned int drain_disabled: 1; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 7c1ef5b6628d..e8321903e3f3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -147,7 +147,6 @@ static bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, bool cancel_all); static void io_queue_sqe(struct io_kiocb *req); -static void __io_submit_flush_completions(struct io_ring_ctx *ctx); struct kmem_cache *req_cachep; @@ -616,7 +615,7 @@ void __io_commit_cqring_flush(struct io_ring_ctx *ctx) static inline void __io_cq_lock(struct io_ring_ctx *ctx) { - if (!ctx->task_complete) + if (!ctx->lockless_cq) spin_lock(&ctx->completion_lock); } @@ -630,8 +629,11 @@ static inline void __io_cq_unlock_post(struct io_ring_ctx *ctx) { io_commit_cqring(ctx); if (!ctx->task_complete) { - spin_unlock(&ctx->completion_lock); - io_cqring_wake(ctx); + if (!ctx->lockless_cq) + spin_unlock(&ctx->completion_lock); + /* IOPOLL rings only need to wake up if it's also SQPOLL */ + if (!ctx->syscall_iopoll) + io_cqring_wake(ctx); } io_commit_cqring_flush(ctx); } @@ -1485,7 +1487,8 @@ void io_queue_next(struct io_kiocb *req) io_req_task_queue(nxt); } -void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node) +static void io_free_batch_list(struct io_ring_ctx *ctx, + struct io_wq_work_node *node) __must_hold(&ctx->uring_lock) { do { @@ -1522,7 +1525,7 @@ void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node) } while (node); } -static void __io_submit_flush_completions(struct io_ring_ctx *ctx) +void __io_submit_flush_completions(struct io_ring_ctx *ctx) __must_hold(&ctx->uring_lock) { struct io_submit_state *state = &ctx->submit_state; @@ -3836,6 +3839,9 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, !(ctx->flags & IORING_SETUP_SQPOLL)) ctx->task_complete = true; + if (ctx->task_complete || (ctx->flags & IORING_SETUP_IOPOLL)) + ctx->lockless_cq = true; + /* * lazy poll_wq activation relies on ->task_complete for synchronisation * purposes, see io_activate_pollwq() diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 2960e35b32a5..07fd185064d2 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -72,7 +72,7 @@ int io_ring_add_registered_file(struct io_uring_task *tctx, struct file *file, int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts); int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr); int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin); -void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node); +void __io_submit_flush_completions(struct io_ring_ctx *ctx); int io_req_prep_async(struct io_kiocb *req); struct io_wq_work *io_wq_free_work(struct io_wq_work *work); diff --git a/io_uring/rw.c b/io_uring/rw.c index 20140d3505f1..0a1e515f0510 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -983,13 +983,6 @@ int io_write(struct io_kiocb *req, unsigned int issue_flags) return ret; } -static void io_cqring_ev_posted_iopoll(struct io_ring_ctx *ctx) -{ - if (ctx->flags & IORING_SETUP_SQPOLL) - io_cqring_wake(ctx); - io_commit_cqring_flush(ctx); -} - void io_rw_fail(struct io_kiocb *req) { int res; @@ -1060,24 +1053,17 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin) if (!smp_load_acquire(&req->iopoll_completed)) break; nr_events++; - if (unlikely(req->flags & REQ_F_CQE_SKIP)) - continue; - req->cqe.flags = io_put_kbuf(req, 0); - if (unlikely(!io_fill_cqe_req(ctx, req))) { - spin_lock(&ctx->completion_lock); - io_req_cqe_overflow(req); - spin_unlock(&ctx->completion_lock); - } } - if (unlikely(!nr_events)) return 0; - io_commit_cqring(ctx); - io_cqring_ev_posted_iopoll(ctx); pos = start ? start->next : ctx->iopoll_list.first; wq_list_cut(&ctx->iopoll_list, prev, start); - io_free_batch_list(ctx, pos); + + if (WARN_ON_ONCE(!wq_list_empty(&ctx->submit_state.compl_reqs))) + return 0; + ctx->submit_state.compl_reqs.first = pos; + __io_submit_flush_completions(ctx); return nr_events; } From patchwork Thu Aug 24 22:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 355B7EE49B2 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243972AbjHXWzo (ORCPT ); Thu, 24 Aug 2023 18:55:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244036AbjHXWzf (ORCPT ); Thu, 24 Aug 2023 18:55:35 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60D731FD4 for ; Thu, 24 Aug 2023 15:55:26 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-99c93638322so82674766b.1 for ; Thu, 24 Aug 2023 15:55:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917724; x=1693522524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f2DEBEI1Rhro87I5c331VMzG9L/kWUItjuQZp5E+Z+g=; b=C6EbkJzQNTy6eC066kxnL3PBgnsbdfomW6X+FCk8KxRaEPS+IA0XdJrwsINO2Vel9O n47N2lDoRo44pJBrbV3nrMJq3Mg/aVLrQqUirYNQFyXA2pkNZVUiiVlBAIubB0/5jq0O tsnkJqTENyoMepfC877D/NrTHU6IrwAPJwIr/1Rxh8raQozIhNf6VsDJ4cnvJChvLgNL Ym+DxuUxP4btP3SaA1z9uxM3o9Xr5tUiROL5mlCxiYgOndekdHHg1m6jBH4TvcDSmIoh Yrwh7eKfJ7grTC3eGSgp99KRz7Byb8fxDfymgcygNsDqgeYtPkVzibbqhkHKRq+xVXMa HCyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917724; x=1693522524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f2DEBEI1Rhro87I5c331VMzG9L/kWUItjuQZp5E+Z+g=; b=WF9rXm9Uxq2iY5N0YGoCCrpwLkHcPcjoSOiAdVe2MH6qtiwYcpkmPwK3B7ZzXatExm 111WxGxvVeetx3V5rXdQM6cAnqaRUGrpKZSkxb1GTDVp+9MSuo5d3AgxjD0pVXHB5sRw DITD8YokFbIfQwvVNW8BFy1cUbAiQUwEMqc2qEZlGAkk4SU8Xxhmh7ZTQivEs3sfhctT Zlf5tOByZw4J5qmGyo8fSAfWMmMhyEILrlCsmoGVhipSmFals/RM42IODcz7YtnhNuyh j77IDbb1iI+9YHQu1APuAuK8+iJpV14oYA78prCa8dqCe0PHJPk5M6lWF1RWyK3Arabf 8SmA== X-Gm-Message-State: AOJu0YzKYXLdKjuC/HC7Y6LfySXIJ+tmxJGuXdRBP15MADe5JI30/n2R DK1086RRdq8vgcRhyr+uqVG/J3njNNo= X-Google-Smtp-Source: AGHT+IGfrOnKQbGVXpbfIeoFCnyMsZrBur8vlDhm2MndWt3c5/JjJZdgA80aBUui9kyz3GijnqKLyA== X-Received: by 2002:a17:906:974a:b0:9a1:c69c:9388 with SMTP id o10-20020a170906974a00b009a1c69c9388mr8709018ejy.37.1692917724666; Thu, 24 Aug 2023 15:55:24 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:24 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 08/15] io_uring: force inline io_fill_cqe_req Date: Thu, 24 Aug 2023 23:53:30 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There are only 2 callers of io_fill_cqe_req left, and one of them is extremely hot. Force inline the function. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 07fd185064d2..547c30582fb8 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -132,7 +132,8 @@ static inline bool io_get_cqe(struct io_ring_ctx *ctx, struct io_uring_cqe **ret return io_get_cqe_overflow(ctx, ret, false); } -static inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, struct io_kiocb *req) +static __always_inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, + struct io_kiocb *req) { struct io_uring_cqe *cqe; From patchwork Thu Aug 24 22:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A5F8EE49B6 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243976AbjHXWzq (ORCPT ); Thu, 24 Aug 2023 18:55:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244037AbjHXWzf (ORCPT ); Thu, 24 Aug 2023 18:55:35 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 128F11FD0 for ; Thu, 24 Aug 2023 15:55:27 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id a640c23a62f3a-9936b3d0286so37412166b.0 for ; Thu, 24 Aug 2023 15:55:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917725; x=1693522525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LRwqn+8y3sMpl97JXgW1Kcuei5VeXjUdmpcF0r4gMS0=; b=DXyyYBC1SzcFWlLWIMc3z5yrf2YD+w6N5PB0/6pdMK2bAFSDSEzcDYS73L0MFyF9Zf MIWQuoFocHzrjT3DMJ9itefeYA657qBPDC6+zuPDQjEcw0Wv74xjfzy8Repge4yChXhp RQyfpD1L9bDlN7zg/vHMzMvM/gxCCx0s31Yuu8Cv9Z7R+UlhjG0q2pL0dWVofIDBZrTG YvjhxqlHNoWqOqAXJeCuPH9xlHm7GdmWCn7svctIwZyuZEBjMa4dcUi6HfsiG/RgdPif 76R/vxUwYGdBj0PnqIbnh/zQYnypaFBo/IQmOuef58rFTr1r7fP2a6Lt4NfbUzaT/qlK yEew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917725; x=1693522525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LRwqn+8y3sMpl97JXgW1Kcuei5VeXjUdmpcF0r4gMS0=; b=jfWfdikrXkFuHo9nRXiiwWUcZvhYNISBkVoxelu0k1SxUfxrLaK54s7DB/pkkYdFvF wKDOuLrWz62nlMtPWbgeMNejVbqHKrfl5PZB5BfQHQya8QUNTV3n2EC4Sbnp25oo50XX LXa5DNFG9Ll8YpiAIpIe9qJAlUeNe1XFsb9BRFEROo1r+jR9lsi6aprxMh32qLXiKWyK pXPSs/bR1frhXIfuVxWSqG8+YpbbqWtXT2Hi9vPNexLp/EUePlcWsP5bcoLWUnV0On32 Ja4RUHeEyfaFpDv78Y4P27MXwHT2QsiKJ0+lqscxTNUgMs6283HjB9sZnTlk30P4IGf0 Wfag== X-Gm-Message-State: AOJu0YyJ1CewVOiqnWlDkJahzifyuN10wmCHpL3Kc+x5RO8g/al72G0V 58VaIH97L8JESWESwZSdV0EcPAQ9Rok= X-Google-Smtp-Source: AGHT+IH+fbo+C1jxd5nu0ioFzFSHzY4NOlGa83eY2ZTy2xfozVwU2qwxVCn9GpiT97THswGXibmpUw== X-Received: by 2002:a17:906:7493:b0:9a1:ca55:b668 with SMTP id e19-20020a170906749300b009a1ca55b668mr6703370ejl.59.1692917725362; Thu, 24 Aug 2023 15:55:25 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 09/15] io_uring: compact SQ/CQ heads/tails Date: Thu, 24 Aug 2023 23:53:31 +0100 Message-ID: <9c8deddf9a7ed32069235a530d1e117fb460bc4c.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Queues heads and tails cache line aligned. That makes sq, cq taking 4 lines or 5 lines if we include the rest of struct io_rings (e.g. sq_flags is frequently accessed). Since modern io_uring is mostly single threaded, it doesn't make much send to spread them as such, it wastes space and puts additional pressure on caches. Put them all into a single line. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index c0c03d8059df..608a8e80e881 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -69,8 +69,8 @@ struct io_uring_task { }; struct io_uring { - u32 head ____cacheline_aligned_in_smp; - u32 tail ____cacheline_aligned_in_smp; + u32 head; + u32 tail; }; /* From patchwork Thu Aug 24 22:53:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 798CDEE49B5 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243975AbjHXWzp (ORCPT ); Thu, 24 Aug 2023 18:55:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46000 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244040AbjHXWzg (ORCPT ); Thu, 24 Aug 2023 18:55:36 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD4061FDF for ; Thu, 24 Aug 2023 15:55:27 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-9a2a4a5472dso110125166b.1 for ; Thu, 24 Aug 2023 15:55:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917726; x=1693522526; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YciofNNRWO0hDofU9tD/NJE19dwmkWDsnusL9jDimeY=; b=cKkXfWWv7Ecm+WNKx6aR2JBoXXIaqW/0aabtghpTyR7JTnZXSDM4s4z6445bFk1GK5 GuCBP2KhoXzlqaknyu0fhRbDsoSbf0n2LbvRWcIiFgxR0sHN9iQUUgt3TiGU3wj7Qsmr C+9OIDlFlmofcuUDXegUPPfPzWwE5doeuAm8cc8Vq7q1Ga1Jgb8LEsASloToTQHclots Hjxk/4U8UVt6e6I/VxEstgVyvR7Ow5OFOyK6LbDESFmJHyVMd9zwWWXsgx9dP01y53Mt b9W1CWSoCxywgs18s7HMmUsYKMM7d8wsZSzWotgOjvQShSLm7+ztvbeEmFByobxdTqI1 9TOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917726; x=1693522526; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YciofNNRWO0hDofU9tD/NJE19dwmkWDsnusL9jDimeY=; b=djUi+glBZHmzmI6Awbuneakxxam72hl2FB6HLeqab8h8feS6Cg0M+yjUpJt7N9ENdm HNNXUWntCxuSrq65oFmJW323UTLHCxcivr/vq35QLXAxADS9ott/nTWDAmRn3/vsUGOW KhK3WUFusRDXMQ1X3b0yGnPIRD7qHSdh4ryHs0Cv8LOF8yidOpPuH9pMSQTTd3aUayXp m9vRV9SiOTXECCW0tpWeDFqI5GJJcHxzvNh7zDS5XWBtToEAK++b6RWLbp8PYFx0RJih N8Ktza/oyQNxhgKlGy1oQrLh4L7Nhp1mEFyR6cAnb2Iuz/9ehEPOJZJbtQ0I0OcjlCrG RLAQ== X-Gm-Message-State: AOJu0Ywso6J48tJCCVGyByjqz49t0PM9dbKfXFvaahB3IhebVPSGIP+o 0y7FY6EvsutgHJtUz+ZFfXZYu0Q3RLg= X-Google-Smtp-Source: AGHT+IFCT6Mns4bDEm4FqcvDSE6yhPqweI01StpmmidW6TBXe4jeQQP+mfV5W427wbRBhmaifxCwUw== X-Received: by 2002:a17:906:748c:b0:9a1:f415:7c23 with SMTP id e12-20020a170906748c00b009a1f4157c23mr5413245ejl.26.1692917726064; Thu, 24 Aug 2023 15:55:26 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 10/15] io_uring: add option to remove SQ indirection Date: Thu, 24 Aug 2023 23:53:32 +0100 Message-ID: <0ffa3268a5ef61d326201ff43a233315c96312e0.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Not many aware, but io_uring submission queue has two levels. The first level usually appears as sq_array and stores indexes into the actual SQ. To my knowledge, no one has ever seriously used it, nor liburing exposes it to users. Add IORING_SETUP_NO_SQARRAY, when set we don't bother creating and using the sq_array and SQ heads/tails will be pointing directly into the SQ. Improves memory footprint, in term of both allocations as well as cache usage, and also should make io_get_sqe() less branchy in the end. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 5 ++++ io_uring/io_uring.c | 52 +++++++++++++++++++++-------------- 2 files changed, 37 insertions(+), 20 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 9fc7195f25df..8e61f8b7c2ce 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -185,6 +185,11 @@ enum { */ #define IORING_SETUP_REGISTERED_FD_ONLY (1U << 15) +/* + * Removes indirection through the SQ index array. + */ +#define IORING_SETUP_NO_SQARRAY (1U << 16) + enum io_uring_op { IORING_OP_NOP, IORING_OP_READV, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e8321903e3f3..a6eea3938802 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2339,8 +2339,21 @@ static void io_commit_sqring(struct io_ring_ctx *ctx) */ static bool io_get_sqe(struct io_ring_ctx *ctx, const struct io_uring_sqe **sqe) { - unsigned head, mask = ctx->sq_entries - 1; - unsigned sq_idx = ctx->cached_sq_head++ & mask; + unsigned mask = ctx->sq_entries - 1; + unsigned head = ctx->cached_sq_head++ & mask; + + if (!(ctx->flags & IORING_SETUP_NO_SQARRAY)) { + head = READ_ONCE(ctx->sq_array[head]); + if (unlikely(head >= ctx->sq_entries)) { + /* drop invalid entries */ + spin_lock(&ctx->completion_lock); + ctx->cq_extra--; + spin_unlock(&ctx->completion_lock); + WRITE_ONCE(ctx->rings->sq_dropped, + READ_ONCE(ctx->rings->sq_dropped) + 1); + return false; + } + } /* * The cached sq head (or cq tail) serves two purposes: @@ -2350,22 +2363,12 @@ static bool io_get_sqe(struct io_ring_ctx *ctx, const struct io_uring_sqe **sqe) * 2) allows the kernel side to track the head on its own, even * though the application is the one updating it. */ - head = READ_ONCE(ctx->sq_array[sq_idx]); - if (likely(head < ctx->sq_entries)) { - /* double index for 128-byte SQEs, twice as long */ - if (ctx->flags & IORING_SETUP_SQE128) - head <<= 1; - *sqe = &ctx->sq_sqes[head]; - return true; - } - /* drop invalid entries */ - spin_lock(&ctx->completion_lock); - ctx->cq_extra--; - spin_unlock(&ctx->completion_lock); - WRITE_ONCE(ctx->rings->sq_dropped, - READ_ONCE(ctx->rings->sq_dropped) + 1); - return false; + /* double index for 128-byte SQEs, twice as long */ + if (ctx->flags & IORING_SETUP_SQE128) + head <<= 1; + *sqe = &ctx->sq_sqes[head]; + return true; } int io_submit_sqes(struct io_ring_ctx *ctx, unsigned int nr) @@ -2734,6 +2737,12 @@ static unsigned long rings_size(struct io_ring_ctx *ctx, unsigned int sq_entries return SIZE_MAX; #endif + if (ctx->flags & IORING_SETUP_NO_SQARRAY) { + if (sq_offset) + *sq_offset = SIZE_MAX; + return off; + } + if (sq_offset) *sq_offset = off; @@ -3710,7 +3719,8 @@ static __cold int io_allocate_scq_urings(struct io_ring_ctx *ctx, return PTR_ERR(rings); ctx->rings = rings; - ctx->sq_array = (u32 *)((char *)rings + sq_array_offset); + if (!(ctx->flags & IORING_SETUP_NO_SQARRAY)) + ctx->sq_array = (u32 *)((char *)rings + sq_array_offset); rings->sq_ring_mask = p->sq_entries - 1; rings->cq_ring_mask = p->cq_entries - 1; rings->sq_ring_entries = p->sq_entries; @@ -3921,7 +3931,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, p->sq_off.ring_entries = offsetof(struct io_rings, sq_ring_entries); p->sq_off.flags = offsetof(struct io_rings, sq_flags); p->sq_off.dropped = offsetof(struct io_rings, sq_dropped); - p->sq_off.array = (char *)ctx->sq_array - (char *)ctx->rings; + if (!(ctx->flags & IORING_SETUP_NO_SQARRAY)) + p->sq_off.array = (char *)ctx->sq_array - (char *)ctx->rings; p->sq_off.resv1 = 0; if (!(ctx->flags & IORING_SETUP_NO_MMAP)) p->sq_off.user_addr = 0; @@ -4010,7 +4021,8 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) IORING_SETUP_COOP_TASKRUN | IORING_SETUP_TASKRUN_FLAG | IORING_SETUP_SQE128 | IORING_SETUP_CQE32 | IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN | - IORING_SETUP_NO_MMAP | IORING_SETUP_REGISTERED_FD_ONLY)) + IORING_SETUP_NO_MMAP | IORING_SETUP_REGISTERED_FD_ONLY | + IORING_SETUP_NO_SQARRAY)) return -EINVAL; return io_uring_create(entries, &p, params); From patchwork Thu Aug 24 22:53:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11D63EE49B0 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243974AbjHXWzp (ORCPT ); Thu, 24 Aug 2023 18:55:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244041AbjHXWzg (ORCPT ); Thu, 24 Aug 2023 18:55:36 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FA0C19A0 for ; Thu, 24 Aug 2023 15:55:28 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-9a2a4a5472dso110126566b.1 for ; Thu, 24 Aug 2023 15:55:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917727; x=1693522527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QfCDFTGPqPDquTSsf5QwXyrmIi5jQ5+Y+5MLL1cbVZg=; b=cRpzkHh4gekBnwGHCVv733wH7otA3KbEzC2tyRMzjnLOo8bzbrAElSFPZw9+YDZiB3 lJsTTYM1TGT2IY3DmiCDtoV0alwZEayhXBHL4McFnEnZwlSHyFv1L7FlCPcIfxyLuqhh 0p7c8AJ8fChKcNcKeaVhYJcQaQlNsrwf67cawm5S6/N7/ukJRLu1sQkU9/Fr/4542DDm 7gKhQnGG6/bPqS1gs3WsvVmXPMMUyhJVPqFoyElUmP3YIEwt2785HSU+LJkpJz/7LTJQ XS+KP3zefm5W/oMESPa7vE/Zr+SrwcL4Q1iPXpM4yjTOcYtw+OCttrcSIrQxcz9Mu2pV b/1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917727; x=1693522527; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QfCDFTGPqPDquTSsf5QwXyrmIi5jQ5+Y+5MLL1cbVZg=; b=eJ+UpfcNoOSIuFrxfZFVKQ0yRLHLHHOX7BZQ/W/nDPIiVJeunjrQ9Y5TPOKsmtzkWJ hBDS4PHBJiuuosJ4biaILYpvM9tDxcd8wOIz86rM5OI7irCzL8ASDU4zEEx9s0hK3MFL IweODwwWVTVPkDlIydin/wg3taFB0SaH7Wj11X8Whtcl80YZ/rhzntuhcU1Hm3VaIW68 w5rCWdYnYEh8moxWrxsmZeJiJl+toUW0TpLe3hL6gEoMumpEB6UfFfWeIaxG0tKXlJDA 9gLKwuNcsss4LCRwImGPt4ptcujW6MRqjCERudGYMhLM1Danl/6+2FMM93mFxBKLyLNI Vihg== X-Gm-Message-State: AOJu0YwWSqOVyernRUA068v/pKwunCHi+6RGtQiTPPBohvXbVh7V3hdD yEnCRZJm57baMLvnVaAHoHR+LKTEvik= X-Google-Smtp-Source: AGHT+IHnF6ehN/QYO4okmI57KzHIK4QRxOX4Gv4bdBLLzdOV/dynBpU7YmMD7WlI4CRx+rg2rJsL2g== X-Received: by 2002:a17:907:948e:b0:9a2:28e2:c347 with SMTP id dm14-20020a170907948e00b009a228e2c347mr1678646ejc.34.1692917726906; Thu, 24 Aug 2023 15:55:26 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:26 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 11/15] io_uring: move non aligned field to the end Date: Thu, 24 Aug 2023 23:53:33 +0100 Message-ID: <518e95d7888e9d481b2c5968dcf3f23db9ea47a5.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Move not cache aligned fields down in io_ring_ctx, should change anything, but makes further refactoring easier. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 36 +++++++++++++++++----------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 608a8e80e881..ad87d6074fb2 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -270,24 +270,6 @@ struct io_ring_ctx { struct io_alloc_cache netmsg_cache; } ____cacheline_aligned_in_smp; - /* IRQ completion list, under ->completion_lock */ - struct io_wq_work_list locked_free_list; - unsigned int locked_free_nr; - - const struct cred *sq_creds; /* cred used for __io_sq_thread() */ - struct io_sq_data *sq_data; /* if using sq thread polling */ - - struct wait_queue_head sqo_sq_wait; - struct list_head sqd_list; - - unsigned long check_cq; - - unsigned int file_alloc_start; - unsigned int file_alloc_end; - - struct xarray personalities; - u32 pers_next; - struct { /* * We cache a range of free CQEs we can use, once exhausted it @@ -332,6 +314,24 @@ struct io_ring_ctx { unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; + /* IRQ completion list, under ->completion_lock */ + struct io_wq_work_list locked_free_list; + unsigned int locked_free_nr; + + const struct cred *sq_creds; /* cred used for __io_sq_thread() */ + struct io_sq_data *sq_data; /* if using sq thread polling */ + + struct wait_queue_head sqo_sq_wait; + struct list_head sqd_list; + + unsigned long check_cq; + + unsigned int file_alloc_start; + unsigned int file_alloc_end; + + struct xarray personalities; + u32 pers_next; + /* Keep this last, we don't need it for the fast path */ struct wait_queue_head poll_wq; struct io_restriction restrictions; From patchwork Thu Aug 24 22:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8918BEE49B7 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243980AbjHXWzr (ORCPT ); Thu, 24 Aug 2023 18:55:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244045AbjHXWzh (ORCPT ); Thu, 24 Aug 2023 18:55:37 -0400 Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE8C71FD5 for ; Thu, 24 Aug 2023 15:55:29 -0700 (PDT) Received: by mail-lf1-x132.google.com with SMTP id 2adb3069b0e04-5007616b756so456966e87.3 for ; Thu, 24 Aug 2023 15:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917728; x=1693522528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8s21GtQ/YbH9JV5FC5+yZFO7uO9zKtUmbFucfHazoQo=; b=pLy1TmVLumYF+nyn1n/Fl1XwLRtDwHPXsv5LL163gWtzLXcXHq5Vre3/aTynPuxL2U Gc6/G59Xn2bVRb/FRoe2J2OCYoR6TXxpKaIUTh5JivX/vTNmAp+Azp15CgnJV0YJlo2P OXeZNs49GkL2Fvsu9OMg6rhIxzDW467SXX+HyjBVSAHVpQtAau3GyqJqWoEEkpIWweT7 6miX7BdI6L+TBxTu6rcnE6CTGRyDCyTlruOkICXmNKgGrevhaKlDlTrVbI6ltzMKReit Gim0MUZLS9c++Gf+7GBe0c03Bm16XjP3Uhbu8MucumZAmdror39PgCzz+6K1iKmC9pJ0 cLyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917728; x=1693522528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8s21GtQ/YbH9JV5FC5+yZFO7uO9zKtUmbFucfHazoQo=; b=a0RLzeCXzPK7MeTPDxY1/2h+pt8HAdRQs58SPbKYKCBsYWM7X7bEDaZHl5Y7IPvNGJ Daw2nz+JQBvF6QH6rTPTzZG0+sCS9wSxajlfZY2Md5CUTr0kAsFvcQgl+NYyAtphfkf9 yvsLze+tVmTxuMlM5I0owMy7KKs2WRsEwHRPQXHhorgFZ5mFMPaS+r3Gczptno/mMt6V bjwev6fHvCRvNLfzPaYkLgJ4EwS+mwQnWQzSZqwzVcjuZbUCXu7UNspCpLP1kNDoLqXz hUsdYYCS35lFMNXwE74tuvo9Q0xFOFEZsTIFiEFEBozE4FhUd2quMRFugqLw4qNgYesi AIOQ== X-Gm-Message-State: AOJu0YxvFDL+sG593/pRnRJc8eTzg4UBtbCCAfbkgBgOVLaiZqd0anL7 VLO2mUwQgrakSu5ubzrfAn4SKD1dIao= X-Google-Smtp-Source: AGHT+IGvRBcloQOORfMKiSYuaM4mwR4+MwGRV26L3QjRScSZ/0DUbC5MDxcFN+5Tka4fVjLeYIIYeQ== X-Received: by 2002:a05:6512:2354:b0:4ff:a8c6:d1aa with SMTP id p20-20020a056512235400b004ffa8c6d1aamr13547679lfu.48.1692917727651; Thu, 24 Aug 2023 15:55:27 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:27 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 12/15] io_uring: banish non-hot data to end of io_ring_ctx Date: Thu, 24 Aug 2023 23:53:34 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Let's move all slow path, setup/init and so on fields to the end of io_ring_ctx, that makes ctx reorganisation later easier. That includes, page arrays used only on tear down, CQ overflow list, old provided buffer caches and used by io-wq poll hashes. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 37 +++++++++++++++++----------------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index ad87d6074fb2..72e609752323 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -211,20 +211,11 @@ struct io_ring_ctx { unsigned int drain_disabled: 1; unsigned int compat: 1; - enum task_work_notify_mode notify_method; + struct task_struct *submitter_task; + struct io_rings *rings; + struct percpu_ref refs; - /* - * If IORING_SETUP_NO_MMAP is used, then the below holds - * the gup'ed pages for the two rings, and the sqes. - */ - unsigned short n_ring_pages; - unsigned short n_sqe_pages; - struct page **ring_pages; - struct page **sqe_pages; - - struct io_rings *rings; - struct task_struct *submitter_task; - struct percpu_ref refs; + enum task_work_notify_mode notify_method; } ____cacheline_aligned_in_smp; /* submission data */ @@ -262,10 +253,8 @@ struct io_ring_ctx { struct io_buffer_list *io_bl; struct xarray io_bl_xa; - struct list_head io_buffers_cache; struct io_hash_table cancel_table_locked; - struct list_head cq_overflow_list; struct io_alloc_cache apoll_cache; struct io_alloc_cache netmsg_cache; } ____cacheline_aligned_in_smp; @@ -298,11 +287,8 @@ struct io_ring_ctx { * manipulate the list, hence no extra locking is needed there. */ struct io_wq_work_list iopoll_list; - struct io_hash_table cancel_table; struct llist_head work_llist; - - struct list_head io_buffers_comp; } ____cacheline_aligned_in_smp; /* timeouts */ @@ -318,6 +304,10 @@ struct io_ring_ctx { struct io_wq_work_list locked_free_list; unsigned int locked_free_nr; + struct list_head io_buffers_comp; + struct list_head cq_overflow_list; + struct io_hash_table cancel_table; + const struct cred *sq_creds; /* cred used for __io_sq_thread() */ struct io_sq_data *sq_data; /* if using sq thread polling */ @@ -332,6 +322,8 @@ struct io_ring_ctx { struct xarray personalities; u32 pers_next; + struct list_head io_buffers_cache; + /* Keep this last, we don't need it for the fast path */ struct wait_queue_head poll_wq; struct io_restriction restrictions; @@ -375,6 +367,15 @@ struct io_ring_ctx { unsigned sq_thread_idle; /* protected by ->completion_lock */ unsigned evfd_last_cq_tail; + + /* + * If IORING_SETUP_NO_MMAP is used, then the below holds + * the gup'ed pages for the two rings, and the sqes. + */ + unsigned short n_ring_pages; + unsigned short n_sqe_pages; + struct page **ring_pages; + struct page **sqe_pages; }; struct io_tw_state { From patchwork Thu Aug 24 22:53:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364852 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BBA0EE49B9 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243979AbjHXWzq (ORCPT ); Thu, 24 Aug 2023 18:55:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244047AbjHXWzh (ORCPT ); Thu, 24 Aug 2023 18:55:37 -0400 Received: from mail-lf1-x131.google.com (mail-lf1-x131.google.com [IPv6:2a00:1450:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DA04B1FDA for ; Thu, 24 Aug 2023 15:55:30 -0700 (PDT) Received: by mail-lf1-x131.google.com with SMTP id 2adb3069b0e04-4fe21e7f3d1so463694e87.3 for ; Thu, 24 Aug 2023 15:55:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917729; x=1693522529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cvhsasKZJnva2wCI0Nz03kKnyZToCj12NVuapLyWr3E=; b=Qpaabl53TVsFlH4ubEcXggqJFVBf3fUOCLwRojsx/crj3sWI0CiegA9RRssNLoqIaO yfCC/CbqktBw1yZFfqbHqIOyFs1msY+a9gs/+5L1dzV8hatE+MHTOOXnd9pAFKHMHpLF KZBx+bXLMU2LUd/AzE9E6ZN4cUMVDEY9WSO068xQjIsAwWO8503EO3888RP2VV5k7n0W DtPY6DpaNHZzKJeAnjig7Y7E1YvIS/hQwFzNsFomaYpP5C8A+d7FKEUVJi1rSAlpqCEE 6nRrT87n7yNrHhohE1uvUeW7IxvJ97PMUwDFsXasmz5jxvqU0yjj0rxaJQvo6mj55KGB d26Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917729; x=1693522529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cvhsasKZJnva2wCI0Nz03kKnyZToCj12NVuapLyWr3E=; b=S8tuQVdbchBqW5LqtXLsFJy91Io0pyd2XNxi5Hm9m0dwrlYvxum33ydqA3MmdXgPs2 fqzJnOdTiAOj702vXUQnE6Kq4V/WZXTEQ81YDffCPZ4cbgqg6wcFwKh0aoEdZ+a275yu iGnpYaV0F4TMVtSCHfPJ5END+Szw7BUPFR/XJyGw5JdHZXNnFlOtUcbB/dvAILgB921e 6RfbNKgUqS8Lags1fy+H/XFD7MPqaGNZFxX3NdhaSHBjLk0JRQTO2yBnfyjNpriNnjnh k5m8PwQBdvGXUbaX7cit2y8LW/eoGCc/L1hAcy/qCpN/DWULLSsojk8cvi63zowUQNAb I6ZQ== X-Gm-Message-State: AOJu0YwCPc09szbv4TLp+QWqMUXjbTeiFdiJuzGiwUw4rObuIqYLk92x T7QOsY/ZYlKnAskjgmUObokmrKyVhLs= X-Google-Smtp-Source: AGHT+IGrj9UpZV3ga84RbF9r2scmGjpDr7xT7aFyBXsUXKkfYnYJfGnrh1f/iV6+Fol08M38yQZ0ZQ== X-Received: by 2002:a05:6512:1093:b0:4fe:a5c:efa3 with SMTP id j19-20020a056512109300b004fe0a5cefa3mr16546421lfg.62.1692917728587; Thu, 24 Aug 2023 15:55:28 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:28 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 13/15] io_uring: separate task_work/waiting cache line Date: Thu, 24 Aug 2023 23:53:35 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org task_work's are typically queued up from IRQ/softirq potentially by a random CPU like in case of networking. Batch ctx fields bouncing as this into a separate cache line. We also move ->cq_timeouts there because waiters have to read and check it. We can also conditionally hide ->cq_timeouts in the future from the CQ wait path as a not really useful rudiment. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 72e609752323..5de5dffe29df 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -270,15 +270,25 @@ struct io_ring_ctx { unsigned cached_cq_tail; unsigned cq_entries; struct io_ev_fd __rcu *io_ev_fd; - struct wait_queue_head cq_wait; unsigned cq_extra; } ____cacheline_aligned_in_smp; + /* + * task_work and async notification delivery cacheline. Expected to + * regularly bounce b/w CPUs. + */ + struct { + struct llist_head work_llist; + unsigned long check_cq; + atomic_t cq_wait_nr; + atomic_t cq_timeouts; + struct wait_queue_head cq_wait; + } ____cacheline_aligned_in_smp; + struct { spinlock_t completion_lock; bool poll_multi_queue; - atomic_t cq_wait_nr; /* * ->iopoll_list is protected by the ctx->uring_lock for @@ -287,14 +297,11 @@ struct io_ring_ctx { * manipulate the list, hence no extra locking is needed there. */ struct io_wq_work_list iopoll_list; - - struct llist_head work_llist; } ____cacheline_aligned_in_smp; /* timeouts */ struct { spinlock_t timeout_lock; - atomic_t cq_timeouts; struct list_head timeout_list; struct list_head ltimeout_list; unsigned cq_last_tm_flush; @@ -314,8 +321,6 @@ struct io_ring_ctx { struct wait_queue_head sqo_sq_wait; struct list_head sqd_list; - unsigned long check_cq; - unsigned int file_alloc_start; unsigned int file_alloc_end; From patchwork Thu Aug 24 22:53:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364848 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 491C9EE49B4 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243978AbjHXWzq (ORCPT ); Thu, 24 Aug 2023 18:55:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244046AbjHXWzh (ORCPT ); Thu, 24 Aug 2023 18:55:37 -0400 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C7491FDB for ; Thu, 24 Aug 2023 15:55:31 -0700 (PDT) Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2bc63e0d8cdso4250631fa.2 for ; Thu, 24 Aug 2023 15:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917729; x=1693522529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5sBb2bmHDwDccJqKXBWjX6d3EaphhndG8TcXEkQq5wI=; b=Nl8FgFfn08R0vxUmcrgzBoQN84qwqOLyN9IkZrxdxca5CDd+Mmq6KxdVDJtv12V01+ OQby5Qqysuqrae3tVJtK2F0bkK3Y8i+5UOFAOpzIe+2EPRtSTPvkm4WoyAYvGVvROdA3 Gsopqn3nIyT06HBjjw3LbXiQvDwg+NNqVVBoOM8XRcSzyqXZ4hv3rHd7HmKLn+Cgx3Qg cR95f6MDMuCQcdbE5oY69Uogj3wDkB5/BRVMDOmnsfbzavwdifrgRyBlshacjHNVEMWq 0+wWiBfueeiJZ8s6eqWPoF8DxRQD7oI/SqYftCCwXLFG7cTutezev9/tenvcyu2/Lpny VlOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917729; x=1693522529; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5sBb2bmHDwDccJqKXBWjX6d3EaphhndG8TcXEkQq5wI=; b=Y0o/uLNNSZB4zUmV374uv1lvuWRRkageoOL+/nd9l3qI8bFjwsEXgIzrgV+UEFOKY+ ahFsYDjUWTHgW8K26lmI1ptdNRGhI/Li/Xmqpd1OGz7ChLa5vPOCGp5tMWtQGNLhK6+w +WDLUUpSkRagHc/Q1I3wwpyDma29jSewl5y+5eXIjGDwEO/djAyqFHtp/LFFpI7WfDRU TT9vchoMOfh5vrqAXsHd1pvipuemRnSePiaEIYZRcmFqNhdOSQGIgiMf0OwcpEvot/jJ lQkZQKuTjwt4h1buUzwd3/t6W3++/2bOtV+3zaSH2+twRKf/zDoyayWUP4UdQ2vvjJ16 x58Q== X-Gm-Message-State: AOJu0YwdZtYqp+SqgZ7q5AR/qCBPKAon9y0LiMSJqM2u8gtEYRDSp/O8 2wc9S12PlrevXz8WAN3FdjPuyCxAfvw= X-Google-Smtp-Source: AGHT+IHHvUvnoXru6w+Yr/noXB95LQT/495H9OaDVSUNJwZJZZEO/L0qyQsecstFzFhnR14aediXtA== X-Received: by 2002:a05:651c:96:b0:2bc:c557:848a with SMTP id 22-20020a05651c009600b002bcc557848amr7779627ljq.50.1692917729332; Thu, 24 Aug 2023 15:55:29 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:29 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 14/15] io_uring: move multishot cqe cache in ctx Date: Thu, 24 Aug 2023 23:53:36 +0100 Message-ID: X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We cache multishot CQEs before flushing them to the CQ in submit_state.cqe. It's a 16 entry cache totalling 256 bytes in the middle of the io_submit_state structure. Move it out of there, it should help with CPU caches for the submission state, and shouldn't affect cached CQEs. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 3 ++- io_uring/io_uring.c | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 5de5dffe29df..01bdbc223edd 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -176,7 +176,6 @@ struct io_submit_state { unsigned short submit_nr; unsigned int cqes_count; struct blk_plug plug; - struct io_uring_cqe cqes[16]; }; struct io_ev_fd { @@ -307,6 +306,8 @@ struct io_ring_ctx { unsigned cq_last_tm_flush; } ____cacheline_aligned_in_smp; + struct io_uring_cqe completion_cqes[16]; + /* IRQ completion list, under ->completion_lock */ struct io_wq_work_list locked_free_list; unsigned int locked_free_nr; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a6eea3938802..88599852af82 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -880,7 +880,7 @@ static void __io_flush_post_cqes(struct io_ring_ctx *ctx) lockdep_assert_held(&ctx->uring_lock); for (i = 0; i < state->cqes_count; i++) { - struct io_uring_cqe *cqe = &state->cqes[i]; + struct io_uring_cqe *cqe = &ctx->completion_cqes[i]; if (!io_fill_cqe_aux(ctx, cqe->user_data, cqe->res, cqe->flags)) { if (ctx->task_complete) { @@ -931,7 +931,7 @@ bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags) lockdep_assert_held(&ctx->uring_lock); - if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->submit_state.cqes)) { + if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) { __io_cq_lock(ctx); __io_flush_post_cqes(ctx); /* no need to flush - flush is deferred */ @@ -945,7 +945,7 @@ bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags) if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq)) return false; - cqe = &ctx->submit_state.cqes[ctx->submit_state.cqes_count++]; + cqe = &ctx->completion_cqes[ctx->submit_state.cqes_count++]; cqe->user_data = user_data; cqe->res = res; cqe->flags = cflags; From patchwork Thu Aug 24 22:53:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13364850 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A4DBEE49B3 for ; Thu, 24 Aug 2023 22:56:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243977AbjHXWzq (ORCPT ); Thu, 24 Aug 2023 18:55:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244052AbjHXWzi (ORCPT ); Thu, 24 Aug 2023 18:55:38 -0400 Received: from mail-lj1-x22a.google.com (mail-lj1-x22a.google.com [IPv6:2a00:1450:4864:20::22a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 110251FC3 for ; Thu, 24 Aug 2023 15:55:32 -0700 (PDT) Received: by mail-lj1-x22a.google.com with SMTP id 38308e7fff4ca-2bcd7a207f7so4223861fa.3 for ; Thu, 24 Aug 2023 15:55:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692917730; x=1693522530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rJ//xnNby4d5jNgpCMvTnltm5MwCd4efW3wW1j315lY=; b=bVsm1vQVokQCMJ1TZ6GHf4zvfglESqR+K3uKqiU/fX5MkFGwJ8jd+ash5jNH2HV1Vj mu9fyUQMOkg9TJyPpY9HFD/BGLYHZPp+z8dBn0aRQuuZrA2lannIchM0pT1hdKbhusfY LAfzhy6z+uyPgLcO7kDJOBuR53ojzM3Oc9Bw60fUmibyuXvFpPpX2s5onWYaTlaF9aNy rx3vTnzA5GW4fYAAApFv/Z2kwCTsMqt4PyH15tUXbjSAxW7MD85BBDTPRLaPrQJOdBKM aMwkrnaeJUzJc2U3HQBqkcGLGtmgzimoBV43OtIcKDKwpdXKpJ0Usb4PUs2DXWVTYqPp WNuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692917730; x=1693522530; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rJ//xnNby4d5jNgpCMvTnltm5MwCd4efW3wW1j315lY=; b=Bsc+2iIjZBBZg7Z1FskYBE9Td5TsrGdygAjRn7EB945aeTnws7cTwRmg8jlsqDs6LC FgAUbxye0oVZQTStLkRcxN8jBkEihbWXQkBGlGSstyDbFHH/oFrPnQtU9wJTuOb8WgR8 PbSDmUVhIy8s7a7bAhWxhjh11Jze5/tVtDUxlBao17WQDhhkwZIjuAIGqURp2Pa73qwy lGbeeAV8FwEcgtheE2Fudx/bJbCIqGEpErMZkG7/1CSeAhvoOpNUXcHUhoXPUoCtvNxZ KiZ1iVj8nNUHNSYz7VfYezfCsTjjCIMrNg1A2P8EpmOyUnKfRqA6jwcxygxqs4WaWmYE 3zUg== X-Gm-Message-State: AOJu0YyhhZz1mz0RRDePocO7KDrNTS4hdLBD03nWhU3dYVCbEIPXK2my uvSe2bunBjDexOjqC5ndwMaLC+Qhi+8= X-Google-Smtp-Source: AGHT+IG1GzITnRPVOSEM5gsTsfoSpU1KTYAwBnsRKseXZI84MjU5hhaivS8TlcEF10J1p7AyQGrnnw== X-Received: by 2002:a2e:9b58:0:b0:2bc:c1d9:6848 with SMTP id o24-20020a2e9b58000000b002bcc1d96848mr9442048ljj.44.1692917730114; Thu, 24 Aug 2023 15:55:30 -0700 (PDT) Received: from 127.0.0.1localhost ([148.252.140.69]) by smtp.gmail.com with ESMTPSA id q4-20020a170906144400b00992f81122e1sm173469ejc.21.2023.08.24.15.55.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Aug 2023 15:55:29 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH v2 15/15] io_uring: move iopoll ctx fields around Date: Thu, 24 Aug 2023 23:53:37 +0100 Message-ID: <5b03cf7e6652e350e6e70a917eec72ba9f33b97b.1692916914.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Move poll_multi_queue and iopoll_list to the submission cache line, it doesn't make much sense to keep them separately, and is better place for it in general. Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 01bdbc223edd..13d19b9be9f4 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -256,6 +256,15 @@ struct io_ring_ctx { struct io_hash_table cancel_table_locked; struct io_alloc_cache apoll_cache; struct io_alloc_cache netmsg_cache; + + /* + * ->iopoll_list is protected by the ctx->uring_lock for + * io_uring instances that don't use IORING_SETUP_SQPOLL. + * For SQPOLL, only the single threaded io_sq_thread() will + * manipulate the list, hence no extra locking is needed there. + */ + struct io_wq_work_list iopoll_list; + bool poll_multi_queue; } ____cacheline_aligned_in_smp; struct { @@ -284,20 +293,6 @@ struct io_ring_ctx { struct wait_queue_head cq_wait; } ____cacheline_aligned_in_smp; - struct { - spinlock_t completion_lock; - - bool poll_multi_queue; - - /* - * ->iopoll_list is protected by the ctx->uring_lock for - * io_uring instances that don't use IORING_SETUP_SQPOLL. - * For SQPOLL, only the single threaded io_sq_thread() will - * manipulate the list, hence no extra locking is needed there. - */ - struct io_wq_work_list iopoll_list; - } ____cacheline_aligned_in_smp; - /* timeouts */ struct { spinlock_t timeout_lock; @@ -308,6 +303,8 @@ struct io_ring_ctx { struct io_uring_cqe completion_cqes[16]; + spinlock_t completion_lock; + /* IRQ completion list, under ->completion_lock */ struct io_wq_work_list locked_free_list; unsigned int locked_free_nr;