From patchwork Mon Sep 5 13:22:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966126 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8101C6FA83 for ; Mon, 5 Sep 2022 13:25:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237362AbiIENZF (ORCPT ); Mon, 5 Sep 2022 09:25:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237442AbiIENZB (ORCPT ); Mon, 5 Sep 2022 09:25:01 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8D8717E30 for ; Mon, 5 Sep 2022 06:25:00 -0700 (PDT) Received: from pps.filterd (m0109334.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2858W5Bh015032 for ; Mon, 5 Sep 2022 06:25:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=gnpoJUJbAC5s4BHYrIp+PoxM0NGyFE9ctD+3e4Y/UDY=; b=rTDgewyin3nuGPYMWhh/iNI5Pb1beLGHCC9StBym4MXikgF6WLGm2edDTpnhYFPx12kZ TrY2/x4XYglZn4JEJ/kH+69pztlZ1lV7FBmgZi53s6u6Re6oHY7GqV5PxbhZqiWEKnWC 9JIS+WdpNTEbZcNcYovuc+YZZ2uVVEhNn/U= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jc684ramf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:25:00 -0700 Received: from twshared8442.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:83::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:24:59 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 1A6535AC516B; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 01/11] Copy defer task run definition from kernel Date: Mon, 5 Sep 2022 06:22:48 -0700 Message-ID: <20220905132258.1858915-2-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: cLUo1LCJYgGSr9a3VWQhGC-shl_hxwI- X-Proofpoint-ORIG-GUID: cLUo1LCJYgGSr9a3VWQhGC-shl_hxwI- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Copy the flag from upstream Signed-off-by: Dylan Yudaken --- src/include/liburing/io_uring.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/include/liburing/io_uring.h b/src/include/liburing/io_uring.h index 6b83177fd41d..972b179bc07a 100644 --- a/src/include/liburing/io_uring.h +++ b/src/include/liburing/io_uring.h @@ -157,6 +157,13 @@ enum { */ #define IORING_SETUP_SINGLE_ISSUER (1U << 12) +/* + * Defer running task work to get events. + * Rather than running bits of task work whenever the task transitions + * try to do it just before it is needed. + */ +#define IORING_SETUP_DEFER_TASKRUN (1U << 13) + enum io_uring_op { IORING_OP_NOP, IORING_OP_READV, From patchwork Mon Sep 5 13:22:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52E2AECAAD3 for ; Mon, 5 Sep 2022 13:25:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236467AbiIENZD (ORCPT ); Mon, 5 Sep 2022 09:25:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48472 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237362AbiIENZB (ORCPT ); Mon, 5 Sep 2022 09:25:01 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0B3F17E2E for ; Mon, 5 Sep 2022 06:25:00 -0700 (PDT) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 285BkDnb009235 for ; Mon, 5 Sep 2022 06:24:59 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=/dwzYe7skYYa7PtXqGvN3HFbnRJ0KJOIyWyVDMKn8H8=; b=aOWduP4IjoZUHB0LduRjGx+uNHvyc/AqdtrigqKNt2XIxMwNcasTjSzDyWObHLZ9dJ7E csagwtwXwGvfFo/jPAyuu0Yt2Q2zrDD3rCOJc4RGDWIurGAZSeDUk9QFEYf5NDuJH+Zp 7wq0PA7gY6p5OAzmb4eW5+uRzStuAApURag= Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3jc2kx0wn2-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:24:59 -0700 Received: from twshared11415.03.ash7.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:24:58 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 1E2C65AC516C; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 02/11] Add documentation for IORING_SETUP_DEFER_TASKRUN flag Date: Mon, 5 Sep 2022 06:22:49 -0700 Message-ID: <20220905132258.1858915-3-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: AzcduOHQ9m3dJj2eQQu0vmrS5ccUfLZ5 X-Proofpoint-GUID: AzcduOHQ9m3dJj2eQQu0vmrS5ccUfLZ5 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add man page entry to io_uring_setup.2 for the new flag Signed-off-by: Dylan Yudaken --- man/io_uring_setup.2 | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/man/io_uring_setup.2 b/man/io_uring_setup.2 index 32a9e2ee89b5..01eb70d95292 100644 --- a/man/io_uring_setup.2 +++ b/man/io_uring_setup.2 @@ -257,6 +257,25 @@ how many userspace tasks do .I io_uring_enter(2). Available since 5.20. +.TP +.B IORING_SETUP_DEFER_TASKRUN +By default, io_uring will process all outstanding work at the end of any system +call or thread interrupt. This can delay the application from making other progress. +Setting this flag will hint to io_uring that it should defer work until an +.BR io_uring_enter(2) +call with the +.B IORING_ENTER_GETEVENTS +flag set. This allows the application to request work to run just before it wants to +process completions. +This flag requires the +.BR IORING_SETUP_SINGLE_ISSUER +flag to be set, and also enforces that the call to +.BR io_uring_enter(2) +is called from the same thread that submitted requests. +Note that if this flag is set then it is the application's responsibility to periodically +trigger work (for example via any of the CQE waiting functions) or else completions may +not be delivered. +Available since 6.1. .PP If no flags are specified, the io_uring instance is setup for interrupt driven I/O. I/O may be submitted using From patchwork Mon Sep 5 13:22:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78923ECAAD5 for ; Mon, 5 Sep 2022 13:25:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237130AbiIENZG (ORCPT ); Mon, 5 Sep 2022 09:25:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236575AbiIENZE (ORCPT ); Mon, 5 Sep 2022 09:25:04 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C10C2186E9 for ; Mon, 5 Sep 2022 06:25:02 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 285BZMXE031641 for ; Mon, 5 Sep 2022 06:25:02 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=WXn5jQ5WJCURHwV9rZatQXxT6A/uG/brVfpgetXdZFA=; b=AlpJvAJzfuWWbWfFI66wlsfoPTrCRqzBA+ufaXxxvk/dxlF6Ojlj3dls5q4QAbSZecPn Tp+y8zu/w+1dkHmKSG7FCdzM3p/uUrc+cViiAbXzi2ZVU+6guyplGIdoF7jnjZ+rCSPD RHzSVyWI0aNSX43o624DRdU/qTfnFOZJuUI= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jcgaeegkn-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:25:02 -0700 Received: from twshared8288.05.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:25:01 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 1FD6D5AC516E; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 03/11] add io_uring_submit_and_get_events and io_uring_get_events Date: Mon, 5 Sep 2022 06:22:50 -0700 Message-ID: <20220905132258.1858915-4-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: NSKTn3QWwPNCuXuu519YWc9SCaIYBUgM X-Proofpoint-GUID: NSKTn3QWwPNCuXuu519YWc9SCaIYBUgM X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org With deferred task running, we would like to be able to combine submit with get events (regardless of if there are CQE's available), or if there is nothing to submit then simply do an enter with IORING_ENTER_GETEVENTS set, in order to process any available work. Expose these APIs Signed-off-by: Dylan Yudaken --- man/io_uring_get_events.3 | 32 ++++++++++++++++++++++++++++ man/io_uring_submit_and_get_events.3 | 31 +++++++++++++++++++++++++++ src/include/liburing.h | 3 +++ src/liburing.map | 2 ++ src/queue.c | 26 +++++++++++++++------- 5 files changed, 86 insertions(+), 8 deletions(-) create mode 100644 man/io_uring_get_events.3 create mode 100644 man/io_uring_submit_and_get_events.3 diff --git a/man/io_uring_get_events.3 b/man/io_uring_get_events.3 new file mode 100644 index 000000000000..2ac3e070473e --- /dev/null +++ b/man/io_uring_get_events.3 @@ -0,0 +1,32 @@ +.\" Copyright (C) 2022 Dylan Yudaken +.\" +.\" SPDX-License-Identifier: LGPL-2.0-or-later +.\" +.TH io_uring_get_events 3 "September 5, 2022" "liburing-2.3" "liburing Manual" +.SH NAME +io_uring_get_events \- Flush outstanding requests to CQE ring +.SH SYNOPSIS +.nf +.B #include +.PP +.BI "int io_uring_get_events(struct io_uring *" ring ");" +.fi +.SH DESCRIPTION +.PP +The +.BR io_uring_get_events (3) +function runs outstanding work and flushes completion events to the CQE ring. + +There can be events needing to be flushed if the ring was full and had overflowed. +Alternatively if the ring was setup with the +.BR IORING_SETUP_DEFER_TASKRUN +flag then this will process outstanding tasks, possibly resulting in more CQEs. + +.SH RETURN VALUE +On success +.BR io_uring_get_events (3) +returns 0. On failure it returns +.BR -errno . +.SH SEE ALSO +.BR io_uring_get_sqe (3), +.BR io_uring_submit_and_get_events (3) diff --git a/man/io_uring_submit_and_get_events.3 b/man/io_uring_submit_and_get_events.3 new file mode 100644 index 000000000000..9e143d1dff47 --- /dev/null +++ b/man/io_uring_submit_and_get_events.3 @@ -0,0 +1,31 @@ +.\" Copyright (C), 2022 dylany +.\" You may distribute this file under the terms of the GNU Free +.\" Documentation License. +.TH io_uring_submit_and_get_events 3 "September 5, 2022" "liburing-2.3" "liburing Manual" +.SH NAME +io_uring_submit_and_get_events \- submit requests to the submission queue and flush completions +.SH SYNOPSIS +.nf +.B #include +.PP +.BI "int io_uring_submit_and_get_events(struct io_uring *" ring ");" +.fi + +.SH DESCRIPTION +The +.BR io_uring_submit_and_get_events (3) +function submits the next events to the submission queue as with +.BR io_uring_submit (3) . +After submission it will flush CQEs as with +.BR io_uring_get_events (3) . + +The benefit of this function is that it does both with only one system call. + +.SH RETURN VALUE +On success +.BR io_uring_submit_and_get_events (3) +returns the number of submitted submission queue entries. On failure it returns +.BR -errno . +.SH SEE ALSO +.BR io_uring_submit (3), +.BR io_uring_get_events (3) diff --git a/src/include/liburing.h b/src/include/liburing.h index 15e93f5bfb4e..765375cb5c6a 100644 --- a/src/include/liburing.h +++ b/src/include/liburing.h @@ -235,6 +235,9 @@ int io_uring_register_sync_cancel(struct io_uring *ring, int io_uring_register_file_alloc_range(struct io_uring *ring, unsigned off, unsigned len); +int io_uring_get_events(struct io_uring *ring); +int io_uring_submit_and_get_events(struct io_uring *ring); + /* * io_uring syscalls. */ diff --git a/src/liburing.map b/src/liburing.map index 8573dfc69cf9..5d08857ff906 100644 --- a/src/liburing.map +++ b/src/liburing.map @@ -66,4 +66,6 @@ LIBURING_2.3 { io_uring_enter2; io_uring_setup; io_uring_register; + io_uring_get_events; + io_uring_submit_and_get_events; } LIBURING_2.2; diff --git a/src/queue.c b/src/queue.c index a670a8ecd20d..b012a3dd950b 100644 --- a/src/queue.c +++ b/src/queue.c @@ -130,6 +130,15 @@ int __io_uring_get_cqe(struct io_uring *ring, struct io_uring_cqe **cqe_ptr, return _io_uring_get_cqe(ring, cqe_ptr, &data); } +int io_uring_get_events(struct io_uring *ring) +{ + int flags = IORING_ENTER_GETEVENTS; + + if (ring->int_flags & INT_FLAG_REG_RING) + flags |= IORING_ENTER_REGISTERED_RING; + return __sys_io_uring_enter(ring->enter_ring_fd, 0, 0, flags, NULL); +} + /* * Fill in an array of IO completions up to count, if any are available. * Returns the amount of IO completions filled. @@ -164,11 +173,7 @@ again: return 0; if (cq_ring_needs_flush(ring)) { - int flags = IORING_ENTER_GETEVENTS; - - if (ring->int_flags & INT_FLAG_REG_RING) - flags |= IORING_ENTER_REGISTERED_RING; - __sys_io_uring_enter(ring->enter_ring_fd, 0, 0, flags, NULL); + io_uring_get_events(ring); overflow_checked = true; goto again; } @@ -340,9 +345,9 @@ int io_uring_wait_cqe_timeout(struct io_uring *ring, * Returns number of sqes submitted */ static int __io_uring_submit(struct io_uring *ring, unsigned submitted, - unsigned wait_nr) + unsigned wait_nr, bool getevents) { - bool cq_needs_enter = wait_nr || cq_ring_needs_enter(ring); + bool cq_needs_enter = getevents || wait_nr || cq_ring_needs_enter(ring); unsigned flags; int ret; @@ -363,7 +368,7 @@ static int __io_uring_submit(struct io_uring *ring, unsigned submitted, static int __io_uring_submit_and_wait(struct io_uring *ring, unsigned wait_nr) { - return __io_uring_submit(ring, __io_uring_flush_sq(ring), wait_nr); + return __io_uring_submit(ring, __io_uring_flush_sq(ring), wait_nr, false); } /* @@ -386,6 +391,11 @@ int io_uring_submit_and_wait(struct io_uring *ring, unsigned wait_nr) return __io_uring_submit_and_wait(ring, wait_nr); } +int io_uring_submit_and_get_events(struct io_uring *ring) +{ + return __io_uring_submit(ring, __io_uring_flush_sq(ring), 0, true); +} + #ifdef LIBURING_INTERNAL struct io_uring_sqe *io_uring_get_sqe(struct io_uring *ring) { From patchwork Mon Sep 5 13:22:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0FD9ECAAD5 for ; Mon, 5 Sep 2022 13:25:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237207AbiIENZB (ORCPT ); Mon, 5 Sep 2022 09:25:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236467AbiIENZA (ORCPT ); Mon, 5 Sep 2022 09:25:00 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4567715726 for ; Mon, 5 Sep 2022 06:24:59 -0700 (PDT) Received: from pps.filterd (m0001303.ppops.net [127.0.0.1]) by m0001303.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 285BkDnY009235 for ; Mon, 5 Sep 2022 06:24:58 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=xQsdTa8TgbkhagguoxnaKU57QbO73paFepvRMH2t8WM=; b=oeR69H0xfS0bSLHxXUigeR76/8vS/vBmsSa3qFuAYLdIgppXMMSdqoPr0ZI1VM0/OrPM LV3HqEEWeLEE0/rL+VhJozyXd62gdCx2xk9L6KaoGE2Up4h4LjbYnm8DIjQZpiy1lm4g k5JsoBGf5gBP3iZc2Z6PBwDzbvnYAOptTyA= Received: from mail.thefacebook.com ([163.114.132.120]) by m0001303.ppops.net (PPS) with ESMTPS id 3jc2kx0wn2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:24:58 -0700 Received: from twshared3888.09.ash9.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:21d::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:24:57 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 268935AC5170; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 04/11] add a t_probe_defer_taskrun helper function for tests Date: Mon, 5 Sep 2022 06:22:51 -0700 Message-ID: <20220905132258.1858915-5-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: YkW_IyWLTTRb6wDjWImbJ6d6uzuRH1HB X-Proofpoint-GUID: YkW_IyWLTTRb6wDjWImbJ6d6uzuRH1HB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Help tests to determine if they can use IORING_SETUP_DEFER_TASKRUN Signed-off-by: Dylan Yudaken --- test/helpers.c | 17 +++++++++++++++-- test/helpers.h | 2 ++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/test/helpers.c b/test/helpers.c index 4d5c402f8fb2..8fb32b8fa982 100644 --- a/test/helpers.c +++ b/test/helpers.c @@ -75,7 +75,7 @@ static void __t_create_file(const char *file, size_t size, char pattern) { ssize_t ret; char *buf; - int fd; + int fd; buf = t_malloc(size); memset(buf, pattern, size); @@ -112,7 +112,7 @@ struct iovec *t_create_buffers(size_t buf_num, size_t buf_size) vecs = t_malloc(buf_num * sizeof(struct iovec)); for (i = 0; i < buf_num; i++) { t_posix_memalign(&vecs[i].iov_base, buf_size, buf_size); - vecs[i].iov_len = buf_size; + vecs[i].iov_len = buf_size; } return vecs; } @@ -253,3 +253,16 @@ errno_cleanup: close(fd[1]); return ret; } + +bool t_probe_defer_taskrun(void) +{ + struct io_uring ring; + int ret; + + ret = io_uring_queue_init(1, &ring, IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN); + if (ret < 0) + return false; + io_uring_queue_exit(&ring); + return true; +} diff --git a/test/helpers.h b/test/helpers.h index 9ad99472b288..5e9bb2613ab2 100644 --- a/test/helpers.h +++ b/test/helpers.h @@ -82,6 +82,8 @@ enum t_setup_ret t_register_buffers(struct io_uring *ring, const struct iovec *iovecs, unsigned nr_iovecs); +bool t_probe_defer_taskrun(void); + #define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0])) #ifdef __cplusplus From patchwork Mon Sep 5 13:22:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34497ECAAD5 for ; Mon, 5 Sep 2022 13:25:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236379AbiIENZO (ORCPT ); Mon, 5 Sep 2022 09:25:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236326AbiIENZN (ORCPT ); Mon, 5 Sep 2022 09:25:13 -0400 Received: from mx0a-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A27582494B for ; Mon, 5 Sep 2022 06:25:11 -0700 (PDT) Received: from pps.filterd (m0089730.ppops.net [127.0.0.1]) by m0089730.ppops.net (8.17.1.5/8.17.1.5) with ESMTP id 2852gq4U004596 for ; Mon, 5 Sep 2022 06:25:10 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=t1Asr9tRDaaI2rFoxKH7e153Co8XTQNQeo+rjD8KwVw=; b=MuEs22dEzouCoNSzn7nWCCwsNCs/VdoHg+gp0POQEWdjJM80L3q/8S2u2wjlz1AL/5xK +eL32jsgyjwht6FqhzKuK52Tmim5Y8ZpnKzFTL9F6RxhAenDqDLnC5gXIA6K8URv5XVz 5wit4t0Xm39fnCTYKTkAucaX3eyVqWh9ihU= Received: from maileast.thefacebook.com ([163.114.130.16]) by m0089730.ppops.net (PPS) with ESMTPS id 3jc2e8gwm0-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:25:10 -0700 Received: from twshared8442.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::c) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:25:09 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 351DF5AC5172; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 05/11] update existing tests for defer taskrun Date: Mon, 5 Sep 2022 06:22:52 -0700 Message-ID: <20220905132258.1858915-6-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: Oj7Kn0SmBJG-EBJguMsIH2sOcSh1fWsi X-Proofpoint-ORIG-GUID: Oj7Kn0SmBJG-EBJguMsIH2sOcSh1fWsi X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add defer_taskrun to a few choice tests that can expose some bad behaviour. This requires adding some io_uring_get_events calls to make sure deferred tasks are run Signed-off-by: Dylan Yudaken --- test/eventfd-disable.c | 33 ++++++++++++++++++++++--- test/iopoll.c | 17 +++++++++---- test/multicqes_drain.c | 50 +++++++++++++++++++++++++++++++++----- test/poll-mshot-overflow.c | 40 +++++++++++++++++++++++++++--- test/recv-multishot.c | 33 ++++++++++++++++--------- test/rsrc_tags.c | 10 ++++++-- 6 files changed, 152 insertions(+), 31 deletions(-) diff --git a/test/eventfd-disable.c b/test/eventfd-disable.c index 2c8cf6dad7c1..162f9f9bc783 100644 --- a/test/eventfd-disable.c +++ b/test/eventfd-disable.c @@ -15,7 +15,7 @@ #include "liburing.h" #include "helpers.h" -int main(int argc, char *argv[]) +static int test(bool defer) { struct io_uring_params p = {}; struct io_uring_sqe *sqe; @@ -28,8 +28,9 @@ int main(int argc, char *argv[]) }; int ret, evfd, i; - if (argc > 1) - return T_EXIT_SKIP; + if (defer) + p.flags |= IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN; ret = io_uring_queue_init_params(64, &ring, &p); if (ret) { @@ -148,5 +149,31 @@ int main(int argc, char *argv[]) io_uring_cqe_seen(&ring, cqe); } + io_uring_queue_exit(&ring); + close(evfd); return T_EXIT_PASS; } + +int main(int argc, char *argv[]) +{ + int ret; + + if (argc > 1) + return T_EXIT_SKIP; + + ret = test(false); + if (ret != T_EXIT_PASS) { + fprintf(stderr, "%s: test(false) failed\n", argv[0]); + return ret; + } + + if (t_probe_defer_taskrun()) { + ret = test(true); + if (ret != T_EXIT_PASS) { + fprintf(stderr, "%s: test(true) failed\n", argv[0]); + return ret; + } + } + + return ret; +} diff --git a/test/iopoll.c b/test/iopoll.c index 91cb71bd2e9c..20f91c7947be 100644 --- a/test/iopoll.c +++ b/test/iopoll.c @@ -274,7 +274,7 @@ ok: } static int test_io(const char *file, int write, int sqthread, int fixed, - int buf_select) + int buf_select, int defer) { struct io_uring ring; int ret, ring_flags = IORING_SETUP_IOPOLL; @@ -282,6 +282,10 @@ static int test_io(const char *file, int write, int sqthread, int fixed, if (no_iopoll) return 0; + if (defer) + ring_flags |= IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN; + ret = t_create_ring(64, &ring, ring_flags); if (ret == T_SETUP_SKIP) return 0; @@ -337,19 +341,22 @@ int main(int argc, char *argv[]) vecs = t_create_buffers(BUFFERS, BS); - nr = 16; + nr = 32; if (no_buf_select) nr = 8; + else if (!t_probe_defer_taskrun()) + nr = 16; for (i = 0; i < nr; i++) { int write = (i & 1) != 0; int sqthread = (i & 2) != 0; int fixed = (i & 4) != 0; int buf_select = (i & 8) != 0; + int defer = (i & 16) != 0; - ret = test_io(fname, write, sqthread, fixed, buf_select); + ret = test_io(fname, write, sqthread, fixed, buf_select, defer); if (ret) { - fprintf(stderr, "test_io failed %d/%d/%d/%d\n", - write, sqthread, fixed, buf_select); + fprintf(stderr, "test_io failed %d/%d/%d/%d/%d\n", + write, sqthread, fixed, buf_select, defer); goto err; } if (no_iopoll) diff --git a/test/multicqes_drain.c b/test/multicqes_drain.c index 6cd03ba5f3f7..f95c4382b3f4 100644 --- a/test/multicqes_drain.c +++ b/test/multicqes_drain.c @@ -233,6 +233,8 @@ static int test_generic_drain(struct io_uring *ring) if (trigger_event(pipes[i])) goto err; + + io_uring_get_events(ring); } sleep(1); i = 0; @@ -246,7 +248,7 @@ static int test_generic_drain(struct io_uring *ring) * compl_bits is a bit map to record completions. * eg. sqe[0], sqe[1], sqe[2] fully completed * then compl_bits is 000...00111b - * + * */ unsigned long long compl_bits = 0; for (j = 0; j < i; j++) { @@ -295,7 +297,12 @@ static int test_simple_drain(struct io_uring *ring) io_uring_prep_poll_add(sqe[1], pipe2[0], POLLIN); sqe[1]->user_data = 1; - ret = io_uring_submit(ring); + /* This test relies on multishot poll to trigger events continually. + * however with IORING_SETUP_DEFER_TASKRUN this will only happen when + * triggered with a get_events. Hence we sprinkle get_events whenever + * there might be work to process in order to get the same result + */ + ret = io_uring_submit_and_get_events(ring); if (ret < 0) { printf("sqe submit failed\n"); goto err; @@ -307,9 +314,11 @@ static int test_simple_drain(struct io_uring *ring) for (i = 0; i < 2; i++) { if (trigger_event(pipe1)) goto err; + io_uring_get_events(ring); } if (trigger_event(pipe2)) goto err; + io_uring_get_events(ring); for (i = 0; i < 2; i++) { sqe[i] = io_uring_get_sqe(ring); @@ -355,15 +364,17 @@ err: return 1; } -int main(int argc, char *argv[]) +static int test(bool defer_taskrun) { struct io_uring ring; int i, ret; + unsigned int flags = 0; - if (argc > 1) - return T_EXIT_SKIP; + if (defer_taskrun) + flags = IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN; - ret = io_uring_queue_init(1024, &ring, 0); + ret = io_uring_queue_init(1024, &ring, flags); if (ret) { printf("ring setup failed\n"); return T_EXIT_FAIL; @@ -384,5 +395,32 @@ int main(int argc, char *argv[]) return T_EXIT_FAIL; } } + + io_uring_queue_exit(&ring); + return T_EXIT_PASS; } + +int main(int argc, char *argv[]) +{ + int ret; + + if (argc > 1) + return T_EXIT_SKIP; + + ret = test(false); + if (ret != T_EXIT_PASS) { + fprintf(stderr, "%s: test(false) failed\n", argv[0]); + return ret; + } + + if (t_probe_defer_taskrun()) { + ret = test(true); + if (ret != T_EXIT_PASS) { + fprintf(stderr, "%s: test(true) failed\n", argv[0]); + return ret; + } + } + + return ret; +} diff --git a/test/poll-mshot-overflow.c b/test/poll-mshot-overflow.c index 360df65d2b15..431a337f19ae 100644 --- a/test/poll-mshot-overflow.c +++ b/test/poll-mshot-overflow.c @@ -42,7 +42,7 @@ int check_final_cqe(struct io_uring *ring) return T_EXIT_PASS; } -int main(int argc, char *argv[]) +static int test(bool defer_taskrun) { struct io_uring_cqe *cqe; struct io_uring_sqe *sqe; @@ -50,9 +50,6 @@ int main(int argc, char *argv[]) int pipe1[2]; int ret, i; - if (argc > 1) - return 0; - if (pipe(pipe1) != 0) { perror("pipe"); return T_EXIT_FAIL; @@ -66,6 +63,10 @@ int main(int argc, char *argv[]) .cq_entries = 2 }; + if (defer_taskrun) + params.flags |= IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN; + ret = io_uring_queue_init_params(2, &ring, ¶ms); if (ret) return T_EXIT_SKIP; @@ -113,6 +114,9 @@ int main(int argc, char *argv[]) io_uring_cqe_seen(&ring, cqe); } + /* make sure everything is processed */ + io_uring_get_events(&ring); + /* now remove the poll */ sqe = io_uring_get_sqe(&ring); io_uring_prep_poll_remove(sqe, 1); @@ -126,5 +130,33 @@ int main(int argc, char *argv[]) ret = check_final_cqe(&ring); + close(pipe1[0]); + close(pipe1[1]); + io_uring_queue_exit(&ring); + + return ret; +} + +int main(int argc, char *argv[]) +{ + int ret; + + if (argc > 1) + return T_EXIT_SKIP; + + ret = test(false); + if (ret != T_EXIT_PASS) { + fprintf(stderr, "%s: test(false) failed\n", argv[0]); + return ret; + } + + if (t_probe_defer_taskrun()) { + ret = test(true); + if (ret != T_EXIT_PASS) { + fprintf(stderr, "%s: test(true) failed\n", argv[0]); + return ret; + } + } + return ret; } diff --git a/test/recv-multishot.c b/test/recv-multishot.c index a322e4317232..1a041f8e865a 100644 --- a/test/recv-multishot.c +++ b/test/recv-multishot.c @@ -29,6 +29,7 @@ struct args { bool wait_each; bool recvmsg; enum early_error_t early_error; + bool defer; }; static int check_sockaddr(struct sockaddr_in *in) @@ -76,19 +77,22 @@ static int test(struct args *args) .tv_sec = 1, }; struct msghdr msg; + struct io_uring_params params = { }; + int n_sqe = 32; memset(recv_buffs, 0, sizeof(recv_buffs)); - if (args->early_error == ERROR_EARLY_OVERFLOW) { - struct io_uring_params params = { - .flags = IORING_SETUP_CQSIZE, - .cq_entries = N_CQE_OVERFLOW - }; + if (args->defer) + params.flags |= IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN; - ret = io_uring_queue_init_params(N_CQE_OVERFLOW, &ring, ¶ms); - } else { - ret = io_uring_queue_init(32, &ring, 0); + if (args->early_error == ERROR_EARLY_OVERFLOW) { + params.flags |= IORING_SETUP_CQSIZE; + params.cq_entries = N_CQE_OVERFLOW; + n_sqe = N_CQE_OVERFLOW; } + + ret = io_uring_queue_init_params(n_sqe, &ring, ¶ms); if (ret) { fprintf(stderr, "queue init failed: %d\n", ret); return ret; @@ -457,23 +461,30 @@ int main(int argc, char *argv[]) int ret; int loop; int early_error = 0; + bool has_defer; if (argc > 1) return T_EXIT_SKIP; - for (loop = 0; loop < 8; loop++) { + has_defer = t_probe_defer_taskrun(); + + for (loop = 0; loop < 16; loop++) { struct args a = { .stream = loop & 0x01, .wait_each = loop & 0x2, .recvmsg = loop & 0x04, + .defer = loop & 0x08, }; + if (a.defer && !has_defer) + continue; for (early_error = 0; early_error < ERROR_EARLY_LAST; early_error++) { a.early_error = (enum early_error_t)early_error; ret = test(&a); if (ret) { fprintf(stderr, - "test stream=%d wait_each=%d recvmsg=%d early_error=%d failed\n", - a.stream, a.wait_each, a.recvmsg, a.early_error); + "test stream=%d wait_each=%d recvmsg=%d early_error=%d " + " defer=%d failed\n", + a.stream, a.wait_each, a.recvmsg, a.early_error, a.defer); return T_EXIT_FAIL; } if (no_recv_mshot) diff --git a/test/rsrc_tags.c b/test/rsrc_tags.c index 22370644b200..047e844acfbd 100644 --- a/test/rsrc_tags.c +++ b/test/rsrc_tags.c @@ -401,7 +401,8 @@ static int test_notag(void) int main(int argc, char *argv[]) { - int ring_flags[] = {0, IORING_SETUP_IOPOLL, IORING_SETUP_SQPOLL}; + int ring_flags[] = {0, IORING_SETUP_IOPOLL, IORING_SETUP_SQPOLL, + IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN}; int i, ret; if (argc > 1) @@ -423,7 +424,12 @@ int main(int argc, char *argv[]) } for (i = 0; i < sizeof(ring_flags) / sizeof(ring_flags[0]); i++) { - ret = test_files(ring_flags[i]); + int flag = ring_flags[i]; + + if (flag & IORING_SETUP_DEFER_TASKRUN && !t_probe_defer_taskrun()) + continue; + + ret = test_files(flag); if (ret) { printf("test_tag failed, type %i\n", i); return ret; From patchwork Mon Sep 5 13:22:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED936ECAAD3 for ; Mon, 5 Sep 2022 13:27:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237809AbiIEN1g (ORCPT ); Mon, 5 Sep 2022 09:27:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237730AbiIEN1b (ORCPT ); Mon, 5 Sep 2022 09:27:31 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8708548EB0 for ; Mon, 5 Sep 2022 06:27:29 -0700 (PDT) Received: from pps.filterd (m0109331.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 285D2wJG012139 for ; Mon, 5 Sep 2022 06:27:28 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=vdLgoPRmihW03RzES3m7kZr3l5AcEv78g4/43Qy3Cuw=; b=ZuVqTkMgb/XffEhvW1dJJkVRieygOhNlP4TPRds5Tkj02xRO7Bdz73FIjb3qK9uaV9VM DXphEbBRt5/z6oECuXmF2Mcw0jIXpqz0HK3aiH5RCs/QiSIHzQT8dmje+WDqgB0QcDhy zt9RuNQR3Ka2sx8/Ld/DnT5SUp7kJu4sPJg= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jc4rurkb3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:27:28 -0700 Received: from twshared0646.06.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:27:27 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 3EBE75AC5174; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 06/11] add a defer-taskrun test Date: Mon, 5 Sep 2022 06:22:53 -0700 Message-ID: <20220905132258.1858915-7-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: JYxONh6KUbdPpTTRgBCqaW8d158KNMcH X-Proofpoint-ORIG-GUID: JYxONh6KUbdPpTTRgBCqaW8d158KNMcH X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a test specifically for IORING_SETUP_DEFER_TASKRUN Signed-off-by: Dylan Yudaken --- test/Makefile | 1 + test/defer-taskrun.c | 333 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 334 insertions(+) create mode 100644 test/defer-taskrun.c diff --git a/test/Makefile b/test/Makefile index 418c11c95875..78a499a357d7 100644 --- a/test/Makefile +++ b/test/Makefile @@ -62,6 +62,7 @@ test_srcs := \ d4ae271dfaae.c \ d77a67ed5f27.c \ defer.c \ + defer-taskrun.c \ double-poll-crash.c \ drop-submit.c \ eeed8b54e0df.c \ diff --git a/test/defer-taskrun.c b/test/defer-taskrun.c new file mode 100644 index 000000000000..aec8c5d3f223 --- /dev/null +++ b/test/defer-taskrun.c @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: MIT +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "liburing.h" +#include "test.h" +#include "helpers.h" + +#define EXEC_FILENAME ".defer-taskrun" +#define EXEC_FILESIZE (1U<<20) + +static bool can_read_t(int fd, int time) +{ + int ret; + struct pollfd p = { + .fd = fd, + .events = POLLIN, + }; + + ret = poll(&p, 1, time); + + return ret == 1; +} + +static bool can_read(int fd) +{ + return can_read_t(fd, 0); +} + +static void eventfd_clear(int fd) +{ + uint64_t val; + int ret; + + assert(can_read(fd)); + ret = read(fd, &val, 8); + assert(ret == 8); +} + +static void eventfd_trigger(int fd) +{ + uint64_t val = 1; + int ret; + + ret = write(fd, &val, sizeof(val)); + assert(ret == sizeof(val)); +} + +#define CHECK(x) if (!(x)) { \ + fprintf(stderr, "%s:%d %s failed\n", __FILE__, __LINE__, #x); \ + return -1; } + +static int test_eventfd(void) +{ + struct io_uring ring; + int ret; + int fda, fdb; + struct io_uring_cqe *cqe; + + ret = io_uring_queue_init(8, &ring, IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN); + if (ret) + return ret; + + fda = eventfd(0, EFD_NONBLOCK); + fdb = eventfd(0, EFD_NONBLOCK); + + CHECK(fda >= 0 && fdb >= 0); + + ret = io_uring_register_eventfd(&ring, fda); + if (ret) + return ret; + + CHECK(!can_read(fda)); + CHECK(!can_read(fdb)); + + io_uring_prep_poll_add(io_uring_get_sqe(&ring), fdb, POLLIN); + io_uring_submit(&ring); + CHECK(!can_read(fda)); /* poll should not have completed */ + + io_uring_prep_nop(io_uring_get_sqe(&ring)); + io_uring_submit(&ring); + CHECK(can_read(fda)); /* nop should have */ + + CHECK(io_uring_peek_cqe(&ring, &cqe) == 0); + CHECK(cqe->res == 0); + io_uring_cqe_seen(&ring, cqe); + eventfd_clear(fda); + + eventfd_trigger(fdb); + /* can take time due to rcu_call */ + CHECK(can_read_t(fda, 1000)); + + /* should not have processed the cqe yet */ + CHECK(io_uring_cq_ready(&ring) == 0); + + io_uring_get_events(&ring); + CHECK(io_uring_cq_ready(&ring) == 1); + + + io_uring_queue_exit(&ring); + return 0; +} + +struct thread_data { + struct io_uring ring; + int efd; + char buff[8]; +}; + +void *thread(void *t) +{ + struct thread_data *td = t; + + io_uring_prep_read(io_uring_get_sqe(&td->ring), td->efd, td->buff, sizeof(td->buff), 0); + io_uring_submit(&td->ring); + + return NULL; +} + +static int test_thread_shutdown(void) +{ + pthread_t t1; + int ret; + struct thread_data td; + struct io_uring_cqe *cqe; + uint64_t val = 1; + + ret = io_uring_queue_init(8, &td.ring, IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN); + if (ret) + return ret; + + /* check that even before submitting we don't get errors */ + CHECK(io_uring_get_events(&td.ring) == 0); + + td.efd = eventfd(0, 0); + CHECK(td.efd >= 0); + + CHECK(pthread_create(&t1, NULL, thread, &td) == 0); + CHECK(pthread_join(t1, NULL) == 0); + + CHECK(write(td.efd, &val, sizeof(val)) == sizeof(val)); + CHECK(io_uring_wait_cqe(&td.ring, &cqe) == -EEXIST); + + close(td.efd); + io_uring_queue_exit(&td.ring); + return 0; +} + +static int test_exec(const char *filename) +{ + int ret; + int fd; + struct io_uring ring; + pid_t fork_pid; + static char * const new_argv[] = {"1", "2", "3", NULL}; + static char * const new_env[] = {NULL}; + char *buff; + + fork_pid = fork(); + CHECK(fork_pid >= 0); + if (fork_pid > 0) { + int wstatus; + + CHECK(waitpid(fork_pid, &wstatus, 0) != (pid_t)-1); + if (!WIFEXITED(wstatus) || WEXITSTATUS(wstatus) != T_EXIT_SKIP) { + fprintf(stderr, "child failed %i\n", WEXITSTATUS(wstatus)); + return -1; + } + return 0; + } + + ret = io_uring_queue_init(8, &ring, IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN); + if (ret) + return ret; + + if (filename) { + fd = open(filename, O_RDONLY | O_DIRECT); + } else { + t_create_file(EXEC_FILENAME, EXEC_FILESIZE); + fd = open(EXEC_FILENAME, O_RDONLY | O_DIRECT); + unlink(EXEC_FILENAME); + } + buff = (char*)malloc(EXEC_FILESIZE); + CHECK(posix_memalign((void **)&buff, 4096, EXEC_FILESIZE) == 0); + CHECK(buff); + + CHECK(fd >= 0); + io_uring_prep_read(io_uring_get_sqe(&ring), fd, buff, EXEC_FILESIZE, 0); + io_uring_submit(&ring); + ret = execve("/proc/self/exe", new_argv, new_env); + /* if we get here it failed anyway */ + fprintf(stderr, "execve failed %d\n", ret); + return -1; +} + +static int test_flag(void) +{ + struct io_uring ring; + int ret; + int fd; + struct io_uring_cqe *cqe; + + ret = io_uring_queue_init(8, &ring, IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN | + IORING_SETUP_TASKRUN_FLAG); + CHECK(!ret); + + fd = eventfd(0, EFD_NONBLOCK); + CHECK(fd >= 0); + + io_uring_prep_poll_add(io_uring_get_sqe(&ring), fd, POLLIN); + io_uring_submit(&ring); + CHECK(!can_read(fd)); /* poll should not have completed */ + + eventfd_trigger(fd); + CHECK(can_read(fd)); + + /* should not have processed the poll cqe yet */ + CHECK(io_uring_cq_ready(&ring) == 0); + + /* flag should be set */ + CHECK(IO_URING_READ_ONCE(*ring.sq.kflags) & IORING_SQ_TASKRUN); + + /* Specifically peek, knowing we have only no cqe + * but because the flag is set, liburing should try and get more + */ + ret = io_uring_peek_cqe(&ring, &cqe); + + CHECK(ret == 0 && cqe); + CHECK(!(IO_URING_READ_ONCE(*ring.sq.kflags) & IORING_SQ_TASKRUN)); + + close(fd); + io_uring_queue_exit(&ring); + return 0; +} + +static int test_ring_shutdown(void) +{ + struct io_uring ring; + int ret; + int fd[2]; + char buff = '\0'; + char send = 'X'; + + ret = io_uring_queue_init(8, &ring, IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN | + IORING_SETUP_TASKRUN_FLAG); + CHECK(!ret); + + ret = t_create_socket_pair(fd, true); + CHECK(!ret); + + io_uring_prep_recv(io_uring_get_sqe(&ring), fd[0], &buff, 1, 0); + io_uring_submit(&ring); + + ret = write(fd[1], &send, 1); + CHECK(ret == 1); + + /* should not have processed the poll cqe yet */ + CHECK(io_uring_cq_ready(&ring) == 0); + io_uring_queue_exit(&ring); + + /* task work should have been processed by now */ + CHECK(buff = 'X'); + + return 0; +} + +int main(int argc, char *argv[]) +{ + int ret; + const char *filename = NULL; + + if (argc > 2) + return T_EXIT_SKIP; + if (argc == 2) { + /* This test exposes interesting behaviour with a null-blk + * device configured like: + * $ modprobe null-blk completion_nsec=100000000 irqmode=2 + * and then run with $ defer-taskrun.t /dev/nullb0 + */ + filename = argv[1]; + } + + if (!t_probe_defer_taskrun()) + return T_EXIT_SKIP; + + ret = test_thread_shutdown(); + if (ret) { + fprintf(stderr, "test_thread_shutdown failed\n"); + return T_EXIT_FAIL; + } + + ret = test_exec(filename); + if (ret) { + fprintf(stderr, "test_exec failed\n"); + return T_EXIT_FAIL; + } + + ret = test_eventfd(); + if (ret) { + fprintf(stderr, "eventfd failed\n"); + return T_EXIT_FAIL; + } + + ret = test_flag(); + if (ret) { + fprintf(stderr, "flag failed\n"); + return T_EXIT_FAIL; + } + + ret = test_ring_shutdown(); + if (ret) { + fprintf(stderr, "test_ring_shutdown failed\n"); + return T_EXIT_FAIL; + } + + return T_EXIT_PASS; +} From patchwork Mon Sep 5 13:22:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67CDAECAAD3 for ; Mon, 5 Sep 2022 13:27:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237748AbiIEN1b (ORCPT ); Mon, 5 Sep 2022 09:27:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237738AbiIEN1a (ORCPT ); Mon, 5 Sep 2022 09:27:30 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 115CF4A112 for ; Mon, 5 Sep 2022 06:27:29 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 284MVFAp022261 for ; Mon, 5 Sep 2022 06:27:28 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=EK9NhxtaoyAmN4a9SiaR96Y291lR6GH5HqekGq5np2o=; b=QCgToWaIJ5gqkSIDihCTOchYpmaK1GGNTH4M8MLgX9jvz1Resunj924ql6iX3xY/p4PW hUObccwlkL0+n5Z6qOFR8L5xarjo+l3MWGok+OjH+5ICfLZFOw/GxJAE/6eWzyCUPZlB GA323sA3/GJldk6jC04tOKfya5T4tthKpr8= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jcgaeegwy-15 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:27:28 -0700 Received: from twshared14074.07.ash9.facebook.com (2620:10d:c085:208::f) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:27:27 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 434FD5AC5176; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 07/11] update io_uring_enter.2 docs for IORING_FEAT_NODROP Date: Mon, 5 Sep 2022 06:22:54 -0700 Message-ID: <20220905132258.1858915-8-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: FBF-COkEVvzZPPGlheY3nNWk3yrT1xLk X-Proofpoint-GUID: FBF-COkEVvzZPPGlheY3nNWk3yrT1xLk X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org The EBUSY docs are out of date, so update them for the IORING_FEAT_NODROP feature flag Signed-off-by: Dylan Yudaken --- man/io_uring_enter.2 | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/man/io_uring_enter.2 b/man/io_uring_enter.2 index 1a9311e20357..4d8d488b5c1f 100644 --- a/man/io_uring_enter.2 +++ b/man/io_uring_enter.2 @@ -1330,7 +1330,18 @@ is a valid file descriptor, but the io_uring ring is not in the right state for details on how to enable the ring. .TP .B EBUSY -The application is attempting to overcommit the number of requests it can have +If the +.B IORING_FEAT_NODROP +feature flag is set, then +.B EBUSY +will be returned if there were overflow entries, +.B IORING_ENTER_GETEVENTS +flag is set and not all of the overflow entries were able to be flushed to +the CQ ring. + +Without +.B IORING_FEAT_NODROP +the application is attempting to overcommit the number of requests it can have pending. The application should wait for some completions and try again. May occur if the application tries to queue more requests than we have room for in the CQ ring, or if the application attempts to wait for more events without From patchwork Mon Sep 5 13:22:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FBB3ECAAD3 for ; Mon, 5 Sep 2022 13:27:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237142AbiIEN1X (ORCPT ); Mon, 5 Sep 2022 09:27:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230314AbiIEN1W (ORCPT ); Mon, 5 Sep 2022 09:27:22 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 148EB48EB0 for ; Mon, 5 Sep 2022 06:27:22 -0700 (PDT) Received: from pps.filterd (m0148461.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 285D6P71011639 for ; Mon, 5 Sep 2022 06:27:21 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=vGyofClJsXWSVgIvO7mgfI8ilfFPtKCRWqY+kse1jg0=; b=SGoe3pn4Q83WDFMA9OGBpZs/ZueEjBMZu/+eYU4/jYFygBfLYI29ieZXu+fIsN/ky7nk Kqxev1qZ5MsyTKEk+bPzci8Bg/I80OI1zS8Drukqh4VPnDJiaaEoW9KPRoZXl5nliv+i rLtiwRycw4kT1d2QqeB3/5q5YDzfDsis4i4= Received: from maileast.thefacebook.com ([163.114.130.16]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jc41tgnb5-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:27:21 -0700 Received: from twshared8442.02.ash8.facebook.com (2620:10d:c0a8:1b::d) by mail.thefacebook.com (2620:10d:c0a8:82::f) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:27:18 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 4B2A45AC5178; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 08/11] add docs for overflow lost errors Date: Mon, 5 Sep 2022 06:22:55 -0700 Message-ID: <20220905132258.1858915-9-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-GUID: 5ktL2IEek8es-8sH7n8akqV2phknKsuS X-Proofpoint-ORIG-GUID: 5ktL2IEek8es-8sH7n8akqV2phknKsuS X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add man docs for return values indicating a CQE was lost. Signed-off-by: Dylan Yudaken --- man/io_uring_enter.2 | 11 +++++++++++ man/io_uring_setup.2 | 11 +++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/man/io_uring_enter.2 b/man/io_uring_enter.2 index 4d8d488b5c1f..3c4aa04016cf 100644 --- a/man/io_uring_enter.2 +++ b/man/io_uring_enter.2 @@ -1329,6 +1329,17 @@ is a valid file descriptor, but the io_uring ring is not in the right state .BR io_uring_register (2) for details on how to enable the ring. .TP +.B EBADR +At least one CQE was dropped even with the +.B IORING_FEAT_NODROP +feature, and there are no otherwise available CQEs. This clears the error state +and so with no other changes the next call to +.BR io_uring_setup (2) +will not have this error. This error should be extremely rare and indicates the +machine is running critically low on memory and. It may be reasonable for the +application to terminate running unless it is able to safely handle any CQE +being lost. +.TP .B EBUSY If the .B IORING_FEAT_NODROP diff --git a/man/io_uring_setup.2 b/man/io_uring_setup.2 index 01eb70d95292..41c9e5f269c5 100644 --- a/man/io_uring_setup.2 +++ b/man/io_uring_setup.2 @@ -298,7 +298,7 @@ call. The SQEs must still be allocated separately. This brings the necessary calls down from three to two. Available since kernel 5.4. .TP .B IORING_FEAT_NODROP -If this flag is set, io_uring supports never dropping completion events. +If this flag is set, io_uring supports almost never dropping completion events. If a completion event occurs and the CQ ring is full, the kernel stores the event internally until such a time that the CQ ring has room for more entries. If this overflow condition is entered, attempting to submit more @@ -306,7 +306,14 @@ IO will fail with the .B -EBUSY error value, if it can't flush the overflown events to the CQ ring. If this happens, the application must reap events from the CQ ring and attempt the -submit again. Available since kernel 5.5. +submit again. If the kernel has no free memory to store the event internally +it will be visible by an increase in the overflow value on the cqring. +Available since kernel 5.5. Additionally +.BR io_uring_enter (2) +will return +.B -EBADR +the next time it would otherwise sleep waiting for completions (since kernel 5.19). + .TP .B IORING_FEAT_SUBMIT_STABLE If this flag is set, applications can be certain that any data for From patchwork Mon Sep 5 13:22:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2C90C54EE9 for ; Mon, 5 Sep 2022 13:27:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237618AbiIEN1Y (ORCPT ); Mon, 5 Sep 2022 09:27:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235444AbiIEN1W (ORCPT ); Mon, 5 Sep 2022 09:27:22 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E141848E8E for ; Mon, 5 Sep 2022 06:27:21 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 284MVFAh022261 for ; Mon, 5 Sep 2022 06:27:21 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=ii840Xnp00Mo5nDn8OqcgKZ9ucEMuml7pYD+plSRjno=; b=qXXPVDMIB2vjPvMxzcsb6i/I9BM4XW8K/zkgXM+xBCEoLn+rC2kznA2oKr/BwcQfjfwA aTuBxZQZy/Oqk2SK7gef4jJDFj8llq69+cD1MW3GqRqDcRRZ1M6OVfxaj8HX1hUEwEmk AfWQ7HtUn6KH88EmweIhTY28wZnUT27MIwg= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jcgaeegwy-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:27:21 -0700 Received: from twshared3888.09.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:27:20 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 5166D5AC517B; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 09/11] expose CQ ring overflow state Date: Mon, 5 Sep 2022 06:22:56 -0700 Message-ID: <20220905132258.1858915-10-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: pfo8JoBye0edUtGZrl_eYtg9LLcS8DX_ X-Proofpoint-GUID: pfo8JoBye0edUtGZrl_eYtg9LLcS8DX_ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Allow the application to easily view if the CQ ring is in overflow state in case it would like to explicitly flush using io_uring_get_events. Explicit flushing can be useful for applications that prefer to have reduced latency on CQs than to process as many as possible. Signed-off-by: Dylan Yudaken --- man/io_uring_cq_has_overflow.3 | 25 +++++++++++++++++++++++++ man/io_uring_get_events.3 | 3 ++- src/include/liburing.h | 9 +++++++++ 3 files changed, 36 insertions(+), 1 deletion(-) create mode 100644 man/io_uring_cq_has_overflow.3 diff --git a/man/io_uring_cq_has_overflow.3 b/man/io_uring_cq_has_overflow.3 new file mode 100644 index 000000000000..e5b352a4f86a --- /dev/null +++ b/man/io_uring_cq_has_overflow.3 @@ -0,0 +1,25 @@ +.\" Copyright (C) 2022 Dylan Yudaken +.\" +.\" SPDX-License-Identifier: LGPL-2.0-or-later +.\" +.TH io_uring_cq_has_overflow 3 "September 5, 2022" "liburing-2.3" "liburing Manual" +.SH NAME +io_uring_cq_has_overflow \- returns if there are overflow entries waiting to move to the CQ ring +.SH SYNOPSIS +.nf +.B #include +.PP +.BI "bool io_uring_cq_has_overflow(const struct io_uring *" ring ");" +.fi +.SH DESCRIPTION +.PP +The +.BR io_uring_cq_has_overflow (3) +function informs the application if CQ entries have overflowed and are waiting to be flushed to +the CQ ring. For example using +.BR io_uring_get_events (3) +. +.SH RETURN VALUE +True if there are CQ entries waiting to be flushed to the CQ ring. +.SH SEE ALSO +.BR io_uring_get_events (3) diff --git a/man/io_uring_get_events.3 b/man/io_uring_get_events.3 index 2ac3e070473e..f2415423953c 100644 --- a/man/io_uring_get_events.3 +++ b/man/io_uring_get_events.3 @@ -29,4 +29,5 @@ returns 0. On failure it returns .BR -errno . .SH SEE ALSO .BR io_uring_get_sqe (3), -.BR io_uring_submit_and_get_events (3) +.BR io_uring_submit_and_get_events (3), +.BR io_uring_cq_has_overflow (3) diff --git a/src/include/liburing.h b/src/include/liburing.h index 765375cb5c6a..ae25e2199264 100644 --- a/src/include/liburing.h +++ b/src/include/liburing.h @@ -1098,6 +1098,15 @@ static inline unsigned io_uring_cq_ready(const struct io_uring *ring) return io_uring_smp_load_acquire(ring->cq.ktail) - *ring->cq.khead; } +/* + * Returns true if there are overflow entries waiting to be flushed onto + * the CQ ring + */ +static inline bool io_uring_cq_has_overflow(const struct io_uring *ring) +{ + return IO_URING_READ_ONCE(*ring->sq.kflags) & IORING_SQ_CQ_OVERFLOW; +} + /* * Returns true if the eventfd notification is currently enabled */ From patchwork Mon Sep 5 13:22:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD905ECAAD5 for ; Mon, 5 Sep 2022 13:27:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237251AbiIEN1j (ORCPT ); Mon, 5 Sep 2022 09:27:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237798AbiIEN1f (ORCPT ); Mon, 5 Sep 2022 09:27:35 -0400 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33D2A4A800 for ; Mon, 5 Sep 2022 06:27:33 -0700 (PDT) Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 285D1Slm025577 for ; Mon, 5 Sep 2022 06:27:32 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=fz5pnw/sUz497BcBgEqTIuHhjU7Ehd04Bem7rgx4F+I=; b=V6ny8Vog4jypm8PQ3vGdUFcdukiAuec+p8eji9El3PEe5oHoLsRQrlencdxLuth1tzp9 iK4QqiIWMYPJlD4ZwJIwLX4A5honHfb/iOBmHw217mQZXpA+KoYaGaQ2h8pWfBxVVVAr aI8S46i6gOJKKrJ3zypuZzXXbIaSnFgxzcM= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jc4p00kpk-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:27:32 -0700 Received: from twshared8288.05.ash9.facebook.com (2620:10d:c085:108::4) by mail.thefacebook.com (2620:10d:c085:21d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:27:30 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 57E645AC517D; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 10/11] overflow: add tests Date: Mon, 5 Sep 2022 06:22:57 -0700 Message-ID: <20220905132258.1858915-11-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: Hjtg-0Flf8cK7uP4AUbcAwY7jtN6tQGH X-Proofpoint-GUID: Hjtg-0Flf8cK7uP4AUbcAwY7jtN6tQGH X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add tests that verify that overflow conditions behave appropriately. Specifically: * if overflow is continually flushed, then CQEs should arrive mostly in order to prevent starvation of some completions * if CQEs are dropped due to GFP_ATOMIC allocation failures it is possible to terminate cleanly. This is not tested by default as it requires debug kernel config, and also has system-wide effects Signed-off-by: Dylan Yudaken --- test/cq-overflow.c | 243 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 236 insertions(+), 7 deletions(-) diff --git a/test/cq-overflow.c b/test/cq-overflow.c index 312b414b2a79..1087eefaacf0 100644 --- a/test/cq-overflow.c +++ b/test/cq-overflow.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "helpers.h" #include "liburing.h" @@ -21,6 +22,32 @@ static struct iovec *vecs; #define ENTRIES 8 +/* + * io_uring has rare cases where CQEs are lost. + * This happens when there is no space in the CQ ring, and also there is no + * GFP_ATOMIC memory available. In reality this probably means that the process + * is about to be killed as many other things might start failing, but we still + * want to test that liburing and the kernel deal with this properly. The fault + * injection framework allows us to test this scenario. Unfortunately this + * requires some system wide changes and so we do not enable this by default. + * The tests in this file should work in both cases (where overflows are queued + * and where they are dropped) on recent kernels. + * + * In order to test dropped CQEs you should enable fault injection in the kernel + * config: + * + * CONFIG_FAULT_INJECTION=y + * CONFIG_FAILSLAB=y + * CONFIG_FAULT_INJECTION_DEBUG_FS=y + * + * and then run the test as follows: + * echo Y > /sys/kernel/debug/failslab/task-filter + * echo 100 > /sys/kernel/debug/failslab/probability + * echo 0 > /sys/kernel/debug/failslab/verbose + * echo 100000 > /sys/kernel/debug/failslab/times + * bash -c "echo 1 > /proc/self/make-it-fail && exec ./cq-overflow.t" + */ + static int test_io(const char *file, unsigned long usecs, unsigned *drops, int fault) { struct io_uring_sqe *sqe; @@ -29,6 +56,7 @@ static int test_io(const char *file, unsigned long usecs, unsigned *drops, int f unsigned reaped, total; struct io_uring ring; int nodrop, i, fd, ret; + bool cqe_dropped = false; fd = open(file, O_RDONLY | O_DIRECT); if (fd < 0) { @@ -104,8 +132,8 @@ static int test_io(const char *file, unsigned long usecs, unsigned *drops, int f reap_it: reaped = 0; do { - if (nodrop) { - /* nodrop should never lose events */ + if (nodrop && !cqe_dropped) { + /* nodrop should never lose events unless cqe_dropped */ if (reaped == total) break; } else { @@ -113,7 +141,10 @@ reap_it: break; } ret = io_uring_wait_cqe(&ring, &cqe); - if (ret) { + if (nodrop && ret == -EBADR) { + cqe_dropped = true; + continue; + } else if (ret) { fprintf(stderr, "wait_cqe=%d\n", ret); goto err; } @@ -133,7 +164,7 @@ reap_it: goto err; } - if (!nodrop) { + if (!nodrop || cqe_dropped) { *drops = *ring.cq.koverflow; } else if (*ring.cq.koverflow) { fprintf(stderr, "Found %u overflows\n", *ring.cq.koverflow); @@ -154,18 +185,29 @@ static int reap_events(struct io_uring *ring, unsigned nr_events, int do_wait) { struct io_uring_cqe *cqe; int i, ret = 0, seq = 0; + unsigned int start_overflow = *ring->cq.koverflow; + bool dropped = false; for (i = 0; i < nr_events; i++) { if (do_wait) ret = io_uring_wait_cqe(ring, &cqe); else ret = io_uring_peek_cqe(ring, &cqe); - if (ret) { + if (do_wait && ret == -EBADR) { + unsigned int this_drop = *ring->cq.koverflow - + start_overflow; + + dropped = true; + start_overflow = *ring->cq.koverflow; + assert(this_drop > 0); + i += (this_drop - 1); + continue; + } else if (ret) { if (ret != -EAGAIN) fprintf(stderr, "cqe peek failed: %d\n", ret); break; } - if (cqe->user_data != seq) { + if (!dropped && cqe->user_data != seq) { fprintf(stderr, "cqe sequence out-of-order\n"); fprintf(stderr, "got %d, wanted %d\n", (int) cqe->user_data, seq); @@ -242,19 +284,206 @@ err: return 1; } + +static void submit_one_nop(struct io_uring *ring, int ud) +{ + struct io_uring_sqe *sqe; + int ret; + + sqe = io_uring_get_sqe(ring); + assert(sqe); + io_uring_prep_nop(sqe); + sqe->user_data = ud; + ret = io_uring_submit(ring); + assert(ret == 1); +} + +/* + * Create an overflow condition and ensure that SQEs are still processed + */ +static int test_overflow_handling(bool batch, int cqe_multiple, bool poll, + bool defer) +{ + struct io_uring ring; + struct io_uring_params p; + int ret, i, j, ud, cqe_count; + unsigned int count; + int const N = 8; + int const LOOPS = 128; + int const QUEUE_LENGTH = 1024; + int completions[N]; + int queue[QUEUE_LENGTH]; + int queued = 0; + int outstanding = 0; + bool cqe_dropped = false; + + memset(&completions, 0, sizeof(int) * N); + memset(&p, 0, sizeof(p)); + p.cq_entries = 2 * cqe_multiple; + p.flags |= IORING_SETUP_CQSIZE; + + if (poll) + p.flags |= IORING_SETUP_IOPOLL; + + if (defer) + p.flags |= IORING_SETUP_SINGLE_ISSUER | + IORING_SETUP_DEFER_TASKRUN; + + ret = io_uring_queue_init_params(2, &ring, &p); + if (ret) { + fprintf(stderr, "io_uring_queue_init failed %d\n", ret); + return 1; + } + + assert(p.cq_entries < N); + /* submit N SQEs, some should overflow */ + for (i = 0; i < N; i++) { + submit_one_nop(&ring, i); + outstanding++; + } + + for (i = 0; i < LOOPS; i++) { + struct io_uring_cqe *cqes[N]; + + if (io_uring_cq_has_overflow(&ring)) { + /* + * Flush any overflowed CQEs and process those. Actively + * flush these to make sure CQEs arrive in vague order + * of being sent. + */ + ret = io_uring_get_events(&ring); + if (ret != 0) { + fprintf(stderr, + "io_uring_get_events returned %d\n", + ret); + goto err; + } + } else if (!cqe_dropped) { + for (j = 0; j < queued; j++) { + submit_one_nop(&ring, queue[j]); + outstanding++; + } + queued = 0; + } + + /* We have lost some random cqes, stop if no remaining. */ + if (cqe_dropped && outstanding == *ring.cq.koverflow) + break; + + ret = io_uring_wait_cqe(&ring, &cqes[0]); + if (ret == -EBADR) { + cqe_dropped = true; + fprintf(stderr, "CQE dropped\n"); + continue; + } else if (ret != 0) { + fprintf(stderr, "io_uring_wait_cqes failed %d\n", ret); + goto err; + } + cqe_count = 1; + if (batch) { + ret = io_uring_peek_batch_cqe(&ring, &cqes[0], 2); + if (ret < 0) { + fprintf(stderr, + "io_uring_peek_batch_cqe failed %d\n", + ret); + goto err; + } + cqe_count = ret; + } + for (j = 0; j < cqe_count; j++) { + assert(cqes[j]->user_data < N); + ud = cqes[j]->user_data; + completions[ud]++; + assert(queued < QUEUE_LENGTH); + queue[queued++] = (int)ud; + } + io_uring_cq_advance(&ring, cqe_count); + outstanding -= cqe_count; + } + + /* See if there were any drops by flushing the CQ ring *and* overflow */ + do { + struct io_uring_cqe *cqe; + + ret = io_uring_get_events(&ring); + if (ret < 0) { + if (ret == -EBADR) { + fprintf(stderr, "CQE dropped\n"); + cqe_dropped = true; + break; + } + goto err; + } + if (outstanding && !io_uring_cq_ready(&ring)) + ret = io_uring_wait_cqe_timeout(&ring, &cqe, NULL); + + if (ret && ret != -ETIME) { + if (ret == -EBADR) { + fprintf(stderr, "CQE dropped\n"); + cqe_dropped = true; + break; + } + fprintf(stderr, "wait_cqe_timeout = %d\n", ret); + goto err; + } + count = io_uring_cq_ready(&ring); + io_uring_cq_advance(&ring, count); + outstanding -= count; + } while (count); + + io_uring_queue_exit(&ring); + + /* Make sure that completions come back in the same order they were + * sent. If they come back unfairly then this will concentrate on a + * couple of indices. + */ + for (i = 1; !cqe_dropped && i < N; i++) { + if (abs(completions[i] - completions[i - 1]) > 1) { + fprintf(stderr, "bad completion size %d %d\n", + completions[i], completions[i - 1]); + goto err; + } + } + return 0; +err: + io_uring_queue_exit(&ring); + return 1; +} + int main(int argc, char *argv[]) { const char *fname = ".cq-overflow"; unsigned iters, drops; unsigned long usecs; int ret; + int i; + bool can_defer; if (argc > 1) return T_EXIT_SKIP; + can_defer = t_probe_defer_taskrun(); + for (i = 0; i < 16; i++) { + bool batch = i & 1; + int mult = (i & 2) ? 1 : 2; + bool poll = i & 4; + bool defer = i & 8; + + if (defer && !can_defer) + continue; + + ret = test_overflow_handling(batch, mult, poll, defer); + if (ret) { + fprintf(stderr, "test_overflow_handling(" + "batch=%d, mult=%d, poll=%d, defer=%d) failed\n", + batch, mult, poll, defer); + goto err; + } + } + ret = test_overflow(); if (ret) { - printf("test_overflow failed\n"); + fprintf(stderr, "test_overflow failed\n"); return ret; } From patchwork Mon Sep 5 13:22:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dylan Yudaken X-Patchwork-Id: 12966133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6579CECAAD3 for ; Mon, 5 Sep 2022 13:27:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237726AbiIEN11 (ORCPT ); Mon, 5 Sep 2022 09:27:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237705AbiIEN1Z (ORCPT ); Mon, 5 Sep 2022 09:27:25 -0400 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DF5A049B61 for ; Mon, 5 Sep 2022 06:27:24 -0700 (PDT) Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 284MVFAl022261 for ; Mon, 5 Sep 2022 06:27:24 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=facebook; bh=ac2oEHDxsKQWx0CIIypZ0l2ErwqOmMqV+RiizuiRdmE=; b=MDSkCcvSsxMY+pdt8AI2IokOlh/B+WtQIZTETAhHDm+dD9UbBGe+GHqgozJj6dOVgpf4 6IGLDeTOexO17KV7rNSpQa8fU42Jvb61hdoKDjuW1yIig85+kY0IiiNlrqp9++v56dKU 8JoNs5eeXrU8cOk+Mlm0wfx78X0a1BESOvk= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3jcgaeegwy-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Mon, 05 Sep 2022 06:27:24 -0700 Received: from twshared3888.09.ash9.facebook.com (2620:10d:c085:208::11) by mail.thefacebook.com (2620:10d:c085:11d::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Mon, 5 Sep 2022 06:27:22 -0700 Received: by devbig038.lla2.facebook.com (Postfix, from userid 572232) id 5E5D85AC517F; Mon, 5 Sep 2022 06:24:50 -0700 (PDT) From: Dylan Yudaken To: , CC: , , Dylan Yudaken Subject: [PATCH liburing v3 11/11] file-verify test: log if short read Date: Mon, 5 Sep 2022 06:22:58 -0700 Message-ID: <20220905132258.1858915-12-dylany@fb.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220905132258.1858915-1-dylany@fb.com> References: <20220905132258.1858915-1-dylany@fb.com> MIME-Version: 1.0 X-FB-Internal: Safe X-Proofpoint-ORIG-GUID: XlFvKfp6LSywIuJWRirHbYlh7iSzC5DR X-Proofpoint-GUID: XlFvKfp6LSywIuJWRirHbYlh7iSzC5DR X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.517,FMLib:17.11.122.1 definitions=2022-09-05_09,2022-09-05_02,2022-06-22_01 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org This test assumes any success is a full read. If it was a short read the test would still fail (as verification would fail) but tracking down the reason can be annoying. For now we can just log if it's short and fail at that point. Signed-off-by: Dylan Yudaken --- test/file-verify.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/file-verify.c b/test/file-verify.c index 595dafd4bd8d..3a395f9dccda 100644 --- a/test/file-verify.c +++ b/test/file-verify.c @@ -430,6 +430,10 @@ static int test(struct io_uring *ring, const char *fname, int buffered, fprintf(stderr, "bad read %d, read %d\n", cqe->res, i); goto err; } + if (cqe->res < CHUNK_SIZE) { + fprintf(stderr, "short read %d, read %d\n", cqe->res, i); + goto err; + } if (cqe->flags & IORING_CQE_F_BUFFER) index = cqe->flags >> 16; else