From patchwork Tue Jul 30 20:29:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13747849 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34C0718CC1F for ; Tue, 30 Jul 2024 20:29:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722371367; cv=none; b=i7/FvORK8ldm+R+1/qgu+8Q+EE0Pxnq4pu0m6n+NivCrNqF5Ogynt6i/Qc8AcVuTjdI7eIuDK9wuMAsIMdbuZ6MNARz8SD10Berg4vt+8b92XSxVoh1WItFLiUSiaVaPwY8Gey9Y6d5kfWOVpib540tjg5Opbn/bw5nhFIgoT0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722371367; c=relaxed/simple; bh=yMnMp6sYGz4/6Z7wV25uL/n2ygpp95FpW6CIeQqXrF8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dHQ28em/yOn9ZnrSCbQTaLA4Srzcgc47Uv6wHw47IgYPwX9ilmaiqJs+RdeZ3jDPGCLwY3PAURA5KJy0U/c+sJTD0zKzMgQlshki9kk6SzCKSxMnEGtx2G+tFHMeApI6AkY9WufErQuzBLaE5w9lZz2BaJESqRUgNbc8axpRn4E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Gx/kbh4u; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Gx/kbh4u" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-42808071810so31122055e9.1 for ; Tue, 30 Jul 2024 13:29:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722371363; x=1722976163; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aCDFSYLRz664AJ+uUNg7sg3G8Ma7AxzpoUVwna2Z1b0=; b=Gx/kbh4uNzrDgx1w4dQkdHjkOB76F2NtwtWaqBJMJms5sEVj9iGFMUp3joXjziUeK4 fFoLYUc/k9J8CTpHEMeVAPrk3DVsnLCOcZvymngeXUJ0u+pkJUMiss69rxmtflOb/Ce2 SeiF0VimHtxa74aFdJ6KmnHL/jN10c81Grp39G6ZygyVcfZRfsuAW1+4zvG/EYhk95x9 EXgjxKpCUW8CxO7dfeIDS1xmtZDWEsmyJxxrHjsohfIyYFtQQX7HXbQb2LnBdCl+7y2H fge3QS+bUY3Xkgk18YglMAsDU1kI/aeHW/JNabyRopxeRMXvKGVNcUZciDM11A5OCMLp NAKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722371363; x=1722976163; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aCDFSYLRz664AJ+uUNg7sg3G8Ma7AxzpoUVwna2Z1b0=; b=wsXKwSbDOSddvTlcNTSMMC89qTJmcWJk8so4Gz4QJxlw+e5HjvUCsfN+JaFxqLckXN DLprsJPYZnk1ILgGGQeQ2kxYzVstIDQTIHzbuFb04CzWAp0A7f0VE6R//oZjqMJEebeu JiUeCp9XNKkUB4c49V4Gks+2Gl7cztV7UEjqCtXt0NsbMnuDSSJlRhYAqs4fpTTGfF7L z/HxQ5Y9CkY+Syj7GOtFhDc24G4LeL3QOUClL5gZTzkof2Tq368zeQ08wwMM6Y7JtsQg rD9DpsY9sBFX7XUp7Nla8+gCsuakPnoZqhY2Mdm4BIcfMsOK7WQiCsCia4ECShZe/y3d 6Y5A== X-Gm-Message-State: AOJu0YxQ8+Zm/Ubx4qlwyDz32YIKP0LwML7YL2KEbrjRbPoqvVQSmCdb L/x+vIWYkYHtf6oChbjxDsKHV7NgRGwAt1rBQPWxKNSAGKNHYWJ/LwPYgg== X-Google-Smtp-Source: AGHT+IHUhBZQXAYok0Vt9EZ5vQ0hNJz0JyYRgiOvPbcYkfYDFTgOXVIVEWdXKVzRDZkpspENfdjmqw== X-Received: by 2002:a05:600c:6c4c:b0:426:5f06:5462 with SMTP id 5b1f17b1804b1-42811e422e4mr71058665e9.37.1722371362714; Tue, 30 Jul 2024 13:29:22 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.235.94]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42808457c7fsm214488065e9.32.2024.07.30.13.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 13:29:22 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH 1/3] io_uring/napi: refactor __io_napi_busy_loop() Date: Tue, 30 Jul 2024 21:29:42 +0100 Message-ID: X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 we don't need to set ->napi_prefer_busy_poll if we're not going to poll, do the checks first and all polling preparation after. Signed-off-by: Pavel Begunkov --- io_uring/napi.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/io_uring/napi.c b/io_uring/napi.c index 4fd6bb331e1e..a670f49e30ef 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -301,10 +301,11 @@ void __io_napi_adjust_timeout(struct io_ring_ctx *ctx, struct io_wait_queue *iow */ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { - iowq->napi_prefer_busy_poll = READ_ONCE(ctx->napi_prefer_busy_poll); + if ((ctx->flags & IORING_SETUP_SQPOLL) || !ctx->napi_enabled) + return; - if (!(ctx->flags & IORING_SETUP_SQPOLL) && ctx->napi_enabled) - io_napi_blocking_busy_loop(ctx, iowq); + iowq->napi_prefer_busy_poll = READ_ONCE(ctx->napi_prefer_busy_poll); + io_napi_blocking_busy_loop(ctx, iowq); } /* From patchwork Tue Jul 30 20:29:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13747850 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9579918DF7D for ; Tue, 30 Jul 2024 20:29:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722371368; cv=none; b=RVtCVnvYCUIz/YilUNgb6FK2isHvB6NOaY991+S8SovigXfP9zzEKicLpsy03oXiDCvx4ujUn3f8k5YGP+XwAuPm5ixC1X2POTNLHsnING9Ptn/BQ4kXrF2RDVPJH3s2y6IeNIHhY660y2lLEJR+gDWakzvCiw3bwlIrBzPBLrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722371368; c=relaxed/simple; bh=zovKaqxO0iZW2msDKkeqZYOYPS1G0opJSaZ5TvPDHgo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ixq+Ew0jGOShHABHJ6KZAV1gZVN4VRq89selXC9UH+TDXlhAOquUwQYzgfenTs6VbPItkhhtgAqLVmxY2A/Cp/CGw0kXZxSSQaI3ZKSHxyZTiWRVWdBxLOYBssKpY7AJxOZz7FW8jeGk1GmzQ1E5QKR2KAMKQ+jq/is4aiE29wA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bwO2s8VF; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bwO2s8VF" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-427b1d4da32so1410845e9.0 for ; Tue, 30 Jul 2024 13:29:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722371364; x=1722976164; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WsQgCMWM9FHgRe4Z+rQk9mNDKnWifQacOoNk8pNYn8U=; b=bwO2s8VFbBAGGjWJ1w1Cg80v/HFVzzEwp+aWEzHPoQaMnQvfKjONSV1twT5jSu3yIJ ulMTDH67/XkZE+0kzmhrj274yj0fYZRZv4bf0IIC+BMn7flLdTiYIGkXj4hv8uFD9vqz DzNUjYKYXyxO2qip1kVNrgSaXMtbuPb82GhIUQAFVtawVaxIWSMFrI+fXHLSvT72LL8Y 70hKEws9iiH5TzVqe6Bg3y3D+rtKtsz67mO75HqsKzRNdVRJFLyRc+LitdsErnETz2R4 2dClx+42WBN7jD+FBH0e72zmkXv8lhBIZLLVqqrVwMV7sfodWFt3C9m+L+yXgXfRuCsp RZOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722371364; x=1722976164; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WsQgCMWM9FHgRe4Z+rQk9mNDKnWifQacOoNk8pNYn8U=; b=hNhs1jLTM/he6BaWEJlliIcr4x5dqTwH9q3JkGc3kGRp7mnEWAepaImuhntQZavjqb GrzPaVGMutie45x4GvrqpdInfAIijgFroD5VN75zHVJxeYFMgI4SaCfWdsfHJ8l2x9Rw L8X+ZdPP6dbkjJgwQInenX3/kcnyIe5djtinevITyrCO9BRPM77+D9uPi4zo2qJMhydb e4rIzsNZkqgalAxzjDlMlugk2Qb5J/KyLmz8LLmExn+e4sQ4yfOvCM1wrJrNl5tFW9Hp WEJyAF1vfyz3z5ddB8xEpJPuXxZ0Oqr2A80lxSjfSm2dSssobhhLk62I9PMbpvYX+KWV Ej5A== X-Gm-Message-State: AOJu0Yy9v0gapgxbHsQoZLAUZChkT/MiP49mU3/4r3wQx955p/Ioehl4 UySYjSpHks16lg9ntkvn4HSemAhdKxS/By+HcY4GPa08YYu3QQSlgfs+7A== X-Google-Smtp-Source: AGHT+IF2PcvlXs4+keKHEMyx0npqCgE8zPdZ3xZdTFhoDxRvctH4z2Yso3CvCX6JwA28OWMnxBb2Lw== X-Received: by 2002:a05:600c:a41:b0:428:9c0:2ad with SMTP id 5b1f17b1804b1-4282445d0c0mr19886095e9.18.1722371364248; Tue, 30 Jul 2024 13:29:24 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.235.94]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42808457c7fsm214488065e9.32.2024.07.30.13.29.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 13:29:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH 2/3] io_uring/napi: delay napi timeout adjustment Date: Tue, 30 Jul 2024 21:29:43 +0100 Message-ID: <2766891d83b5d1ff231210ee3d09387398ef29cc.1722357468.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Instead adjusting busy polling time in io_cqring_wait(), rely on the deadline value and delay the check until io_napi_busy_loop_timeout() is called inside the napi busy polling loop. There is a side effect of comparing a cpu local clock with ktime_get(), however waiting timeouts are usually long enough not to care and napi.c is already careless about mixing time flavours, i.e. io_napi_blocking_busy_loop() gets a cpu local time before disabling preemption. Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 1 - io_uring/napi.c | 30 ++++++------------------------ io_uring/napi.h | 16 ---------------- 3 files changed, 6 insertions(+), 41 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3942db160f18..9ec07f76ad19 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2423,7 +2423,6 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, dt = timespec64_to_ktime(ts); iowq.timeout = ktime_add(dt, ktime_get()); - io_napi_adjust_timeout(ctx, &iowq, dt); } if (sig) { diff --git a/io_uring/napi.c b/io_uring/napi.c index a670f49e30ef..c5c1177e2fb4 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -109,12 +109,15 @@ static inline void io_napi_remove_stale(struct io_ring_ctx *ctx, bool is_stale) } static inline bool io_napi_busy_loop_timeout(ktime_t start_time, - ktime_t bp) + struct io_wait_queue *iowq) { + ktime_t bp = iowq->napi_busy_poll_dt; + if (bp) { ktime_t end_time = ktime_add(start_time, bp); ktime_t now = net_to_ktime(busy_loop_current_time()); + end_time = min(end_time, iowq->timeout); return ktime_after(now, end_time); } @@ -130,8 +133,7 @@ static bool io_napi_busy_loop_should_end(void *data, return true; if (io_should_wake(iowq) || io_has_work(iowq->ctx)) return true; - if (io_napi_busy_loop_timeout(net_to_ktime(start_time), - iowq->napi_busy_poll_dt)) + if (io_napi_busy_loop_timeout(net_to_ktime(start_time), iowq)) return true; return false; @@ -271,27 +273,6 @@ int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg) return 0; } -/* - * __io_napi_adjust_timeout() - adjust busy loop timeout - * @ctx: pointer to io-uring context structure - * @iowq: pointer to io wait queue - * @ts: pointer to timespec or NULL - * - * Adjust the busy loop timeout according to timespec and busy poll timeout. - * If the specified NAPI timeout is bigger than the wait timeout, then adjust - * the NAPI timeout accordingly. - */ -void __io_napi_adjust_timeout(struct io_ring_ctx *ctx, struct io_wait_queue *iowq, - ktime_t to_wait) -{ - ktime_t poll_dt = READ_ONCE(ctx->napi_busy_poll_dt); - - if (to_wait) - poll_dt = min(poll_dt, to_wait); - - iowq->napi_busy_poll_dt = poll_dt; -} - /* * __io_napi_busy_loop() - execute busy poll loop * @ctx: pointer to io-uring context structure @@ -304,6 +285,7 @@ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) if ((ctx->flags & IORING_SETUP_SQPOLL) || !ctx->napi_enabled) return; + iowq->napi_busy_poll_dt = READ_ONCE(ctx->napi_busy_poll_dt); iowq->napi_prefer_busy_poll = READ_ONCE(ctx->napi_prefer_busy_poll); io_napi_blocking_busy_loop(ctx, iowq); } diff --git a/io_uring/napi.h b/io_uring/napi.h index 88f1c21d5548..87e30b4f8d9e 100644 --- a/io_uring/napi.h +++ b/io_uring/napi.h @@ -17,8 +17,6 @@ int io_unregister_napi(struct io_ring_ctx *ctx, void __user *arg); void __io_napi_add(struct io_ring_ctx *ctx, struct socket *sock); -void __io_napi_adjust_timeout(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, ktime_t to_wait); void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq); int io_napi_sqpoll_busy_poll(struct io_ring_ctx *ctx); @@ -27,15 +25,6 @@ static inline bool io_napi(struct io_ring_ctx *ctx) return !list_empty(&ctx->napi_list); } -static inline void io_napi_adjust_timeout(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, - ktime_t to_wait) -{ - if (!io_napi(ctx)) - return; - __io_napi_adjust_timeout(ctx, iowq, to_wait); -} - static inline void io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { @@ -86,11 +75,6 @@ static inline bool io_napi(struct io_ring_ctx *ctx) static inline void io_napi_add(struct io_kiocb *req) { } -static inline void io_napi_adjust_timeout(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq, - ktime_t to_wait) -{ -} static inline void io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { From patchwork Tue Jul 30 20:29:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 13747851 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D666618DF9A for ; Tue, 30 Jul 2024 20:29:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722371369; cv=none; b=QQpGDNSbFDGgWhN/JYGOyYKPijQZfaXkoA7vOp7Ai1vzq4u31MDXWiuKGtS5Tg0ENDU91kKFYIIqe9FMvWBeNF+ST/CEzUm1Aq0yblxkabbQQXkhuNXmV8p8ygvQwquapRFrZFhhm6gwYfgD9oEJqvfOeibAhKh6V4cw7r4RvZs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722371369; c=relaxed/simple; bh=JZOOWWe04FmpSsGhdrgPhCnpAbAOD6Rq2mMhZw+BKjs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uEhHuiC4waSaQhyotR88Uo4R5BvJwve9iBoOeLvHhLACBGDftA/fl7WkDLQTBNxq9voER8sGQhP/EdGBDFhRv+W5zKbOAOmN2WhbMU1YLpz9Uf9wlKaDYtbfBaZhmRPhMsKU91897X6pjyMKTTrzF9WMteMGFTpILDSRzucva8I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bdZf2IGh; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bdZf2IGh" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-428243f928fso13266165e9.0 for ; Tue, 30 Jul 2024 13:29:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722371366; x=1722976166; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xb8u2LzTqPXQGpAJX7Eawxnuj0oeSkULgKH2OZy+MqQ=; b=bdZf2IGhhWUq1vfE+U1V//rnjSwPgO/D5afOHe3MHePFaG/GYrcahmk7rFXYmPZZgA V3NepThcyv8iY0Iny5+d17KD2Tv8S21qvV04k4RovLOqlNHYT1dLKl4HkZn1bWZN8ueR Dfprs3cr0+D+3nNni5LnU9Zfmqs2Z6L0TPvLFTkeNHl1NkVDUzoQzIVwnC3VnTEyQlBx XL2SUmciWJ3YEclMhJ8TebD4f5741CxN3RNi21jCa3x1GxCvvOnZ24HdC1AhiU6WpAPM wzD80zL3gamMvZqZRI7BmarulhcizH1TXHdoMSzOYC1XlKalOsKSxdW9AtBFBuBoXFuB NzXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722371366; x=1722976166; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xb8u2LzTqPXQGpAJX7Eawxnuj0oeSkULgKH2OZy+MqQ=; b=IrBlm+t8YfrGPv7LuJol2bqaclcAGDSL7Tode+vj5s/kQ/W/Q7Ppsjb28/Fu671vp8 30xQ1tBSogGt9b4vLhTL18SbYasA90Xb1I3Co5lAT1GlQfz1wcK1YWX1IBRl1COT1gmj rvCPCXT5TLjTPEPQaKDoK5CdDtSZ7SnYHrDczqhpAGMG4hNwvRW3K1pv8bdUxPRKc4KK wy5PD5vlFPwtSoJRyWSCdvZQk07AzeFyzYlLXkf9OeMMbpXElYiZ8EAnDeHSqLrpK8Eb Exl1BDALpvqKIZ1AXEeYEiGdGqX5bNE/Cwmba/19ApfhgH/tkGjG0SBsdhfCjfZ7h2Sy fEDA== X-Gm-Message-State: AOJu0Yzrtmf2neYcu0gzZWRrNZKnBJaxZbzldMIwB99RexRGGopTt3xN 0Y8Qug7QMSMD0OcctRrSa3Vr+hO/ekSfcW4aSXFremGxHzYVpqJ5UOQ2og== X-Google-Smtp-Source: AGHT+IFEi5Vr8c/q+mEPUHBtUOwriUpfA5Hz1dQuRJSM0ghM5w/qzu7YdG4QIN8ekovfZXSVMbqFNQ== X-Received: by 2002:a05:600c:4fcd:b0:426:545b:ec00 with SMTP id 5b1f17b1804b1-42811da0861mr103358135e9.19.1722371365568; Tue, 30 Jul 2024 13:29:25 -0700 (PDT) Received: from 127.0.0.1localhost ([85.255.235.94]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42808457c7fsm214488065e9.32.2024.07.30.13.29.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 13:29:24 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Lewis Baker Subject: [PATCH 3/3] io_uring: add absolute mode wait timeouts Date: Tue, 30 Jul 2024 21:29:44 +0100 Message-ID: <8db7af5ac6f6c81dc731e214f927c58e53d2d015.1722357468.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: References: Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In addition to current relative timeouts for the waiting loop, where the timespec argument specifies the maximum time it can wait for, add support for the absolute mode, with the value carrying a CLOCK_MONOTONIC absolute time until which we should return control back to the user. Link: https://github.com/axboe/liburing/issues/1162 Suggested-by: Lewis Baker Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 13 +++++++------ 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 2aaf7ee256ac..afc901502804 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -507,6 +507,7 @@ struct io_cqring_offsets { #define IORING_ENTER_SQ_WAIT (1U << 2) #define IORING_ENTER_EXT_ARG (1U << 3) #define IORING_ENTER_REGISTERED_RING (1U << 4) +#define IORING_ENTER_ABS_TIMER (1U << 5) /* * Passed in for io_uring_setup(2). Copied back with updated info on success diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9ec07f76ad19..5940bd8f5630 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2387,7 +2387,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, * Wait until events become available, if we don't already have some. The * application must reap them itself, as they reside on the shared cq ring. */ -static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, +static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, const sigset_t __user *sig, size_t sigsz, struct __kernel_timespec __user *uts) { @@ -2416,13 +2416,13 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (uts) { struct timespec64 ts; - ktime_t dt; if (get_timespec64(&ts, uts)) return -EFAULT; - dt = timespec64_to_ktime(ts); - iowq.timeout = ktime_add(dt, ktime_get()); + iowq.timeout = timespec64_to_ktime(ts); + if (!(flags & IORING_ENTER_ABS_TIMER)) + iowq.timeout = ktime_add(iowq.timeout, ktime_get()); } if (sig) { @@ -3153,7 +3153,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP | IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG | - IORING_ENTER_REGISTERED_RING))) + IORING_ENTER_REGISTERED_RING | + IORING_ENTER_ABS_TIMER))) return -EINVAL; /* @@ -3252,7 +3253,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, min_complete = min(min_complete, ctx->cq_entries); ret2 = io_cqring_wait(ctx, min_complete, sig, - argsz, ts); + flags, argsz, ts); } }