From patchwork Thu Feb 15 16:06:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13558773 Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7367C13A24E for ; Thu, 15 Feb 2024 16:10:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013409; cv=none; b=YPjm5nPAadTvH4qoQyHw/6XThf7Z6c6/9TcQDW30LU5b0SsprkT2eTsaKk+iJOIWu73lIlguHU19cH10+3m7Fq4O4lFxruFKbzl5WBM+ckanBsG+zGQh7jyU1b5BzhOtNvHPWlhP+rrdzjen/vXfpBqitOsAQ8RaIHaU2CoZVlg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013409; c=relaxed/simple; bh=bmsxmG+OqlGTcY/3NXZ+G6esoExlSkE8JVHVGK3CfKE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hsTB34FqPE3aBEGy7MQ9YVlKrOKriLcuhQ2f4ObLGlx+fQg9P9hYdq7HoCG7kHfZp957wFPXRVpcXuttmN+V81d8eYfOfVfyygGvVZZqiVWuJuHWyWs/LTnxiYXnqKvND+WzalSMn20INku/cOeq3RJkhwfOzeZnUoHuJNuPbr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=xwyU+dEG; arc=none smtp.client-ip=209.85.166.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="xwyU+dEG" Received: by mail-il1-f173.google.com with SMTP id e9e14a558f8ab-363acc3bbd8so535565ab.1 for ; Thu, 15 Feb 2024 08:10:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708013406; x=1708618206; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RODZUjyaVV50YYBvMGYdEqYxM/1CrYMy8Opg6CHor5g=; b=xwyU+dEGC5iQoaKmNJIoDJXupqYCY6XELsMbQp0KoLG+q3bi6II84v+j2Bu0PaFbqn CUuBUj9fWfB1TUnjF9Lfshtc9GvsYvBMR4kqAIw7jcDlf8iiKo30t6epz79gLn0vkTa4 14eqOHWAtIwN95LmSc+hQ+HXaslMFasbTP7vIEc2BWPk+IQONCLf30p9fOVHPQJifned shJ9StR7RIZ+f97GQz3TWStptI5Un1VLXRxKGLxVu3ehBLr886yMuY+W/Ij+FJxljxRm as/HdMNCzib3YCuell3XHrmJfNvGGvVx+4GkqalsjdSNya3GEGHdHCRLpozwaxClDmh2 zB+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708013406; x=1708618206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RODZUjyaVV50YYBvMGYdEqYxM/1CrYMy8Opg6CHor5g=; b=veMwTCBRYKkcNg8qxhBD6PNFGJzJ8yi3sW98JyikS5ME6XR9lP0/Mo4XsAWfsCBjKf 5uj9ZYFrMN26a4ajKFSDdorL9MsmFlYF4xanrapo311ddDvt0wjw4jnc9gvpZsR8yaEv WCNOlq5gIRqDs7SoYE7wJpXO3777VPSkHiM6OP4VlP8bgMYxuMG+dLO0MrYKhvxgjTrH u1KYXDk3TbsitTaY9JVcnNSMncWA34B2iwkNcLWdQGiez/VYa4O2oQYN5XdNtrloxazf OTQKX7S5AOPd/ebGU+HDss05y/JivdruLk5IPbgDX4+ShIVnllnOPb3p063wht6O+s8C aBag== X-Gm-Message-State: AOJu0YwR4Wltof2tNuTtZqaIgwlpeVjaHKsfBSgGYPAE59vrPTV5fySm L01gkdcGca+XcdvBuspx114VP1zU9RZL+6ck9s1gLeyMxLMTK+xqOcO8JmqujWQDMAEfNkH1Ax9 4 X-Google-Smtp-Source: AGHT+IGYV3MEI69MAMPuDiabhbXQZVf2mkXgoJNiEmmu3Crd7ms9Q5eTNs3gHkpGLdp5vNoELDWidA== X-Received: by 2002:a05:6e02:1d0a:b0:363:b545:3a97 with SMTP id i10-20020a056e021d0a00b00363b5453a97mr2235087ila.0.1708013406181; Thu, 15 Feb 2024 08:10:06 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id x4-20020a056e02074400b0036275404ab3sm458524ils.85.2024.02.15.08.10.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 08:10:04 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/4] io_uring: move schedule wait logic into helper Date: Thu, 15 Feb 2024 09:06:56 -0700 Message-ID: <20240215161002.3044270-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240215161002.3044270-1-axboe@kernel.dk> References: <20240215161002.3044270-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for expanding how we handle waits, move the actual schedule and schedule_timeout() handling into a helper. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 45a2f8f3a77c..8f52acc14ebc 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2516,22 +2516,10 @@ static bool current_pending_io(void) return percpu_counter_read_positive(&tctx->inflight); } -/* when returns >0, the caller should retry */ -static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq) +static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx, + struct io_wait_queue *iowq) { - int io_wait, ret; - - if (unlikely(READ_ONCE(ctx->check_cq))) - return 1; - if (unlikely(!llist_empty(&ctx->work_llist))) - return 1; - if (unlikely(test_thread_flag(TIF_NOTIFY_SIGNAL))) - return 1; - if (unlikely(task_sigpending(current))) - return -EINTR; - if (unlikely(io_should_wake(iowq))) - return 0; + int io_wait, ret = 0; /* * Mark us as being in io_wait if we have pending requests, so cpufreq @@ -2541,7 +2529,6 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, io_wait = current->in_iowait; if (current_pending_io()) current->in_iowait = 1; - ret = 0; if (iowq->timeout == KTIME_MAX) schedule(); else if (!schedule_hrtimeout(&iowq->timeout, HRTIMER_MODE_ABS)) @@ -2550,6 +2537,24 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, return ret; } +/* when returns >0, the caller should retry */ +static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, + struct io_wait_queue *iowq) +{ + if (unlikely(READ_ONCE(ctx->check_cq))) + return 1; + if (unlikely(!llist_empty(&ctx->work_llist))) + return 1; + if (unlikely(test_thread_flag(TIF_NOTIFY_SIGNAL))) + return 1; + if (unlikely(task_sigpending(current))) + return -EINTR; + if (unlikely(io_should_wake(iowq))) + return 0; + + return __io_cqring_wait_schedule(ctx, iowq); +} + /* * Wait until events become available, if we don't already have some. The * application must reap them itself, as they reside on the shared cq ring. From patchwork Thu Feb 15 16:06:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13558774 Received: from mail-il1-f172.google.com (mail-il1-f172.google.com [209.85.166.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC7A984FC2 for ; Thu, 15 Feb 2024 16:10:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013411; cv=none; b=srK4sqBV8gYKnda4RhiQMeuSS4eXlnEVsObC2CdQj4Pf33LdGSs1Pm+dADGE52fCAKOuKPaDBrHVX9JtWri6EBOWeDg42AAWEVdwmdErISnZq3BIptXS3ObMsWKCRAJlLXhR1qq+g7oMOTEaoG4AZKkXvXn2LpJvkVuKJw59Y2E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013411; c=relaxed/simple; bh=SPbtbd/VPyQSoyZUrc2YiFvw71pEWWTOy06FiwBC7q8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AgPAMIC/ugVbDbxnhll2VZkkb3MWMBIVix7p8TyyMiZ/GFX8gCB3xhxEYd8FAMTKLITBLxpwHL4dRhZWUMWYtE82RVsfA2ANQNPDnhy1rUwKGSgyjkYjXkazt3+EhiG+iWYMrzoKsJGaWonJ25Zp8nUmiUy899ETtPb+DvF0p+A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=DwbEp+JO; arc=none smtp.client-ip=209.85.166.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="DwbEp+JO" Received: by mail-il1-f172.google.com with SMTP id e9e14a558f8ab-363e7d542f4so417675ab.0 for ; Thu, 15 Feb 2024 08:10:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708013407; x=1708618207; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I4cnl52ova8W38aWfWiW7EBOK/wVRhZpBo5coezzJxs=; b=DwbEp+JOMBngwgRB7qvVFIDUlDdrVgP+ER7wTBtmwoQooOvpulcB5+Ato1sTuoJL3F tGIkEwFwsw0KyV3+CVPkej/eBnQjcITaT94ovbNucYjsD9FIF+088I37zbXxyw61WmV1 FkT9bXOVCUtmP9/LPSjqg1Oej9t3XoaFYpx+YVtTiV7X5FjzUkJI0HIEbUQmMn/Fk0hg qtjY9BBX/DRXXes75jwxmmCPpztN1KUg1/R0lRREToecNN111QH9XHt18aPma/QeK0cX bxiko+XfZLjQxQ1AUf5levCqOk+8cg4yvUyDUXycYAsD6QQbsSKgDd7UgDcN4KsTUxIM ysSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708013407; x=1708618207; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I4cnl52ova8W38aWfWiW7EBOK/wVRhZpBo5coezzJxs=; b=Xmw/uira9X19Hqt6mFqxdAao7hvC2OblX43Vd4NYXRdOt+kSu9MKjEWM2w6Q1fZABh FpM9j/AeafHgL2Z5Sxugor5dIjdDoghQbGxkckY29DHKUOh/vQ1F3/m5/Y8aYM2M77kg 6ziPFc35+iH2/BW+Go44aPO91pDSZ1qEN1WVTGuWrzJTw6qke2OrSWNbYOG57xt7gmPv OrlgMTEj/03KGpDyv6+ygWrows3A78S8c8MGwi+XO/Y2zhKfWKNhh0cH+nXgbFRTkfSD zizZcKriQ4EEfmLY05MlEaxt85mJu8sLAfhXD+xuQh5sSFhXJQMv463y7sVl8LCbjAFL NnzQ== X-Gm-Message-State: AOJu0Yz1qIFHgE2YkNFYUkN26t6l4AR18Yc5lbSDDYUwyoZbC4TnhLzr FJJ8q7J2k1MGCnF2UKcwpKmGfXBWcnY/rNWnT7oubk3VXsWe75Mi01aXPe001SoWbI/MNMZpzqC 2 X-Google-Smtp-Source: AGHT+IGTONfYW91A5x0netz5WqT0dw0qb1cMfHofxh3YJD0GIHTVvwSfxmnSsFXemk2kDDE0as1TXg== X-Received: by 2002:a92:c26c:0:b0:363:c7c6:4f5f with SMTP id h12-20020a92c26c000000b00363c7c64f5fmr2115741ild.1.1708013407468; Thu, 15 Feb 2024 08:10:07 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id x4-20020a056e02074400b0036275404ab3sm458524ils.85.2024.02.15.08.10.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 08:10:06 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/4] io_uring: implement our own schedule timeout handling Date: Thu, 15 Feb 2024 09:06:57 -0700 Message-ID: <20240215161002.3044270-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240215161002.3044270-1-axboe@kernel.dk> References: <20240215161002.3044270-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for having two distinct timeouts and avoid waking the task if we don't need to. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 39 +++++++++++++++++++++++++++++++++++---- io_uring/io_uring.h | 2 ++ 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8f52acc14ebc..ebc646ad6acf 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2488,7 +2488,7 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, * Cannot safely flush overflowed CQEs from here, ensure we wake up * the task, and the next invocation will do it. */ - if (io_should_wake(iowq) || io_has_work(iowq->ctx)) + if (io_should_wake(iowq) || io_has_work(iowq->ctx) || iowq->hit_timeout) return autoremove_wake_function(curr, mode, wake_flags, key); return -1; } @@ -2516,6 +2516,37 @@ static bool current_pending_io(void) return percpu_counter_read_positive(&tctx->inflight); } +static enum hrtimer_restart io_cqring_timer_wakeup(struct hrtimer *timer) +{ + struct io_wait_queue *iowq = container_of(timer, struct io_wait_queue, t); + struct io_ring_ctx *ctx = iowq->ctx; + + WRITE_ONCE(iowq->hit_timeout, 1); + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + wake_up_process(ctx->submitter_task); + else + io_cqring_wake(ctx); + return HRTIMER_NORESTART; +} + +static int io_cqring_schedule_timeout(struct io_wait_queue *iowq) +{ + iowq->hit_timeout = 0; + hrtimer_init_on_stack(&iowq->t, CLOCK_MONOTONIC, HRTIMER_MODE_ABS); + iowq->t.function = io_cqring_timer_wakeup; + hrtimer_set_expires_range_ns(&iowq->t, iowq->timeout, 0); + hrtimer_start_expires(&iowq->t, HRTIMER_MODE_ABS); + + if (!READ_ONCE(iowq->hit_timeout)) + schedule(); + + hrtimer_cancel(&iowq->t); + destroy_hrtimer_on_stack(&iowq->t); + __set_current_state(TASK_RUNNING); + + return READ_ONCE(iowq->hit_timeout) ? -ETIME : 0; +} + static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) { @@ -2529,10 +2560,10 @@ static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx, io_wait = current->in_iowait; if (current_pending_io()) current->in_iowait = 1; - if (iowq->timeout == KTIME_MAX) + if (iowq->timeout != KTIME_MAX) + ret = io_cqring_schedule_timeout(iowq); + else schedule(); - else if (!schedule_hrtimeout(&iowq->timeout, HRTIMER_MODE_ABS)) - ret = -ETIME; current->in_iowait = io_wait; return ret; } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 6426ee382276..9d1045bdc505 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -40,7 +40,9 @@ struct io_wait_queue { struct io_ring_ctx *ctx; unsigned cq_tail; unsigned nr_timeouts; + int hit_timeout; ktime_t timeout; + struct hrtimer t; #ifdef CONFIG_NET_RX_BUSY_POLL unsigned int napi_busy_poll_to; From patchwork Thu Feb 15 16:06:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13558775 Received: from mail-il1-f175.google.com (mail-il1-f175.google.com [209.85.166.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A8D713A25B for ; Thu, 15 Feb 2024 16:10:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013412; cv=none; b=tEyOxhJv+n5XOEVEWUyG8vwHLD2TWClK/OcAP8MhQg+zoDY7BEMa89kXnajSkkpi086+oIcEtyBqsGcExHPWLz1BRWj+mADaB9qe5VdPIVB/FCktv1oTmQw3e6z3zrwPwBZKpFWfZLBgqTcM3nt1581BcNl5puc5d6t/UEqJZLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013412; c=relaxed/simple; bh=J3Ud47rfdIGk4jBd05Wc+uUJyjx4SrgqJaDmEybi4W4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Hstr8yBMAjjtRFq62SOZNr1pfkBC0fuqaTvuKYrauvlJrJ6HIcHLHMtm/4mK5THQkIwo1UTKuI3Y8W8cdr5E8dqXyh3HmS0OYKJLTdUISQZGdupIZYqVqyiT8W6ugrgiX8wdPDyQIwADsusmwshuW5/wbzfVTAhByZXNSpzWDws= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=tNU9qCdQ; arc=none smtp.client-ip=209.85.166.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="tNU9qCdQ" Received: by mail-il1-f175.google.com with SMTP id e9e14a558f8ab-363e7d542f4so417715ab.0 for ; Thu, 15 Feb 2024 08:10:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708013409; x=1708618209; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QKp+xF2jAsd8VIFw0Ca+T76+SiSx2nSK3CsAVAuuGTo=; b=tNU9qCdQiBXBvmhJtLkltusmh7rvVponFCGHQkhXm6BiT+UNE0m3PQXAr91lCKW01c IypqwwAiqT+P+mIwiUMW1qMc7kcgI7IpNDzQwUvASO2+5ls49HzfV0YubZRcURgsm873 nWtx3UP9MsdeS6Z2rg4xFINXfnpWA4q/XrRR+NQiczzsadJdZk5/6a/FrhKvyaiJfXjX tushxOsVCFxr+PeGaTomuX/ElrU0y16Q0qwOS7an3+mXj81fVNKI1qYQdeqQxtA/5S6m DFFaeFjTDTRUEaewQFoVKza+FwSNef+BoRTGg21MlwLpXWiWUrpfVcRjj1aKlDgD1Gwu oObw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708013409; x=1708618209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QKp+xF2jAsd8VIFw0Ca+T76+SiSx2nSK3CsAVAuuGTo=; b=TmTgDOA8nKRb0lOu2wsU5HiMZOqyR/Ubi+mgkMkpV/KmlwkZJYoKeVQIhQ5CcYc6Jb 63uN38IKYALwUz+bCBqkw6GpdGX6WKPxUBXwfhemVFHPjyXZeufCpTxI89Rve7Azs+Df tZkYyDO7kv/64zjJtE/oE7lVxp2HHXTk1ktkYW0v9abq36DIBQNPoqj9bsYZdXCiKc/9 P6rcrWXZ1igr2xOX1xFs9i0gyvLuuc+sCdOficrRtnbBUf2jn2sYt2iZkPmKQMJGuoA6 u1e1u5Ogmr0+nDOYcOTaoUQ9HhD59QxW7HSsPVb/C1n1jQAnxkuhXVXL6BZza4hFGgRa 3/Ag== X-Gm-Message-State: AOJu0YyPoy1klX8yzsHsRakJZW3wzvEqNajJKcAF1w9kELvUqXxeuSoE B0y9BLKOUjpOyJD1ZI700LnGL7P93slmFw8o5AmalmhA45VKgvRT8qkhUDkOIAyLcojd6arTIqC r X-Google-Smtp-Source: AGHT+IGCKKngCo+MwZXEaSRdd0M3ktWd6kQrWsKTusFqAusWbZbxURU8PPicMGC+cQV+DzbJAsKjuA== X-Received: by 2002:a05:6e02:1d0a:b0:363:b545:3a97 with SMTP id i10-20020a056e021d0a00b00363b5453a97mr2235247ila.0.1708013409094; Thu, 15 Feb 2024 08:10:09 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id x4-20020a056e02074400b0036275404ab3sm458524ils.85.2024.02.15.08.10.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 08:10:07 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/4] io_uring: add support for batch wait timeout Date: Thu, 15 Feb 2024 09:06:58 -0700 Message-ID: <20240215161002.3044270-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240215161002.3044270-1-axboe@kernel.dk> References: <20240215161002.3044270-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Waiting for events with io_uring has two knobs that can be set: 1) The number of events to wake for 2) The timeout associated with the event Waiting will abort when either of those conditions are met, as expected. This adds support for a third event, which is associated with the number of events to wait for. Applications generally like to handle batches of completions, and right now they'd set a number of events to wait for and the timeout for that. If no events have been received but the timeout triggers, control is returned to the application and it can wait again. However, if the application doesn't have anything to do until events are reaped, then it's possible to make this waiting more efficient. For example, the application may have a latency time of 50 usecs and wanting to handle a batch of 8 requests at the time. If it uses 50 usecs as the timeout, then it'll be doing 20K context switches per second even if nothing is happening. This introduces the notion of min batch wait time. If the min batch wait time expires, then we'll return to userspace if we have any events at all. If none are available, the general wait time is applied. Any request arriving after the min batch wait time will cause waiting to stop and return control to the application. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 79 +++++++++++++++++++++++++++++++++++++++------ io_uring/io_uring.h | 2 ++ 2 files changed, 71 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ebc646ad6acf..e72261f280a7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2529,12 +2529,65 @@ static enum hrtimer_restart io_cqring_timer_wakeup(struct hrtimer *timer) return HRTIMER_NORESTART; } -static int io_cqring_schedule_timeout(struct io_wait_queue *iowq) +/* + * Doing min_timeout portion. If we saw any timeouts, events, or have work, + * wake up. If not, and we have a normal timeout, switch to that and keep + * sleeping. + */ +static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer) { + struct io_wait_queue *iowq = container_of(timer, struct io_wait_queue, t); + struct io_ring_ctx *ctx = iowq->ctx; + ktime_t timeout; + + /* no general timeout, or shorter, we are done */ + if (iowq->timeout == KTIME_MAX || + ktime_after(iowq->min_timeout, iowq->timeout)) + goto out_wake; + /* work we may need to run, wake function will see if we need to wake */ + if (io_has_work(ctx)) + goto out_wake; + /* got events since we started waiting, min timeout is done */ + if (iowq->cq_min_tail != READ_ONCE(ctx->rings->cq.tail)) + goto out_wake; + /* if we have any events and min timeout expired, we're done */ + if (io_cqring_events(ctx)) + goto out_wake; + + /* + * If using deferred task_work running and application is waiting on + * more than one request, ensure we reset it now where we are switching + * to normal sleeps. Any request completion post min_wait should wake + * the task and return. + */ + if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) + atomic_set(&ctx->cq_wait_nr, 1); + + timeout = ktime_sub_ns(iowq->timeout, iowq->min_timeout); + iowq->t.function = io_cqring_timer_wakeup; + hrtimer_set_expires(timer, ktime_add_ns(timeout, ktime_get_ns())); + return HRTIMER_RESTART; +out_wake: + return io_cqring_timer_wakeup(timer); +} + +static int io_cqring_schedule_timeout(struct io_wait_queue *iowq, + ktime_t start_time) +{ + ktime_t timeout; + iowq->hit_timeout = 0; hrtimer_init_on_stack(&iowq->t, CLOCK_MONOTONIC, HRTIMER_MODE_ABS); - iowq->t.function = io_cqring_timer_wakeup; - hrtimer_set_expires_range_ns(&iowq->t, iowq->timeout, 0); + + if (iowq->min_timeout != KTIME_MAX) { + timeout = ktime_add_ns(iowq->min_timeout, start_time); + iowq->t.function = io_cqring_min_timer_wakeup; + } else { + timeout = ktime_add_ns(iowq->timeout, start_time); + iowq->t.function = io_cqring_timer_wakeup; + } + + hrtimer_set_expires_range_ns(&iowq->t, timeout, 0); hrtimer_start_expires(&iowq->t, HRTIMER_MODE_ABS); if (!READ_ONCE(iowq->hit_timeout)) @@ -2548,7 +2601,8 @@ static int io_cqring_schedule_timeout(struct io_wait_queue *iowq) } static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq) + struct io_wait_queue *iowq, + ktime_t start_time) { int io_wait, ret = 0; @@ -2560,8 +2614,8 @@ static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx, io_wait = current->in_iowait; if (current_pending_io()) current->in_iowait = 1; - if (iowq->timeout != KTIME_MAX) - ret = io_cqring_schedule_timeout(iowq); + if (iowq->timeout != KTIME_MAX || iowq->min_timeout != KTIME_MAX) + ret = io_cqring_schedule_timeout(iowq, start_time); else schedule(); current->in_iowait = io_wait; @@ -2570,7 +2624,8 @@ static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx, /* when returns >0, the caller should retry */ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, - struct io_wait_queue *iowq) + struct io_wait_queue *iowq, + ktime_t start_time) { if (unlikely(READ_ONCE(ctx->check_cq))) return 1; @@ -2583,7 +2638,7 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, if (unlikely(io_should_wake(iowq))) return 0; - return __io_cqring_wait_schedule(ctx, iowq); + return __io_cqring_wait_schedule(ctx, iowq, start_time); } /* @@ -2596,6 +2651,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, { struct io_wait_queue iowq; struct io_rings *rings = ctx->rings; + ktime_t start_time; int ret; if (!io_allowed_run_tw(ctx)) @@ -2626,8 +2682,11 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, INIT_LIST_HEAD(&iowq.wq.entry); iowq.ctx = ctx; iowq.nr_timeouts = atomic_read(&ctx->cq_timeouts); + iowq.cq_min_tail = READ_ONCE(ctx->rings->cq.tail); iowq.cq_tail = READ_ONCE(ctx->rings->cq.head) + min_events; + iowq.min_timeout = KTIME_MAX; iowq.timeout = KTIME_MAX; + start_time = ktime_get_ns(); if (uts) { struct timespec64 ts; @@ -2635,7 +2694,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, if (get_timespec64(&ts, uts)) return -EFAULT; - iowq.timeout = ktime_add_ns(timespec64_to_ktime(ts), ktime_get_ns()); + iowq.timeout = timespec64_to_ktime(ts); io_napi_adjust_timeout(ctx, &iowq, &ts); } @@ -2654,7 +2713,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, TASK_INTERRUPTIBLE); } - ret = io_cqring_wait_schedule(ctx, &iowq); + ret = io_cqring_wait_schedule(ctx, &iowq, start_time); __set_current_state(TASK_RUNNING); atomic_set(&ctx->cq_wait_nr, IO_CQ_WAKE_INIT); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 9d1045bdc505..f385c7e36cb7 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -39,8 +39,10 @@ struct io_wait_queue { struct wait_queue_entry wq; struct io_ring_ctx *ctx; unsigned cq_tail; + unsigned cq_min_tail; unsigned nr_timeouts; int hit_timeout; + ktime_t min_timeout; ktime_t timeout; struct hrtimer t; From patchwork Thu Feb 15 16:06:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13558776 Received: from mail-il1-f176.google.com (mail-il1-f176.google.com [209.85.166.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF76213A25F for ; Thu, 15 Feb 2024 16:10:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013414; cv=none; b=nNsYvzXWHVlfZSK4utyZIAMQQLGwrl2Xkyk1HOcjlI5QhOK+r2R6cHSMvDJhv+19yMx/3q1yJCIbU3YX4lxKSEpmwAViWWJbw3JMlZ4Gor3pQSF+NtZErrokkQe0a1k4blw/S5HIfGcXfZAaaA/a0ZltHofWt1EDNi7bqUIVJ90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708013414; c=relaxed/simple; bh=+01ggezWDlBXaMZzUYyNL+qL7BUdm/4u6nKXttGq8+0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=p3hDT257/MorZG4fWNzecD5rsIsVdm7dDTvKdzz7fVPzpL7LUdEHvujtIM/4lt34O3kdGl+EWOdfoU5j4jSkYn4DLYHIeDnA4A5o4BXT9ISgecHI6kbAVDw3GT7OnJn9cW3tceo1JhA9AuD9mC3oAHpHy22MApk4NsrGtioTYXE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=HEPOxFkj; arc=none smtp.client-ip=209.85.166.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="HEPOxFkj" Received: by mail-il1-f176.google.com with SMTP id e9e14a558f8ab-35d374bebe3so1535565ab.1 for ; Thu, 15 Feb 2024 08:10:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1708013410; x=1708618210; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/ktOLqvWoXFqusXcGOV12iqejJlkHbShshkXyRrkD7I=; b=HEPOxFkjIl1NVaronn9hJDlY1pmPb5ugOy5uDemE5ML9aOFzKfKiZYvRAu9e26be3W tHGpF+0o+cRAo881nSbQ0uoILrlEdkSkwFGo59QIfoLvrOtN8L07dfOc1QUmnA89Kg+H 5cKEbWh/ULKvupS7StMFlMaT+3QRV+pZ/QfZlmoOq27m5lQ93IyQea3f15GZJ9gXNytE 4LxzqEzjOLwUBtNpZQseJ7jCf2L1GTqrK4z8MY5F/MFIVi0LVwjBUNTV5B1cbTIEns/a mSrYqSRDl5XQ7evbCY7JBedYQxp6wLWZK+IoIuwyEq+4aEifdgEVlpGQnXc+OQYg+Crd opXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708013410; x=1708618210; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/ktOLqvWoXFqusXcGOV12iqejJlkHbShshkXyRrkD7I=; b=Atdgza9t2cybHZN4DU4DoeTeqw97q57DzhLOJecm3VvpMFYtwPrMnntXRKzin0oCPY xn7G0HXOJSnSGlPXueHyA9GBoIr/2ZMmnfm/Q3Y2NPbmz6D9JUn8yN6Pl4TFPtH+8NSR MC5uU1OkVtN6tMxYt40U++L210CitJs6t0Ju50EO4lZ+FMRJGWGrFr7azOXn05D2xm/Z FJGhjNPqRhfz0VAdStmKuVNO+rxWkCSJDKzv6zXCQZeBe9PNQhVL/aNnYipTXI/JAUsB vSu//09PARfiMdDHvxfdzZ5v68ldsl+d1hgdqLaYnzot1vyKM+kEh5EIHAXagDUGULVU OSzw== X-Gm-Message-State: AOJu0YyGsyuq54sgI/9OUHzgYnBpEPdjcdsnr33smC6x3XWdFeM24yGH hhIgXP+WjvfzCqa/d5KZXe7CWlv1f32j+qF2wpAK/7v376imyQMafE1rTCmKjkPo3vx1L/TukBa u X-Google-Smtp-Source: AGHT+IHsDy/2kOKfUoYJaZ7ZpxFoBJ58qi3Xt5H1Eo7RshjEM7KPtdx2WyZ8VeBI841MuixFZZt1JQ== X-Received: by 2002:a92:c24b:0:b0:363:c919:eee3 with SMTP id k11-20020a92c24b000000b00363c919eee3mr2240616ilo.0.1708013410519; Thu, 15 Feb 2024 08:10:10 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id x4-20020a056e02074400b0036275404ab3sm458524ils.85.2024.02.15.08.10.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Feb 2024 08:10:09 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/4] io_uring: wire up min batch wake timeout Date: Thu, 15 Feb 2024 09:06:59 -0700 Message-ID: <20240215161002.3044270-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240215161002.3044270-1-axboe@kernel.dk> References: <20240215161002.3044270-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Expose min_wait_usec in io_uring_getevents_arg, replacing the pad member that is currently in there. The value is in usecs, which is explained in the name as well. Note that if min_wait_usec and a normal timeout is used in conjunction, the normal timeout is still relative to the base time. For example, if min_wait_usec is set to 100 and the normal timeout is 1000, the max total time waited is still 1000. This also means that if the normal timeout is shorter than min_wait_usec, then only the min_wait_usec will take effect. See previous commit for an explanation of how this works. IORING_FEAT_MIN_TIMEOUT is added as a feature flag for this, as applications doing submit_and_wait_timeout() style operations will generally not see the -EINVAL from the wait side as they return the number of IOs submitted. Only if no IOs are submitted will the -EINVAL bubble back up to the application. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 3 ++- io_uring/io_uring.c | 21 +++++++++++++-------- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 7bd10201a02b..dbefda14d087 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -522,6 +522,7 @@ struct io_uring_params { #define IORING_FEAT_CQE_SKIP (1U << 11) #define IORING_FEAT_LINKED_FILE (1U << 12) #define IORING_FEAT_REG_REG_RING (1U << 13) +#define IORING_FEAT_MIN_TIMEOUT (1U << 14) /* * io_uring_register(2) opcodes and arguments @@ -738,7 +739,7 @@ enum { struct io_uring_getevents_arg { __u64 sigmask; __u32 sigmask_sz; - __u32 pad; + __u32 min_wait_usec; __u64 ts; }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index e72261f280a7..8dd5eb647b43 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2647,7 +2647,8 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, */ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, const sigset_t __user *sig, size_t sigsz, - struct __kernel_timespec __user *uts) + struct __kernel_timespec __user *uts, + ktime_t min_time) { struct io_wait_queue iowq; struct io_rings *rings = ctx->rings; @@ -2684,7 +2685,7 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, iowq.nr_timeouts = atomic_read(&ctx->cq_timeouts); iowq.cq_min_tail = READ_ONCE(ctx->rings->cq.tail); iowq.cq_tail = READ_ONCE(ctx->rings->cq.head) + min_events; - iowq.min_timeout = KTIME_MAX; + iowq.min_timeout = min_time; iowq.timeout = KTIME_MAX; start_time = ktime_get_ns(); @@ -3634,10 +3635,12 @@ static int io_validate_ext_arg(unsigned flags, const void __user *argp, size_t a static int io_get_ext_arg(unsigned flags, const void __user *argp, size_t *argsz, struct __kernel_timespec __user **ts, - const sigset_t __user **sig) + const sigset_t __user **sig, ktime_t *min_time) { struct io_uring_getevents_arg arg; + *min_time = KTIME_MAX; + /* * If EXT_ARG isn't set, then we have no timespec and the argp pointer * is just a pointer to the sigset_t. @@ -3656,8 +3659,8 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, size_t *argsz return -EINVAL; if (copy_from_user(&arg, argp, sizeof(arg))) return -EFAULT; - if (arg.pad) - return -EINVAL; + if (arg.min_wait_usec) + *min_time = arg.min_wait_usec * NSEC_PER_USEC; *sig = u64_to_user_ptr(arg.sigmask); *argsz = arg.sigmask_sz; *ts = u64_to_user_ptr(arg.ts); @@ -3769,13 +3772,14 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, } else { const sigset_t __user *sig; struct __kernel_timespec __user *ts; + ktime_t min_time; - ret2 = io_get_ext_arg(flags, argp, &argsz, &ts, &sig); + ret2 = io_get_ext_arg(flags, argp, &argsz, &ts, &sig, &min_time); if (likely(!ret2)) { min_complete = min(min_complete, ctx->cq_entries); ret2 = io_cqring_wait(ctx, min_complete, sig, - argsz, ts); + argsz, ts, min_time); } } @@ -4058,7 +4062,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, IORING_FEAT_POLL_32BITS | IORING_FEAT_SQPOLL_NONFIXED | IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS | IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP | - IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING; + IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING | + IORING_FEAT_MIN_TIMEOUT; if (copy_to_user(params, p, sizeof(*p))) { ret = -EFAULT;