From patchwork Wed Oct 23 15:54:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847375 Received: from mail-il1-f170.google.com (mail-il1-f170.google.com [209.85.166.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA31A1C9DCE for ; Wed, 23 Oct 2024 15:56:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729699005; cv=none; b=WCa4ijN9iWFNMzslHNq+E37xnoslTiJy2PsHjc8MCQN9h7ZK8via0Oiof8EODfmpc6mY4+5rWaBt7A/+yxWvnPAat7/CzTbq+9l3IJaUSaOINi2WGogMoMasfXicHGV04KdtgkjAo0Yzdu53Zc+64xWdPMkP9r53ZxPo+PQZasI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729699005; c=relaxed/simple; bh=IaJHGPEnG4FwAl81uj6oQFgeVUEh5ilo+0SBvR1Qvsw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rNqOBuCLb6qycSty4aS8DxmZa5trnnyjCpOX2YJzp8q+Nl+y5+SGYzWaVcV4lAGIeSkcYtUY8sTQLLk69zOPcwNWGR6mr5fBzrOZ6mO+5/SGps++JgNxHxvNr1BQ+HZK1nRZ9pddcwbDjt1wWxBhepQKLPdVfaBdh7tpz9aX0+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=O77vwv4O; arc=none smtp.client-ip=209.85.166.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="O77vwv4O" Received: by mail-il1-f170.google.com with SMTP id e9e14a558f8ab-3a3a5cd2a3bso28402215ab.3 for ; Wed, 23 Oct 2024 08:56:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729699002; x=1730303802; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=877Civ4zwX5SMS6lpgNa8Hw02O/Hug6rwORSlmT0bwA=; b=O77vwv4O90i74SUnGC0Kevuv/vRulQzC5QDvZnrGtn9hUk/iHzHBlqQwTDrptWv6OU UHiho1ieDw7UpDkDcZajFq1nBq1cHMdktqZte6uJIu8Gzfv9ufAZC023Y52aO0sTZboK C2UJgQoBDtu5Ho6DcgyjQVeHNyRIFV7oIfXls0u1EAtAuJVdFa3LA9tv1HmH/8qktf0Q DnSKyCeDl4Spknl1pWVLUpcFtgYGLVXvxQQ+6w8dEkwruFnez59gdQ5/A3HHp/qw0UbF udt1Je+ebDfKjdi9s8AlPaZh/OHdiYA1jHqIehyztlsZsCHDkHPIulY4Tzh7u1tmV02o 9+eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729699002; x=1730303802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=877Civ4zwX5SMS6lpgNa8Hw02O/Hug6rwORSlmT0bwA=; b=t+WTjCEYq6ruw1+XR2wb8ekWtn9hAQf8i3MNNupZlvIkMdnQ+w9BvJciIl1ngo701M NXTaxN57mjNKRb61MOpvRXTS6Z+HteNXr463eOQvJGr2BgtntbmUkuC8ZM2XAO0eJe1o Z65P+PHMB7uRD+Yhojlw8dgGniak0lexDiV4VevUSgKPyfqKIS2fphIbHviwvKnCd2ui o0ztYWzigXYzRiepVDhOxOVFaaSgLVYXfO13t3LoFGOthcdRmqdFgbJc3tKvlllez4dt RrZQk3U08+gEerMmDBo3Y+u/2gWC5xt1VOZv81wK0rqyCBTEx/H5/Owcjc+dfQGtkH9R zYdA== X-Gm-Message-State: AOJu0YxjwF2SHJVwzGEScj0+0iFeozeKSM6kEZjXJVwHO9XcfrDcixRM yT1Ww1xsCnYbiG/6JuZ/pD+uhsXwkO3ad6fOb7PxEpEaWpM7T55lcbTPfDc2RNHlzhXFrI+O+hr g X-Google-Smtp-Source: AGHT+IHX+tF2oMT96nYi0O8pS0cbcc8f6jDNgOnQnKHRBaMyNW7Sxveb6MyzKBqwIHx2DCuvjOHybg== X-Received: by 2002:a05:6e02:138a:b0:39f:558a:e404 with SMTP id e9e14a558f8ab-3a4d5930270mr40829845ab.4.1729699002237; Wed, 23 Oct 2024 08:56:42 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a6091bfsm2131572173.97.2024.10.23.08.56.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 08:56:41 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/3] io_uring: switch struct ext_arg from __kernel_timespec to timespec64 Date: Wed, 23 Oct 2024 09:54:32 -0600 Message-ID: <20241023155639.1124650-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023155639.1124650-1-axboe@kernel.dk> References: <20241023155639.1124650-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This avoids intermediate storage for turning a __kernel_timespec user pointer into an on-stack struct timespec64, only then to turn it into a ktime_t. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b5974bdad48b..8952453ea807 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2494,9 +2494,10 @@ static inline int io_cqring_wait_schedule(struct io_ring_ctx *ctx, struct ext_arg { size_t argsz; - struct __kernel_timespec __user *ts; + struct timespec64 ts; const sigset_t __user *sig; ktime_t min_time; + bool ts_set; }; /* @@ -2534,13 +2535,8 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, iowq.timeout = KTIME_MAX; start_time = io_get_time(ctx); - if (ext_arg->ts) { - struct timespec64 ts; - - if (get_timespec64(&ts, ext_arg->ts)) - return -EFAULT; - - iowq.timeout = timespec64_to_ktime(ts); + if (ext_arg->ts_set) { + iowq.timeout = timespec64_to_ktime(ext_arg->ts); if (!(flags & IORING_ENTER_ABS_TIMER)) iowq.timeout = ktime_add(iowq.timeout, start_time); } @@ -3251,7 +3247,6 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, */ if (!(flags & IORING_ENTER_EXT_ARG)) { ext_arg->sig = (const sigset_t __user *) argp; - ext_arg->ts = NULL; return 0; } @@ -3266,7 +3261,11 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, ext_arg->min_time = arg.min_wait_usec * NSEC_PER_USEC; ext_arg->sig = u64_to_user_ptr(arg.sigmask); ext_arg->argsz = arg.sigmask_sz; - ext_arg->ts = u64_to_user_ptr(arg.ts); + if (arg.ts) { + if (get_timespec64(&ext_arg->ts, u64_to_user_ptr(arg.ts))) + return -EFAULT; + ext_arg->ts_set = true; + } return 0; } From patchwork Wed Oct 23 15:54:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847377 Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BB7179C4 for ; Wed, 23 Oct 2024 15:56:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729699007; cv=none; b=BEAT3nbzyOMP503KUlUdrLRocp6PzdoVh6GVJZLYytlj//lXXGQlMod27nh8/A7RNEZ1e88f1OOd7ZgsWyWIA6xhiRS/0qrUhlp3rw79yvWMMBhnFQkPy2OsAqmqgXCY3stIVWQAk1P8GFqYu2IDTwQ4ufK1jCgU0MTO9rB6R1Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729699007; c=relaxed/simple; bh=azReeC6JDM9vl2bxhBAJCZSFC7xE/Zte+PZEVTChuyQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=K223ZX23ByalPa1nZt0GIM602iyTFQ25kewGb8UM4C7O2l/rzA0peSJviNgTb35dI8XSeVOOvTEVr6ENm7q8jr1EvinepiJ689gdJzR8d6oTMs53ugqCsj1pqz15r0HkSt/lOYvO1IPQAFNw+Ci+BoK0aM7LLFk+Z6smNC4DyJY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=vTRJxjFc; arc=none smtp.client-ip=209.85.166.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="vTRJxjFc" Received: by mail-io1-f45.google.com with SMTP id ca18e2360f4ac-83aba237c03so237279639f.3 for ; Wed, 23 Oct 2024 08:56:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729699004; x=1730303804; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HYtrenvy20ANrIpWXUBY1U73jDZWvmvz372HYOtJ4d8=; b=vTRJxjFctGVmtgyW8ZQoyA76GiEG/YEvbqzfBaOoawrAD3WeZcI1e019R4bkdsTw2j xt97pAchLia7/WcI4C7B0v2HHBjwrGBWECbrp+oi9L/ySTcgpfLpWLVvsjyy6N5sEEsw rYtsN+1228ugJYDMoAFw/ZRDpLpOceuR/o07u9PduBMcg1ZrJnY38pVjj1+WGw16I6FL SWgM3iL6nKZJpQaU+z2dQAXX2uQK1a/eYG5EyCN/gbgpos2WboEDLK5Om9oMBDDIhVj6 EyADRRY1QGiL4fOGvHLEQXYpI59tnOOBp1QowgC44K5lTtSc5VP23caAKOpkxIMakr2N llgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729699004; x=1730303804; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HYtrenvy20ANrIpWXUBY1U73jDZWvmvz372HYOtJ4d8=; b=nVyAAv4oJCKzz0Ef9NsAtpUd6MQR3k1vA280A+Zsoln2Py7fZXgRRXzdZm4ocbRYH2 eS88iqmchOXL0r6HJj/V0LvzV00tfruDt5ouFekPXnYEijULTa/2Np1TCaQcPGL4mv6s tmic3YmGfRzhgk0umJ7GgDuXTgYeb5wt5kKSSR+Dao+eLMg+z+u9kjwexLuZDqIC5hPB Zm4s6i+dLPQQoLBskXoio+q0aC0c3RYwOnK5/JNqGK+wKQisOOqVG34356+wBwGLZP6d QGG2YxskkB98nFqq2gn+lfcLG3TUioR5tQswQzdRJA8//gD9M5/s8cr77UfIpeQs3fKw xPMw== X-Gm-Message-State: AOJu0Yw7qzMkfQAbNi8lGvgjyFL1gGpeasrtPdMrqTDCTcGvHmo2t0ii 7LtJoSeVnRuYLnRzsvIAzD9nPx7Cdk388Q+80rs5xFcKDZDCeWnqX8es5J3AMVIRc1RzGAHbvQJ a X-Google-Smtp-Source: AGHT+IHQcEcSEF9VOZ6cZo69i3GjwSflhGG+Be1HLboJwZyBhji+Z4AeUZu5mroCxg/SYUO2iWmLyg== X-Received: by 2002:a05:6602:1482:b0:83a:b364:ff10 with SMTP id ca18e2360f4ac-83af619a291mr316848339f.9.1729699004255; Wed, 23 Oct 2024 08:56:44 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a6091bfsm2131572173.97.2024.10.23.08.56.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 08:56:42 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe , Keith Busch Subject: [PATCH 2/3] io_uring: change io_get_ext_arg() to use uaccess begin + end Date: Wed, 23 Oct 2024 09:54:33 -0600 Message-ID: <20241023155639.1124650-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023155639.1124650-1-axboe@kernel.dk> References: <20241023155639.1124650-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In scenarios where a high frequency of wait events are seen, the copy of the struct io_uring_getevents_arg is quite noticeable in the profiles in terms of time spent. It can be seen as up to 3.5-4.5%. Rewrite the copy-in logic, saving about 0.5% of the time. Reviewed-by: Keith Busch Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 8952453ea807..bfea5d1fbc67 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3239,6 +3239,7 @@ static int io_validate_ext_arg(unsigned flags, const void __user *argp, size_t a static int io_get_ext_arg(unsigned flags, const void __user *argp, struct ext_arg *ext_arg) { + const struct io_uring_getevents_arg __user *uarg = argp; struct io_uring_getevents_arg arg; /* @@ -3256,8 +3257,19 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, */ if (ext_arg->argsz != sizeof(arg)) return -EINVAL; - if (copy_from_user(&arg, argp, sizeof(arg))) + if (!user_access_begin(uarg, sizeof(*uarg))) return -EFAULT; +#ifdef CONFIG_64BIT + unsafe_get_user(arg.sigmask, &uarg->sigmask, uaccess_end); + unsafe_get_user(arg.ts, &uarg->ts, uaccess_end); +#else + unsafe_copy_from_user(&arg.sigmask, &uarg->sigmask, sizeof(arg.sigmask), + uaccess_end); + unsafe_copy_from_user(&arg.ts, &uarg->ts, sizeof(arg.ts), uaccess_end); +#endif + unsafe_get_user(arg.min_wait_usec, &uarg->min_wait_usec, uaccess_end); + unsafe_get_user(arg.sigmask_sz, &uarg->sigmask_sz, uaccess_end); + user_access_end(); ext_arg->min_time = arg.min_wait_usec * NSEC_PER_USEC; ext_arg->sig = u64_to_user_ptr(arg.sigmask); ext_arg->argsz = arg.sigmask_sz; @@ -3267,6 +3279,9 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, ext_arg->ts_set = true; } return 0; +uaccess_end: + user_access_end(); + return -EFAULT; } SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, From patchwork Wed Oct 23 15:54:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13847378 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81CED1CACC8 for ; Wed, 23 Oct 2024 15:56:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729699009; cv=none; b=YWtbCk5euzPgjHkdvsHrO8RcWble16rUPYOYEW2qdJ8gKae+2bdPjtE5f6jkt5rr2Wq6UhysQRs2rUPQ+VZ+6ZBksh/7neYMf2xXQwJVa26YBJWFNZP4cTtCZ6reCmMpHUOrN2UoUQsTp2KRdPaHM79aSMge7+fj7ECoq/vp/KY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729699009; c=relaxed/simple; bh=s/E2tW7QCtv4OtGMUfwLNwN2hLwrAjCX1lcaRY4vE2M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=D+TjLfm3qFg/OhPYWQVyKZW8r7I9rr+vo9cL1a5/thCzQvVRyKnLwdL2DLuR0wFSVG3dXU8fSAvG8QRW/ffG7wWZ9u5ADZKJIsBSXBnqP0LstCySPZv6qumWY44mTTar6heivHyT75YFFmrhL/+KV14n1Li7twh8b04ZD0Nx9Vo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=xbzRKFKP; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="xbzRKFKP" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-83ab94452a7so237480939f.3 for ; Wed, 23 Oct 2024 08:56:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1729699006; x=1730303806; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w+Zh1gzaSOjXXONr7aRvoYvCGpH5OrTj4pnUiP1o7ws=; b=xbzRKFKPiaokN74iFq9wB8xV1wZOe9Jkr/hjtwSibUjeSv4L8LldQMkHX0upKGnuYY 6aDLYILbJJ8+klQiaDWaXfrJqXYYT9FEqG9PHG+Y2OjRz/OT9DX2fDs6+D6sAGFjGpUC ZI4BQ7rHnuQoBC1WXgE/zllF3dGIzNiNjAsI5qTDzodiVk+mrEAKmTRldk1r5W2skk24 X3hGam2XJUsoFh/bawzNJFEs1h3Pe53+LLoGkiriytP6X7I/4rr2gQ5qDdgknVJZ/8Z6 IsbLwV5z/snTLuUmN/CVv6kNA+otLko3iSSB8P3SegeLm1yDjgCUkB6qUHSfOFy55xjT ujkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729699006; x=1730303806; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w+Zh1gzaSOjXXONr7aRvoYvCGpH5OrTj4pnUiP1o7ws=; b=ac9PkfWGvrHfm83GWqGnhWCl0zNO5mVcrhhzVMTSOGb6rH5sewqOXYT7QaBQL0CoWa fCE0+j1PLmLWr+OwD81xnXnjmPLN+fBGGHrKCc77FCr66oy6cpGtyebu7qhYaWV+F3b/ yMHrZvhUPTahciIkZRHE+vJBHgDOjvcc1mrX3GviGdG8RyGwzj0KXQU2Jd7x0ELl58Xm PsB711RNXN3BLUN1a9zVnG+UMqbx3RiiDYatryqaqjU72YSmNS8/SH11dhuDBnAE/O8s 0q4zA7yZSnEV7kTMtnSUmFLOjiqfCswtRJZUBieBpEL62Ne/YPHGFaOeFbmltGT3A/lc RgIQ== X-Gm-Message-State: AOJu0YyxMSbhDcYDy8NVAD/LYlANrFUXe63RaSgNi1I0B90XlgZb9Wx0 V5Ft9ujy0nj+3cdd84Aolmkdf0L8gPZC5pQW2bQ0tM3nKZqwlmpYYwl/IfF83D6JTcLqMnw/4rY 4 X-Google-Smtp-Source: AGHT+IH/N+KcaVg7ZmoLS5MqNbL9ENVHAFq0Ju/QYKq3EifZhLzAx9Bq7547dUFhWpYIfUG3NoeosA== X-Received: by 2002:a05:6e02:1d02:b0:3a0:9fd8:f70b with SMTP id e9e14a558f8ab-3a4d594419dmr33799355ab.6.1729699006050; Wed, 23 Oct 2024 08:56:46 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc2a6091bfsm2131572173.97.2024.10.23.08.56.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 08:56:44 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/3] io_uring: add support for fixed wait regions Date: Wed, 23 Oct 2024 09:54:34 -0600 Message-ID: <20241023155639.1124650-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241023155639.1124650-1-axboe@kernel.dk> References: <20241023155639.1124650-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Generally applications have 1 or a few waits of waiting, yet they pass in a struct io_uring_getevents_arg every time. This needs to get copied and, in turn, the timeout value needs to get copied. Rather than do this for every invocation, allow the application to register a fixed set of wait regions that can simply be indexed when asking the kernel to wait on events. At ring setup time, the application can register a number of these wait regions ala: struct io_uring_reg_wait *reg; posix_memalign((void **) ®, page_size, page_size); memset(reg, 0, page_size); /* set timeout and mark it as set, sigmask/sigmask_sz as needed */ reg->ts.tv_sec = 0; reg->ts.tv_nsec = 100000; reg->flags = IORING_REG_WAIT_TS; io_uring_register_cqwait_reg(ring, reg, nr_regions); and instead of doing: struct __kernel_timespec timeout = { .tv_nsec = 100000, }; io_uring_submit_and_wait_timeout(ring, &cqe, nr, &t, NULL); for each submit_and_wait, or just wait, operation, it can just reference the above region at offset 0 and do: io_uring_submit_and_wait_reg(ring, &cqe, nr, 0); to achieve the same goal of waiting 100usec without needing to copy both struct io_uring_getevents_arg (24b) and struct __kernel_timeout (16b) for each invocation. Struct io_uring_reg_wait looks as follows: struct io_uring_reg_wait { struct __kernel_timespec ts; __u32 min_wait_usec; __u32 flags; __u64 sigmask; __u32 sigmask_sz; __u32 pad[3]; __u64 pad2[2]; }; embedding the timeout itself in the region, rather than passing it as a pointer as well. Note that the signal mask is still passed as a pointer, both for compatability reasons, but also because there doesn't seem to be a lot of high frequency waits scenarios that involve setting and resetting the signal mask for each wait. The application is free to modify any region before a wait call, or it can use keep multiple regions with different settings to avoid needing to modify the same one for wait calls. Up to a page size of regions is mapped by default, allowing PAGE_SIZE / 64 available regions for use. The registered region must fit within a page. On a 4kb page size system, that allows for 64 wait regions if a full page is used, as the size of struct io_uring_reg_wait is 64b. The region registered must be aligned to io_uring_reg_wait in size. It's valid to register less than 64 entries. In network performance testing with zero-copy, this reduced the time spent waiting on the TX side from 3.12% to 0.3% and the RX side from 4.4% to 0.3%. Wait regions are fixed for the lifetime of the ring - once registered, they are persistent until the ring is torn down. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 10 +++++ include/uapi/linux/io_uring.h | 18 ++++++++ io_uring/io_uring.c | 68 ++++++++++++++++++++++++----- io_uring/register.c | 79 ++++++++++++++++++++++++++++++++++ io_uring/register.h | 1 + 5 files changed, 165 insertions(+), 11 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 6d3ee71bd832..9746f4bb5182 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -327,6 +327,14 @@ struct io_ring_ctx { atomic_t cq_wait_nr; atomic_t cq_timeouts; struct wait_queue_head cq_wait; + + /* + * If registered with IORING_REGISTER_CQWAIT_REG, a single + * page holds N entries, mapped in cq_wait_arg. cq_wait_index + * is the maximum allowable index. + */ + struct io_uring_reg_wait *cq_wait_arg; + unsigned char cq_wait_index; } ____cacheline_aligned_in_smp; /* timeouts */ @@ -423,6 +431,8 @@ struct io_ring_ctx { unsigned short n_sqe_pages; struct page **ring_pages; struct page **sqe_pages; + + struct page **cq_wait_page; }; struct io_tw_state { diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index c4737892c7cd..4ead32fa9275 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -518,6 +518,7 @@ struct io_cqring_offsets { #define IORING_ENTER_EXT_ARG (1U << 3) #define IORING_ENTER_REGISTERED_RING (1U << 4) #define IORING_ENTER_ABS_TIMER (1U << 5) +#define IORING_ENTER_EXT_ARG_REG (1U << 6) /* * Passed in for io_uring_setup(2). Copied back with updated info on success @@ -618,6 +619,9 @@ enum io_uring_register_op { /* resize CQ ring */ IORING_REGISTER_RESIZE_RINGS = 33, + /* register fixed io_uring_reg_wait arguments */ + IORING_REGISTER_CQWAIT_REG = 34, + /* this goes last */ IORING_REGISTER_LAST, @@ -801,6 +805,20 @@ enum io_uring_register_restriction_op { IORING_RESTRICTION_LAST }; +enum { + IORING_REG_WAIT_TS = (1U << 0), +}; + +struct io_uring_reg_wait { + struct __kernel_timespec ts; + __u32 min_wait_usec; + __u32 flags; + __u64 sigmask; + __u32 sigmask_sz; + __u32 pad[3]; + __u64 pad2[2]; +}; + struct io_uring_getevents_arg { __u64 sigmask; __u32 sigmask_sz; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index bfea5d1fbc67..27ea988e19ba 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2735,6 +2735,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->msg_cache, io_msg_cache_free); io_futex_cache_free(ctx); io_destroy_buffers(ctx); + io_unregister_cqwait_reg(ctx); mutex_unlock(&ctx->uring_lock); if (ctx->sq_creds) put_cred(ctx->sq_creds); @@ -3223,21 +3224,43 @@ void __io_uring_cancel(bool cancel_all) io_uring_cancel_generic(cancel_all, NULL); } -static int io_validate_ext_arg(unsigned flags, const void __user *argp, size_t argsz) +static struct io_uring_reg_wait *io_get_ext_arg_fixed(struct io_ring_ctx *ctx, + const struct io_uring_getevents_arg __user *uarg) { - if (flags & IORING_ENTER_EXT_ARG) { - struct io_uring_getevents_arg arg; + struct io_uring_reg_wait *arg = READ_ONCE(ctx->cq_wait_arg); - if (argsz != sizeof(arg)) + if (arg) { + unsigned int index = (unsigned int) (uintptr_t) uarg; + + if (index <= ctx->cq_wait_index) + return arg + index; + } + + return ERR_PTR(-EFAULT); +} + +static int io_validate_ext_arg(struct io_ring_ctx *ctx, unsigned flags, + const void __user *argp, size_t argsz) +{ + struct io_uring_getevents_arg arg; + + if (!(flags & IORING_ENTER_EXT_ARG)) + return 0; + + if (flags & IORING_ENTER_EXT_ARG_REG) { + if (argsz != sizeof(struct io_uring_reg_wait)) return -EINVAL; - if (copy_from_user(&arg, argp, sizeof(arg))) - return -EFAULT; + return PTR_ERR(io_get_ext_arg_fixed(ctx, argp)); } + if (argsz != sizeof(arg)) + return -EINVAL; + if (copy_from_user(&arg, argp, sizeof(arg))) + return -EFAULT; return 0; } -static int io_get_ext_arg(unsigned flags, const void __user *argp, - struct ext_arg *ext_arg) +static int io_get_ext_arg(struct io_ring_ctx *ctx, unsigned flags, + const void __user *argp, struct ext_arg *ext_arg) { const struct io_uring_getevents_arg __user *uarg = argp; struct io_uring_getevents_arg arg; @@ -3251,6 +3274,28 @@ static int io_get_ext_arg(unsigned flags, const void __user *argp, return 0; } + if (flags & IORING_ENTER_EXT_ARG_REG) { + struct io_uring_reg_wait *w; + + if (ext_arg->argsz != sizeof(struct io_uring_reg_wait)) + return -EINVAL; + w = io_get_ext_arg_fixed(ctx, argp); + if (IS_ERR(w)) + return PTR_ERR(w); + + if (w->flags & ~IORING_REG_WAIT_TS) + return -EINVAL; + ext_arg->min_time = READ_ONCE(w->min_wait_usec) * NSEC_PER_USEC; + ext_arg->sig = u64_to_user_ptr(READ_ONCE(w->sigmask)); + ext_arg->argsz = READ_ONCE(w->sigmask_sz); + if (w->flags & IORING_REG_WAIT_TS) { + ext_arg->ts.tv_sec = READ_ONCE(w->ts.tv_sec); + ext_arg->ts.tv_nsec = READ_ONCE(w->ts.tv_nsec); + ext_arg->ts_set = true; + } + return 0; + } + /* * EXT_ARG is set - ensure we agree on the size of it and copy in our * timespec and sigset_t pointers if good. @@ -3295,7 +3340,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP | IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG | IORING_ENTER_REGISTERED_RING | - IORING_ENTER_ABS_TIMER))) + IORING_ENTER_ABS_TIMER | + IORING_ENTER_EXT_ARG_REG))) return -EINVAL; /* @@ -3378,7 +3424,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, */ mutex_lock(&ctx->uring_lock); iopoll_locked: - ret2 = io_validate_ext_arg(flags, argp, argsz); + ret2 = io_validate_ext_arg(ctx, flags, argp, argsz); if (likely(!ret2)) { min_complete = min(min_complete, ctx->cq_entries); @@ -3388,7 +3434,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, } else { struct ext_arg ext_arg = { .argsz = argsz }; - ret2 = io_get_ext_arg(flags, argp, &ext_arg); + ret2 = io_get_ext_arg(ctx, flags, argp, &ext_arg); if (likely(!ret2)) { min_complete = min(min_complete, ctx->cq_entries); diff --git a/io_uring/register.c b/io_uring/register.c index e38d83c8bbf1..8cc293c6c7a7 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -532,6 +532,79 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) return 0; } +void io_unregister_cqwait_reg(struct io_ring_ctx *ctx) +{ + unsigned short npages = 1; + + if (!ctx->cq_wait_page) + return; + + io_pages_unmap(ctx->cq_wait_arg, &ctx->cq_wait_page, &npages, true); + ctx->cq_wait_arg = NULL; + if (ctx->user) + __io_unaccount_mem(ctx->user, 1); +} + +/* + * Register a page holding N entries of struct io_uring_reg_wait, which can + * be used via io_uring_enter(2) if IORING_GETEVENTS_EXT_ARG_REG is set. + * If that is set with IORING_GETEVENTS_EXT_ARG, then instead of passing + * in a pointer for a struct io_uring_getevents_arg, an index into this + * registered array is passed, avoiding two (arg + timeout) copies per + * invocation. + */ +static int io_register_cqwait_reg(struct io_ring_ctx *ctx, void __user *uarg, + unsigned nr_args) +{ + struct io_uring_reg_wait *arg; + struct page **pages; + unsigned long len; + int nr_pages, poff; + int ret; + + if (ctx->cq_wait_page || ctx->cq_wait_arg) + return -EBUSY; + /* must be >= 1 and must fit within a page */ + if (!nr_args || nr_args > PAGE_SIZE / sizeof(*arg)) + return -EINVAL; + if (check_mul_overflow(sizeof(*arg), nr_args, &len)) + return -EOVERFLOW; + if (len > PAGE_SIZE) + return -EINVAL; + /* offset + len must fit within a page, and must be reg_wait aligned */ + poff = (unsigned long) uarg & ~PAGE_MASK; + if (len + poff > PAGE_SIZE) + return -EINVAL; + if (poff % sizeof(struct io_uring_reg_wait)) + return -EINVAL; + + pages = io_pin_pages((unsigned long) uarg, len, &nr_pages); + if (IS_ERR(pages)) + return PTR_ERR(pages); + ret = -EINVAL; + if (nr_pages != 1) + goto out_free; + if (ctx->user) { + ret = __io_account_mem(ctx->user, 1); + if (ret) + goto out_free; + } + + arg = vmap(pages, 1, VM_MAP, PAGE_KERNEL); + if (arg) { + ctx->cq_wait_index = nr_args - 1; + WRITE_ONCE(ctx->cq_wait_page, pages); + WRITE_ONCE(ctx->cq_wait_arg, (void *) arg + poff); + return 0; + } + ret = -ENOMEM; + if (ctx->user) + __io_unaccount_mem(ctx->user, 1); +out_free: + io_pages_free(&pages, nr_pages); + return ret; +} + static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, void __user *arg, unsigned nr_args) __releases(ctx->uring_lock) @@ -726,6 +799,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_register_resize_rings(ctx, arg); break; + case IORING_REGISTER_CQWAIT_REG: + ret = -EINVAL; + if (!arg) + break; + ret = io_register_cqwait_reg(ctx, arg, nr_args); + break; default: ret = -EINVAL; break; diff --git a/io_uring/register.h b/io_uring/register.h index a5f39d5ef9e0..3e935e8fa4b2 100644 --- a/io_uring/register.h +++ b/io_uring/register.h @@ -5,5 +5,6 @@ int io_eventfd_unregister(struct io_ring_ctx *ctx); int io_unregister_personality(struct io_ring_ctx *ctx, unsigned id); struct file *io_uring_register_get_file(unsigned int fd, bool registered); +void io_unregister_cqwait_reg(struct io_ring_ctx *ctx); #endif