From patchwork Thu Aug 13 15:32:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Garzarella X-Patchwork-Id: 11712603 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B32A5109B for ; Thu, 13 Aug 2020 15:33:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9646C20866 for ; Thu, 13 Aug 2020 15:33:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ZO1pXnIf" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726684AbgHMPd3 (ORCPT ); Thu, 13 Aug 2020 11:33:29 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:22028 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726427AbgHMPd2 (ORCPT ); Thu, 13 Aug 2020 11:33:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1597332807; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3NyHfHbF6ww7AOILglJMvLxTHqil+Il52XX3iacaG1c=; b=ZO1pXnIf+yadrE8FVwvSWUCf9Jt5IRUNMh4IyKlOxGOyXDPduPuzzNzBPsrlkX9vbuiNqH kHpLqOogUAdoWL830BUo5yjuAWYRmaS43v+Q8khEKEyFl2WsQvMQhNnVg+U/vd6kRoiOqH ErsdQPW+bD3A9XWsbqjkVdXyOSbmG60= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-277-P_6uw1VhOiiPCWSeelheKQ-1; Thu, 13 Aug 2020 11:33:25 -0400 X-MC-Unique: P_6uw1VhOiiPCWSeelheKQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id A36F5807323; Thu, 13 Aug 2020 15:33:23 +0000 (UTC) Received: from steredhat.redhat.com (ovpn-113-140.ams2.redhat.com [10.36.113.140]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4E52260C04; Thu, 13 Aug 2020 15:33:18 +0000 (UTC) From: Stefano Garzarella To: Jens Axboe Cc: Christian Brauner , Jann Horn , Jeff Moyer , linux-fsdevel@vger.kernel.org, Sargun Dhillon , Kees Cook , Alexander Viro , Kernel Hardening , Stefan Hajnoczi , linux-kernel@vger.kernel.org, Aleksa Sarai , io-uring@vger.kernel.org Subject: [PATCH v4 1/3] io_uring: use an enumeration for io_uring_register(2) opcodes Date: Thu, 13 Aug 2020 17:32:52 +0200 Message-Id: <20200813153254.93731-2-sgarzare@redhat.com> In-Reply-To: <20200813153254.93731-1-sgarzare@redhat.com> References: <20200813153254.93731-1-sgarzare@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The enumeration allows us to keep track of the last io_uring_register(2) opcode available. Behaviour and opcodes names don't change. Signed-off-by: Stefano Garzarella Reviewed-by: Kees Cook --- include/uapi/linux/io_uring.h | 27 ++++++++++++++++----------- 1 file changed, 16 insertions(+), 11 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index d65fde732518..cdc98afbacc3 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -255,17 +255,22 @@ struct io_uring_params { /* * io_uring_register(2) opcodes and arguments */ -#define IORING_REGISTER_BUFFERS 0 -#define IORING_UNREGISTER_BUFFERS 1 -#define IORING_REGISTER_FILES 2 -#define IORING_UNREGISTER_FILES 3 -#define IORING_REGISTER_EVENTFD 4 -#define IORING_UNREGISTER_EVENTFD 5 -#define IORING_REGISTER_FILES_UPDATE 6 -#define IORING_REGISTER_EVENTFD_ASYNC 7 -#define IORING_REGISTER_PROBE 8 -#define IORING_REGISTER_PERSONALITY 9 -#define IORING_UNREGISTER_PERSONALITY 10 +enum { + IORING_REGISTER_BUFFERS, + IORING_UNREGISTER_BUFFERS, + IORING_REGISTER_FILES, + IORING_UNREGISTER_FILES, + IORING_REGISTER_EVENTFD, + IORING_UNREGISTER_EVENTFD, + IORING_REGISTER_FILES_UPDATE, + IORING_REGISTER_EVENTFD_ASYNC, + IORING_REGISTER_PROBE, + IORING_REGISTER_PERSONALITY, + IORING_UNREGISTER_PERSONALITY, + + /* this goes last */ + IORING_REGISTER_LAST +}; struct io_uring_files_update { __u32 offset; From patchwork Thu Aug 13 15:32:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Garzarella X-Patchwork-Id: 11712611 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22ACB739 for ; Thu, 13 Aug 2020 15:34:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EC6492087C for ; Thu, 13 Aug 2020 15:34:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="MkI5t4BT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726807AbgHMPd7 (ORCPT ); Thu, 13 Aug 2020 11:33:59 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:42348 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726732AbgHMPd5 (ORCPT ); Thu, 13 Aug 2020 11:33:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1597332835; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zz3etVbVEOjvsPy3a1rmULc9GOmqKniLRh7neDKkaXw=; b=MkI5t4BTb5auEeLyfUa/WVmdtrwmE5zY5LCdEhLyUILI0SKnRH5Ait22CuQyEZkls/UFCp 2Koq4t9/kGb5MigeN3nlA7yFCw5G1hTlcpXa+JaKEmepu3Y8V5syluWQKtdr2+wxMj8BMg B8D0P5YWP5Zr9TGcHAx706e0npRc/0E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-149-NF_LSg7jPDCoEHuwC5VjVA-1; Thu, 13 Aug 2020 11:33:51 -0400 X-MC-Unique: NF_LSg7jPDCoEHuwC5VjVA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6C8E4183586C; Thu, 13 Aug 2020 15:33:49 +0000 (UTC) Received: from steredhat.redhat.com (ovpn-113-140.ams2.redhat.com [10.36.113.140]) by smtp.corp.redhat.com (Postfix) with ESMTP id 26D7560C04; Thu, 13 Aug 2020 15:33:23 +0000 (UTC) From: Stefano Garzarella To: Jens Axboe Cc: Christian Brauner , Jann Horn , Jeff Moyer , linux-fsdevel@vger.kernel.org, Sargun Dhillon , Kees Cook , Alexander Viro , Kernel Hardening , Stefan Hajnoczi , linux-kernel@vger.kernel.org, Aleksa Sarai , io-uring@vger.kernel.org Subject: [PATCH v4 2/3] io_uring: add IOURING_REGISTER_RESTRICTIONS opcode Date: Thu, 13 Aug 2020 17:32:53 +0200 Message-Id: <20200813153254.93731-3-sgarzare@redhat.com> In-Reply-To: <20200813153254.93731-1-sgarzare@redhat.com> References: <20200813153254.93731-1-sgarzare@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org The new io_uring_register(2) IOURING_REGISTER_RESTRICTIONS opcode permanently installs a feature allowlist on an io_ring_ctx. The io_ring_ctx can then be passed to untrusted code with the knowledge that only operations present in the allowlist can be executed. The allowlist approach ensures that new features added to io_uring do not accidentally become available when an existing application is launched on a newer kernel version. Currently is it possible to restrict sqe opcodes, sqe flags, and register opcodes. IOURING_REGISTER_RESTRICTIONS can only be made once. Afterwards it is not possible to change restrictions anymore. This prevents untrusted code from removing restrictions. Suggested-by: Stefan Hajnoczi Signed-off-by: Stefano Garzarella Reported-by: kernel test robot Reviewed-by: Kees Cook --- v3: - added IORING_RESTRICTION_SQE_FLAGS_ALLOWED and IORING_RESTRICTION_SQE_FLAGS_REQUIRED - removed IORING_RESTRICTION_FIXED_FILES_ONLY RFC v2: - added 'restricted' flag in the ctx [Jens] - added IORING_MAX_RESTRICTIONS define - returned EBUSY instead of EINVAL when restrictions are already registered - reset restrictions if an error happened during the registration --- fs/io_uring.c | 112 +++++++++++++++++++++++++++++++++- include/uapi/linux/io_uring.h | 31 ++++++++++ 2 files changed, 142 insertions(+), 1 deletion(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 1ec25ee71372..cb365e6e0af7 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -98,6 +98,8 @@ #define IORING_MAX_FILES_TABLE (1U << IORING_FILE_TABLE_SHIFT) #define IORING_FILE_TABLE_MASK (IORING_MAX_FILES_TABLE - 1) #define IORING_MAX_FIXED_FILES (64 * IORING_MAX_FILES_TABLE) +#define IORING_MAX_RESTRICTIONS (IORING_RESTRICTION_LAST + \ + IORING_REGISTER_LAST + IORING_OP_LAST) struct io_uring { u32 head ____cacheline_aligned_in_smp; @@ -219,6 +221,13 @@ struct io_buffer { __u16 bid; }; +struct io_restriction { + DECLARE_BITMAP(register_op, IORING_REGISTER_LAST); + DECLARE_BITMAP(sqe_op, IORING_OP_LAST); + u8 sqe_flags_allowed; + u8 sqe_flags_required; +}; + struct io_ring_ctx { struct { struct percpu_ref refs; @@ -231,6 +240,7 @@ struct io_ring_ctx { unsigned int cq_overflow_flushed: 1; unsigned int drain_next: 1; unsigned int eventfd_async: 1; + unsigned int restricted: 1; /* * Ring buffer of indices into array of io_uring_sqe, which is @@ -338,6 +348,7 @@ struct io_ring_ctx { struct llist_head file_put_llist; struct work_struct exit_work; + struct io_restriction restrictions; }; /* @@ -6353,6 +6364,19 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, if (unlikely(sqe_flags & ~SQE_VALID_FLAGS)) return -EINVAL; + if (unlikely(ctx->restricted)) { + if (!test_bit(req->opcode, ctx->restrictions.sqe_op)) + return -EACCES; + + if ((sqe_flags & ctx->restrictions.sqe_flags_required) != + ctx->restrictions.sqe_flags_required) + return -EACCES; + + if (sqe_flags & ~(ctx->restrictions.sqe_flags_allowed | + ctx->restrictions.sqe_flags_required)) + return -EACCES; + } + if ((sqe_flags & IOSQE_BUFFER_SELECT) && !io_op_defs[req->opcode].buffer_select) return -EOPNOTSUPP; @@ -8650,6 +8674,76 @@ static int io_unregister_personality(struct io_ring_ctx *ctx, unsigned id) return -EINVAL; } +static int io_register_restrictions(struct io_ring_ctx *ctx, void __user *arg, + unsigned int nr_args) +{ + struct io_uring_restriction *res; + size_t size; + int i, ret; + + /* We allow only a single restrictions registration */ + if (ctx->restricted) + return -EBUSY; + + if (!arg || nr_args > IORING_MAX_RESTRICTIONS) + return -EINVAL; + + size = array_size(nr_args, sizeof(*res)); + if (size == SIZE_MAX) + return -EOVERFLOW; + + res = kmalloc(size, GFP_KERNEL); + if (!res) + return -ENOMEM; + + if (copy_from_user(res, arg, size)) { + ret = -EFAULT; + goto out; + } + + for (i = 0; i < nr_args; i++) { + switch (res[i].opcode) { + case IORING_RESTRICTION_REGISTER_OP: + if (res[i].register_op >= IORING_REGISTER_LAST) { + ret = -EINVAL; + goto out; + } + + __set_bit(res[i].register_op, + ctx->restrictions.register_op); + break; + case IORING_RESTRICTION_SQE_OP: + if (res[i].sqe_op >= IORING_OP_LAST) { + ret = -EINVAL; + goto out; + } + + __set_bit(res[i].sqe_op, ctx->restrictions.sqe_op); + break; + case IORING_RESTRICTION_SQE_FLAGS_ALLOWED: + ctx->restrictions.sqe_flags_allowed = res[i].sqe_flags; + break; + case IORING_RESTRICTION_SQE_FLAGS_REQUIRED: + ctx->restrictions.sqe_flags_required = res[i].sqe_flags; + break; + default: + ret = -EINVAL; + goto out; + } + } + + ctx->restricted = 1; + + ret = 0; +out: + /* Reset all restrictions if an error happened */ + if (ret != 0) + memset(&ctx->restrictions, 0, sizeof(ctx->restrictions)); + + kfree(res); + return ret; +} + static bool io_register_op_must_quiesce(int op) { switch (op) { @@ -8696,6 +8790,18 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, if (ret) { percpu_ref_resurrect(&ctx->refs); ret = -EINTR; + goto out_quiesce; + } + } + + if (ctx->restricted) { + if (opcode >= IORING_REGISTER_LAST) { + ret = -EINVAL; + goto out; + } + + if (!test_bit(opcode, ctx->restrictions.register_op)) { + ret = -EACCES; goto out; } } @@ -8759,15 +8865,19 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_unregister_personality(ctx, nr_args); break; + case IORING_REGISTER_RESTRICTIONS: + ret = io_register_restrictions(ctx, arg, nr_args); + break; default: ret = -EINVAL; break; } +out: if (io_register_op_must_quiesce(opcode)) { /* bring the ctx back to life */ percpu_ref_reinit(&ctx->refs); -out: +out_quiesce: reinit_completion(&ctx->ref_comp); } return ret; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index cdc98afbacc3..be54bc3cf173 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -267,6 +267,7 @@ enum { IORING_REGISTER_PROBE, IORING_REGISTER_PERSONALITY, IORING_UNREGISTER_PERSONALITY, + IORING_REGISTER_RESTRICTIONS, /* this goes last */ IORING_REGISTER_LAST @@ -295,4 +296,34 @@ struct io_uring_probe { struct io_uring_probe_op ops[0]; }; +struct io_uring_restriction { + __u16 opcode; + union { + __u8 register_op; /* IORING_RESTRICTION_REGISTER_OP */ + __u8 sqe_op; /* IORING_RESTRICTION_SQE_OP */ + __u8 sqe_flags; /* IORING_RESTRICTION_SQE_FLAGS_* */ + }; + __u8 resv; + __u32 resv2[3]; +}; + +/* + * io_uring_restriction->opcode values + */ +enum { + /* Allow an io_uring_register(2) opcode */ + IORING_RESTRICTION_REGISTER_OP, + + /* Allow an sqe opcode */ + IORING_RESTRICTION_SQE_OP, + + /* Allow sqe flags */ + IORING_RESTRICTION_SQE_FLAGS_ALLOWED, + + /* Require sqe flags (these flags must be set on each submission) */ + IORING_RESTRICTION_SQE_FLAGS_REQUIRED, + + IORING_RESTRICTION_LAST +}; + #endif From patchwork Thu Aug 13 15:32:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefano Garzarella X-Patchwork-Id: 11712617 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 53C7A739 for ; Thu, 13 Aug 2020 15:34:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2AB2920781 for ; Thu, 13 Aug 2020 15:34:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="bkVQWdea" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726870AbgHMPeR (ORCPT ); Thu, 13 Aug 2020 11:34:17 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:47468 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726844AbgHMPeQ (ORCPT ); Thu, 13 Aug 2020 11:34:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1597332854; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xzivr37W6Sz/OFkat8WISQrphMPdi2PlV4uzOfubSgg=; b=bkVQWdeaeCSY2Xu/HmtuTS+Gio4ZCrfUgEDQl82PWa2+ZcmsDiNupzq3plmaghmyh/PiBC X03J9tUG7Jhqkti1vaAzubsqa5iU5igYOm1OjrqGBA9F3U+Xak4thA5+7FM0ipu3rhp7h4 Wb4Fno/f08dsnJ9U/lzg2g0gDJnbxzg= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-391-14sqnvJ0PGe7-Ac8Ox3DLQ-1; Thu, 13 Aug 2020 11:34:10 -0400 X-MC-Unique: 14sqnvJ0PGe7-Ac8Ox3DLQ-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id C53A1807321; Thu, 13 Aug 2020 15:34:08 +0000 (UTC) Received: from steredhat.redhat.com (ovpn-113-140.ams2.redhat.com [10.36.113.140]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6443C60C04; Thu, 13 Aug 2020 15:33:49 +0000 (UTC) From: Stefano Garzarella To: Jens Axboe Cc: Christian Brauner , Jann Horn , Jeff Moyer , linux-fsdevel@vger.kernel.org, Sargun Dhillon , Kees Cook , Alexander Viro , Kernel Hardening , Stefan Hajnoczi , linux-kernel@vger.kernel.org, Aleksa Sarai , io-uring@vger.kernel.org Subject: [PATCH v4 3/3] io_uring: allow disabling rings during the creation Date: Thu, 13 Aug 2020 17:32:54 +0200 Message-Id: <20200813153254.93731-4-sgarzare@redhat.com> In-Reply-To: <20200813153254.93731-1-sgarzare@redhat.com> References: <20200813153254.93731-1-sgarzare@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org This patch adds a new IORING_SETUP_R_DISABLED flag to start the rings disabled, allowing the user to register restrictions, buffers, files, before to start processing SQEs. When IORING_SETUP_R_DISABLED is set, SQE are not processed and SQPOLL kthread is not started. The restrictions registration are allowed only when the rings are disable to prevent concurrency issue while processing SQEs. The rings can be enabled using IORING_REGISTER_ENABLE_RINGS opcode with io_uring_register(2). Suggested-by: Jens Axboe Signed-off-by: Stefano Garzarella Reviewed-by: Kees Cook --- v4: - fixed io_uring_enter() exit path when ring is disabled v3: - enabled restrictions only when the rings start RFC v2: - removed return value of io_sq_offload_start() --- fs/io_uring.c | 52 ++++++++++++++++++++++++++++++----- include/uapi/linux/io_uring.h | 2 ++ 2 files changed, 47 insertions(+), 7 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index cb365e6e0af7..09fedc380a41 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -226,6 +226,7 @@ struct io_restriction { DECLARE_BITMAP(sqe_op, IORING_OP_LAST); u8 sqe_flags_allowed; u8 sqe_flags_required; + bool registered; }; struct io_ring_ctx { @@ -7420,8 +7421,8 @@ static int io_init_wq_offload(struct io_ring_ctx *ctx, return ret; } -static int io_sq_offload_start(struct io_ring_ctx *ctx, - struct io_uring_params *p) +static int io_sq_offload_create(struct io_ring_ctx *ctx, + struct io_uring_params *p) { int ret; @@ -7458,7 +7459,6 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx, ctx->sqo_thread = NULL; goto err; } - wake_up_process(ctx->sqo_thread); } else if (p->flags & IORING_SETUP_SQ_AFF) { /* Can't have SQ_AFF without SQPOLL */ ret = -EINVAL; @@ -7479,6 +7479,12 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx, return ret; } +static void io_sq_offload_start(struct io_ring_ctx *ctx) +{ + if ((ctx->flags & IORING_SETUP_SQPOLL) && ctx->sqo_thread) + wake_up_process(ctx->sqo_thread); +} + static inline void __io_unaccount_mem(struct user_struct *user, unsigned long nr_pages) { @@ -8218,6 +8224,9 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, if (!percpu_ref_tryget(&ctx->refs)) goto out_fput; + if (ctx->flags & IORING_SETUP_R_DISABLED) + goto out_fput; + /* * For SQ polling, the thread will do all submissions and completions. * Just return the requested submit count, and wake the thread if @@ -8532,10 +8541,13 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p, if (ret) goto err; - ret = io_sq_offload_start(ctx, p); + ret = io_sq_offload_create(ctx, p); if (ret) goto err; + if (!(p->flags & IORING_SETUP_R_DISABLED)) + io_sq_offload_start(ctx); + memset(&p->sq_off, 0, sizeof(p->sq_off)); p->sq_off.head = offsetof(struct io_rings, sq.head); p->sq_off.tail = offsetof(struct io_rings, sq.tail); @@ -8598,7 +8610,8 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params) if (p.flags & ~(IORING_SETUP_IOPOLL | IORING_SETUP_SQPOLL | IORING_SETUP_SQ_AFF | IORING_SETUP_CQSIZE | - IORING_SETUP_CLAMP | IORING_SETUP_ATTACH_WQ)) + IORING_SETUP_CLAMP | IORING_SETUP_ATTACH_WQ | + IORING_SETUP_R_DISABLED)) return -EINVAL; return io_uring_create(entries, &p, params); @@ -8681,8 +8694,12 @@ static int io_register_restrictions(struct io_ring_ctx *ctx, void __user *arg, size_t size; int i, ret; + /* Restrictions allowed only if rings started disabled */ + if (!(ctx->flags & IORING_SETUP_R_DISABLED)) + return -EINVAL; + /* We allow only a single restrictions registration */ - if (ctx->restricted) + if (ctx->restrictions.registered) return -EBUSY; if (!arg || nr_args > IORING_MAX_RESTRICTIONS) @@ -8732,7 +8749,7 @@ static int io_register_restrictions(struct io_ring_ctx *ctx, void __user *arg, } } - ctx->restricted = 1; + ctx->restrictions.registered = true; ret = 0; out: @@ -8744,6 +8761,21 @@ static int io_register_restrictions(struct io_ring_ctx *ctx, void __user *arg, return ret; } +static int io_register_enable_rings(struct io_ring_ctx *ctx) +{ + if (!(ctx->flags & IORING_SETUP_R_DISABLED)) + return -EINVAL; + + if (ctx->restrictions.registered) + ctx->restricted = 1; + + ctx->flags &= ~IORING_SETUP_R_DISABLED; + + io_sq_offload_start(ctx); + + return 0; +} + static bool io_register_op_must_quiesce(int op) { switch (op) { @@ -8865,6 +8897,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode, break; ret = io_unregister_personality(ctx, nr_args); break; + case IORING_REGISTER_ENABLE_RINGS: + ret = -EINVAL; + if (arg || nr_args) + break; + ret = io_register_enable_rings(ctx); + break; case IORING_REGISTER_RESTRICTIONS: ret = io_register_restrictions(ctx, arg, nr_args); break; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index be54bc3cf173..ddb30513e027 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -95,6 +95,7 @@ enum { #define IORING_SETUP_CQSIZE (1U << 3) /* app defines CQ size */ #define IORING_SETUP_CLAMP (1U << 4) /* clamp SQ/CQ ring sizes */ #define IORING_SETUP_ATTACH_WQ (1U << 5) /* attach to existing wq */ +#define IORING_SETUP_R_DISABLED (1U << 6) /* start with ring disabled */ enum { IORING_OP_NOP, @@ -268,6 +269,7 @@ enum { IORING_REGISTER_PERSONALITY, IORING_UNREGISTER_PERSONALITY, IORING_REGISTER_RESTRICTIONS, + IORING_REGISTER_ENABLE_RINGS, /* this goes last */ IORING_REGISTER_LAST