From patchwork Mon Oct 28 14:52:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853647 Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 138521D9681 for ; Mon, 28 Oct 2024 15:04:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127888; cv=none; b=tROblNcPxvgDiF2efMTihmZgUcU/foaCcg6BbkJBp9u7puQIVUeikKtF33tykG5GfN7Bg6e/PeprBZPxITVn28DHh0BzH2/1ym49KXaULCfnUtD1IeogwLPp2B4wioz9Qd8RXCWi6w6DnQZEPHp4lk0dWI+tTETYmwuouE/MgzI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127888; c=relaxed/simple; bh=HPnvI+rOjDAWZqF0Xd42uWJKF/3qtcinc1VyXx9MXy4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h4+q+sL/2w9oqU5ztGVf8EJlafPhez3pcPKQO16BNWkhnr/hDvObhBRZmk0KXd+DjU4QAn1NQ1ksOwpn3ZGyB1cuE2OMwQZsQikh4C425qqnnLICfYcVHhDbgrlrIQZ6/xQcBGuf6xI8V+4U7KWmOMx7GqU+BEp/+7Srxpmg4Gs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=WJz+OxFU; arc=none smtp.client-ip=209.85.166.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="WJz+OxFU" Received: by mail-il1-f181.google.com with SMTP id e9e14a558f8ab-3a3aeb19ea2so16000385ab.0 for ; Mon, 28 Oct 2024 08:04:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127884; x=1730732684; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lk/+RVPeoujW1QBGROEzXF5c7w5OPLyPUXaokbUSoj0=; b=WJz+OxFUIe+DfhHgsVZ0nDBs6PpiZ9xQetozra0lsfAHbXOcUwe2CUhsljaVxa0LL3 rojUQsuP2kaeGntrltjjE3FXRERIu1BjvTKlkXectjSwGjaD8s01yy3nrVuKukLZMgiI q7E+S1P8ZHUpxGwF4Jt24MAIL17iX/lk0eQao9DC96m4hi3NRPWZv35YDYNW75MkHu7+ PnknCmt4osvTH1cRIdzzFpoJuJfVkEai/e4LWYmyEW14M7qOeogFndL2fB+SQ/GvPJoK sBbSk3PtmM+Fw+4/IcDN+wP+U103//dK6I+fX81nSzLxvSPHBOTeAc9X+tvEc9pBYH0v QCLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127884; x=1730732684; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lk/+RVPeoujW1QBGROEzXF5c7w5OPLyPUXaokbUSoj0=; b=qBzKTkilGtL4PrkJ/EvmAYc6+juaWSvAI6Zl6TXjQZSWNQ+R5Sxqhlu0IWzBDxOCxF F8rzpI8b7Kix75bfQK+lP9vyC9jSfkWrerY1pwW7LZdt/nNQBbs4C1AakvdP8RALjiSb 8kCUmrK6NPNf+ce577SdxZzQ3JDU41JSHJGPkCDxz4eAnXqrhkl+ghMsalaup7Jzk/JV e456BcAT1KLHdnVxTj11D/byoOqwbzx5LbGb2GUUwyHhwrcuQqL3k7id+mkqbZSE1dg/ HljMZwWmUfs3edUj5DDkPVlUtfwauF/pSfMsDc/W2/qFBKfjD+aJoIBWM2ZUaTPOSvN7 zW7g== X-Gm-Message-State: AOJu0YxLu+VyqiR7ZB5zzvSsKrBWvtPiJbIuC8W0WdpKMIuoW9znyR0g Pf1EezLnS5mndmxO1NFYYYNZjzH4Z8q3Du2mNoGLNRukuOyr4ww13Yz+GRyQeLr8dHIgJs2C/ZR A X-Google-Smtp-Source: AGHT+IG5i6GgjRUwZBstb32BYDeJPkTd3KNhTLPLHcgtEo1HaqZMq7xT07taUC8OMvEXruMNl5W2zQ== X-Received: by 2002:a05:6e02:1d01:b0:3a0:9829:100b with SMTP id e9e14a558f8ab-3a4ed2fcbc6mr66461455ab.21.1730127884126; Mon, 28 Oct 2024 08:04:44 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:41 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 01/13] io_uring/nop: add support for testing registered files and buffers Date: Mon, 28 Oct 2024 08:52:31 -0600 Message-ID: <20241028150437.387667-2-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Useful for testing performance/efficiency impact of registered files and buffers, vs (particularly) non-registered files. Signed-off-by: Jens Axboe --- include/uapi/linux/io_uring.h | 3 +++ io_uring/nop.c | 49 +++++++++++++++++++++++++++++++---- 2 files changed, 47 insertions(+), 5 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 65b7417c1b05..024745283783 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -416,6 +416,9 @@ enum io_uring_msg_ring_flags { * IORING_NOP_INJECT_RESULT Inject result from sqe->result */ #define IORING_NOP_INJECT_RESULT (1U << 0) +#define IORING_NOP_FILE (1U << 1) +#define IORING_NOP_FIXED_FILE (1U << 2) +#define IORING_NOP_FIXED_BUFFER (1U << 3) /* * IO completion data structure (Completion Queue Entry) diff --git a/io_uring/nop.c b/io_uring/nop.c index a5bcf3d6984f..2c7a22ba4053 100644 --- a/io_uring/nop.c +++ b/io_uring/nop.c @@ -8,35 +8,74 @@ #include #include "io_uring.h" +#include "rsrc.h" #include "nop.h" struct io_nop { /* NOTE: kiocb has the file as the first member, so don't do it here */ struct file *file; int result; + int fd; + int buffer; + unsigned int flags; }; +#define NOP_FLAGS (IORING_NOP_INJECT_RESULT | IORING_NOP_FIXED_FILE | \ + IORING_NOP_FIXED_BUFFER | IORING_NOP_FILE) + int io_nop_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { - unsigned int flags; struct io_nop *nop = io_kiocb_to_cmd(req, struct io_nop); - flags = READ_ONCE(sqe->nop_flags); - if (flags & ~IORING_NOP_INJECT_RESULT) + nop->flags = READ_ONCE(sqe->nop_flags); + if (nop->flags & ~NOP_FLAGS) return -EINVAL; - if (flags & IORING_NOP_INJECT_RESULT) + if (nop->flags & IORING_NOP_INJECT_RESULT) nop->result = READ_ONCE(sqe->len); else nop->result = 0; + if (nop->flags & IORING_NOP_FIXED_FILE) + nop->fd = READ_ONCE(sqe->fd); + if (nop->flags & IORING_NOP_FIXED_BUFFER) + nop->buffer = READ_ONCE(sqe->buf_index); return 0; } int io_nop(struct io_kiocb *req, unsigned int issue_flags) { struct io_nop *nop = io_kiocb_to_cmd(req, struct io_nop); + int ret = nop->result; + + if (nop->flags & IORING_NOP_FILE) { + if (nop->flags & IORING_NOP_FIXED_FILE) { + req->file = io_file_get_fixed(req, nop->fd, issue_flags); + req->flags |= REQ_F_FIXED_FILE; + } else { + req->file = io_file_get_normal(req, nop->fd); + } + if (!req->file) { + ret = -EBADF; + goto done; + } + } + if (nop->flags & IORING_NOP_FIXED_BUFFER) { + struct io_ring_ctx *ctx = req->ctx; + struct io_mapped_ubuf *imu; + int idx; - if (nop->result < 0) + ret = -EFAULT; + io_ring_submit_lock(ctx, issue_flags); + if (nop->buffer < ctx->nr_user_bufs) { + idx = array_index_nospec(nop->buffer, ctx->nr_user_bufs); + imu = READ_ONCE(ctx->user_bufs[idx]); + io_req_set_rsrc_node(req, ctx); + ret = 0; + } + io_ring_submit_unlock(ctx, issue_flags); + } +done: + if (ret < 0) req_set_fail(req); io_req_set_res(req, nop->result, 0); return IOU_OK; From patchwork Mon Oct 28 14:52:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853648 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ACACB1DD864 for ; Mon, 28 Oct 2024 15:04:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127889; cv=none; b=O0PnlSLg0VMtN7r3Y8O+iuYbOMAAOkiSgheTOX5+cgEQhLy2TujSEcfwPrNh+H9KiXztTyhCqqIJaW+fJeM1a9YDhEFhFBjnaVWrNINfUhDAlCjydg7m3O3XKVRPagwvgkr0qJROLNR6roFPWj0DIhdkT+bNScLHMpCJPLFcjaQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127889; c=relaxed/simple; bh=gbtUQsMeQx/ihpjCfNEVueNKiMVBzIIvZfHP4mZ+ic0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E/y2IWBUMpUpRCW/WH8sawOp9S37XRQWZjyc5fD/HH/EcWw+rT4Y2EMiVho89x8UGGAjRsY88lKB7sj9hRiWcwwtegG97nkAssUVjumQYR0UNtEng6fQ9eahuYf25jjD39+rkTtHPIxktYVL5gxjC6g5LDLtsGBP4w6/vpdeRw8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=qYuTdNWw; arc=none smtp.client-ip=209.85.210.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="qYuTdNWw" Received: by mail-ot1-f50.google.com with SMTP id 46e09a7af769-71808b6246bso2337520a34.2 for ; Mon, 28 Oct 2024 08:04:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127885; x=1730732685; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kWafocjdEh3Pob5N+inSXBHHJdCi0UP8BjhLCBT+Hd8=; b=qYuTdNWw1k4tUwU4s0qc12pSojRHVEq2y/7V/pZ4tIDt0ZjMMwiEf26wAN7LN8lQh9 hlmxuxkg8MNXuf5+hMNfCb0w7DqOmQhbzoBnaCNhENjQwuhnc6iWrJFTIFaIoMGEcDT4 ZC1QhyLmG1LMQgCXaXS7wjo+NFtmehTD3LvhaaZ90HFh6t11c+v5Dp/2q/LjHc6KgDyy s3DO+AkENyuNiatu14svFrEwMUO7EV87+FKv1zIymAdNe+/PROCmQWS+3r0V5IyMStyA ageY8ATbreqXrLk1Zaa1gC//bAn4u6A1TvRXEO39ZoUi61O4BKSpBzZkGmnAt2Rgev2H B9RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127885; x=1730732685; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kWafocjdEh3Pob5N+inSXBHHJdCi0UP8BjhLCBT+Hd8=; b=ebb2EbYkP9ki8DjPGJHCyRxaeE2QEmPxZDdXkTQwEE4jd/g1foPvlI4Tl12ERzkHHI nZiSkveZtsm3zxeOjBZ7D1Tt2ZXm8stIDVB8c148vyCfqVZfEGVwNJ+f/09J2cNS6O8W Q0FPvD2hSaEMjMSyCWFPYBTFSMBlH9csZABEwoX8ohDuxPRNPyko3kEE7zTd3FVYaLF1 W9EyprUxQHnMarYK7wROnsg01J0QSjdLA1nscOyBl8/UTi5VhB6m1+f5v8YG84F8aNKs Vc9tRDHyf8vGKkKOMdAknAlVSgZFFlGUdjT2qSYtC6gtYsyaYqtE9MRnGTE/12cIDun5 VP9g== X-Gm-Message-State: AOJu0YykXF+cHfzS6NqmGTF3pS3MlHf8bVPJw+FhkhGvk2UgXya1EmNg poUR6xUmVMyxspjJZtBbr2VirD6aaQEB8MQ5e1fPI2Rup1UVPlMgBvYYGOBH5FgkyeWdnLoXTmw 0 X-Google-Smtp-Source: AGHT+IF07h74mphVF++z2fvF+VTcvBgW8RPaoFZNx69QCaQ3VmHKuJdNd7hVbdbtlerhGFGFu+jBbg== X-Received: by 2002:a05:6830:6311:b0:716:ab1b:6473 with SMTP id 46e09a7af769-7186829708amr7521071a34.30.1730127884954; Mon, 28 Oct 2024 08:04:44 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:44 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 02/13] io_uring/rsrc: move struct io_fixed_file to rsrc.h header Date: Mon, 28 Oct 2024 08:52:32 -0600 Message-ID: <20241028150437.387667-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 There's no need for this internal structure to be visible, move it to the private rsrc.h header instead. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 5 ----- io_uring/filetable.h | 1 + io_uring/rsrc.h | 5 +++++ 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 2f12828b22a4..d4ba4ae480d6 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -55,11 +55,6 @@ struct io_wq_work { int cancel_seq; }; -struct io_fixed_file { - /* file * with additional FFS_* flags */ - unsigned long file_ptr; -}; - struct io_file_table { struct io_fixed_file *files; unsigned long *bitmap; diff --git a/io_uring/filetable.h b/io_uring/filetable.h index b2435c4dca1f..c027ed4ad68d 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -4,6 +4,7 @@ #include #include +#include "rsrc.h" bool io_alloc_file_tables(struct io_file_table *table, unsigned nr_files); void io_free_file_tables(struct io_file_table *table); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index c50d4be4aa6d..e072fb3ee351 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -40,6 +40,11 @@ struct io_rsrc_node { struct io_rsrc_put item; }; +struct io_fixed_file { + /* file * with additional FFS_* flags */ + unsigned long file_ptr; +}; + struct io_mapped_ubuf { u64 ubuf; unsigned int len; From patchwork Mon Oct 28 14:52:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853649 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E52EE1DE2C6 for ; Mon, 28 Oct 2024 15:04:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127889; cv=none; b=FkW6UAMsDu89iV4keUirYV9ohmwO1YWA+bFECgL6NE+mjKkQq3PYe1TC5F0tkn74Bz6wXN7xeslqYv2MzSaRYrrVHsZkUHW+XBPD4a2t04VevUFF3KUeqKqoAL+iH9o9IPzzDt1HSXjPelLD4mBUQs4+ybbD5c+XWgxU+CAOfSM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127889; c=relaxed/simple; bh=SYpQSxC5Dp9fLpjQdEWdmbvundwGSG12kF2zAC1HBac=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RP9frZS8r6xiqH6gEiLlZnaBqji57LKHbC01ZINJohXlOGLCzZiPU90q60Xf9M3XHwCNw4TxcUdRWcmknul16DxY3TkoNm8PjjnILc0w+FV1ArF1IqwrgOrhbvoz5IzGYbZI3rDcgDon2z/8uUdazEEQCd6/3yDEIdmmdCQWcos= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=qAfKAvfo; arc=none smtp.client-ip=209.85.166.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="qAfKAvfo" Received: by mail-io1-f47.google.com with SMTP id ca18e2360f4ac-83ace760016so157744439f.1 for ; Mon, 28 Oct 2024 08:04:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127886; x=1730732686; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yp+EnMS7ATl15usRfPUg+8nUDFsAhh02UXOXNhYJd0A=; b=qAfKAvfo8FtbVrktofBt13AMrXqVAMKSaCam3EhPfKhXLUSGlZVjwmhPwfJAqAfo91 JcWq2n9/sfxL8/wbotN2Y22b15GOCRtu7kQSfUI24NhFCMlLvA1WkZ8ecppx35y2NlNa zYo5+3LkTPj9zhRh+/wfKfvivML3GS1ymj+NOVjxJNIapcA44wYHbhZL0FMGV5PshHId pis6CKaPwbbz4wsfTTbC6xwB1KM09QandpERGisEgcesFvj/QejtyUDtv43h8WxwEj+h Y2At8cVGbT5Rye7HBY8FnoO65GBlHmFYyEeZPdkad9//MDoQSZ3LE+HxLNdQvKgyUPy0 Hm7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127886; x=1730732686; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yp+EnMS7ATl15usRfPUg+8nUDFsAhh02UXOXNhYJd0A=; b=IN28OFljrU1Pc3WI95O1GRl6dSDaNkkIl/5/csn3zvn1EhfNErr6XcrXPOGv10hxw2 jh0FrVHMK6SsxaR5FMpherZbf1AcQOL5cAQgxIxnOCha2Pujrt29oFGM1DY9VyubtaPG j7myCef5+80TZwBxEzwaRbcdw05twxHH5v8bVx8rrfgiEjQWccWhToGes95LVS7k/koc 9Dh9YI8OYu8QqE76BHgWz2U/D9oTpUQMl7XZ5SUwGMT7E7/iLH3xTC3noR0bgOYpU2Lo Dee8aT6scxicjNrIL+1jap9hle8xTBQYrM+rjEnsvFtZTnbHrAnVkv2FzuHVm76elUKF Gt3w== X-Gm-Message-State: AOJu0Ywh5/kFb4bGg8o0bP+/KnmGeNtYI7Lk/QnInOAD4lhuW19X0muT IkhYbXjAHWQwfv+xQ38cyBPvn+DKo498rQR7o8CmmvPKR+GK/bPT6yunkqjQpBUudNxZqNxOGP4 s X-Google-Smtp-Source: AGHT+IFPCVJRr0o8bfaLXcln47vxsWTNjV+fyNh4dc9tz72yFUVwk9sRjZssZ6go0BfeofQuCk45IQ== X-Received: by 2002:a05:6e02:198f:b0:39d:3c87:1435 with SMTP id e9e14a558f8ab-3a50896d7cemr1204655ab.1.1730127886634; Mon, 28 Oct 2024 08:04:46 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:45 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 03/13] io_uring: specify freeptr usage for SLAB_TYPESAFE_BY_RCU io_kiocb cache Date: Mon, 28 Oct 2024 08:52:33 -0600 Message-ID: <20241028150437.387667-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Doesn't matter right now as there's still some bytes left for it, but let's prepare for the io_kiocb potentially growing and add a specific freeptr offset for it. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 2863b957e373..a09c67b38c1b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -3846,6 +3846,8 @@ static int __init io_uring_init(void) struct kmem_cache_args kmem_args = { .useroffset = offsetof(struct io_kiocb, cmd.data), .usersize = sizeof_field(struct io_kiocb, cmd.data), + .freeptr_offset = offsetof(struct io_kiocb, work), + .use_freeptr_offset = true, }; #define __BUILD_BUG_VERIFY_OFFSET_SIZE(stype, eoffset, esize, ename) do { \ From patchwork Mon Oct 28 14:52:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853650 Received: from mail-il1-f179.google.com (mail-il1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 276E41DE3AA for ; Mon, 28 Oct 2024 15:04:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127892; cv=none; b=GkLnqAJ7GFl1Xe+7UTFnhELVpijblxLy0SCLlm3BXKoIfPn4DCxz/LTjdg7wLWY5SU/82oawidfsEt1tBditDe1+R9d46dLOIdo2tsdnuETUhku9zPJyX1efBgh1ihLKj7vsT3g6/FmnYAItR1nS0CSsRL07VZO77Y9a5BQ8Yu0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127892; c=relaxed/simple; bh=MFnBn3Pt+fR8V94brBcdw32DLbRG+lgrEKVj+CSb2kM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Wsa6GAgFlCMPaii+fUxzi/KqOu8nmxJd7XsmJ85MDxVkjIC/JXLDGbYxrChDsIuSvfPl4+5EPzsscwcsZUSGpoRmral6zlprD/x3REx7q5fxWGW+2WDDuTdUSMiN/0FPRv1cYvI1yojK+aoz8nJQssZ8KvlCdcEdVNhNi2HunHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Z7scMJp5; arc=none smtp.client-ip=209.85.166.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Z7scMJp5" Received: by mail-il1-f179.google.com with SMTP id e9e14a558f8ab-3a3b6b281d4so17752255ab.0 for ; Mon, 28 Oct 2024 08:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127888; x=1730732688; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1wzBUpdtpa3hgDqW+hLOo7TP8QbExzAyYMIVCGHCt2s=; b=Z7scMJp56K9VQEnVsD2ADUbRNDU7wpCeg7XJOa8pAtOkzwnerfw8tSD2wDkoXgnOUN 8RDr2mDxqW5Ld7uxCLuDfEFu5HKfsz0N6jaZYmrYq7N77KPi7qrBZcQt+H+lQIMJ3hZo utvYMIWNAlget9C7cWjRtd4c40H/U3gVk4vGRKOvIMGs+jD8wURUmWwGL7mCugp3MJC4 b27qOzwjeXcRnUFCaZsm6R5w5FaY9MF59THorbvHEmq80ykhFqGO5v3WIsPReXpXE5C4 sDgxWdtU+WVPH8U4QtF1ZoFpbXdSfOJ7M2zNm1a/ngH5jUNgrMOrgNvyM3zuxqCroTMZ 284A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127888; x=1730732688; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1wzBUpdtpa3hgDqW+hLOo7TP8QbExzAyYMIVCGHCt2s=; b=JPN0OZGFx/jRdmmN1421J+FJvuk+wE6veLUvr4OmDPyYwssXGRLmbnjzYBn2RhI9Ii D0Pkj3cX8bqmRg/jAnP365DOTpXuywbkQZI6J9Sy4CyBlwcKcggZ78cFCTqIfoT3p1Bk sJDBhIS+WSWi/jIzUd/cJJQuQOqd7Y/DCMUhp/hs6GGH1Jc+U9YkDWaMECKfaK5IadOG J9PyHT/bLCXL0Irmr9UC4GvZNdELoYH8gqRAkvhJ8z8lfbInolav7nddxi7O0wnF4SMN G0/HTeI3BJ6pJ63Vdk2lZkcxETVPh7UVdaXMykO9dOyyqY5ZI/IaCGFcQgxEUTj0ztDv qQWg== X-Gm-Message-State: AOJu0Yy9Pm54+ktAj4u2nhVJ/HWwiwnnuBs3K5RBG9Wnb38aquG4Bhyp wu6gmUOhF8zSfMNkLG3Gh0jJMcePOMpuJspaIexbkSYGREKZVqovwFlxd+z+aeQAPgf6V/NQSwf X X-Google-Smtp-Source: AGHT+IEWlGS4ppe1fXn3O0knG6L0TbNFLTODTpYR8w3WdANEUF4ILqYV4UC4qlstFwSo9j2Tli8atw== X-Received: by 2002:a92:c261:0:b0:3a4:e726:58a6 with SMTP id e9e14a558f8ab-3a4ed27dbdbmr73315645ab.8.1730127888400; Mon, 28 Oct 2024 08:04:48 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:46 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 04/13] io_uring/splice: open code 2nd direct file assignment Date: Mon, 28 Oct 2024 08:52:34 -0600 Message-ID: <20241028150437.387667-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for not pinning the whole registered file table, open code the second potential direct file assignment. This will be handled by appropriate helpers in the future, for now just do it manually. Signed-off-by: Jens Axboe --- io_uring/opdef.c | 2 ++ io_uring/splice.c | 44 ++++++++++++++++++++++++++++++++++++-------- io_uring/splice.h | 1 + 3 files changed, 39 insertions(+), 8 deletions(-) diff --git a/io_uring/opdef.c b/io_uring/opdef.c index a2be3bbca5ff..3de75eca1c92 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -641,6 +641,7 @@ const struct io_cold_def io_cold_defs[] = { }, [IORING_OP_SPLICE] = { .name = "SPLICE", + .cleanup = io_splice_cleanup, }, [IORING_OP_PROVIDE_BUFFERS] = { .name = "PROVIDE_BUFFERS", @@ -650,6 +651,7 @@ const struct io_cold_def io_cold_defs[] = { }, [IORING_OP_TEE] = { .name = "TEE", + .cleanup = io_splice_cleanup, }, [IORING_OP_SHUTDOWN] = { .name = "SHUTDOWN", diff --git a/io_uring/splice.c b/io_uring/splice.c index 3b659cd23e9d..e62bc6497a94 100644 --- a/io_uring/splice.c +++ b/io_uring/splice.c @@ -21,6 +21,7 @@ struct io_splice { u64 len; int splice_fd_in; unsigned int flags; + struct io_rsrc_node *rsrc_node; }; static int __io_splice_prep(struct io_kiocb *req, @@ -34,6 +35,7 @@ static int __io_splice_prep(struct io_kiocb *req, if (unlikely(sp->flags & ~valid_flags)) return -EINVAL; sp->splice_fd_in = READ_ONCE(sqe->splice_fd_in); + sp->rsrc_node = NULL; req->flags |= REQ_F_FORCE_ASYNC; return 0; } @@ -45,6 +47,38 @@ int io_tee_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return __io_splice_prep(req, sqe); } +void io_splice_cleanup(struct io_kiocb *req) +{ + struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); + + io_put_rsrc_node(req->ctx, sp->rsrc_node); +} + +static struct file *io_splice_get_file(struct io_kiocb *req, + unsigned int issue_flags) +{ + struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); + struct io_ring_ctx *ctx = req->ctx; + struct io_fixed_file *slot; + struct file *file = NULL; + + if (!(sp->flags & SPLICE_F_FD_IN_FIXED)) + return io_file_get_normal(req, sp->splice_fd_in); + + io_ring_submit_lock(ctx, issue_flags); + if (unlikely(sp->splice_fd_in >= ctx->nr_user_files)) + goto out; + sp->splice_fd_in = array_index_nospec(sp->splice_fd_in, ctx->nr_user_files); + slot = &ctx->file_table.files[sp->splice_fd_in]; + if (!req->rsrc_node) + __io_req_set_rsrc_node(req, ctx); + file = io_slot_file(slot); + req->flags |= REQ_F_NEED_CLEANUP; +out: + io_ring_submit_unlock(ctx, issue_flags); + return file; +} + int io_tee(struct io_kiocb *req, unsigned int issue_flags) { struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); @@ -55,10 +89,7 @@ int io_tee(struct io_kiocb *req, unsigned int issue_flags) WARN_ON_ONCE(issue_flags & IO_URING_F_NONBLOCK); - if (sp->flags & SPLICE_F_FD_IN_FIXED) - in = io_file_get_fixed(req, sp->splice_fd_in, issue_flags); - else - in = io_file_get_normal(req, sp->splice_fd_in); + in = io_splice_get_file(req, issue_flags); if (!in) { ret = -EBADF; goto done; @@ -96,10 +127,7 @@ int io_splice(struct io_kiocb *req, unsigned int issue_flags) WARN_ON_ONCE(issue_flags & IO_URING_F_NONBLOCK); - if (sp->flags & SPLICE_F_FD_IN_FIXED) - in = io_file_get_fixed(req, sp->splice_fd_in, issue_flags); - else - in = io_file_get_normal(req, sp->splice_fd_in); + in = io_splice_get_file(req, issue_flags); if (!in) { ret = -EBADF; goto done; diff --git a/io_uring/splice.h b/io_uring/splice.h index 542f94168ad3..b9b2848327fb 100644 --- a/io_uring/splice.h +++ b/io_uring/splice.h @@ -3,5 +3,6 @@ int io_tee_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_tee(struct io_kiocb *req, unsigned int issue_flags); +void io_splice_cleanup(struct io_kiocb *req); int io_splice_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_splice(struct io_kiocb *req, unsigned int issue_flags); From patchwork Mon Oct 28 14:52:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853651 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 467961DE2C4 for ; Mon, 28 Oct 2024 15:04:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127893; cv=none; b=b7k+lJumPDYbznUsZ1FzBfErZ16H4T6rWLrFfcbYEA0riR2PS2ovReYtmLPLM57QRYIwfnluzL48IcxZASJkJ5+9h1Rf3P5r9K0Hqp9wRDimozYpoM09BDDFlH5wviUmU0d9mc3/kO1/mJe0Rorl/gZXWD74wNzc2aJtOfsDjC8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127893; c=relaxed/simple; bh=/fVZOAilfDDBwA4j0A3RHIJMhEBKne+Py4wXuj7ry3I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VmBs/J4J/h2ZlcIm5ybj0ESbC6TkSHUZOYMeLxTTjkJ3yGylbyF92z1f6KKNEcGvuCCxgQfzOduq8NdKmaH9gTzxdBIUcyxNOW0PyYFwHTGqkjptZnKEkoN0QucKDC8cwj2xGGBipi9bH1XE+sJnPH0OxCOnWVNo4ZABwpeUDl0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=SIAiFtTR; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="SIAiFtTR" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3e600add5dcso1976154b6e.2 for ; Mon, 28 Oct 2024 08:04:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127889; x=1730732689; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KdiXgBsgaIl4AHuNvVMJAsYeocIhfg9+qcEY/qVH01A=; b=SIAiFtTRvHmdD9ba3Q96OY6EAsM6kNOOr9oFPsU7RZhfIDfdstQJnL8hzueNrigCtD KzsFT71AE0gGQ3493fFhc+fwILNusmnDVGI7qpTqoPG4zvuYwr1tzqMq12AW+VwxTJfE n5OPKkth0FzNAKNEXNaFfhS/yUaSYuhnoVRIzXlGh8Jvv1eorbpsJU4qMU44UXb7VEzd eCpV+Y1APUJKGEbTgXlc0gV+j9v9wZfFz3JzZkX4oWOJSf7daikU5teJbgYyAi/Ecjd3 hiQJdCrcYLgl7faPAEzSiudeXqObdYn7fSd23+G7iHWZGr9+DB8qPW3tNVbesEkqYN/9 9FdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127889; x=1730732689; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KdiXgBsgaIl4AHuNvVMJAsYeocIhfg9+qcEY/qVH01A=; b=KmPVUyF6P1qvAYZSCBX45bZ+pO91bKK8Z2IHDH1ifVGPIn6VnPqEtp2CaNLfKIM0TD ieZBbYMG1jpGfYhhGEEiMd4vLMv8GDHT/nRjIjzBpp6qnRHpvi/7O5EBm4u5Yvb4wEp3 jTSPBA0ADvXbsSmE+YtJnSTiBwXj6ZzcjF6+n3SphsphaNcMDLtgveen79uMhINMK8dW hLomHlg2TLYj0v40sZCGe9M2RFhIM8RQKqZzEgGHsM8d3Eac8i9VtjGO29ubwtyPKFWP OklGJpRGV7uMagj+ECVmTk3Ymp5Fu9AlFAjGC/vSCXQ0s3UvaabHNqGphtnLWf5VqFuz N9tw== X-Gm-Message-State: AOJu0YxT/XKIrDaldxPuiyY+q3R4RGL/0/aFt5LwImbpikiMKfPoAM5n HV9VUC/NpK8SHLLg5dNZhI0DjCNxqsRF9oFTzlkTFhJi3PFfaZQaCxUdR5026UvMeeb/dT01+on k X-Google-Smtp-Source: AGHT+IHDuek0JyhGxEoFTTdBKhuSsJzzkJ3KCVWX1+6YQUMd8Ea9vTNp2KUojk9ZRjNgHLFrO7W+TA== X-Received: by 2002:a05:6808:182a:b0:3e6:2d97:ca67 with SMTP id 5614622812f47-3e6384ce4fcmr6009617b6e.44.1730127889154; Mon, 28 Oct 2024 08:04:49 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:48 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 05/13] io_uring/rsrc: kill io_charge_rsrc_node() Date: Mon, 28 Oct 2024 08:52:35 -0600 Message-ID: <20241028150437.387667-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's only used from __io_req_set_rsrc_node(), and it takes both the ctx and node itself, while never using the ctx. Just open-code the basic refs++ in __io_req_set_rsrc_node() instead. Signed-off-by: Jens Axboe --- io_uring/rsrc.h | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index e072fb3ee351..1589c9740083 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -97,18 +97,12 @@ static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node io_rsrc_node_ref_zero(node); } -static inline void io_charge_rsrc_node(struct io_ring_ctx *ctx, - struct io_rsrc_node *node) -{ - node->refs++; -} - static inline void __io_req_set_rsrc_node(struct io_kiocb *req, struct io_ring_ctx *ctx) { lockdep_assert_held(&ctx->uring_lock); req->rsrc_node = ctx->rsrc_node; - io_charge_rsrc_node(ctx, ctx->rsrc_node); + ctx->rsrc_node->refs++; } static inline void io_req_set_rsrc_node(struct io_kiocb *req, From patchwork Mon Oct 28 14:52:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853655 Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D8B01DE2C6 for ; Mon, 28 Oct 2024 15:04:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127897; cv=none; b=YlTdazmoB02zQqDn7HtHSH75e+jTzARotVAmQD1kWTXpcxRwUfycC7B4xnRjdiFkCnMppXZE+YOw/IlldIBfN27Abots+B4uFqijr1MQIMqDThk65DL3mcgHyiemdfXQvdlQIt9myjkKf+6rwf8MU8DmiKQRGFCXDxTU3yWZhcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127897; c=relaxed/simple; bh=PKSiK9vZxzhf5QMKXcBUZRvZ7oYrRilxGgNUXSkKQtI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ODnixPfU2fwYmMxkeBfS/qUj9EG1295ZNGKQFNpLseKZyiNUxuljeTybkDp1NAH93yUT5tjySNvG8DKnrHaPj4sAL83c6RkCQaJ7M8LGjip0EiM93CtXFJeE+6kNHiaPzyoouuWSanlUAdux6XfGeB37YLoxGCfVFkibtECyvXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=hRf96Mdo; arc=none smtp.client-ip=209.85.166.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="hRf96Mdo" Received: by mail-il1-f181.google.com with SMTP id e9e14a558f8ab-3a4e58e3f9bso11186365ab.1 for ; Mon, 28 Oct 2024 08:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127891; x=1730732691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8/+Qt93/kosL+hc1ElAFP2eVvQr4EvT1v9SHBA/KA+I=; b=hRf96MdodU+6ZMupX4eRhu3+ROWp8iKJgR0fObk3FIeO9JQpcQHAFMSaT/WzjzNhrl O+lfqIaIOg2hjW80DQqytKOCS8MsSXbzDGykP5J/XEaZ4g/5lWSshKYqMY8hwx+biD+9 MSkDJi61VCOusOA3ik8xszvVvHvXWLNpFL3H7N44VhQCONPvVH/TvegdlD/kuzqZI8ZT ABRPfRk8fHD7pERMjph1adhTgSGtG6opUmyPwbScEEw54b27XYUz+4iLJSV7+e1Oo2dR gasNA8ejKTYxuLMc0gyvbULKmymGS/9G2zu/d1N360KwSjbmpA/Py3+8FSDF3qg7fZoX Hs/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127891; x=1730732691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8/+Qt93/kosL+hc1ElAFP2eVvQr4EvT1v9SHBA/KA+I=; b=rSwFXKw+Z30hERIhZqzasWoOsFkoY9Dp/NUcVR+uq+aiuDm8Eb3L/Pi/uak9eXyDVt ghyxrgdUzLB/EmLSIxJMKrFYMQlphbiEUh+xfsCuYRQhuQH94c06NkCTjshgoJo1OHOQ Rrqe9zIxu4nPs6Z/hjOpghGrWpqP9qbWTce63iMPuYREosqu+0VCSdUnzrq+gnQECgI2 WQ4S8hzC0u99F9rNTI1g2uhw/6FltvkBwtv2XTGHOs3SombZDiXKe+Sb6wBKq74eysHt /psik1OJfxgEH/6W0iYKwaMQQBrJR7h53Gm5TFnHhwm8g8IpHvooLZty8tRFy+7dGxCx RWYA== X-Gm-Message-State: AOJu0YxRnQ/bouS9euO0h2/JGiueSlJViWA+WhxEf0DuQpj2r74Q+mqY Vr5+urWRUo/uh8I7fKJMhpoG72vFH7pQ/i4GkOj9GbrdzHMCr1xhqnGYdtx2ykpiK8sQSxytKYB j X-Google-Smtp-Source: AGHT+IFe9y9sZ/86SmWA6WITWEBC8nx0BuEX0sKCrJU+ewngoOF07gKSWQ0qlLSzZ4GHzuNzMBd9Yg== X-Received: by 2002:a05:6e02:1c26:b0:3a3:4175:79d2 with SMTP id e9e14a558f8ab-3a4ed2badbbmr79454925ab.14.1730127890240; Mon, 28 Oct 2024 08:04:50 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:49 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 06/13] io_uring/rsrc: get rid of per-ring io_rsrc_node list Date: Mon, 28 Oct 2024 08:52:36 -0600 Message-ID: <20241028150437.387667-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Work in progress, but get rid of the per-ring serialization of resource nodes, like registered buffers and files. Main issue here is that one node can otherwise hold up a bunch of other nodes from getting freed, which is especially a problem for file resource nodes and networked workloads where some descriptors may not see activity in a long time. As an example, instantiate an io_uring ring fd and create a sparse registered file table. Even 2 will do. Then create a socket and register it as fixed file 0, F0. The number of open files in the app is now 5, with 0/1/2 being the usual stdin/out/err, 3 being the ring fd, and 4 being the socket. Register this socket (eg "the listener") in slot 0 of the registered file table. Now add an operation on the socket that uses slot 0. Finally, loop N times, where each loop creates a new socket, registers said socket as a file, then unregisters the socket, and finally closes the socket. This is roughly similar to what a basic accept loop would look like. At the end of this loop, it's not unreasonable to expect that there would still be 5 open files. Each socket created and registered in the loop is also unregistered and closed. But since the listener socket registered first still has references to its resource node due to still being active, each subsequent socket unregistration is stuck behind it for reclaim. Hence 5 + N files are still open at that point, where N is awaiting the final put held up by the listener socket. Rewrite the io_rsrc_node handling to NOT rely on serialization. Struct io_kiocb now gets explicit resource nodes assigned, with each holding a reference to the parent node. A parent node is either of type FILE or BUFFER, which are the two types of nodes that exist. A request can have two nodes assigned, if it's using both registered files and buffers. Since request issue and task_work completion is both under the ring private lock, no atomics are needed to handle these references. It's a simple unlocked inc/dec. As before, the registered buffer or file table each hold a reference as well to the registered nodes. Final put of the node will remove the node and free the underlying resource, eg unmap the buffer or put the file. Outside of removing the stall in resource reclaim described above, it has the following advantages: 1) It's a lot simpler than the previous scheme, and easier to follow. No need to specific quiesce handling anymore. 2) There are no resource node allocations in the fast path, all of that happens at resource registration time. 3) The structs related to resource handling can all get simplified quite a bit, like io_rsrc_node and io_rsrc_data. io_rsrc_put can go away completely. 4) Handling of resource tags is much simpler, and doesn't require persistent storage as it can simply get assigned up front at registration time. Just copy them in one-by-one at registration time and assign to the resource node. The only real downside is that a request is now explicitly limited to pinning 2 resources, one file and one buffer, where before just assigning a resource node to a request would pin all of them. The upside is that it's easier to follow now, as an individual resource is explicitly referenced and assigned to the request. With this in place, the above mentioned example will be using exactly 5 files at the end of the loop, not N. Needs to get broken up a bit and there are certainly rough edges, but that's why it's a work in progress... But it does remove a ton more code than it adds, and passes the liburing tests. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 10 +- io_uring/fdinfo.c | 2 +- io_uring/filetable.c | 52 ++-- io_uring/filetable.h | 25 +- io_uring/io_uring.c | 38 +-- io_uring/net.c | 11 +- io_uring/nop.c | 6 +- io_uring/notif.c | 3 +- io_uring/rsrc.c | 488 +++++++++++++-------------------- io_uring/rsrc.h | 75 ++--- io_uring/rw.c | 8 +- io_uring/splice.c | 16 +- io_uring/uring_cmd.c | 12 +- 13 files changed, 280 insertions(+), 466 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index d4ba4ae480d6..42c5f2c992c4 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -56,7 +56,7 @@ struct io_wq_work { }; struct io_file_table { - struct io_fixed_file *files; + struct io_rsrc_node **nodes; unsigned long *bitmap; unsigned int alloc_hint; }; @@ -264,7 +264,6 @@ struct io_ring_ctx { * Fixed resources fast path, should be accessed only under * uring_lock, and updated through io_uring_register(2) */ - struct io_rsrc_node *rsrc_node; atomic_t cancel_seq; /* @@ -277,7 +276,7 @@ struct io_ring_ctx { struct io_wq_work_list iopoll_list; struct io_file_table file_table; - struct io_mapped_ubuf **user_bufs; + struct io_rsrc_node **user_bufs; unsigned nr_user_files; unsigned nr_user_bufs; @@ -372,10 +371,7 @@ struct io_ring_ctx { struct io_rsrc_data *buf_data; /* protected by ->uring_lock */ - struct list_head rsrc_ref_list; struct io_alloc_cache rsrc_node_cache; - struct wait_queue_head rsrc_quiesce_wq; - unsigned rsrc_quiesce; u32 pers_next; struct xarray personalities; @@ -642,7 +638,7 @@ struct io_kiocb { __poll_t apoll_events; }; - struct io_rsrc_node *rsrc_node; + struct io_rsrc_node *rsrc_nodes[2]; atomic_t refs; bool cancel_seq_set; diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index a6bac533edbe..064a79475c5f 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -176,7 +176,7 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file) } seq_printf(m, "UserBufs:\t%u\n", ctx->nr_user_bufs); for (i = 0; has_lock && i < ctx->nr_user_bufs; i++) { - struct io_mapped_ubuf *buf = ctx->user_bufs[i]; + struct io_mapped_ubuf *buf = ctx->user_bufs[i]->buf; seq_printf(m, "%5u: 0x%llx/%u\n", i, buf->ubuf, buf->len); } diff --git a/io_uring/filetable.c b/io_uring/filetable.c index 997c56d32ee6..c18e37b495a1 100644 --- a/io_uring/filetable.c +++ b/io_uring/filetable.c @@ -38,14 +38,14 @@ static int io_file_bitmap_get(struct io_ring_ctx *ctx) bool io_alloc_file_tables(struct io_file_table *table, unsigned nr_files) { - table->files = kvcalloc(nr_files, sizeof(table->files[0]), - GFP_KERNEL_ACCOUNT); - if (unlikely(!table->files)) + table->nodes = kvmalloc_array(nr_files, sizeof(struct io_src_node *), + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (unlikely(!table->nodes)) return false; table->bitmap = bitmap_zalloc(nr_files, GFP_KERNEL_ACCOUNT); if (unlikely(!table->bitmap)) { - kvfree(table->files); + kvfree(table->nodes); return false; } @@ -54,9 +54,9 @@ bool io_alloc_file_tables(struct io_file_table *table, unsigned nr_files) void io_free_file_tables(struct io_file_table *table) { - kvfree(table->files); + kvfree(table->nodes); bitmap_free(table->bitmap); - table->files = NULL; + table->nodes = NULL; table->bitmap = NULL; } @@ -64,8 +64,7 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, u32 slot_index) __must_hold(&req->ctx->uring_lock) { - struct io_fixed_file *file_slot; - int ret; + struct io_rsrc_node *node; if (io_is_uring_fops(file)) return -EBADF; @@ -74,22 +73,18 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, if (slot_index >= ctx->nr_user_files) return -EINVAL; - slot_index = array_index_nospec(slot_index, ctx->nr_user_files); - file_slot = io_fixed_file_slot(&ctx->file_table, slot_index); - - if (file_slot->file_ptr) { - ret = io_queue_rsrc_removal(ctx->file_data, slot_index, - io_slot_file(file_slot)); - if (ret) - return ret; + node = io_rsrc_node_alloc(ctx, ctx->file_data, slot_index, IORING_RSRC_FILE); + if (IS_ERR(node)) + return -ENOMEM; - file_slot->file_ptr = 0; - } else { + slot_index = array_index_nospec(slot_index, ctx->nr_user_files); + if (ctx->file_table.nodes[slot_index]) + io_put_rsrc_node(ctx->file_table.nodes[slot_index]); + else io_file_bitmap_set(&ctx->file_table, slot_index); - } - *io_get_tag_slot(ctx->file_data, slot_index) = 0; - io_fixed_file_set(file_slot, file); + ctx->file_table.nodes[slot_index] = node; + io_fixed_file_set(node, file); return 0; } @@ -134,25 +129,16 @@ int io_fixed_fd_install(struct io_kiocb *req, unsigned int issue_flags, int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset) { - struct io_fixed_file *file_slot; - int ret; - if (unlikely(!ctx->file_data)) return -ENXIO; if (offset >= ctx->nr_user_files) return -EINVAL; offset = array_index_nospec(offset, ctx->nr_user_files); - file_slot = io_fixed_file_slot(&ctx->file_table, offset); - if (!file_slot->file_ptr) + if (!ctx->file_table.nodes[offset]) return -EBADF; - - ret = io_queue_rsrc_removal(ctx->file_data, offset, - io_slot_file(file_slot)); - if (ret) - return ret; - - file_slot->file_ptr = 0; + io_put_rsrc_node(ctx->file_table.nodes[offset]); + ctx->file_table.nodes[offset] = NULL; io_file_bitmap_clear(&ctx->file_table, offset); return 0; } diff --git a/io_uring/filetable.h b/io_uring/filetable.h index c027ed4ad68d..47616079abaa 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -34,36 +34,35 @@ static inline void io_file_bitmap_set(struct io_file_table *table, int bit) table->alloc_hint = bit + 1; } -static inline struct io_fixed_file * -io_fixed_file_slot(struct io_file_table *table, unsigned i) -{ - return &table->files[i]; -} - #define FFS_NOWAIT 0x1UL #define FFS_ISREG 0x2UL #define FFS_MASK ~(FFS_NOWAIT|FFS_ISREG) -static inline unsigned int io_slot_flags(struct io_fixed_file *slot) +static inline unsigned int io_slot_flags(struct io_rsrc_node *node) { - return (slot->file_ptr & ~FFS_MASK) << REQ_F_SUPPORT_NOWAIT_BIT; + + return (node->file_ptr & ~FFS_MASK) << REQ_F_SUPPORT_NOWAIT_BIT; } -static inline struct file *io_slot_file(struct io_fixed_file *slot) +static inline struct file *io_slot_file(struct io_rsrc_node *node) { - return (struct file *)(slot->file_ptr & FFS_MASK); + return (struct file *)(node->file_ptr & FFS_MASK); } static inline struct file *io_file_from_index(struct io_file_table *table, int index) { - return io_slot_file(io_fixed_file_slot(table, index)); + struct io_rsrc_node *node = table->nodes[index]; + + if (node) + return io_slot_file(node); + return NULL; } -static inline void io_fixed_file_set(struct io_fixed_file *file_slot, +static inline void io_fixed_file_set(struct io_rsrc_node *node, struct file *file) { - file_slot->file_ptr = (unsigned long)file | + node->file_ptr = (unsigned long)file | (io_file_get_flags(file) >> REQ_F_SUPPORT_NOWAIT_BIT); } diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a09c67b38c1b..0876aa74c739 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -333,7 +333,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) mutex_init(&ctx->uring_lock); init_waitqueue_head(&ctx->cq_wait); init_waitqueue_head(&ctx->poll_wq); - init_waitqueue_head(&ctx->rsrc_quiesce_wq); spin_lock_init(&ctx->completion_lock); spin_lock_init(&ctx->timeout_lock); INIT_WQ_LIST(&ctx->iopoll_list); @@ -341,7 +340,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->defer_list); INIT_LIST_HEAD(&ctx->timeout_list); INIT_LIST_HEAD(&ctx->ltimeout_list); - INIT_LIST_HEAD(&ctx->rsrc_ref_list); init_llist_head(&ctx->work_llist); INIT_LIST_HEAD(&ctx->tctx_list); ctx->submit_state.free_list.next = NULL; @@ -1415,7 +1413,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx, io_clean_op(req); } io_put_file(req); - io_put_rsrc_node(ctx, req->rsrc_node); + io_req_put_rsrc_nodes(req); io_put_task(req->task); node = req->comp_list.next; @@ -1878,7 +1876,7 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd, unsigned int issue_flags) { struct io_ring_ctx *ctx = req->ctx; - struct io_fixed_file *slot; + struct io_rsrc_node *node; struct file *file = NULL; io_ring_submit_lock(ctx, issue_flags); @@ -1886,11 +1884,12 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd, if (unlikely((unsigned int)fd >= ctx->nr_user_files)) goto out; fd = array_index_nospec(fd, ctx->nr_user_files); - slot = io_fixed_file_slot(&ctx->file_table, fd); - if (!req->rsrc_node) - __io_req_set_rsrc_node(req, ctx); - req->flags |= io_slot_flags(slot); - file = io_slot_file(slot); + node = ctx->file_table.nodes[fd]; + if (node) { + io_req_assign_rsrc_node(req, node); + req->flags |= io_slot_flags(node); + file = io_slot_file(node); + } out: io_ring_submit_unlock(ctx, issue_flags); return file; @@ -2036,7 +2035,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->flags = (__force io_req_flags_t) sqe_flags; req->cqe.user_data = READ_ONCE(sqe->user_data); req->file = NULL; - req->rsrc_node = NULL; + req->rsrc_nodes[IORING_RSRC_FILE] = NULL; + req->rsrc_nodes[IORING_RSRC_BUFFER] = NULL; req->task = current; req->cancel_seq_set = false; @@ -2718,15 +2718,10 @@ static void io_req_caches_free(struct io_ring_ctx *ctx) static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) { io_sq_thread_finish(ctx); - /* __io_rsrc_put_work() may need uring_lock to progress, wait w/o it */ - if (WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list))) - return; mutex_lock(&ctx->uring_lock); - if (ctx->buf_data) - __io_sqe_buffers_unregister(ctx); - if (ctx->file_data) - __io_sqe_files_unregister(ctx); + io_sqe_buffers_unregister(ctx); + io_sqe_files_unregister(ctx); io_cqring_overflow_kill(ctx); io_eventfd_unregister(ctx); io_alloc_cache_free(&ctx->apoll_cache, kfree); @@ -2743,11 +2738,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) if (ctx->submitter_task) put_task_struct(ctx->submitter_task); - /* there are no registered resources left, nobody uses it */ - if (ctx->rsrc_node) - io_rsrc_node_destroy(ctx, ctx->rsrc_node); - - WARN_ON_ONCE(!list_empty(&ctx->rsrc_ref_list)); WARN_ON_ONCE(!list_empty(&ctx->ltimeout_list)); io_alloc_cache_free(&ctx->rsrc_node_cache, kfree); @@ -3729,10 +3719,6 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, if (ret) goto err; - ret = io_rsrc_init(ctx); - if (ret) - goto err; - p->features = IORING_FEAT_SINGLE_MMAP | IORING_FEAT_NODROP | IORING_FEAT_SUBMIT_STABLE | IORING_FEAT_RW_CUR_POS | IORING_FEAT_CUR_PERSONALITY | IORING_FEAT_FAST_POLL | diff --git a/io_uring/net.c b/io_uring/net.c index 2040195e33ab..ce1156551d10 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1342,15 +1342,15 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) if (sr->flags & IORING_RECVSEND_FIXED_BUF) { struct io_ring_ctx *ctx = req->ctx; - struct io_mapped_ubuf *imu; + struct io_rsrc_node *node; int idx; ret = -EFAULT; io_ring_submit_lock(ctx, issue_flags); if (sr->buf_index < ctx->nr_user_bufs) { idx = array_index_nospec(sr->buf_index, ctx->nr_user_bufs); - imu = READ_ONCE(ctx->user_bufs[idx]); - io_req_set_rsrc_node(sr->notif, ctx); + node = ctx->user_bufs[idx]; + io_req_assign_rsrc_node(sr->notif, node); ret = 0; } io_ring_submit_unlock(ctx, issue_flags); @@ -1358,8 +1358,9 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret)) return ret; - ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, imu, - (u64)(uintptr_t)sr->buf, sr->len); + ret = io_import_fixed(ITER_SOURCE, &kmsg->msg.msg_iter, + node->buf, (u64)(uintptr_t)sr->buf, + sr->len); if (unlikely(ret)) return ret; kmsg->msg.sg_from_iter = io_sg_from_iter; diff --git a/io_uring/nop.c b/io_uring/nop.c index 2c7a22ba4053..de91600a3bc6 100644 --- a/io_uring/nop.c +++ b/io_uring/nop.c @@ -61,15 +61,15 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags) } if (nop->flags & IORING_NOP_FIXED_BUFFER) { struct io_ring_ctx *ctx = req->ctx; - struct io_mapped_ubuf *imu; + struct io_rsrc_node *node; int idx; ret = -EFAULT; io_ring_submit_lock(ctx, issue_flags); if (nop->buffer < ctx->nr_user_bufs) { idx = array_index_nospec(nop->buffer, ctx->nr_user_bufs); - imu = READ_ONCE(ctx->user_bufs[idx]); - io_req_set_rsrc_node(req, ctx); + node = READ_ONCE(ctx->user_bufs[idx]); + io_req_assign_rsrc_node(req, node); ret = 0; } io_ring_submit_unlock(ctx, issue_flags); diff --git a/io_uring/notif.c b/io_uring/notif.c index 28859ae3ee6e..4f02e969cf08 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -117,7 +117,8 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->file = NULL; notif->task = current; io_get_task_refs(1); - notif->rsrc_node = NULL; + notif->rsrc_nodes[IORING_RSRC_FILE] = NULL; + notif->rsrc_nodes[IORING_RSRC_BUFFER] = NULL; nd = io_notif_to_data(notif); nd->zc_report = false; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index ca2ec8a018be..cc93d0c2df4c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -26,10 +26,8 @@ struct io_rsrc_update { u32 offset; }; -static void io_rsrc_buf_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc); -static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, - struct io_mapped_ubuf **pimu, - struct page **last_hpage); +static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, + struct iovec *iov, int index, struct page **last_hpage); /* only define max */ #define IORING_MAX_FIXED_FILES (1U << 20) @@ -110,13 +108,13 @@ static int io_buffer_validate(struct iovec *iov) return 0; } -static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slot) +static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_rsrc_node *node) { - struct io_mapped_ubuf *imu = *slot; unsigned int i; - *slot = NULL; - if (imu != &dummy_ubuf) { + if (node->buf != &dummy_ubuf) { + struct io_mapped_ubuf *imu = node->buf; + if (!refcount_dec_and_test(&imu->refs)) return; for (i = 0; i < imu->nr_bvecs; i++) @@ -125,207 +123,62 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo io_unaccount_mem(ctx, imu->acct_pages); kvfree(imu); } + node->rsrc = NULL; } -static void io_rsrc_put_work(struct io_rsrc_node *node) -{ - struct io_rsrc_put *prsrc = &node->item; - - if (prsrc->tag) - io_post_aux_cqe(node->ctx, prsrc->tag, 0, 0); - - switch (node->type) { - case IORING_RSRC_FILE: - fput(prsrc->file); - break; - case IORING_RSRC_BUFFER: - io_rsrc_buf_put(node->ctx, prsrc); - break; - default: - WARN_ON_ONCE(1); - break; - } -} - -void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *node) -{ - if (!io_alloc_cache_put(&ctx->rsrc_node_cache, node)) - kfree(node); -} - -void io_rsrc_node_ref_zero(struct io_rsrc_node *node) - __must_hold(&node->ctx->uring_lock) -{ - struct io_ring_ctx *ctx = node->ctx; - - while (!list_empty(&ctx->rsrc_ref_list)) { - node = list_first_entry(&ctx->rsrc_ref_list, - struct io_rsrc_node, node); - /* recycle ref nodes in order */ - if (node->refs) - break; - list_del(&node->node); - - if (likely(!node->empty)) - io_rsrc_put_work(node); - io_rsrc_node_destroy(ctx, node); - } - if (list_empty(&ctx->rsrc_ref_list) && unlikely(ctx->rsrc_quiesce)) - wake_up_all(&ctx->rsrc_quiesce_wq); -} - -struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx) +struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, + struct io_rsrc_data *data, int index, + int type) { - struct io_rsrc_node *ref_node; + struct io_rsrc_node *node; - ref_node = io_alloc_cache_get(&ctx->rsrc_node_cache); - if (!ref_node) { - ref_node = kzalloc(sizeof(*ref_node), GFP_KERNEL); - if (!ref_node) + node = io_alloc_cache_get(&ctx->rsrc_node_cache); + if (!node) { + node = kzalloc(sizeof(*node), GFP_KERNEL); + if (!node) return NULL; } - ref_node->ctx = ctx; - ref_node->empty = 0; - ref_node->refs = 1; - return ref_node; -} - -__cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data, - struct io_ring_ctx *ctx) -{ - struct io_rsrc_node *backup; - DEFINE_WAIT(we); - int ret; - - /* As We may drop ->uring_lock, other task may have started quiesce */ - if (data->quiesce) - return -ENXIO; - - backup = io_rsrc_node_alloc(ctx); - if (!backup) - return -ENOMEM; - ctx->rsrc_node->empty = true; - ctx->rsrc_node->type = -1; - list_add_tail(&ctx->rsrc_node->node, &ctx->rsrc_ref_list); - io_put_rsrc_node(ctx, ctx->rsrc_node); - ctx->rsrc_node = backup; - - if (list_empty(&ctx->rsrc_ref_list)) - return 0; - - if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - atomic_set(&ctx->cq_wait_nr, 1); - smp_mb(); - } - - ctx->rsrc_quiesce++; - data->quiesce = true; - do { - prepare_to_wait(&ctx->rsrc_quiesce_wq, &we, TASK_INTERRUPTIBLE); - mutex_unlock(&ctx->uring_lock); - - ret = io_run_task_work_sig(ctx); - if (ret < 0) { - finish_wait(&ctx->rsrc_quiesce_wq, &we); - mutex_lock(&ctx->uring_lock); - if (list_empty(&ctx->rsrc_ref_list)) - ret = 0; - break; - } - - schedule(); - mutex_lock(&ctx->uring_lock); - ret = 0; - } while (!list_empty(&ctx->rsrc_ref_list)); - - finish_wait(&ctx->rsrc_quiesce_wq, &we); - data->quiesce = false; - ctx->rsrc_quiesce--; - - if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - atomic_set(&ctx->cq_wait_nr, 0); - smp_mb(); - } - return ret; -} - -static void io_free_page_table(void **table, size_t size) -{ - unsigned i, nr_tables = DIV_ROUND_UP(size, PAGE_SIZE); - - for (i = 0; i < nr_tables; i++) - kfree(table[i]); - kfree(table); + node->ctx = ctx; + node->refs = 1; + node->type = type; + node->tag = 0; + node->rsrc = NULL; + return node; } static void io_rsrc_data_free(struct io_rsrc_data *data) { - size_t size = data->nr * sizeof(data->tags[0][0]); - - if (data->tags) - io_free_page_table((void **)data->tags, size); - kfree(data); -} - -static __cold void **io_alloc_page_table(size_t size) -{ - unsigned i, nr_tables = DIV_ROUND_UP(size, PAGE_SIZE); - size_t init_size = size; - void **table; - - table = kcalloc(nr_tables, sizeof(*table), GFP_KERNEL_ACCOUNT); - if (!table) - return NULL; + int i; - for (i = 0; i < nr_tables; i++) { - unsigned int this_size = min_t(size_t, size, PAGE_SIZE); + for (i = 0; i < data->nr; i++) { + struct io_rsrc_node *node = data->nodes[i]; - table[i] = kzalloc(this_size, GFP_KERNEL_ACCOUNT); - if (!table[i]) { - io_free_page_table(table, init_size); - return NULL; - } - size -= this_size; + io_put_rsrc_node(node); } - return table; + kvfree(data->nodes); + kfree(data); } -__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, int type, - u64 __user *utags, - unsigned nr, struct io_rsrc_data **pdata) +__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, unsigned nr, + struct io_rsrc_data **pdata) { struct io_rsrc_data *data; - int ret = 0; - unsigned i; data = kzalloc(sizeof(*data), GFP_KERNEL); if (!data) return -ENOMEM; - data->tags = (u64 **)io_alloc_page_table(nr * sizeof(data->tags[0][0])); - if (!data->tags) { - kfree(data); + + data->nodes = kvmalloc_array(nr, sizeof(struct io_rsrc_node *), + GFP_KERNEL | __GFP_ZERO); + if (!data->nodes) { + io_rsrc_data_free(data); return -ENOMEM; } data->nr = nr; - data->ctx = ctx; - data->rsrc_type = type; - if (utags) { - ret = -EFAULT; - for (i = 0; i < nr; i++) { - u64 *tag_slot = io_get_tag_slot(data, i); - - if (copy_from_user(tag_slot, &utags[i], - sizeof(*tag_slot))) - goto fail; - } - } *pdata = data; return 0; -fail: - io_rsrc_data_free(data); - return ret; } static int __io_sqe_files_update(struct io_ring_ctx *ctx, @@ -334,8 +187,6 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, { u64 __user *tags = u64_to_user_ptr(up->tags); __s32 __user *fds = u64_to_user_ptr(up->data); - struct io_rsrc_data *data = ctx->file_data; - struct io_fixed_file *file_slot; int fd, i, err = 0; unsigned int done; @@ -360,18 +211,14 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, continue; i = array_index_nospec(up->offset + done, ctx->nr_user_files); - file_slot = io_fixed_file_slot(&ctx->file_table, i); - - if (file_slot->file_ptr) { - err = io_queue_rsrc_removal(data, i, - io_slot_file(file_slot)); - if (err) - break; - file_slot->file_ptr = 0; + if (ctx->file_table.nodes[i]) { + io_put_rsrc_node(ctx->file_table.nodes[i]); + ctx->file_table.nodes[i] = NULL; io_file_bitmap_clear(&ctx->file_table, i); } if (fd != -1) { struct file *file = fget(fd); + struct io_rsrc_node *node; if (!file) { err = -EBADF; @@ -385,8 +232,15 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, err = -EBADF; break; } - *io_get_tag_slot(data, i) = tag; - io_fixed_file_set(file_slot, file); + node = io_rsrc_node_alloc(ctx, ctx->file_data, i, IORING_RSRC_FILE); + if (!node) { + err = -ENOMEM; + fput(file); + break; + } + ctx->file_table.nodes[i] = node; + node->tag = tag; + io_fixed_file_set(node, file); io_file_bitmap_set(&ctx->file_table, i); } } @@ -411,7 +265,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, return -EINVAL; for (done = 0; done < nr_args; done++) { - struct io_mapped_ubuf *imu; + struct io_rsrc_node *node; u64 tag = 0; uvec = u64_to_user_ptr(user_data); @@ -431,23 +285,16 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, err = -EINVAL; break; } - err = io_sqe_buffer_register(ctx, iov, &imu, &last_hpage); - if (err) - break; - i = array_index_nospec(up->offset + done, ctx->nr_user_bufs); - if (ctx->user_bufs[i] != &dummy_ubuf) { - err = io_queue_rsrc_removal(ctx->buf_data, i, - ctx->user_bufs[i]); - if (unlikely(err)) { - io_buffer_unmap(ctx, &imu); - break; - } - ctx->user_bufs[i] = (struct io_mapped_ubuf *)&dummy_ubuf; + node = io_sqe_buffer_register(ctx, iov, i, &last_hpage); + if (IS_ERR(node)) { + err = PTR_ERR(node); + break; } + io_put_rsrc_node(ctx->user_bufs[i]); - ctx->user_bufs[i] = imu; - *io_get_tag_slot(ctx->buf_data, i) = tag; + ctx->user_bufs[i] = node; + node->tag = tag; if (ctx->compat) user_data += sizeof(struct compat_iovec); else @@ -622,38 +469,47 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags) return IOU_OK; } -int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc) +void io_free_rsrc_node(struct io_rsrc_node *node) { - struct io_ring_ctx *ctx = data->ctx; - struct io_rsrc_node *node = ctx->rsrc_node; - u64 *tag_slot = io_get_tag_slot(data, idx); + struct io_ring_ctx *ctx = node->ctx; - ctx->rsrc_node = io_rsrc_node_alloc(ctx); - if (unlikely(!ctx->rsrc_node)) { - ctx->rsrc_node = node; - return -ENOMEM; + lockdep_assert_held(&ctx->uring_lock); + + if (node->tag) + io_post_aux_cqe(node->ctx, node->tag, 0, 0); + + switch (node->type) { + case IORING_RSRC_FILE: + if (io_slot_file(node)) + fput(io_slot_file(node)); + break; + case IORING_RSRC_BUFFER: + if (node->buf) + io_buffer_unmap(node->ctx, node); + break; + default: + WARN_ON_ONCE(1); + break; } - node->item.rsrc = rsrc; - node->type = data->rsrc_type; - node->item.tag = *tag_slot; - *tag_slot = 0; - list_add_tail(&node->node, &ctx->rsrc_ref_list); - io_put_rsrc_node(ctx, node); - return 0; + if (!io_alloc_cache_put(&ctx->rsrc_node_cache, node)) + kfree(node); } -void __io_sqe_files_unregister(struct io_ring_ctx *ctx) +static void __io_sqe_files_unregister(struct io_ring_ctx *ctx) { int i; + lockdep_assert_held(&ctx->uring_lock); + for (i = 0; i < ctx->nr_user_files; i++) { - struct file *file = io_file_from_index(&ctx->file_table, i); + struct io_rsrc_node *node = ctx->file_table.nodes[i]; - if (!file) - continue; - io_file_bitmap_clear(&ctx->file_table, i); - fput(file); + if (node) { + io_put_rsrc_node(node); + io_file_bitmap_clear(&ctx->file_table, i); + ctx->file_table.nodes[i] = NULL; + } } io_free_file_tables(&ctx->file_table); @@ -665,22 +521,11 @@ void __io_sqe_files_unregister(struct io_ring_ctx *ctx) int io_sqe_files_unregister(struct io_ring_ctx *ctx) { - unsigned nr = ctx->nr_user_files; - int ret; - if (!ctx->file_data) return -ENXIO; - /* - * Quiesce may unlock ->uring_lock, and while it's not held - * prevent new requests using the table. - */ - ctx->nr_user_files = 0; - ret = io_rsrc_ref_quiesce(ctx->file_data, ctx); - ctx->nr_user_files = nr; - if (!ret) - __io_sqe_files_unregister(ctx); - return ret; + __io_sqe_files_unregister(ctx); + return 0; } int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, @@ -699,8 +544,7 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, return -EMFILE; if (nr_args > rlimit(RLIMIT_NOFILE)) return -EMFILE; - ret = io_rsrc_data_alloc(ctx, IORING_RSRC_FILE, tags, nr_args, - &ctx->file_data); + ret = io_rsrc_data_alloc(ctx, nr_args, &ctx->file_data); if (ret) return ret; @@ -711,16 +555,18 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, } for (i = 0; i < nr_args; i++, ctx->nr_user_files++) { - struct io_fixed_file *file_slot; + struct io_rsrc_node *node; + u64 tag = 0; - if (fds && copy_from_user(&fd, &fds[i], sizeof(fd))) { - ret = -EFAULT; + ret = -EFAULT; + if (tags && copy_from_user(&tag, &tags[i], sizeof(tag))) + goto fail; + if (fds && copy_from_user(&fd, &fds[i], sizeof(fd))) goto fail; - } /* allow sparse sets */ if (!fds || fd == -1) { ret = -EINVAL; - if (unlikely(*io_get_tag_slot(ctx->file_data, i))) + if (tag) goto fail; continue; } @@ -737,8 +583,16 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, fput(file); goto fail; } - file_slot = io_fixed_file_slot(&ctx->file_table, i); - io_fixed_file_set(file_slot, file); + ret = -ENOMEM; + node = io_rsrc_node_alloc(ctx, ctx->file_data, i, IORING_RSRC_FILE); + if (!node) { + fput(file); + goto fail; + } + if (tag) + node->tag = tag; + ctx->file_table.nodes[i] = node; + io_fixed_file_set(node, file); io_file_bitmap_set(&ctx->file_table, i); } @@ -750,43 +604,30 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, return ret; } -static void io_rsrc_buf_put(struct io_ring_ctx *ctx, struct io_rsrc_put *prsrc) -{ - io_buffer_unmap(ctx, &prsrc->buf); - prsrc->buf = NULL; -} - -void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx) +static void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx) { unsigned int i; - for (i = 0; i < ctx->nr_user_bufs; i++) - io_buffer_unmap(ctx, &ctx->user_bufs[i]); - kfree(ctx->user_bufs); - io_rsrc_data_free(ctx->buf_data); + lockdep_assert_held(&ctx->uring_lock); + + for (i = 0; i < ctx->nr_user_bufs; i++) { + io_put_rsrc_node(ctx->user_bufs[i]); + ctx->user_bufs[i] = NULL; + } + kvfree(ctx->user_bufs); ctx->user_bufs = NULL; + io_rsrc_data_free(ctx->buf_data); ctx->buf_data = NULL; ctx->nr_user_bufs = 0; } int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) { - unsigned nr = ctx->nr_user_bufs; - int ret; - if (!ctx->buf_data) return -ENXIO; - /* - * Quiesce may unlock ->uring_lock, and while it's not held - * prevent new requests using the table. - */ - ctx->nr_user_bufs = 0; - ret = io_rsrc_ref_quiesce(ctx->buf_data, ctx); - ctx->nr_user_bufs = nr; - if (!ret) - __io_sqe_buffers_unregister(ctx); - return ret; + __io_sqe_buffers_unregister(ctx); + return 0; } /* @@ -813,7 +654,8 @@ static bool headpage_already_acct(struct io_ring_ctx *ctx, struct page **pages, /* check previously registered pages */ for (i = 0; i < ctx->nr_user_bufs; i++) { - struct io_mapped_ubuf *imu = ctx->user_bufs[i]; + struct io_rsrc_node *node = ctx->user_bufs[i]; + struct io_mapped_ubuf *imu = node->buf; for (j = 0; j < imu->nr_bvecs; j++) { if (!PageCompound(imu->bvec[j].bv_page)) @@ -950,21 +792,28 @@ static bool io_try_coalesce_buffer(struct page ***pages, int *nr_pages, return io_do_coalesce_buffer(pages, nr_pages, data, nr_folios); } -static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, - struct io_mapped_ubuf **pimu, - struct page **last_hpage) +static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, + struct iovec *iov, + int index, + struct page **last_hpage) { struct io_mapped_ubuf *imu = NULL; struct page **pages = NULL; + struct io_rsrc_node *node; unsigned long off; size_t size; int ret, nr_pages, i; struct io_imu_folio_data data; bool coalesced; - *pimu = (struct io_mapped_ubuf *)&dummy_ubuf; - if (!iov->iov_base) - return 0; + node = io_rsrc_node_alloc(ctx, ctx->buf_data, index, IORING_RSRC_BUFFER); + if (!node) + return ERR_PTR(-ENOMEM); + + if (!iov->iov_base) { + node->buf = (struct io_mapped_ubuf *) &dummy_ubuf; + return node; + } ret = -ENOMEM; pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, @@ -998,7 +847,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->folio_shift = data.folio_shift; refcount_set(&imu->refs, 1); off = (unsigned long) iov->iov_base & ((1UL << imu->folio_shift) - 1); - *pimu = imu; + node->buf = imu; ret = 0; for (i = 0; i < nr_pages; i++) { @@ -1010,10 +859,14 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, size -= vec_len; } done: - if (ret) + if (ret) { kvfree(imu); + if (node) + io_put_rsrc_node(node); + node = ERR_PTR(ret); + } kvfree(pages); - return ret; + return node; } static int io_buffers_map_alloc(struct io_ring_ctx *ctx, unsigned int nr_args) @@ -1037,7 +890,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, return -EBUSY; if (!nr_args || nr_args > IORING_MAX_REG_BUFFERS) return -EINVAL; - ret = io_rsrc_data_alloc(ctx, IORING_RSRC_BUFFER, tags, nr_args, &data); + ret = io_rsrc_data_alloc(ctx, nr_args, &data); if (ret) return ret; ret = io_buffers_map_alloc(ctx, nr_args); @@ -1050,6 +903,9 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, memset(iov, 0, sizeof(*iov)); for (i = 0; i < nr_args; i++, ctx->nr_user_bufs++) { + struct io_rsrc_node *node; + u64 tag = 0; + if (arg) { uvec = (struct iovec __user *) arg; iov = iovec_from_user(uvec, 1, 1, &fast_iov, ctx->compat); @@ -1066,15 +922,24 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, arg += sizeof(struct iovec); } - if (!iov->iov_base && *io_get_tag_slot(data, i)) { - ret = -EINVAL; - break; + if (tags) { + if (copy_from_user(&tag, &tags[i], sizeof(tag))) { + ret = -EFAULT; + break; + } + if (tag && !iov->iov_base) { + ret = -EINVAL; + break; + } } - ret = io_sqe_buffer_register(ctx, iov, &ctx->user_bufs[i], - &last_hpage); - if (ret) + node = io_sqe_buffer_register(ctx, iov, i, &last_hpage); + if (IS_ERR(node)) { + ret = PTR_ERR(node); break; + } + node->tag = tag; + ctx->user_bufs[i] = node; } WARN_ON_ONCE(ctx->buf_data); @@ -1148,7 +1013,7 @@ int io_import_fixed(int ddir, struct iov_iter *iter, static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx) { - struct io_mapped_ubuf **user_bufs; + struct io_rsrc_node **user_bufs; struct io_rsrc_data *data; int i, ret, nbufs; @@ -1163,21 +1028,31 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx nbufs = src_ctx->nr_user_bufs; if (!nbufs) goto out_unlock; - ret = io_rsrc_data_alloc(ctx, IORING_RSRC_BUFFER, NULL, nbufs, &data); + ret = io_rsrc_data_alloc(ctx, nbufs, &data); if (ret) goto out_unlock; ret = -ENOMEM; - user_bufs = kcalloc(nbufs, sizeof(*ctx->user_bufs), GFP_KERNEL); + user_bufs = kvmalloc_array(nbufs, sizeof(struct io_rsrc_node *), + GFP_KERNEL | __GFP_ZERO); if (!user_bufs) goto out_free_data; for (i = 0; i < nbufs; i++) { - struct io_mapped_ubuf *src = src_ctx->user_bufs[i]; + struct io_mapped_ubuf *imu = src_ctx->user_bufs[i]->buf; + struct io_rsrc_node *dst_node; - if (src != &dummy_ubuf) - refcount_inc(&src->refs); - user_bufs[i] = src; + dst_node = io_rsrc_node_alloc(ctx, data, i, IORING_RSRC_BUFFER); + if (!dst_node) + goto out_put_free; + + if (imu == &dummy_ubuf) { + dst_node->buf = (struct io_mapped_ubuf *) &dummy_ubuf; + } else { + refcount_inc(&imu->refs); + dst_node->buf = imu; + } + user_bufs[i] = dst_node; } /* Have a ref on the bufs now, drop src lock and re-grab our own lock */ @@ -1190,12 +1065,17 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx return 0; } + mutex_unlock(&ctx->uring_lock); + mutex_lock(&src_ctx->uring_lock); /* someone raced setting up buffers, dump ours */ - for (i = 0; i < nbufs; i++) - io_buffer_unmap(ctx, &user_bufs[i]); - io_rsrc_data_free(data); - kfree(user_bufs); - return -EBUSY; + ret = -EBUSY; + i = nbufs; +out_put_free: + while (i--) { + io_buffer_unmap(src_ctx, user_bufs[i]); + kfree(user_bufs[i]); + } + kvfree(user_bufs); out_free_data: io_rsrc_data_free(data); out_unlock: diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 1589c9740083..37de08d34b78 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -11,38 +11,25 @@ enum { IORING_RSRC_FILE = 0, IORING_RSRC_BUFFER = 1, -}; - -struct io_rsrc_put { - u64 tag; - union { - void *rsrc; - struct file *file; - struct io_mapped_ubuf *buf; - }; + IORING_RSRC_INVALID = 2, }; struct io_rsrc_data { - struct io_ring_ctx *ctx; - - u64 **tags; unsigned int nr; - u16 rsrc_type; - bool quiesce; + struct io_rsrc_node **nodes; }; struct io_rsrc_node { struct io_ring_ctx *ctx; int refs; - bool empty; u16 type; - struct list_head node; - struct io_rsrc_put item; -}; -struct io_fixed_file { - /* file * with additional FFS_* flags */ - unsigned long file_ptr; + u64 tag; + union { + void *rsrc; + unsigned long file_ptr; + struct io_mapped_ubuf *buf; + }; }; struct io_mapped_ubuf { @@ -63,21 +50,19 @@ struct io_imu_folio_data { unsigned int folio_shift; }; -void io_rsrc_node_ref_zero(struct io_rsrc_node *node); -void io_rsrc_node_destroy(struct io_ring_ctx *ctx, struct io_rsrc_node *ref_node); -struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx); -int io_queue_rsrc_removal(struct io_rsrc_data *data, unsigned idx, void *rsrc); +struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, + struct io_rsrc_data *data, int index, + int type); +void io_free_rsrc_node(struct io_rsrc_node *node); int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, u64 buf_addr, size_t len); int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg); -void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx); int io_sqe_buffers_unregister(struct io_ring_ctx *ctx); int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, unsigned int nr_args, u64 __user *tags); -void __io_sqe_files_unregister(struct io_ring_ctx *ctx); int io_sqe_files_unregister(struct io_ring_ctx *ctx); int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, unsigned nr_args, u64 __user *tags); @@ -89,41 +74,23 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); -static inline void io_put_rsrc_node(struct io_ring_ctx *ctx, struct io_rsrc_node *node) +static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - lockdep_assert_held(&ctx->uring_lock); - if (node && !--node->refs) - io_rsrc_node_ref_zero(node); + io_free_rsrc_node(node); } -static inline void __io_req_set_rsrc_node(struct io_kiocb *req, - struct io_ring_ctx *ctx) +static inline void io_req_put_rsrc_nodes(struct io_kiocb *req) { - lockdep_assert_held(&ctx->uring_lock); - req->rsrc_node = ctx->rsrc_node; - ctx->rsrc_node->refs++; -} - -static inline void io_req_set_rsrc_node(struct io_kiocb *req, - struct io_ring_ctx *ctx) -{ - if (!req->rsrc_node) - __io_req_set_rsrc_node(req, ctx); -} - -static inline u64 *io_get_tag_slot(struct io_rsrc_data *data, unsigned int idx) -{ - unsigned int off = idx & IO_RSRC_TAG_TABLE_MASK; - unsigned int table_idx = idx >> IO_RSRC_TAG_TABLE_SHIFT; - - return &data->tags[table_idx][off]; + io_put_rsrc_node(req->rsrc_nodes[IORING_RSRC_FILE]); + io_put_rsrc_node(req->rsrc_nodes[IORING_RSRC_BUFFER]); } -static inline int io_rsrc_init(struct io_ring_ctx *ctx) +static inline void io_req_assign_rsrc_node(struct io_kiocb *req, + struct io_rsrc_node *node) { - ctx->rsrc_node = io_rsrc_node_alloc(ctx); - return ctx->rsrc_node ? 0 : -ENOMEM; + node->refs++; + req->rsrc_nodes[node->type] = node; } int io_files_update(struct io_kiocb *req, unsigned int issue_flags); diff --git a/io_uring/rw.c b/io_uring/rw.c index 8080ffd6d571..65491f4f2c7e 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -330,7 +330,7 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); struct io_ring_ctx *ctx = req->ctx; - struct io_mapped_ubuf *imu; + struct io_rsrc_node *node; struct io_async_rw *io; u16 index; int ret; @@ -342,11 +342,11 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe if (unlikely(req->buf_index >= ctx->nr_user_bufs)) return -EFAULT; index = array_index_nospec(req->buf_index, ctx->nr_user_bufs); - imu = ctx->user_bufs[index]; - io_req_set_rsrc_node(req, ctx); + node = ctx->user_bufs[index]; + io_req_assign_rsrc_node(req, node); io = req->async_data; - ret = io_import_fixed(ddir, &io->iter, imu, rw->addr, rw->len); + ret = io_import_fixed(ddir, &io->iter, node->buf, rw->addr, rw->len); iov_iter_save_state(&io->iter, &io->iter_state); return ret; } diff --git a/io_uring/splice.c b/io_uring/splice.c index e62bc6497a94..a0b4e0435b8b 100644 --- a/io_uring/splice.c +++ b/io_uring/splice.c @@ -51,7 +51,7 @@ void io_splice_cleanup(struct io_kiocb *req) { struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); - io_put_rsrc_node(req->ctx, sp->rsrc_node); + io_put_rsrc_node(sp->rsrc_node); } static struct file *io_splice_get_file(struct io_kiocb *req, @@ -59,7 +59,7 @@ static struct file *io_splice_get_file(struct io_kiocb *req, { struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); struct io_ring_ctx *ctx = req->ctx; - struct io_fixed_file *slot; + struct io_rsrc_node *node; struct file *file = NULL; if (!(sp->flags & SPLICE_F_FD_IN_FIXED)) @@ -69,11 +69,13 @@ static struct file *io_splice_get_file(struct io_kiocb *req, if (unlikely(sp->splice_fd_in >= ctx->nr_user_files)) goto out; sp->splice_fd_in = array_index_nospec(sp->splice_fd_in, ctx->nr_user_files); - slot = &ctx->file_table.files[sp->splice_fd_in]; - if (!req->rsrc_node) - __io_req_set_rsrc_node(req, ctx); - file = io_slot_file(slot); - req->flags |= REQ_F_NEED_CLEANUP; + node = ctx->file_table.nodes[sp->splice_fd_in]; + if (node) { + node->refs++; + sp->rsrc_node = node; + file = io_slot_file(node); + req->flags |= REQ_F_NEED_CLEANUP; + } out: io_ring_submit_unlock(ctx, issue_flags); return file; diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 6994f60d7ec7..0899c71008ae 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -220,7 +220,7 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) * being called. This prevents destruction of the mapped buffer * we'll need at actual import time. */ - io_req_set_rsrc_node(req, ctx); + io_req_assign_rsrc_node(req, ctx->user_bufs[req->buf_index]); } ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); @@ -276,15 +276,11 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw, struct iov_iter *iter, void *ioucmd) { struct io_kiocb *req = cmd_to_io_kiocb(ioucmd); - struct io_ring_ctx *ctx = req->ctx; + struct io_rsrc_node *node = req->rsrc_nodes[IORING_RSRC_BUFFER]; /* Must have had rsrc_node assigned at prep time */ - if (req->rsrc_node) { - struct io_mapped_ubuf *imu; - - imu = READ_ONCE(ctx->user_bufs[req->buf_index]); - return io_import_fixed(rw, iter, imu, ubuf, len); - } + if (node) + return io_import_fixed(rw, iter, node->buf, ubuf, len); return -EFAULT; } From patchwork Mon Oct 28 14:52:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853652 Received: from mail-il1-f172.google.com (mail-il1-f172.google.com [209.85.166.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 089361DE3B4 for ; Mon, 28 Oct 2024 15:04:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127895; cv=none; b=K9hy441yU/zg3ALMJtN5zpd++oGFERRSeVUwKJaNrZGnVySZp7yAJZL3br4qEwhCNRaSZB5d7Q13/ZhSJaAlfMRWWtPthkonZifi+jySN+sG8EBPDoZx2KiIXfyllE1sdXKkFXCIyJdri3TZXqXSWfjlyQJM6aVQxLepKILyrQQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127895; c=relaxed/simple; bh=Fo/frzSKlv4OyiJGKpbBRm4RdvJKEoEHG1D8QYdklYc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QCs7pttQHPx/eWdmXQYx1K0Tzt8LOKYLQYhE1z5Kkx0/YUc/Flfr8M71uiCdeK4WkeKBT+ytTU58oYcuF46ZTSo7qkuiw7hpg+IZi10rVZrpy+029Uj7ooxNV51+Vsgx5KoTCoDKjW6XffsGdxtRJfhlhzZsXarY4wEen2xTnY8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=aBTtE8fl; arc=none smtp.client-ip=209.85.166.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="aBTtE8fl" Received: by mail-il1-f172.google.com with SMTP id e9e14a558f8ab-3a4d2d028b3so16367175ab.1 for ; Mon, 28 Oct 2024 08:04:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127891; x=1730732691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GJ3VbKxyariUHBQbk+XVcQESHHWp/zX87MdRysrKFDA=; b=aBTtE8fl0Vb0S1nbBvG9fpxL8muszqJTmQldY1OoX7Cw1TWsBW3aL2CCVMwRXPqm1I h+KwiPSthUluOBjl+WXAb4xB+2A5vTPiij07taztjaCjuXTGjLd5RhALVGPJej8ivmhD +fPA0svzLTNonlJnXjiKZrXrDQFHWeCF1Ip6uj+WwTgfonnxVuNB75U+CuEvLFOx/E9H dQ6i/6iuQnSWpwVIhDwDwAS0AbwHdqmhiEv4gBWx0rSInW4C6lk3p7kj4dUhPSZr2GqI ajxE+GHGwk6a4UWr9NnfDmOO6k9tQRohOS00HG0xTPeUYf1rLqi16c/qYCDvGoHVFrsC VbRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127891; x=1730732691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GJ3VbKxyariUHBQbk+XVcQESHHWp/zX87MdRysrKFDA=; b=dNCJF/VL9y2JA54/4ZJivld+WIa1gGClt+FDf+mwnPJR25mPBLRMpCPjTaic6BIr05 CIy6erZgOqhikuBJmafoIhCX0e74+2pqYZh+oVEJf+x90whqLplMsIHdBtmb31eEnvhp yOPm/vraJqq8YRoKk1uBCs4v+Xto7St+KZ4R+PePkISH+2Odr76PKiIgNbpg4FreR/Z2 C9/OHAoq7GsR8d+sh31Uo6L3E/qLBRTWTtzjSgnByL7edVHuOl7lRyCz4E5iqlh5++HE QgRxAbK49RYGzebVBE61T8gRzOm+UANZeuP6lvEfRq/j76q9vkOx/9xtEJdVlhM7++C2 TNYg== X-Gm-Message-State: AOJu0YwkZgsNlMXjnG9HaSv8ZgnRQp+MRooDChjaNIJKKMUw8AthSvHZ B/7CfuMaR89ir8ANygSIXu/OOrSfp93qch7NdmZv+rWml0aJvLZ+hY3etHglGiheYwGojJTlD5E g X-Google-Smtp-Source: AGHT+IES7Jd6cCdJRy1MXc+PrwXrR23NQwRLXweEj2Rq4VrwSLsz0xX2jxdzSutkXImW2Nl3eMClCQ== X-Received: by 2002:a05:6e02:17ce:b0:3a3:b254:ca2c with SMTP id e9e14a558f8ab-3a4ed3524e1mr78114455ab.25.1730127891270; Mon, 28 Oct 2024 08:04:51 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:50 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 07/13] io_uring/rsrc: get rid of io_rsrc_node allocation cache Date: Mon, 28 Oct 2024 08:52:37 -0600 Message-ID: <20241028150437.387667-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's not going to be needed in the fast path going forward, so kill it off. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 --- io_uring/io_uring.c | 6 +----- io_uring/rsrc.c | 22 ++++++++-------------- 3 files changed, 9 insertions(+), 22 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 42c5f2c992c4..696f2a05a98b 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -370,9 +370,6 @@ struct io_ring_ctx { struct io_rsrc_data *file_data; struct io_rsrc_data *buf_data; - /* protected by ->uring_lock */ - struct io_alloc_cache rsrc_node_cache; - u32 pers_next; struct xarray personalities; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0876aa74c739..094788cca47f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -312,9 +312,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); - ret = io_alloc_cache_init(&ctx->rsrc_node_cache, IO_NODE_ALLOC_CACHE_MAX, - sizeof(struct io_rsrc_node)); - ret |= io_alloc_cache_init(&ctx->apoll_cache, IO_POLL_ALLOC_CACHE_MAX, + ret = io_alloc_cache_init(&ctx->apoll_cache, IO_POLL_ALLOC_CACHE_MAX, sizeof(struct async_poll)); ret |= io_alloc_cache_init(&ctx->netmsg_cache, IO_ALLOC_CACHE_MAX, sizeof(struct io_async_msghdr)); @@ -358,7 +356,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) free_ref: percpu_ref_exit(&ctx->refs); err: - io_alloc_cache_free(&ctx->rsrc_node_cache, kfree); io_alloc_cache_free(&ctx->apoll_cache, kfree); io_alloc_cache_free(&ctx->netmsg_cache, io_netmsg_cache_free); io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); @@ -2740,7 +2737,6 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) WARN_ON_ONCE(!list_empty(&ctx->ltimeout_list)); - io_alloc_cache_free(&ctx->rsrc_node_cache, kfree); if (ctx->mm_account) { mmdrop(ctx->mm_account); ctx->mm_account = NULL; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index cc93d0c2df4c..1ba7f3e55947 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -13,7 +13,6 @@ #include #include "io_uring.h" -#include "alloc_cache.h" #include "openclose.h" #include "rsrc.h" #include "memmap.h" @@ -132,18 +131,14 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, { struct io_rsrc_node *node; - node = io_alloc_cache_get(&ctx->rsrc_node_cache); - if (!node) { - node = kzalloc(sizeof(*node), GFP_KERNEL); - if (!node) - return NULL; + node = kzalloc(sizeof(*node), GFP_KERNEL); + if (node) { + node->ctx = ctx; + node->refs = 1; + node->type = type; + node->tag = 0; + node->rsrc = NULL; } - - node->ctx = ctx; - node->refs = 1; - node->type = type; - node->tag = 0; - node->rsrc = NULL; return node; } @@ -492,8 +487,7 @@ void io_free_rsrc_node(struct io_rsrc_node *node) break; } - if (!io_alloc_cache_put(&ctx->rsrc_node_cache, node)) - kfree(node); + kfree(node); } static void __io_sqe_files_unregister(struct io_ring_ctx *ctx) From patchwork Mon Oct 28 14:52:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853654 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07E251DD894 for ; Mon, 28 Oct 2024 15:04:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127896; cv=none; b=cgs6zmyMgG5IsMIfwXHXr99RDvidh/gIUMrBmeopl/sD9fkhguM12GuafvZQYH3tvNVKY3uujQN8i6lmjFiuD6XCoRJpPU2tRJwtv5IcU+JeGkRROCSIWR31f5AcD2i1YX+oo23R5ywFBsg7k7GnJjaKkCLlKOMqBIwQntn4Pss= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127896; c=relaxed/simple; bh=J9Bqst/kwlzUhBvdftqt01lWwWoE1NQ1Zq+4lTD9fck=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UW3DB0I4BYS/Nkh/yihws9a1vHvn7Nk4XgG00VXjq5LsDvjC8rIGBmBxXM545g9ZL1qt7tdOLPCYljcXlJxIZ+tgLCPX8hpWzVGKJMhDVHz4QMYxbRjNUD9WsjePCryUFoQltqrylIHoxenZDZPjRx0k0z7+Ucby77gGZVXvDrM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=DYgZhT3a; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="DYgZhT3a" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-83ab00438acso123568539f.0 for ; Mon, 28 Oct 2024 08:04:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127892; x=1730732692; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wPZlXwAr3l63Q1issR7TuoyIsfSx3r6dCl3WpbSrH2o=; b=DYgZhT3aVWuUG9B+vjivUGCL8/HFYkVeUzYyMYpXp38f5VI+Yg6IOSiIYifEyxs1xg qu3xmnEzLG25vH+A0g47LjO6Xn59hyGGLHWgHPECKArmlR9IW4kMSl7lwAYCn/ymDzgB PdfEml8SdwtlX++JTncMPle76uNm2YCTMwunTQEjL16+pZNrfELAS7UBX8IL5N/ZzkJD dR47nnvS+5D7zh54OgNArbp0ClOtKj7nI4Wjpa6J+Lk5HmsI8h7bQMQ0/WZmSW66WfW2 qWWyZmIwIdiNe/yNTwr+U4DAdPBofgNTyAJ1oeSN0hQsWr5gGXHDahnpk2q8700Txeyp WfKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127892; x=1730732692; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wPZlXwAr3l63Q1issR7TuoyIsfSx3r6dCl3WpbSrH2o=; b=isCgqej08cNIs+QvWE4Mdd5HtES8gFtCOMl3Xn2GZD/YyRq0Ml4O1QTpvUmzLW4bn9 /J5sztM0qg9iJuSNFfFT2+EP9PGihr2hNZ/4WRSN1P7JgInaMwjowoXnPOGF+kYYFTUv vDopjVyDgUkmjWIn2KG/tcnAE2BvPL3Wp7UxjeS79tu+gNLIRjmch6y8LplA3aFTUUKL /dJmxL/9GWQzqqGyyyamoHtQVdC9gCXJTi5J8Sn2MW0WU/IJX4b6A9vs4aNQrmwKO0FL rVbePEVBQeAX3Mh8JBJRcsLFfqf4q9M4B3VPP/VkU3RFxzxlJ4/wMYKTOkJuReNRsTpO SZtA== X-Gm-Message-State: AOJu0YwWyF/JhT5uDVSLWURCDHCtuTxJzvwJ9cTPH3hiUjYp9a9MbWUU qg9cltf4AfJUa9T8mjKv5lJTYm7/xJKMijWdRINtjwOSnQqxyOZtHx6LbnoWR32RMhXUMtPDpLt j X-Google-Smtp-Source: AGHT+IH6i8pwbSF76o202g9n8Ayzg8ljXADVkH6qS2J6d8EpJ+XjEE3I/gsFtWrcMKPFyPsMJ689xA== X-Received: by 2002:a05:6e02:214a:b0:3a3:dadc:12f7 with SMTP id e9e14a558f8ab-3a50896d7e6mr1214355ab.3.1730127892224; Mon, 28 Oct 2024 08:04:52 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:51 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 08/13] io_uring/rsrc: add an empty io_rsrc_node for sparse buffer entries Date: Mon, 28 Oct 2024 08:52:38 -0600 Message-ID: <20241028150437.387667-9-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than allocate an io_rsrc_node for an empty/sparse buffer entry, add a const entry that can be used for that. This just needs checking for writing the tag, and the put check needs to check for that sparse node rather than NULL for validity. This avoids allocating rsrc nodes for sparse buffer entries. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 4 ++-- io_uring/notif.c | 4 ++-- io_uring/rsrc.c | 52 +++++++++++++++++++++++++++------------------ io_uring/rsrc.h | 11 +++++++--- io_uring/splice.c | 2 +- 5 files changed, 44 insertions(+), 29 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 094788cca47f..9282d5fa45d3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2032,8 +2032,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->flags = (__force io_req_flags_t) sqe_flags; req->cqe.user_data = READ_ONCE(sqe->user_data); req->file = NULL; - req->rsrc_nodes[IORING_RSRC_FILE] = NULL; - req->rsrc_nodes[IORING_RSRC_BUFFER] = NULL; + req->rsrc_nodes[IORING_RSRC_FILE] = rsrc_empty_node; + req->rsrc_nodes[IORING_RSRC_BUFFER] = rsrc_empty_node; req->task = current; req->cancel_seq_set = false; diff --git a/io_uring/notif.c b/io_uring/notif.c index 4f02e969cf08..44bf21c0f810 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -117,8 +117,8 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx) notif->file = NULL; notif->task = current; io_get_task_refs(1); - notif->rsrc_nodes[IORING_RSRC_FILE] = NULL; - notif->rsrc_nodes[IORING_RSRC_BUFFER] = NULL; + notif->rsrc_nodes[IORING_RSRC_FILE] = rsrc_empty_node; + notif->rsrc_nodes[IORING_RSRC_BUFFER] = rsrc_empty_node; nd = io_notif_to_data(notif); nd->zc_report = false; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1ba7f3e55947..08c57332c783 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -38,6 +38,11 @@ static const struct io_mapped_ubuf dummy_ubuf = { .len = UINT_MAX, }; +const struct io_rsrc_node empty_node = { + .type = IORING_RSRC_BUFFER, + .buf = (struct io_mapped_ubuf *) &dummy_ubuf, +}; + int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -149,7 +154,8 @@ static void io_rsrc_data_free(struct io_rsrc_data *data) for (i = 0; i < data->nr; i++) { struct io_rsrc_node *node = data->nodes[i]; - io_put_rsrc_node(node); + if (node) + io_put_rsrc_node(node); } kvfree(data->nodes); kfree(data); @@ -234,7 +240,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, break; } ctx->file_table.nodes[i] = node; - node->tag = tag; + if (tag) + node->tag = tag; io_fixed_file_set(node, file); io_file_bitmap_set(&ctx->file_table, i); } @@ -286,10 +293,12 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, err = PTR_ERR(node); break; } - io_put_rsrc_node(ctx->user_bufs[i]); + if (ctx->user_bufs[i]) + io_put_rsrc_node(ctx->user_bufs[i]); ctx->user_bufs[i] = node; - node->tag = tag; + if (tag) + node->tag = tag; if (ctx->compat) user_data += sizeof(struct compat_iovec); else @@ -605,8 +614,10 @@ static void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx) lockdep_assert_held(&ctx->uring_lock); for (i = 0; i < ctx->nr_user_bufs; i++) { - io_put_rsrc_node(ctx->user_bufs[i]); - ctx->user_bufs[i] = NULL; + if (ctx->user_bufs[i]) { + io_put_rsrc_node(ctx->user_bufs[i]); + ctx->user_bufs[i] = NULL; + } } kvfree(ctx->user_bufs); ctx->user_bufs = NULL; @@ -800,15 +811,13 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct io_imu_folio_data data; bool coalesced; + if (!iov->iov_base) + return rsrc_empty_node; + node = io_rsrc_node_alloc(ctx, ctx->buf_data, index, IORING_RSRC_BUFFER); if (!node) return ERR_PTR(-ENOMEM); - if (!iov->iov_base) { - node->buf = (struct io_mapped_ubuf *) &dummy_ubuf; - return node; - } - ret = -ENOMEM; pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, &nr_pages); @@ -932,7 +941,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, ret = PTR_ERR(node); break; } - node->tag = tag; + if (tag) + node->tag = tag; ctx->user_bufs[i] = node; } @@ -1033,18 +1043,18 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx goto out_free_data; for (i = 0; i < nbufs; i++) { - struct io_mapped_ubuf *imu = src_ctx->user_bufs[i]->buf; + struct io_rsrc_node *src_node = src_ctx->user_bufs[i]; struct io_rsrc_node *dst_node; - dst_node = io_rsrc_node_alloc(ctx, data, i, IORING_RSRC_BUFFER); - if (!dst_node) - goto out_put_free; - - if (imu == &dummy_ubuf) { - dst_node->buf = (struct io_mapped_ubuf *) &dummy_ubuf; + if (src_node == rsrc_empty_node) { + dst_node = rsrc_empty_node; } else { - refcount_inc(&imu->refs); - dst_node->buf = imu; + dst_node = io_rsrc_node_alloc(ctx, data, i, IORING_RSRC_BUFFER); + if (!dst_node) + goto out_put_free; + + refcount_inc(&src_node->buf->refs); + dst_node->buf = src_node->buf; } user_bufs[i] = dst_node; } diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 37de08d34b78..fb86f080ae5c 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -74,9 +74,12 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg, int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, unsigned int size, unsigned int type); +extern const struct io_rsrc_node empty_node; +#define rsrc_empty_node (struct io_rsrc_node *) &empty_node + static inline void io_put_rsrc_node(struct io_rsrc_node *node) { - if (node && !--node->refs) + if (node != &empty_node && !--node->refs) io_free_rsrc_node(node); } @@ -89,8 +92,10 @@ static inline void io_req_put_rsrc_nodes(struct io_kiocb *req) static inline void io_req_assign_rsrc_node(struct io_kiocb *req, struct io_rsrc_node *node) { - node->refs++; - req->rsrc_nodes[node->type] = node; + if (node != rsrc_empty_node) { + node->refs++; + req->rsrc_nodes[node->type] = node; + } } int io_files_update(struct io_kiocb *req, unsigned int issue_flags); diff --git a/io_uring/splice.c b/io_uring/splice.c index a0b4e0435b8b..f78afb575ae6 100644 --- a/io_uring/splice.c +++ b/io_uring/splice.c @@ -35,7 +35,7 @@ static int __io_splice_prep(struct io_kiocb *req, if (unlikely(sp->flags & ~valid_flags)) return -EINVAL; sp->splice_fd_in = READ_ONCE(sqe->splice_fd_in); - sp->rsrc_node = NULL; + sp->rsrc_node = rsrc_empty_node; req->flags |= REQ_F_FORCE_ASYNC; return 0; } From patchwork Mon Oct 28 14:52:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853653 Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC90218FDB0 for ; Mon, 28 Oct 2024 15:04:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127896; cv=none; b=a2LBOHgq6qf5esxD3wJ5Qnj1YIRMo9D8AvjmR/w+Z+dL7yiI8N1I3lAu557MAwzQizMQcQmNDeV0XGZKVLXOD/Mn8ANMbVr/gJcBbSmFosHFKWBNobc7CIw9i7sujAoWqzFIx7x0MifitxHZ226gOvgGFuOfps1RDu7j0dy+aMg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127896; c=relaxed/simple; bh=aONGbiXI68zJwmEw4baPUiPbjsy3j7TKGRE5bSfm3Wc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nvmlmegQtP6VnCijD7mbsurMv68XILMlPtw3UWeBL4hz0biycPfJfLlg0CtxkNRhN9nAKzoviZ6GYR1hb7hg2Ia/zld7SjZmwG+E7h+hK10TJas5Wy8Z5FUv8IIAdtvenaTsHieOXqunKLiyj9108zPRxIVOtDzs2ecDihyXRe8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=kEongFck; arc=none smtp.client-ip=209.85.166.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="kEongFck" Received: by mail-il1-f173.google.com with SMTP id e9e14a558f8ab-3a4ccfde548so12022955ab.1 for ; Mon, 28 Oct 2024 08:04:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127893; x=1730732693; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MeRQPNzBbEmZmIGWPgoBj0EMrrHCu/JDI5w42FbaIPQ=; b=kEongFckB9Sw4JPjbh3vRPYDynahNg2QeQDkoCGU/mcjsQP0iazT3lbwoEhjBl6mqK MvfnRzm5gSai63qoRDQ35SfvDqO9vW+cJJyyZBWQtrgghMyAzJq7eiy2YxoNYTonfjLo cVr9qJq0f6sRQEEQCb4LqKBBd5rYB32nBAPqx8dIaAN625tFCBCNwtOx9PvaJJ1Jn43d yMDzvKjxhopilucQwFbD79BzEKaCKPuIC0L4C2bqkW7//17MouRHeO1E6r/Q5ppz97ev TItfCvyyhaSZSnxP9MHmAqzG7anKeVi/bg0SLnRQM3wygkydveWxgQluZfafdXTAx+nO lO6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127893; x=1730732693; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MeRQPNzBbEmZmIGWPgoBj0EMrrHCu/JDI5w42FbaIPQ=; b=i9uLt+4bcNMehFnTGwCdJ30n9xUtT7sxmaFmWxRSqI5yr9FSnm2jT/9ONKVRotivX2 kCcl85F6vvQkzwfBTj7JJ8kphxb7VmdJTAvyXU6MojOB/j26ve/tN5BEJWce1g2OHnG5 ST1AIS/HudCBekk0D7GY6pltS0jNh65D4vXHxmjKOQhbwgtI1AaTQBrmAQd9X4njrHxK sHPnRuiEV6LoFM3vwTH0DfzDjGetGAWosJNjputDObr0pmvegkBVEQMVQHl8rH4W0nm8 FpliGuu4EGUizJcL6nRcoGBA0sk1xPF1z+P4pUjijtNz+isFeYk6Jq6jJmbCge2yAN4e u3fw== X-Gm-Message-State: AOJu0YyZNTfToiN6TciDIOsskoAk5F9Wvdycn3ef6ScPyLqiSy4LXOkC dK0mKInyi4Ak8LO7QDAiAlisOYfz7s6dk7sYMy0lX8TcABDwUxt/PmiBsOWBFRThj3EYN86w4em o X-Google-Smtp-Source: AGHT+IGRb8sZ7gH5u1I6F9N7pvTnXCIt+fv3o1Z3QPo9orEJLP/rP67NbJ/L6Fb8/VwFl4gaiP58tw== X-Received: by 2002:a05:6e02:198f:b0:39d:3c87:1435 with SMTP id e9e14a558f8ab-3a50896d7cemr1209965ab.1.1730127893156; Mon, 28 Oct 2024 08:04:53 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:52 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 09/13] io_uring: only initialize io_kiocb rsrc_nodes when needed Date: Mon, 28 Oct 2024 08:52:39 -0600 Message-ID: <20241028150437.387667-10-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add the empty node initializing to the preinit part of the io_kiocb allocation, and reset them if they have been used. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 4 ++-- io_uring/rsrc.h | 10 ++++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 9282d5fa45d3..60c947114fa3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -947,6 +947,8 @@ void io_req_defer_failed(struct io_kiocb *req, s32 res) static void io_preinit_req(struct io_kiocb *req, struct io_ring_ctx *ctx) { req->ctx = ctx; + req->rsrc_nodes[IORING_RSRC_FILE] = rsrc_empty_node; + req->rsrc_nodes[IORING_RSRC_BUFFER] = rsrc_empty_node; req->link = NULL; req->async_data = NULL; /* not necessary, but safer to zero */ @@ -2032,8 +2034,6 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->flags = (__force io_req_flags_t) sqe_flags; req->cqe.user_data = READ_ONCE(sqe->user_data); req->file = NULL; - req->rsrc_nodes[IORING_RSRC_FILE] = rsrc_empty_node; - req->rsrc_nodes[IORING_RSRC_BUFFER] = rsrc_empty_node; req->task = current; req->cancel_seq_set = false; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index fb86f080ae5c..53907e142ae0 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -85,8 +85,14 @@ static inline void io_put_rsrc_node(struct io_rsrc_node *node) static inline void io_req_put_rsrc_nodes(struct io_kiocb *req) { - io_put_rsrc_node(req->rsrc_nodes[IORING_RSRC_FILE]); - io_put_rsrc_node(req->rsrc_nodes[IORING_RSRC_BUFFER]); + if (req->rsrc_nodes[IORING_RSRC_FILE] != rsrc_empty_node) { + io_put_rsrc_node(req->rsrc_nodes[IORING_RSRC_FILE]); + req->rsrc_nodes[IORING_RSRC_FILE] = rsrc_empty_node; + } + if (req->rsrc_nodes[IORING_RSRC_BUFFER] != rsrc_empty_node) { + io_put_rsrc_node(req->rsrc_nodes[IORING_RSRC_BUFFER]); + req->rsrc_nodes[IORING_RSRC_BUFFER] = rsrc_empty_node; + } } static inline void io_req_assign_rsrc_node(struct io_kiocb *req, From patchwork Mon Oct 28 14:52:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853657 Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F04091DE3A4 for ; Mon, 28 Oct 2024 15:04:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127900; cv=none; b=GCIT5ce68YqHddUV+upXhueizj+p0S6C1ax5M3sD6pDsOiM1rqvuQTsxfcPmZQp5zWSilSOSaYHuPLtg6Cof5gJnKFGnFxLoDItQY4Dj3inSrG4JUzUNeOetmseA8d08B+Sakip+xEf0cSGpB8EnKITxN7WHgqVxbiBp3GA1x+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127900; c=relaxed/simple; bh=0BnyqkamtuUuQn4QJVLoOghGjVWM62N0LYqrMKR2z2s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CiKrvvrKflQKvwbrdtUlvVHnJ39iBdFvsKhfrkg8lFWor/5J8hkmYPMbNBMEpTES6mzII7XSF5pYFrsFEY797dIMUzIUrnLEq4WRvsIrireP+8H1iLaQEhzzgsgYZ2I01Jn/s4NCzPAHaQJb7eVUljub5nA0tUqzTUj3HWXrBsA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=QCnnT6/Y; arc=none smtp.client-ip=209.85.210.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="QCnnT6/Y" Received: by mail-ot1-f42.google.com with SMTP id 46e09a7af769-7180a5ba498so2204504a34.2 for ; Mon, 28 Oct 2024 08:04:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127895; x=1730732695; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1BS+W7w4HDWk3Ej95MgpffIdqXXe0wni3SuXIa6yK7c=; b=QCnnT6/YWO1wPU3XaheW3H/FVi7Hq7B5wKRuVTZDl4ObGjxQaprgrKi+yD3l5t8ZAr zIAnybH8bBWttjLc0M4kTC11r1/QpTb0a4VVocX6wql4D4BENGdjrFov/zKlqoj+tm+X /bVIqYSJGlsP6g/eQ2uJ7eld6yr+jj1KETTGKfWSK3hlikQ2/IzQKK9LBKtc/EaJ3N99 eisC5hTR2qIWMvlkIu7NYvA5FL1m38Y94EUbfrGtZHaey/DEYRv+yjOyj65+n/fuxhl1 1z5uSjsE8Nah/Taf5kz/+vftBOvQeUQzexMhKqma6/YXByIlYSnUSmtgWJuxN5MAUR9O IjvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127895; x=1730732695; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1BS+W7w4HDWk3Ej95MgpffIdqXXe0wni3SuXIa6yK7c=; b=Lhr0SawWIywaTaplt8VpCeTHxPlVWOguqV2Mw0NF3v7JHbsoiMQ72/CQ7HJEvB9Y7z mhTRwiSzhY9zExhqNE/dLRGT3c/U14U/0ZUd15jWJ3W3f6DqxgirJwM6ErT2RrxNIGKB NE8rqpoMNm9o5HFfeccn4FQyoywloy7AijidA/vcwGV8hz1mJtWACg8Za5AdpyaRbuZR 5zaGaOb3c0fUY4UjBN+Cq7P86tM/IEww7y0KjEEJz8rOQvFaX+4ZiHLMfhi/Ku8FncQm BLAZtpvQXQQFeZdB2rNEbTjXg/Xn6OOoSLHjmyrak2YKYqaT3E7eIQ2UfHt60YRHCaxO 9fIw== X-Gm-Message-State: AOJu0YyAG8+ct2wIYc6jxYAgrFb7XQKRgmc2dOtBCHfRcSXGShhffTpm 01PATuifcBVk+tmnqrIZzgLEltpIBhFBVg3658LqGXpAYrb82VwV/1HqZj5t4oREl16LWBkZZxa D X-Google-Smtp-Source: AGHT+IHnOV6xaqEJSWOXHTJZ7zKxWgNQM+3boCY1wjVjbFtMwiS2VgNWVeQMEsXhyy5ZM3IDD/IaoQ== X-Received: by 2002:a05:6830:610a:b0:717:f7b9:e408 with SMTP id 46e09a7af769-7186829a607mr9788883a34.28.1730127895159; Mon, 28 Oct 2024 08:04:55 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:53 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 10/13] io_uring/rsrc: unify file and buffer resource tables Date: Mon, 28 Oct 2024 08:52:40 -0600 Message-ID: <20241028150437.387667-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For files, there's nr_user_files/file_table/file_data, and buffers have nr_user_bufs/user_bufs/buf_data. There's no reason why file_table and file_data can't be the same thing, and ditto for the buffer side. That gets rid of more io_ring_ctx state that's in two spots rather than just being in one spot, as it should be. Put all the registered file data in one locations, and ditto on the buffer front. This also avoids having both io_rsrc_data->nodes being an allocated array, and ->user_bufs[] or ->file_table.nodes. There's no reason to have this information duplicated. Keep it in one spot, io_rsrc_data, along with how many resources are available. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 15 ++- io_uring/cancel.c | 4 +- io_uring/fdinfo.c | 10 +- io_uring/filetable.c | 47 ++++---- io_uring/filetable.h | 2 +- io_uring/io_uring.c | 7 +- io_uring/msg_ring.c | 4 +- io_uring/net.c | 6 +- io_uring/nop.c | 6 +- io_uring/register.c | 3 +- io_uring/rsrc.c | 212 ++++++++++----------------------- io_uring/rsrc.h | 17 ++- io_uring/rw.c | 6 +- io_uring/splice.c | 6 +- io_uring/uring_cmd.c | 6 +- 15 files changed, 131 insertions(+), 220 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 696f2a05a98b..77fd508d043a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -55,8 +55,13 @@ struct io_wq_work { int cancel_seq; }; +struct io_rsrc_data { + unsigned int nr; + struct io_rsrc_node **nodes; +}; + struct io_file_table { - struct io_rsrc_node **nodes; + struct io_rsrc_data data; unsigned long *bitmap; unsigned int alloc_hint; }; @@ -276,9 +281,7 @@ struct io_ring_ctx { struct io_wq_work_list iopoll_list; struct io_file_table file_table; - struct io_rsrc_node **user_bufs; - unsigned nr_user_files; - unsigned nr_user_bufs; + struct io_rsrc_data buf_table; struct io_submit_state submit_state; @@ -366,10 +369,6 @@ struct io_ring_ctx { struct wait_queue_head poll_wq; struct io_restriction restrictions; - /* slow path rsrc auxilary data, used by update/register */ - struct io_rsrc_data *file_data; - struct io_rsrc_data *buf_data; - u32 pers_next; struct xarray personalities; diff --git a/io_uring/cancel.c b/io_uring/cancel.c index cc3475b22ae5..3a2996307025 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -240,9 +240,9 @@ static int __io_sync_cancel(struct io_uring_task *tctx, /* fixed must be grabbed every time since we drop the uring_lock */ if ((cd->flags & IORING_ASYNC_CANCEL_FD) && (cd->flags & IORING_ASYNC_CANCEL_FD_FIXED)) { - if (unlikely(fd >= ctx->nr_user_files)) + if (unlikely(fd >= ctx->file_table.data.nr)) return -EBADF; - fd = array_index_nospec(fd, ctx->nr_user_files); + fd = array_index_nospec(fd, ctx->file_table.data.nr); cd->file = io_file_from_index(&ctx->file_table, fd); if (!cd->file) return -EBADF; diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index 064a79475c5f..e3f5e9fe5562 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -165,8 +165,8 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file) seq_printf(m, "SqThreadCpu:\t%d\n", sq_cpu); seq_printf(m, "SqTotalTime:\t%llu\n", sq_total_time); seq_printf(m, "SqWorkTime:\t%llu\n", sq_work_time); - seq_printf(m, "UserFiles:\t%u\n", ctx->nr_user_files); - for (i = 0; has_lock && i < ctx->nr_user_files; i++) { + seq_printf(m, "UserFiles:\t%u\n", ctx->file_table.data.nr); + for (i = 0; has_lock && i < ctx->file_table.data.nr; i++) { struct file *f = io_file_from_index(&ctx->file_table, i); if (f) @@ -174,9 +174,9 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file) else seq_printf(m, "%5u: \n", i); } - seq_printf(m, "UserBufs:\t%u\n", ctx->nr_user_bufs); - for (i = 0; has_lock && i < ctx->nr_user_bufs; i++) { - struct io_mapped_ubuf *buf = ctx->user_bufs[i]->buf; + seq_printf(m, "UserBufs:\t%u\n", ctx->buf_table.nr); + for (i = 0; has_lock && i < ctx->buf_table.nr; i++) { + struct io_mapped_ubuf *buf = ctx->buf_table.nodes[i]->buf; seq_printf(m, "%5u: 0x%llx/%u\n", i, buf->ubuf, buf->len); } diff --git a/io_uring/filetable.c b/io_uring/filetable.c index c18e37b495a1..c4c31f8b4f5d 100644 --- a/io_uring/filetable.c +++ b/io_uring/filetable.c @@ -38,25 +38,19 @@ static int io_file_bitmap_get(struct io_ring_ctx *ctx) bool io_alloc_file_tables(struct io_file_table *table, unsigned nr_files) { - table->nodes = kvmalloc_array(nr_files, sizeof(struct io_src_node *), - GFP_KERNEL_ACCOUNT | __GFP_ZERO); - if (unlikely(!table->nodes)) + if (io_rsrc_data_alloc(&table->data, nr_files)) return false; - table->bitmap = bitmap_zalloc(nr_files, GFP_KERNEL_ACCOUNT); - if (unlikely(!table->bitmap)) { - kvfree(table->nodes); - return false; - } - - return true; + if (table->bitmap) + return true; + io_rsrc_data_free(&table->data); + return false; } void io_free_file_tables(struct io_file_table *table) { - kvfree(table->nodes); + io_rsrc_data_free(&table->data); bitmap_free(table->bitmap); - table->nodes = NULL; table->bitmap = NULL; } @@ -68,22 +62,23 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, if (io_is_uring_fops(file)) return -EBADF; - if (!ctx->file_data) + if (!ctx->file_table.data.nr) return -ENXIO; - if (slot_index >= ctx->nr_user_files) + if (slot_index >= ctx->file_table.data.nr) return -EINVAL; - node = io_rsrc_node_alloc(ctx, ctx->file_data, slot_index, IORING_RSRC_FILE); + node = io_rsrc_node_alloc(ctx, &ctx->file_table.data, slot_index, + IORING_RSRC_FILE); if (IS_ERR(node)) return -ENOMEM; - slot_index = array_index_nospec(slot_index, ctx->nr_user_files); - if (ctx->file_table.nodes[slot_index]) - io_put_rsrc_node(ctx->file_table.nodes[slot_index]); + slot_index = array_index_nospec(slot_index, ctx->file_table.data.nr); + if (ctx->file_table.data.nodes[slot_index]) + io_put_rsrc_node(ctx->file_table.data.nodes[slot_index]); else io_file_bitmap_set(&ctx->file_table, slot_index); - ctx->file_table.nodes[slot_index] = node; + ctx->file_table.data.nodes[slot_index] = node; io_fixed_file_set(node, file); return 0; } @@ -129,16 +124,16 @@ int io_fixed_fd_install(struct io_kiocb *req, unsigned int issue_flags, int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset) { - if (unlikely(!ctx->file_data)) + if (unlikely(!ctx->file_table.data.nr)) return -ENXIO; - if (offset >= ctx->nr_user_files) + if (offset >= ctx->file_table.data.nr) return -EINVAL; - offset = array_index_nospec(offset, ctx->nr_user_files); - if (!ctx->file_table.nodes[offset]) + offset = array_index_nospec(offset, ctx->file_table.data.nr); + if (!ctx->file_table.data.nodes[offset]) return -EBADF; - io_put_rsrc_node(ctx->file_table.nodes[offset]); - ctx->file_table.nodes[offset] = NULL; + io_put_rsrc_node(ctx->file_table.data.nodes[offset]); + ctx->file_table.data.nodes[offset] = NULL; io_file_bitmap_clear(&ctx->file_table, offset); return 0; } @@ -153,7 +148,7 @@ int io_register_file_alloc_range(struct io_ring_ctx *ctx, return -EFAULT; if (check_add_overflow(range.off, range.len, &end)) return -EOVERFLOW; - if (range.resv || end > ctx->nr_user_files) + if (range.resv || end > ctx->file_table.data.nr) return -EINVAL; io_file_table_set_alloc_range(ctx, range.off, range.len); diff --git a/io_uring/filetable.h b/io_uring/filetable.h index 47616079abaa..664c31502dbb 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -52,7 +52,7 @@ static inline struct file *io_slot_file(struct io_rsrc_node *node) static inline struct file *io_file_from_index(struct io_file_table *table, int index) { - struct io_rsrc_node *node = table->nodes[index]; + struct io_rsrc_node *node = table->data.nodes[index]; if (node) return io_slot_file(node); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 60c947114fa3..3a535e9e8ac3 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1879,17 +1879,12 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd, struct file *file = NULL; io_ring_submit_lock(ctx, issue_flags); - - if (unlikely((unsigned int)fd >= ctx->nr_user_files)) - goto out; - fd = array_index_nospec(fd, ctx->nr_user_files); - node = ctx->file_table.nodes[fd]; + node = io_rsrc_node_lookup(&ctx->file_table.data, fd); if (node) { io_req_assign_rsrc_node(req, node); req->flags |= io_slot_flags(node); file = io_slot_file(node); } -out: io_ring_submit_unlock(ctx, issue_flags); return file; } diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index edea1ffd501c..b90ab3b8f5e0 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -180,8 +180,8 @@ static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_fl int idx = msg->src_fd; io_ring_submit_lock(ctx, issue_flags); - if (likely(idx < ctx->nr_user_files)) { - idx = array_index_nospec(idx, ctx->nr_user_files); + if (likely(idx < ctx->file_table.data.nr)) { + idx = array_index_nospec(idx, ctx->file_table.data.nr); file = io_file_from_index(&ctx->file_table, idx); if (file) get_file(file); diff --git a/io_uring/net.c b/io_uring/net.c index ce1156551d10..3e1f31574abb 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1347,9 +1347,9 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) ret = -EFAULT; io_ring_submit_lock(ctx, issue_flags); - if (sr->buf_index < ctx->nr_user_bufs) { - idx = array_index_nospec(sr->buf_index, ctx->nr_user_bufs); - node = ctx->user_bufs[idx]; + if (sr->buf_index < ctx->buf_table.nr) { + idx = array_index_nospec(sr->buf_index, ctx->buf_table.nr); + node = ctx->buf_table.nodes[idx]; io_req_assign_rsrc_node(sr->notif, node); ret = 0; } diff --git a/io_uring/nop.c b/io_uring/nop.c index de91600a3bc6..0dac01127de5 100644 --- a/io_uring/nop.c +++ b/io_uring/nop.c @@ -66,9 +66,9 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags) ret = -EFAULT; io_ring_submit_lock(ctx, issue_flags); - if (nop->buffer < ctx->nr_user_bufs) { - idx = array_index_nospec(nop->buffer, ctx->nr_user_bufs); - node = READ_ONCE(ctx->user_bufs[idx]); + if (nop->buffer < ctx->buf_table.nr) { + idx = array_index_nospec(nop->buffer, ctx->buf_table.nr); + node = READ_ONCE(ctx->buf_table.nodes[idx]); io_req_assign_rsrc_node(req, node); ret = 0; } diff --git a/io_uring/register.c b/io_uring/register.c index 1eb686eaa310..45edfc57963a 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -937,7 +937,8 @@ SYSCALL_DEFINE4(io_uring_register, unsigned int, fd, unsigned int, opcode, mutex_lock(&ctx->uring_lock); ret = __io_uring_register(ctx, opcode, arg, nr_args); mutex_unlock(&ctx->uring_lock); - trace_io_uring_register(ctx, opcode, ctx->nr_user_files, ctx->nr_user_bufs, ret); + trace_io_uring_register(ctx, opcode, ctx->file_table.data.nr, + ctx->buf_table.nr, ret); if (!use_registered_ring) fput(file); return ret; diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 08c57332c783..108c94428e7c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -147,39 +147,28 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, return node; } -static void io_rsrc_data_free(struct io_rsrc_data *data) +__cold void io_rsrc_data_free(struct io_rsrc_data *data) { - int i; - - for (i = 0; i < data->nr; i++) { - struct io_rsrc_node *node = data->nodes[i]; - - if (node) - io_put_rsrc_node(node); + if (!data->nr) + return; + while (data->nr--) { + if (data->nodes[data->nr]) + io_put_rsrc_node(data->nodes[data->nr]); } kvfree(data->nodes); - kfree(data); + data->nodes = NULL; + data->nr = 0; } -__cold static int io_rsrc_data_alloc(struct io_ring_ctx *ctx, unsigned nr, - struct io_rsrc_data **pdata) +__cold int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr) { - struct io_rsrc_data *data; - - data = kzalloc(sizeof(*data), GFP_KERNEL); - if (!data) - return -ENOMEM; - data->nodes = kvmalloc_array(nr, sizeof(struct io_rsrc_node *), - GFP_KERNEL | __GFP_ZERO); - if (!data->nodes) { - io_rsrc_data_free(data); - return -ENOMEM; + GFP_KERNEL_ACCOUNT | __GFP_ZERO); + if (data->nodes) { + data->nr = nr; + return 0; } - - data->nr = nr; - *pdata = data; - return 0; + return -ENOMEM; } static int __io_sqe_files_update(struct io_ring_ctx *ctx, @@ -191,9 +180,9 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, int fd, i, err = 0; unsigned int done; - if (!ctx->file_data) + if (!ctx->file_table.data.nr) return -ENXIO; - if (up->offset + nr_args > ctx->nr_user_files) + if (up->offset + nr_args > ctx->file_table.data.nr) return -EINVAL; for (done = 0; done < nr_args; done++) { @@ -211,10 +200,10 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, if (fd == IORING_REGISTER_FILES_SKIP) continue; - i = array_index_nospec(up->offset + done, ctx->nr_user_files); - if (ctx->file_table.nodes[i]) { - io_put_rsrc_node(ctx->file_table.nodes[i]); - ctx->file_table.nodes[i] = NULL; + i = array_index_nospec(up->offset + done, ctx->file_table.data.nr); + if (ctx->file_table.data.nodes[i]) { + io_put_rsrc_node(ctx->file_table.data.nodes[i]); + ctx->file_table.data.nodes[i] = NULL; io_file_bitmap_clear(&ctx->file_table, i); } if (fd != -1) { @@ -233,13 +222,14 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, err = -EBADF; break; } - node = io_rsrc_node_alloc(ctx, ctx->file_data, i, IORING_RSRC_FILE); + node = io_rsrc_node_alloc(ctx, &ctx->file_table.data, i, + IORING_RSRC_FILE); if (!node) { err = -ENOMEM; fput(file); break; } - ctx->file_table.nodes[i] = node; + ctx->file_table.data.nodes[i] = node; if (tag) node->tag = tag; io_fixed_file_set(node, file); @@ -261,9 +251,9 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, __u32 done; int i, err; - if (!ctx->buf_data) + if (!ctx->buf_table.nr) return -ENXIO; - if (up->offset + nr_args > ctx->nr_user_bufs) + if (up->offset + nr_args > ctx->buf_table.nr) return -EINVAL; for (done = 0; done < nr_args; done++) { @@ -287,16 +277,16 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx, err = -EINVAL; break; } - i = array_index_nospec(up->offset + done, ctx->nr_user_bufs); + i = array_index_nospec(up->offset + done, ctx->buf_table.nr); node = io_sqe_buffer_register(ctx, iov, i, &last_hpage); if (IS_ERR(node)) { err = PTR_ERR(node); break; } - if (ctx->user_bufs[i]) - io_put_rsrc_node(ctx->user_bufs[i]); + if (ctx->buf_table.nodes[i]) + io_put_rsrc_node(ctx->buf_table.nodes[i]); - ctx->user_bufs[i] = node; + ctx->buf_table.nodes[i] = node; if (tag) node->tag = tag; if (ctx->compat) @@ -414,7 +404,7 @@ static int io_files_update_with_index_alloc(struct io_kiocb *req, struct file *file; int ret, fd; - if (!req->ctx->file_data) + if (!req->ctx->file_table.data.nr) return -ENXIO; for (done = 0; done < up->nr_args; done++) { @@ -499,35 +489,13 @@ void io_free_rsrc_node(struct io_rsrc_node *node) kfree(node); } -static void __io_sqe_files_unregister(struct io_ring_ctx *ctx) -{ - int i; - - lockdep_assert_held(&ctx->uring_lock); - - for (i = 0; i < ctx->nr_user_files; i++) { - struct io_rsrc_node *node = ctx->file_table.nodes[i]; - - if (node) { - io_put_rsrc_node(node); - io_file_bitmap_clear(&ctx->file_table, i); - ctx->file_table.nodes[i] = NULL; - } - } - - io_free_file_tables(&ctx->file_table); - io_file_table_set_alloc_range(ctx, 0, 0); - io_rsrc_data_free(ctx->file_data); - ctx->file_data = NULL; - ctx->nr_user_files = 0; -} - int io_sqe_files_unregister(struct io_ring_ctx *ctx) { - if (!ctx->file_data) + if (!ctx->file_table.data.nr) return -ENXIO; - __io_sqe_files_unregister(ctx); + io_free_file_tables(&ctx->file_table); + io_file_table_set_alloc_range(ctx, 0, 0); return 0; } @@ -539,7 +507,7 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, int fd, ret; unsigned i; - if (ctx->file_data) + if (ctx->file_table.data.nr) return -EBUSY; if (!nr_args) return -EINVAL; @@ -547,17 +515,10 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, return -EMFILE; if (nr_args > rlimit(RLIMIT_NOFILE)) return -EMFILE; - ret = io_rsrc_data_alloc(ctx, nr_args, &ctx->file_data); - if (ret) - return ret; - - if (!io_alloc_file_tables(&ctx->file_table, nr_args)) { - io_rsrc_data_free(ctx->file_data); - ctx->file_data = NULL; + if (!io_alloc_file_tables(&ctx->file_table, nr_args)) return -ENOMEM; - } - for (i = 0; i < nr_args; i++, ctx->nr_user_files++) { + for (i = 0; i < nr_args; i++) { struct io_rsrc_node *node; u64 tag = 0; @@ -587,51 +548,32 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg, goto fail; } ret = -ENOMEM; - node = io_rsrc_node_alloc(ctx, ctx->file_data, i, IORING_RSRC_FILE); + node = io_rsrc_node_alloc(ctx, &ctx->file_table.data, i, + IORING_RSRC_FILE); if (!node) { fput(file); goto fail; } if (tag) node->tag = tag; - ctx->file_table.nodes[i] = node; + ctx->file_table.data.nodes[i] = node; io_fixed_file_set(node, file); io_file_bitmap_set(&ctx->file_table, i); } /* default it to the whole table */ - io_file_table_set_alloc_range(ctx, 0, ctx->nr_user_files); + io_file_table_set_alloc_range(ctx, 0, ctx->file_table.data.nr); return 0; fail: - __io_sqe_files_unregister(ctx); + io_sqe_files_unregister(ctx); return ret; } -static void __io_sqe_buffers_unregister(struct io_ring_ctx *ctx) -{ - unsigned int i; - - lockdep_assert_held(&ctx->uring_lock); - - for (i = 0; i < ctx->nr_user_bufs; i++) { - if (ctx->user_bufs[i]) { - io_put_rsrc_node(ctx->user_bufs[i]); - ctx->user_bufs[i] = NULL; - } - } - kvfree(ctx->user_bufs); - ctx->user_bufs = NULL; - io_rsrc_data_free(ctx->buf_data); - ctx->buf_data = NULL; - ctx->nr_user_bufs = 0; -} - int io_sqe_buffers_unregister(struct io_ring_ctx *ctx) { - if (!ctx->buf_data) + if (!ctx->buf_table.nr) return -ENXIO; - - __io_sqe_buffers_unregister(ctx); + io_rsrc_data_free(&ctx->buf_table); return 0; } @@ -658,8 +600,8 @@ static bool headpage_already_acct(struct io_ring_ctx *ctx, struct page **pages, } /* check previously registered pages */ - for (i = 0; i < ctx->nr_user_bufs; i++) { - struct io_rsrc_node *node = ctx->user_bufs[i]; + for (i = 0; i < ctx->buf_table.nr; i++) { + struct io_rsrc_node *node = ctx->buf_table.nodes[i]; struct io_mapped_ubuf *imu = node->buf; for (j = 0; j < imu->nr_bvecs; j++) { @@ -814,7 +756,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, if (!iov->iov_base) return rsrc_empty_node; - node = io_rsrc_node_alloc(ctx, ctx->buf_data, index, IORING_RSRC_BUFFER); + node = io_rsrc_node_alloc(ctx, &ctx->buf_table, index, IORING_RSRC_BUFFER); if (!node) return ERR_PTR(-ENOMEM); @@ -872,40 +814,29 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, return node; } -static int io_buffers_map_alloc(struct io_ring_ctx *ctx, unsigned int nr_args) -{ - ctx->user_bufs = kcalloc(nr_args, sizeof(*ctx->user_bufs), GFP_KERNEL); - return ctx->user_bufs ? 0 : -ENOMEM; -} - int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, unsigned int nr_args, u64 __user *tags) { struct page *last_hpage = NULL; - struct io_rsrc_data *data; + struct io_rsrc_data data; struct iovec fast_iov, *iov = &fast_iov; const struct iovec __user *uvec; int i, ret; BUILD_BUG_ON(IORING_MAX_REG_BUFFERS >= (1u << 16)); - if (ctx->user_bufs) + if (ctx->buf_table.nr) return -EBUSY; if (!nr_args || nr_args > IORING_MAX_REG_BUFFERS) return -EINVAL; - ret = io_rsrc_data_alloc(ctx, nr_args, &data); + ret = io_rsrc_data_alloc(&data, nr_args); if (ret) return ret; - ret = io_buffers_map_alloc(ctx, nr_args); - if (ret) { - io_rsrc_data_free(data); - return ret; - } if (!arg) memset(iov, 0, sizeof(*iov)); - for (i = 0; i < nr_args; i++, ctx->nr_user_bufs++) { + for (i = 0; i < nr_args; i++) { struct io_rsrc_node *node; u64 tag = 0; @@ -943,14 +874,12 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg, } if (tag) node->tag = tag; - ctx->user_bufs[i] = node; + data.nodes[i] = node; } - WARN_ON_ONCE(ctx->buf_data); - - ctx->buf_data = data; + ctx->buf_table = data; if (ret) - __io_sqe_buffers_unregister(ctx); + io_sqe_buffers_unregister(ctx); return ret; } @@ -1017,8 +946,7 @@ int io_import_fixed(int ddir, struct iov_iter *iter, static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx) { - struct io_rsrc_node **user_bufs; - struct io_rsrc_data *data; + struct io_rsrc_data data; int i, ret, nbufs; /* @@ -1029,43 +957,35 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx mutex_lock(&src_ctx->uring_lock); ret = -ENXIO; - nbufs = src_ctx->nr_user_bufs; + nbufs = src_ctx->buf_table.nr; if (!nbufs) goto out_unlock; - ret = io_rsrc_data_alloc(ctx, nbufs, &data); + ret = io_rsrc_data_alloc(&data, nbufs); if (ret) goto out_unlock; - ret = -ENOMEM; - user_bufs = kvmalloc_array(nbufs, sizeof(struct io_rsrc_node *), - GFP_KERNEL | __GFP_ZERO); - if (!user_bufs) - goto out_free_data; - for (i = 0; i < nbufs; i++) { - struct io_rsrc_node *src_node = src_ctx->user_bufs[i]; + struct io_rsrc_node *src_node = src_ctx->buf_table.nodes[i]; struct io_rsrc_node *dst_node; if (src_node == rsrc_empty_node) { dst_node = rsrc_empty_node; } else { - dst_node = io_rsrc_node_alloc(ctx, data, i, IORING_RSRC_BUFFER); + dst_node = io_rsrc_node_alloc(ctx, &data, i, IORING_RSRC_BUFFER); if (!dst_node) goto out_put_free; refcount_inc(&src_node->buf->refs); dst_node->buf = src_node->buf; } - user_bufs[i] = dst_node; + data.nodes[i] = dst_node; } /* Have a ref on the bufs now, drop src lock and re-grab our own lock */ mutex_unlock(&src_ctx->uring_lock); mutex_lock(&ctx->uring_lock); - if (!ctx->user_bufs) { - ctx->user_bufs = user_bufs; - ctx->buf_data = data; - ctx->nr_user_bufs = nbufs; + if (!ctx->buf_table.nr) { + ctx->buf_table = data; return 0; } @@ -1076,12 +996,10 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx i = nbufs; out_put_free: while (i--) { - io_buffer_unmap(src_ctx, user_bufs[i]); - kfree(user_bufs[i]); + io_buffer_unmap(src_ctx, data.nodes[i]); + kfree(data.nodes[i]); } - kvfree(user_bufs); -out_free_data: - io_rsrc_data_free(data); + io_rsrc_data_free(&data); out_unlock: mutex_unlock(&src_ctx->uring_lock); mutex_lock(&ctx->uring_lock); @@ -1102,7 +1020,7 @@ int io_register_clone_buffers(struct io_ring_ctx *ctx, void __user *arg) struct file *file; int ret; - if (ctx->user_bufs || ctx->nr_user_bufs) + if (ctx->buf_table.nr) return -EBUSY; if (copy_from_user(&buf, arg, sizeof(buf))) return -EFAULT; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 53907e142ae0..dcc1fb185035 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -14,11 +14,6 @@ enum { IORING_RSRC_INVALID = 2, }; -struct io_rsrc_data { - unsigned int nr; - struct io_rsrc_node **nodes; -}; - struct io_rsrc_node { struct io_ring_ctx *ctx; int refs; @@ -54,6 +49,8 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, struct io_rsrc_data *data, int index, int type); void io_free_rsrc_node(struct io_rsrc_node *node); +void io_rsrc_data_free(struct io_rsrc_data *data); +int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr); int io_import_fixed(int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, @@ -77,6 +74,16 @@ int io_register_rsrc(struct io_ring_ctx *ctx, void __user *arg, extern const struct io_rsrc_node empty_node; #define rsrc_empty_node (struct io_rsrc_node *) &empty_node +static inline struct io_rsrc_node *io_rsrc_node_lookup(struct io_rsrc_data *data, + int index) +{ + if (index < data->nr) { + index = array_index_nospec(index, data->nr); + return data->nodes[index]; + } + return NULL; +} + static inline void io_put_rsrc_node(struct io_rsrc_node *node) { if (node != &empty_node && !--node->refs) diff --git a/io_uring/rw.c b/io_uring/rw.c index 65491f4f2c7e..28fff18ebb19 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -339,10 +339,10 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe if (unlikely(ret)) return ret; - if (unlikely(req->buf_index >= ctx->nr_user_bufs)) + if (unlikely(req->buf_index >= ctx->buf_table.nr)) return -EFAULT; - index = array_index_nospec(req->buf_index, ctx->nr_user_bufs); - node = ctx->user_bufs[index]; + index = array_index_nospec(req->buf_index, ctx->buf_table.nr); + node = ctx->buf_table.nodes[index]; io_req_assign_rsrc_node(req, node); io = req->async_data; diff --git a/io_uring/splice.c b/io_uring/splice.c index f78afb575ae6..deeb8bb18651 100644 --- a/io_uring/splice.c +++ b/io_uring/splice.c @@ -66,17 +66,13 @@ static struct file *io_splice_get_file(struct io_kiocb *req, return io_file_get_normal(req, sp->splice_fd_in); io_ring_submit_lock(ctx, issue_flags); - if (unlikely(sp->splice_fd_in >= ctx->nr_user_files)) - goto out; - sp->splice_fd_in = array_index_nospec(sp->splice_fd_in, ctx->nr_user_files); - node = ctx->file_table.nodes[sp->splice_fd_in]; + node = io_rsrc_node_lookup(&ctx->file_table.data, sp->splice_fd_in); if (node) { node->refs++; sp->rsrc_node = node; file = io_slot_file(node); req->flags |= REQ_F_NEED_CLEANUP; } -out: io_ring_submit_unlock(ctx, issue_flags); return file; } diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 0899c71008ae..17d5f5004702 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -212,15 +212,15 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) u16 index; index = READ_ONCE(sqe->buf_index); - if (unlikely(index >= ctx->nr_user_bufs)) + if (unlikely(index >= ctx->buf_table.nr)) return -EFAULT; - req->buf_index = array_index_nospec(index, ctx->nr_user_bufs); + req->buf_index = array_index_nospec(index, ctx->buf_table.nr); /* * Pi node upfront, prior to io_uring_cmd_import_fixed() * being called. This prevents destruction of the mapped buffer * we'll need at actual import time. */ - io_req_assign_rsrc_node(req, ctx->user_bufs[req->buf_index]); + io_req_assign_rsrc_node(req, ctx->buf_table.nodes[req->buf_index]); } ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); From patchwork Mon Oct 28 14:52:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853656 Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 513B418FDB0 for ; Mon, 28 Oct 2024 15:04:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127900; cv=none; b=kszmi56N5hoqk4GcOYRgHX3lJs3QK1incT6CK97UlamvbIshhhgIB+SwSu9cA6s/vyaJ4NnjOJwjGQWNLfGuMUzmrYwOHPSaPA9fJbZTFdSkr49w6XpgJ+CNhux7KXoUpHfBGgYhbDHr+FnPY+Ohbjtk/HW7bieqymNqbSb4/3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127900; c=relaxed/simple; bh=Y/5ecBE1jL3xh3QKH+3W783u8h2aWPU7wFOADYD1rkw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WhDJUaTzDCuViEF3aTamPB7c4ghi3dF1aIZdkPMVamEOqDaSu49jxbvREX46iIOYhyuhDe9VDLI3oz95C5GKUTS8X5c1Fa8+tr83dAITAmQ9G7mdJFpPB7pX83JVXcWYBJxsvodlE5UlKOJYDyB9LzYw3vVv/52doZCtpDxc5MM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Gy+8Sl/p; arc=none smtp.client-ip=209.85.166.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Gy+8Sl/p" Received: by mail-io1-f54.google.com with SMTP id ca18e2360f4ac-83ac05206f6so172907839f.2 for ; Mon, 28 Oct 2024 08:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127896; x=1730732696; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RwY82xqK1igH3RTsjaPKoZuhmLqEqFBjzQh0nVw9IyA=; b=Gy+8Sl/pKu6x/2KSDu6MPecssJwbXSVynx9BKXojj0bW3GRbrr6xSUTqJq6YlCuXp2 Kru6T6FIH6GVnM1zCXPourhgJx5AKgw4gza6MLplgn6vyXLL983143YVyMa36/qMePs8 MqpP3S2l8q/bQZzyS+9Cnnp7i3i9f3zIBkGPYGPYgyI7MCQJBSNWGs2nlhKRntjR94GM EiiA176OxTz/qrLGx4+zdTv3koVSssg076S6awnRS0cLEss16v2r7okt8gQH9MRikP7p GSRyQBDws1eX4AI7p4nep22YjeBNgMh4XUsiEIM15ickm35Axh/AQ/h8iHOeWgJGKO9+ 63Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127896; x=1730732696; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RwY82xqK1igH3RTsjaPKoZuhmLqEqFBjzQh0nVw9IyA=; b=AqFbqnACT97o2l4qCXbUzuT7vjxF6jzRViLZnCoxbMOBvz/G18kbJUR3WM+gReY6zl MB6xxwNFAokyUAd4tjPJdf44abAxpm7BZW+KGDAbajBz25FCLeetDuHb0ZJnIDTzMUdH T1NPx9K6iBL2GX2qIu9q20X+MD5U9MvOwK5BoStemDdJRKv+HqgT2lSKptS2zJg9HwVm KJgjnTpamPoTqX+PpkWIceFR6uqrVZwqlIL6ZMOBL+bzV2Rtx/0iAYUgakfyUxdhocnD UuYfeLX89QmGwk5rlLyuvLiUilNvoe2p1YMQ/B3eOfVIC0KYj9tOCdaQFWCsXMISK6sD 1vvw== X-Gm-Message-State: AOJu0Yzv5w6/oa70v/yMWvYjVYnYV5x/a5XYGQenmQIX2GZgr4KDh1nI UwGboswjlthyJGRe+AVVLhdk77uHb7gNVH1RcQK/tbCeh9FGuY6xe+VEqnaHcxKBnXsqpMHm2+8 h X-Google-Smtp-Source: AGHT+IFYzwXOqYQk3hyoLvd3o/mb3AnZXSTR8b9Ir0GT5mOjdVxfRIFBEaae/Y57sBBoOxCfh4EM7g== X-Received: by 2002:a05:6e02:16c7:b0:3a0:8edc:d133 with SMTP id e9e14a558f8ab-3a4ed28f225mr69881875ab.9.1730127895959; Mon, 28 Oct 2024 08:04:55 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:55 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 11/13] io_uring/rsrc: use io_rsrc_node_lookup() consistently Date: Mon, 28 Oct 2024 08:52:41 -0600 Message-ID: <20241028150437.387667-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Various spots open code this helper, use the provided one. The helper now returns the adjusted, if needed, index into the array, for the cases where the caller needs that. Signed-off-by: Jens Axboe --- io_uring/cancel.c | 8 +++++--- io_uring/filetable.c | 24 +++++++++++++----------- io_uring/filetable.h | 9 +++++---- io_uring/io_uring.c | 2 +- io_uring/msg_ring.c | 33 ++++++++++++++++----------------- io_uring/net.c | 7 +++---- io_uring/nop.c | 6 ++---- io_uring/rsrc.c | 15 +++++++++------ io_uring/rsrc.h | 8 ++++---- io_uring/rw.c | 9 ++++----- io_uring/splice.c | 2 +- io_uring/uring_cmd.c | 10 +++++----- 12 files changed, 68 insertions(+), 65 deletions(-) diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 3a2996307025..cc2cd16832ef 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -240,10 +240,12 @@ static int __io_sync_cancel(struct io_uring_task *tctx, /* fixed must be grabbed every time since we drop the uring_lock */ if ((cd->flags & IORING_ASYNC_CANCEL_FD) && (cd->flags & IORING_ASYNC_CANCEL_FD_FIXED)) { - if (unlikely(fd >= ctx->file_table.data.nr)) + struct io_rsrc_node *node; + + node = io_rsrc_node_lookup(&ctx->file_table.data, &fd); + if (unlikely(!node)) return -EBADF; - fd = array_index_nospec(fd, ctx->file_table.data.nr); - cd->file = io_file_from_index(&ctx->file_table, fd); + cd->file = io_slot_file(node); if (!cd->file) return -EBADF; } diff --git a/io_uring/filetable.c b/io_uring/filetable.c index c4c31f8b4f5d..7b6e4df7cef9 100644 --- a/io_uring/filetable.c +++ b/io_uring/filetable.c @@ -55,10 +55,10 @@ void io_free_file_tables(struct io_file_table *table) } static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, - u32 slot_index) + int slot_index) __must_hold(&req->ctx->uring_lock) { - struct io_rsrc_node *node; + struct io_rsrc_node *node, *old_node; if (io_is_uring_fops(file)) return -EBADF; @@ -72,9 +72,9 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, if (IS_ERR(node)) return -ENOMEM; - slot_index = array_index_nospec(slot_index, ctx->file_table.data.nr); - if (ctx->file_table.data.nodes[slot_index]) - io_put_rsrc_node(ctx->file_table.data.nodes[slot_index]); + old_node = io_rsrc_node_lookup(&ctx->file_table.data, &slot_index); + if (old_node) + io_put_rsrc_node(old_node); else io_file_bitmap_set(&ctx->file_table, slot_index); @@ -84,7 +84,7 @@ static int io_install_fixed_file(struct io_ring_ctx *ctx, struct file *file, } int __io_fixed_fd_install(struct io_ring_ctx *ctx, struct file *file, - unsigned int file_slot) + int file_slot) { bool alloc_slot = file_slot == IORING_FILE_INDEX_ALLOC; int ret; @@ -108,7 +108,7 @@ int __io_fixed_fd_install(struct io_ring_ctx *ctx, struct file *file, * fput() is called correspondingly. */ int io_fixed_fd_install(struct io_kiocb *req, unsigned int issue_flags, - struct file *file, unsigned int file_slot) + struct file *file, int file_slot) { struct io_ring_ctx *ctx = req->ctx; int ret; @@ -122,17 +122,19 @@ int io_fixed_fd_install(struct io_kiocb *req, unsigned int issue_flags, return ret; } -int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset) +int io_fixed_fd_remove(struct io_ring_ctx *ctx, int offset) { + struct io_rsrc_node *node; + if (unlikely(!ctx->file_table.data.nr)) return -ENXIO; if (offset >= ctx->file_table.data.nr) return -EINVAL; - offset = array_index_nospec(offset, ctx->file_table.data.nr); - if (!ctx->file_table.data.nodes[offset]) + node = io_rsrc_node_lookup(&ctx->file_table.data, &offset); + if (!node) return -EBADF; - io_put_rsrc_node(ctx->file_table.data.nodes[offset]); + io_put_rsrc_node(node); ctx->file_table.data.nodes[offset] = NULL; io_file_bitmap_clear(&ctx->file_table, offset); return 0; diff --git a/io_uring/filetable.h b/io_uring/filetable.h index 664c31502dbb..f68ffc24061c 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -10,10 +10,10 @@ bool io_alloc_file_tables(struct io_file_table *table, unsigned nr_files); void io_free_file_tables(struct io_file_table *table); int io_fixed_fd_install(struct io_kiocb *req, unsigned int issue_flags, - struct file *file, unsigned int file_slot); + struct file *file, int file_slot); int __io_fixed_fd_install(struct io_ring_ctx *ctx, struct file *file, - unsigned int file_slot); -int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset); + int file_slot); +int io_fixed_fd_remove(struct io_ring_ctx *ctx, int offset); int io_register_file_alloc_range(struct io_ring_ctx *ctx, struct io_uring_file_index_range __user *arg); @@ -52,8 +52,9 @@ static inline struct file *io_slot_file(struct io_rsrc_node *node) static inline struct file *io_file_from_index(struct io_file_table *table, int index) { - struct io_rsrc_node *node = table->data.nodes[index]; + struct io_rsrc_node *node; + node = io_rsrc_node_lookup(&table->data, &index); if (node) return io_slot_file(node); return NULL; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 3a535e9e8ac3..4514644fdf52 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1879,7 +1879,7 @@ inline struct file *io_file_get_fixed(struct io_kiocb *req, int fd, struct file *file = NULL; io_ring_submit_lock(ctx, issue_flags); - node = io_rsrc_node_lookup(&ctx->file_table.data, fd); + node = io_rsrc_node_lookup(&ctx->file_table.data, &fd); if (node) { io_req_assign_rsrc_node(req, node); req->flags |= io_slot_flags(node); diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index b90ab3b8f5e0..8e783aa75064 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -25,7 +25,7 @@ struct io_msg { u64 user_data; u32 len; u32 cmd; - u32 src_fd; + int src_fd; union { u32 dst_fd; u32 cqe_flags; @@ -172,22 +172,24 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) return __io_msg_ring_data(target_ctx, msg, issue_flags); } -static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flags) +static int io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flags) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); struct io_ring_ctx *ctx = req->ctx; - struct file *file = NULL; - int idx = msg->src_fd; + struct io_rsrc_node *node; + int ret = -EBADF; io_ring_submit_lock(ctx, issue_flags); - if (likely(idx < ctx->file_table.data.nr)) { - idx = array_index_nospec(idx, ctx->file_table.data.nr); - file = io_file_from_index(&ctx->file_table, idx); - if (file) - get_file(file); + node = io_rsrc_node_lookup(&ctx->file_table.data, &msg->src_fd); + if (node) { + msg->src_file = io_slot_file(node); + if (msg->src_file) + get_file(msg->src_file); + req->flags |= REQ_F_NEED_CLEANUP; + ret = 0; } io_ring_submit_unlock(ctx, issue_flags); - return file; + return ret; } static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flags) @@ -256,7 +258,6 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); struct io_ring_ctx *ctx = req->ctx; - struct file *src_file = msg->src_file; if (msg->len) return -EINVAL; @@ -264,12 +265,10 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) return -EINVAL; if (target_ctx->flags & IORING_SETUP_R_DISABLED) return -EBADFD; - if (!src_file) { - src_file = io_msg_grab_file(req, issue_flags); - if (!src_file) - return -EBADF; - msg->src_file = src_file; - req->flags |= REQ_F_NEED_CLEANUP; + if (!msg->src_file) { + int ret = io_msg_grab_file(req, issue_flags); + if (unlikely(ret)) + return ret; } if (io_msg_need_remote(target_ctx)) diff --git a/io_uring/net.c b/io_uring/net.c index 3e1f31574abb..87b9ae07d647 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1343,13 +1343,12 @@ static int io_send_zc_import(struct io_kiocb *req, unsigned int issue_flags) if (sr->flags & IORING_RECVSEND_FIXED_BUF) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *node; - int idx; + int idx = sr->buf_index; ret = -EFAULT; io_ring_submit_lock(ctx, issue_flags); - if (sr->buf_index < ctx->buf_table.nr) { - idx = array_index_nospec(sr->buf_index, ctx->buf_table.nr); - node = ctx->buf_table.nodes[idx]; + node = io_rsrc_node_lookup(&ctx->buf_table, &idx); + if (node) { io_req_assign_rsrc_node(sr->notif, node); ret = 0; } diff --git a/io_uring/nop.c b/io_uring/nop.c index 0dac01127de5..97150c0e6c25 100644 --- a/io_uring/nop.c +++ b/io_uring/nop.c @@ -62,13 +62,11 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags) if (nop->flags & IORING_NOP_FIXED_BUFFER) { struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *node; - int idx; ret = -EFAULT; io_ring_submit_lock(ctx, issue_flags); - if (nop->buffer < ctx->buf_table.nr) { - idx = array_index_nospec(nop->buffer, ctx->buf_table.nr); - node = READ_ONCE(ctx->buf_table.nodes[idx]); + node = io_rsrc_node_lookup(&ctx->buf_table, &nop->buffer); + if (node) { io_req_assign_rsrc_node(req, node); ret = 0; } diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 108c94428e7c..c3251a80d4c9 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -186,6 +186,7 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, return -EINVAL; for (done = 0; done < nr_args; done++) { + struct io_rsrc_node *node; u64 tag = 0; if ((tags && copy_from_user(&tag, &tags[done], sizeof(tag))) || @@ -200,9 +201,10 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx, if (fd == IORING_REGISTER_FILES_SKIP) continue; - i = array_index_nospec(up->offset + done, ctx->file_table.data.nr); - if (ctx->file_table.data.nodes[i]) { - io_put_rsrc_node(ctx->file_table.data.nodes[i]); + i = up->offset + done; + node = io_rsrc_node_lookup(&ctx->file_table.data, &i); + if (node) { + io_put_rsrc_node(node); ctx->file_table.data.nodes[i] = NULL; io_file_bitmap_clear(&ctx->file_table, i); } @@ -965,13 +967,14 @@ static int io_clone_buffers(struct io_ring_ctx *ctx, struct io_ring_ctx *src_ctx goto out_unlock; for (i = 0; i < nbufs; i++) { - struct io_rsrc_node *src_node = src_ctx->buf_table.nodes[i]; - struct io_rsrc_node *dst_node; + struct io_rsrc_node *dst_node, *src_node; + int index = i; + src_node = io_rsrc_node_lookup(&src_ctx->buf_table, &index); if (src_node == rsrc_empty_node) { dst_node = rsrc_empty_node; } else { - dst_node = io_rsrc_node_alloc(ctx, &data, i, IORING_RSRC_BUFFER); + dst_node = io_rsrc_node_alloc(ctx, &data, index, IORING_RSRC_BUFFER); if (!dst_node) goto out_put_free; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index dcc1fb185035..5897306bcc35 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -75,11 +75,11 @@ extern const struct io_rsrc_node empty_node; #define rsrc_empty_node (struct io_rsrc_node *) &empty_node static inline struct io_rsrc_node *io_rsrc_node_lookup(struct io_rsrc_data *data, - int index) + int *index) { - if (index < data->nr) { - index = array_index_nospec(index, data->nr); - return data->nodes[index]; + if (*index < data->nr) { + *index = array_index_nospec(*index, data->nr); + return data->nodes[*index]; } return NULL; } diff --git a/io_uring/rw.c b/io_uring/rw.c index 28fff18ebb19..7ce1cbc048fa 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -332,17 +332,16 @@ static int io_prep_rw_fixed(struct io_kiocb *req, const struct io_uring_sqe *sqe struct io_ring_ctx *ctx = req->ctx; struct io_rsrc_node *node; struct io_async_rw *io; - u16 index; - int ret; + int index, ret; ret = io_prep_rw(req, sqe, ddir, false); if (unlikely(ret)) return ret; - if (unlikely(req->buf_index >= ctx->buf_table.nr)) + index = req->buf_index; + node = io_rsrc_node_lookup(&ctx->buf_table, &index); + if (!node) return -EFAULT; - index = array_index_nospec(req->buf_index, ctx->buf_table.nr); - node = ctx->buf_table.nodes[index]; io_req_assign_rsrc_node(req, node); io = req->async_data; diff --git a/io_uring/splice.c b/io_uring/splice.c index deeb8bb18651..dfc84896da8a 100644 --- a/io_uring/splice.c +++ b/io_uring/splice.c @@ -66,7 +66,7 @@ static struct file *io_splice_get_file(struct io_kiocb *req, return io_file_get_normal(req, sp->splice_fd_in); io_ring_submit_lock(ctx, issue_flags); - node = io_rsrc_node_lookup(&ctx->file_table.data, sp->splice_fd_in); + node = io_rsrc_node_lookup(&ctx->file_table.data, &sp->splice_fd_in); if (node) { node->refs++; sp->rsrc_node = node; diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 17d5f5004702..6bbf2b603765 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -209,18 +209,18 @@ int io_uring_cmd_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (ioucmd->flags & IORING_URING_CMD_FIXED) { struct io_ring_ctx *ctx = req->ctx; - u16 index; + int index = req->buf_index; + struct io_rsrc_node *node; - index = READ_ONCE(sqe->buf_index); - if (unlikely(index >= ctx->buf_table.nr)) + node = io_rsrc_node_lookup(&ctx->buf_table, &index); + if (unlikely(!node)) return -EFAULT; - req->buf_index = array_index_nospec(index, ctx->buf_table.nr); /* * Pi node upfront, prior to io_uring_cmd_import_fixed() * being called. This prevents destruction of the mapped buffer * we'll need at actual import time. */ - io_req_assign_rsrc_node(req, ctx->buf_table.nodes[req->buf_index]); + io_req_assign_rsrc_node(req, node); } ioucmd->cmd_op = READ_ONCE(sqe->cmd_op); From patchwork Mon Oct 28 14:52:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853658 Received: from mail-il1-f169.google.com (mail-il1-f169.google.com [209.85.166.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BDA31DE3AC for ; Mon, 28 Oct 2024 15:04:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127901; cv=none; b=LuEFCNty5rE3gvHByhIVCpC7qsnIUgsqH5nRjdWt/lIKYdXXQh1knayiQCqaFoSbgA7RcVi5bmrOxl6FtOgrlYV/D9NKEp6bF3w001MNVwhpZzIBU09Ly1LIm0VPeZOQYdh7GpDrU8Jmmjey7wqZqu9ZbBxeFRH89Y3orKJQhGE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127901; c=relaxed/simple; bh=pkYcWRssBfizJinpMOsKyjVzc8YHB6Nvuut57LJm0pg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ArZlp8AujCEaf4KPENOn+UDx8D6YnwdFgOkcHgXYFb/Yz9NYnQxPfmomQksmdom1BNy9nGmCmfjhEv9Zk1nRP9BHjswH37e628ceNcYUUFE6wEQgVryn3yf8JA+QrbB/daWeMFMZjQY+jyxYQ2m+L8cg5ymFf4gO83/S5qOIR0k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=fn3FPivD; arc=none smtp.client-ip=209.85.166.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="fn3FPivD" Received: by mail-il1-f169.google.com with SMTP id e9e14a558f8ab-3a3a6abcbf1so17541585ab.2 for ; Mon, 28 Oct 2024 08:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127897; x=1730732697; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=R4ABfE88V2XiWFzjVdtoTBBUyKwlbamhOB0YbTSimnQ=; b=fn3FPivDrWM85jQvt7yBpxtSFsnQpRC6WG4YBiA8td4bVnxfdo4FfE3lEmEm3kmTlF Qk6NXNuj1UJpdMDkIACfYuIrnh0W0wNJSi9ya3EOUmyhnH1iks4gFz7VEgYmCkyHyRx/ vGCxbF2ICntv/3K+YjH2rBbbOGoiiilWMneoEKITXtvGOSyqtsyE42GV/o+crA5LXPeL NS67/p0Qsn+nkQ122UZa/pHHbLXFk3cc/r/SAut++Y3IYk+nxExbHJyG/7a1DBOkPewd z7+yciPNZQwbHhmfYnIuNcmgW6ohC4+WaE6NA7JS+9IAseM/iUk6IO6WKNHJ68R19q+/ AHww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127897; x=1730732697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R4ABfE88V2XiWFzjVdtoTBBUyKwlbamhOB0YbTSimnQ=; b=OCyLiuXWPUj2TepkSIpos0+I9gHO1xklwOElL4q+DMG9a8WGeaxcIDiF/mG28q9P33 il/yWeumSm7TLZzo0gjNZkr8WZemksW5pb77UMwcNboN35JsD5z6gsfoJ3ZfbZMMG9g7 S2gGb/gldBIxYDLPo5rI3y+V3Khes6YCCWhGefawI3o8/kANXtMxYvVLtZnhgFQIUe79 Ob9rnaziv0xvQFKFaPMb9gR44/JOofNjL/UVSl9KwYpRxq6HFkgsBOOWZSYW9AwZP0dV G10glM7eWo+2/RMucMT/6iFazNYHesYlZt+kgxnUw1LU9HPPkqUxfjX9/0yjSrp0Y1Pu CtYQ== X-Gm-Message-State: AOJu0YzwYFv4sLsq3DFkjQxfEtdldo1z1iNeyRKdD9UYo5j3N7997/Bm FxPrsne1C/zVo+5sJHjH3yiioTrwv6gul5qFmWhTVI0zREFKkdli7vwESHqojGxL7abH6kzkIA4 Q X-Google-Smtp-Source: AGHT+IEZ4HRSR4Vtbj31I0C/aubFKUmJNXF2IXtNxrqztTPR+46jqMcsgMSlm7ZcBxoIp3ZVralPrA== X-Received: by 2002:a05:6e02:1d85:b0:3a3:445d:f711 with SMTP id e9e14a558f8ab-3a4ed1c2ea6mr83401225ab.0.1730127897604; Mon, 28 Oct 2024 08:04:57 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:56 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 12/13] io_uring/filetable: remove io_file_from_index() helper Date: Mon, 28 Oct 2024 08:52:42 -0600 Message-ID: <20241028150437.387667-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's only used in fdinfo, nothing really gained from having this helper. Signed-off-by: Jens Axboe --- io_uring/fdinfo.c | 4 +++- io_uring/filetable.h | 11 ----------- 2 files changed, 3 insertions(+), 12 deletions(-) diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index e3f5e9fe5562..9d96481e2eb6 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -167,8 +167,10 @@ __cold void io_uring_show_fdinfo(struct seq_file *m, struct file *file) seq_printf(m, "SqWorkTime:\t%llu\n", sq_work_time); seq_printf(m, "UserFiles:\t%u\n", ctx->file_table.data.nr); for (i = 0; has_lock && i < ctx->file_table.data.nr; i++) { - struct file *f = io_file_from_index(&ctx->file_table, i); + struct file *f = NULL; + if (ctx->file_table.data.nodes[i]) + f = io_slot_file(ctx->file_table.data.nodes[i]); if (f) seq_printf(m, "%5u: %s\n", i, file_dentry(f)->d_iname); else diff --git a/io_uring/filetable.h b/io_uring/filetable.h index f68ffc24061c..782d28d66245 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -49,17 +49,6 @@ static inline struct file *io_slot_file(struct io_rsrc_node *node) return (struct file *)(node->file_ptr & FFS_MASK); } -static inline struct file *io_file_from_index(struct io_file_table *table, - int index) -{ - struct io_rsrc_node *node; - - node = io_rsrc_node_lookup(&table->data, &index); - if (node) - return io_slot_file(node); - return NULL; -} - static inline void io_fixed_file_set(struct io_rsrc_node *node, struct file *file) { From patchwork Mon Oct 28 14:52:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13853659 Received: from mail-il1-f172.google.com (mail-il1-f172.google.com [209.85.166.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C93AE154433 for ; Mon, 28 Oct 2024 15:04:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127901; cv=none; b=WKQ3bJ1E4zKXslJQEhExEKrXbPpyloOxJ9GWUIGAop0O1aIZCyDUuT2EmLMGb0nzQKVteAShPyvXPiSCQhw62JsumvW1ofJZYCUtAmirqaXEieqwHXFxOhV7Z9N+twnx2Au/hLWFVLAyEr53T9X/wjvnlsOqOJ11ScpPsN9jcaU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730127901; c=relaxed/simple; bh=VfbDmtv5aW5PNQ30l++4tRQtbgzBFwi3W06NW3vdzIo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Tr4TAg/LFJ2xYNOe+gExgjWme5UUXpiosZE/BxksRHxlWVtmjCul51GYtP+hZPu5TYm/NfDo2gKnqu3EntcmCh9xqcCHjN2x7SKq4Ngq7YepEXghlFIbRp2L6YhFh+E8rNndbtx4/nPHgd7XUr/teaSp9VzOnfBJPsYQ6/3Xt2Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=j+wn2+Q8; arc=none smtp.client-ip=209.85.166.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="j+wn2+Q8" Received: by mail-il1-f172.google.com with SMTP id e9e14a558f8ab-3a3aeb19ea2so16001245ab.0 for ; Mon, 28 Oct 2024 08:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1730127898; x=1730732698; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z/2dKOrmPK4w2j6buyCUdzsOQjVIHjTxGl9nrPXboIs=; b=j+wn2+Q8cUYz3RoVcsyWtNNRgpBXkfdN84NLcynPgun20VWV1BvTbVaN3d2yMdFZlK Bom57CvsuElMj8nwWAAwqgxxv56CWm5TxprPm9/cOWB0B9g7sL3bYvK8VIhjYyGTLk7i i/5HxkwWezt9Pj9Tevhqy56k31UzJkEifDq9mEDuyTX26+5fk4VotZsYf4bq9oiUY8At NF5NbLdg/AHjsf6FDPJOZr2fyi2OOCD+zRjaqaKuBV+GC3UHGAdtgdm4PvWcOGzV1kf9 X153RkL8GIVfCfoi/5mOHR7mbWF9V8hfCO/JMOTZYf8MKH0k4Wh9z46GUGVwwWLnM7+X j1lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730127898; x=1730732698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z/2dKOrmPK4w2j6buyCUdzsOQjVIHjTxGl9nrPXboIs=; b=TwiA+u4lL8PtlVWwPNj5c9cV/Fko+FmK/M7E8Z0o+JrjRYQUFOjOWFtQs8FgGO8FRo r5Zg/+TW0fVp3vU0sWrSvT7i92HjikWruJ4ZyBZ2vEFk+eRNS2KyKeLEZdTy0XtfwbOR tigv2gFRkdQ0UDfnBZLgwCCkx30gsmBHitsv57+gFypPPh4PTNYA2kzcwDap9E1MjSu0 PaiFQZpauwOSwDBq4j3vw+1FdEj5GO3VSSqa12TH+1d8UKoi6MbQanl8825tpbHjQ2cN 6L/FY5wbVtUw1fXM3NXwB7EkJNXH/DsjSueRDkDXOj83cXrBuuAr4ZxK6nLgcNlzhht7 Hifw== X-Gm-Message-State: AOJu0Yykb66vogWlE0YFTEZ6lwioQypzV806ZULqDup9+eFiE0ZsTsS/ BG1sR7/mKI4o95vsfp8fv7zIj2JpasVxEtdB9iiu15qvnFnE4imsfx+OCwB+7PYLhnh281/nuoa + X-Google-Smtp-Source: AGHT+IHbkpyQleAXSnTQthfZWkj0Tejcsba4wqO5vJXrous1OtUEisMD2MhHlsCzn5xHT6j0fmz7hQ== X-Received: by 2002:a05:6e02:1568:b0:3a1:a163:ba64 with SMTP id e9e14a558f8ab-3a4ed264feemr75401085ab.3.1730127898327; Mon, 28 Oct 2024 08:04:58 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4dc7261c7e3sm1721616173.72.2024.10.28.08.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Oct 2024 08:04:57 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 13/13] io_uring/filetable: kill io_reset_alloc_hint() helper Date: Mon, 28 Oct 2024 08:52:43 -0600 Message-ID: <20241028150437.387667-14-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241028150437.387667-1-axboe@kernel.dk> References: <20241028150437.387667-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 It's only used internally, and in one spot, just open-code ti. Signed-off-by: Jens Axboe --- io_uring/filetable.h | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/io_uring/filetable.h b/io_uring/filetable.h index 782d28d66245..2a30e0db94c0 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -56,17 +56,12 @@ static inline void io_fixed_file_set(struct io_rsrc_node *node, (io_file_get_flags(file) >> REQ_F_SUPPORT_NOWAIT_BIT); } -static inline void io_reset_alloc_hint(struct io_ring_ctx *ctx) -{ - ctx->file_table.alloc_hint = ctx->file_alloc_start; -} - static inline void io_file_table_set_alloc_range(struct io_ring_ctx *ctx, unsigned off, unsigned len) { ctx->file_alloc_start = off; ctx->file_alloc_end = off + len; - io_reset_alloc_hint(ctx); + ctx->file_table.alloc_hint = ctx->file_alloc_start; } #endif