From patchwork Wed Feb 7 17:17:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13548759 Received: from mail-il1-f179.google.com (mail-il1-f179.google.com [209.85.166.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A769823B8 for ; Wed, 7 Feb 2024 17:19:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326388; cv=none; b=E3ivWpAY4my/689uJKxOsigyzTEfkocA4XQwyDyJbw9Rgmn5dbBZrR48jSliwEPjlQ8tgc9r+3ynS/Uy8sLnadnPwxIBaI+eHTGjzHCtM+8OPv5WbIsn4BTZWUBKtXa2HSHAjVrjyMaKB3gP5401Fu9anJKRU6+1UBtP7brh1zg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326388; c=relaxed/simple; bh=48Gx8/Jp8WOdUwoot/MAsP5S47K9J5qjBMZcBNaSZWg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QxtCYiQZ128sLI/t/lV9+vIa+ixp0CK8eeNG3xofs/nyv7XluwglL080uIKI6PUrGA3NxkwfUa78JXNjy55ISQWaRW296AmjeB8090BtrLyb+O46hHBTB0/npymhvX0+TuupxmcfE4wfl5SSAjpHSINvej/K3mrHIPCc1jIX6ws= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=TFqIJnYH; arc=none smtp.client-ip=209.85.166.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="TFqIJnYH" Received: by mail-il1-f179.google.com with SMTP id e9e14a558f8ab-361a8f20e22so635555ab.0 for ; Wed, 07 Feb 2024 09:19:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707326385; x=1707931185; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FKB6xz72wVSOfpvPNPF4mIDyX2PnEOtIciWC2C6rd4Q=; b=TFqIJnYHCv4tlBG80AU3T7AKyshyqXTrQfnoy2yDB0DIEtlns3JENG2P2O5/U6ecjz PZHO8Mbpj+AO4kcNAHUTU53xVFDNFUWr6/2+7+NY18yoi9F1PO4bk1Xq/ROAOie/0kuo ISj3faoQfkdXQVPlzwaYjKS3FZ/yVeZFS7+GifdpATpRPumA+J85rNvG4S0jR+0dX/jH t239H7zm+z8gxktSYTsikEl2BbNT2RdQ5VjFvHEPIsHeDj/847ceygg47GhvTZ5zCS7X Xc0wCH+th0xKYY4IqxW7NMeH0HCei/WMrXnNGKTEUs8Ld9x7zqIZbWHzwi3Zj+fYajOe 8EnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707326385; x=1707931185; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FKB6xz72wVSOfpvPNPF4mIDyX2PnEOtIciWC2C6rd4Q=; b=UtaTH0QErVK/6wBhkOJJLdUNlXiX/eNfeWlC/HQcVcbiYnIC/gSW+ReyWOradkhA8N x7FnnrrBC4YzGbdJ0Mj1Q3AmWkEVPA14wjyfOqG6A4tZrL4BaobrvIpCf9XR3W8Bu/S0 +13P8jB0TiHPVmlOuNxlYSu1pqxM4aiRFQmL5HeX2LmfMhU3273t1O0y1oXnD/Of3+M8 dPvZTKQAW6ptTwo3mPdXn8RI0N7hEbxoKbbtoQqcUvhi/SfeZJXcdKfHLmBDmMDpmUe/ woquhmAh6zRjhwXqubpucRJhc/+d0AuZ3gBIMCP/FLQTz8aO5GJ2S9e1CYYObnGKeuNw 89wQ== X-Gm-Message-State: AOJu0YzKNqh/QsVbiyosi1K+oeTj8YF7/ZdB8yDOdcKkwz2+qbfjzPsD KJgoTRGu7UWe3TjaPQLh7CH1NLx4xbsgoPy8+ibrrMUckLTAzgxdp8ue/GAyXD9xsycMq071ABQ VaKw= X-Google-Smtp-Source: AGHT+IHAqgQycM8yIy9Gu20gr+5BC3KJssjmO1JDIZG9dvxVVKbyvE25xJH7BnAy53NRquwVUZhRyQ== X-Received: by 2002:a6b:dc16:0:b0:7c3:f955:ada6 with SMTP id s22-20020a6bdc16000000b007c3f955ada6mr4059750ioc.1.1707326384892; Wed, 07 Feb 2024 09:19:44 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id g22-20020a6b7616000000b007bc4622d199sm421131iom.22.2024.02.07.09.19.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 09:19:43 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/6] io_uring: expand main struct io_kiocb flags to 64-bits Date: Wed, 7 Feb 2024 10:17:35 -0700 Message-ID: <20240207171941.1091453-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207171941.1091453-1-axboe@kernel.dk> References: <20240207171941.1091453-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We're out of space here, and none of the flags are easily reclaimable. Bump it to 64-bits and re-arrange the struct a bit to avoid gaps. Add a specific bitwise type for the request flags, io_request_flags_t. This will help catch violations of casting this value to a smaller type on 32-bit archs, like unsigned int. This creates a hole in the io_kiocb, so move nr_tw up and rsrc_node down to retain needing only cacheline 0 and 1 for non-polled opcodes. No functional changes intended in this patch. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 77 ++++++++++++++++++--------------- include/trace/events/io_uring.h | 14 +++--- io_uring/filetable.h | 2 +- io_uring/io_uring.c | 9 ++-- 4 files changed, 55 insertions(+), 47 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 854ad67a5f70..56bf733d3ee6 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -468,70 +468,73 @@ enum { __REQ_F_LAST_BIT, }; +typedef u64 __bitwise io_req_flags_t; +#define IO_REQ_FLAG(bitno) ((__force io_req_flags_t) BIT_ULL((bitno))) + enum { /* ctx owns file */ - REQ_F_FIXED_FILE = BIT(REQ_F_FIXED_FILE_BIT), + REQ_F_FIXED_FILE = IO_REQ_FLAG(REQ_F_FIXED_FILE_BIT), /* drain existing IO first */ - REQ_F_IO_DRAIN = BIT(REQ_F_IO_DRAIN_BIT), + REQ_F_IO_DRAIN = IO_REQ_FLAG(REQ_F_IO_DRAIN_BIT), /* linked sqes */ - REQ_F_LINK = BIT(REQ_F_LINK_BIT), + REQ_F_LINK = IO_REQ_FLAG(REQ_F_LINK_BIT), /* doesn't sever on completion < 0 */ - REQ_F_HARDLINK = BIT(REQ_F_HARDLINK_BIT), + REQ_F_HARDLINK = IO_REQ_FLAG(REQ_F_HARDLINK_BIT), /* IOSQE_ASYNC */ - REQ_F_FORCE_ASYNC = BIT(REQ_F_FORCE_ASYNC_BIT), + REQ_F_FORCE_ASYNC = IO_REQ_FLAG(REQ_F_FORCE_ASYNC_BIT), /* IOSQE_BUFFER_SELECT */ - REQ_F_BUFFER_SELECT = BIT(REQ_F_BUFFER_SELECT_BIT), + REQ_F_BUFFER_SELECT = IO_REQ_FLAG(REQ_F_BUFFER_SELECT_BIT), /* IOSQE_CQE_SKIP_SUCCESS */ - REQ_F_CQE_SKIP = BIT(REQ_F_CQE_SKIP_BIT), + REQ_F_CQE_SKIP = IO_REQ_FLAG(REQ_F_CQE_SKIP_BIT), /* fail rest of links */ - REQ_F_FAIL = BIT(REQ_F_FAIL_BIT), + REQ_F_FAIL = IO_REQ_FLAG(REQ_F_FAIL_BIT), /* on inflight list, should be cancelled and waited on exit reliably */ - REQ_F_INFLIGHT = BIT(REQ_F_INFLIGHT_BIT), + REQ_F_INFLIGHT = IO_REQ_FLAG(REQ_F_INFLIGHT_BIT), /* read/write uses file position */ - REQ_F_CUR_POS = BIT(REQ_F_CUR_POS_BIT), + REQ_F_CUR_POS = IO_REQ_FLAG(REQ_F_CUR_POS_BIT), /* must not punt to workers */ - REQ_F_NOWAIT = BIT(REQ_F_NOWAIT_BIT), + REQ_F_NOWAIT = IO_REQ_FLAG(REQ_F_NOWAIT_BIT), /* has or had linked timeout */ - REQ_F_LINK_TIMEOUT = BIT(REQ_F_LINK_TIMEOUT_BIT), + REQ_F_LINK_TIMEOUT = IO_REQ_FLAG(REQ_F_LINK_TIMEOUT_BIT), /* needs cleanup */ - REQ_F_NEED_CLEANUP = BIT(REQ_F_NEED_CLEANUP_BIT), + REQ_F_NEED_CLEANUP = IO_REQ_FLAG(REQ_F_NEED_CLEANUP_BIT), /* already went through poll handler */ - REQ_F_POLLED = BIT(REQ_F_POLLED_BIT), + REQ_F_POLLED = IO_REQ_FLAG(REQ_F_POLLED_BIT), /* buffer already selected */ - REQ_F_BUFFER_SELECTED = BIT(REQ_F_BUFFER_SELECTED_BIT), + REQ_F_BUFFER_SELECTED = IO_REQ_FLAG(REQ_F_BUFFER_SELECTED_BIT), /* buffer selected from ring, needs commit */ - REQ_F_BUFFER_RING = BIT(REQ_F_BUFFER_RING_BIT), + REQ_F_BUFFER_RING = IO_REQ_FLAG(REQ_F_BUFFER_RING_BIT), /* caller should reissue async */ - REQ_F_REISSUE = BIT(REQ_F_REISSUE_BIT), + REQ_F_REISSUE = IO_REQ_FLAG(REQ_F_REISSUE_BIT), /* supports async reads/writes */ - REQ_F_SUPPORT_NOWAIT = BIT(REQ_F_SUPPORT_NOWAIT_BIT), + REQ_F_SUPPORT_NOWAIT = IO_REQ_FLAG(REQ_F_SUPPORT_NOWAIT_BIT), /* regular file */ - REQ_F_ISREG = BIT(REQ_F_ISREG_BIT), + REQ_F_ISREG = IO_REQ_FLAG(REQ_F_ISREG_BIT), /* has creds assigned */ - REQ_F_CREDS = BIT(REQ_F_CREDS_BIT), + REQ_F_CREDS = IO_REQ_FLAG(REQ_F_CREDS_BIT), /* skip refcounting if not set */ - REQ_F_REFCOUNT = BIT(REQ_F_REFCOUNT_BIT), + REQ_F_REFCOUNT = IO_REQ_FLAG(REQ_F_REFCOUNT_BIT), /* there is a linked timeout that has to be armed */ - REQ_F_ARM_LTIMEOUT = BIT(REQ_F_ARM_LTIMEOUT_BIT), + REQ_F_ARM_LTIMEOUT = IO_REQ_FLAG(REQ_F_ARM_LTIMEOUT_BIT), /* ->async_data allocated */ - REQ_F_ASYNC_DATA = BIT(REQ_F_ASYNC_DATA_BIT), + REQ_F_ASYNC_DATA = IO_REQ_FLAG(REQ_F_ASYNC_DATA_BIT), /* don't post CQEs while failing linked requests */ - REQ_F_SKIP_LINK_CQES = BIT(REQ_F_SKIP_LINK_CQES_BIT), + REQ_F_SKIP_LINK_CQES = IO_REQ_FLAG(REQ_F_SKIP_LINK_CQES_BIT), /* single poll may be active */ - REQ_F_SINGLE_POLL = BIT(REQ_F_SINGLE_POLL_BIT), + REQ_F_SINGLE_POLL = IO_REQ_FLAG(REQ_F_SINGLE_POLL_BIT), /* double poll may active */ - REQ_F_DOUBLE_POLL = BIT(REQ_F_DOUBLE_POLL_BIT), + REQ_F_DOUBLE_POLL = IO_REQ_FLAG(REQ_F_DOUBLE_POLL_BIT), /* request has already done partial IO */ - REQ_F_PARTIAL_IO = BIT(REQ_F_PARTIAL_IO_BIT), + REQ_F_PARTIAL_IO = IO_REQ_FLAG(REQ_F_PARTIAL_IO_BIT), /* fast poll multishot mode */ - REQ_F_APOLL_MULTISHOT = BIT(REQ_F_APOLL_MULTISHOT_BIT), + REQ_F_APOLL_MULTISHOT = IO_REQ_FLAG(REQ_F_APOLL_MULTISHOT_BIT), /* recvmsg special flag, clear EPOLLIN */ - REQ_F_CLEAR_POLLIN = BIT(REQ_F_CLEAR_POLLIN_BIT), + REQ_F_CLEAR_POLLIN = IO_REQ_FLAG(REQ_F_CLEAR_POLLIN_BIT), /* hashed into ->cancel_hash_locked, protected by ->uring_lock */ - REQ_F_HASH_LOCKED = BIT(REQ_F_HASH_LOCKED_BIT), + REQ_F_HASH_LOCKED = IO_REQ_FLAG(REQ_F_HASH_LOCKED_BIT), /* don't use lazy poll wake for this request */ - REQ_F_POLL_NO_LAZY = BIT(REQ_F_POLL_NO_LAZY_BIT), + REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); @@ -592,15 +595,17 @@ struct io_kiocb { * and after selection it points to the buffer ID itself. */ u16 buf_index; - unsigned int flags; + + unsigned nr_tw; + + /* REQ_F_* flags */ + io_req_flags_t flags; struct io_cqe cqe; struct io_ring_ctx *ctx; struct task_struct *task; - struct io_rsrc_node *rsrc_node; - union { /* store used ubuf, so we can prevent reloading */ struct io_mapped_ubuf *imu; @@ -621,10 +626,12 @@ struct io_kiocb { /* cache ->apoll->events */ __poll_t apoll_events; }; + + struct io_rsrc_node *rsrc_node; + atomic_t refs; atomic_t poll_refs; struct io_task_work io_task_work; - unsigned nr_tw; /* for polled requests, i.e. IORING_OP_POLL_ADD and async armed poll */ struct hlist_node hash_node; /* internal polling, see IORING_FEAT_FAST_POLL */ diff --git a/include/trace/events/io_uring.h b/include/trace/events/io_uring.h index 69454f1f98b0..3d7704a52b73 100644 --- a/include/trace/events/io_uring.h +++ b/include/trace/events/io_uring.h @@ -148,7 +148,7 @@ TRACE_EVENT(io_uring_queue_async_work, __field( void *, req ) __field( u64, user_data ) __field( u8, opcode ) - __field( unsigned int, flags ) + __field( io_req_flags_t, flags ) __field( struct io_wq_work *, work ) __field( int, rw ) @@ -167,10 +167,10 @@ TRACE_EVENT(io_uring_queue_async_work, __assign_str(op_str, io_uring_get_opcode(req->opcode)); ), - TP_printk("ring %p, request %p, user_data 0x%llx, opcode %s, flags 0x%x, %s queue, work %p", + TP_printk("ring %p, request %p, user_data 0x%llx, opcode %s, flags 0x%lx, %s queue, work %p", __entry->ctx, __entry->req, __entry->user_data, - __get_str(op_str), - __entry->flags, __entry->rw ? "hashed" : "normal", __entry->work) + __get_str(op_str), (long) __entry->flags, + __entry->rw ? "hashed" : "normal", __entry->work) ); /** @@ -378,7 +378,7 @@ TRACE_EVENT(io_uring_submit_req, __field( void *, req ) __field( unsigned long long, user_data ) __field( u8, opcode ) - __field( u32, flags ) + __field( io_req_flags_t, flags ) __field( bool, sq_thread ) __string( op_str, io_uring_get_opcode(req->opcode) ) @@ -395,10 +395,10 @@ TRACE_EVENT(io_uring_submit_req, __assign_str(op_str, io_uring_get_opcode(req->opcode)); ), - TP_printk("ring %p, req %p, user_data 0x%llx, opcode %s, flags 0x%x, " + TP_printk("ring %p, req %p, user_data 0x%llx, opcode %s, flags 0x%lx, " "sq_thread %d", __entry->ctx, __entry->req, __entry->user_data, __get_str(op_str), - __entry->flags, __entry->sq_thread) + (long) __entry->flags, __entry->sq_thread) ); /* diff --git a/io_uring/filetable.h b/io_uring/filetable.h index b47adf170c31..b2435c4dca1f 100644 --- a/io_uring/filetable.h +++ b/io_uring/filetable.h @@ -17,7 +17,7 @@ int io_fixed_fd_remove(struct io_ring_ctx *ctx, unsigned int offset); int io_register_file_alloc_range(struct io_ring_ctx *ctx, struct io_uring_file_index_range __user *arg); -unsigned int io_file_get_flags(struct file *file); +io_req_flags_t io_file_get_flags(struct file *file); static inline void io_file_bitmap_clear(struct io_file_table *table, int bit) { diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index cd9a137ad6ce..b8ca907b77eb 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1768,9 +1768,9 @@ static void io_iopoll_req_issued(struct io_kiocb *req, unsigned int issue_flags) } } -unsigned int io_file_get_flags(struct file *file) +io_req_flags_t io_file_get_flags(struct file *file) { - unsigned int res = 0; + io_req_flags_t res = 0; if (S_ISREG(file_inode(file)->i_mode)) res |= REQ_F_ISREG; @@ -2171,7 +2171,8 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, /* req is partially pre-initialised, see io_preinit_req() */ req->opcode = opcode = READ_ONCE(sqe->opcode); /* same numerical values with corresponding REQ_F_*, safe to copy */ - req->flags = sqe_flags = READ_ONCE(sqe->flags); + sqe_flags = READ_ONCE(sqe->flags); + req->flags = (io_req_flags_t) sqe_flags; req->cqe.user_data = READ_ONCE(sqe->user_data); req->file = NULL; req->rsrc_node = NULL; @@ -4153,7 +4154,7 @@ static int __init io_uring_init(void) BUILD_BUG_ON(SQE_COMMON_FLAGS >= (1 << 8)); BUILD_BUG_ON((SQE_VALID_FLAGS | SQE_COMMON_FLAGS) != SQE_VALID_FLAGS); - BUILD_BUG_ON(__REQ_F_LAST_BIT > 8 * sizeof(int)); + BUILD_BUG_ON(__REQ_F_LAST_BIT > 8 * sizeof_field(struct io_kiocb, flags)); BUILD_BUG_ON(sizeof(atomic_t) != sizeof(u32)); From patchwork Wed Feb 7 17:17:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13548760 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CCF87A70E for ; Wed, 7 Feb 2024 17:19:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326389; cv=none; b=RyTpXS3utOJSuKrFT1Ai8yol6CFSceBBA8/Bk9xtfF1hXko5C0qcMjEGXLevhHMPykqxDf3ZETuag/hQojKBAWrNTZFvMIFD55eZqCB2NHS47dokt6K3QAnpEuU1fdF+3rZkLnSTqWA22kcxynNDNTg/YySrrKsciFZEBqCDq+E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326389; c=relaxed/simple; bh=i8Qdi8ws8Pdi0xD/DCla8fIwg3048KoTZDGnwE9ukZw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QqcIsen8E0JHbzxW6HFu/wNJaAUs0vm6yDLPXpdSbcsl+AcWCsWN9efApdL9Cs7kEdCQsZVEYA1my2JbUsqpZp4exSP5E6db+Rmo5S+PyhxzCnQZG4FxGfjYkizCJeiBeouv+keHGU22T6ODjb2qTacjfgIRqa69ZsvGq/P/DGo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=dzuuLnUQ; arc=none smtp.client-ip=209.85.166.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="dzuuLnUQ" Received: by mail-io1-f53.google.com with SMTP id ca18e2360f4ac-7bbdd28a52aso17088839f.1 for ; Wed, 07 Feb 2024 09:19:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707326386; x=1707931186; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Z+BvvT/bLt+Dtd4p+a8eUflunBIdVoGheioMCSSBG2o=; b=dzuuLnUQ0XIITZBcl37S8zN5RG8cQMZSp/aMCoYV8m78I9WDJD55/cwWWAuH6y9oaS I7oNBQOb8eCd1h1uACSHr9f/oostXAUbR62297QkfZOuceUFPuxDqN8h/hG6xAY6xQgi 8OUczQjl1vZYxRNtSuf2RDN8BDScgmxoHh5A/CMA+P4m4/o6kepCI+bNg7L+lXA4GoHP Iu0VOOLul1q/jI+cZbYCjrjBla+b/E/q7XFj2ygfCudx950Q887OgLHXBc7OXNlQGUXq nlnXGdf/mc/A5s/gpJtIx09YDIbMtredCg9ImZZshQylhPbYsMGLVPvZLhjOAe4kUb1m xgXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707326386; x=1707931186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Z+BvvT/bLt+Dtd4p+a8eUflunBIdVoGheioMCSSBG2o=; b=Xw+MIfZfzfOhkMVj53Dby1Lm2Xs7f1u1x3RCcnpECcyL6OKyaIbXR7Hp1clFsoF7ew YGgU2iBsI8KIWh4K1etinFgq4MKjal2b0ebnYE9TWuDxQ0iQGlGL9ZovWfziUrC1BO92 QZhHfwG14u2MJRGbI6d0u+bBLM4zGgHI3cPAiKZ7JUrpdhGUIksTMLZ7EyROnca5ZzWf xRUpqm/svTFWpeoWkGuo57kanFluYhsPnbzT9AgYg+POeATSaO6B7FYMwGWkisHhXrHY dsCKiVuOWVomzPGkR55Vo8EnOhpxJdH0aoMzg7eCiqo7FbFYbGurwcSoKl7IeDWh1CSB K+qA== X-Gm-Message-State: AOJu0Yy9LkllmKxDV99yTisaEyt5+ceuOoIJtjQvBW7xl3D9fifXqPCH q39IFFrgVDVWhvMiJxoLptYDVuz2s/TpXssvjulu0a8xX3dTYjUwYPDGk7VkSGdT51G/an8YN2Y Gpow= X-Google-Smtp-Source: AGHT+IHccnQl1OtD19qC7e2B/NbF0WJmELn0D5flhBJM0XNTL/7fjod7hH2kiTMUgAfwPCW1HXGLiA== X-Received: by 2002:a05:6602:155:b0:7c4:655:6e05 with SMTP id v21-20020a056602015500b007c406556e05mr472627iot.2.1707326386238; Wed, 07 Feb 2024 09:19:46 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id g22-20020a6b7616000000b007bc4622d199sm421131iom.22.2024.02.07.09.19.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 09:19:45 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/6] io_uring/cancel: don't default to setting req->work.cancel_seq Date: Wed, 7 Feb 2024 10:17:36 -0700 Message-ID: <20240207171941.1091453-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207171941.1091453-1-axboe@kernel.dk> References: <20240207171941.1091453-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just leave it unset by default, avoiding dipping into the last cacheline (which is otherwise untouched) for the fast path of using poll to drive networked traffic. Add a flag that tells us if the sequence is valid or not, and then we can defer actually assigning the flag and sequence until someone runs cancelations. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/cancel.c | 3 +-- io_uring/cancel.h | 10 ++++++++++ io_uring/io_uring.c | 1 - io_uring/poll.c | 6 +----- 5 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 56bf733d3ee6..e19698daae1a 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -463,6 +463,7 @@ enum { REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, REQ_F_POLL_NO_LAZY_BIT, + REQ_F_CANCEL_SEQ_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -535,6 +536,8 @@ enum { REQ_F_HASH_LOCKED = IO_REQ_FLAG(REQ_F_HASH_LOCKED_BIT), /* don't use lazy poll wake for this request */ REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), + /* cancel sequence is set and valid */ + REQ_F_CANCEL_SEQ = IO_REQ_FLAG(REQ_F_CANCEL_SEQ_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 8a8b07dfc444..acfcdd7f059a 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -58,9 +58,8 @@ bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd) return false; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { check_seq: - if (cd->seq == req->work.cancel_seq) + if (io_cancel_match_sequence(req, cd->seq)) return false; - req->work.cancel_seq = cd->seq; } return true; diff --git a/io_uring/cancel.h b/io_uring/cancel.h index c0a8e7c520b6..76b32e65c03c 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -25,4 +25,14 @@ void init_hash_table(struct io_hash_table *table, unsigned size); int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg); bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd); +static inline bool io_cancel_match_sequence(struct io_kiocb *req, int sequence) +{ + if ((req->flags & REQ_F_CANCEL_SEQ) && sequence == req->work.cancel_seq) + return true; + + req->flags |= REQ_F_CANCEL_SEQ; + req->work.cancel_seq = sequence; + return false; +} + #endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b8ca907b77eb..fd552b260eef 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -463,7 +463,6 @@ static void io_prep_async_work(struct io_kiocb *req) req->work.list.next = NULL; req->work.flags = 0; - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); if (req->flags & REQ_F_FORCE_ASYNC) req->work.flags |= IO_WQ_WORK_CONCURRENT; diff --git a/io_uring/poll.c b/io_uring/poll.c index 7513afc7b702..c2b0a2d0762b 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -588,10 +588,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll_table *ipt, __poll_t mask, unsigned issue_flags) { - struct io_ring_ctx *ctx = req->ctx; - INIT_HLIST_NODE(&req->hash_node); - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); io_init_poll_iocb(poll, mask); poll->file = req->file; req->apoll_events = poll->events; @@ -818,9 +815,8 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, if (poll_only && req->opcode != IORING_OP_POLL_ADD) continue; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { - if (cd->seq == req->work.cancel_seq) + if (io_cancel_match_sequence(req, cd->seq)) continue; - req->work.cancel_seq = cd->seq; } *out_bucket = hb; return req; From patchwork Wed Feb 7 17:17:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13548761 Received: from mail-il1-f182.google.com (mail-il1-f182.google.com [209.85.166.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 02806823D9 for ; Wed, 7 Feb 2024 17:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326390; cv=none; b=STePHle7zCByi2WAMXUjVg5I1J5LtIhH6dVRbe3RLUERvbE8IYDSIfBPStLOUwvrw5s0F66+ytEFHKnybvW0z6dC9c+C4glX9VIs7m/kOS9vo46OiHro4PalUkftQD1a7m09KcU/DzJebs40MMD93NYAEH8xAkDXfafLFMPGDDA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326390; c=relaxed/simple; bh=Vx2z+uKGYqAboyyvt/fRTruzvgWgXrTGSrxi8X8tqm0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G6H56WNZ8lJCO3AM6Io8oqwQCB5DGGPwnFEkJy2C7vbnEfvhJct8c3sh8s3EDbZ02TjTbTAyfAgNAKBMBT9FVrM0J+ffY6BbnInwvfTo9c7tfhxFvfOkjbEv2VT6zB42IaV/8BVIbONjAKhhTTuLrdps8PO5KgTG6n9je3536Rs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=M3nelOSY; arc=none smtp.client-ip=209.85.166.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="M3nelOSY" Received: by mail-il1-f182.google.com with SMTP id e9e14a558f8ab-363ce7a4823so1208595ab.0 for ; Wed, 07 Feb 2024 09:19:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707326387; x=1707931187; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=A0u+eIWzkQ4RfJkBwW7+VLDljeFRsQAnKe6PYkaLIvo=; b=M3nelOSY06VjrBvu4ihUcZhJxXWJMqVxnRPyqVvVyWKaQTBydJbxLRYr0SmtB19I61 JvTHYAVMHMpLxrCsOewKsq/JcRiHi7uy82xPLrZQqhvMWpcY4mMeZ9hGENJXgyNMbitn 8c/VBoO6am4QuCgERkN+NPs60Ch2A5WCbiHEWZ/0PBtQRDmCbetk5rg/qLVYrDnyg1LF R2KHNAHKSyi8QrPKP+a9xKBdn4bky2C/HJAeix6x5XtnnFO3pER3zug0poVuRvvbGze2 tYylZ8zD8I6mBw1aAW7o/gDXnbDCb/rXh6JPh7poiM4dabFSykahhv5s2f290fJ1qk64 2nbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707326387; x=1707931187; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=A0u+eIWzkQ4RfJkBwW7+VLDljeFRsQAnKe6PYkaLIvo=; b=rDTdpzR+tAP+nJTDwmRY9FL7w0OHGEW+kgr+IYAyDtalBQB8s00VLEDPvw7q/M0VU0 V+kzZv0fjD2WHzp/xjJz08plhhHAu3FLJWQHbcL84JUNB7QDg7qFrcVr5x7OUPb3IymT 2lV4rSvE7lh70frCF7yyrYrZEsG0vhN2v4qz+MgPIfuvHO0m6BbQzffvnp1KPm5Vz3p2 z214HDsmGjCoRpIFrJkRyCaxi67Jw0mD8QFaK78HYUbeakgPclSsCtXKUkhiOCYQy3yY 8SRYBY9YBfe+EaLoKYq7rNuqYWHKssVNHYu6E225UNw4pWvy4SjCEaZ2yc/hW7fL0Ypp QIjg== X-Gm-Message-State: AOJu0YwmY3C7nNp8KmcwG5mHoRI8OADqi58TOxd/uTdBvbufk04ISI+b eOVPgG4w6UsDktIVK9URaKYVzn+SGFdhLotn/SEbfOQI0tcO/wZxITYpDzF0UnArrfcBoJwS4Gi wRkE= X-Google-Smtp-Source: AGHT+IFMcJyYz1ZfvNNdLAPGFscCz9QhBJBkOeq1LVGqeqAvdsiKH6GBuwfdGasrvnYTLhkOA9RZug== X-Received: by 2002:a6b:3e42:0:b0:7c4:606:6501 with SMTP id l63-20020a6b3e42000000b007c406066501mr610204ioa.2.1707326387542; Wed, 07 Feb 2024 09:19:47 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id g22-20020a6b7616000000b007bc4622d199sm421131iom.22.2024.02.07.09.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 09:19:46 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/6] io_uring: add io_file_can_poll() helper Date: Wed, 7 Feb 2024 10:17:37 -0700 Message-ID: <20240207171941.1091453-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207171941.1091453-1-axboe@kernel.dk> References: <20240207171941.1091453-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This adds a flag to avoid dipping dereferencing file and then f_op to figure out if the file has a poll handler defined or not. We generally call this at least twice for networked workloads, and if using ring provided buffers, we do it on every buffer selection. Particularly the latter is troublesome, as it's otherwise a very fast operation. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 3 +++ io_uring/io_uring.c | 2 +- io_uring/io_uring.h | 12 ++++++++++++ io_uring/kbuf.c | 2 +- io_uring/poll.c | 2 +- io_uring/rw.c | 6 +++--- 6 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e19698daae1a..4ddc7b3168f3 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -464,6 +464,7 @@ enum { REQ_F_ISREG_BIT, REQ_F_POLL_NO_LAZY_BIT, REQ_F_CANCEL_SEQ_BIT, + REQ_F_CAN_POLL_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -538,6 +539,8 @@ enum { REQ_F_POLL_NO_LAZY = IO_REQ_FLAG(REQ_F_POLL_NO_LAZY_BIT), /* cancel sequence is set and valid */ REQ_F_CANCEL_SEQ = IO_REQ_FLAG(REQ_F_CANCEL_SEQ_BIT), + /* file is pollable */ + REQ_F_CAN_POLL = IO_REQ_FLAG(REQ_F_CAN_POLL_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index fd552b260eef..17bd16be1dfd 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1968,7 +1968,7 @@ void io_wq_submit_work(struct io_wq_work *work) if (req->flags & REQ_F_FORCE_ASYNC) { bool opcode_poll = def->pollin || def->pollout; - if (opcode_poll && file_can_poll(req->file)) { + if (opcode_poll && io_file_can_poll(req)) { needs_poll = true; issue_flags |= IO_URING_F_NONBLOCK; } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index d5495710c178..2952551fe345 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include "io-wq.h" @@ -398,4 +399,15 @@ static inline size_t uring_sqe_size(struct io_ring_ctx *ctx) return 2 * sizeof(struct io_uring_sqe); return sizeof(struct io_uring_sqe); } + +static inline bool io_file_can_poll(struct io_kiocb *req) +{ + if (req->flags & REQ_F_CAN_POLL) + return true; + if (file_can_poll(req->file)) { + req->flags |= REQ_F_CAN_POLL; + return true; + } + return false; +} #endif diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 18df5a9d2f5e..71880615bb78 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -180,7 +180,7 @@ static void __user *io_ring_buffer_select(struct io_kiocb *req, size_t *len, req->buf_list = bl; req->buf_index = buf->bid; - if (issue_flags & IO_URING_F_UNLOCKED || !file_can_poll(req->file)) { + if (issue_flags & IO_URING_F_UNLOCKED || !io_file_can_poll(req)) { /* * If we came in unlocked, we have no choice but to consume the * buffer here, otherwise nothing ensures that the buffer won't diff --git a/io_uring/poll.c b/io_uring/poll.c index c2b0a2d0762b..3f3380dc5f68 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -724,7 +724,7 @@ int io_arm_poll_handler(struct io_kiocb *req, unsigned issue_flags) if (!def->pollin && !def->pollout) return IO_APOLL_ABORTED; - if (!file_can_poll(req->file)) + if (!io_file_can_poll(req)) return IO_APOLL_ABORTED; if (!(req->flags & REQ_F_APOLL_MULTISHOT)) mask |= EPOLLONESHOT; diff --git a/io_uring/rw.c b/io_uring/rw.c index d5e79d9bdc71..0fb7a045163a 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -682,7 +682,7 @@ static bool io_rw_should_retry(struct io_kiocb *req) * just use poll if we can, and don't attempt if the fs doesn't * support callback based unlocks */ - if (file_can_poll(req->file) || !(req->file->f_mode & FMODE_BUF_RASYNC)) + if (io_file_can_poll(req) || !(req->file->f_mode & FMODE_BUF_RASYNC)) return false; wait->wait.func = io_async_buf_func; @@ -831,7 +831,7 @@ static int __io_read(struct io_kiocb *req, unsigned int issue_flags) * If we can poll, just do that. For a vectored read, we'll * need to copy state first. */ - if (file_can_poll(req->file) && !io_issue_defs[req->opcode].vectored) + if (io_file_can_poll(req) && !io_issue_defs[req->opcode].vectored) return -EAGAIN; /* IOPOLL retry should happen for io-wq threads */ if (!force_nonblock && !(req->ctx->flags & IORING_SETUP_IOPOLL)) @@ -930,7 +930,7 @@ int io_read_mshot(struct io_kiocb *req, unsigned int issue_flags) /* * Multishot MUST be used on a pollable file */ - if (!file_can_poll(req->file)) + if (!io_file_can_poll(req)) return -EBADFD; ret = __io_read(req, issue_flags); From patchwork Wed Feb 7 17:17:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13548762 Received: from mail-il1-f182.google.com (mail-il1-f182.google.com [209.85.166.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D113823D2 for ; Wed, 7 Feb 2024 17:19:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326391; cv=none; b=O5rpZ6HigZkKKntR+Rtl57etLp6RLhQdQ23EtF8r2Mfmz8haEpl5zusb5RXj4Bt7D987KM1xwJrMx0RTGmIb8x0ujHv7h8uLJzGz0b7ESTIVDSqM02uWOm0zVJEOLguZFjMEfmXtCKCf9rE7sAHOPr6EpjMYmda+47RHOPUkINQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326391; c=relaxed/simple; bh=uimgy7+wl4DXZV7rrq9F0Z4HxdrzWqdEH39e6n8xHJ4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DZbfbPd+jzXOt6Pfsx+gSa0N7oK8svxzfek2f5hZosClW1BBVNJUJLsfnRIKKrQO4n7uup6wee6bDiJHBeCAQJA2UyvB//eZ9D+GBMEJ7FjqrIg2uqBhzVf8XBq54MWxlhobQQ3NNBGfIEj/z8HXtPDD2KSZRyeuT6BR7LzYKFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=tHW0/q54; arc=none smtp.client-ip=209.85.166.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="tHW0/q54" Received: by mail-il1-f182.google.com with SMTP id e9e14a558f8ab-363d6b409b1so1231275ab.1 for ; Wed, 07 Feb 2024 09:19:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707326389; x=1707931189; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GXvepwDyq3Lcyc7QSbbVA75c61s/YgsZV8j5neg7n9s=; b=tHW0/q54iQpQjTXx0ItGG4KCQHn3P84p2vhXICxTq2OGmzHdlumR0vPrDOZLdZXsgI sCRNhIp09+nTjZh4c1nbnU7bI9Zyc6FO+Ecw9IERvapE9hwOAES2VyuxKQluPJ83TTYa pysQLfVH5sN2Sj3mqIAppzU4QDAVb7NPXqfd0T1WVqzImV2eF84U74qRu//fP0jOpMDB BA2tLSRB5thcut/hcuziRYTkBQQTh8i0OxhL/BQqdHMrQ5kHe195q949kSNvA+EqSYMY dGVoxfNBCur9TpSIIfQ3tzlyLdpEM9lzdCHJ+2u0qZlE6/pORinb8ywiKaLLkCXZts7U XGww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707326389; x=1707931189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GXvepwDyq3Lcyc7QSbbVA75c61s/YgsZV8j5neg7n9s=; b=hSIiXR7PgqXjmhB/bf07K996bQoUQunPkLTKnRDTZsv64l8awTSJplKoLBpkqEK6Oo ZqaOPlI3TukoUFK5veWCgg5hH/Ks3Ck6NoDXNbIn+Bok4GaED+vBoSGrdXiefIGRbXxv tlp6kEsZ/57Fe1s15n77mE9vssuwuFtA5Gv0MRbXM7bMCpReY6N0hAa0MkdIq38U9Fzv GFFWf5ZEjQG/1qwB9bTGBE7Eo3xdfwA8REDbc/L1SFH2eGLjSz+la22QfVORV88ACCy3 XO8rliVSLULuugZ7OUUXPsZ4XHKZZVRfC9pZbboQI0ZwXYbenx8J7zUVx1V9tCHsVx4z qAEA== X-Gm-Message-State: AOJu0YyH2pqeC4vC1+4p7GBSBm3wYw34+tJArV/qISPtKVJtEzVHFny1 euEXDAGXqKd4AbiM+q8NlOfqx9AqAGNhX/PAlhtUPOIDMFtK3OyhorwKsfLuFqS6PqmFAd175aW RRFY= X-Google-Smtp-Source: AGHT+IH4AuyVfppAQj4Nn8Obhg0o86xsDq7rycKP7iq4TKvotoQ3bEum136qrgjMjMpi14R8HYwF8w== X-Received: by 2002:a5d:938a:0:b0:7c3:f631:a18f with SMTP id c10-20020a5d938a000000b007c3f631a18fmr4803455iol.1.1707326388954; Wed, 07 Feb 2024 09:19:48 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id g22-20020a6b7616000000b007bc4622d199sm421131iom.22.2024.02.07.09.19.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 09:19:47 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/6] io_uring: mark the need to lock/unlock the ring as unlikely Date: Wed, 7 Feb 2024 10:17:38 -0700 Message-ID: <20240207171941.1091453-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207171941.1091453-1-axboe@kernel.dk> References: <20240207171941.1091453-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Any of the fast paths will already have this locked, this helper only exists to deal with io-wq invoking request issue where we do not have the ctx->uring_lock held already. This means that any common or fast path will already have this locked, mark it as such. Signed-off-by: Jens Axboe --- io_uring/io_uring.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 2952551fe345..46795ee462df 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -208,7 +208,7 @@ static inline void io_ring_submit_unlock(struct io_ring_ctx *ctx, unsigned issue_flags) { lockdep_assert_held(&ctx->uring_lock); - if (issue_flags & IO_URING_F_UNLOCKED) + if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) mutex_unlock(&ctx->uring_lock); } @@ -221,7 +221,7 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx, * The only exception is when we've detached the request and issue it * from an async worker thread, grab the lock for that case. */ - if (issue_flags & IO_URING_F_UNLOCKED) + if (unlikely(issue_flags & IO_URING_F_UNLOCKED)) mutex_lock(&ctx->uring_lock); lockdep_assert_held(&ctx->uring_lock); } From patchwork Wed Feb 7 17:17:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13548763 Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89CBF823D9 for ; Wed, 7 Feb 2024 17:19:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326393; cv=none; b=ut1dYsC4Bl57vahOKj+yCBjuyCA9kf+2ENiQoiPD2NXKGSKb5bC5cRS7G+VdoEkdiUlcXVJqKh/olIOk74Pkx7hZYS6nkIyZZixIhWvecH4U2uD41ZRp16RfSkRf/i4EV4ks7c0I9K8aKujZRUK8plSykojOjhxWixbQnTfMErM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326393; c=relaxed/simple; bh=t+UUO9ziUzttYNdzy1UEtG1LdRlHQTp7m6IMRSUs1to=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EoMm8kV8Nd1YvMfSWgK/KOvikZOuReeB1eJfxZwR/cP/L3KSb+UlR/gLborsFnf+u+lZD682aI3Bg+p2Pvx0w4fzrAcMVLqtzQw55hzdcd8YRc54l5v5+2YLhxTvM3ik7+gKCg1SFuN5o3RSUHWvFJYYaMIcTAVJolZu/ZavISQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=QOM0jQSa; arc=none smtp.client-ip=209.85.166.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="QOM0jQSa" Received: by mail-io1-f47.google.com with SMTP id ca18e2360f4ac-7bbdd28a52aso17090039f.1 for ; Wed, 07 Feb 2024 09:19:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707326390; x=1707931190; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jdSGsBvnf6lEB/G2k9YOuPATM50KlQkAbmUJzXz9v+E=; b=QOM0jQSaMhFno/705z6WVzj95Ml/FL2fZtn6xANpwm/FZWmz+XnlUpf6qI1r15xv0V nTTHxhH5CPoDyL7vjOoqxoLzTHyxdnEhbqV4ratgdmOIsbBZ6CeglK6hAjQO+oW/m4lr qfAGSngWyaUvuLC6paRUDqoxKwLslwOaarod05J2HCZRiRtdT/y5ais7MDG7uqS1Q25l SRyYucbTThGQgZv+qiX9lZSr5t/G+xmC0N9+EA0Q4c6OdomX05l8IJMWyDeamL0QNKax qiITzIrJdbfD3MvcQrRIULhxfhk5CCnAfPYQdkrRVLgRwwON3xcxWxMNFAegMvzDX5Qt EvbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707326390; x=1707931190; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jdSGsBvnf6lEB/G2k9YOuPATM50KlQkAbmUJzXz9v+E=; b=Qxe433yJjJd/DNTmev2mdza2p+iCZq3JsrQO7Y+0Ve4F453S04DOztxoettRH0i1Ao ktnwyill8wRo+IICFfc8gfe6Ow0Fg7loEatRVzzKssaJ451K7TbsE8iVoH2+EKXVmlv1 14fqJ54z67HzfhWQnvd0MroF1kio52HcsNqJUMXNtz2LNIsm5pR41ASiM20651qUier9 FUntnufkfH1J7jW1a8KygsJg3OCVSYqWCxO9omrOthSxr1upbn90Vv+SXrCwxI0fV4NZ 2pcmClKXTiiUkAw5Kh3l1y8rRWOxbtAKwOWdiO/CwImgqJsinX+CaXN2KWKlIke5OQH+ 5XXg== X-Gm-Message-State: AOJu0YxfHe51JXi8sin5VrMvp7BMhYVEERIrFAL15iSeG/tIsf0BF/tQ QeANsS2TTqxc7038kQzmgskBSSE18p0QXnbTKoydZrugvtzZWpRw7VduM5m7mffsfyZ8h2i3GYu 4VWQ= X-Google-Smtp-Source: AGHT+IFESC5GmgQoA3BAteoM6wTRGVBPg3BqlAkC6g1NqKzbqviFuZ3t8qOvS91WjB88HhTo2Jrltg== X-Received: by 2002:a05:6602:155:b0:7c4:655:6e05 with SMTP id v21-20020a056602015500b007c406556e05mr472905iot.2.1707326390372; Wed, 07 Feb 2024 09:19:50 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id g22-20020a6b7616000000b007bc4622d199sm421131iom.22.2024.02.07.09.19.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 09:19:49 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 5/6] io_uring: cleanup io_req_complete_post() Date: Wed, 7 Feb 2024 10:17:39 -0700 Message-ID: <20240207171941.1091453-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207171941.1091453-1-axboe@kernel.dk> References: <20240207171941.1091453-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Move the ctx declaration and assignment up to be generally available in the function, as we use req->ctx at the top anyway. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 17bd16be1dfd..704df7e23734 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1024,15 +1024,15 @@ static void __io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags) { - if (req->ctx->task_complete && req->ctx->submitter_task != current) { + struct io_ring_ctx *ctx = req->ctx; + + if (ctx->task_complete && ctx->submitter_task != current) { req->io_task_work.func = io_req_task_complete; io_req_task_work_add(req); } else if (!(issue_flags & IO_URING_F_UNLOCKED) || - !(req->ctx->flags & IORING_SETUP_IOPOLL)) { + !(ctx->flags & IORING_SETUP_IOPOLL)) { __io_req_complete_post(req, issue_flags); } else { - struct io_ring_ctx *ctx = req->ctx; - mutex_lock(&ctx->uring_lock); __io_req_complete_post(req, issue_flags & ~IO_URING_F_UNLOCKED); mutex_unlock(&ctx->uring_lock); From patchwork Wed Feb 7 17:17:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13548764 Received: from mail-il1-f173.google.com (mail-il1-f173.google.com [209.85.166.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 051BD823C7 for ; Wed, 7 Feb 2024 17:19:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326395; cv=none; b=o5GRlZzg7l0+jwaQrpkHoovY8Kz13DJL+8TM7APt3GWjg3Kq47i1IkM6D/TwXBduTL1o6nng5ZvflqcdxPaO+Vm0ZyIpBryyu+1+lEM7LGKSWsW+WyKzE7tS+59fr3BYpehTjY1mCO2LLdov18YbZFsJ67FwGboYlEf4gxhRJM4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707326395; c=relaxed/simple; bh=iW+NquMWGt9Gqk/vv4bbPeGDI6UBStnYtNVupziAj8A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rKOJAzEOQJyK7DlL3dP/pGRkT/2s8yV8gy9jU/53O8Lw8iF4ZPQ9DtXUO5F07oqKeH+Htd/Xrhj/mcFNkgxeQw+nzogw5pMa4R8RNq2S3Kl3MyqPzdVjTj7SURwkvDX2Ei6iuW5RT4libPGXj65wGgfoV2UJOIieLqD5fVZ6wA8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=IUWkaF3w; arc=none smtp.client-ip=209.85.166.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="IUWkaF3w" Received: by mail-il1-f173.google.com with SMTP id e9e14a558f8ab-363bf960762so259775ab.0 for ; Wed, 07 Feb 2024 09:19:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1707326391; x=1707931191; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UdoaioIXngn6g/+xmxUBVqHGk8dPSTC8+pDTCg3h3uM=; b=IUWkaF3wkynlNdh304l+lgXsb9/nuN+dmPcwumsh19VDzeUjssehJXGqLPYiE28Lfj fjnNcsUTvTabKAznKz6UQhGGRsRAq7k4CXW2mJ2Thx3ISsJS/h6pfRyvG4meI8wSl3Fk 5cpbQASrt95t/345IVeh1j5kCAto4DoDLgOvBYuXMGLGeJ+XLZBsi85rRJGA/lwEGZWA 2fxA/8/D2cBUM4ToXDpGBGh+MHPGed9IfVoQThGDgAwwoeXjXQdVS13InXe9Yn84PkVk tPUEziMjuvLhEbyQQwZUadizEThNJPPUvguVOfJj9j5VL3KahWwSeKhtrjGc7ex3jT3A dCOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707326391; x=1707931191; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UdoaioIXngn6g/+xmxUBVqHGk8dPSTC8+pDTCg3h3uM=; b=C7Tq29vg4BV6h464sPWzMxkh5vDwP8NIXTbrxBNdrt258aIw0ndnACBIU7w8m7RX8W OfbiIPREnJDVTTyKZOxbTm6Z6ZjEtVMhFJ/Eja7ySg1d3YcteeHfF5Wos+mMPt9KFwez 0eUZDSYDvMQSCKTyhf1lNVTpA/doY/B5392ENEI9puoZBJqb44KQ1K2yahc7qq0KMrDN NM0HV2X0VY7WmAU5rlyQyOUbVfF7lgDr8mrxU1xLG9rwVTYvUd62c00YAT+Rq+PbBMl8 nExePJ+Z0z6wgDLRFolN81IBA3xkaYYd8Bqouii3Eg2+IwW9w0Cjqm2jLcHMPirWZbmX guaQ== X-Gm-Message-State: AOJu0Yy9HZOS09Jm1dLpN4h0PYJH9vfLncwO3kBfTILrqMs5I72IOlFb 6dwr0ZGR0UOp+1Ki+1J1w2GmqM4Jb5aUfTRh4yqio0lY/+icCAVfLKYTbZZ/TqrhC1Lf0ml+JR0 RuaQ= X-Google-Smtp-Source: AGHT+IFFC/UUpA08qMxhDve3FFc8a23sy2Ubvkj360wSkt5Hz5rJDLA76fijlM6UCT+aj3c/YWnoTA== X-Received: by 2002:a05:6602:2410:b0:7c3:f2c1:e8aa with SMTP id s16-20020a056602241000b007c3f2c1e8aamr5896417ioa.0.1707326391654; Wed, 07 Feb 2024 09:19:51 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id g22-20020a6b7616000000b007bc4622d199sm421131iom.22.2024.02.07.09.19.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Feb 2024 09:19:50 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 6/6] io_uring/rw: remove dead file == NULL check Date: Wed, 7 Feb 2024 10:17:40 -0700 Message-ID: <20240207171941.1091453-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240207171941.1091453-1-axboe@kernel.dk> References: <20240207171941.1091453-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Any read/write opcode has needs_file == true, which means that we would've failed the request long before reaching the issue stage if we didn't successfully assign a file. This check has been dead forever, and is really a leftover from generic code. Signed-off-by: Jens Axboe --- io_uring/rw.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/rw.c b/io_uring/rw.c index 0fb7a045163a..8ba93fffc23a 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -721,7 +721,7 @@ static int io_rw_init_file(struct io_kiocb *req, fmode_t mode) struct file *file = req->file; int ret; - if (unlikely(!file || !(file->f_mode & mode))) + if (unlikely(!(file->f_mode & mode))) return -EBADF; if (!(req->flags & REQ_F_FIXED_FILE))