From patchwork Wed Jun 5 13:51:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687011 Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E8CE2770E for ; Wed, 5 Jun 2024 14:19:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597186; cv=none; b=oaE0K2auBhK1/Q3RUpMRbsDRM2HMUJEJgxOCUGhW+FLfXyBwLtsd4pQXb8UyVDfV0M8loHWM0cFMgn7InPxDFYnVAOgLVvtb005UPkDjHyVel1vAEUy3gJ3qK4CAsm/oIASthnAuf1pfU/FUkVWLBeB+PD1RV+X5RRzbMMLi8SY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597186; c=relaxed/simple; bh=Dg7eZOnIgLth5/bbTSu+ZhOWjL8S1TMLSv1i4QJE9bY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MbvyFnwHkoGQwBj/41+e7QwjSh1jcWDUgpYj3/hDTxQlgkLmGFIvnQ0dM0bW4vZiPUgWLO13W9oL2gv8KZAOHd8lyRrtgFFNov7WwrCPnlFRx/0KcVkOA3FJWmNXVEs6Q+yt7KaZNeXdVJC7J94zLDN1wsXTIQhyhJJ2BEkUMRI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=B3BXqBWt; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="B3BXqBWt" Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3d1b8bacf67so685415b6e.2 for ; Wed, 05 Jun 2024 07:19:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597181; x=1718201981; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=T1EmDC2bGMmU0XeSvchLBl6+1crQIfO4g6MqrY1wcIM=; b=B3BXqBWtM5mcSYMWhIVcPloEK2EUxqw5THlfWUWQLCvKTaLgZH0s1/Bum/PWSF4TFt sU0ifRCZsUo6PqxlnAVZ95x0Bj8vKSf0I7UzsLekUISDoJ7fpeN70W5ucxPb6a+tqgAr 1y7pr7+8Fkf+nJNWWnbVku8ISEZs9vxsMeyqB2ZLLaDKxmaKCx91d4fnnwFR/WGHr97P Qh8rA8EqQCivj7VRjPzm+NoqGVstqmee48P0rVXpAsK09Fa2CCulV4XU8eryqLp5AQJE jVUemBzko9nGjvn4T8SSpiztxatXPONhdoqZrE4gnhGx2eFbMJ8wuCKwwnUujgmPcZKw VNzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597181; x=1718201981; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T1EmDC2bGMmU0XeSvchLBl6+1crQIfO4g6MqrY1wcIM=; b=QFFHeKjyeg+rW7PLkr8jwWPnVfv4ex6l/b+iROVpfAMesPibap+z2Ekm220w+queZ9 Df5aeI8wljJiB0YBCiT2qFcaDBsbbEqxp1tCcpUzXWgYkWR+O8DfntVLHMnirlQALULS AfrPcFuE02xHAT3jyNeLtfkN40JCtIH4PrFy7CW65MhYzGiYsBZOOZmLH3EOldv+akhi SIuqcIn/4DCmWgJbpmSlhu4ijT5KCDI1Pij5qCj2Y1XDJD6VafKkbHnq/RtGV8uOC2aB GHurdNsWVmDPXhJNL063Ls/fWAtXj9yXQKCwd6i3pbWyCDuWkC8YKoCAHQXb4rEaYKs3 tMCg== X-Gm-Message-State: AOJu0Yx+1EuKWZZ4xxR8HvC0aUlt/BM2b1uhyaO5RbYqiWvVGdgQIkIa vLtD13kpWkNmewuzV0AnK0rd1XJULr7oGbIRTju0t1xpv2Op2dICatNsB5LxmoLJUXz/4QmgTpS 5 X-Google-Smtp-Source: AGHT+IFfzknKUeqA2p6Cc3z31GXrzHks2X8zBt5i7RTXo1DvEKpPQWvvFdhtue3KNAa46L6r+VILzw== X-Received: by 2002:a05:6870:4d18:b0:24c:b092:fd38 with SMTP id 586e51a60fabf-25121cf42eamr3315374fac.1.1717597180911; Wed, 05 Jun 2024 07:19:40 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:39 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 1/9] io_uring/msg_ring: tighten requirement for remote posting Date: Wed, 5 Jun 2024 07:51:09 -0600 Message-ID: <20240605141933.11975-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Currently this is gated on whether or not the target ring needs a local completion - and if so, whether or not we're running on the right task. The use case for same thread cross posting is probably a lot less relevant than remote posting. And since we're going to improve this situation anyway, just gate it on local posting and ignore what task we're currently running on. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 81c4a9d43729..9fdb0cc19bfd 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -68,9 +68,7 @@ void io_msg_ring_cleanup(struct io_kiocb *req) static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) { - if (!target_ctx->task_complete) - return false; - return current != target_ctx->submitter_task; + return target_ctx->task_complete; } static int io_msg_exec_remote(struct io_kiocb *req, task_work_func_t func) From patchwork Wed Jun 5 13:51:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687012 Received: from mail-oa1-f43.google.com (mail-oa1-f43.google.com [209.85.160.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBA371EB56 for ; Wed, 5 Jun 2024 14:19:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597186; cv=none; b=hFFebUWbNv30ZiX8sLrhGyKGHbOL4VabnoVZoKubL9mvBcdROVokIhZtj/bnUrBwN8IUnH5OTh5UHMzebm5ETEyto6o7Wm3+owaghSQtgMZcHnaig/0n5GQqANO4wAksDTfaRCCg+obp3jzPwTBwQJfI+B8oNT++yhG9UYngkDk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597186; c=relaxed/simple; bh=DbCyZ2luzl89+Eh8uaBBmuMYg9vSkfpTwL8ayHGzEBY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Lc7dWDXK7bGuBXEcv3ZdLQtekuSbj/alntISG0jGh1QOER+R/iJbCRAKHIWOOVLZjtOCySxnNwAPlWTYvWDpOc8Gz7FgODxqz3XL4LyekdwfjEPYT2zwG4CCwf4JSLHtTyT4w0pocoQWNjvYvXOXgRVc2MYKzzgr4aQWstJmypk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=03NiykMr; arc=none smtp.client-ip=209.85.160.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="03NiykMr" Received: by mail-oa1-f43.google.com with SMTP id 586e51a60fabf-250932751b9so112014fac.0 for ; Wed, 05 Jun 2024 07:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597183; x=1718201983; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wOQkpDBR8PoqEAncRvt11g7sOP8fj8jrMBX5gQ7aO/o=; b=03NiykMrQwFIW3sNZDnK29vlf8nIg4ujkv8kCPvcYY/QnvDgAkMQ+i+jz1bHbrvQn3 NsF/jm1yTigpmfyWDHGdjtjQSbKcL8LOsn5DXFsS/+DZzzy8/dequMFg/IxPiSLO6MRJ k41hzoWBmw4K75GRVYEZHo/Ge24RIrJQgVhJEr6oCBTLrWxo98VfgJF870Fw5SFb2dnE G3HOto/OI+rY3d3P5RBnBeidZkCUTTxzZiILJf2waq57CD9mzYT8ZIwd8CVyrzwcUiSr 0hxq5sLkU8wuv882rX7efLxPocd3vN2lo8Og20ZiHR2F1IynLwYo6NmOIpbPoz8ceyys u3QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597183; x=1718201983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wOQkpDBR8PoqEAncRvt11g7sOP8fj8jrMBX5gQ7aO/o=; b=Ta9dGgGbVSp5pQ83hNzB/DjfZkoJZvSmC7FyAa1cNNrOg+bqObKuxgGSxE/ZziYN7Q a4jr0prgNiSgvQP3wu/5h4t34koeuMd/t6qseZOhGI1+nME8ww8MvVokACpRGcF4VOhC Ox50oLdO6nQjdHEntaeIP5ZtMuE6gJKlD6ZE5B581x6MhRu4ZIbOrfIq8Eelcy5YeJVh 0ptK8+EJnZUS+2DlQKVIKUu1BmFJZyLX9S50xcQCy38JrPnJ9XpQTWHm9hfTroKIGEtD gUTWNDiLEvpp7f2TuQB3df51VuU2TDY14O0nWCJRvx1grZnBLYr2/ClstsZmSj0Lwo/+ vtcA== X-Gm-Message-State: AOJu0Yw0nsiZzOAKZtBphfR92rtr8Jmyd5X3XmcFs3RgQyLVMorYl17z XG/t+qza9u5tnZP3ZOseURVNcz5J9jMXgQgUbT+n4JE+7Su/bGoQFXL6yx/+ViciQip1uaW52nX k X-Google-Smtp-Source: AGHT+IGeepSW53VyIRITsVkpvjPOs3mUr4KvPSIJdsWpHSFTFlmCT+T8s+90HyE8K7O4aMvnehsoPA== X-Received: by 2002:a05:6870:548f:b0:250:7c91:2bef with SMTP id 586e51a60fabf-25121f10bdfmr3041101fac.2.1717597182468; Wed, 05 Jun 2024 07:19:42 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:41 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 2/9] io_uring: keep track of overflow entry count Date: Wed, 5 Jun 2024 07:51:10 -0600 Message-ID: <20240605141933.11975-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Not needed just yet, but in preparation for factoring these in for the wakeup batching. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 1 + io_uring/io_uring.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index a2227ab7fd16..d795f3f3a705 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -325,6 +325,7 @@ struct io_ring_ctx { unsigned long check_cq; atomic_t cq_wait_nr; atomic_t cq_timeouts; + int nr_overflow; struct wait_queue_head cq_wait; } ____cacheline_aligned_in_smp; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 96f6da0bf5cd..94af56dd5344 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -618,6 +618,7 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool dying) memcpy(cqe, &ocqe->cqe, cqe_size); } list_del(&ocqe->list); + ctx->nr_overflow--; kfree(ocqe); } @@ -724,6 +725,7 @@ static bool io_cqring_event_overflow(struct io_ring_ctx *ctx, u64 user_data, ocqe->cqe.big_cqe[0] = extra1; ocqe->cqe.big_cqe[1] = extra2; } + ctx->nr_overflow++; list_add_tail(&ocqe->list, &ctx->cq_overflow_list); return true; } From patchwork Wed Jun 5 13:51:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687013 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 538192837A for ; Wed, 5 Jun 2024 14:19:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597187; cv=none; b=bkSzNALpLZ8BoBHhIRiopGXvm7C35YPKDkDd6RwxPqQ4sljIe+wpPB5I+XYuuDGK0j58Kv7baNmQxgv6G/0mNtlotR7o6IqULLQyVhXnwTPZuxWv+MqwKHHj16u2simWlg3ZbJcgbLXFi/0t3PwTIcmyzp+nDlXvOxXk53oCHXE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597187; c=relaxed/simple; bh=+jLUBktFTPJ7GG9oqx0iEin3WExYTbOfnclwL0wT+gs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MJ/CzlLVG3NNfl0DwkttRVmXGQOhU/CrXDdWs6bAy7KRpZU7KqCdF+C3S2y/+iWkk0oE1HHo/jHr+8/5p5uFJeMbljKB47NoGQJ/4XktDqRSI4ENEEZGU8gJnkOIChhEM3W3ZRxWqlBLVmQcfCz2rvuOMPRO4owmQWZTCZ/S61w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=SwDFbvOG; arc=none smtp.client-ip=209.85.167.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="SwDFbvOG" Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3d1ffa160b8so254290b6e.1 for ; Wed, 05 Jun 2024 07:19:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597184; x=1718201984; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0Pr/GgWGhD1e0y7bJe1ugy4uzP4rrsfTEk9Z0eKw/mo=; b=SwDFbvOGkrrmyxKFRN8JvCEtuMXDtRJdsro7V/FRMquKQ0KmgLUAMHyUoAHSfzZMtL gFOmd2nVN6ru24jl6vQrgk7Kv4V0CLRnL3g3Ui+McmnuWOLtdMJsM+z3/XzpVtSLMYWN nsTYptQNaimLkzgt8OMy+O6swWcnf4EyblzZLSIslAPDimfoNK92KpdIqzYwsygSJBXV g9wXa7EO85zeu0+DBywbFW6PwFZiGarvW/IEiWWwuX5GcetT1hFRGXhwqKVrc6Rl1+Lf yqbyaQcZxG0SWUeKJI65BxrqCHO4F1LtPxDbtALtxlepA8FNrNfosSlK/5FKcV2UZuWH 4+TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597184; x=1718201984; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0Pr/GgWGhD1e0y7bJe1ugy4uzP4rrsfTEk9Z0eKw/mo=; b=G+ZPkCVtPHjf2XqkpBTAb+QrgswqO6gGOlXuemffMUEfSpYXOjIbvq6rp7EgWhFLKr wOoJ+kHrx7C58qhgMqOdBR60BS805iJyq1EDWiUbm3VWUDMyMhljHhTXsHtHQTTQjBDN pLUZpxTBke9bInW+Yz2wF5Hqf6LMipx1xiYt7pyfeSkgBWj3hZnFopzldEXtglNZxVst ppdOb3HXJobTg7+/0BZlrraOIQYlRFgxQ2sc/cvCVYCXGCG23GZG1hVNWjV4ZZO16KIS GJc/oRLKIoJhBt1FzuQvMji1lNGrLyoAk7IjQZ/utzylkhWQuAixtodhqTVGfBZqm/eG SMUg== X-Gm-Message-State: AOJu0Yz37DVbVzs+7A4pgCFyLL201RJo2UqqEHOSy5BAcDPceGBhZfOf y0sBwcS7eBsk2vhMyewfGesFe66LVuGbCMmKqrOyKjRU+tNmEhzmk0YWfuh3QyVEJYaKNwvMNuK s X-Google-Smtp-Source: AGHT+IHv9C+08cTmcfHjGY1m9OiVLlNxxtb6bBKkIFIYxI2aFDVrHK7rWPWoDUv4oPAFUEMyHvyeeg== X-Received: by 2002:a05:6870:4d18:b0:24c:b092:fd38 with SMTP id 586e51a60fabf-25121cf42eamr3315507fac.1.1717597183583; Wed, 05 Jun 2024 07:19:43 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:43 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 3/9] io_uring: abstract out helpers for DEFER_TASKRUN wakeup batching Date: Wed, 5 Jun 2024 07:51:11 -0600 Message-ID: <20240605141933.11975-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for being able to use these two elsewhere, factor out the helpers that io_req_local_work_add() uses to do wakeup batching. Signed-off-by: Jens Axboe --- io_uring/io_uring.c | 24 +++--------------------- io_uring/io_uring.h | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 21 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 94af56dd5344..499255ef62c7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1103,7 +1103,7 @@ void tctx_task_work(struct callback_head *cb) static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) { struct io_ring_ctx *ctx = req->ctx; - unsigned nr_wait, nr_tw, nr_tw_prev; + unsigned nr_tw, nr_tw_prev; struct llist_node *head; /* See comment above IO_CQ_WAKE_INIT */ @@ -1116,19 +1116,8 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) if (req->flags & (REQ_F_LINK | REQ_F_HARDLINK)) flags &= ~IOU_F_TWQ_LAZY_WAKE; - head = READ_ONCE(ctx->work_llist.first); do { - nr_tw_prev = 0; - if (head) { - struct io_kiocb *first_req = container_of(head, - struct io_kiocb, - io_task_work.node); - /* - * Might be executed at any moment, rely on - * SLAB_TYPESAFE_BY_RCU to keep it alive. - */ - nr_tw_prev = READ_ONCE(first_req->nr_tw); - } + head = io_defer_tw_count(ctx, &nr_tw_prev); /* * Theoretically, it can overflow, but that's fine as one of @@ -1158,14 +1147,7 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) io_eventfd_signal(ctx); } - nr_wait = atomic_read(&ctx->cq_wait_nr); - /* not enough or no one is waiting */ - if (nr_tw < nr_wait) - return; - /* the previous add has already woken it up */ - if (nr_tw_prev >= nr_wait) - return; - wake_up_state(ctx->submitter_task, TASK_INTERRUPTIBLE); + io_defer_wake(ctx, nr_tw, nr_tw_prev); } static void io_req_normal_work_add(struct io_kiocb *req) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index cd43924eed04..fdcf1a2a6b8a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -444,4 +444,48 @@ static inline bool io_has_work(struct io_ring_ctx *ctx) return test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq) || !llist_empty(&ctx->work_llist); } + +/* + * Return first request nr_tw field. Only applicable for users of + * ctx->work_llist, which is DEFER_TASKRUN. Must be called with the RCU read + * lock held. Returns the current task_work count and head of list, if any. + */ +static inline struct llist_node *io_defer_tw_count(struct io_ring_ctx *ctx, + unsigned *nr_tw_prev) +{ + struct llist_node *head = READ_ONCE(ctx->work_llist.first); + + *nr_tw_prev = 0; + if (head) { + struct io_kiocb *first; + + first = container_of(head, struct io_kiocb, io_task_work.node); + /* + * Might be executed at any moment, rely on + * SLAB_TYPESAFE_BY_RCU to keep it alive. + */ + *nr_tw_prev = READ_ONCE(first->nr_tw); + } + + return head; +} + +static inline void io_defer_wake(struct io_ring_ctx *ctx, unsigned nr_tw, + unsigned nr_tw_prev) +{ + struct task_struct *task = READ_ONCE(ctx->submitter_task); + unsigned nr_wait; + + /* add pending overflows, for MSG_RING */ + nr_tw += READ_ONCE(ctx->nr_overflow); + + nr_wait = atomic_read(&ctx->cq_wait_nr); + /* not enough or no one is waiting */ + if (nr_tw < nr_wait) + return; + /* the previous add has already woken it up */ + if (nr_tw_prev >= nr_wait) + return; + wake_up_state(task, TASK_INTERRUPTIBLE); +} #endif From patchwork Wed Jun 5 13:51:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687014 Received: from mail-oa1-f43.google.com (mail-oa1-f43.google.com [209.85.160.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 148D0199A2 for ; Wed, 5 Jun 2024 14:19:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597188; cv=none; b=V2DGlvMePRooO4MQSDBhNHSLEC2fP4+VPqzV1ljS3vGYTNFPX/QzBAGnLZpCUT6fOXO2JpKZvpsrvgeS7U1gOo5PHV2x6eZ6DzbdOuYD8qFPQLEWgHWpmIsnQZ6/A91rDc95bCmpb44Sk8sA176IDn0VCSs8OL6BB1rP8GXaJgY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597188; c=relaxed/simple; bh=vHYUrcMtnuCg2wrAbOmWy2FCmOHpz5C0fyDjYykFbj0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hRPizw29YX0eTYpCQ88Aqxv3VdSyHJkfpduAVufRIP5l6uwhXpVE9D3P6RYA81QpbNi2qJmc5nEMASqo5Wlp6Ag6xI0/h/vwdfUYrwkta8Na/GtezAf+BslC7hGdvnHciMLIn2Spw2q+EpEfBagY4vnyfGRX8MRDv6Ac5GqE7Yo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=HvZYE6zE; arc=none smtp.client-ip=209.85.160.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="HvZYE6zE" Received: by mail-oa1-f43.google.com with SMTP id 586e51a60fabf-24c582673a5so193707fac.2 for ; Wed, 05 Jun 2024 07:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597186; x=1718201986; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ET9g4vu0P/A3wT5hqKQuO00wFCpSRk4smWZdVZtC6W0=; b=HvZYE6zEs2PxK1A3Axdqv/o8/GzhLxWwdnkjTJ+ssPo7AqyaaNcZp5xLKBtNEJuG4b tOhTX47/pMbRjhsHsxsTFQXlt9EO/dpPyc9JsWV30bqtDeqVH2OI9pd//ZHOkWyjiQmE Hlg/14bywvyJMv5JiCSxwdXM5cKVEt5JcFnlCydSeTEM12vJpPXkjL9T+/I/b8q84ulh lDBOIUUaqGxqqZDaG9HZC5RzaoyY4EPptv0mzagmqiFT6FINKxik5z89Wq2Hr8YOD5ys j4M9CG7/s0D3UuA+OLl/hmH+caxkM8oGrUocTVxMK1pHWiS1O4MYBrpj5JB6ncxVOBYu /r0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597186; x=1718201986; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ET9g4vu0P/A3wT5hqKQuO00wFCpSRk4smWZdVZtC6W0=; b=obfssl+1BnkMYK0ld6CvG/PG2dHlIxoyLvytv8BRHmZ9gD1IHAM8daCuPsYCJ3X5p2 3rXADmvCnJS6Rk9+H4IvMMmuu1iq6cbg3oVtfP/1xY/uLWh3m9zy8rouH2Xex2t/EJ5a 6piBfx9HfPL4hxUsBbIS2p2Hk72FqXlisEECh/2lfomfOUh7ugVIgzZfSEGuhU278CsM /CMpTdCGjPsAYuxFP6hG9occQSuCBBgqDwouGEYUQ9weGS5y5E08cc1sg76eWPs1VWAh mXOt1aUn0alVLZA9joeoBpEoaC/1VuXRCiGnZhJOj5SEWJY8g/MVQaYVMNxK57LBH53j 6RJA== X-Gm-Message-State: AOJu0YznfEq7yH7DQwspfU84p+wh+VcusLnyIbPIXmtk3NZ6e7h1f/jv X6X3r/jnHC2cSVyeVPW1pDrFPZBKUN+LB1rZ6fiVC4rXnTMpp/XNHSn5qjM6lu3xqJ9NHfhqGa1 z X-Google-Smtp-Source: AGHT+IFC3CEAmLGrbI6oTWiB6bqIVyxAr09Sz8ovhGrAiTjOOVn7rqYj2Ouhgk2uNCgdPd1vpq3Nuw== X-Received: by 2002:a05:6871:2882:b0:250:826d:5202 with SMTP id 586e51a60fabf-25122078d9amr2800548fac.3.1717597185473; Wed, 05 Jun 2024 07:19:45 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:44 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 4/9] io_uring/msg_ring: avoid double indirection task_work for data messages Date: Wed, 5 Jun 2024 07:51:12 -0600 Message-ID: <20240605141933.11975-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If IORING_SETUP_DEFER_TASKRUN is set, then we can't post CQEs remotely to the target ring. Instead, task_work is queued for the target ring, which is used to post the CQE. To make matters worse, once the target CQE has been posted, task_work is then queued with the originator to fill the completion. This obviously adds a bunch of overhead and latency. Instead of relying on generic kernel task_work for this, fill an overflow entry on the target ring and flag it as such that the target ring will flush it. This avoids both the task_work for posting the CQE, and it means that the originator CQE can be filled inline as well. In local testing, this reduces the latency on the sender side by 5-6x. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 88 ++++++++++++++++++++++++++++----------------- 1 file changed, 56 insertions(+), 32 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 9fdb0cc19bfd..2b649087fe5c 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -87,38 +87,62 @@ static int io_msg_exec_remote(struct io_kiocb *req, task_work_func_t func) return IOU_ISSUE_SKIP_COMPLETE; } -static void io_msg_tw_complete(struct callback_head *head) +static struct io_overflow_cqe *io_alloc_overflow(struct io_ring_ctx *target_ctx) { - struct io_msg *msg = container_of(head, struct io_msg, tw); - struct io_kiocb *req = cmd_to_io_kiocb(msg); - struct io_ring_ctx *target_ctx = req->file->private_data; - int ret = 0; - - if (current->flags & PF_EXITING) { - ret = -EOWNERDEAD; - } else { - u32 flags = 0; - - if (msg->flags & IORING_MSG_RING_FLAGS_PASS) - flags = msg->cqe_flags; - - /* - * If the target ring is using IOPOLL mode, then we need to be - * holding the uring_lock for posting completions. Other ring - * types rely on the regular completion locking, which is - * handled while posting. - */ - if (target_ctx->flags & IORING_SETUP_IOPOLL) - mutex_lock(&target_ctx->uring_lock); - if (!io_post_aux_cqe(target_ctx, msg->user_data, msg->len, flags)) - ret = -EOVERFLOW; - if (target_ctx->flags & IORING_SETUP_IOPOLL) - mutex_unlock(&target_ctx->uring_lock); + bool is_cqe32 = target_ctx->flags & IORING_SETUP_CQE32; + size_t cqe_size = sizeof(struct io_overflow_cqe); + struct io_overflow_cqe *ocqe; + + if (is_cqe32) + cqe_size += sizeof(struct io_uring_cqe); + + ocqe = kmalloc(cqe_size, GFP_ATOMIC | __GFP_ACCOUNT); + if (!ocqe) + return NULL; + + if (is_cqe32) + ocqe->cqe.big_cqe[0] = ocqe->cqe.big_cqe[1] = 0; + + return ocqe; +} + +/* + * Entered with the target uring_lock held, and will drop it before + * returning. Adds a previously allocated ocqe to the overflow list on + * the target, and marks it appropriately for flushing. + */ +static void io_msg_add_overflow(struct io_msg *msg, + struct io_ring_ctx *target_ctx, + struct io_overflow_cqe *ocqe, int ret, + u32 flags) + __releases(&target_ctx->completion_lock) +{ + if (list_empty(&target_ctx->cq_overflow_list)) { + set_bit(IO_CHECK_CQ_OVERFLOW_BIT, &target_ctx->check_cq); + atomic_or(IORING_SQ_TASKRUN, &target_ctx->rings->sq_flags); } - if (ret < 0) - req_set_fail(req); - io_req_queue_tw_complete(req, ret); + ocqe->cqe.user_data = msg->user_data; + ocqe->cqe.res = ret; + ocqe->cqe.flags = flags; + target_ctx->nr_overflow++; + list_add_tail(&ocqe->list, &target_ctx->cq_overflow_list); + spin_unlock(&target_ctx->completion_lock); + wake_up_state(target_ctx->submitter_task, TASK_INTERRUPTIBLE); +} + +static int io_msg_fill_remote(struct io_msg *msg, unsigned int issue_flags, + struct io_ring_ctx *target_ctx, u32 flags) +{ + struct io_overflow_cqe *ocqe; + + ocqe = io_alloc_overflow(target_ctx); + if (!ocqe) + return -ENOMEM; + + spin_lock(&target_ctx->completion_lock); + io_msg_add_overflow(msg, target_ctx, ocqe, msg->len, flags); + return 0; } static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) @@ -135,12 +159,12 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) if (target_ctx->flags & IORING_SETUP_R_DISABLED) return -EBADFD; - if (io_msg_need_remote(target_ctx)) - return io_msg_exec_remote(req, io_msg_tw_complete); - if (msg->flags & IORING_MSG_RING_FLAGS_PASS) flags = msg->cqe_flags; + if (io_msg_need_remote(target_ctx)) + return io_msg_fill_remote(msg, issue_flags, target_ctx, flags); + ret = -EOVERFLOW; if (target_ctx->flags & IORING_SETUP_IOPOLL) { if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) From patchwork Wed Jun 5 13:51:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687015 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6414F199A2 for ; Wed, 5 Jun 2024 14:19:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597190; cv=none; b=NXv6KBYtBqqKZNiXv7z/udcwntIYs1nuyuAch2RPS0m1S3ZJvSacmlNBBy+SlnwWTQkc+i6lGQfgMYKpN+VLppXcV6FT/mW173AJBGJ3H/AovjmF1Nww6LT0zX8+DukfOIiz4fAdA5V8m0gtQ/xqJ7LnAsyb+3I+iZ9gdMBqoRk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597190; c=relaxed/simple; bh=eILMDD2M1md0Mdp9L+ODJ6EXRvVNtT3efZfywN1wms0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JdVDtWG8ja/MK9971+K87Gg4oyyuQpk0BB+vqTO3b3hmLrV6bWiDwK0Mp2WENvlC0DuKcgqzdAA2BSwRpcj++JYse3nN3V0blDqx8dVHw12CLn80tWOPpuIdERIBBmrb80NrFlNZ4fRfm9KZU81PmbIj8/r5Rk1Wl3ck/GQbSNc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=sRgfyWaG; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="sRgfyWaG" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3d1ffc2d76aso234479b6e.3 for ; Wed, 05 Jun 2024 07:19:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597188; x=1718201988; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c76WKEiXpJFnCssoMjw+Z6qRfmEcY3UQINBr/vE+k24=; b=sRgfyWaGz/FZ6C9W/b58mohhPnyGRR2sCstpxqniHnao8gxDbAkyDlfmj9KU7fuLrf K8plQa/pS3rSDy7scKM4T846q0kDrVwVDj4fmtLjaZ3M3keEcttEL9W6Um/uW9BFmvlN gfbEVkRepEscexKzrz2vzybesUmCie8v4mwY1LpHR/9sOZgXmvk3DbxJWA4pRyq0PLsT Om15ttMKcWtnUw6D3JRGHEcGPnz4SV2wtVPu29xrEDxTp7kxjGuOdQ0aOg8lxeE2Tizs p1tOLMyzINkjsrJBWkovM6vJK3kMvG5xrZ1Zz64CXCJu0a3lr1onIWc88/9iR6qn74u6 g8bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597188; x=1718201988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c76WKEiXpJFnCssoMjw+Z6qRfmEcY3UQINBr/vE+k24=; b=ra5akSjtcbaKtRGbJ16Y41/8LVupci6e8pgyJXmV4az8jyTVTvonAjDI5ppKAkqiuS fcXT4dA4lDVBRhqIydnZ3LNVdJI6wr0j4/ayuvs2UYl1hVVPuIJZZioFNM0CzwGnnGTM K6Gk7cEolSyxG8vEp/vn+ya/a3cF1hjg6DpIzh0cPche+jIN7OePETtH3YyzCKf+2ZIr /dZcbpMMLb1JjPp2/MobWldzWDLXJ/iXwOt3391+p/mScQac2jdKjtqGGPU9PknEjwo+ boyLKKbgS9WzXLnYontoSs/1dNNIX26Zl//dp/EuuNAIp93bGkCV/H/xMjVIDUfFmaR8 B05A== X-Gm-Message-State: AOJu0Ywprf5CSEQ6g11FpxSU11dCS1F7saI2jioeDopa9vm0lZT1ICaQ 3teEVyyT6IP51sk/agCMb2+K+D/n+XeuhiverB9FKgmRi/iDlL0Kc+Ll7YMepZbajufLwcKNflC Z X-Google-Smtp-Source: AGHT+IHlGyJXLrHMvgXlt9jdjXGo6JA0KLFJR56Aly6wiQ85+gN66DWS0nroTJlUIn0ts/bNzbJCRQ== X-Received: by 2002:a05:6870:9689:b0:250:6be3:3406 with SMTP id 586e51a60fabf-2512207ac4fmr2956894fac.3.1717597187245; Wed, 05 Jun 2024 07:19:47 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:46 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 5/9] io_uring/msg_ring: avoid double indirection task_work for fd passing Date: Wed, 5 Jun 2024 07:51:13 -0600 Message-ID: <20240605141933.11975-6-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Like what was done for MSG_RING data passing avoiding a double task_work roundtrip for IORING_SETUP_DEFER_TASKRUN, implement the same model for fd passing. File descriptor passing is separately locked anyway, so the only remaining issue is CQE posting, just like it was for data passing. And for that, we can use the same approach. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 57 ++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 26 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 2b649087fe5c..1ee89bdbbb5b 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -71,22 +71,6 @@ static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) return target_ctx->task_complete; } -static int io_msg_exec_remote(struct io_kiocb *req, task_work_func_t func) -{ - struct io_ring_ctx *ctx = req->file->private_data; - struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); - struct task_struct *task = READ_ONCE(ctx->submitter_task); - - if (unlikely(!task)) - return -EOWNERDEAD; - - init_task_work(&msg->tw, func); - if (task_work_add(task, &msg->tw, TWA_SIGNAL)) - return -EOWNERDEAD; - - return IOU_ISSUE_SKIP_COMPLETE; -} - static struct io_overflow_cqe *io_alloc_overflow(struct io_ring_ctx *target_ctx) { bool is_cqe32 = target_ctx->flags & IORING_SETUP_CQE32; @@ -227,17 +211,38 @@ static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flag return ret; } -static void io_msg_tw_fd_complete(struct callback_head *head) +static int io_msg_install_remote(struct io_kiocb *req, unsigned int issue_flags, + struct io_ring_ctx *target_ctx) { - struct io_msg *msg = container_of(head, struct io_msg, tw); - struct io_kiocb *req = cmd_to_io_kiocb(msg); - int ret = -EOWNERDEAD; + struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); + struct io_overflow_cqe *ocqe = NULL; + int ret; - if (!(current->flags & PF_EXITING)) - ret = io_msg_install_complete(req, IO_URING_F_UNLOCKED); - if (ret < 0) - req_set_fail(req); - io_req_queue_tw_complete(req, ret); + if (!(msg->flags & IORING_MSG_RING_CQE_SKIP)) { + ocqe = io_alloc_overflow(target_ctx); + if (!ocqe) + return -ENOMEM; + } + + if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) { + kfree(ocqe); + return -EAGAIN; + } + + ret = __io_fixed_fd_install(target_ctx, msg->src_file, msg->dst_fd); + mutex_unlock(&target_ctx->uring_lock); + + if (ret >= 0) { + msg->src_file = NULL; + req->flags &= ~REQ_F_NEED_CLEANUP; + if (ocqe) { + spin_lock(&target_ctx->completion_lock); + io_msg_add_overflow(msg, target_ctx, ocqe, ret, 0); + return 0; + } + } + kfree(ocqe); + return ret; } static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) @@ -262,7 +267,7 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) } if (io_msg_need_remote(target_ctx)) - return io_msg_exec_remote(req, io_msg_tw_fd_complete); + return io_msg_install_remote(req, issue_flags, target_ctx); return io_msg_install_complete(req, issue_flags); } From patchwork Wed Jun 5 13:51:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687016 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27C271BC3C for ; Wed, 5 Jun 2024 14:19:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597192; cv=none; b=MavDvkDi9mGHqZ/2sr0r8up2O0UzP6l51d2nRn6EmDXRyHQnrOaLcJPAXLJsXp99ObkAEvgcVB/J9Uw1QtWLCTWyWV2zrkOhU2CBpcPtQBCvbg56T5GyUMPnoFvFuP/oo2MiYPogoQ8kOb9YnKgDUiD8zABcwbVXyRNl/kvx4eY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597192; c=relaxed/simple; bh=+sGrQpS/TqRgBayPAV7HC0UunlbFm9y/mjMTsCzMl2E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BBgtSMw+5HlA+tYyOX3fziSepAe5bwiNlIQLY7LPyB7JPoQmo9B8SnZUIIaOd9S1ZTX0M7PPMqDL4i9j72MI0Evo8jsg5JxXZ9aJ/4B0vXGLIbMEMGw8kcIE+OREKRLdkEvMCKMJuVWaHJqZsd4Hr6unFovk3jC/93y3GZjQMko= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=nZNV+Cxj; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="nZNV+Cxj" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3d1dbf0d2deso686280b6e.0 for ; Wed, 05 Jun 2024 07:19:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597188; x=1718201988; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KdSk0OgXTLJoOarsefZ+rdTaqGB+4c8ZJAicy/5Kyq8=; b=nZNV+CxjL3X2KJDwKVlcsUlqw08tBnw9fgBYxa1Upt/ixlDPED9uEA18Q8KBW3+ffw 0C6i5uUaEuM8vksx1pXOUrjq5gjEoNfZwAAYwuO38upJOu/dO4ksLMxYy80CJ5PamR8U G7uBwgIcvyn4Y9lX1hFnpGbIQiKhnt4YK7KpRdDqtVxihPtRRw/Sb87PrkMd+zEI+i1i QqEUVE6JRKJ1gIxBHUQ/cEMCPVd3Z05nmxxGRNQbORLhNHBdwZzen0rK3P2TGkg9p9Y3 +43ZuXs/P9ys8hOwL1DvPF2GId3WunbGZF88i4yYKsyMXPeoILAW29qKGeufhE6im8qi Lz7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597188; x=1718201988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KdSk0OgXTLJoOarsefZ+rdTaqGB+4c8ZJAicy/5Kyq8=; b=bWl+SsbwN1PQi9LfSNYnvmYL9MXhdgMxtgXwr2pi/PAx0vEQptxrCYZ9Ks5uGicUNd 6k4W9xU+UyevtOpKg7RdIqCh6BF6oKQVaQPW1eGJx6mmhWK9G/5T9ObVTq39wNH4qUuX kEx3l2iqL2fLdMHfBx5xAQohlQgSx3/sB1o1CzXr8BqLj2A8HL1vm+s4U2eChP2d+Trh oCU90S41Dy1t259Ydy3/geNQHLrPPI2AuXoxZgzBQG3IsBFCpvYtOxamjUNs6Or75fcY 6oeEXvGRLETJ2zJ7YYu2tC2zWhjq7VcsBknGfvvD4ZffNLJk5J5yZNnPFeQ2fLNwXeE9 ORnw== X-Gm-Message-State: AOJu0YzN/GdwcU+5NAOCpeADJSNo/TPlxcLJxkGEfHK9dCKxYW8KZ3f4 ZG00U+HYBZ23MP/8VjZ/EziMIsFBUZXrKYEMgyOyAsxFKSabj2LXI3kCVOOrLewZHlrYYDSKwJy N X-Google-Smtp-Source: AGHT+IFMhnrJHSuDv4a2Px3hpNxUJIfLc0co5dJKch4g0zc3JjzgI7na/L5QHJue2BY4Bgt5pqYZeQ== X-Received: by 2002:a05:6870:4d18:b0:24c:b092:fd38 with SMTP id 586e51a60fabf-25121cf42eamr3315780fac.1.1717597188389; Wed, 05 Jun 2024 07:19:48 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:47 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 6/9] io_uring/msg_ring: add an alloc cache for CQE entries Date: Wed, 5 Jun 2024 07:51:14 -0600 Message-ID: <20240605141933.11975-7-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 io_uring accounts the memory allocated, which is quite expensive. Wrap the allocation and frees in the provided alloc cache framework. The target ctx needs to be locked anyway for posting the overflow entry, so just move the overflow alloc inside that section. Flushing the entries has it locked as well, so io_cache_alloc_free() can be used. In a simple test, most of the overhead of DEFER_TASKRUN message passing ends up being accounting for allocation and free, and with this change it's completely gone. Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 7 ++++ io_uring/io_uring.c | 7 +++- io_uring/msg_ring.c | 76 +++++++++++++++++++++++----------- io_uring/msg_ring.h | 3 ++ 4 files changed, 68 insertions(+), 25 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index d795f3f3a705..c7f3a330482d 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -350,6 +350,13 @@ struct io_ring_ctx { struct io_alloc_cache futex_cache; #endif + /* + * Unlike the other caches, this one is used by the sender of messages + * to this ring, not by the ring itself. As such, protection for this + * cache is under ->completion_lock, not ->uring_lock. + */ + struct io_alloc_cache msg_cache; + const struct cred *sq_creds; /* cred used for __io_sq_thread() */ struct io_sq_data *sq_data; /* if using sq thread polling */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 499255ef62c7..3c77d96fc6d7 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -95,6 +95,7 @@ #include "futex.h" #include "napi.h" #include "uring_cmd.h" +#include "msg_ring.h" #include "memmap.h" #include "timeout.h" @@ -316,6 +317,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) ret |= io_alloc_cache_init(&ctx->uring_cache, IO_ALLOC_CACHE_MAX, sizeof(struct uring_cache)); ret |= io_futex_cache_init(ctx); + ret |= io_msg_cache_init(ctx); if (ret) goto err; init_completion(&ctx->ref_comp); @@ -352,6 +354,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); io_alloc_cache_free(&ctx->uring_cache, kfree); io_futex_cache_free(ctx); + io_msg_cache_free(ctx); kfree(ctx->cancel_table.hbs); kfree(ctx->cancel_table_locked.hbs); xa_destroy(&ctx->io_bl_xa); @@ -619,7 +622,8 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool dying) } list_del(&ocqe->list); ctx->nr_overflow--; - kfree(ocqe); + if (!io_alloc_cache_put(&ctx->msg_cache, ocqe)) + kfree(ocqe); } if (list_empty(&ctx->cq_overflow_list)) { @@ -2556,6 +2560,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) io_alloc_cache_free(&ctx->rw_cache, io_rw_cache_free); io_alloc_cache_free(&ctx->uring_cache, kfree); io_futex_cache_free(ctx); + io_msg_cache_free(ctx); io_destroy_buffers(ctx); mutex_unlock(&ctx->uring_lock); if (ctx->sq_creds) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 1ee89bdbbb5b..a33228f8c364 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -11,6 +11,7 @@ #include "io_uring.h" #include "rsrc.h" #include "filetable.h" +#include "alloc_cache.h" #include "msg_ring.h" @@ -72,20 +73,28 @@ static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) } static struct io_overflow_cqe *io_alloc_overflow(struct io_ring_ctx *target_ctx) + __acquires(&target_ctx->completion_lock) { - bool is_cqe32 = target_ctx->flags & IORING_SETUP_CQE32; - size_t cqe_size = sizeof(struct io_overflow_cqe); struct io_overflow_cqe *ocqe; - if (is_cqe32) - cqe_size += sizeof(struct io_uring_cqe); + spin_lock(&target_ctx->completion_lock); + + ocqe = io_alloc_cache_get(&target_ctx->msg_cache); + if (!ocqe) { + bool is_cqe32 = target_ctx->flags & IORING_SETUP_CQE32; + size_t cqe_size = sizeof(struct io_overflow_cqe); - ocqe = kmalloc(cqe_size, GFP_ATOMIC | __GFP_ACCOUNT); - if (!ocqe) - return NULL; + if (is_cqe32) + cqe_size += sizeof(struct io_uring_cqe); + + ocqe = kmalloc(cqe_size, GFP_ATOMIC | __GFP_ACCOUNT); + if (!ocqe) + return NULL; - if (is_cqe32) - ocqe->cqe.big_cqe[0] = ocqe->cqe.big_cqe[1] = 0; + /* just init at alloc time, won't change */ + if (is_cqe32) + ocqe->cqe.big_cqe[0] = ocqe->cqe.big_cqe[1] = 0; + } return ocqe; } @@ -121,12 +130,13 @@ static int io_msg_fill_remote(struct io_msg *msg, unsigned int issue_flags, struct io_overflow_cqe *ocqe; ocqe = io_alloc_overflow(target_ctx); - if (!ocqe) - return -ENOMEM; + if (ocqe) { + io_msg_add_overflow(msg, target_ctx, ocqe, msg->len, flags); + return 0; + } - spin_lock(&target_ctx->completion_lock); - io_msg_add_overflow(msg, target_ctx, ocqe, msg->len, flags); - return 0; + spin_unlock(&target_ctx->completion_lock); + return -ENOMEM; } static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) @@ -216,17 +226,17 @@ static int io_msg_install_remote(struct io_kiocb *req, unsigned int issue_flags, { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); struct io_overflow_cqe *ocqe = NULL; - int ret; + int ret = -ENOMEM; + + if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) + return -EAGAIN; if (!(msg->flags & IORING_MSG_RING_CQE_SKIP)) { ocqe = io_alloc_overflow(target_ctx); - if (!ocqe) - return -ENOMEM; - } - - if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) { - kfree(ocqe); - return -EAGAIN; + if (unlikely(!ocqe)) { + mutex_unlock(&target_ctx->uring_lock); + goto err; + } } ret = __io_fixed_fd_install(target_ctx, msg->src_file, msg->dst_fd); @@ -236,12 +246,15 @@ static int io_msg_install_remote(struct io_kiocb *req, unsigned int issue_flags, msg->src_file = NULL; req->flags &= ~REQ_F_NEED_CLEANUP; if (ocqe) { - spin_lock(&target_ctx->completion_lock); io_msg_add_overflow(msg, target_ctx, ocqe, ret, 0); return 0; } } - kfree(ocqe); + if (ocqe) { +err: + spin_unlock(&target_ctx->completion_lock); + kfree(ocqe); + } return ret; } @@ -321,3 +334,18 @@ int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags) io_req_set_res(req, ret, 0); return IOU_OK; } + +int io_msg_cache_init(struct io_ring_ctx *ctx) +{ + size_t size = sizeof(struct io_overflow_cqe); + + if (ctx->flags & IORING_SETUP_CQE32) + size += sizeof(struct io_uring_cqe); + + return io_alloc_cache_init(&ctx->msg_cache, IO_ALLOC_CACHE_MAX, size); +} + +void io_msg_cache_free(struct io_ring_ctx *ctx) +{ + io_alloc_cache_free(&ctx->msg_cache, kfree); +} diff --git a/io_uring/msg_ring.h b/io_uring/msg_ring.h index 3987ee6c0e5f..94f5716d522e 100644 --- a/io_uring/msg_ring.h +++ b/io_uring/msg_ring.h @@ -3,3 +3,6 @@ int io_msg_ring_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_msg_ring(struct io_kiocb *req, unsigned int issue_flags); void io_msg_ring_cleanup(struct io_kiocb *req); + +int io_msg_cache_init(struct io_ring_ctx *ctx); +void io_msg_cache_free(struct io_ring_ctx *ctx); From patchwork Wed Jun 5 13:51:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687017 Received: from mail-oa1-f47.google.com (mail-oa1-f47.google.com [209.85.160.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A45A71C686 for ; Wed, 5 Jun 2024 14:19:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597193; cv=none; b=u4QQOTWxMsN/2WUcgJHLW4ZV6AAaC2Eu7YFB+AOxlJKYoPt7zLlke29N8aNUlvSnxxwyKB+AxKy1m1r66tb2uJuf+hoVjk2ktETuYIz8eZluOfigBrkfrdJyqEvTq0jcT/Zdd3kwRJZm6O+3nacirzBQGxNWiO16ewkp7m4KUTM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597193; c=relaxed/simple; bh=uwQF30IimSlEKlp8+Eiz1XztFwx4xSw6qoS56QzvNNk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UE7mru5DbNlcU3BJo9OLWJw3vGEtUmknmjKNXmJ7El7E7WV5MD4zlgkstdhF84dCPPYEFczWuRRaiCGmLJ33MkY1W/mG21ky4Oxnl2DrlKpmE56aZL7c1Rjzg6p2VxuT5FHknIJCz3h7wm28CvaYGBS4scY+AzO2GNofLxusfwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=i0xNEMEx; arc=none smtp.client-ip=209.85.160.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="i0xNEMEx" Received: by mail-oa1-f47.google.com with SMTP id 586e51a60fabf-2511c3a4061so32107fac.3 for ; Wed, 05 Jun 2024 07:19:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597190; x=1718201990; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WgrIC9T62eJ/Kd42KI92TTHwOB/mMkk2A1Xe2LS7R8E=; b=i0xNEMExyu1ChRYOJCFdM0IYOefJ7Dr8QwNE+ENRu9AFMo4+Jy3RJ9Wnm/3KvQAAez e3gMuHX/ooTTtgelgS5HchAvr9vAhQYwQWM9J7e2EkdLt2hBhSyd9fNtphkChSZEBihh 21yiN5gUfuv7F69cbNm/HkYOB2uHoPMXcLMyKNim2Y82df2b/m3J/vgQY4rerelA+Ezu LVLVw5rLuhzXp0lM/I0zBSGUqd5ddjjbzB67ddFt4Bb30m7fkljdo5bsTu+rLgHNxeKF AdpzOmQkbj9KnXIAL6cQkR/PIb1qye2rhJJSWqt/Y7z4SpKIx/AFHQ6WWoDKoXe5gDWd LlMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597190; x=1718201990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WgrIC9T62eJ/Kd42KI92TTHwOB/mMkk2A1Xe2LS7R8E=; b=KDdO7/SOp2cbwBLYdEM4rPZeBT1LuBgTxzmca7qmQPsGKYnZTLhzBX1N2RSaBOL1A0 Xo9VKgBYmsdZIm4rL6P5+2Fo7ZLjV23hPwav1rfuyXcaj0A5HywBQt/RZW45seiYZa1H PheMRxHeQA6YGPhSYWERYDRsO2DAKvzeqA6AAiQptud0LktPEAD77tDRBOKE21GXmy9c k+b0cvmj7qYofuyWBjvyi3T7hKiYVIkx05UC5u7i/2TtVBwmvXlWnwi5iu0KA9u/bdal cjvqu4gQbYa/ooxbt3BNeGi//Tp3jJ52hrmybKh7j3ZHfFWA2YiGOjGRRQv1PQWHQZDI YgaA== X-Gm-Message-State: AOJu0YyrYbF2TPF6dRw5N1ZrdrN/2DTqkz5skFCBkQ2mrtb2VrM4p8zB UBXhv/u9Ag4S1yop6psbATcrRVeeRf2TtSot5C551gC+KgLYFUXMLQ/ERGH5eMQQpjwd+wAYo1L N X-Google-Smtp-Source: AGHT+IF/Y2vPLHKUd7t9OppUqhtnbutzFk+SFKeLZrG2Gw79sS0vj3+r/+BntSDB56v5T/SUzSeipQ== X-Received: by 2002:a05:6870:a981:b0:24f:c055:fc52 with SMTP id 586e51a60fabf-25121856b56mr2879477fac.0.1717597190212; Wed, 05 Jun 2024 07:19:50 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:48 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 7/9] io_uring/msg_ring: remove callback_head from struct io_msg Date: Wed, 5 Jun 2024 07:51:15 -0600 Message-ID: <20240605141933.11975-8-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 This is now unused, get rid of it. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 1 - 1 file changed, 1 deletion(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index a33228f8c364..9a7c63f38c46 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -22,7 +22,6 @@ struct io_msg { struct file *file; struct file *src_file; - struct callback_head tw; u64 user_data; u32 len; u32 cmd; From patchwork Wed Jun 5 13:51:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687018 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 722591D524 for ; Wed, 5 Jun 2024 14:19:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597194; cv=none; b=XDjVzetg6ksqNi0MGn/0sGDVFf1BOJ5pM4/Rj9J8n5UeNyGNXctheb8n0yH7qVZgrLumBITcSPOGtD0ULBCBDZnt7TgXr8RzqIoxIOpY2JM+FuQqJYQWXoiU68hyahifSuJOFF03GFGD/NcnDQ9pf31DjufLGKIQnPzf70W5u4A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597194; c=relaxed/simple; bh=ZxUqacrgcB+gIX50cKma8/WkESJan7gE3fo++Ia6ESc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QOR9QSiwHVeOrfLHU8GA51J6DcXJJKeNFdzmQvUN40FnSt91ZWCup5W4wNc4Dxz3M9M2pTPVXoQZQhUxkSZC977T/Ih7TzyDiuI7iJmDkqWFON4HW9e2eUEDcejugpYatfRtniD+uC0sKms9XsOMkNUhgrdfo793kh4G1QENojg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ju2enn2B; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ju2enn2B" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-6f93fe6d11aso74195a34.2 for ; Wed, 05 Jun 2024 07:19:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597192; x=1718201992; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vvVkg3/KW9P0mu9B789CugsGlim6x0bnl0gXJoooKGI=; b=ju2enn2BhhRMK3JcCTuMng1OJPelppH/gyd7bdjVuBehC2X44G9/2xpKB+rvTLACo2 IweSZz0TwMrFChAVaWcvFF559iHaaSfosFQSsBkbJ7+KMZUe8KtxoFeBZHZYP9AkP4dA hWPwbIj8HpvVKa3lqT9GksoAvdp/WIZjuivdrHwor2ifX0RZTUQ3nCVO2NiqkaGF340w 4kKmJd71AJL0HIGYSqDGJSVqIMhYInrBbGZ+jnJVbJfr4IoXyh/zzJuwgQjxho9G+RHq lJ4JQ5YAqdYE9CbFNhxZVK3B+Gk71Xo1JX4wy1vmwRUJ4hWkxfOrmDQJHZSgI76Anlyn b14w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597192; x=1718201992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vvVkg3/KW9P0mu9B789CugsGlim6x0bnl0gXJoooKGI=; b=F5g7ydSqOUoImJxuRWr/cLUlvat+RzwEvPuY8BOKNnc9PFsruv7J6gClehXHRiHmsR OtU7MUkMFLVhi3FYEzMwksyddhQtMiONO8O9EsKP6fKUZ2pKKQFedrlMeAOp1mRpfDmb uSECc1y/rg6hgvnuOloJGkTnjzs8NJv7n2lF4auqKcsAUJfeVoXW65gVXLcAYLy4gp65 /AqtHMdezXZA8A5PCDQUuSEzH5Q9AZ2SGSnbqcO+JjI5Vf/XrSqBdVVITzDX6jJ2vSl/ Of4SDc9GtfTaT6N0LkIWHpBpJTqXXRWH3A6jaidYX0jiGvlgTM3rvuNFejT2jsyAACPL fMlA== X-Gm-Message-State: AOJu0YzKg9CVJlBBgujI2Ydau4ij3IYUl1hc8DpC2ajdFHokP1Hzhnl1 SwUkn7P4eYFjSOZyM/UXyU1nFNcdyOHuUfi4uIstH2305byAMSeiZ0IJR1dW9amQpp8sFmSRyqh W X-Google-Smtp-Source: AGHT+IEp+Gn0OJtHxlgK0ZaEj0MFNY8oX6fLSDxjMLd2QQpjWB67yHb8a4gRdyLmic+2Kflmqi1QCg== X-Received: by 2002:a05:6871:b06:b0:24f:e599:9168 with SMTP id 586e51a60fabf-25121cda5dcmr2663221fac.1.1717597191890; Wed, 05 Jun 2024 07:19:51 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:50 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 8/9] io_uring/msg_ring: add basic wakeup batch support Date: Wed, 5 Jun 2024 07:51:16 -0600 Message-ID: <20240605141933.11975-9-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Factor in the number of overflow entries waiting, on both the msg ring and local task_work add side. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index 9a7c63f38c46..eeca1563ceed 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -109,6 +109,8 @@ static void io_msg_add_overflow(struct io_msg *msg, u32 flags) __releases(&target_ctx->completion_lock) { + unsigned nr_prev, nr_wait; + if (list_empty(&target_ctx->cq_overflow_list)) { set_bit(IO_CHECK_CQ_OVERFLOW_BIT, &target_ctx->check_cq); atomic_or(IORING_SQ_TASKRUN, &target_ctx->rings->sq_flags); @@ -117,10 +119,14 @@ static void io_msg_add_overflow(struct io_msg *msg, ocqe->cqe.user_data = msg->user_data; ocqe->cqe.res = ret; ocqe->cqe.flags = flags; - target_ctx->nr_overflow++; + nr_prev = target_ctx->nr_overflow++; list_add_tail(&ocqe->list, &target_ctx->cq_overflow_list); spin_unlock(&target_ctx->completion_lock); - wake_up_state(target_ctx->submitter_task, TASK_INTERRUPTIBLE); + rcu_read_lock(); + io_defer_tw_count(target_ctx, &nr_wait); + nr_prev += nr_wait; + io_defer_wake(target_ctx, nr_prev + 1, nr_prev); + rcu_read_unlock(); } static int io_msg_fill_remote(struct io_msg *msg, unsigned int issue_flags, From patchwork Wed Jun 5 13:51:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13687019 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E313B1DA58 for ; Wed, 5 Jun 2024 14:19:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597196; cv=none; b=ckpSVFhsNdlHHoZZ44RNVlDl/RFS+MTSMmgesiggkfmM5pBbeQgkWOct6RcEUR91qwr/UMcTbgkvA9O3VRBeCIPf1gxXcBKmHXb2mrhdgvPe/Yi2KwNzHjH4W3x2j1eLRuysz46w827MhgRzxd63hO4d5IDCHrVvIz4LfSDpN+U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717597196; c=relaxed/simple; bh=HBWZSW6GaRg+Bo8l1b/YRXXah+qHcJJASJWS1lbRkjw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N1yq79GPR3kv9crfGInnkbioWg8+xvetrjbdKpU8+Bsz901sLByi5V6uhVZPh03DvFz7tp1IUZb9wlfwuZXEFEPVPjm3HhyrM1BW7Zf3G5iCjicIShRI0dGnAgGzf0kKcBKBQwugirU4HVKIneDI4RP9rxfhJgxxEDtTpCwDYwM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=o2nGBF1+; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="o2nGBF1+" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-6f94ab45376so11279a34.2 for ; Wed, 05 Jun 2024 07:19:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1717597193; x=1718201993; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D8v+/zxeBFDAYrEYgAGJcLmN2TRS+9P3Sfsn7WN/CWE=; b=o2nGBF1+zQWJN2R1Ec3kg2lcTxS5IBp4G1Jt1As3i8KjwewgBwIK+jj831HB4mKiI8 s9S1eeRZUSmyWadI+QSXOCNqkxLoizt3PBbhFksrQ7NxP9C3ZLA57HPboH123r0JLzwC MmaxGrfTjnXrxmBhfZcj8rtjlq5llETkb+BU3Shp4V5RmEjXCbAv2ZXfgynIq3zf9jcL jaZGw7U9WsCAG+d9UX48s5wil3IbGGL3AQOFRQ5BhQPua50UJ0YgeA9vn5VRDrkrsScc VMUriLU3Lj6u4NBOk8M8Gt7EULVzpDL8obZ1SqXKQjygC72r2FnTF+PeBRvcLP2zMCQA a71Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717597193; x=1718201993; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D8v+/zxeBFDAYrEYgAGJcLmN2TRS+9P3Sfsn7WN/CWE=; b=M7+u0ZffoybPRHkuvz21lCExiVMngGTigsnE3zHYFoRx71szW8XIKKGO1a1zJnfYw2 2oNVho8V6MaKFKmMZiByMuZnLZsu7oIq1b+dr7R07pCdn2wX3dYt/rW1FyPZLuzhtbwj W6bUMl6A7CzJ+wAskLGEMhN+e7YWgp+3L8lhB4ROOJ5bSqquNVHDm6TTKyj/aquKGi4B LFVycW7gWVpR9+v98sX5Vj6KR1AteLKJAtObxvRQ6XBLG3mvCo2xJ5RZm0exG3kovfUN /7g9WDo+ajEQLnxlEduIH67nRY+OXBReqB2fCakJB4i4Xmr0Zy06CyeQ/oBwzCdoD5dC A5yA== X-Gm-Message-State: AOJu0YyqbYYsQmR229+vJivE3ItNR19EWngusnMB+yKjT5uES9WE27cU wG9t/LGTgdXWgB3JUVJpocNIi+VhWhd84q+dSOkAlO2wP8klaKbfL3i0OL6ieqdU4vJglEAz1o+ a X-Google-Smtp-Source: AGHT+IFJRphuUXx7n0loteYVj0MdhSfj5eB26Mgkm5UX1a9bctHC3mJj/YlOh7sUh6QVWCXssbhFCA== X-Received: by 2002:a05:6871:294:b0:24f:ea59:4e1c with SMTP id 586e51a60fabf-2512242cc12mr2986101fac.4.1717597193430; Wed, 05 Jun 2024 07:19:53 -0700 (PDT) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 586e51a60fabf-250853ca28esm4048918fac.55.2024.06.05.07.19.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 07:19:52 -0700 (PDT) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 9/9] io_uring/msg_ring: remove non-remote message passing Date: Wed, 5 Jun 2024 07:51:17 -0600 Message-ID: <20240605141933.11975-10-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240605141933.11975-1-axboe@kernel.dk> References: <20240605141933.11975-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Now that the overflow approach works well, there's no need to retain the double locking for direct CQ posting on the target ring. Just have any kind of target ring use the same messaging mechanism. Signed-off-by: Jens Axboe --- io_uring/msg_ring.c | 79 ++++++++------------------------------------- 1 file changed, 14 insertions(+), 65 deletions(-) diff --git a/io_uring/msg_ring.c b/io_uring/msg_ring.c index eeca1563ceed..9fb355b7e736 100644 --- a/io_uring/msg_ring.c +++ b/io_uring/msg_ring.c @@ -33,11 +33,6 @@ struct io_msg { u32 flags; }; -static void io_double_unlock_ctx(struct io_ring_ctx *octx) -{ - mutex_unlock(&octx->uring_lock); -} - static int io_double_lock_ctx(struct io_ring_ctx *octx, unsigned int issue_flags) { @@ -66,11 +61,6 @@ void io_msg_ring_cleanup(struct io_kiocb *req) msg->src_file = NULL; } -static inline bool io_msg_need_remote(struct io_ring_ctx *target_ctx) -{ - return target_ctx->task_complete; -} - static struct io_overflow_cqe *io_alloc_overflow(struct io_ring_ctx *target_ctx) __acquires(&target_ctx->completion_lock) { @@ -109,7 +99,7 @@ static void io_msg_add_overflow(struct io_msg *msg, u32 flags) __releases(&target_ctx->completion_lock) { - unsigned nr_prev, nr_wait; + unsigned nr_prev; if (list_empty(&target_ctx->cq_overflow_list)) { set_bit(IO_CHECK_CQ_OVERFLOW_BIT, &target_ctx->check_cq); @@ -122,11 +112,17 @@ static void io_msg_add_overflow(struct io_msg *msg, nr_prev = target_ctx->nr_overflow++; list_add_tail(&ocqe->list, &target_ctx->cq_overflow_list); spin_unlock(&target_ctx->completion_lock); - rcu_read_lock(); - io_defer_tw_count(target_ctx, &nr_wait); - nr_prev += nr_wait; - io_defer_wake(target_ctx, nr_prev + 1, nr_prev); - rcu_read_unlock(); + if (target_ctx->flags & IORING_SETUP_DEFER_TASKRUN) { + unsigned nr_wait; + + rcu_read_lock(); + io_defer_tw_count(target_ctx, &nr_wait); + nr_prev += nr_wait; + io_defer_wake(target_ctx, nr_prev + 1, nr_prev); + rcu_read_unlock(); + } else if (wq_has_sleeper(&target_ctx->cq_wait)) { + wake_up(&target_ctx->cq_wait); + } } static int io_msg_fill_remote(struct io_msg *msg, unsigned int issue_flags, @@ -149,7 +145,6 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); u32 flags = 0; - int ret; if (msg->src_fd || msg->flags & ~IORING_MSG_RING_FLAGS_PASS) return -EINVAL; @@ -161,19 +156,7 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) if (msg->flags & IORING_MSG_RING_FLAGS_PASS) flags = msg->cqe_flags; - if (io_msg_need_remote(target_ctx)) - return io_msg_fill_remote(msg, issue_flags, target_ctx, flags); - - ret = -EOVERFLOW; - if (target_ctx->flags & IORING_SETUP_IOPOLL) { - if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) - return -EAGAIN; - } - if (io_post_aux_cqe(target_ctx, msg->user_data, msg->len, flags)) - ret = 0; - if (target_ctx->flags & IORING_SETUP_IOPOLL) - io_double_unlock_ctx(target_ctx); - return ret; + return io_msg_fill_remote(msg, issue_flags, target_ctx, flags); } static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_flags) @@ -194,38 +177,6 @@ static struct file *io_msg_grab_file(struct io_kiocb *req, unsigned int issue_fl return file; } -static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flags) -{ - struct io_ring_ctx *target_ctx = req->file->private_data; - struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); - struct file *src_file = msg->src_file; - int ret; - - if (unlikely(io_double_lock_ctx(target_ctx, issue_flags))) - return -EAGAIN; - - ret = __io_fixed_fd_install(target_ctx, src_file, msg->dst_fd); - if (ret < 0) - goto out_unlock; - - msg->src_file = NULL; - req->flags &= ~REQ_F_NEED_CLEANUP; - - if (msg->flags & IORING_MSG_RING_CQE_SKIP) - goto out_unlock; - /* - * If this fails, the target still received the file descriptor but - * wasn't notified of the fact. This means that if this request - * completes with -EOVERFLOW, then the sender must ensure that a - * later IORING_OP_MSG_RING delivers the message. - */ - if (!io_post_aux_cqe(target_ctx, msg->user_data, ret, 0)) - ret = -EOVERFLOW; -out_unlock: - io_double_unlock_ctx(target_ctx); - return ret; -} - static int io_msg_install_remote(struct io_kiocb *req, unsigned int issue_flags, struct io_ring_ctx *target_ctx) { @@ -284,9 +235,7 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags) req->flags |= REQ_F_NEED_CLEANUP; } - if (io_msg_need_remote(target_ctx)) - return io_msg_install_remote(req, issue_flags, target_ctx); - return io_msg_install_complete(req, issue_flags); + return io_msg_install_remote(req, issue_flags, target_ctx); } int io_msg_ring_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)