From patchwork Mon Jan 29 20:23:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13536351 Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAC4812B86 for ; Mon, 29 Jan 2024 20:30:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560234; cv=none; b=CIx2xqqimlMFGuQbjqWGG9IecYm88Ck2MEpPKhAZ9AmDiKjJliNzYIQQnD8nD9+bMenbKHqNHY2iUg898IIG4Ec2EwF23pvJBIYk8nlt5WoAUMwPRWUthjtrEp4vbSb9zbzvsm5UGrzY6udiJzng+KwY65HKSqnNRkqolNmRKH4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560234; c=relaxed/simple; bh=pR97qf/01SSuJ9StJBQqDjHNrvMuF05pBDnaNBcL02U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SDpMmtIql12f74wkxogq/iGLan+DLt/y2d1BqYZD05FaHrt9cb5zKbug2gpI+1aBmKme/thMuPvC8k2pyt/ohXpSErrgSHLGBa8K645HXuoqBxg/nIMa8AR46D/vZ7pYsa1kKc7ajUIqPYCj45iGVTBg1TMM7TMaFD5rfnVOV2o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=J9IUmv5y; arc=none smtp.client-ip=209.85.166.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="J9IUmv5y" Received: by mail-io1-f52.google.com with SMTP id ca18e2360f4ac-7bf3283c18dso42978439f.0 for ; Mon, 29 Jan 2024 12:30:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706560229; x=1707165029; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8ux8x8eUe7gD1jQ5pGHo2InXRKpLdO69yR0/M6Q9qck=; b=J9IUmv5y6a5J+J0ZHkeCVSeDbvTTPJKY3wG+y/GpXdLrQq/sYWJnF2zPTGhPwBgBYq Zg4YhVvnlnLFp1b6Gy8cR0d2QoUBT3G7BnIX1WqADUTtXv9h4ovybYVEyEPJJ7/aO/e5 IICvLY7lZuk1FrFhk+y/v9KeDD1SvwTKKCpeFn5tDAzktRIbZoNXJoZVwjTImRP6/45u hhBLYxWWmKVhOWGqB1wlhHIedqckKJO7B8O6E4fKVIBQLLc4Jr1+mYw1uuxqyDe72Yd5 VsENIKKhkuAUzmBlVPHMqITciBDp+013AnTuS7XUUen9tOFqjyCzqKKdMI6hlG7HuTj1 +/6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706560229; x=1707165029; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8ux8x8eUe7gD1jQ5pGHo2InXRKpLdO69yR0/M6Q9qck=; b=TstmTkx2BUOzJbrIgzU4D1wAQYfc1lcFXkq4wcBhNqXiUXpbImidyyTtNCiWTDsJFo f49O7ZxVmaPFj2sjX1LOGnHLIh4RsZFqGtLHlq4iQkjR2S5wk77voGzd1W6Hg3XC1f65 AvV/3Df49f/I9asaH9TNL4i6WHh8/hyhT3Gci9Hc6pjLxLXZlKf9GBQ05v9ffmWIHut5 DFu33Qko1/0C9oLD9WfgFwEtTMHXxA3XPkBhkLdtokgQFUUUWqnvae5CFfET62Yv/b4k OJlqm1aK5M2Mg4ocVyu/lXvKQhF2O6Wee/rPzz3/bz712mCRxj0U8vgE/+vKq2rdtZaU BxUA== X-Gm-Message-State: AOJu0YxgnhSlFaLPktviTDap5afI3cEOSqqBYzC1DsibsiDn33+JRLVx hI3w19hzD9NSTst17nPhgdqVc9djBm94zrXpaCBK9a6Ss2R00VKnIzZNBRZ0BU7oD03jk4VYYt9 K1Gw= X-Google-Smtp-Source: AGHT+IEiFJbTediERcLa3Q9EYApjHSJX2Rq1NXSiJa430u1iLG+hWypjhRk66EeYyl2IRjgyE6Zl0A== X-Received: by 2002:a5e:870b:0:b0:7bf:b18e:fccc with SMTP id y11-20020a5e870b000000b007bfb18efcccmr7692849ioj.1.1706560229514; Mon, 29 Jan 2024 12:30:29 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id i8-20020a05663815c800b0046e6a6482d2sm1952510jat.97.2024.01.29.12.30.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 12:30:29 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 1/4] io_uring/poll: move poll execution helpers higher up Date: Mon, 29 Jan 2024 13:23:44 -0700 Message-ID: <20240129203025.3214152-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240129203025.3214152-1-axboe@kernel.dk> References: <20240129203025.3214152-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for calling __io_poll_execute() higher up, move the functions to avoid forward declarations. No functional changes in this patch. Signed-off-by: Jens Axboe --- io_uring/poll.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/io_uring/poll.c b/io_uring/poll.c index d59b74a99d4e..785a5b191003 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -228,6 +228,26 @@ enum { IOU_POLL_REISSUE = 3, }; +static void __io_poll_execute(struct io_kiocb *req, int mask) +{ + unsigned flags = 0; + + io_req_set_res(req, mask, 0); + req->io_task_work.func = io_poll_task_func; + + trace_io_uring_task_add(req, mask); + + if (!(req->flags & REQ_F_POLL_NO_LAZY)) + flags = IOU_F_TWQ_LAZY_WAKE; + __io_req_task_work_add(req, flags); +} + +static inline void io_poll_execute(struct io_kiocb *req, int res) +{ + if (io_poll_get_ownership(req)) + __io_poll_execute(req, res); +} + /* * All poll tw should go through this. Checks for poll events, manages * references, does rewait, etc. @@ -364,26 +384,6 @@ void io_poll_task_func(struct io_kiocb *req, struct io_tw_state *ts) } } -static void __io_poll_execute(struct io_kiocb *req, int mask) -{ - unsigned flags = 0; - - io_req_set_res(req, mask, 0); - req->io_task_work.func = io_poll_task_func; - - trace_io_uring_task_add(req, mask); - - if (!(req->flags & REQ_F_POLL_NO_LAZY)) - flags = IOU_F_TWQ_LAZY_WAKE; - __io_req_task_work_add(req, flags); -} - -static inline void io_poll_execute(struct io_kiocb *req, int res) -{ - if (io_poll_get_ownership(req)) - __io_poll_execute(req, res); -} - static void io_poll_cancel_req(struct io_kiocb *req) { io_poll_mark_cancelled(req); From patchwork Mon Jan 29 20:23:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13536350 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8135EA21 for ; Mon, 29 Jan 2024 20:30:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560234; cv=none; b=RWHty3znyYkAlt83cMH4+Z1PXNy96iPQIJpcDeCOCkfZpZjSY5bCdzF/PtxZ3NbSUpFotclKpXDHfjzHmREPl+XSALUxmCbD+mdXIhzsjBT854F03oRGBvpvDUTQGrhHgWnO39tsrCtwzJuuBDTCxkeIvpDK4P/zImqDwcCf2pA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560234; c=relaxed/simple; bh=YS03xbG0odVO0lGQxdTvwAnf2ZC5giEb8iXYWz4CaWs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DyyRZYSYUoh2aetmi1A8I5E9bd/DlflaVrQGZq9kVvgigCDr7c6HxQQY/+rWQAbwCrABBx4ksRGBfAWkWy6sA8aG0t+cfVCA+HGwAUlfff6yX0T8AaXqZkxsMrVDVwCRDN+GJ39qA+YooTOXWjxuvblRCEfL1ACVGSUE5PcBiFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=L39Um0l5; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="L39Um0l5" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7bff2f6080aso7654839f.1 for ; Mon, 29 Jan 2024 12:30:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706560231; x=1707165031; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vCfxKu0+1vsIk+M+VwSWxtuM8+I3PGc7NT21G792nOk=; b=L39Um0l5DY5Xe/QCGabNwq/RZy4Meefm7himAioeb2XvI+LnvAFogM+3Qolwnjxede yRPPCjEtgtORo0fcj0Q0ndCoYk2jc90RdOcHl6hNNuYE+Lb04dhe40jJYbW0di6kuW7k p8oSCOypW8IV5u/T8gqlsmI/CnPuMJw8zcWOUQgiMw3uBqMoDUs/WMlNcO4XgU+YC7hR 1w1UHEYXNkeZykoDBKdh9cn4t98CmxBE2q3r8PAXRKLbLXL7cKHZbuxTSXPsuE8bHbs/ eszWhJLlSD2u1ne//ItJ15aBVHc9CzD5P4PgL/I1Npd4u6eAzLdV3p0pVdkugcQUVJ9Q 0uPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706560231; x=1707165031; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vCfxKu0+1vsIk+M+VwSWxtuM8+I3PGc7NT21G792nOk=; b=kLcVY98Nv1Dm5WzqbKXv55UFH5AJY06yYGaTs3lPXhBPZ3HSBmDriOQGWVjO0wGNmc qunmDNJnm48vg7PHzhOalS+a4mGwqWPvXYfm+UcvFizqUDYKVXPdhW8Ohmv9ReOhnuR8 lYbNBCuZ5D3Uv9ndnKdFvZGUuCwxk/uVGGFOEaFFDqHtMNgnenBEQqP8vLrg+IOdGqRk yUDNkdlyoUM7FcbQFQ3etMz0D2ky0JSoFU2uRJ7I6+cpj03AWtdlljXyYszAzGwm/G3t +d7UTjaUO7X3GVyxDBpMbumxu+hss6aQGsbBBfU11lOYfT0f0Krdl+4bgUBX6V0fRqNO lYvw== X-Gm-Message-State: AOJu0YzpzZwaqVJZiGkaj+IipwNUTXzzlXlKZWTSxqOH/0kvEJ/IEGIm fj+Cz/GbrUkEo4Or62jOsZFCqB8Daqj70uIGN3E+sMJuV6PTRWYdODDBp6ulrE6Igq7Aquizslz 94LA= X-Google-Smtp-Source: AGHT+IF/HlY95NUt4TUcWUUQBtX11tZ0PFBDQQPfijzg3eii4ZqIk8kVnv26alW9QM/qF7Uv14WofQ== X-Received: by 2002:a05:6602:2352:b0:7bf:dfac:316d with SMTP id r18-20020a056602235200b007bfdfac316dmr6628565iot.1.1706560231208; Mon, 29 Jan 2024 12:30:31 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id i8-20020a05663815c800b0046e6a6482d2sm1952510jat.97.2024.01.29.12.30.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 12:30:29 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 2/4] io_uring/net: un-indent mshot retry path in io_recv_finish() Date: Mon, 29 Jan 2024 13:23:45 -0700 Message-ID: <20240129203025.3214152-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240129203025.3214152-1-axboe@kernel.dk> References: <20240129203025.3214152-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 In preparation for putting some retry logic in there, have the done path just skip straight to the end rather than have too much nesting in here. No functional changes in this patch. Signed-off-by: Jens Axboe --- io_uring/net.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 75d494dad7e2..740c6bfa5b59 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -645,23 +645,27 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, return true; } - if (!mshot_finished) { - if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, - *ret, cflags | IORING_CQE_F_MORE)) { - io_recv_prep_retry(req); - /* Known not-empty or unknown state, retry */ - if (cflags & IORING_CQE_F_SOCK_NONEMPTY || - msg->msg_inq == -1) - return false; - if (issue_flags & IO_URING_F_MULTISHOT) - *ret = IOU_ISSUE_SKIP_COMPLETE; - else - *ret = -EAGAIN; - return true; - } - /* Otherwise stop multishot but use the current result. */ - } + if (mshot_finished) + goto finish; + /* + * Fill CQE for this receive and see if we should keep trying to + * receive from this socket. + */ + if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, + *ret, cflags | IORING_CQE_F_MORE)) { + io_recv_prep_retry(req); + /* Known not-empty or unknown state, retry */ + if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq == -1) + return false; + if (issue_flags & IO_URING_F_MULTISHOT) + *ret = IOU_ISSUE_SKIP_COMPLETE; + else + *ret = -EAGAIN; + return true; + } + /* Otherwise stop multishot but use the current result. */ +finish: io_req_set_res(req, *ret, cflags); if (issue_flags & IO_URING_F_MULTISHOT) From patchwork Mon Jan 29 20:23:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13536352 Received: from mail-io1-f49.google.com (mail-io1-f49.google.com [209.85.166.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6460241E9 for ; Mon, 29 Jan 2024 20:30:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560235; cv=none; b=J+OV3uG/Y8gITtx700Gnlwtd/ijuG+s0Cqu5dysdopZ1PZw8n6VHlUll7Tim7pe+gcAs20Mz1FpXaklv2jwvA/v02cS8Vnx5S05B1NCshaEfbLLqqf0pm+uLt1tzO1rEdJgsfCe3IfkiUSBfRiEMfib+LHby0xM7/ZgZruNUAqE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560235; c=relaxed/simple; bh=0+Rd3c+2VrjAHvhG2SXGTxJvV0LNchhXjiL1XoN1hYA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cwQJuveLzh0b8x0dPFom9NL6VrfBSdwsh/xvsHfTkJkGMC3aaNj0O8NGsdVK2gnD4cfreLGgoFTqDDbjrweHEcx/h6ZdhzSL5TevaBpR9Z0PAHOnSBbV1ZVPHybvEG6Rg6YEVJmIewZBh5ifK2eNCBvJLCLvfyQ5Js69wsgZRLs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=g79Wba8k; arc=none smtp.client-ip=209.85.166.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="g79Wba8k" Received: by mail-io1-f49.google.com with SMTP id ca18e2360f4ac-7beeeb1ba87so49128939f.0 for ; Mon, 29 Jan 2024 12:30:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706560232; x=1707165032; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Uq6s8O+pYHZy+YqyeW9aGwqRrlELW919jI2ZvhUwBE=; b=g79Wba8krWHgWUkmqXySe4oo9o+vRUuaYWY/csaO3wprde+CEZL2+5Bl1evgNwFK9p P52vyeY5M2+ZlZTHM9Cm5AtOPL975xWNMSkzte3D1r39cfYJ/tVZTyOv+/3fI5B23ddQ 4Hay+JenYL6EDG7DPJ3rMEoARuVbNKXgjzASoIsf0cJ+67O7Gqy4J7aB00SBxJJ29ZgY Z7CN/RV8qw72VySOm83DwkWLGnB0GjTOKANzV6+QRrd5Yy714pp5t9Oo9YW/t5xP5ExU 6HQrR52r1qvkVAk/ib9qabiApI4NcgFGLnAUvz7RUL3M4ah8sSVwqE+bKoQacNLp4KtG eXJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706560232; x=1707165032; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Uq6s8O+pYHZy+YqyeW9aGwqRrlELW919jI2ZvhUwBE=; b=MWQLrZvMVgm/jmVwJUIlDGqAyHramERUQ3j5NZG73jBBZetTWeP8kyWwvW9L+52mz0 QFu544mNjKzs5vb8XFzeP0YfczzMrK3VOQ17ZMm7K23fMTbfWav+0pyT8WkFjQ1aCVhG cxcoRKWbqEoJkHLhEIpOv5QRCsVZjy2HuizWhjPNRcUhgQF7J6pR6v1+F9FERjmvv8Dx hTr8Bca6ajrnQ3bYfoGE0nx+6CGx7XOEEOuXi3gaSan+IYuaiFiZgZUW0N2dibMpT/I5 8dYKN4HiTcc0RkiuUiYu5WMdep7KUb8zddoa+QWqoclGWu+cvIs3MbdqOftjUe4+t+0l 4Fgg== X-Gm-Message-State: AOJu0Yxa1Jkp+Nl4etxOqWOup+4tZZEgev4nhau3GaC+jeEuIvh6SBbp 0yFfoyW/Sm4LmR/6aWrJgD7ZmLYOJ4Jm6eqUUTJieiAk0c/rYebaowXhL18k/xMj6LomFseuGvj ah3I= X-Google-Smtp-Source: AGHT+IFI9DGTXsdE+TE2omu5IVWlSJ8jTtoDnY7xieNFXkdniHwWsNW+nz2DK80hHJLKG3RGXcU2UQ== X-Received: by 2002:a05:6602:1508:b0:7bf:f336:8032 with SMTP id g8-20020a056602150800b007bff3368032mr5204169iow.2.1706560232543; Mon, 29 Jan 2024 12:30:32 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id i8-20020a05663815c800b0046e6a6482d2sm1952510jat.97.2024.01.29.12.30.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 12:30:31 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe Subject: [PATCH 3/4] io_uring/poll: add requeue return code from poll multishot handling Date: Mon, 29 Jan 2024 13:23:46 -0700 Message-ID: <20240129203025.3214152-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240129203025.3214152-1-axboe@kernel.dk> References: <20240129203025.3214152-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since our poll handling is edge triggered, multishot handlers retry internally until they know that no more data is available. In preparation for limiting these retries, add an internal return code, IOU_REQUEUE, which can be used to inform the poll backend about the handler wanting to retry, but that this should happen through a normal task_work requeue rather than keep hammering on the issue side for this one request. No functional changes in this patch, nobody is using this return code just yet. Signed-off-by: Jens Axboe --- io_uring/io_uring.h | 8 +++++++- io_uring/poll.c | 9 ++++++++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 04e33f25919c..d5495710c178 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -15,11 +15,17 @@ #include #endif - enum { IOU_OK = 0, IOU_ISSUE_SKIP_COMPLETE = -EIOCBQUEUED, + /* + * Requeue the task_work to restart operations on this request. The + * actual value isn't important, should just be not an otherwise + * valid error code, yet less than -MAX_ERRNO and valid internally. + */ + IOU_REQUEUE = -3072, + /* * Intended only when both IO_URING_F_MULTISHOT is passed * to indicate to the poll runner that multishot should be diff --git a/io_uring/poll.c b/io_uring/poll.c index 785a5b191003..7513afc7b702 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -226,6 +226,7 @@ enum { IOU_POLL_NO_ACTION = 1, IOU_POLL_REMOVE_POLL_USE_RES = 2, IOU_POLL_REISSUE = 3, + IOU_POLL_REQUEUE = 4, }; static void __io_poll_execute(struct io_kiocb *req, int mask) @@ -329,6 +330,8 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) int ret = io_poll_issue(req, ts); if (ret == IOU_STOP_MULTISHOT) return IOU_POLL_REMOVE_POLL_USE_RES; + else if (ret == IOU_REQUEUE) + return IOU_POLL_REQUEUE; if (ret < 0) return ret; } @@ -351,8 +354,12 @@ void io_poll_task_func(struct io_kiocb *req, struct io_tw_state *ts) int ret; ret = io_poll_check_events(req, ts); - if (ret == IOU_POLL_NO_ACTION) + if (ret == IOU_POLL_NO_ACTION) { return; + } else if (ret == IOU_POLL_REQUEUE) { + __io_poll_execute(req, 0); + return; + } io_poll_remove_entries(req); io_poll_tw_hash_eject(req, ts); From patchwork Mon Jan 29 20:23:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13536353 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A69E51552F2 for ; Mon, 29 Jan 2024 20:30:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560237; cv=none; b=CWs3HzQRE3BTwVofZBny/l2A6u/93iep/0Zsafj8KnIbo+L5obmm9OToMrV+xMLgRE1vhiJgP2WudrCD34O33rvW5SY+CI5OYSd5PHIZ+Pox0FdOYyhRXX3IYXyPf3ERO8Sktuig70XsV4XfO8FTYnITpnxzVIDRPzLcImy4BFk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706560237; c=relaxed/simple; bh=F0L6R3HMTsvU5QrGV1jKpmdDPnL8PDZ9XptAIhf9OaQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jqx2GtQDbZotJIClDeGGG0Air9knBprLXoX8ejmMLDXH0wyl8KR6kT83zlATmM84IE+kXgDREUZUUmq8vcxEXQJK/omUUsLxU1VyFvGJBVKdMZriUEuzJpdQ2ySt7TtMZE7PgrJhlZ78gfgYPBOL5Pu+7KnTfgghCerL9UyMLmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=2ObtooWg; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="2ObtooWg" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7beeeb1ba87so49129239f.0 for ; Mon, 29 Jan 2024 12:30:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1706560234; x=1707165034; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NDZQtmapdAhbbStZz6bk8XOpRhvQFEopwTej5qLfkwg=; b=2ObtooWgTYxq3GyuCs0QVPrKGTuq2T/IPJUIFQfdFb3+n14JJ6iMoPXw8k4Sfg97nk IMoGdoXDwSN7m2nL0/4dKzYfRUGHvRksmXG/3vCtuGUo2RYOiqId8e8ItFBtQBSETUQ7 iLelrvfBja4rBbYZbWH5dB2cK0SyL8SS6Sq+8ZWB3RRec3EUJMXrftuPQIAlvpePi2T1 Mtb4Wc1rJ114f4j4ciLdto4QYaJbsuZJUIbrGVLS4kYP/mtsjXA2Y0750hJWKMiCY53l /HQiGsrRy15VbnLc4WmyyOebh55nifbeaAKych9mipHGm7l02AEMzAdFGWDxVsXo9a2Y 9htA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706560234; x=1707165034; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NDZQtmapdAhbbStZz6bk8XOpRhvQFEopwTej5qLfkwg=; b=gfX8XMRvX5Y2k5K87Uq3aIY2rxpe3uaItrtwuzLLMMr+Ur79D4GO4jeUWifFrpEH15 AZxhOA2gr8m/30visYPHR4yNE7loGGHSZ9SUoUvLYPHU3HMALuZqXeDj+0r7ifyNDFib 2LPcnviQw+WOvsdr4bHAPsxuv22scRTeXiYlYcIxeNMZ1Jd59R+7jKNLLocqnjElPotY TKVIpxge+bf2oZIt4/49OxbmapIdJiIE8GjMg0AaYn3MjBT0ly6slJ7BiAqePeaAp5jI eOx49ZdbLZ2N/0TSuyZrnVfZmInaYYzEiDIiauCkDA5ZhK0LBQ90yszEFWKDTH6wCnLp PN9Q== X-Gm-Message-State: AOJu0Yzmsks8iZeiKIfVXtyrPUDojoawK+RGtX4g1hs/nCozZbjGPcZ8 I4poaUA9/e3Hree912PRAmq68OU1zP4Yqgdy5CEnfO+c22iBxzaVUjP3SI/3IPz7mnTHIOX2OMu o4jg= X-Google-Smtp-Source: AGHT+IEzuwIW8l3BA3QcrZVmR/kStX9HAjIfJYx53Es5wNkmu14HVNKS5mUQJ4BxrwCQX+uFFhnTqQ== X-Received: by 2002:a6b:f814:0:b0:7bf:4758:2a12 with SMTP id o20-20020a6bf814000000b007bf47582a12mr7198091ioh.0.1706560233931; Mon, 29 Jan 2024 12:30:33 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id i8-20020a05663815c800b0046e6a6482d2sm1952510jat.97.2024.01.29.12.30.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 12:30:32 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: asml.silence@gmail.com, Jens Axboe , stable@vger.kernel.org Subject: [PATCH 4/4] io_uring/net: limit inline multishot retries Date: Mon, 29 Jan 2024 13:23:47 -0700 Message-ID: <20240129203025.3214152-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240129203025.3214152-1-axboe@kernel.dk> References: <20240129203025.3214152-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If we have multiple clients and some/all are flooding the receives to such an extent that we can retry a LOT handling multishot receives, then we can be starving some clients and hence serving traffic in an imbalanced fashion. Limit multishot retry attempts to some arbitrary value, whose only purpose serves to ensure that we don't keep serving a single connection for way too long. We default to 32 retries, which should be more than enough to provide fairness, yet not so small that we'll spend too much time requeuing rather than handling traffic. Cc: stable@vger.kernel.org Depends-on: 704ea888d646 ("io_uring/poll: add requeue return code from poll multishot handling") Depends-on: 1e5d765a82f ("io_uring/net: un-indent mshot retry path in io_recv_finish()") Depends-on: e84b01a880f6 ("io_uring/poll: move poll execution helpers higher up") Fixes: b3fdea6ecb55 ("io_uring: multishot recv") Fixes: 9bb66906f23e ("io_uring: support multishot in recvmsg") Link: https://github.com/axboe/liburing/issues/1043 Signed-off-by: Jens Axboe --- io_uring/net.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 740c6bfa5b59..a12ff69e6843 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -60,6 +60,7 @@ struct io_sr_msg { unsigned len; unsigned done_io; unsigned msg_flags; + unsigned nr_multishot_loops; u16 flags; /* initialised and used only by !msg send variants */ u16 addr_len; @@ -70,6 +71,13 @@ struct io_sr_msg { struct io_kiocb *notif; }; +/* + * Number of times we'll try and do receives if there's more data. If we + * exceed this limit, then add us to the back of the queue and retry from + * there. This helps fairness between flooding clients. + */ +#define MULTISHOT_MAX_RETRY 32 + static inline bool io_check_multishot(struct io_kiocb *req, unsigned int issue_flags) { @@ -611,6 +619,7 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->msg_flags |= MSG_CMSG_COMPAT; #endif sr->done_io = 0; + sr->nr_multishot_loops = 0; return 0; } @@ -654,12 +663,20 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, */ if (io_fill_cqe_req_aux(req, issue_flags & IO_URING_F_COMPLETE_DEFER, *ret, cflags | IORING_CQE_F_MORE)) { + struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + int mshot_retry_ret = IOU_ISSUE_SKIP_COMPLETE; + io_recv_prep_retry(req); /* Known not-empty or unknown state, retry */ - if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq == -1) - return false; + if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq == -1) { + if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY) + return false; + /* mshot retries exceeded, force a requeue */ + sr->nr_multishot_loops = 0; + mshot_retry_ret = IOU_REQUEUE; + } if (issue_flags & IO_URING_F_MULTISHOT) - *ret = IOU_ISSUE_SKIP_COMPLETE; + *ret = mshot_retry_ret; else *ret = -EAGAIN; return true;