From patchwork Thu Mar 7 20:30:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13586239 Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C77F922EFB for ; Thu, 7 Mar 2024 20:31:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843481; cv=none; b=pYg70gZXLeNapGXWVtCzZBoGTucvWCxqBwtkKkYYnzuN9J/MF2IoAzMe+cdKCtwmVm8me1E2qUAOSY4/FSOFOBoVcgVrosMvffun6Me4ywI7VBCnrLmjA20g6RNa9Vff+EkeC4NZFfTrG98ohjmSDkBSR5YlMY9pHd9i1wI6LmM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843481; c=relaxed/simple; bh=MOJbzZU3ZTQ0Za5RasSdNuJI5PbJkI0sYcdd5Wx+Dek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Adl+I40rYbrKQ90abbk0PDxDhJgmZfDJHozIcw5UVn4kcjCb8UlPNzbStKj7cDzhLBooTeUV9VNZbDWwKGbGQ30oAcS5hyVO2X927i/7UeeIEtjK2MmABw/Dd/4FYuQt+JdV3o+F5Bf52RUMD5NzKxux6bNtFaEIoxgYnrtuKuw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=UuB7cZ1S; arc=none smtp.client-ip=209.85.166.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="UuB7cZ1S" Received: by mail-io1-f50.google.com with SMTP id ca18e2360f4ac-7c840d5aab4so4376039f.0 for ; Thu, 07 Mar 2024 12:31:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709843477; x=1710448277; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5NyTP/TpabmOMMvIFRa2yEBaWnqiDJyFsY60N+kaF1k=; b=UuB7cZ1S1LQUdX+zqEPqWqHwfYxNo+aGGJBrKKW6Qiu9+5Z1bypf3DjQKYHt8oc2ri bSS4aNv0IFMXfTznMoibzgXqbX1+tjUBgWZXtog9anCUMOVmwHWE+PIAJOqiDQVtDdPE +B7uyH7hjEz72EGyOSaKHawXGoeam6GMmE9CSCCsSD4lieLxeWpbZdCsDOG1kZQsrD+h FOcb9rXQND/4VfnF+1ghIHBdRUCQFOHyaWCvGZFxXhOd8X8n9Xk6HnkXfDi+zaZix6q1 9CM2WiC8dGH99GD0redltYgQse3rdWle/Em1P5YeR5aatKURki3tqLJhTIVnMM7puM1E 59Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709843477; x=1710448277; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5NyTP/TpabmOMMvIFRa2yEBaWnqiDJyFsY60N+kaF1k=; b=bn9baHEsnAqRh3Sqyov3IybQyVMab7pGRQjY/WKO9BKV5hIQBQfh4sO1vO9JUTmhrN v3020duiJldzMPzJOxjt222nqmmzITTCC12nOILVaOz5LAbFa9fTs7damZZPKOlyGOP9 6K1SaAJAy/g+Q7bqOmurPdh7aeHNuOMGqEN+V3E6c/99jXtEtWtr30s3RQD/xGKSHkaj YrUJYTlDl+3Uk+hfb76BMCHPF+PVhV2m/+g1Ne8mUdeltG4HQ9ZxKZduAtKeIs51JFoZ UuHV0q0doVWM+vs8wXILfW7u5UEFN1Nxd68BpAUdokZ3GnEBdQ0qqRadJKi7rVCvlCvy +1jA== X-Gm-Message-State: AOJu0Yzn8468Gtvl8NigHGdFv77VY/HPj6pfNIZRfPk1skmLdeTsEAAY tbEkaFeIkFFKJH+ZNNvkNxTTqB1pDrih0xHa/plrE8SlNagSicaKRnOXLWBUxrAEmsHuwHvmxJe 3 X-Google-Smtp-Source: AGHT+IFUAbNDQ7RPY9u7SGA+noLrb/TZKq13Rz7LNWN7M/KL6i1bpcW8V1P2yrR/Cj4A0H1QwA2XyQ== X-Received: by 2002:a6b:dd11:0:b0:7c8:718b:cff5 with SMTP id f17-20020a6bdd11000000b007c8718bcff5mr3557294ioc.2.1709843477525; Thu, 07 Mar 2024 12:31:17 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id f1-20020a028481000000b0047469b04c35sm4198921jai.65.2024.03.07.12.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 12:31:15 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 1/4] io_uring/net: remove dependency on REQ_F_PARTIAL_IO for sr->done_io Date: Thu, 7 Mar 2024 13:30:24 -0700 Message-ID: <20240307203113.575893-2-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240307203113.575893-1-axboe@kernel.dk> References: <20240307203113.575893-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Ensure that prep handlers always initialize sr->done_io before any potential failure conditions, and with that, we now it's always been set even for the failure case. With that, we don't need to use the REQ_F_PARTIAL_IO flag to gate on that. Additionally, we should not overwrite req->cqe.res unless sr->done_io is actually positive. Signed-off-by: Jens Axboe --- io_uring/net.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 0d545f71dc79..eacbe9295a7f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -387,6 +387,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + sr->done_io = 0; + if (req->opcode == IORING_OP_SEND) { if (READ_ONCE(sqe->__pad3[0])) return -EINVAL; @@ -409,7 +411,6 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->ctx->compat) sr->msg_flags |= MSG_CMSG_COMPAT; #endif - sr->done_io = 0; return 0; } @@ -631,6 +632,8 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); + sr->done_io = 0; + if (unlikely(sqe->file_index || sqe->addr2)) return -EINVAL; @@ -667,7 +670,6 @@ int io_recvmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (req->ctx->compat) sr->msg_flags |= MSG_CMSG_COMPAT; #endif - sr->done_io = 0; sr->nr_multishot_loops = 0; return 0; } @@ -1054,6 +1056,8 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) struct io_ring_ctx *ctx = req->ctx; struct io_kiocb *notif; + zc->done_io = 0; + if (unlikely(READ_ONCE(sqe->__pad2[0]) || READ_ONCE(sqe->addr3))) return -EINVAL; /* we don't support IOSQE_CQE_SKIP_SUCCESS just yet */ @@ -1106,8 +1110,6 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (zc->msg_flags & MSG_DONTWAIT) req->flags |= REQ_F_NOWAIT; - zc->done_io = 0; - #ifdef CONFIG_COMPAT if (req->ctx->compat) zc->msg_flags |= MSG_CMSG_COMPAT; @@ -1352,7 +1354,7 @@ void io_sendrecv_fail(struct io_kiocb *req) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); - if (req->flags & REQ_F_PARTIAL_IO) + if (sr->done_io) req->cqe.res = sr->done_io; if ((req->flags & REQ_F_NEED_CLEANUP) && From patchwork Thu Mar 7 20:30:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13586240 Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91DA7250F8 for ; Thu, 7 Mar 2024 20:31:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843483; cv=none; b=GFnBdvl5IEXJWj7IEcjNSxyS6VSka6DS8WL/g+uvP10sK97vNYfubyr+DUueE7fLeICMItWBtoWoKcLKL+pGtSnqW/JZY1dtcVUZlQCp3Hl5UooVHbEc13JTlaRAIWLJ+WNmRKuc0Cdm7K58JTYhdVvy1floiikW3SYBeMHkL4U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843483; c=relaxed/simple; bh=2+RPus01m4Xw6sTEhw8vNksYC2MghJDb7pcAejpJNUc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=n4R55EPOr4C106Z2OAiKLQusolJ87cyi2xBCwVgSz1jjdH/sZ8fsCDo/1xk4ytdAsWu9YKXbjggUx7PPnvVJ75G0p5vL2u3p2xuBeP9kzDvRqFFoT5KC7p+4WEcGnxwlVPSkZdS3DaJ2XgbB/oZDE/XPERC6W2nIJziMucDyTRY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=urmOzs3j; arc=none smtp.client-ip=209.85.166.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="urmOzs3j" Received: by mail-io1-f51.google.com with SMTP id ca18e2360f4ac-7c495be1924so20176139f.1 for ; Thu, 07 Mar 2024 12:31:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709843479; x=1710448279; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SnMcoyCrC2eEkuIEAAmWnSMiZWHvwaAKzGPEpmxj7Mc=; b=urmOzs3j1NC0ZFOUVMm7CMCUVJrU4Uy+d8TJ/n/a/8IYnDxZ6+wENnG0lkdtvD5KjW JD4nSgsLX4VW7jlq4C2HpelBzRllYO16Qr+iBXqKIMxFv51c+r8PzhPUMb47edHNr+Sb WqEarWVCZOUcpyazxiNt1p3vtlyLLIHXXDR/TYS+Pz2RjCJXv0tc22uXNFxU1g3maDsc lTNWNuV5otIUckv8j6dFhCWQFV7YQTIXeZe3ZWXPzEzsvZBFjSbPDljhphg8O/RRcqrk 82qYoM5ynfdp1E2UVhyFu4ip6nOHu87h0QNS0S9UVx64jvjCiELj1ieZwpbAW93AMv3Z igAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709843479; x=1710448279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SnMcoyCrC2eEkuIEAAmWnSMiZWHvwaAKzGPEpmxj7Mc=; b=a+YykBFEUOKQ2eQE4ll/PNtKNgHHS/CPToLDqHWmjikFoR3+mlXEs/yTQ9WBLnGubF J3YqmpAL0qpqag7txhTWiaQCc4/9s/KryDPZT5lGpGlUtac8H6MBi0J3RWZsiDprnHZR msOrTwWPZue0a02WYFPocJ3r2/M2QK0S1Giw1KvIR37BbZ39DrHJZJWQqEH/59Pw6bVD V9ZTXs7Xxp26fROZVBjg1dr/xztLjY0lxl4Dj6MU5sNMF5oHhKhzIT6gBok/b8BVfHok KYkvJxRjCXBZfbrOxqMu1sM7LvqvqcqQXSwSbXubMMdK+3ldFVnogyVo/soxFxhrMbGX fVWg== X-Gm-Message-State: AOJu0YyBlEqg+7sGYCnhZ0OECYoL3dBZSOdSX3v4W8AdYhzQtHmz2v3S bH25CLI5OKEMleZpnFraf4Vf6eBXsSPbMFuLq1TtrMIOf4EWgcgbQpUwAjvo3Hb5UCoc/e883rV 2 X-Google-Smtp-Source: AGHT+IFju2+pk1q+U7m1W7ga4Ad0dN1Zyf4JTX6Yk1Z5OOYA0bxwfzdhqguG9xWeCYl2HzztIfnrVg== X-Received: by 2002:a6b:f20c:0:b0:7c8:8a21:7156 with SMTP id q12-20020a6bf20c000000b007c88a217156mr1918722ioh.2.1709843479260; Thu, 07 Mar 2024 12:31:19 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id f1-20020a028481000000b0047469b04c35sm4198921jai.65.2024.03.07.12.31.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 12:31:17 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 2/4] io_uring/kbuf: rename REQ_F_PARTIAL_IO to REQ_F_BL_NO_RECYCLE Date: Thu, 7 Mar 2024 13:30:25 -0700 Message-ID: <20240307203113.575893-3-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240307203113.575893-1-axboe@kernel.dk> References: <20240307203113.575893-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 We only use the flag for this purpose, so rename it accordingly. This further prevents various other use cases of it, keeping it clean and consistent. Then we can also check it in one spot, when it's being attempted recycled, and remove some dead code in io_kbuf_recycle_ring(). Signed-off-by: Jens Axboe --- include/linux/io_uring_types.h | 6 +++--- io_uring/kbuf.c | 9 --------- io_uring/kbuf.h | 20 +++++--------------- io_uring/net.c | 12 ++++++------ io_uring/rw.c | 4 ++-- 5 files changed, 16 insertions(+), 35 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index d8111d64812b..e24893625085 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -470,7 +470,6 @@ enum { REQ_F_SKIP_LINK_CQES_BIT, REQ_F_SINGLE_POLL_BIT, REQ_F_DOUBLE_POLL_BIT, - REQ_F_PARTIAL_IO_BIT, REQ_F_APOLL_MULTISHOT_BIT, REQ_F_CLEAR_POLLIN_BIT, REQ_F_HASH_LOCKED_BIT, @@ -481,6 +480,7 @@ enum { REQ_F_CANCEL_SEQ_BIT, REQ_F_CAN_POLL_BIT, REQ_F_BL_EMPTY_BIT, + REQ_F_BL_NO_RECYCLE_BIT, /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -543,8 +543,6 @@ enum { REQ_F_SINGLE_POLL = IO_REQ_FLAG(REQ_F_SINGLE_POLL_BIT), /* double poll may active */ REQ_F_DOUBLE_POLL = IO_REQ_FLAG(REQ_F_DOUBLE_POLL_BIT), - /* request has already done partial IO */ - REQ_F_PARTIAL_IO = IO_REQ_FLAG(REQ_F_PARTIAL_IO_BIT), /* fast poll multishot mode */ REQ_F_APOLL_MULTISHOT = IO_REQ_FLAG(REQ_F_APOLL_MULTISHOT_BIT), /* recvmsg special flag, clear EPOLLIN */ @@ -559,6 +557,8 @@ enum { REQ_F_CAN_POLL = IO_REQ_FLAG(REQ_F_CAN_POLL_BIT), /* buffer list was empty after selection of buffer */ REQ_F_BL_EMPTY = IO_REQ_FLAG(REQ_F_BL_EMPTY_BIT), + /* don't recycle provided buffers for this request */ + REQ_F_BL_NO_RECYCLE = IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT), }; typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts); diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 3d257ed9031b..9be42bff936b 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -81,15 +81,6 @@ bool io_kbuf_recycle_legacy(struct io_kiocb *req, unsigned issue_flags) struct io_buffer_list *bl; struct io_buffer *buf; - /* - * For legacy provided buffer mode, don't recycle if we already did - * IO to this buffer. For ring-mapped provided buffer mode, we should - * increment ring->head to explicitly monopolize the buffer to avoid - * multiple use. - */ - if (req->flags & REQ_F_PARTIAL_IO) - return false; - io_ring_submit_lock(ctx, issue_flags); buf = req->kbuf; diff --git a/io_uring/kbuf.h b/io_uring/kbuf.h index f74c910b83f4..5218bfd79e87 100644 --- a/io_uring/kbuf.h +++ b/io_uring/kbuf.h @@ -73,21 +73,9 @@ static inline bool io_kbuf_recycle_ring(struct io_kiocb *req) * to monopolize the buffer. */ if (req->buf_list) { - if (req->flags & REQ_F_PARTIAL_IO) { - /* - * If we end up here, then the io_uring_lock has - * been kept held since we retrieved the buffer. - * For the io-wq case, we already cleared - * req->buf_list when the buffer was retrieved, - * hence it cannot be set here for that case. - */ - req->buf_list->head++; - req->buf_list = NULL; - } else { - req->buf_index = req->buf_list->bgid; - req->flags &= ~REQ_F_BUFFER_RING; - return true; - } + req->buf_index = req->buf_list->bgid; + req->flags &= ~REQ_F_BUFFER_RING; + return true; } return false; } @@ -101,6 +89,8 @@ static inline bool io_do_buffer_select(struct io_kiocb *req) static inline bool io_kbuf_recycle(struct io_kiocb *req, unsigned issue_flags) { + if (req->flags & REQ_F_BL_NO_RECYCLE) + return false; if (req->flags & REQ_F_BUFFER_SELECTED) return io_kbuf_recycle_legacy(req, issue_flags); if (req->flags & REQ_F_BUFFER_RING) diff --git a/io_uring/net.c b/io_uring/net.c index eacbe9295a7f..f8495f6a0bda 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -456,7 +456,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_controllen = 0; kmsg->msg.msg_control = NULL; sr->done_io += ret; - req->flags |= REQ_F_PARTIAL_IO; + req->flags |= REQ_F_BL_NO_RECYCLE; return io_setup_async_msg(req, kmsg, issue_flags); } if (ret == -ERESTARTSYS) @@ -535,7 +535,7 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) sr->len -= ret; sr->buf += ret; sr->done_io += ret; - req->flags |= REQ_F_PARTIAL_IO; + req->flags |= REQ_F_BL_NO_RECYCLE; return io_setup_async_addr(req, &__address, issue_flags); } if (ret == -ERESTARTSYS) @@ -907,7 +907,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) } if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; - req->flags |= REQ_F_PARTIAL_IO; + req->flags |= REQ_F_BL_NO_RECYCLE; return io_setup_async_msg(req, kmsg, issue_flags); } if (ret == -ERESTARTSYS) @@ -1006,7 +1006,7 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) sr->len -= ret; sr->buf += ret; sr->done_io += ret; - req->flags |= REQ_F_PARTIAL_IO; + req->flags |= REQ_F_BL_NO_RECYCLE; return -EAGAIN; } if (ret == -ERESTARTSYS) @@ -1249,7 +1249,7 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) zc->len -= ret; zc->buf += ret; zc->done_io += ret; - req->flags |= REQ_F_PARTIAL_IO; + req->flags |= REQ_F_BL_NO_RECYCLE; return io_setup_async_addr(req, &__address, issue_flags); } if (ret == -ERESTARTSYS) @@ -1319,7 +1319,7 @@ int io_sendmsg_zc(struct io_kiocb *req, unsigned int issue_flags) if (ret > 0 && io_net_retry(sock, flags)) { sr->done_io += ret; - req->flags |= REQ_F_PARTIAL_IO; + req->flags |= REQ_F_BL_NO_RECYCLE; return io_setup_async_msg(req, kmsg, issue_flags); } if (ret == -ERESTARTSYS) diff --git a/io_uring/rw.c b/io_uring/rw.c index 7733449271f2..5651a5ad4e11 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -275,7 +275,7 @@ static bool __io_complete_rw_common(struct io_kiocb *req, long res) * current cycle. */ io_req_io_end(req); - req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; + req->flags |= REQ_F_REISSUE | REQ_F_BL_NO_RECYCLE; return true; } req_set_fail(req); @@ -342,7 +342,7 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res) io_req_end_write(req); if (unlikely(res != req->cqe.res)) { if (res == -EAGAIN && io_rw_should_reissue(req)) { - req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; + req->flags |= REQ_F_REISSUE | REQ_F_BL_NO_RECYCLE; return; } req->cqe.res = res; From patchwork Thu Mar 7 20:30:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13586241 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E9E96312F for ; Thu, 7 Mar 2024 20:31:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843484; cv=none; b=AgROkmHA0BdO5A/hq1ZVJZEC2Z+jiOocAJK9d6idGIMPOEnpZd56eb3pJ4J30M3zNSWw1xm8FjL4S5hf4YXfBNEBdytJq/1qarmBWX2FH2vBMrE+gfknybDppWA55UWQI/Nu+vqyKEvjBy43IbxNCGRim5FTY7EqJEaeq5BjbfE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843484; c=relaxed/simple; bh=x62x4eLeT00le194Rzh+3OzyARrf7VSLrMQH0L+XvxA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XuwoP6oljOCgo50ZPXkqW5PnqH9Dji0AcfqGa1ePra1S7ZXW/Tzi4QeIs9ll9vTBtrRfdlJyFBgltaEViBo59ZlhMXotW7f968KGwbtKVigtgdOqctWrIKaNnNK9pM7FEZJxwlWiJMa6EN5k6m+kvzOUajfSDWuKzNoAgyi0Fn8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ES9Lkqpe; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ES9Lkqpe" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7c495be1924so20176839f.1 for ; Thu, 07 Mar 2024 12:31:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709843481; x=1710448281; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y73OfLbNdbzIkG7Jv1Ebw5Y0V7KrlnEGSBYyOhB57Qk=; b=ES9LkqpeNhVM05PONoBMZUs7kEAaMWc8qZyioDLxWIUSUpSiORu2FtwOGMJaL6g1cx +qkbBK/pWSUtmwQxwIXsib92MOcK75tPLRqSpQ5XQ/+H9/l+beyuyUyy6I/EX8DKTjge Ca7ijxog+epoNZlxFSf+548mdZzk/51Qu6HnI8WeUvHCoPraRgSUOwhK1r0eEcXUDD4H Pg8XaYmrP+wwvbInnwpZvUR5sAOlTBSd6IcFt+6SWwNxwEZoRIo/DISkrQngCstSQPjg Kp5HZL/+q1L5mp8QvgaQaLFRTXuYi8jJm9tDzAz89OMpeqU7NCmahxDTo4x3SWqrb5qG iJZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709843481; x=1710448281; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y73OfLbNdbzIkG7Jv1Ebw5Y0V7KrlnEGSBYyOhB57Qk=; b=us7+7DNCyXdC7PUIPvbtjivk9vy3UrhyF718yMzQQ4JwBkQAvHeNPEyfeOFxfpVaMz K8JNWyF8Xnwh1EETkmJ1TerjPibv3kI5QMpRMDiz085zOMQLi2rnx7sGGJAXT3FFxNq9 oJMjBbpY2OP7LXXrQKnwH6ZHwYAdtapTdzfJV3c0MTgHZHRuCEVHgOJaGqVwiyRHZxXt Qm/IbCKOCdPN0wCV/BRO+h/7GZ/kyUj1lha4LeEm3zArNCKMTA/v2AfcwapdciXMiWSV ++HEpnW5CAq1Y5kHLaTeKEhEtA8C4IZsGxNJZtW5rJyMPXyCb8w90IaiJZhw/evAgB2R 505A== X-Gm-Message-State: AOJu0YymMfRGzOWHZUzyXEOwKMb8wHBHDQJFF8Hdig5xI7FkYbHR4GpL E9jDC0u1x5+mVF0yct8/pVbKNDd23RDQCTdLv0IYTOlVnKh8kdP0hOicJ0axYikcY3K4wubjHGw / X-Google-Smtp-Source: AGHT+IHDev+cvBNNO7tS1/RXggpurV8R27G07Fc9HCN4EPQwop1dym7u8kxnIXFMzV3VKQFveBP/Lg== X-Received: by 2002:a5d:974e:0:b0:7c8:6f1f:d44 with SMTP id c14-20020a5d974e000000b007c86f1f0d44mr3352981ioo.1.1709843481107; Thu, 07 Mar 2024 12:31:21 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id f1-20020a028481000000b0047469b04c35sm4198921jai.65.2024.03.07.12.31.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 12:31:19 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 3/4] io_uring/net: simplify msghd->msg_inq checking Date: Thu, 7 Mar 2024 13:30:26 -0700 Message-ID: <20240307203113.575893-4-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240307203113.575893-1-axboe@kernel.dk> References: <20240307203113.575893-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just check for larger than zero rather than check for non-zero and not -1. This is easier to read, and also protects against any errants < 0 values that aren't -1. Signed-off-by: Jens Axboe --- io_uring/net.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index f8495f6a0bda..e24baf765c0e 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -697,7 +697,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, unsigned int cflags; cflags = io_put_kbuf(req, issue_flags); - if (msg->msg_inq && msg->msg_inq != -1) + if (msg->msg_inq > 0) cflags |= IORING_CQE_F_SOCK_NONEMPTY; if (!(req->flags & REQ_F_APOLL_MULTISHOT)) { @@ -720,7 +720,7 @@ static inline bool io_recv_finish(struct io_kiocb *req, int *ret, io_recv_prep_retry(req); /* Known not-empty or unknown state, retry */ - if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq == -1) { + if (cflags & IORING_CQE_F_SOCK_NONEMPTY || msg->msg_inq < 0) { if (sr->nr_multishot_loops++ < MULTISHOT_MAX_RETRY) return false; /* mshot retries exceeded, force a requeue */ From patchwork Thu Mar 7 20:30:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13586242 Received: from mail-io1-f42.google.com (mail-io1-f42.google.com [209.85.166.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EE5264CEC for ; Thu, 7 Mar 2024 20:31:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843486; cv=none; b=DdwFLaQJZ3iSr63lCXweAWYhLMdUJRkWBAbsyzyd09JCknkUGrm/u2ywPtmhcNylQq6242mClhIkxHLtng54cXtRvUf3A1qb5OWuolP3SH+SzRtLyYFtKeXh06/dl3DaX4KmYvTnnWB/c4dyQfJXaeV5If9GdKyrEkmJo5ncA9I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709843486; c=relaxed/simple; bh=qGjoEIIbEt53uegd1JPCjUDPb12JVfDqcK4UZILwJqg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hkJqoKO8OVkTqKsUf+AZwha9q+J1eIchapj/sjE/x25MnfZqEBKyn2cjIuY14a6BrMF7AEI8i4XNjLKiDbOpSnMklTogT+mNrpT2qtEC/pOCZlzSiuZin+17fPd7t/f7qEF5SAmFv1O5LTBLoI+BiK8t6T0VKTHMR9UKOyz42kQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=x/fu2mBQ; arc=none smtp.client-ip=209.85.166.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="x/fu2mBQ" Received: by mail-io1-f42.google.com with SMTP id ca18e2360f4ac-7bff8f21b74so19315039f.0 for ; Thu, 07 Mar 2024 12:31:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709843483; x=1710448283; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VPeh0CbqPDrdMwbZbcHzwzoJ4zR73fb7DgUBw9dW0fU=; b=x/fu2mBQRNKE19IXyu/rVsMQhP7POGwWLNs2gmH/2yXYpsI98Lqy5Lo2DRaKTeP1c8 kelyqa3j0hNQL4JvhRE2cpRNuUWVNPRXZqjRMT7hn4L2RKihpV6VbhEveQlXqrH831ea nxoI1bVXcuBRSaanaO5PZtymV9YbfZKWiKAcKOQOhQEvAxMajd5zGOasoOEiIPVKjtIc gK5ue2VA76CarSBKOgZGK8M6M1irz8QDD1EJGFw67nwDvkfcHNL9e5/mQsBnnbHs9cL5 uMDua81w5DoxXzZTnAx8GvDKw0UQXmUvNILHp1iScsra0j+ICdGFRfuEUcxkxPA9xFSI M7Ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709843483; x=1710448283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VPeh0CbqPDrdMwbZbcHzwzoJ4zR73fb7DgUBw9dW0fU=; b=HbRBZDbQVUYbeJxCEVqAojaZmMsMlgRblPNgdGABcCMWMj2ph49m73HoQ78/mrtPhn 4YOWV4yrKtUanjhlrPtHzUQxtAAw0JTQTIwJ2xScVHdrYcgr2F8gBYY6WRhqdIcWqYYd itQqShEGPEJvXrdm1zcxL4S+sRK1i+ugicmwvQ9dJOvSw7HKqx3DFN2VhrQAJF+4tN2a HQ4imyHP7ZOpQMowdyfimjdWVd8e/K3NJgiRnp1Y2AYT9yZJEHQhrdtraM+fZbz5alOh NdZ5U2ADDtAEU3yLQWHcFZ54lG4/sWLIqf4lS7K/EDeo2F9gfbQL7MAtqVam/R4j+qoj 4h4Q== X-Gm-Message-State: AOJu0YwPLR+QFfaDTFLwtD4jpsX1At4y7P0Fuydjnx296X2P5DcDF2HB uXIC00UWDrkjElVmUWitvGQWsYnxG/xdw3AtgEitVnhw7iogjvtSnA3RDe2teTzwE+72SEJ0uH1 7 X-Google-Smtp-Source: AGHT+IE9a2HatPfNa3yv1+gUPtyYY1lQnR8M8/8mhmyGd5dJ62B+/PUWpQ1c7WnyPPPLmdMiIK559A== X-Received: by 2002:a5d:990c:0:b0:7c8:789b:b3d8 with SMTP id x12-20020a5d990c000000b007c8789bb3d8mr6087228iol.0.1709843482917; Thu, 07 Mar 2024 12:31:22 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id f1-20020a028481000000b0047469b04c35sm4198921jai.65.2024.03.07.12.31.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Mar 2024 12:31:21 -0800 (PST) From: Jens Axboe To: io-uring@vger.kernel.org Cc: Jens Axboe Subject: [PATCH 4/4] io_uring/net: add io_req_msg_cleanup() helper Date: Thu, 7 Mar 2024 13:30:27 -0700 Message-ID: <20240307203113.575893-5-axboe@kernel.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240307203113.575893-1-axboe@kernel.dk> References: <20240307203113.575893-1-axboe@kernel.dk> Precedence: bulk X-Mailing-List: io-uring@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 For the fast inline path, we manually recycle the io_async_msghdr and free the iovec, and then clear the REQ_F_NEED_CLEANUP flag to avoid that needing doing in the slower path. We already do that in 2 spots, and in preparation for adding more, add a helper and use it. Signed-off-by: Jens Axboe --- io_uring/net.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index e24baf765c0e..848dc14060b2 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -414,6 +414,17 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +static void io_req_msg_cleanup(struct io_kiocb *req, + struct io_async_msghdr *kmsg, + unsigned int issue_flags) +{ + req->flags &= ~REQ_F_NEED_CLEANUP; + /* fast path, check for non-NULL to avoid function call */ + if (kmsg->free_iov) + kfree(kmsg->free_iov); + io_netmsg_recycle(req, issue_flags); +} + int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) { struct io_sr_msg *sr = io_kiocb_to_cmd(req, struct io_sr_msg); @@ -463,11 +474,7 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret = -EINTR; req_set_fail(req); } - /* fast path, check for non-NULL to avoid function call */ - if (kmsg->free_iov) - kfree(kmsg->free_iov); - req->flags &= ~REQ_F_NEED_CLEANUP; - io_netmsg_recycle(req, issue_flags); + io_req_msg_cleanup(req, kmsg, issue_flags); if (ret >= 0) ret += sr->done_io; else if (sr->done_io) @@ -927,13 +934,8 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) if (!io_recv_finish(req, &ret, &kmsg->msg, mshot_finished, issue_flags)) goto retry_multishot; - if (mshot_finished) { - /* fast path, check for non-NULL to avoid function call */ - if (kmsg->free_iov) - kfree(kmsg->free_iov); - io_netmsg_recycle(req, issue_flags); - req->flags &= ~REQ_F_NEED_CLEANUP; - } + if (mshot_finished) + io_req_msg_cleanup(req, kmsg, issue_flags); return ret; }