From patchwork Wed Aug 24 12:07:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12953324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6AA1C00140 for ; Wed, 24 Aug 2022 12:10:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237241AbiHXMKf (ORCPT ); Wed, 24 Aug 2022 08:10:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237230AbiHXMK0 (ORCPT ); Wed, 24 Aug 2022 08:10:26 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AF833FA26 for ; Wed, 24 Aug 2022 05:10:24 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id n7so14948242ejh.2 for ; Wed, 24 Aug 2022 05:10:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=ce5/ubV4JELaaR3vDudEhMmzRHtnYRw2aHkc66qf1mA=; b=M0p935OtryPrFypp1boj49PbiHEA8oxepnlQ0NWB9pp363fD6LN6iLdnuRucEF1Vah wizrRqdMtxkaJ7PttyymfCXyifeMP0tSBUEVTxien27PSaDw3RLKjbI3X4/l5BloFfAw 1B+5tmznAJ6Q/V3ng/WqU2ldBG00uUD1w59zMFnIBtPBYUSlJ/yHSkd4dRp6gPOwC3wi 5tAeH7WzxlVu0vrs6hIStE7sY0CRI6kwyiGTzsO5+WkTrDg02+lxcqCtChSC8kGdgfNS 0S5QtoDIK0wEkrsc9NUKu94Owrc6l2Xx+d9tuBJL8krrC8nIygv9iQIJd9xAB7wb2bUu u8ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=ce5/ubV4JELaaR3vDudEhMmzRHtnYRw2aHkc66qf1mA=; b=iO12mvMN/iKjhnXYe5Vbit7C+w6zIO+fr7ZjF3eWPag2lF6F6ibfllNDegbShG81+9 9fOXaLE8Y7w7Cf5T7HpZrLl71k3jTtzOYYOwM1palR3CcSMauKc45YkKuHplSgrbONoZ 56UYbR7C6kPE2030HhVJzj8ZIhOMM+P3WLPI4/MNwmFS/RGr1OJTMtmq0aznsp1+2cIC XEJtcivxfnCTPatfVM5hokT7P6fypSb7538T5vhD2BJgAQmrCfwXWN22hYV0MgFTtDaV 9KA7TpMl6fS5CTw5rVXP1PccJ9qg7SR/k8qX9H+mDoWYXFY+A0KRA+x100wqBB1x/FnP yULw== X-Gm-Message-State: ACgBeo2UnSCQHnIEmWiKDbuyJVDLZF5VKNaSvJXy69DnvUgYP2RJeg6B pM+KXs5cN21t+sqPj987Ici0/PFBYtkWOw== X-Google-Smtp-Source: AA6agR7PRntyC7yomtGca141ex0cm0203vySe20sxgk/RUO6B41agskQ/sHtg230Xya6v1hfYI5/qA== X-Received: by 2002:a17:907:da0:b0:730:d0ba:7b13 with SMTP id go32-20020a1709070da000b00730d0ba7b13mr2866249ejc.332.1661343022183; Wed, 24 Aug 2022 05:10:22 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:7067]) by smtp.gmail.com with ESMTPSA id j2-20020a170906410200b007308bdef04bsm1094626ejk.103.2022.08.24.05.10.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:10:21 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Stefan Metzmacher Subject: [PATCH 1/6] io_uring/net: fix must_hold annotation Date: Wed, 24 Aug 2022 13:07:38 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Fix up the io_alloc_notif()'s __must_hold as we don't have a ctx argument there but should get it from the slot instead. Reported-by: Stefan Metzmacher Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 977736e82c1a..568ff17dc552 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -73,7 +73,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, } void io_notif_slot_flush(struct io_notif_slot *slot) - __must_hold(&ctx->uring_lock) + __must_hold(&slot->notif->ctx->uring_lock) { struct io_kiocb *notif = slot->notif; struct io_notif_data *nd = io_notif_to_data(notif); From patchwork Wed Aug 24 12:07:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12953325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D290DC32793 for ; Wed, 24 Aug 2022 12:10:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236032AbiHXMKf (ORCPT ); Wed, 24 Aug 2022 08:10:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237231AbiHXMK0 (ORCPT ); Wed, 24 Aug 2022 08:10:26 -0400 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62FE1BE35 for ; Wed, 24 Aug 2022 05:10:25 -0700 (PDT) Received: by mail-ed1-x535.google.com with SMTP id q2so19637945edb.6 for ; Wed, 24 Aug 2022 05:10:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=dkkPq0XV2TYz8ilMw2LmdxWaWpjg8zabfGHRPH9WEQY=; b=XXVyQGoTC0yc5L0h4l8EL51ss8h/GivENIfnKZDzs3sAOwD+Tcc5oDBNRghydDwnzQ z2awE8dCTe5xcRKX5aMpTrKaTJw2HRKv73Mnc5zzHZUuzWA1RAWFQtMV2XuG2kQN9hJX LeXrzqzIMn6uRq6aIIRrC5POzWXyflA0+4xlXxzTD9cQ1TLLVfCNOOQCmKMxDipJ3vIn ltykUdNMz5obpgu0O4FoC0vDs4/LqpX9nA/ZrHMDbz7nJqpaGSnTxagQUuSgq73aFcb0 4ISgh2QbQfoeDcvmnN9P0PUyoMSm+6pYihSYxWBaCdiwWw951aXBGGXWS/Z7LKv4tQgb ei1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=dkkPq0XV2TYz8ilMw2LmdxWaWpjg8zabfGHRPH9WEQY=; b=UCF+W1Fs2sm3tWliw7berh0iGw+IhG5uA/aKVh2nk1bWoSEZe6r7Lv5ZZ7SLERdwua dQgJ4D9ibpWNj9Ap0tqd5cAT9hDOP25+J1qHwZzaeX7Ed9zP1VSDegOKQ85Ku4Pkdk5p tYdE0GtHQ6lexcptBduFGfztvW6fd/2sQLwhmqTp1R0YxkrFUdZQJtPRdtaVEyqe9ELr R9kWz7AP6tW2X+7aRc6dxa6K9oJcVV5W7o1UbUHKx5L7IGJyy0YHoLvY40ovWbtGjecd FfItMC3Iju/5niStPt0GxjJvmYmroQIGfipqIwiZ9+9SiQEJCu6+Wfyi2lsBFDRQI7j9 12Rw== X-Gm-Message-State: ACgBeo3vI8xWVjLsYtnf+wxHVAuXkmYjywhAVRd0Mp3eHJ5uBYcxrDbW 5Rbzm+E3cmQlA/lVUQpI7e6H/BNO1gHS/Q== X-Google-Smtp-Source: AA6agR4qfd/gbn/82ux3uqS6lhAppAxVIOV/DaWxLjidSLpSyg93OwRhSWQgGsUTJsKkPVsS0sIZ4A== X-Received: by 2002:aa7:d990:0:b0:447:8c86:a8ab with SMTP id u16-20020aa7d990000000b004478c86a8abmr1347291eds.157.1661343023153; Wed, 24 Aug 2022 05:10:23 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:7067]) by smtp.gmail.com with ESMTPSA id j2-20020a170906410200b007308bdef04bsm1094626ejk.103.2022.08.24.05.10.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:10:22 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 2/6] io_uring/net: fix zc send link failing Date: Wed, 24 Aug 2022 13:07:39 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Failed requests should be marked with req_set_fail(), so links and cqe skipping work correctly, which is missing in io_sendzc(). Note, io_sendzc() return IOU_OK on failure, so the core code won't do the cleanup for us. Fixes: 06a5464be84e4 ("io_uring: wire send zc request type") Signed-off-by: Pavel Begunkov --- io_uring/net.c | 1 + 1 file changed, 1 insertion(+) diff --git a/io_uring/net.c b/io_uring/net.c index f8cdf1dc3863..d6310c655a0f 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -1023,6 +1023,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) } if (ret == -ERESTARTSYS) ret = -EINTR; + req_set_fail(req); } else if (zc->flags & IORING_RECVSEND_NOTIF_FLUSH) { io_notif_slot_flush_submit(notif_slot, 0); } From patchwork Wed Aug 24 12:07:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12953323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBEDAC3814E for ; Wed, 24 Aug 2022 12:10:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236738AbiHXMKe (ORCPT ); Wed, 24 Aug 2022 08:10:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237241AbiHXMK0 (ORCPT ); Wed, 24 Aug 2022 08:10:26 -0400 Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B545F2A for ; Wed, 24 Aug 2022 05:10:26 -0700 (PDT) Received: by mail-ej1-x635.google.com with SMTP id n7so14948372ejh.2 for ; Wed, 24 Aug 2022 05:10:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=D/r0bwGS4j43HMvjNKZ5xY8Hl8R6rMXkcIwjNTHFPZs=; b=N8FdgVacw5y4q3O7zBRiLOFYnR3WPHc1qN+nYeQ9oPcwHiZAlNykkU68jhPY81fZdT a/YarC4GUBgOQeWftmx+xVlZZ8gcDcBF+VPxPA4uH7AiI0AM5oH4A4hYFYHJMpuoesPp ky+cGoZMRGR/Fj/DnLaGWfM1oNJ7jdECEnnw62c2kAigfyqvni5fGs/iT5qkshtbOsro s7oTRKZ2IJSCtsGpNV6J810Mo7PW/VEZ3bm2OFD4xnOlB/El8YxaoIuRTbu6Tjk8qk2H vLLchYcaZfR/ZcveDsV1+aKfSPYEI9uJ/E37YdvIjvIRF0Mo3EqpAMMfMmXEQHZHo3xU EnAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=D/r0bwGS4j43HMvjNKZ5xY8Hl8R6rMXkcIwjNTHFPZs=; b=kY1X/d7GP9ilziBRYGvIF2+E1I5yCPRLQV2leC/5HThHXIhip5uevbaz1gaEQDiWg7 eGGjGYU0dAh43WP8vpbI0YP1AEoef+m+LOHjhsq3ilucBqOCSXUkOICWCfL1AeF3TwTf tNe07EVTY6toFutfzjHvFGFl5ZempHJ1pN2tRKEulNpd9m5oTKsBL/FRlh2QAJNzy9ep nrAgJOMOse8TSf9AZwFygWgQ8dB8uwzgAIkdRzTlKBI0wV+9ikEXdiC0dQ6AeZC7xOtA z6xUTSkw5Zy0gOOIaO7HYDQfKL/O3mMrAgZKkNhMKN8vmxNg1hVw+q2becPviGw8Bo+e rweA== X-Gm-Message-State: ACgBeo1ikIBq5VHDrw0Fr22+PzeQpKB8j2iV/IzaCXwL3+TtOwRzdI/k 25FsgxETMxOXApOaeeOl2kKY1XYOrXW4Xw== X-Google-Smtp-Source: AA6agR7cvIGW9228m/Q2vriyFjoqFD4uQnukd7a1zrareBLfwZhnXeihMNlYOdB3k3BW92Ieqd6w1g== X-Received: by 2002:a17:907:3e1d:b0:73d:a9c9:819d with SMTP id hp29-20020a1709073e1d00b0073da9c9819dmr2759830ejc.170.1661343024004; Wed, 24 Aug 2022 05:10:24 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:7067]) by smtp.gmail.com with ESMTPSA id j2-20020a170906410200b007308bdef04bsm1094626ejk.103.2022.08.24.05.10.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:10:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 3/6] io_uring/net: fix indention Date: Wed, 24 Aug 2022 13:07:40 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Fix up indention before we get complaints from tooling. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/net.c b/io_uring/net.c index d6310c655a0f..3adcb09ae264 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -989,7 +989,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) ret = io_import_fixed(WRITE, &msg.msg_iter, req->imu, (u64)(uintptr_t)zc->buf, zc->len); if (unlikely(ret)) - return ret; + return ret; } else { ret = import_single_range(WRITE, zc->buf, zc->len, &iov, &msg.msg_iter); From patchwork Wed Aug 24 12:07:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12953322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A44EC38142 for ; Wed, 24 Aug 2022 12:10:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237268AbiHXMKd (ORCPT ); Wed, 24 Aug 2022 08:10:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237262AbiHXMK2 (ORCPT ); Wed, 24 Aug 2022 08:10:28 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0A003AB30 for ; Wed, 24 Aug 2022 05:10:26 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id y3so6743472ejc.1 for ; Wed, 24 Aug 2022 05:10:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=53NZJ58ZuojeGaGvbnjakOC5xmXKAG4gfl0fQbTWQPA=; b=h4Wei7FE2fKcj/Yq/rbfD+wW+nKUwkOfY+o84FieLhD4ZQieYDGBTfSjOMwHQ/R/yS ZBk0TbOMWjTk/z5HFjmU0KWtBDkbglrxnF713ICuuMlSBU+4KVDHIbEzuP08q9V31q2V PEyB5mzaljXiHpYs3/PGXMkUyWcMof/7Qv3lw2BnVZkGrliT0IcK0pOC4Yiqoqna3KGy njBiw8E1bgxYNOLptAkryLTSAusqeYKktjvh+9+GpM5l/qo0AnL94phBLkr8fX9LzShg gN0ld57Gm103BexBWhonj2p0QmZ6UQFCMwi6jUC+s3hu4QenhPNm9OTICHCmErN6Zmua W4pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=53NZJ58ZuojeGaGvbnjakOC5xmXKAG4gfl0fQbTWQPA=; b=C96Zd8+IpmzsbdREHZ+Bj8Axfl06rQ5NcvJ08iKCGjok2l83MjCmvZsR6dxUvE12o/ L8mSgacRcNMUxUjzlth8zCr0PxVb24lu7uRoH55pRTAsMeN1fm/C3AjzivMwROCuyYCo +Gp7R+CneCU91VluJ3riYvo++VlGXTpN2y+1LbyJsfPR2tEvkissId53WoZmhtBB21g4 ZjU8R6ncLhRn8Dv+HfzeQNXEXBZLnyRxQUfmQA2OnEDZ57/erF+atXlEcxbveaDih8BK jyhTkgopHbmTYq65yH4p+6FZ/liG+jYcYS7ILVEdkg31XD/NQ6GmpgjUZ8iXFjdsXdwn rstw== X-Gm-Message-State: ACgBeo1cGSgkCOepmXYqbWFpJjkiN4UNNTTFklyehtqH2QFp8ZwNMQ/f vAgpruALC8azPDU94DUXWhFUaNf6BMyxcg== X-Google-Smtp-Source: AA6agR62MB20F/aGE51dHVCXFP1MtM7myn8zwwtQelNPnKh09AX6Myldyd8SFH8NTbVyMg/vgW2vhA== X-Received: by 2002:a17:907:6818:b0:730:d99f:7b91 with SMTP id qz24-20020a170907681800b00730d99f7b91mr2638275ejc.496.1661343024944; Wed, 24 Aug 2022 05:10:24 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:7067]) by smtp.gmail.com with ESMTPSA id j2-20020a170906410200b007308bdef04bsm1094626ejk.103.2022.08.24.05.10.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:10:24 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 4/6] io_uring/notif: order notif vs send CQEs Date: Wed, 24 Aug 2022 13:07:41 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Currently, there is no ordering between notification CQEs and completions of the send flushing it, this quite complicates the userspace, especially since we don't flush notification when the send(+flush) request fails, i.e. there will be only one CQE. What we can do is to make sure that notification completions come only after sends. The easiest way to achieve this is to not try to complete a notification inline from io_sendzc() but defer it to task_work, considering that io-wq sendzc is disallowed CQEs will be naturally ordered because task_works will only be executed after we're done with submission and so inline completion. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 568ff17dc552..96f076b175e0 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -81,8 +81,10 @@ void io_notif_slot_flush(struct io_notif_slot *slot) slot->notif = NULL; /* drop slot's master ref */ - if (refcount_dec_and_test(&nd->uarg.refcnt)) - io_notif_complete(notif); + if (refcount_dec_and_test(&nd->uarg.refcnt)) { + notif->io_task_work.func = __io_notif_complete_tw; + io_req_task_work_add(notif); + } } __cold int io_notif_unregister(struct io_ring_ctx *ctx) From patchwork Wed Aug 24 12:07:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12953321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04D2EC00140 for ; Wed, 24 Aug 2022 12:10:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237185AbiHXMKc (ORCPT ); Wed, 24 Aug 2022 08:10:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237280AbiHXMK3 (ORCPT ); Wed, 24 Aug 2022 08:10:29 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B207A3E779 for ; Wed, 24 Aug 2022 05:10:27 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id sd33so11771279ejc.8 for ; Wed, 24 Aug 2022 05:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=vT2SMsi+70IiyW0tafUfZBivZ5hq8kKCBP0oxP+zUEc=; b=Y8ySVizk0AyS4lAF6YnNM991COxMujbDwBh4Pbzd7AcHhphSZoR2P/8iNJyXFe0p0I H8j3HR5+4MpmAZBWduM78BanFTbKEKLZglcd225aJNvoRjUnKJUZIJCKhu9KeaSlgWWm flnVM6mfO/TC0mIxjHSbCgK1nkbruyaGJcvPFU88bEBcAlcCX7hksQt8Qs/QrNjFBBZC 5ICjlcXGu90X/v4RU9rAyQ0YAdNRjg4RT24FXrMlNNDCbX5A4BG6zhH+sgcRk4PivRWg ixtuFKBBoZmB1FYkmuBvL2a6q62IvcO7KfUNf5lQEHTCsJZvyN/iRPJnF8kUjD1qyBDl lLaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=vT2SMsi+70IiyW0tafUfZBivZ5hq8kKCBP0oxP+zUEc=; b=vQ7BoKw2BHOwMuCABSQkuTKsGvHpRZ+eZpdnyz8EV6/YjFnmRhahWZBeaSVO3QB0/y jqD3h1AtbFDEWJf2qZNz4H3dv6bdaol1fNZFEw0+jZH3hIMqFSJ6Js71RPCigd1Nbint iYjYKr4/smzcESqbPalYXdq125C+99fjqvWnMnnnULusBmFxCQNUSvcdBnSiUM+9SigJ OCCVrp1y21nHdp1TwbW/8ujUrw6woztqUprSa67rIn/nPIY0IaFaMvPIPX0F6EpQeOIV m35RrWNFj1U553Kro1ohWmToN1UchqZqNQMti5C5JEBUbaOVhoN+8FL2qI5dInxeJxOu YpCA== X-Gm-Message-State: ACgBeo3hr7qUijqtKS8C3a98uERVr057lU4+7ebFQquqreeMI2zLSrIi 4PMs+p4EFSxz0KrGEApi74w7B/xJZkhVJA== X-Google-Smtp-Source: AA6agR7uHl1AvUn8D92HXpiIlF8Jw5CEf/3eLFqxXVRtM7esHGp/z3q9DydbWmzNSQ6EHtAxBi9fXQ== X-Received: by 2002:a17:907:60c7:b0:731:14e2:af10 with SMTP id hv7-20020a17090760c700b0073114e2af10mr2733827ejc.92.1661343026014; Wed, 24 Aug 2022 05:10:26 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:7067]) by smtp.gmail.com with ESMTPSA id j2-20020a170906410200b007308bdef04bsm1094626ejk.103.2022.08.24.05.10.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:10:25 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com Subject: [PATCH 5/6] io_uring: conditional ->async_data allocation Date: Wed, 24 Aug 2022 13:07:42 +0100 Message-Id: <9dc62be9e88dd0ed63c48365340e8922d2498293.1661342812.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There are opcodes that need ->async_data only in some cases and allocation it unconditionally may hurt performance. Add an option to opdef to make move the allocation part from the core io_uring to opcode specific code. Note, we can't just set opdef->async_size to zero because there are other helpers that rely on it, e.g. io_alloc_async_data(). Signed-off-by: Pavel Begunkov --- io_uring/io_uring.c | 7 ++++--- io_uring/opdef.h | 2 ++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index ebfdb2212ec2..77616279000b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1450,9 +1450,10 @@ int io_req_prep_async(struct io_kiocb *req) return 0; if (WARN_ON_ONCE(req_has_async_data(req))) return -EFAULT; - if (io_alloc_async_data(req)) - return -EAGAIN; - + if (!io_op_defs[req->opcode].manual_alloc) { + if (io_alloc_async_data(req)) + return -EAGAIN; + } return def->prep_async(req); } diff --git a/io_uring/opdef.h b/io_uring/opdef.h index ece8ed4f96c4..763c6e54e2ee 100644 --- a/io_uring/opdef.h +++ b/io_uring/opdef.h @@ -25,6 +25,8 @@ struct io_op_def { unsigned ioprio : 1; /* supports iopoll */ unsigned iopoll : 1; + /* opcode specific path will handle ->async_data allocation if needed */ + unsigned manual_alloc : 1; /* size of async data needed, if any */ unsigned short async_size; From patchwork Wed Aug 24 12:07:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12953327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DB1BC00140 for ; Wed, 24 Aug 2022 12:10:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237289AbiHXMKm (ORCPT ); Wed, 24 Aug 2022 08:10:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37946 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237325AbiHXMKa (ORCPT ); Wed, 24 Aug 2022 08:10:30 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC29A402CC for ; Wed, 24 Aug 2022 05:10:28 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id w19so33141268ejc.7 for ; Wed, 24 Aug 2022 05:10:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=OTyuNR9mG76hXCIvBRUfIr/jVV32q+g4El0FvvVYohU=; b=RbTPM2PJchJZ/g5k8ym3yrfgMKb8i5/gz6RvPRJEWOF3YyfHeX5cIXMeEqbnXTk8Ik UdySW7/GglDSZjacKflnIkYG1cEWe5Y9tyTuAzLJI6aaEwrGeLM16TMr0+EUKrttesxH v8zqKFXHoAbuAx7gGw/PqSJ2hyVLDYx1/USKuto7fEifjNP5OBHpsW5fb8jq69je6PEU 1TGsOqhFmuqiKmagR0mXxzyivNiubiIhTl2aa/YO6HdJPR07MmNjuUFWjp+HXxA7CWtf GBwld94GYDVAD1Wtu00X6JjQRk+h9qaXSQJ5T0AWyhK7D7iKKPviAD1zDbcrg5f1Gl0t tReg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=OTyuNR9mG76hXCIvBRUfIr/jVV32q+g4El0FvvVYohU=; b=TcJMWiu9gxxajc+i7e1ywGZnY2X2THmOMHrh7EkDqmE80gImlKFql96Psh+x0O69WW o3A8qRndWP9sNzN/bH2WIDQEekO3qcx+QDmcCOKiTL40YdiPc5UeNbSMSl+HtQAYJjbm sMCZQTuNx8R+W//OAlS4e18/iAi5UqrWu5FzXYzxOEWlZMOWOYVgOqc5bpJ7WkpVIRBt viUiMSnKxjFzkkL7uPTA6pp1hHKl19vDhRLDPePKBCfxbjSNIAn5Zxmqg42+3bxGdqVo yt2/Rvo9YMlrkHQ0Jg9qDLqhOqNfKwMmIxsi15Q3uX5VWZ7YuRSWKWtyAzPE+I6bfeBm /QqA== X-Gm-Message-State: ACgBeo0Tfi2d/xTVe6E5h+pgVQ83YxVsRL9Un1ejVqBD/VZdFY0UJdn/ 5DjcwxS0Q+3kUeDOLznrRfp93hyajvLGJw== X-Google-Smtp-Source: AA6agR6x9JLPjnPBNm3ZyR8w9u2sTiN97L2vpWc4dQAesZRONGjlTkFLOCMSote+CsxnOtPbGzXurA== X-Received: by 2002:a17:907:e8c:b0:73d:8146:9aa1 with SMTP id ho12-20020a1709070e8c00b0073d81469aa1mr2671746ejc.253.1661343026978; Wed, 24 Aug 2022 05:10:26 -0700 (PDT) Received: from 127.0.0.1localhost.com ([2620:10d:c092:600::2:7067]) by smtp.gmail.com with ESMTPSA id j2-20020a170906410200b007308bdef04bsm1094626ejk.103.2022.08.24.05.10.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Aug 2022 05:10:26 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, Stefan Metzmacher Subject: [PATCH 6/6] io_uring/net: save address for sendzc async execution Date: Wed, 24 Aug 2022 13:07:43 +0100 Message-Id: X-Mailer: git-send-email 2.37.2 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We usually copy all bits that a request needs from the userspace for async execution, so the userspace can keep them on the stack. However, send zerocopy violates this pattern for addresses and may reloads it e.g. from io-wq. Save the address if any in ->async_data as usual. Reported-by: Stefan Metzmacher Signed-off-by: Pavel Begunkov --- io_uring/net.c | 52 +++++++++++++++++++++++++++++++++++++++++------- io_uring/net.h | 1 + io_uring/opdef.c | 4 +++- 3 files changed, 49 insertions(+), 8 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index 3adcb09ae264..4eaeb805e720 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -182,6 +182,37 @@ static int io_sendmsg_copy_hdr(struct io_kiocb *req, &iomsg->free_iov); } +int io_sendzc_prep_async(struct io_kiocb *req) +{ + struct io_sendzc *zc = io_kiocb_to_cmd(req, struct io_sendzc); + struct io_async_msghdr *io; + int ret; + + if (!zc->addr || req_has_async_data(req)) + return 0; + if (io_alloc_async_data(req)) + return -ENOMEM; + + io = req->async_data; + ret = move_addr_to_kernel(zc->addr, zc->addr_len, &io->addr); + return ret; +} + +static int io_setup_async_addr(struct io_kiocb *req, + struct sockaddr_storage *addr, + unsigned int issue_flags) +{ + struct io_async_msghdr *io; + + if (!addr || req_has_async_data(req)) + return -EAGAIN; + if (io_alloc_async_data(req)) + return -ENOMEM; + io = req->async_data; + memcpy(&io->addr, addr, sizeof(io->addr)); + return -EAGAIN; +} + int io_sendmsg_prep_async(struct io_kiocb *req) { int ret; @@ -944,7 +975,7 @@ static int io_sg_from_iter(struct sock *sk, struct sk_buff *skb, int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) { - struct sockaddr_storage address; + struct sockaddr_storage __address, *addr; struct io_ring_ctx *ctx = req->ctx; struct io_sendzc *zc = io_kiocb_to_cmd(req, struct io_sendzc); struct io_notif_slot *notif_slot; @@ -978,10 +1009,16 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) msg.msg_namelen = 0; if (zc->addr) { - ret = move_addr_to_kernel(zc->addr, zc->addr_len, &address); - if (unlikely(ret < 0)) - return ret; - msg.msg_name = (struct sockaddr *)&address; + if (req_has_async_data(req)) { + struct io_async_msghdr *io = req->async_data; + + msg.msg_name = &io->addr; + } else { + ret = move_addr_to_kernel(zc->addr, zc->addr_len, &__address); + if (unlikely(ret < 0)) + return ret; + msg.msg_name = (struct sockaddr *)&__address; + } msg.msg_namelen = zc->addr_len; } @@ -1013,13 +1050,14 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) - return -EAGAIN; + return io_setup_async_addr(req, addr, issue_flags); + if (ret > 0 && io_net_retry(sock, msg.msg_flags)) { zc->len -= ret; zc->buf += ret; zc->done_io += ret; req->flags |= REQ_F_PARTIAL_IO; - return -EAGAIN; + return io_setup_async_addr(req, addr, issue_flags); } if (ret == -ERESTARTSYS) ret = -EINTR; diff --git a/io_uring/net.h b/io_uring/net.h index 7c438d39c089..f91f56c6eeac 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -31,6 +31,7 @@ struct io_async_connect { int io_shutdown_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); int io_shutdown(struct io_kiocb *req, unsigned int issue_flags); +int io_sendzc_prep_async(struct io_kiocb *req); int io_sendmsg_prep_async(struct io_kiocb *req); void io_sendmsg_recvmsg_cleanup(struct io_kiocb *req); int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); diff --git a/io_uring/opdef.c b/io_uring/opdef.c index 72dd2b2d8a9d..41410126c1c6 100644 --- a/io_uring/opdef.c +++ b/io_uring/opdef.c @@ -478,13 +478,15 @@ const struct io_op_def io_op_defs[] = { .pollout = 1, .audit_skip = 1, .ioprio = 1, + .manual_alloc = 1, #if defined(CONFIG_NET) + .async_size = sizeof(struct io_async_msghdr), .prep = io_sendzc_prep, .issue = io_sendzc, + .prep_async = io_sendzc_prep_async, #else .prep = io_eopnotsupp_prep, #endif - }, };