From patchwork Wed Aug 10 15:49:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68A59C00140 for ; Wed, 10 Aug 2022 15:51:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232049AbiHJPvm (ORCPT ); Wed, 10 Aug 2022 11:51:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231928AbiHJPvk (ORCPT ); Wed, 10 Aug 2022 11:51:40 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56B8349B60; Wed, 10 Aug 2022 08:51:39 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id z12so18222301wrs.9; Wed, 10 Aug 2022 08:51:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=mveucMTZdZPLlN4D5K1kqfZw5UNWDwp9Ns7O7VRFK+I=; b=FeHdakLvXxtWgjpPF+aflE98THue/ZKUpEuSPYY/vwKS6pLky2/FycS8L1DaAOP7p2 gbmwq+ySuj1nbkojJhrktc86MeoeDdNZHmrmEXeui4TvX9dWZtL5o1pn/n2SFPu2UKqJ LwG8Bt1+6Z9HVd12TbOjrwFHHO7Xbx+0NZ4knAvVP00VnvWpTA1vdoX8qFWZUEU1a0Hc WTzV9hRGh/acAWBr/K4D/fYN2WXLMaChMzaEobWBKN01dp0GD77rhc2rXo7Zw1Rse8vI EoYL8E2KpOU5sN0NjoXacILSTjNTJslWSxRKaYX/is4eawuPkbRaWRlUK6vsWA5Q252i fHCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=mveucMTZdZPLlN4D5K1kqfZw5UNWDwp9Ns7O7VRFK+I=; b=VEJoyOrdLQBPaiK440vKvKQmYJQZF1w3S1FoRCeD0gVrFfwZfWOBqtEY2hYwzg4NkJ Xx5pj3Dzp8i3vpjdmqQE9JoE2PQX37BVb7tnI96hrFCIvtqjCUOQvZfiV1fJOLoCkHIZ ImlOjofUB33KtvWVYkiiDaWT6RTEYcMcaOJpHYnzufh3xb1orTfrxPWtp4Mjf1ZpCt97 3o23hZvZpwVJKPoQ9/tQsF7s5MwlvgKaA0k4HuipYXDyP98uf4C/0j5gaWGgfC8O2684 vNvxyXRlvp9Ob1i5nsgv1ekzSO+CCtVFWBTqkklDP67eqYa1Djy+n/PzSxa2pdRW3Sz6 9aYQ== X-Gm-Message-State: ACgBeo3Bi4n6B4w3tWe7h4HZ9hId0z23fBeDjuDtGKTM9w/nJ0/6xyK3 Iz2Beuv8R8OWYXfg7dF6Wx31QtWH9Ts= X-Google-Smtp-Source: AA6agR6IVj+Term37QFPdVaJx8duP8zGzs+p0efJL76vLIQCk5ivpP0E95/E2mMnqsqquEDgBKZAtg== X-Received: by 2002:a05:6000:3c6:b0:220:5efd:423c with SMTP id b6-20020a05600003c600b002205efd423cmr18135634wrg.214.1660146697602; Wed, 10 Aug 2022 08:51:37 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:37 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 01/11] net: introduce struct ubuf_info_msgzc Date: Wed, 10 Aug 2022 16:49:09 +0100 Message-Id: X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We're going to split struct ubuf_info and leave there only mandatory fields. Users are free to extend it. Add struct ubuf_info_msgzc, which will be an extended version for MSG_ZEROCOPY and some other users. It duplicates of struct ubuf_info for now and will be removed in a couple of patches. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index ca8afa382bf2..f8ac3678dab8 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -554,7 +554,28 @@ struct ubuf_info { } mmp; }; +struct ubuf_info_msgzc { + struct ubuf_info ubuf; + + union { + struct { + unsigned long desc; + void *ctx; + }; + struct { + u32 id; + u16 len; + u16 zerocopy:1; + u32 bytelen; + }; + }; + + struct mmpin mmp; +}; + #define skb_uarg(SKB) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg)) +#define uarg_to_msgzc(ubuf_ptr) container_of((ubuf_ptr), struct ubuf_info_msgzc, \ + ubuf) int mm_account_pinned_pages(struct mmpin *mmp, size_t size); void mm_unaccount_pinned_pages(struct mmpin *mmp); From patchwork Wed Aug 10 15:49:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56003C00140 for ; Wed, 10 Aug 2022 15:51:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232187AbiHJPvy (ORCPT ); Wed, 10 Aug 2022 11:51:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231687AbiHJPvm (ORCPT ); Wed, 10 Aug 2022 11:51:42 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDD724C617; Wed, 10 Aug 2022 08:51:40 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id bv3so18211594wrb.5; Wed, 10 Aug 2022 08:51:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=RS63Hb0HKwQLj+yY8E+g8nWuxDYTXWs65j0+OKPH1OE=; b=ZQdOMbpO/VHyANe5mleak9CanXYG2CBuqoaXodVPqSxfN0jYX3Wpv/vcb6iubMBW1u 4Ble+mSyTPc7xm8K/FN13yhEm7K3hDV1my/RtaOdys3+jH8lzdROVrs3yday1zidVe+x bMEKgzVG6yfs6m4/lp21crX1YCY0qdX1LO/SN3r1GTpQB97R7NYdjhtr4vcyw0X1MhlZ Rsr3NqzS5od350DfBp0/rWIgXTPVwoJA57tT5sN5ssmtZG7YNPmWvIuqmqqLbLD3mFiz LM3JoK9cwamnuJVbkAxMoi+VNXdtwhLPXE2PCWt/23HgKhwIUuswHMgcOCy1QLV1mNAn Xu2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=RS63Hb0HKwQLj+yY8E+g8nWuxDYTXWs65j0+OKPH1OE=; b=X2vFMomXGDOwWZ+7D9vzD4/w+Cdz9e9ao4ouwVdY0iSrp454nW7gGRDMIaIIJnL0Zw DB19kKAhXYO5YD+GwCmu3IT+UiDhxR1t9nxhks+Gazino8z8qwGl5AX0LLn3HIsE38Ko WI019uls1D1ZZuLb2ENgtb1IuYs5D/I7NF5FT/C0K1PMP1DCi45WuECxM2JTUktlR/Ry 4WUtk1zQ+PcxwQnkfeKLL5hN3xJI0pXDP+rEv84l0Y8Es9Uw5lXRxReIJhEqDJgxeBZ+ 6+eJZct4tJoVFAPIB3FafyqUdSWKuZpKFbCZXdfd41rauCidMJJPG3njtlkoem6OSlHX /rWg== X-Gm-Message-State: ACgBeo1vNFjBUpYM3+sGv98bHJfQo5xses+P9BBJL39T+F52ZJCyee6Y cqLy69RCJReRNZU1NxfEDmIn90inKbw= X-Google-Smtp-Source: AA6agR4m7a1ZpBafnXxHxc3Qgb+mIUi9eJAFTWTIHDXvxrcaOuXHorpWp86zCAS5S/bxu6e6MTh00Q== X-Received: by 2002:a5d:64aa:0:b0:21e:be27:6dfb with SMTP id m10-20020a5d64aa000000b0021ebe276dfbmr18180217wrp.456.1660146698952; Wed, 10 Aug 2022 08:51:38 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:38 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 02/11] xen/netback: use struct ubuf_info_msgzc Date: Wed, 10 Aug 2022 16:49:10 +0100 Message-Id: <9d13d0d17d672ce55a3d5cfd06d5c521270a7d4d.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org struct ubuf_info will be changed, use ubuf_info_msgzc instead. Signed-off-by: Pavel Begunkov --- drivers/net/xen-netback/common.h | 2 +- drivers/net/xen-netback/interface.c | 4 ++-- drivers/net/xen-netback/netback.c | 7 ++++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/drivers/net/xen-netback/common.h b/drivers/net/xen-netback/common.h index 8174d7b2966c..1545cbee77a4 100644 --- a/drivers/net/xen-netback/common.h +++ b/drivers/net/xen-netback/common.h @@ -62,7 +62,7 @@ struct pending_tx_info { * ubuf_to_vif is a helper which finds the struct xenvif from a pointer * to this field. */ - struct ubuf_info callback_struct; + struct ubuf_info_msgzc callback_struct; }; #define XEN_NETIF_TX_RING_SIZE __CONST_RING_SIZE(xen_netif_tx, XEN_PAGE_SIZE) diff --git a/drivers/net/xen-netback/interface.c b/drivers/net/xen-netback/interface.c index fb32ae82d9b0..e579ecd40b74 100644 --- a/drivers/net/xen-netback/interface.c +++ b/drivers/net/xen-netback/interface.c @@ -591,8 +591,8 @@ int xenvif_init_queue(struct xenvif_queue *queue) } for (i = 0; i < MAX_PENDING_REQS; i++) { - queue->pending_tx_info[i].callback_struct = (struct ubuf_info) - { .callback = xenvif_zerocopy_callback, + queue->pending_tx_info[i].callback_struct = (struct ubuf_info_msgzc) + { { .callback = xenvif_zerocopy_callback }, { { .ctx = NULL, .desc = i } } }; queue->grant_tx_handle[i] = NETBACK_INVALID_HANDLE; diff --git a/drivers/net/xen-netback/netback.c b/drivers/net/xen-netback/netback.c index a256695fc89e..3d2081bbbc86 100644 --- a/drivers/net/xen-netback/netback.c +++ b/drivers/net/xen-netback/netback.c @@ -133,7 +133,7 @@ static inline unsigned long idx_to_kaddr(struct xenvif_queue *queue, /* Find the containing VIF's structure from a pointer in pending_tx_info array */ -static inline struct xenvif_queue *ubuf_to_queue(const struct ubuf_info *ubuf) +static inline struct xenvif_queue *ubuf_to_queue(const struct ubuf_info_msgzc *ubuf) { u16 pending_idx = ubuf->desc; struct pending_tx_info *temp = @@ -1228,11 +1228,12 @@ static int xenvif_tx_submit(struct xenvif_queue *queue) return work_done; } -void xenvif_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *ubuf, +void xenvif_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *ubuf_base, bool zerocopy_success) { unsigned long flags; pending_ring_idx_t index; + struct ubuf_info_msgzc *ubuf = uarg_to_msgzc(ubuf_base); struct xenvif_queue *queue = ubuf_to_queue(ubuf); /* This is the only place where we grab this lock, to protect callbacks @@ -1241,7 +1242,7 @@ void xenvif_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *ubuf, spin_lock_irqsave(&queue->callback_lock, flags); do { u16 pending_idx = ubuf->desc; - ubuf = (struct ubuf_info *) ubuf->ctx; + ubuf = (struct ubuf_info_msgzc *) ubuf->ctx; BUG_ON(queue->dealloc_prod - queue->dealloc_cons >= MAX_PENDING_REQS); index = pending_index(queue->dealloc_prod); From patchwork Wed Aug 10 15:49:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87637C19F2A for ; Wed, 10 Aug 2022 15:51:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232231AbiHJPvz (ORCPT ); Wed, 10 Aug 2022 11:51:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232099AbiHJPvo (ORCPT ); Wed, 10 Aug 2022 11:51:44 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2349A4E873; Wed, 10 Aug 2022 08:51:42 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id a18-20020a05600c349200b003a30de68697so1852910wmq.0; Wed, 10 Aug 2022 08:51:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=zKsqv0p5Obkg1TLb6/kUNEYbyvoYoWpfWSOapsGCJxI=; b=DFTQlY8ddG8nxlheZEdY1AhOzyhFf8i+cYlGvYaQA2jXHoy604PEPidno8824FFofg YVuRVy3aoGvd+VxIlbyTojzVgA6ObPKrHOe3N+BtwnJ8/SiHLCzaUthyY809sB53Uy0k A80hQEmxtBOQP5ObevE2QDwC8UdnYAyiC/9NRGBeNLRJhgqxJeTUEIEq3lyZCycgRAKv YeRgjQWebysXI2hCFaZdjrhd9yqY/GnXi1hw7QfcxVhAyGjXJUcOFXn8ZTBLEhYDSjyB tkKzffcoleVU5wSv7BENj8Sif4EsQ6FQBYNVk3yJnkXWSuO8uva7e43fGJIcAy/AtU+c ghIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=zKsqv0p5Obkg1TLb6/kUNEYbyvoYoWpfWSOapsGCJxI=; b=G9aK6Gly5GgY0QpJ8fb3amum62meRzCgj4rlDdGRL7cdUR0YivFKfTWAhAvu7vxpOp 0wB/xQk8WsXnhzymZnVag0UME1xBRGnUX35uKYhOLVt2gMswBcrvRxTi1UwYf5QEtFyX SRvB+EZX4NiRCVpY3QMF/uxofU3cOxNSvBBvpV5N4qLU0sdBvfoFgZvXuvDBEBlY77mu fmWALm5IHL7PBE+kuxowScTaRIq7D0ik75XB40oS1CDQYscfYfWeCEhueqWjiB87ht7v eBAW/nhhiIhU17GEZXgVO2VBMNegPdbDShUgkE9feIvukg6MyiWXTsMCVIuzuwiFTIHz rvvw== X-Gm-Message-State: ACgBeo0/yTQ7g7bWh9HGB6rBfLllfriWPCVLpm39gwfO0vwOqw9ijL4E 0oPjeNib66T+mzxDVPa807JLE5xt5KY= X-Google-Smtp-Source: AA6agR7LF0ZwQ1BlKaVDicKs5LglruTEZwAxo7zyYgmWusQVZHSM24Cst0gJV7pB4XoOeH/qUhDY2A== X-Received: by 2002:a05:600c:28cd:b0:3a5:4f45:b927 with SMTP id h13-20020a05600c28cd00b003a54f45b927mr2964921wmd.90.1660146700292; Wed, 10 Aug 2022 08:51:40 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:39 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 03/11] vhost/net: use struct ubuf_info_msgzc Date: Wed, 10 Aug 2022 16:49:11 +0100 Message-Id: <87cfb629defa270e5ed953c6e501a47278f916c9.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org struct ubuf_info will be changed, use ubuf_info_msgzc instead. Signed-off-by: Pavel Begunkov --- drivers/vhost/net.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 68e4ecd1cc0e..9b616536dd9e 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -118,7 +118,7 @@ struct vhost_net_virtqueue { /* Number of XDP frames batched */ int batched_xdp; /* an array of userspace buffers info */ - struct ubuf_info *ubuf_info; + struct ubuf_info_msgzc *ubuf_info; /* Reference counting for outstanding ubufs. * Protected by vq mutex. Writers must also take device mutex. */ struct vhost_net_ubuf_ref *ubufs; @@ -288,7 +288,7 @@ static int vhost_net_set_ubuf_info(struct vhost_net *n) n->vqs[i].ubuf_info = kmalloc_array(UIO_MAXIOV, sizeof(*n->vqs[i].ubuf_info), - GFP_KERNEL); + GFP_KERNEL | __GFP_ZERO); if (!n->vqs[i].ubuf_info) goto err; } @@ -382,8 +382,9 @@ static void vhost_zerocopy_signal_used(struct vhost_net *net, } static void vhost_zerocopy_callback(struct sk_buff *skb, - struct ubuf_info *ubuf, bool success) + struct ubuf_info *ubuf_base, bool success) { + struct ubuf_info_msgzc *ubuf = uarg_to_msgzc(ubuf_base); struct vhost_net_ubuf_ref *ubufs = ubuf->ctx; struct vhost_virtqueue *vq = ubufs->vq; int cnt; @@ -871,7 +872,7 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) size_t len, total_len = 0; int err; struct vhost_net_ubuf_ref *ubufs; - struct ubuf_info *ubuf; + struct ubuf_info_msgzc *ubuf; bool zcopy_used; int sent_pkts = 0; @@ -907,14 +908,14 @@ static void handle_tx_zerocopy(struct vhost_net *net, struct socket *sock) ubuf = nvq->ubuf_info + nvq->upend_idx; vq->heads[nvq->upend_idx].id = cpu_to_vhost32(vq, head); vq->heads[nvq->upend_idx].len = VHOST_DMA_IN_PROGRESS; - ubuf->callback = vhost_zerocopy_callback; ubuf->ctx = nvq->ubufs; ubuf->desc = nvq->upend_idx; - ubuf->flags = SKBFL_ZEROCOPY_FRAG; - refcount_set(&ubuf->refcnt, 1); + ubuf->ubuf.callback = vhost_zerocopy_callback; + ubuf->ubuf.flags = SKBFL_ZEROCOPY_FRAG; + refcount_set(&ubuf->ubuf.refcnt, 1); msg.msg_control = &ctl; ctl.type = TUN_MSG_UBUF; - ctl.ptr = ubuf; + ctl.ptr = &ubuf->ubuf; msg.msg_controllen = sizeof(ctl); ubufs = nvq->ubufs; atomic_inc(&ubufs->refcount); From patchwork Wed Aug 10 15:49:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0255C00140 for ; Wed, 10 Aug 2022 15:52:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232384AbiHJPv7 (ORCPT ); Wed, 10 Aug 2022 11:51:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232198AbiHJPvz (ORCPT ); Wed, 10 Aug 2022 11:51:55 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 92B895005A; Wed, 10 Aug 2022 08:51:43 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id z16so18165782wrh.12; Wed, 10 Aug 2022 08:51:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=k7b9egffbhRjX5/OjtOx/by90hRcurZmtiI07zTQ9r0=; b=DO5S08zBi3vihZLd+TPnKuLvUBVq1/APvJ0DNR/ZVpJjz4DlQk8pVoCf++lDJQ+qCe J5GiO2q7rRcXgkXHqvH+Xa7gnatlgZJV5TwqmNquXIn+gL91mG56ATFSsq9BpXpc7zr0 /1S04KGFgf42+WhssmeRXUhlovG5xP8Cx3T/uzDl82B5aLUk0pPU9HQVxh+wNlScY3LK CgQH74Kc4vSw87U7lXPsLH2C2ORUZZBxNzu/uVo2SSLsJLJNwcM+/ACew/XMVXYxRLdn VTO2yH9kAmRsQtleJsYuYUQbcBz0OVWz31KhDO6+31seXNLpFkbd/CzwI333S10XzntZ GzzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=k7b9egffbhRjX5/OjtOx/by90hRcurZmtiI07zTQ9r0=; b=q/aeco9XRi2IFPIcejKflxKd0Om9wO1ELYLp3YbomKutJtSwFGvsWVq9k9/l16CDLc XQKpyTBmiwaW4DsAOen0R36wRG+WZcmykT/q+RXHpcSJ8MSa8x/KKCtmjPJQJZjV4hxZ mtAid9W4kcyVLLFjSddAjIORntqpW2OaP9RFkzyjT5OG+AS2yKQm1DBiNR+gwFjuczWT Z4ZhBKmPE/SLMyM4YCQDfizARq/IzqMDVaMjEruok/2Da3IVg1M8me7Sj9XrY9Ai2TpV fSMpgT6zqA6w5Y5OIEfSXr3DjUkRS80tG/ipAi+r/9ns3fvZLeAEpim8ju5onVNEiUHm GmEA== X-Gm-Message-State: ACgBeo3z5VrdV0PPMyT1yNPjbjeWoQGP7HnojBNr/CS509fS7Yx8Z+Pn EIORSP9dx6HuVcpB681N+XUIzfWqTAw= X-Google-Smtp-Source: AA6agR6suRfvAaYpMQRv+C0nU04QPftK1r02sWI7OLgKf0C0mEfWd53MX+qfOk7MeFvdNRw94olKdA== X-Received: by 2002:adf:db8f:0:b0:220:6059:5284 with SMTP id u15-20020adfdb8f000000b0022060595284mr18041360wri.33.1660146701918; Wed, 10 Aug 2022 08:51:41 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:41 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 04/11] net: shrink struct ubuf_info Date: Wed, 10 Aug 2022 16:49:12 +0100 Message-Id: <1e2d0bef44cd68b0e3dd106283f4536645c2a124.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We can benefit from a smaller struct ubuf_info, so leave only mandatory fields and let users to decide how they want to extend it. Convert MSG_ZEROCOPY to struct ubuf_info_msgzc and remove duplicated fields. This reduces the size from 48 bytes to just 16. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 22 ++++------------------ net/core/skbuff.c | 38 +++++++++++++++++++++----------------- net/ipv4/ip_output.c | 2 +- net/ipv4/tcp.c | 2 +- net/ipv6/ip6_output.c | 2 +- 5 files changed, 28 insertions(+), 38 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index f8ac3678dab8..afd7400d7f62 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -533,25 +533,8 @@ enum { struct ubuf_info { void (*callback)(struct sk_buff *, struct ubuf_info *, bool zerocopy_success); - union { - struct { - unsigned long desc; - void *ctx; - }; - struct { - u32 id; - u16 len; - u16 zerocopy:1; - u32 bytelen; - }; - }; refcount_t refcnt; u8 flags; - - struct mmpin { - struct user_struct *user; - unsigned int num_pg; - } mmp; }; struct ubuf_info_msgzc { @@ -570,7 +553,10 @@ struct ubuf_info_msgzc { }; }; - struct mmpin mmp; + struct mmpin { + struct user_struct *user; + unsigned int num_pg; + } mmp; }; #define skb_uarg(SKB) ((struct ubuf_info *)(skb_shinfo(SKB)->destructor_arg)) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 974bbbbe7138..b047a773acd7 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1183,7 +1183,7 @@ EXPORT_SYMBOL_GPL(mm_unaccount_pinned_pages); static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) { - struct ubuf_info *uarg; + struct ubuf_info_msgzc *uarg; struct sk_buff *skb; WARN_ON_ONCE(!in_task()); @@ -1201,19 +1201,19 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) return NULL; } - uarg->callback = msg_zerocopy_callback; + uarg->ubuf.callback = msg_zerocopy_callback; uarg->id = ((u32)atomic_inc_return(&sk->sk_zckey)) - 1; uarg->len = 1; uarg->bytelen = size; uarg->zerocopy = 1; - uarg->flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; - refcount_set(&uarg->refcnt, 1); + uarg->ubuf.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + refcount_set(&uarg->ubuf.refcnt, 1); sock_hold(sk); - return uarg; + return &uarg->ubuf; } -static inline struct sk_buff *skb_from_uarg(struct ubuf_info *uarg) +static inline struct sk_buff *skb_from_uarg(struct ubuf_info_msgzc *uarg) { return container_of((void *)uarg, struct sk_buff, cb); } @@ -1222,6 +1222,7 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, struct ubuf_info *uarg) { if (uarg) { + struct ubuf_info_msgzc *uarg_zc; const u32 byte_limit = 1 << 19; /* limit to a few TSO */ u32 bytelen, next; @@ -1237,8 +1238,9 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, return NULL; } - bytelen = uarg->bytelen + size; - if (uarg->len == USHRT_MAX - 1 || bytelen > byte_limit) { + uarg_zc = uarg_to_msgzc(uarg); + bytelen = uarg_zc->bytelen + size; + if (uarg_zc->len == USHRT_MAX - 1 || bytelen > byte_limit) { /* TCP can create new skb to attach new uarg */ if (sk->sk_type == SOCK_STREAM) goto new_alloc; @@ -1246,11 +1248,11 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, } next = (u32)atomic_read(&sk->sk_zckey); - if ((u32)(uarg->id + uarg->len) == next) { - if (mm_account_pinned_pages(&uarg->mmp, size)) + if ((u32)(uarg_zc->id + uarg_zc->len) == next) { + if (mm_account_pinned_pages(&uarg_zc->mmp, size)) return NULL; - uarg->len++; - uarg->bytelen = bytelen; + uarg_zc->len++; + uarg_zc->bytelen = bytelen; atomic_set(&sk->sk_zckey, ++next); /* no extra ref when appending to datagram (MSG_MORE) */ @@ -1286,7 +1288,7 @@ static bool skb_zerocopy_notify_extend(struct sk_buff *skb, u32 lo, u16 len) return true; } -static void __msg_zerocopy_callback(struct ubuf_info *uarg) +static void __msg_zerocopy_callback(struct ubuf_info_msgzc *uarg) { struct sk_buff *tail, *skb = skb_from_uarg(uarg); struct sock_exterr_skb *serr; @@ -1339,19 +1341,21 @@ static void __msg_zerocopy_callback(struct ubuf_info *uarg) void msg_zerocopy_callback(struct sk_buff *skb, struct ubuf_info *uarg, bool success) { - uarg->zerocopy = uarg->zerocopy & success; + struct ubuf_info_msgzc *uarg_zc = uarg_to_msgzc(uarg); + + uarg_zc->zerocopy = uarg_zc->zerocopy & success; if (refcount_dec_and_test(&uarg->refcnt)) - __msg_zerocopy_callback(uarg); + __msg_zerocopy_callback(uarg_zc); } EXPORT_SYMBOL_GPL(msg_zerocopy_callback); void msg_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref) { - struct sock *sk = skb_from_uarg(uarg)->sk; + struct sock *sk = skb_from_uarg(uarg_to_msgzc(uarg))->sk; atomic_dec(&sk->sk_zckey); - uarg->len--; + uarg_to_msgzc(uarg)->len--; if (have_uref) msg_zerocopy_callback(NULL, uarg, true); diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index d7bd1daf022b..546897a4b4fa 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1043,7 +1043,7 @@ static int __ip_append_data(struct sock *sk, paged = true; zc = true; } else { - uarg->zerocopy = 0; + uarg_to_msgzc(uarg)->zerocopy = 0; skb_zcopy_set(skb, uarg, &extra_uref); } } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 970e9a2cca4a..3152da8f4763 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1239,7 +1239,7 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) } zc = sk->sk_route_caps & NETIF_F_SG; if (!zc) - uarg->zerocopy = 0; + uarg_to_msgzc(uarg)->zerocopy = 0; } } diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 897ca4f9b791..6d4f01a0cf6e 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1568,7 +1568,7 @@ static int __ip6_append_data(struct sock *sk, paged = true; zc = true; } else { - uarg->zerocopy = 0; + uarg_to_msgzc(uarg)->zerocopy = 0; skb_zcopy_set(skb, uarg, &extra_uref); } } From patchwork Wed Aug 10 15:49:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82342C19F2A for ; Wed, 10 Aug 2022 15:52:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232432AbiHJPwA (ORCPT ); Wed, 10 Aug 2022 11:52:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232209AbiHJPvz (ORCPT ); Wed, 10 Aug 2022 11:51:55 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BAADD51407; Wed, 10 Aug 2022 08:51:45 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id v131-20020a1cac89000000b003a4bb3f786bso1236841wme.0; Wed, 10 Aug 2022 08:51:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=aYyDEkTXwWBCvoGHlIPqb+/BxSLKuFl8ndVR481kS1w=; b=hZ9U5hOp4VB5FaNjAUnUraKpOCA2Wpyj5HNp6Hhmv23m66TTZRgMWtl/dQOYhFz/E9 36lXEdGbdo0OOYp9ub05DxHU0JI2ixvQfkVCuWQmznlpIUTvzJs+XnGqaJ2CMBu+LqhP BnPg0B38v5clDnOrD+9TKJLttKJu9NRZchU1iQUWn8KW95x2YDQFqqAhiIbkPwedVKlK NGR7PWWBnUvj5HLI2Nv9zCRnQk5F7LOPjdp0zI7zbkzXr3sK9gOJiNthyiUh1Y08msvy 8rfhtGaZQIdqI/4wm9WgW+yQdrbS+/dsw22V6yBOrwzgjM4RRxcF9hYdLnEirsOA1u+p JGmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=aYyDEkTXwWBCvoGHlIPqb+/BxSLKuFl8ndVR481kS1w=; b=QBns0V45GWQUMi1ZmaGEx8Y6fb0wUQ0OWtC7jbFynvW4jijhJwIiojXybTQHMTbvJK BEzqN8FFKpC4Zfjqz8wcCGKajYk7Ye9/6V9jdGn98kqVHmufM7CMgI26vOknJf6KB3ZK 4tLSC/7sesifMWuy7M+UyFrTezQtZhXaKX6YeNDNzP4VnvVNdZgQlhRfZjGmx0WtF0gU Ex7jzEVmbN+TOZRyPwW4yXqDv//z09OMLheSUvqoNq26wOTK6Tp91Hc0obOYFeokMsO9 dvoInxYi6iVPONmhCuJGzdc/ThVCLapPdij2h0ovPS3/0R8aEIMdfMYDYWXDg7lVrtjt Y4lA== X-Gm-Message-State: ACgBeo0JDZ4ZzFuD3rcqQhJjshQuRlf0NKqWv+xJMhfVC4rGTxKlIMil oVfNns6OzFdI6cGLNNKm/a/sylMtxDY= X-Google-Smtp-Source: AA6agR4pIC1FSo3wRp5p1MAJqhHvgQE9nDnk6Uk00rTTOcb22zy1nLrULEj+sKAA+/DezVNGQLcv4Q== X-Received: by 2002:a05:600c:1e8d:b0:3a5:74d:c61c with SMTP id be13-20020a05600c1e8d00b003a5074dc61cmr2953296wmb.70.1660146703329; Wed, 10 Aug 2022 08:51:43 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:42 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 05/11] net: rename ubuf_info's flags Date: Wed, 10 Aug 2022 16:49:13 +0100 Message-Id: <7d43d8dd1c7e7c9964d40202482f325ecd06642b.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org ubuf_info::flags contains SKBFL_* flags that we copy into skbs, change the field name to stress that it keeps skb flags. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 4 ++-- io_uring/notif.c | 2 +- net/core/skbuff.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index afd7400d7f62..e749b5d3868d 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -534,7 +534,7 @@ struct ubuf_info { void (*callback)(struct sk_buff *, struct ubuf_info *, bool zerocopy_success); refcount_t refcnt; - u8 flags; + u8 skb_flags; }; struct ubuf_info_msgzc { @@ -1664,7 +1664,7 @@ static inline void net_zcopy_get(struct ubuf_info *uarg) static inline void skb_zcopy_init(struct sk_buff *skb, struct ubuf_info *uarg) { skb_shinfo(skb)->destructor_arg = uarg; - skb_shinfo(skb)->flags |= uarg->flags; + skb_shinfo(skb)->flags |= uarg->skb_flags; } static inline void skb_zcopy_set(struct sk_buff *skb, struct ubuf_info *uarg, diff --git a/io_uring/notif.c b/io_uring/notif.c index b5f989dff9de..97cb4a7e8849 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -65,7 +65,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd = io_notif_to_data(notif); nd->account_pages = 0; - nd->uarg.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; nd->uarg.callback = io_uring_tx_zerocopy_callback; /* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set(&nd->uarg.refcnt, 1); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b047a773acd7..40bb84986800 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1206,7 +1206,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) uarg->len = 1; uarg->bytelen = size; uarg->zerocopy = 1; - uarg->ubuf.flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + uarg->ubuf.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; refcount_set(&uarg->ubuf.refcnt, 1); sock_hold(sk); From patchwork Wed Aug 10 15:49:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 677D1C25B07 for ; Wed, 10 Aug 2022 15:52:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232331AbiHJPw2 (ORCPT ); Wed, 10 Aug 2022 11:52:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232171AbiHJPv4 (ORCPT ); Wed, 10 Aug 2022 11:51:56 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F251552DF0; Wed, 10 Aug 2022 08:51:46 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id q1-20020a05600c040100b003a52db97fffso1192915wmb.4; Wed, 10 Aug 2022 08:51:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=I6HOvD/OHCC6U7VTSloH8Bzb7BhYYX9aNTArIgD6mXg=; b=CvZ+o5GyX9Cl58rR5XQmVQ/WnA/GRIRdZxP33Dzgj6r9DPlQRphs6yOhzWqdULKen9 gMmXHh/unAvJbQnQA9hJTCa2uvfKeDglP05vwYf7ACs1xU/sY79GVh0ja78nBVBuQos9 At2FAi3HtK9PCPmiPaAo3PGgSrGuJe4zvs8ZzOggBwh7U2mW5q2DaPPvVs8/T9wnfdXz 94FcpAUm2zJ0X95SveH+CL7kQAQJnVwYQOQrhReSHaP1tz8rIPU18lNRtSV+dGg7djft 87OokEhFNIW6wyCD9lMh9IYzKCch26fi5GrPIxaxnKIQEuRdYWpJJ4Lz0KAThc69LX2g ITgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=I6HOvD/OHCC6U7VTSloH8Bzb7BhYYX9aNTArIgD6mXg=; b=JPNyq2yj6MpV1IiFffvq9BmCGwVRPHU7LAqfCAsfawNsW/wD0yH5eiYhp0wpqurYGg CwK1a3vk4awqM/tfP4oRtwUGnkC504bC5Tgy6BfesMebMKJ9UlIZ6KUyz1RsH4nu3Nb7 P0t90kIZoE5LrnQw2jpXhOyBvhBC9nDzBoz+Xp39RQTdAtwug0r89aApV6y/+VuIps5t ++NIlvDRlGn4fKnp/suZBSJYr84uRzwbrKOwSuIBGmK3X6lL9BoRno0ViwUeKFRWrhxi s8qtnq0OLV3tCjcPJhY7+xBbEhO0auz2Z+MiU9o1IJmWIakpA1GZG2QUybqOXkBTy+zk eQiQ== X-Gm-Message-State: ACgBeo2/r5lwgiNUro3kkCV8765Dgm0nl8ZBna97gcP+fy3+FNFP4rfC Ym8YynHN3wXXODR0ZSUP36HavzYC/DE= X-Google-Smtp-Source: AA6agR45he4dyQKI12poNM8AGSPq8F4/kGUNUT0s11gtoLe3WPA20HuKlateIe0Lr4uUnp3aBCdobA== X-Received: by 2002:a05:600c:1e05:b0:3a5:b441:e9c with SMTP id ay5-20020a05600c1e0500b003a5b4410e9cmr2221515wmb.24.1660146704771; Wed, 10 Aug 2022 08:51:44 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:44 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 06/11] net: add flags for controlling ubuf_info Date: Wed, 10 Aug 2022 16:49:14 +0100 Message-Id: <433971d77b5a757b11ce5683ef1d0377efcc8544.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org There are already skb_flags in ubuf_info, which enhancing skbs. Also add flags controlling ubuf_info, mainly to hint about various referencing aspects of it, which will be introduced in later patches. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 1 + io_uring/notif.c | 1 + net/core/skbuff.c | 1 + 3 files changed, 3 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index e749b5d3868d..2b2e0020030b 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -535,6 +535,7 @@ struct ubuf_info { bool zerocopy_success); refcount_t refcnt; u8 skb_flags; + u8 flags; }; struct ubuf_info_msgzc { diff --git a/io_uring/notif.c b/io_uring/notif.c index 97cb4a7e8849..a2ba1e35a59f 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -66,6 +66,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd = io_notif_to_data(notif); nd->account_pages = 0; nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + nd->uarg.flags = 0; nd->uarg.callback = io_uring_tx_zerocopy_callback; /* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set(&nd->uarg.refcnt, 1); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 40bb84986800..7e102373482c 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1207,6 +1207,7 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) uarg->bytelen = size; uarg->zerocopy = 1; uarg->ubuf.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; + uarg->ubuf.flags = 0; refcount_set(&uarg->ubuf.refcnt, 1); sock_hold(sk); From patchwork Wed Aug 10 15:49:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C384C00140 for ; Wed, 10 Aug 2022 15:52:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232316AbiHJPw1 (ORCPT ); Wed, 10 Aug 2022 11:52:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232330AbiHJPv4 (ORCPT ); Wed, 10 Aug 2022 11:51:56 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CA7F65A889; Wed, 10 Aug 2022 08:51:48 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id z16so18166031wrh.12; Wed, 10 Aug 2022 08:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=h9HHmdtmbDaBgAkYhJr0MaG2LsaIMYK3/tGmura3Q/c=; b=RkQDnrNrgozMvW1slfCx3exnfLZSPNL0SE3J0CmXcn4NPQmzKH1nUhVoCi4rJ/bGhQ qWkiTJzALDavZu1xxA3ht8UtZU4n6tL4iHm1Zx/Em0r6K4Qe5KsTNb1lHxoLPs9CxUrL ZNGihmk1LOvOohQDQ1AY3Ulw8RiLhtDxqyohg/rvq0SO7NQvrO7saW7gPAaH7++iBvJP Jl1azWCXpOvZRwom0TDdfqiZB33KjKODuKkdpzGyNypg8/uceMD9cO1ihA+jI9r5ZcC3 TYYoAUSpD/Fkw5BOO1iG7WSOK8lPFUPa6a5TlPYgWBB6P/+e3jyrNxZORTfBjtmz0zUS rYzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=h9HHmdtmbDaBgAkYhJr0MaG2LsaIMYK3/tGmura3Q/c=; b=0LfNvTPH2qZ+KX9ACd8KRmyHu9J1ET1iWSsVtX1cN61HxUyjR9FbT5JSOkpa1bPx2S sxCPDRDRDlQ4GGmJlrsXs+rbQTmjkqNDx40vzRJWxxt0+U1d5ZM3anwNdN73QdW3vQXR /nJS+y5M90Vw9bgkkLhL4T/KqOHvHrdlyqg/3x7l12O73dJ8mJj7w2TvLQOxOv3HFb5K mQ9Han2qcLIC39zUZP5yuMvUvYRJANsZvKT3kfmR5PcV3EIPERASNER8bR9jDJtcGqEf agid8xSFJHCyTZeQnYhEH32LOiEaSRrS0AJzxMspbVInrC3luObq0+0wJDnVV4NwXkLJ HjZQ== X-Gm-Message-State: ACgBeo0BANQt2Qbz9Ye2kVCVr/kJlX1xk75O+ccuY2UjekERRsrqBDKA ig4nRVo23PzU0PDWahba4+LyHuL69N8= X-Google-Smtp-Source: AA6agR6ZNn9gZ3GD3qgxjdOVbL8JXumKhuxwUdsZoAGuCONXxJ2F64wM4uAcQJPM/4Jm+GN1m/z5nQ== X-Received: by 2002:a5d:634d:0:b0:220:5ff7:3969 with SMTP id b13-20020a5d634d000000b002205ff73969mr16857681wrw.709.1660146706405; Wed, 10 Aug 2022 08:51:46 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:46 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 07/11] net/tcp: optimise tcp ubuf refcounting Date: Wed, 10 Aug 2022 16:49:15 +0100 Message-Id: <9a42187cdc9ce034fd23179c7b31d7cc6a54bd45.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add UARGFL_CALLER_PINNED letting protocols know that the caller holds a reference to the ubuf_info and so it doesn't need additional refcounting for purposes of keeping it alive. With that TCP can save a refcount put/get pair per send when used with ->msg_ubuf. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 7 +++++++ net/ipv4/tcp.c | 9 ++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 2b2e0020030b..45fe7f0648d0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -522,6 +522,13 @@ enum { #define SKBFL_ALL_ZEROCOPY (SKBFL_ZEROCOPY_FRAG | SKBFL_PURE_ZEROCOPY | \ SKBFL_DONT_ORPHAN | SKBFL_MANAGED_FRAG_REFS) +enum { + /* The caller holds a reference during the submission so the ubuf won't + * be freed until we return. + */ + UARGFL_CALLER_PINNED = BIT(0), +}; + /* * The callback notifies userspace to release buffers when skb DMA is done in * lower device, the skb last reference should be 0 when calling this. diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index 3152da8f4763..4925107de57d 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1229,7 +1229,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (msg->msg_ubuf) { uarg = msg->msg_ubuf; - net_zcopy_get(uarg); + if (!(uarg->flags & UARGFL_CALLER_PINNED)) + net_zcopy_get(uarg); zc = sk->sk_route_caps & NETIF_F_SG; } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, size, skb_zcopy(skb)); @@ -1455,7 +1456,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) tcp_push(sk, flags, mss_now, tp->nonagle, size_goal); } out_nopush: - net_zcopy_put(uarg); + if (uarg && !(uarg->flags & UARGFL_CALLER_PINNED)) + net_zcopy_put(uarg); return copied + copied_syn; do_error: @@ -1464,7 +1466,8 @@ int tcp_sendmsg_locked(struct sock *sk, struct msghdr *msg, size_t size) if (copied + copied_syn) goto out; out_err: - net_zcopy_put_abort(uarg, true); + if (uarg && !(uarg->flags & UARGFL_CALLER_PINNED)) + net_zcopy_put_abort(uarg, true); err = sk_stream_error(sk, flags, err); /* make sure we wake any epoll edge trigger waiter */ if (unlikely(tcp_rtx_and_write_queues_empty(sk) && err == -EAGAIN)) { From patchwork Wed Aug 10 15:49:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92318C25B0D for ; Wed, 10 Aug 2022 15:52:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232209AbiHJPw4 (ORCPT ); Wed, 10 Aug 2022 11:52:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232597AbiHJPwJ (ORCPT ); Wed, 10 Aug 2022 11:52:09 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D5AF85C9E7; Wed, 10 Aug 2022 08:51:49 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id i128-20020a1c3b86000000b003a536d58f73so1212098wma.4; Wed, 10 Aug 2022 08:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=itYGv2JZK8WBFRrJm44D9H1rA7HcrDCnR5GlKfK0jhs=; b=g4eoVr1fu36ewbAmfG/t6UJBFOF1oHPXDiKrqdGSPIInTe4GhXxac8aY/xl6qj4Ry6 hjJIrOjXgd6Yh8flP1MMruBaz19LaJGjliJaVdqyUgKhS5Iv3yZ5EXnaluD0BcKphXnA a5VMobfClu6mDyYL8AgAxvtUKvP55/JPpMBJHMu3wEu4WaWTzPCZhMYmHWz8b0MVDXhx GJ1Mj+/b/tEIWLH+pz8xBbXVEHEgTx42XMs5iv3QJxf8DSyr0HdmFD+Q1l3bx4FME1BL RC5BlkaAdauVJCjSBnJ19eyaQPimR9ZGZKGI4e8RTaSx6XcGE3jjRf3+QIMjbpvGpNmL Nr8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=itYGv2JZK8WBFRrJm44D9H1rA7HcrDCnR5GlKfK0jhs=; b=SAstj7p49yuUTPY70/jB0k7C8YHoEWu1Mg2ea6ACYeDa3nXmArMRkeNQEuA3k4rGKV MdyCAXyDk+xVpkzYYMc5Ou1YnRpQH54kjkXVqcJMyvYwsYkEs4NdLE0hTl+qjLlft1Fw pJ4BkeKnDnuBUsFgCKgN83Pd/+PsTK03GtVlSdpg/zkxGsGliZvxslRUdbMv1y04rM1q G/M7qhOLLQvkIbLO7aykmsjWWQ0fkggkWaqUDil61VK03B2+jrIQOe23VQ+9kzC1qkmC dWqEX1eGkFtAc/JBAJdFZNo/7JiAQM+kTamNbOnRL3xdtT5LJxTPFs545uiy/0i9fqGM /kpA== X-Gm-Message-State: ACgBeo2QsgzYzYGh799JV0RxiVAuM3yZwpDvLmSeiDlv9wDsTt3aMybs l8v/dMgQ22BolbXvyXHzgdW/+1gX/1U= X-Google-Smtp-Source: AA6agR5tWoJNmIKULSv6qED4kYicFbD6aj95a2kTMgyfbOltADMEJMZMCY4KEhAPbmXEqnmelWqLpA== X-Received: by 2002:a05:600c:6009:b0:3a5:b069:5d34 with SMTP id az9-20020a05600c600900b003a5b0695d34mr2478342wmb.115.1660146707842; Wed, 10 Aug 2022 08:51:47 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:47 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 08/11] net: let callers provide ->msg_ubuf refs Date: Wed, 10 Aug 2022 16:49:16 +0100 Message-Id: <526fe4cb9cda287bedfc92b3888b48a4f3b0250b.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Some msg_ubuf providers like io_uring can keep elaborated ubuf_info reference batching and caching, so it will be of benefit to let the network layer to optionally steal some of the cached refs. Add UARGFL_GIFT_REF, if set the caller has at least one extra reference that it can gift away. If the network decides to take the ref it should clear the flag. Signed-off-by: Pavel Begunkov --- include/linux/skbuff.h | 14 ++++++++++++++ net/ipv4/ip_output.c | 1 + net/ipv6/ip6_output.c | 1 + 3 files changed, 16 insertions(+) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 45fe7f0648d0..972ec676e222 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -527,6 +527,11 @@ enum { * be freed until we return. */ UARGFL_CALLER_PINNED = BIT(0), + + /* The caller can gift one ubuf reference. The flag should be cleared + * when the reference is taken. + */ + UARGFL_GIFT_REF = BIT(1), }; /* @@ -1709,6 +1714,15 @@ static inline void net_zcopy_put(struct ubuf_info *uarg) uarg->callback(NULL, uarg, true); } +static inline bool net_zcopy_get_gift_ref(struct ubuf_info *uarg) +{ + bool has_ref; + + has_ref = uarg->flags & UARGFL_GIFT_REF; + uarg->flags &= ~UARGFL_GIFT_REF; + return has_ref; +} + static inline void net_zcopy_put_abort(struct ubuf_info *uarg, bool have_uref) { if (uarg) { diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c index 546897a4b4fa..9d42b6dd6b78 100644 --- a/net/ipv4/ip_output.c +++ b/net/ipv4/ip_output.c @@ -1032,6 +1032,7 @@ static int __ip_append_data(struct sock *sk, paged = true; zc = true; uarg = msg->msg_ubuf; + extra_uref = net_zcopy_get_gift_ref(uarg); } } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 6d4f01a0cf6e..8d8a8bbdb8df 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -1557,6 +1557,7 @@ static int __ip6_append_data(struct sock *sk, paged = true; zc = true; uarg = msg->msg_ubuf; + extra_uref = net_zcopy_get_gift_ref(uarg); } } else if (sock_flag(sk, SOCK_ZEROCOPY)) { uarg = msg_zerocopy_realloc(sk, length, skb_zcopy(skb)); From patchwork Wed Aug 10 15:49:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51D2FC00140 for ; Wed, 10 Aug 2022 15:53:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232518AbiHJPx2 (ORCPT ); Wed, 10 Aug 2022 11:53:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232726AbiHJPwK (ORCPT ); Wed, 10 Aug 2022 11:52:10 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59ED360686; Wed, 10 Aug 2022 08:51:51 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id s11-20020a1cf20b000000b003a52a0945e8so1194722wmc.1; Wed, 10 Aug 2022 08:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=P8aAKAj6QbUGM3qiLbxj6rCIuSDWisCzVUCts73nG7k=; b=iw4zMeJIq6Iduj5Ai23kLZLWv6lH2honhKvhFD/8RjX03J6CC6OT3GFZdBOkedBUdt n+4sEAA+jY/L/94w0veZ6+BigRGb/z+iiaiJDK2kbDqMnLkYpIKAVeEpBGeRkUhR8ZY4 3C2zmmMBUj+haew1vR4ZvQFcsNz2n5JD1clVwf0uHdPmnBF3QMz0ffFPSnpU76csdYvx WghdcQ+nasGK3chHwCDoPTmwBdCerSaeh8eCSRiiRqlv5NxV5e1DoaVA1cZ7GwqNoWTR Iu2g0zXQ4q7bhvEbKDfbxuVUgmKMV7FWlYgxIhokbOIyG0ObnwT0cf/lwdYDKKmbRxeE YrCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=P8aAKAj6QbUGM3qiLbxj6rCIuSDWisCzVUCts73nG7k=; b=QvOBTC59h5AidkNvLgHwm0uXicB5kDWjDfifkrQuGZHvcHAlxYa2pVqEozHcvcWVK0 5RKQFESxiGqP+5q7eJiwG/7szw9K0/Jn2ILHpUr20OLHFzPRZkrP75HkwGcUS8FWyscU NlE2SMxG+sfdtxoX7PDYKIqWw8hQUOMHn6V74tUxrUNpOQ5wLnvbrBiQavxHvd+lUzpn OKmfMkjNE8HytOwBtssE/8E7vUmxEoWUgLJ8kmAnxcbhvL0P8HD2JLdLlloLh6hieDQx qv2QyY/IUH7+vCvPbPBLATdEvLJIatDKo1oW0xbvWOyjVypt8hYouWtqG/F0u2TOEN7m g9jA== X-Gm-Message-State: ACgBeo2P6UPpAZNYtaZCVFybX8ftDCLXHDDVig6uA3pgtQhA08nDNhiG xs6e9o9BeG6NjqNLyxc3TeynOjLSSGM= X-Google-Smtp-Source: AA6agR51LI/17gzAJrVldihDliUyILNuNHqftq15rGqHxVJN5cbfccCeChTUV2Xc8UnWpC6KUoJk0Q== X-Received: by 2002:a1c:f209:0:b0:3a4:f42c:9ffb with SMTP id s9-20020a1cf209000000b003a4f42c9ffbmr2964735wmc.62.1660146709173; Wed, 10 Aug 2022 08:51:49 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:48 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 09/11] io_uring/notif: add helper for flushing refs Date: Wed, 10 Aug 2022 16:49:17 +0100 Message-Id: X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Add a helper for dropping notification references during flush. It's a preparation patch, currently it's only one master ref, but we're going to add ref caching. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index a2ba1e35a59f..5661681b3b44 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -73,6 +73,13 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, return notif; } +static inline bool io_notif_drop_refs(struct io_notif_data *nd) +{ + int refs = 1; + + return refcount_sub_and_test(refs, &nd->uarg.refcnt); +} + void io_notif_slot_flush(struct io_notif_slot *slot) __must_hold(&ctx->uring_lock) { @@ -81,8 +88,7 @@ void io_notif_slot_flush(struct io_notif_slot *slot) slot->notif = NULL; - /* drop slot's master ref */ - if (refcount_dec_and_test(&nd->uarg.refcnt)) + if (io_notif_drop_refs(nd)) io_notif_complete(notif); } @@ -97,13 +103,11 @@ __cold int io_notif_unregister(struct io_ring_ctx *ctx) for (i = 0; i < ctx->nr_notif_slots; i++) { struct io_notif_slot *slot = &ctx->notif_slots[i]; struct io_kiocb *notif = slot->notif; - struct io_notif_data *nd; if (!notif) continue; - nd = io_kiocb_to_cmd(notif); slot->notif = NULL; - if (!refcount_dec_and_test(&nd->uarg.refcnt)) + if (!io_notif_drop_refs(io_kiocb_to_cmd(notif))) continue; notif->io_task_work.func = __io_notif_complete_tw; io_req_task_work_add(notif); From patchwork Wed Aug 10 15:49:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 765CBC00140 for ; Wed, 10 Aug 2022 15:52:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231928AbiHJPwZ (ORCPT ); Wed, 10 Aug 2022 11:52:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232545AbiHJPwJ (ORCPT ); Wed, 10 Aug 2022 11:52:09 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C81D765541; Wed, 10 Aug 2022 08:51:52 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id c187-20020a1c35c4000000b003a30d88fe8eso1227700wma.2; Wed, 10 Aug 2022 08:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=KeuLRkJHac7T18EvJz7V2qjxDYwlB1JNeLqOJx59kSU=; b=NwaYFZmlXQkYLPFX3ZTPko8HcLgEHLwlrcyFvHq9hm7vQV2IwxrttH0/p8/BnnmZQl BTO0b3I01yD4gVwVuca1h/jbv+Cfysmr/pwkJd2AgCZmA4nNJaqDDMtHbPuEcJweTVo2 579S6P3KHIkfq4qhjgykgSQx3GEN7jkMGKOWCq1fqwahARcWVFoKAElwF2/ur8B6ClIo 4VUYL64JbAwxyO16ryqLgQ35TM4VOjpGgJE30dWaAoTd8zwN9ejtzD9yLFz3cxaGQo9r I+xFxR/ul8X0T8pEmed+ACyAEzgpzPI6KoqWeaF80317jz0w1JJi3GWEirlZLAj1Zjjz DcIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=KeuLRkJHac7T18EvJz7V2qjxDYwlB1JNeLqOJx59kSU=; b=RZ8EOPfH4zHrwmQxojqmbzpzfAX88SLw2BiN77K8tAA0hFNufyQsKeQHIt8NHAefZB 8U9nhdE+zB/lVRky3cASaN+fF2+PYfJOaOCJzEZ7G8MhBHX+vVPC/qrFg4sP5FumvUwQ +3tyDHiBfWVlzId+XeUlBtgnEQHXe9xD6aEjIzJcGSJVeGfGrdoltei9wRB7Y0Nnjx/f itK+XycLz9yp2tW9DoXq0tk8wQ9CqJWsl8IEVM4EZ+QdS5YG655EhqdQBDiudKV9iBHq E0RSRzphbWSCV/Tc9F9pwZH1pYbuBlwN7nYk8flxk5DM/A9cVITM5sxWZ68DUort9mmR uzhQ== X-Gm-Message-State: ACgBeo3hAES7nOXwi4NiBTWzmqN7h7vBETrXLSBrS60UjJI1+OiIK5KV /BCIvYY+2SoEZkSviKhmjVobRdZr4i4= X-Google-Smtp-Source: AA6agR6Ld0dgt8VkL4NPznZl6LE5uIBV6RhDIhjeXAmILYOIjoMjQERHnQMZ5pvfGHOhFOpmDqlmgQ== X-Received: by 2002:a7b:c852:0:b0:3a5:407a:76df with SMTP id c18-20020a7bc852000000b003a5407a76dfmr2851591wml.101.1660146710481; Wed, 10 Aug 2022 08:51:50 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:50 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 10/11] io_uring/notif: mark notifs with UARGFL_CALLER_PINNED Date: Wed, 10 Aug 2022 16:49:18 +0100 Message-Id: <19cad183acc3a44b17b76c1719ad30c80aeff1ef.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org We always keep references to active notifications and drop them only when we flush, so they're always pinned during sock_sendmsg() and we can add UARGFL_CALLER_PINNED. Signed-off-by: Pavel Begunkov --- io_uring/notif.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/io_uring/notif.c b/io_uring/notif.c index 5661681b3b44..dd346ea67580 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -66,7 +66,7 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd = io_notif_to_data(notif); nd->account_pages = 0; nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; - nd->uarg.flags = 0; + nd->uarg.flags = UARGFL_CALLER_PINNED; nd->uarg.callback = io_uring_tx_zerocopy_callback; /* master ref owned by io_notif_slot, will be dropped on flush */ refcount_set(&nd->uarg.refcnt, 1); From patchwork Wed Aug 10 15:49:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 12940724 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73DFAC25B07 for ; Wed, 10 Aug 2022 15:53:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232755AbiHJPx3 (ORCPT ); Wed, 10 Aug 2022 11:53:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232846AbiHJPwM (ORCPT ); Wed, 10 Aug 2022 11:52:12 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D94C96B664; Wed, 10 Aug 2022 08:51:53 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id s11-20020a1cf20b000000b003a52a0945e8so1194784wmc.1; Wed, 10 Aug 2022 08:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=8bvrycIBhC98AAS95EkRU1aUeIYv5cmtQ6g1fcH8Ls4=; b=NXRHpYfu0yj454B9TNqMSmsanOBuZYbEWiz+69QOiDC+DiJ2G+DWR8eQVlY70gQLu2 d0y7fh2jjsSTdciIvCwRVN4S9hece9UJOGa8XGh2v8vTiQdh6mNLdQQ8Fav5YfXJjcUU khtZG1kVvwRuto8hvHjfgINz8qFheZKolPC7tRow4X5InG3ks03RinhMEB69q/HPIghp RR52RfGpkiacBVtnyrW+hJzjBXI1YrXWzEp3tbUt8VcNjh69N5Ru/lSwr+rsyQJoHgS+ 05TXvir/g7FILT8dTPmjkGDy/QS+ouQWtNodRMedLwj3LN6GVEXfdRJ6j4sUIlZrGHUR ilkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=8bvrycIBhC98AAS95EkRU1aUeIYv5cmtQ6g1fcH8Ls4=; b=rswghZnQ8v3r55AeXW+qUVezH8KK9xYl4oBb1lBQJ8v0ARPYXK82lMP50ozbzN2XE5 +twmQfsX1n2gV1+TdnhXy93q2+CaCtnOKNvUYps0vadIy4FdhUgKqgK9a+K20PiPkuLA fWPzSn1/WhOI5kFsie4VSwn7wozR1sWFoz5zPgt3nKwc11mchLMPv+TiOWmf0RDgm7ez amPkRz4HUmvSkDbLnnVkZxiQNtBf41E+zxzwc7SfHqdwwCzrjeul4CBYClxpzOlW5FQe gKDYXI81RR8chsd/Y6x3CRjf8RSxyLB4kkZ4WeKMmyb24YmDL+F6nWRh1rOPK8hvhyOB GT5A== X-Gm-Message-State: ACgBeo3aaVrtdPUcnhVnj/DNQT+k8ELoo9o3tTyaH6Gw87YTm1VnXeOh NpMFzr5lNGCK4dThRlvtstWcIaeIq7M= X-Google-Smtp-Source: AA6agR6ysliwRQwjHrbtgRjuvt8Ybcqj6h9fHpK3wukZ1ZPxN88XDbQ1aaLBOWU/0aq+IGK8xrnzTA== X-Received: by 2002:a05:600c:4ecd:b0:3a3:3eb0:9101 with SMTP id g13-20020a05600c4ecd00b003a33eb09101mr3007084wmq.49.1660146711936; Wed, 10 Aug 2022 08:51:51 -0700 (PDT) Received: from 127.0.0.1localhost (188.28.126.24.threembb.co.uk. [188.28.126.24]) by smtp.gmail.com with ESMTPSA id ay1-20020a05600c1e0100b003a342933727sm3004519wmb.3.2022.08.10.08.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Aug 2022 08:51:51 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org, netdev@vger.kernel.org Cc: Jens Axboe , "David S . Miller" , Jakub Kicinski , kernel-team@fb.com, linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Wei Liu , Paul Durrant , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, "Michael S . Tsirkin" , Jason Wang , Pavel Begunkov Subject: [RFC net-next io_uring 11/11] io_uring/notif: add ubuf_info ref caching Date: Wed, 10 Aug 2022 16:49:19 +0100 Message-Id: <9514ddeea83902c36887ce764bc0306a518af73c.1660124059.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: io-uring@vger.kernel.org Cache some active notifier references at the io_uring side and get them in batches, so the ammortised cost is low. Then these references can be given away to the network layer using UARGFL_GIFT_REF. Signed-off-by: Pavel Begunkov --- io_uring/net.c | 8 +++++++- io_uring/notif.c | 6 ++++-- io_uring/notif.h | 22 +++++++++++++++++++++- 3 files changed, 32 insertions(+), 4 deletions(-) diff --git a/io_uring/net.c b/io_uring/net.c index e6fc9748fbd2..bdaf9b10bd1b 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -949,6 +949,7 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) struct io_sendzc *zc = io_kiocb_to_cmd(req); struct io_notif_slot *notif_slot; struct io_kiocb *notif; + struct ubuf_info *ubuf; struct msghdr msg; struct iovec iov; struct socket *sock; @@ -1007,10 +1008,15 @@ int io_sendzc(struct io_kiocb *req, unsigned int issue_flags) min_ret = iov_iter_count(&msg.msg_iter); msg.msg_flags = msg_flags; - msg.msg_ubuf = &io_notif_to_data(notif)->uarg; msg.sg_from_iter = io_sg_from_iter; + msg.msg_ubuf = ubuf = &io_notif_to_data(notif)->uarg; + ubuf->flags |= UARGFL_GIFT_REF; ret = sock_sendmsg(sock, &msg); + /* check if the send consumed an additional ref */ + if (likely(!(ubuf->flags & UARGFL_GIFT_REF))) + io_notif_consume_ref(notif); + if (unlikely(ret < min_ret)) { if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK)) return -EAGAIN; diff --git a/io_uring/notif.c b/io_uring/notif.c index dd346ea67580..73bbda5de07d 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -68,15 +68,17 @@ struct io_kiocb *io_alloc_notif(struct io_ring_ctx *ctx, nd->uarg.skb_flags = SKBFL_ZEROCOPY_FRAG | SKBFL_DONT_ORPHAN; nd->uarg.flags = UARGFL_CALLER_PINNED; nd->uarg.callback = io_uring_tx_zerocopy_callback; + nd->cached_refs = IO_NOTIF_REF_CACHE_NR; /* master ref owned by io_notif_slot, will be dropped on flush */ - refcount_set(&nd->uarg.refcnt, 1); + refcount_set(&nd->uarg.refcnt, IO_NOTIF_REF_CACHE_NR + 1); return notif; } static inline bool io_notif_drop_refs(struct io_notif_data *nd) { - int refs = 1; + int refs = nd->cached_refs + 1; + nd->cached_refs = 0; return refcount_sub_and_test(refs, &nd->uarg.refcnt); } diff --git a/io_uring/notif.h b/io_uring/notif.h index 0819304d7e00..2a263055a53b 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -9,11 +9,14 @@ #define IO_NOTIF_SPLICE_BATCH 32 #define IORING_MAX_NOTIF_SLOTS (1U << 10) +#define IO_NOTIF_REF_CACHE_NR 64 struct io_notif_data { struct file *file; - struct ubuf_info uarg; unsigned long account_pages; + /* extra uarg->refcnt refs */ + int cached_refs; + struct ubuf_info uarg; }; struct io_notif_slot { @@ -88,3 +91,20 @@ static inline int io_notif_account_mem(struct io_kiocb *notif, unsigned len) } return 0; } + +static inline void io_notif_consume_ref(struct io_kiocb *notif) + __must_hold(&ctx->uring_lock) +{ + struct io_notif_data *nd = io_notif_to_data(notif); + + nd->cached_refs--; + + /* + * Issue sends without looking at notif->cached_refs first, so we + * always have to have at least one ref cached + */ + if (unlikely(!nd->cached_refs)) { + refcount_add(IO_NOTIF_REF_CACHE_NR, &nd->uarg.refcnt); + nd->cached_refs += IO_NOTIF_REF_CACHE_NR; + } +}